[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 02:03:40.620293 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:42883,127.19.126.61:45487,127.19.126.60:33683
I20240503 02:03:40.621888 19960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:03:40.622762 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:03:40.640683 19966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:40.642424 19969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:40.643349 19960 server_base.cc:1034] running on GCE node
W20240503 02:03:40.641259 19967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:41.758136 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:41.758355 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:41.758502 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701821758485 us; error 0 us; skew 500 ppm
I20240503 02:03:41.759229 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:41.764835 19960 webserver.cc:460] Webserver started at http://127.19.126.62:42413/ using document root <none> and password file <none>
I20240503 02:03:41.765620 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:41.765820 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:41.766212 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:41.772418 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "298676057a0f41abb61b00b9ba48f18e"
format_stamp: "Formatted at 2024-05-03 02:03:41 on dist-test-slave-8rwr"
I20240503 02:03:41.779902 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20240503 02:03:41.785256 19975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:41.786309 19960 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20240503 02:03:41.786593 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "298676057a0f41abb61b00b9ba48f18e"
format_stamp: "Formatted at 2024-05-03 02:03:41 on dist-test-slave-8rwr"
I20240503 02:03:41.786906 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:41.847712 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:41.849572 19960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:03:41.849970 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:41.924860 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:42883
I20240503 02:03:41.925441 20026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:42883 every 8 connection(s)
I20240503 02:03:41.931092 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:03:41.931876 20027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:03:41.939283 20029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:41.943619 20030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:41.945164 20032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:41.946357 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:41.946952 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:41.947134 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:41.947279 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701821947265 us; error 0 us; skew 500 ppm
I20240503 02:03:41.947778 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:41.951833 19960 webserver.cc:460] Webserver started at http://127.19.126.61:34405/ using document root <none> and password file <none>
I20240503 02:03:41.952286 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:41.952461 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:41.952720 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:41.953788 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-1-root/instance:
uuid: "2e003817faf249309d96c38579008173"
format_stamp: "Formatted at 2024-05-03 02:03:41 on dist-test-slave-8rwr"
I20240503 02:03:41.959236 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240503 02:03:41.956645 20027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:03:41.964977 20037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:41.965910 19960 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240503 02:03:41.966323 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-1-root
uuid: "2e003817faf249309d96c38579008173"
format_stamp: "Formatted at 2024-05-03 02:03:41 on dist-test-slave-8rwr"
I20240503 02:03:41.966812 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:41.984303 20027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:03:41.986848 19979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.126.61:45487: connect: Connection refused (error 111)
W20240503 02:03:41.990757 20027 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.19.126.61:45487: Network error: Client connection negotiation failed: client connection to 127.19.126.61:45487: connect: Connection refused (error 111)
I20240503 02:03:42.006522 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:42.007803 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:42.040238 20027 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } attempt: 1
W20240503 02:03:42.044991 20027 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.19.126.61:45487: Network error: Client connection negotiation failed: client connection to 127.19.126.61:45487: connect: Connection refused (error 111)
I20240503 02:03:42.084475 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.61:45487
I20240503 02:03:42.084627 20091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.61:45487 every 8 connection(s)
I20240503 02:03:42.089130 20092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:42.089825 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:03:42.097025 20092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:03:42.097236 20094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:42.100281 20095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:42.106086 19960 server_base.cc:1034] running on GCE node
W20240503 02:03:42.107256 20097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:42.108086 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:42.108282 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:42.108446 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701822108430 us; error 0 us; skew 500 ppm
I20240503 02:03:42.109040 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:42.113212 19960 webserver.cc:460] Webserver started at http://127.19.126.60:43455/ using document root <none> and password file <none>
I20240503 02:03:42.113485 20027 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } attempt: 2
I20240503 02:03:42.113790 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:42.113996 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:42.114329 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:42.115952 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-2-root/instance:
uuid: "7951382277a446008b68b8c6ceb4a20d"
format_stamp: "Formatted at 2024-05-03 02:03:42 on dist-test-slave-8rwr"
I20240503 02:03:42.117905 20092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:03:42.125026 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.008s
I20240503 02:03:42.128429 20027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:03:42.132580 20105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:42.133682 19960 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.003s
I20240503 02:03:42.134033 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-2-root
uuid: "7951382277a446008b68b8c6ceb4a20d"
format_stamp: "Formatted at 2024-05-03 02:03:42 on dist-test-slave-8rwr"
W20240503 02:03:42.134575 20027 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.19.126.60:33683: Network error: Client connection negotiation failed: client connection to 127.19.126.60:33683: connect: Connection refused (error 111)
I20240503 02:03:42.134400 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:42.136338 20092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:03:42.144469 20092 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.19.126.60:33683: Network error: Client connection negotiation failed: client connection to 127.19.126.60:33683: connect: Connection refused (error 111)
I20240503 02:03:42.166777 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:42.168015 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:42.175923 20092 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } attempt: 1
I20240503 02:03:42.177985 20027 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } attempt: 1
W20240503 02:03:42.180521 20092 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.19.126.60:33683: Network error: Client connection negotiation failed: client connection to 127.19.126.60:33683: connect: Connection refused (error 111)
W20240503 02:03:42.182555 20027 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.19.126.60:33683: Network error: Client connection negotiation failed: client connection to 127.19.126.60:33683: connect: Connection refused (error 111)
I20240503 02:03:42.243230 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.60:33683
I20240503 02:03:42.244112 20157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.60:33683 every 8 connection(s)
I20240503 02:03:42.247946 20158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:42.248006 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:03:42.250030 20027 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } attempt: 2
I20240503 02:03:42.254860 20158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:03:42.255911 20092 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } attempt: 2
I20240503 02:03:42.271770 20158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:03:42.272663 20027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e: Bootstrap starting.
I20240503 02:03:42.279989 20027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:42.281062 20158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:03:42.282187 20092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173: Bootstrap starting.
I20240503 02:03:42.282374 20027 log.cc:824] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e: Log is configured to *not* fsync() on all Append() calls
I20240503 02:03:42.287372 20027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e: No bootstrap required, opened a new log
I20240503 02:03:42.289216 20092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:42.296020 20092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173: No bootstrap required, opened a new log
I20240503 02:03:42.296687 20158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d: Bootstrap starting.
I20240503 02:03:42.301260 20158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:42.306576 20158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d: No bootstrap required, opened a new log
I20240503 02:03:42.312565 20158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } }
I20240503 02:03:42.312688 20027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } }
I20240503 02:03:42.312583 20092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } }
I20240503 02:03:42.313350 20158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:42.313469 20027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:42.313550 20092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:42.313694 20158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7951382277a446008b68b8c6ceb4a20d, State: Initialized, Role: FOLLOWER
I20240503 02:03:42.313836 20027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298676057a0f41abb61b00b9ba48f18e, State: Initialized, Role: FOLLOWER
I20240503 02:03:42.313853 20092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e003817faf249309d96c38579008173, State: Initialized, Role: FOLLOWER
I20240503 02:03:42.314687 20158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } }
I20240503 02:03:42.314694 20027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } }
I20240503 02:03:42.314693 20092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } }
I20240503 02:03:42.320109 20166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } } }
I20240503 02:03:42.322108 20166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:42.322954 20092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:03:42.323146 20158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:03:42.322919 20167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } } }
I20240503 02:03:42.324024 20167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:42.323668 20168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } } }
I20240503 02:03:42.324513 20168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:42.324653 20027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [sys.catalog]: configured and running, proceeding with master startup.
W20240503 02:03:42.355356 19981 tablet.cc:2369] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:03:42.376284 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240503 02:03:42.386097 20199 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:03:42.386482 20199 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:03:42.387395 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 02:03:42.390148 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:03:42.394857 20201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:03:42.394943 20200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:03:42.395210 20201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 02:03:42.395385 20200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 02:03:42.400615 20203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:42.406442 20204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:42.407163 20206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:42.408007 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:42.408739 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:42.408921 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:42.409075 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701822409058 us; error 0 us; skew 500 ppm
I20240503 02:03:42.409610 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:42.412346 19960 webserver.cc:460] Webserver started at http://127.19.126.1:43945/ using document root <none> and password file <none>
I20240503 02:03:42.412797 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:42.412963 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:42.413201 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:42.414381 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-0-root/instance:
uuid: "4a498427fe2d48a089dc9e141f990b76"
format_stamp: "Formatted at 2024-05-03 02:03:42 on dist-test-slave-8rwr"
I20240503 02:03:42.418891 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:03:42.422178 20211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:42.422952 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:03:42.423197 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "4a498427fe2d48a089dc9e141f990b76"
format_stamp: "Formatted at 2024-05-03 02:03:42 on dist-test-slave-8rwr"
I20240503 02:03:42.423462 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:42.434407 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:42.435596 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:42.439353 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:03:42.444619 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:03:42.444901 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:42.445199 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:03:42.445432 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 02:03:42.544806 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:36865
I20240503 02:03:42.545377 20273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:36865 every 8 connection(s)
I20240503 02:03:42.573323 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:03:42.574247 20274 heartbeater.cc:347] Connected to a master server at 127.19.126.60:33683
I20240503 02:03:42.574569 20166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:42.575235 20274 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:42.575188 20166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } }
I20240503 02:03:42.576551 20274 heartbeater.cc:511] Master 127.19.126.60:33683 requested a full tablet report, sending...
I20240503 02:03:42.581224 20166 leader_election.cc:288] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 298676057a0f41abb61b00b9ba48f18e (127.19.126.62:42883), 2e003817faf249309d96c38579008173 (127.19.126.61:45487)
I20240503 02:03:42.579336 20002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7951382277a446008b68b8c6ceb4a20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "298676057a0f41abb61b00b9ba48f18e" is_pre_election: true
I20240503 02:03:42.585343 20002 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7951382277a446008b68b8c6ceb4a20d in term 0.
I20240503 02:03:42.585920 20067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7951382277a446008b68b8c6ceb4a20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e003817faf249309d96c38579008173" is_pre_election: true
I20240503 02:03:42.588594 20067 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7951382277a446008b68b8c6ceb4a20d in term 0.
I20240503 02:03:42.590422 20123 ts_manager.cc:194] Registered new tserver with Master: 4a498427fe2d48a089dc9e141f990b76 (127.19.126.1:36865)
I20240503 02:03:42.590816 20110 leader_election.cc:302] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e003817faf249309d96c38579008173, 7951382277a446008b68b8c6ceb4a20d; no voters:
W20240503 02:03:42.594743 20281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:42.594769 20276 heartbeater.cc:347] Connected to a master server at 127.19.126.62:42883
I20240503 02:03:42.595839 20276 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:42.593685 20166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:03:42.597776 20276 heartbeater.cc:511] Master 127.19.126.62:42883 requested a full tablet report, sending...
I20240503 02:03:42.597160 20166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:03:42.599596 20275 heartbeater.cc:347] Connected to a master server at 127.19.126.61:45487
I20240503 02:03:42.599721 20166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:42.600217 20275 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:42.601770 20275 heartbeater.cc:511] Master 127.19.126.61:45487 requested a full tablet report, sending...
I20240503 02:03:42.603866 19992 ts_manager.cc:194] Registered new tserver with Master: 4a498427fe2d48a089dc9e141f990b76 (127.19.126.1:36865)
I20240503 02:03:42.607354 20057 ts_manager.cc:194] Registered new tserver with Master: 4a498427fe2d48a089dc9e141f990b76 (127.19.126.1:36865)
W20240503 02:03:42.609448 20282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:42.613026 20166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } }
I20240503 02:03:42.616848 20002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7951382277a446008b68b8c6ceb4a20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "298676057a0f41abb61b00b9ba48f18e"
I20240503 02:03:42.617831 20002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:42.618754 20166 leader_election.cc:288] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [CANDIDATE]: Term 1 election: Requested vote from peers 298676057a0f41abb61b00b9ba48f18e (127.19.126.62:42883), 2e003817faf249309d96c38579008173 (127.19.126.61:45487)
I20240503 02:03:42.619941 20067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7951382277a446008b68b8c6ceb4a20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e003817faf249309d96c38579008173"
I20240503 02:03:42.620522 20067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:03:42.625730 20284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:42.627910 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:42.628252 20067 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7951382277a446008b68b8c6ceb4a20d in term 1.
I20240503 02:03:42.627892 20002 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7951382277a446008b68b8c6ceb4a20d in term 1.
I20240503 02:03:42.629266 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:42.629637 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:42.629458 20110 leader_election.cc:302] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e003817faf249309d96c38579008173, 7951382277a446008b68b8c6ceb4a20d; no voters:
I20240503 02:03:42.629921 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701822629898 us; error 0 us; skew 500 ppm
I20240503 02:03:42.630533 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:42.630545 20166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:42.631747 20166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 1 LEADER]: Becoming Leader. State: Replica: 7951382277a446008b68b8c6ceb4a20d, State: Running, Role: LEADER
I20240503 02:03:42.633047 20166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } }
I20240503 02:03:42.641705 19960 webserver.cc:460] Webserver started at http://127.19.126.2:44765/ using document root <none> and password file <none>
I20240503 02:03:42.642150 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:42.642377 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:42.642830 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:42.644403 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-1-root/instance:
uuid: "9ec0bede00584270bb54719c6d50f549"
format_stamp: "Formatted at 2024-05-03 02:03:42 on dist-test-slave-8rwr"
I20240503 02:03:42.647267 20286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7951382277a446008b68b8c6ceb4a20d. Latest consensus state: current_term: 1 leader_uuid: "7951382277a446008b68b8c6ceb4a20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } } }
I20240503 02:03:42.647987 20286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [sys.catalog]: This master's current role is: LEADER
I20240503 02:03:42.651166 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20240503 02:03:42.652536 20290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:03:42.656100 20292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:42.657038 19960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:03:42.657406 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "9ec0bede00584270bb54719c6d50f549"
format_stamp: "Formatted at 2024-05-03 02:03:42 on dist-test-slave-8rwr"
I20240503 02:03:42.657768 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:42.659060 20290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:03:42.675029 20067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [term 1 FOLLOWER]: Refusing update from remote peer 7951382277a446008b68b8c6ceb4a20d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:03:42.676817 20002 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [term 1 FOLLOWER]: Refusing update from remote peer 7951382277a446008b68b8c6ceb4a20d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:03:42.676581 20286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:42.678329 20286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:42.684031 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:42.687225 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:42.705581 20167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7951382277a446008b68b8c6ceb4a20d. Latest consensus state: current_term: 1 leader_uuid: "7951382277a446008b68b8c6ceb4a20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } } }
I20240503 02:03:42.706157 20167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:42.707238 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:03:42.714146 20168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7951382277a446008b68b8c6ceb4a20d. Latest consensus state: current_term: 1 leader_uuid: "7951382277a446008b68b8c6ceb4a20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } } }
I20240503 02:03:42.714929 20168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:42.720822 20298 mvcc.cc:205] Tried to move back new op lower bound from 7023418665661349888 to 7023418665518043136. Current Snapshot: MvccSnapshot[applied={T|T < 7023418665661349888}]
I20240503 02:03:42.723260 20168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7951382277a446008b68b8c6ceb4a20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } } }
I20240503 02:03:42.724170 20168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:42.726435 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:03:42.726728 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:42.727022 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:03:42.727241 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:42.734711 20167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7951382277a446008b68b8c6ceb4a20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } } }
I20240503 02:03:42.735528 20167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:42.743655 20290 catalog_manager.cc:1347] Generated new cluster ID: cd4e65360eaf471eaaaa721b34b8fe91
I20240503 02:03:42.744081 20290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:03:42.745560 20166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7951382277a446008b68b8c6ceb4a20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } } }
I20240503 02:03:42.746323 20166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [sys.catalog]: This master's current role is: LEADER
I20240503 02:03:42.746956 20166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7951382277a446008b68b8c6ceb4a20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298676057a0f41abb61b00b9ba48f18e" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 42883 } } peers { permanent_uuid: "2e003817faf249309d96c38579008173" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 45487 } } peers { permanent_uuid: "7951382277a446008b68b8c6ceb4a20d" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 33683 } } }
I20240503 02:03:42.747645 20166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [sys.catalog]: This master's current role is: LEADER
I20240503 02:03:42.785660 20290 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:03:42.788832 20290 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:03:42.853657 20290 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d: Generated new TSK 0
I20240503 02:03:42.854857 20290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:03:42.905206 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:38243
I20240503 02:03:42.905393 20363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:38243 every 8 connection(s)
I20240503 02:03:42.933207 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:03:42.949478 20365 heartbeater.cc:347] Connected to a master server at 127.19.126.61:45487
I20240503 02:03:42.950440 20365 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:42.956049 20364 heartbeater.cc:347] Connected to a master server at 127.19.126.60:33683
I20240503 02:03:42.956105 20365 heartbeater.cc:511] Master 127.19.126.61:45487 requested a full tablet report, sending...
I20240503 02:03:42.956539 20364 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:42.956936 20367 heartbeater.cc:347] Connected to a master server at 127.19.126.62:42883
I20240503 02:03:42.957453 20367 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:42.958154 20364 heartbeater.cc:511] Master 127.19.126.60:33683 requested a full tablet report, sending...
I20240503 02:03:42.958972 20367 heartbeater.cc:511] Master 127.19.126.62:42883 requested a full tablet report, sending...
I20240503 02:03:42.960335 20057 ts_manager.cc:194] Registered new tserver with Master: 9ec0bede00584270bb54719c6d50f549 (127.19.126.2:38243)
I20240503 02:03:42.960593 20123 ts_manager.cc:194] Registered new tserver with Master: 9ec0bede00584270bb54719c6d50f549 (127.19.126.2:38243)
I20240503 02:03:42.963618 20123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54862
W20240503 02:03:42.966392 20371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:42.967792 19992 ts_manager.cc:194] Registered new tserver with Master: 9ec0bede00584270bb54719c6d50f549 (127.19.126.2:38243)
W20240503 02:03:42.970328 20372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:42.972335 20374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:42.973449 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:42.974267 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:42.974464 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:42.974630 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701822974617 us; error 0 us; skew 500 ppm
I20240503 02:03:42.975104 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:42.977613 19960 webserver.cc:460] Webserver started at http://127.19.126.3:46605/ using document root <none> and password file <none>
I20240503 02:03:42.978044 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:42.978224 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:42.978454 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:42.979446 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-2-root/instance:
uuid: "a269f5186b8b4b93a5c905862aa055f2"
format_stamp: "Formatted at 2024-05-03 02:03:42 on dist-test-slave-8rwr"
I20240503 02:03:42.983738 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240503 02:03:42.987392 20379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:42.988121 19960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:03:42.988385 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "a269f5186b8b4b93a5c905862aa055f2"
format_stamp: "Formatted at 2024-05-03 02:03:42 on dist-test-slave-8rwr"
I20240503 02:03:42.988647 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:43.005730 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:43.006914 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:43.009369 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:03:43.013108 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:03:43.013315 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240503 02:03:43.013535 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:03:43.013685 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:43.119971 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:39303
I20240503 02:03:43.120255 20441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:39303 every 8 connection(s)
I20240503 02:03:43.144865 20442 heartbeater.cc:347] Connected to a master server at 127.19.126.60:33683
I20240503 02:03:43.145546 20442 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:43.147367 20442 heartbeater.cc:511] Master 127.19.126.60:33683 requested a full tablet report, sending...
I20240503 02:03:43.150440 20123 ts_manager.cc:194] Registered new tserver with Master: a269f5186b8b4b93a5c905862aa055f2 (127.19.126.3:39303)
I20240503 02:03:43.152267 20123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54874
I20240503 02:03:43.161055 20444 heartbeater.cc:347] Connected to a master server at 127.19.126.62:42883
I20240503 02:03:43.161489 20444 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:43.161648 20443 heartbeater.cc:347] Connected to a master server at 127.19.126.61:45487
I20240503 02:03:43.162014 20443 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:43.162258 20444 heartbeater.cc:511] Master 127.19.126.62:42883 requested a full tablet report, sending...
I20240503 02:03:43.162690 20443 heartbeater.cc:511] Master 127.19.126.61:45487 requested a full tablet report, sending...
I20240503 02:03:43.164882 19992 ts_manager.cc:194] Registered new tserver with Master: a269f5186b8b4b93a5c905862aa055f2 (127.19.126.3:39303)
I20240503 02:03:43.165324 20057 ts_manager.cc:194] Registered new tserver with Master: a269f5186b8b4b93a5c905862aa055f2 (127.19.126.3:39303)
I20240503 02:03:43.166441 19960 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014080169s
I20240503 02:03:43.390731 20199 catalog_manager.cc:1259] Loaded cluster ID: cd4e65360eaf471eaaaa721b34b8fe91
I20240503 02:03:43.390986 20199 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e: loading cluster ID for follower catalog manager: success
I20240503 02:03:43.395479 20199 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e: acquiring CA information for follower catalog manager: success
I20240503 02:03:43.399022 20200 catalog_manager.cc:1259] Loaded cluster ID: cd4e65360eaf471eaaaa721b34b8fe91
I20240503 02:03:43.399282 20200 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173: loading cluster ID for follower catalog manager: success
I20240503 02:03:43.404593 20199 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:03:43.405965 20200 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173: acquiring CA information for follower catalog manager: success
I20240503 02:03:43.409301 20200 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:03:43.596361 20123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54846
I20240503 02:03:43.968204 20364 heartbeater.cc:503] Master 127.19.126.60:33683 was elected leader, sending a full tablet report...
I20240503 02:03:44.156070 20442 heartbeater.cc:503] Master 127.19.126.60:33683 was elected leader, sending a full tablet report...
I20240503 02:03:44.378381 20196 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:03:44.473731 19960 test_util.cc:268] Using random seed: 556055145
I20240503 02:03:44.536669 20123 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54890:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:03:44.598984 20274 heartbeater.cc:503] Master 127.19.126.60:33683 was elected leader, sending a full tablet report...
I20240503 02:03:44.617050 20407 tablet_service.cc:1467] Processing CreateTablet for tablet 6dca41841be945c8a84c8265e2b569c7 (DEFAULT_TABLE table=test-workload [id=a13fd1f8a9144706825dd36f7d14183f]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:03:44.618726 20407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dca41841be945c8a84c8265e2b569c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:44.622614 20239 tablet_service.cc:1467] Processing CreateTablet for tablet bb0a8fc5749f41d0ac8606c631826dcd (DEFAULT_TABLE table=test-workload [id=a13fd1f8a9144706825dd36f7d14183f]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:03:44.623706 20239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0a8fc5749f41d0ac8606c631826dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:44.623463 20238 tablet_service.cc:1467] Processing CreateTablet for tablet c5b1c3805d4e4ce298421539c82b5c01 (DEFAULT_TABLE table=test-workload [id=a13fd1f8a9144706825dd36f7d14183f]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:03:44.624792 20238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b1c3805d4e4ce298421539c82b5c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:44.632268 20329 tablet_service.cc:1467] Processing CreateTablet for tablet 56227184c2604ffd82b9b46381138bc1 (DEFAULT_TABLE table=test-workload [id=a13fd1f8a9144706825dd36f7d14183f]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:03:44.633631 20329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56227184c2604ffd82b9b46381138bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:44.650509 20478 tablet_bootstrap.cc:492] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2: Bootstrap starting.
I20240503 02:03:44.654695 20477 tablet_bootstrap.cc:492] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76: Bootstrap starting.
I20240503 02:03:44.658597 20479 tablet_bootstrap.cc:492] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549: Bootstrap starting.
I20240503 02:03:44.660552 20478 tablet_bootstrap.cc:654] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:44.662544 20477 tablet_bootstrap.cc:654] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:44.665865 20479 tablet_bootstrap.cc:654] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:44.668736 20478 tablet_bootstrap.cc:492] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2: No bootstrap required, opened a new log
I20240503 02:03:44.669342 20478 ts_tablet_manager.cc:1367] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20240503 02:03:44.670676 20477 tablet_bootstrap.cc:492] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76: No bootstrap required, opened a new log
I20240503 02:03:44.671144 20477 ts_tablet_manager.cc:1367] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s
I20240503 02:03:44.672235 20479 tablet_bootstrap.cc:492] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549: No bootstrap required, opened a new log
I20240503 02:03:44.672698 20479 ts_tablet_manager.cc:1367] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240503 02:03:44.672343 20478 raft_consensus.cc:348] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a269f5186b8b4b93a5c905862aa055f2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 39303 } }
I20240503 02:03:44.673081 20478 raft_consensus.cc:374] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:44.673483 20478 raft_consensus.cc:729] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a269f5186b8b4b93a5c905862aa055f2, State: Initialized, Role: FOLLOWER
I20240503 02:03:44.674197 20478 consensus_queue.cc:260] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a269f5186b8b4b93a5c905862aa055f2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 39303 } }
I20240503 02:03:44.674880 20478 raft_consensus.cc:388] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:03:44.674996 20479 raft_consensus.cc:348] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec0bede00584270bb54719c6d50f549" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38243 } }
I20240503 02:03:44.675302 20478 raft_consensus.cc:482] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:03:44.675522 20479 raft_consensus.cc:374] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:44.675817 20478 raft_consensus.cc:3037] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:44.675869 20479 raft_consensus.cc:729] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec0bede00584270bb54719c6d50f549, State: Initialized, Role: FOLLOWER
I20240503 02:03:44.676581 20479 consensus_queue.cc:260] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec0bede00584270bb54719c6d50f549" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38243 } }
I20240503 02:03:44.677043 20479 raft_consensus.cc:388] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:03:44.674721 20477 raft_consensus.cc:348] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a498427fe2d48a089dc9e141f990b76" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 36865 } }
I20240503 02:03:44.677268 20479 raft_consensus.cc:482] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:03:44.677472 20477 raft_consensus.cc:374] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:44.677675 20479 raft_consensus.cc:3037] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:44.677845 20477 raft_consensus.cc:729] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a498427fe2d48a089dc9e141f990b76, State: Initialized, Role: FOLLOWER
I20240503 02:03:44.678680 20477 consensus_queue.cc:260] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a498427fe2d48a089dc9e141f990b76" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 36865 } }
I20240503 02:03:44.679345 20477 raft_consensus.cc:388] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:03:44.679687 20477 raft_consensus.cc:482] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:03:44.680563 20477 raft_consensus.cc:3037] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:44.682036 20478 raft_consensus.cc:504] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a269f5186b8b4b93a5c905862aa055f2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 39303 } }
I20240503 02:03:44.682770 20478 leader_election.cc:302] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a269f5186b8b4b93a5c905862aa055f2; no voters:
I20240503 02:03:44.682914 20479 raft_consensus.cc:504] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec0bede00584270bb54719c6d50f549" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38243 } }
I20240503 02:03:44.683490 20479 leader_election.cc:302] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ec0bede00584270bb54719c6d50f549; no voters:
I20240503 02:03:44.686182 20478 leader_election.cc:288] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:03:44.686636 20477 raft_consensus.cc:504] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a498427fe2d48a089dc9e141f990b76" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 36865 } }
I20240503 02:03:44.687587 20483 raft_consensus.cc:2781] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:44.688041 20477 leader_election.cc:302] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a498427fe2d48a089dc9e141f990b76; no voters:
I20240503 02:03:44.692734 20479 leader_election.cc:288] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:03:44.694065 20484 raft_consensus.cc:2781] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:44.696866 20478 ts_tablet_manager.cc:1398] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2: Time spent starting tablet: real 0.027s user 0.013s sys 0.012s
I20240503 02:03:44.701993 20479 ts_tablet_manager.cc:1398] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549: Time spent starting tablet: real 0.029s user 0.013s sys 0.013s
I20240503 02:03:44.703384 20484 raft_consensus.cc:686] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [term 1 LEADER]: Becoming Leader. State: Replica: 9ec0bede00584270bb54719c6d50f549, State: Running, Role: LEADER
I20240503 02:03:44.703989 20484 consensus_queue.cc:237] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec0bede00584270bb54719c6d50f549" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38243 } }
I20240503 02:03:44.708498 20483 raft_consensus.cc:686] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [term 1 LEADER]: Becoming Leader. State: Replica: a269f5186b8b4b93a5c905862aa055f2, State: Running, Role: LEADER
I20240503 02:03:44.709800 20477 leader_election.cc:288] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:03:44.709457 20483 consensus_queue.cc:237] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a269f5186b8b4b93a5c905862aa055f2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 39303 } }
I20240503 02:03:44.710733 20487 raft_consensus.cc:2781] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:44.716408 20477 ts_tablet_manager.cc:1398] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76: Time spent starting tablet: real 0.044s user 0.027s sys 0.015s
I20240503 02:03:44.718904 20477 tablet_bootstrap.cc:492] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76: Bootstrap starting.
I20240503 02:03:44.732336 20487 raft_consensus.cc:686] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [term 1 LEADER]: Becoming Leader. State: Replica: 4a498427fe2d48a089dc9e141f990b76, State: Running, Role: LEADER
I20240503 02:03:44.735257 20487 consensus_queue.cc:237] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a498427fe2d48a089dc9e141f990b76" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 36865 } }
I20240503 02:03:44.740628 20123 catalog_manager.cc:5484] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ec0bede00584270bb54719c6d50f549 (127.19.126.2). New cstate: current_term: 1 leader_uuid: "9ec0bede00584270bb54719c6d50f549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec0bede00584270bb54719c6d50f549" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38243 } health_report { overall_health: HEALTHY } } }
I20240503 02:03:44.747287 20477 tablet_bootstrap.cc:654] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:44.753247 20122 catalog_manager.cc:5484] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a269f5186b8b4b93a5c905862aa055f2 (127.19.126.3). New cstate: current_term: 1 leader_uuid: "a269f5186b8b4b93a5c905862aa055f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a269f5186b8b4b93a5c905862aa055f2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 39303 } health_report { overall_health: HEALTHY } } }
I20240503 02:03:44.779990 20121 catalog_manager.cc:5484] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a498427fe2d48a089dc9e141f990b76 (127.19.126.1). New cstate: current_term: 1 leader_uuid: "4a498427fe2d48a089dc9e141f990b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a498427fe2d48a089dc9e141f990b76" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 36865 } health_report { overall_health: HEALTHY } } }
I20240503 02:03:44.814551 20477 tablet_bootstrap.cc:492] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76: No bootstrap required, opened a new log
I20240503 02:03:44.815068 20477 ts_tablet_manager.cc:1367] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76: Time spent bootstrapping tablet: real 0.096s user 0.023s sys 0.019s
I20240503 02:03:44.823681 20477 raft_consensus.cc:348] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a498427fe2d48a089dc9e141f990b76" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 36865 } }
I20240503 02:03:44.824396 20477 raft_consensus.cc:374] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:44.825486 20477 raft_consensus.cc:729] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a498427fe2d48a089dc9e141f990b76, State: Initialized, Role: FOLLOWER
I20240503 02:03:44.826308 20477 consensus_queue.cc:260] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a498427fe2d48a089dc9e141f990b76" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 36865 } }
I20240503 02:03:44.826947 20477 raft_consensus.cc:388] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:03:44.827243 20477 raft_consensus.cc:482] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:03:44.827579 20477 raft_consensus.cc:3037] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:44.833271 20477 raft_consensus.cc:504] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a498427fe2d48a089dc9e141f990b76" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 36865 } }
I20240503 02:03:44.833966 20477 leader_election.cc:302] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a498427fe2d48a089dc9e141f990b76; no voters:
I20240503 02:03:44.835263 20492 raft_consensus.cc:2781] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:44.834548 20477 leader_election.cc:288] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:03:44.835770 20492 raft_consensus.cc:686] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [term 1 LEADER]: Becoming Leader. State: Replica: 4a498427fe2d48a089dc9e141f990b76, State: Running, Role: LEADER
I20240503 02:03:44.837023 20492 consensus_queue.cc:237] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a498427fe2d48a089dc9e141f990b76" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 36865 } }
I20240503 02:03:44.837458 20477 ts_tablet_manager.cc:1398] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76: Time spent starting tablet: real 0.022s user 0.014s sys 0.002s
I20240503 02:03:44.845553 20122 catalog_manager.cc:5484] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a498427fe2d48a089dc9e141f990b76 (127.19.126.1). New cstate: current_term: 1 leader_uuid: "4a498427fe2d48a089dc9e141f990b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a498427fe2d48a089dc9e141f990b76" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 36865 } health_report { overall_health: HEALTHY } } }
I20240503 02:03:44.912794 19960 tablet_server.cc:178] TabletServer@127.19.126.1:0 shutting down...
I20240503 02:03:44.930855 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:03:44.931739 19960 tablet_replica.cc:330] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76: stopping tablet replica
I20240503 02:03:44.932245 19960 raft_consensus.cc:2229] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:44.932636 19960 raft_consensus.cc:2258] T c5b1c3805d4e4ce298421539c82b5c01 P 4a498427fe2d48a089dc9e141f990b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:44.935209 19960 tablet_replica.cc:330] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76: stopping tablet replica
I20240503 02:03:44.935628 19960 raft_consensus.cc:2229] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:44.936007 19960 raft_consensus.cc:2258] T bb0a8fc5749f41d0ac8606c631826dcd P 4a498427fe2d48a089dc9e141f990b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:45.081970 19960 tablet_server.cc:194] TabletServer@127.19.126.1:0 shutdown complete.
I20240503 02:03:45.090500 19960 tablet_server.cc:178] TabletServer@127.19.126.2:0 shutting down...
I20240503 02:03:45.104782 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:03:45.105509 19960 tablet_replica.cc:330] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549: stopping tablet replica
I20240503 02:03:45.105994 19960 raft_consensus.cc:2229] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:45.106354 19960 raft_consensus.cc:2258] T 56227184c2604ffd82b9b46381138bc1 P 9ec0bede00584270bb54719c6d50f549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:45.124181 19960 tablet_server.cc:194] TabletServer@127.19.126.2:0 shutdown complete.
I20240503 02:03:45.131282 19960 tablet_server.cc:178] TabletServer@127.19.126.3:0 shutting down...
I20240503 02:03:45.145757 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:03:45.146409 19960 tablet_replica.cc:330] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2: stopping tablet replica
I20240503 02:03:45.146908 19960 raft_consensus.cc:2229] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:45.147295 19960 raft_consensus.cc:2258] T 6dca41841be945c8a84c8265e2b569c7 P a269f5186b8b4b93a5c905862aa055f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:45.165716 19960 tablet_server.cc:194] TabletServer@127.19.126.3:0 shutdown complete.
I20240503 02:03:45.172672 19960 master.cc:537] Master@127.19.126.62:42883 shutting down...
I20240503 02:03:45.272244 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:45.272737 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:45.273044 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 298676057a0f41abb61b00b9ba48f18e: stopping tablet replica
I20240503 02:03:45.292240 19960 master.cc:552] Master@127.19.126.62:42883 shutdown complete.
I20240503 02:03:45.324767 19960 master.cc:537] Master@127.19.126.61:45487 shutting down...
I20240503 02:03:45.338421 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:45.338888 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:45.339126 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2e003817faf249309d96c38579008173: stopping tablet replica
I20240503 02:03:45.376751 19960 master.cc:552] Master@127.19.126.61:45487 shutdown complete.
I20240503 02:03:45.385964 19960 master.cc:537] Master@127.19.126.60:33683 shutting down...
I20240503 02:03:45.397634 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:45.398387 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:45.398660 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7951382277a446008b68b8c6ceb4a20d: stopping tablet replica
I20240503 02:03:45.415417 19960 master.cc:552] Master@127.19.126.60:33683 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (4824 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240503 02:03:45.441512 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:34943
I20240503 02:03:45.442387 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:03:45.447515 20503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:45.448298 20504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:45.449383 20506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:45.452576 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:45.453208 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:45.453415 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:45.453553 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701825453536 us; error 0 us; skew 500 ppm
I20240503 02:03:45.454007 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:45.456110 19960 webserver.cc:460] Webserver started at http://127.19.126.62:44243/ using document root <none> and password file <none>
I20240503 02:03:45.456524 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:45.456686 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:45.456921 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:45.457954 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "b5a12468480143ea9365bbd30798bc57"
format_stamp: "Formatted at 2024-05-03 02:03:45 on dist-test-slave-8rwr"
I20240503 02:03:45.462085 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240503 02:03:45.464825 20511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:45.465551 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240503 02:03:45.465883 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "b5a12468480143ea9365bbd30798bc57"
format_stamp: "Formatted at 2024-05-03 02:03:45 on dist-test-slave-8rwr"
I20240503 02:03:45.466137 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:45.487277 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:45.488198 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:45.520018 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:34943
I20240503 02:03:45.520112 20562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:34943 every 8 connection(s)
I20240503 02:03:45.523586 20563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:45.532267 20563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57: Bootstrap starting.
I20240503 02:03:45.535869 20563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:45.539307 20563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57: No bootstrap required, opened a new log
I20240503 02:03:45.540937 20563 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a12468480143ea9365bbd30798bc57" member_type: VOTER }
I20240503 02:03:45.541330 20563 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:45.541530 20563 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a12468480143ea9365bbd30798bc57, State: Initialized, Role: FOLLOWER
I20240503 02:03:45.541987 20563 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a12468480143ea9365bbd30798bc57" member_type: VOTER }
I20240503 02:03:45.542382 20563 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:03:45.542716 20563 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:03:45.542955 20563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:45.546875 20563 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a12468480143ea9365bbd30798bc57" member_type: VOTER }
I20240503 02:03:45.547349 20563 leader_election.cc:302] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5a12468480143ea9365bbd30798bc57; no voters:
I20240503 02:03:45.548317 20563 leader_election.cc:288] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:03:45.548602 20566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:45.549877 20566 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [term 1 LEADER]: Becoming Leader. State: Replica: b5a12468480143ea9365bbd30798bc57, State: Running, Role: LEADER
I20240503 02:03:45.550493 20566 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a12468480143ea9365bbd30798bc57" member_type: VOTER }
I20240503 02:03:45.551064 20563 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:03:45.553382 20568 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5a12468480143ea9365bbd30798bc57. Latest consensus state: current_term: 1 leader_uuid: "b5a12468480143ea9365bbd30798bc57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a12468480143ea9365bbd30798bc57" member_type: VOTER } }
I20240503 02:03:45.553344 20567 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5a12468480143ea9365bbd30798bc57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a12468480143ea9365bbd30798bc57" member_type: VOTER } }
I20240503 02:03:45.554100 20568 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [sys.catalog]: This master's current role is: LEADER
I20240503 02:03:45.554394 20567 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [sys.catalog]: This master's current role is: LEADER
I20240503 02:03:45.557709 20572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:03:45.566249 20572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:03:45.567193 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:03:45.574684 20572 catalog_manager.cc:1347] Generated new cluster ID: 39721831251c4543b48fa98d3b29c013
I20240503 02:03:45.574930 20572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:03:45.603542 20572 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:03:45.604705 20572 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:03:45.627954 20572 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57: Generated new TSK 0
I20240503 02:03:45.628474 20572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:03:45.634336 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:03:45.640286 20584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:45.641770 20585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:45.642884 20587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:45.643636 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:45.644327 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:45.644505 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:45.644651 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701825644636 us; error 0 us; skew 500 ppm
I20240503 02:03:45.645151 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:45.647333 19960 webserver.cc:460] Webserver started at http://127.19.126.1:41969/ using document root <none> and password file <none>
I20240503 02:03:45.647751 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:45.647925 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:45.648164 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:45.649144 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-0-root/instance:
uuid: "b5c4d76c502647cbae2946723165732b"
format_stamp: "Formatted at 2024-05-03 02:03:45 on dist-test-slave-8rwr"
I20240503 02:03:45.653378 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:03:45.656178 20592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:45.656854 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:03:45.657137 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "b5c4d76c502647cbae2946723165732b"
format_stamp: "Formatted at 2024-05-03 02:03:45 on dist-test-slave-8rwr"
I20240503 02:03:45.657434 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:45.685855 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:45.686794 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:45.688151 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:03:45.690265 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:03:45.690450 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:45.690673 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:03:45.690822 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:45.725327 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:35423
I20240503 02:03:45.725420 20654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:35423 every 8 connection(s)
I20240503 02:03:45.729501 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:03:45.737176 20659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:45.738370 20660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:45.739408 20662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:45.740301 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:45.740933 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:45.741125 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:45.741308 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701825741266 us; error 0 us; skew 500 ppm
I20240503 02:03:45.741784 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:45.744314 19960 webserver.cc:460] Webserver started at http://127.19.126.2:35647/ using document root <none> and password file <none>
I20240503 02:03:45.744823 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:45.744999 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:45.745330 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:45.745685 20655 heartbeater.cc:347] Connected to a master server at 127.19.126.62:34943
I20240503 02:03:45.745985 20655 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:45.746446 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-1-root/instance:
uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2"
format_stamp: "Formatted at 2024-05-03 02:03:45 on dist-test-slave-8rwr"
I20240503 02:03:45.746585 20655 heartbeater.cc:511] Master 127.19.126.62:34943 requested a full tablet report, sending...
I20240503 02:03:45.748636 20528 ts_manager.cc:194] Registered new tserver with Master: b5c4d76c502647cbae2946723165732b (127.19.126.1:35423)
I20240503 02:03:45.750905 20528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46824
I20240503 02:03:45.751780 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:03:45.754806 20667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:45.755491 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:03:45.755739 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2"
format_stamp: "Formatted at 2024-05-03 02:03:45 on dist-test-slave-8rwr"
I20240503 02:03:45.755980 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:45.769512 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:45.770416 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:45.771667 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:03:45.773680 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:03:45.773845 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:45.774076 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:03:45.774210 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:45.808840 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:37755
I20240503 02:03:45.808920 20729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:37755 every 8 connection(s)
I20240503 02:03:45.813535 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:03:45.820840 20733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:45.821750 20734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:45.823909 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:45.824469 20730 heartbeater.cc:347] Connected to a master server at 127.19.126.62:34943
I20240503 02:03:45.824882 20730 heartbeater.cc:464] Registering TS with master...
W20240503 02:03:45.824833 20736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:45.825820 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:03:45.825834 20730 heartbeater.cc:511] Master 127.19.126.62:34943 requested a full tablet report, sending...
W20240503 02:03:45.826104 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:45.826328 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701825826309 us; error 0 us; skew 500 ppm
I20240503 02:03:45.827003 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:45.827940 20528 ts_manager.cc:194] Registered new tserver with Master: 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:45.829706 19960 webserver.cc:460] Webserver started at http://127.19.126.3:40621/ using document root <none> and password file <none>
I20240503 02:03:45.829819 20528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46830
I20240503 02:03:45.830329 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:45.830596 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:45.830931 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:45.831964 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-2-root/instance:
uuid: "1f03f282c3ee4622ab83463143d2bb72"
format_stamp: "Formatted at 2024-05-03 02:03:45 on dist-test-slave-8rwr"
I20240503 02:03:45.835906 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 02:03:45.838709 20741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:45.839407 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:03:45.839674 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "1f03f282c3ee4622ab83463143d2bb72"
format_stamp: "Formatted at 2024-05-03 02:03:45 on dist-test-slave-8rwr"
I20240503 02:03:45.839939 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:45.869366 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:45.870297 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:45.871596 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:03:45.873690 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:03:45.873871 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:45.874079 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:03:45.874222 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:45.909389 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:38097
I20240503 02:03:45.909466 20803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:38097 every 8 connection(s)
I20240503 02:03:45.921630 20804 heartbeater.cc:347] Connected to a master server at 127.19.126.62:34943
I20240503 02:03:45.921989 20804 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:45.922664 20804 heartbeater.cc:511] Master 127.19.126.62:34943 requested a full tablet report, sending...
I20240503 02:03:45.924345 20528 ts_manager.cc:194] Registered new tserver with Master: 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097)
I20240503 02:03:45.924499 19960 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012191116s
I20240503 02:03:45.926072 20528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46832
I20240503 02:03:46.753335 20655 heartbeater.cc:503] Master 127.19.126.62:34943 was elected leader, sending a full tablet report...
I20240503 02:03:46.832021 20730 heartbeater.cc:503] Master 127.19.126.62:34943 was elected leader, sending a full tablet report...
I20240503 02:03:46.928364 20804 heartbeater.cc:503] Master 127.19.126.62:34943 was elected leader, sending a full tablet report...
I20240503 02:03:46.956673 19960 test_util.cc:268] Using random seed: 558538094
I20240503 02:03:46.977185 20528 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49284:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:03:46.979508 20528 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:03:47.036823 20620 tablet_service.cc:1467] Processing CreateTablet for tablet e3adc50685334bef9080b3ef04a50252 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 02:03:47.037429 20619 tablet_service.cc:1467] Processing CreateTablet for tablet 0eabb3b19a3744efa5234554306453b9 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 02:03:47.038252 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3adc50685334bef9080b3ef04a50252. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.039516 20618 tablet_service.cc:1467] Processing CreateTablet for tablet 70e5e4d17b9b47b485d6ee938059c45b (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 02:03:47.040827 20618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e5e4d17b9b47b485d6ee938059c45b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.041630 20619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eabb3b19a3744efa5234554306453b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.047761 20617 tablet_service.cc:1467] Processing CreateTablet for tablet 38f3f4ef8a70481592fc168f64dedaf8 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 02:03:47.049101 20617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f3f4ef8a70481592fc168f64dedaf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.053778 20690 tablet_service.cc:1467] Processing CreateTablet for tablet ea2d295a06734d2b8bd5bfae41bf51d3 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 02:03:47.053911 20695 tablet_service.cc:1467] Processing CreateTablet for tablet e3adc50685334bef9080b3ef04a50252 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 02:03:47.059327 20768 tablet_service.cc:1467] Processing CreateTablet for tablet 0eabb3b19a3744efa5234554306453b9 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 02:03:47.060621 20690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2d295a06734d2b8bd5bfae41bf51d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.061228 20695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3adc50685334bef9080b3ef04a50252. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.062454 20766 tablet_service.cc:1467] Processing CreateTablet for tablet 38f3f4ef8a70481592fc168f64dedaf8 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 02:03:47.064667 20765 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe84da27587452aa191b1d7eafafad9 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 02:03:47.066946 20764 tablet_service.cc:1467] Processing CreateTablet for tablet ea2d295a06734d2b8bd5bfae41bf51d3 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 02:03:47.068663 20613 tablet_service.cc:1467] Processing CreateTablet for tablet 866cc7b0167a4994bb6ff0a24fbadefb (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 02:03:47.069186 20763 tablet_service.cc:1467] Processing CreateTablet for tablet 908d384adc744a38a456ffbac58a0463 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 02:03:47.069989 20613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866cc7b0167a4994bb6ff0a24fbadefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.074323 20767 tablet_service.cc:1467] Processing CreateTablet for tablet 70e5e4d17b9b47b485d6ee938059c45b (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 02:03:47.070601 20762 tablet_service.cc:1467] Processing CreateTablet for tablet 866cc7b0167a4994bb6ff0a24fbadefb (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 02:03:47.073901 20693 tablet_service.cc:1467] Processing CreateTablet for tablet 70e5e4d17b9b47b485d6ee938059c45b (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 02:03:47.079134 20693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e5e4d17b9b47b485d6ee938059c45b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.073041 20691 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe84da27587452aa191b1d7eafafad9 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 02:03:47.082721 20691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe84da27587452aa191b1d7eafafad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.056242 20694 tablet_service.cc:1467] Processing CreateTablet for tablet 0eabb3b19a3744efa5234554306453b9 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 02:03:47.086620 20694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eabb3b19a3744efa5234554306453b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.055115 20615 tablet_service.cc:1467] Processing CreateTablet for tablet ea2d295a06734d2b8bd5bfae41bf51d3 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 02:03:47.056219 20688 tablet_service.cc:1467] Processing CreateTablet for tablet 866cc7b0167a4994bb6ff0a24fbadefb (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 02:03:47.054440 20689 tablet_service.cc:1467] Processing CreateTablet for tablet 908d384adc744a38a456ffbac58a0463 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 02:03:47.090113 20688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866cc7b0167a4994bb6ff0a24fbadefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.090853 20689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 908d384adc744a38a456ffbac58a0463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.056532 20616 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe84da27587452aa191b1d7eafafad9 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 02:03:47.058362 20769 tablet_service.cc:1467] Processing CreateTablet for tablet e3adc50685334bef9080b3ef04a50252 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 02:03:47.066844 20614 tablet_service.cc:1467] Processing CreateTablet for tablet 908d384adc744a38a456ffbac58a0463 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 02:03:47.057531 20692 tablet_service.cc:1467] Processing CreateTablet for tablet 38f3f4ef8a70481592fc168f64dedaf8 (DEFAULT_TABLE table=test-workload [id=9033f98ba7a84814984c5be5df889557]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 02:03:47.060629 20768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eabb3b19a3744efa5234554306453b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.093866 20692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f3f4ef8a70481592fc168f64dedaf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.097646 20769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3adc50685334bef9080b3ef04a50252. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.090117 20615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2d295a06734d2b8bd5bfae41bf51d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.099700 20763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 908d384adc744a38a456ffbac58a0463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.101647 20616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe84da27587452aa191b1d7eafafad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.104000 20614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 908d384adc744a38a456ffbac58a0463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.104322 20764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2d295a06734d2b8bd5bfae41bf51d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.107205 20767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e5e4d17b9b47b485d6ee938059c45b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.110239 20824 tablet_bootstrap.cc:492] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Bootstrap starting.
I20240503 02:03:47.135010 20766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f3f4ef8a70481592fc168f64dedaf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.136313 20762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866cc7b0167a4994bb6ff0a24fbadefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.137701 20765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe84da27587452aa191b1d7eafafad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:47.150110 20825 tablet_bootstrap.cc:492] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b: Bootstrap starting.
I20240503 02:03:47.164558 20825 tablet_bootstrap.cc:654] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.175710 20824 tablet_bootstrap.cc:654] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.204905 20827 tablet_bootstrap.cc:492] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72: Bootstrap starting.
I20240503 02:03:47.207429 20825 tablet_bootstrap.cc:492] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b: No bootstrap required, opened a new log
I20240503 02:03:47.207933 20825 ts_tablet_manager.cc:1367] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b: Time spent bootstrapping tablet: real 0.058s user 0.022s sys 0.017s
I20240503 02:03:47.211728 20825 raft_consensus.cc:348] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } }
I20240503 02:03:47.212451 20827 tablet_bootstrap.cc:654] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.212498 20825 raft_consensus.cc:374] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.212951 20825 raft_consensus.cc:729] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c4d76c502647cbae2946723165732b, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.215878 20825 consensus_queue.cc:260] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } }
I20240503 02:03:47.221983 20825 ts_tablet_manager.cc:1398] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b: Time spent starting tablet: real 0.014s user 0.006s sys 0.004s
I20240503 02:03:47.223089 20825 tablet_bootstrap.cc:492] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b: Bootstrap starting.
I20240503 02:03:47.225873 20827 tablet_bootstrap.cc:492] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72: No bootstrap required, opened a new log
I20240503 02:03:47.226322 20827 ts_tablet_manager.cc:1367] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.001s
I20240503 02:03:47.228384 20825 tablet_bootstrap.cc:654] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.230844 20827 raft_consensus.cc:348] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } }
I20240503 02:03:47.231700 20827 raft_consensus.cc:374] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.232055 20827 raft_consensus.cc:729] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.233032 20827 consensus_queue.cc:260] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } }
I20240503 02:03:47.237207 20824 tablet_bootstrap.cc:492] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2: No bootstrap required, opened a new log
I20240503 02:03:47.237628 20824 ts_tablet_manager.cc:1367] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent bootstrapping tablet: real 0.128s user 0.011s sys 0.015s
I20240503 02:03:47.239205 20827 ts_tablet_manager.cc:1398] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72: Time spent starting tablet: real 0.013s user 0.002s sys 0.006s
I20240503 02:03:47.239390 20824 raft_consensus.cc:348] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } }
I20240503 02:03:47.239984 20824 raft_consensus.cc:374] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.240245 20824 raft_consensus.cc:729] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d3adb9fdfc47a8a9760ac9be6f19b2, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.240872 20824 consensus_queue.cc:260] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } }
I20240503 02:03:47.242352 20827 tablet_bootstrap.cc:492] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72: Bootstrap starting.
I20240503 02:03:47.250917 20827 tablet_bootstrap.cc:654] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.252072 20825 tablet_bootstrap.cc:492] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b: No bootstrap required, opened a new log
I20240503 02:03:47.252557 20825 ts_tablet_manager.cc:1367] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.015s
I20240503 02:03:47.253793 20824 ts_tablet_manager.cc:1398] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20240503 02:03:47.254688 20824 tablet_bootstrap.cc:492] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Bootstrap starting.
I20240503 02:03:47.255580 20825 raft_consensus.cc:348] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.256506 20825 raft_consensus.cc:374] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.256860 20825 raft_consensus.cc:729] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c4d76c502647cbae2946723165732b, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.257722 20825 consensus_queue.cc:260] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.260367 20825 ts_tablet_manager.cc:1398] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240503 02:03:47.260500 20824 tablet_bootstrap.cc:654] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.261330 20825 tablet_bootstrap.cc:492] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b: Bootstrap starting.
I20240503 02:03:47.262203 20827 tablet_bootstrap.cc:492] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72: No bootstrap required, opened a new log
I20240503 02:03:47.262650 20827 ts_tablet_manager.cc:1367] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.007s
I20240503 02:03:47.264950 20827 raft_consensus.cc:348] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.265681 20827 raft_consensus.cc:374] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.265964 20827 raft_consensus.cc:729] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.266698 20825 tablet_bootstrap.cc:654] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.266817 20827 consensus_queue.cc:260] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.272044 20827 ts_tablet_manager.cc:1398] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72: Time spent starting tablet: real 0.009s user 0.001s sys 0.006s
I20240503 02:03:47.273177 20827 tablet_bootstrap.cc:492] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72: Bootstrap starting.
I20240503 02:03:47.277595 20825 tablet_bootstrap.cc:492] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b: No bootstrap required, opened a new log
I20240503 02:03:47.278085 20825 ts_tablet_manager.cc:1367] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240503 02:03:47.279915 20827 tablet_bootstrap.cc:654] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.280469 20825 raft_consensus.cc:348] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.281250 20825 raft_consensus.cc:374] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.281558 20825 raft_consensus.cc:729] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c4d76c502647cbae2946723165732b, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.282457 20825 consensus_queue.cc:260] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.285233 20825 ts_tablet_manager.cc:1398] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:03:47.286506 20825 tablet_bootstrap.cc:492] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b: Bootstrap starting.
I20240503 02:03:47.291229 20827 tablet_bootstrap.cc:492] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72: No bootstrap required, opened a new log
I20240503 02:03:47.291689 20827 ts_tablet_manager.cc:1367] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.003s
I20240503 02:03:47.292205 20825 tablet_bootstrap.cc:654] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.294231 20827 raft_consensus.cc:348] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.295289 20827 raft_consensus.cc:374] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.296340 20832 raft_consensus.cc:482] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.296916 20832 raft_consensus.cc:504] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } }
I20240503 02:03:47.300160 20832 leader_election.cc:288] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:47.295675 20827 raft_consensus.cc:729] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.301584 20827 consensus_queue.cc:260] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.304644 20827 ts_tablet_manager.cc:1398] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20240503 02:03:47.305838 20827 tablet_bootstrap.cc:492] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72: Bootstrap starting.
I20240503 02:03:47.313714 20832 raft_consensus.cc:482] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.314424 20827 tablet_bootstrap.cc:654] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.314178 20832 raft_consensus.cc:504] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.316080 20832 leader_election.cc:288] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:47.318295 20825 tablet_bootstrap.cc:492] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b: No bootstrap required, opened a new log
I20240503 02:03:47.318897 20825 ts_tablet_manager.cc:1367] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.012s
I20240503 02:03:47.320680 20828 raft_consensus.cc:482] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.321321 20828 raft_consensus.cc:504] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.321965 20825 raft_consensus.cc:348] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.322918 20825 raft_consensus.cc:374] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.323192 20825 raft_consensus.cc:729] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c4d76c502647cbae2946723165732b, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.323889 20825 consensus_queue.cc:260] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.325397 20827 tablet_bootstrap.cc:492] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72: No bootstrap required, opened a new log
I20240503 02:03:47.325946 20827 ts_tablet_manager.cc:1367] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.009s
I20240503 02:03:47.327670 20828 leader_election.cc:288] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755), 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097)
I20240503 02:03:47.331107 20827 raft_consensus.cc:348] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.332059 20827 raft_consensus.cc:374] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.332851 20827 raft_consensus.cc:729] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.333454 20825 ts_tablet_manager.cc:1398] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b: Time spent starting tablet: real 0.014s user 0.007s sys 0.009s
I20240503 02:03:47.333591 20827 consensus_queue.cc:260] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.334389 20825 tablet_bootstrap.cc:492] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b: Bootstrap starting.
I20240503 02:03:47.335932 20827 ts_tablet_manager.cc:1398] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s
I20240503 02:03:47.336825 20827 tablet_bootstrap.cc:492] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72: Bootstrap starting.
I20240503 02:03:47.339977 20825 tablet_bootstrap.cc:654] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.342216 20827 tablet_bootstrap.cc:654] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.351219 20825 tablet_bootstrap.cc:492] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b: No bootstrap required, opened a new log
I20240503 02:03:47.351671 20825 ts_tablet_manager.cc:1367] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.005s
I20240503 02:03:47.352084 20827 tablet_bootstrap.cc:492] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72: No bootstrap required, opened a new log
I20240503 02:03:47.352631 20827 ts_tablet_manager.cc:1367] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20240503 02:03:47.354123 20825 raft_consensus.cc:348] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.354892 20825 raft_consensus.cc:374] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.355177 20825 raft_consensus.cc:729] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c4d76c502647cbae2946723165732b, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.355437 20827 raft_consensus.cc:348] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.356124 20827 raft_consensus.cc:374] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.355823 20825 consensus_queue.cc:260] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.356416 20827 raft_consensus.cc:729] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.357336 20827 consensus_queue.cc:260] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.358089 20825 ts_tablet_manager.cc:1398] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240503 02:03:47.359246 20825 tablet_bootstrap.cc:492] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b: Bootstrap starting.
I20240503 02:03:47.359592 20827 ts_tablet_manager.cc:1398] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240503 02:03:47.360469 20827 tablet_bootstrap.cc:492] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72: Bootstrap starting.
I20240503 02:03:47.366237 20841 raft_consensus.cc:482] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.366710 20841 raft_consensus.cc:504] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } }
I20240503 02:03:47.373694 20841 leader_election.cc:288] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097), 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:47.377138 20827 tablet_bootstrap.cc:654] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.384716 20832 raft_consensus.cc:482] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.385172 20832 raft_consensus.cc:504] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.387466 20825 tablet_bootstrap.cc:654] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.387866 20832 leader_election.cc:288] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:47.397805 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3adc50685334bef9080b3ef04a50252" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b" is_pre_election: true
I20240503 02:03:47.398555 20630 raft_consensus.cc:2454] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 0.
I20240503 02:03:47.398833 20824 tablet_bootstrap.cc:492] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2: No bootstrap required, opened a new log
I20240503 02:03:47.399302 20824 ts_tablet_manager.cc:1367] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent bootstrapping tablet: real 0.145s user 0.064s sys 0.062s
I20240503 02:03:47.400764 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e5e4d17b9b47b485d6ee938059c45b" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b" is_pre_election: true
I20240503 02:03:47.401446 20629 raft_consensus.cc:2454] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 0.
I20240503 02:03:47.401785 20824 raft_consensus.cc:348] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.402489 20824 raft_consensus.cc:374] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.402799 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908d384adc744a38a456ffbac58a0463" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b" is_pre_election: true
I20240503 02:03:47.403735 20778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3adc50685334bef9080b3ef04a50252" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f282c3ee4622ab83463143d2bb72" is_pre_election: true
I20240503 02:03:47.404580 20778 raft_consensus.cc:2454] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c4d76c502647cbae2946723165732b in term 0.
I20240503 02:03:47.406641 20596 leader_election.cc:302] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b; no voters:
I20240503 02:03:47.408097 20841 raft_consensus.cc:2781] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:03:47.408558 20841 raft_consensus.cc:482] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:03:47.408959 20841 raft_consensus.cc:3037] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.409330 20705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866cc7b0167a4994bb6ff0a24fbadefb" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" is_pre_election: true
I20240503 02:03:47.410784 20704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3adc50685334bef9080b3ef04a50252" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" is_pre_election: true
I20240503 02:03:47.411454 20704 raft_consensus.cc:2454] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c4d76c502647cbae2946723165732b in term 0.
I20240503 02:03:47.402812 20824 raft_consensus.cc:729] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d3adb9fdfc47a8a9760ac9be6f19b2, State: Initialized, Role: FOLLOWER
W20240503 02:03:47.412936 20594 leader_election.cc:341] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:03:47.402634 20779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866cc7b0167a4994bb6ff0a24fbadefb" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f282c3ee4622ab83463143d2bb72" is_pre_election: true
I20240503 02:03:47.413547 20824 consensus_queue.cc:260] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
W20240503 02:03:47.414460 20596 leader_election.cc:341] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:03:47.414862 20596 leader_election.cc:302] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5c4d76c502647cbae2946723165732b; no voters: 1f03f282c3ee4622ab83463143d2bb72, 49d3adb9fdfc47a8a9760ac9be6f19b2
I20240503 02:03:47.415972 20824 ts_tablet_manager.cc:1398] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent starting tablet: real 0.016s user 0.006s sys 0.002s
I20240503 02:03:47.416905 20824 tablet_bootstrap.cc:492] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2: Bootstrap starting.
I20240503 02:03:47.418159 20828 raft_consensus.cc:2726] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:03:47.421826 20841 raft_consensus.cc:504] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } }
I20240503 02:03:47.423589 20827 tablet_bootstrap.cc:492] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72: No bootstrap required, opened a new log
I20240503 02:03:47.424041 20827 ts_tablet_manager.cc:1367] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.011s
I20240503 02:03:47.425246 20779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3adc50685334bef9080b3ef04a50252" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f282c3ee4622ab83463143d2bb72"
I20240503 02:03:47.425915 20779 raft_consensus.cc:3037] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.427719 20827 raft_consensus.cc:348] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.428694 20827 raft_consensus.cc:374] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.429044 20827 raft_consensus.cc:729] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.430393 20827 consensus_queue.cc:260] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.431979 20841 leader_election.cc:288] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 election: Requested vote from peers 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097), 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:47.433516 20827 ts_tablet_manager.cc:1398] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20240503 02:03:47.433317 20704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3adc50685334bef9080b3ef04a50252" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2"
I20240503 02:03:47.417207 20743 leader_election.cc:302] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b; no voters:
I20240503 02:03:47.433866 20779 raft_consensus.cc:2454] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c4d76c502647cbae2946723165732b in term 1.
I20240503 02:03:47.434358 20704 raft_consensus.cc:3037] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.435319 20832 raft_consensus.cc:2741] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 02:03:47.435250 20596 leader_election.cc:302] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b; no voters:
I20240503 02:03:47.436388 20841 raft_consensus.cc:2781] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:47.437376 20832 raft_consensus.cc:482] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.437463 20841 raft_consensus.cc:686] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 1 LEADER]: Becoming Leader. State: Replica: b5c4d76c502647cbae2946723165732b, State: Running, Role: LEADER
I20240503 02:03:47.437906 20832 raft_consensus.cc:504] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.438412 20824 tablet_bootstrap.cc:654] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.439359 20841 consensus_queue.cc:237] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } }
I20240503 02:03:47.440089 20825 tablet_bootstrap.cc:492] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b: No bootstrap required, opened a new log
I20240503 02:03:47.440465 20832 leader_election.cc:288] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:47.440723 20825 ts_tablet_manager.cc:1367] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b: Time spent bootstrapping tablet: real 0.082s user 0.017s sys 0.021s
I20240503 02:03:47.443121 20743 leader_election.cc:302] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b; no voters:
I20240503 02:03:47.443389 20825 raft_consensus.cc:348] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.443710 20827 tablet_bootstrap.cc:492] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72: Bootstrap starting.
I20240503 02:03:47.444049 20825 raft_consensus.cc:374] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.444380 20825 raft_consensus.cc:729] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c4d76c502647cbae2946723165732b, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.444854 20832 raft_consensus.cc:2781] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:03:47.444723 20704 raft_consensus.cc:2454] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c4d76c502647cbae2946723165732b in term 1.
I20240503 02:03:47.445469 20832 raft_consensus.cc:482] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:03:47.446064 20832 raft_consensus.cc:3037] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.448029 20824 tablet_bootstrap.cc:492] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2: No bootstrap required, opened a new log
I20240503 02:03:47.448387 20824 ts_tablet_manager.cc:1367] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.002s
I20240503 02:03:47.445032 20825 consensus_queue.cc:260] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.450773 20825 ts_tablet_manager.cc:1398] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240503 02:03:47.450546 20824 raft_consensus.cc:348] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.450922 20832 raft_consensus.cc:504] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.451239 20824 raft_consensus.cc:374] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.451664 20825 tablet_bootstrap.cc:492] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b: Bootstrap starting.
I20240503 02:03:47.451618 20824 raft_consensus.cc:729] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d3adb9fdfc47a8a9760ac9be6f19b2, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.452739 20832 leader_election.cc:288] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 election: Requested vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:47.452524 20824 consensus_queue.cc:260] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.454634 20824 ts_tablet_manager.cc:1398] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20240503 02:03:47.455192 20743 leader_election.cc:341] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b5c4d76c502647cbae2946723165732b (127.19.126.1:35423): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:03:47.455626 20824 tablet_bootstrap.cc:492] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Bootstrap starting.
I20240503 02:03:47.458166 20825 tablet_bootstrap.cc:654] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.458649 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eabb3b19a3744efa5234554306453b9" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b" is_pre_election: true
I20240503 02:03:47.459324 20630 raft_consensus.cc:2454] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 0.
I20240503 02:03:47.461656 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e5e4d17b9b47b485d6ee938059c45b" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b"
I20240503 02:03:47.462378 20630 raft_consensus.cc:3037] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.462620 20828 raft_consensus.cc:482] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.463002 20828 raft_consensus.cc:504] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.464222 20824 tablet_bootstrap.cc:654] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.464876 20828 leader_election.cc:288] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755), 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097)
I20240503 02:03:47.467933 20704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f3f4ef8a70481592fc168f64dedaf8" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" is_pre_election: true
I20240503 02:03:47.469811 20705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3adc50685334bef9080b3ef04a50252" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" is_pre_election: true
I20240503 02:03:47.470151 20779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f3f4ef8a70481592fc168f64dedaf8" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f282c3ee4622ab83463143d2bb72" is_pre_election: true
W20240503 02:03:47.471706 20594 leader_election.cc:341] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:03:47.476891 20705 raft_consensus.cc:2379] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f03f282c3ee4622ab83463143d2bb72 in current term 1: Already voted for candidate b5c4d76c502647cbae2946723165732b in this term.
I20240503 02:03:47.484505 20827 tablet_bootstrap.cc:654] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.485957 20704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908d384adc744a38a456ffbac58a0463" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" is_pre_election: true
I20240503 02:03:47.476536 20779 raft_consensus.cc:2454] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c4d76c502647cbae2946723165732b in term 0.
I20240503 02:03:47.490000 20596 leader_election.cc:302] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b; no voters: 49d3adb9fdfc47a8a9760ac9be6f19b2
I20240503 02:03:47.490128 20705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e5e4d17b9b47b485d6ee938059c45b" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" is_pre_election: true
I20240503 02:03:47.491019 20705 raft_consensus.cc:2454] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 0.
I20240503 02:03:47.492846 20841 raft_consensus.cc:2781] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:03:47.492230 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e5e4d17b9b47b485d6ee938059c45b" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2"
I20240503 02:03:47.493969 20841 raft_consensus.cc:482] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:03:47.494343 20703 raft_consensus.cc:3037] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.485577 20743 leader_election.cc:302] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b; no voters:
I20240503 02:03:47.494415 20841 raft_consensus.cc:3037] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.495802 20832 raft_consensus.cc:2781] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:03:47.494064 20527 catalog_manager.cc:5484] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b reported cstate change: term changed from 0 to 1, leader changed from <none> to b5c4d76c502647cbae2946723165732b (127.19.126.1). New cstate: current_term: 1 leader_uuid: "b5c4d76c502647cbae2946723165732b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } health_report { overall_health: UNKNOWN } } }
I20240503 02:03:47.496202 20832 raft_consensus.cc:482] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:03:47.497262 20832 raft_consensus.cc:3037] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:03:47.499341 20743 leader_election.cc:341] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:03:47.494091 20704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eabb3b19a3744efa5234554306453b9" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" is_pre_election: true
I20240503 02:03:47.499779 20743 leader_election.cc:302] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72; no voters: 49d3adb9fdfc47a8a9760ac9be6f19b2, b5c4d76c502647cbae2946723165732b
I20240503 02:03:47.501740 20703 raft_consensus.cc:2454] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 1.
I20240503 02:03:47.503330 20841 raft_consensus.cc:504] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.504333 20832 raft_consensus.cc:504] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.505812 20832 leader_election.cc:288] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 election: Requested vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:47.506299 20832 raft_consensus.cc:2726] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:03:47.507548 20779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f3f4ef8a70481592fc168f64dedaf8" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f282c3ee4622ab83463143d2bb72"
I20240503 02:03:47.507725 20630 raft_consensus.cc:2454] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 1.
I20240503 02:03:47.508132 20779 raft_consensus.cc:3037] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.510088 20824 tablet_bootstrap.cc:492] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2: No bootstrap required, opened a new log
I20240503 02:03:47.510243 20825 tablet_bootstrap.cc:492] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b: No bootstrap required, opened a new log
I20240503 02:03:47.512089 20824 ts_tablet_manager.cc:1367] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent bootstrapping tablet: real 0.057s user 0.014s sys 0.013s
I20240503 02:03:47.512104 20825 ts_tablet_manager.cc:1367] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b: Time spent bootstrapping tablet: real 0.061s user 0.032s sys 0.005s
I20240503 02:03:47.513346 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f3f4ef8a70481592fc168f64dedaf8" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2"
W20240503 02:03:47.514449 20594 leader_election.cc:341] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:03:47.514959 20824 raft_consensus.cc:348] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.515734 20825 raft_consensus.cc:348] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.516975 20825 raft_consensus.cc:374] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.517321 20825 raft_consensus.cc:729] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c4d76c502647cbae2946723165732b, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.518208 20825 consensus_queue.cc:260] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.520174 20779 raft_consensus.cc:2454] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c4d76c502647cbae2946723165732b in term 1.
I20240503 02:03:47.515703 20824 raft_consensus.cc:374] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.521302 20824 raft_consensus.cc:729] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d3adb9fdfc47a8a9760ac9be6f19b2, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.521220 20596 leader_election.cc:302] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b; no voters: 49d3adb9fdfc47a8a9760ac9be6f19b2
I20240503 02:03:47.522714 20828 raft_consensus.cc:2781] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:47.522330 20824 consensus_queue.cc:260] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.525197 20824 ts_tablet_manager.cc:1398] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 02:03:47.525530 20825 ts_tablet_manager.cc:1398] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240503 02:03:47.526108 20824 tablet_bootstrap.cc:492] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Bootstrap starting.
I20240503 02:03:47.526554 20825 tablet_bootstrap.cc:492] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b: Bootstrap starting.
I20240503 02:03:47.528550 20827 tablet_bootstrap.cc:492] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72: No bootstrap required, opened a new log
I20240503 02:03:47.529033 20827 ts_tablet_manager.cc:1367] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72: Time spent bootstrapping tablet: real 0.095s user 0.011s sys 0.010s
I20240503 02:03:47.532294 20841 leader_election.cc:288] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 election: Requested vote from peers 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755), 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097)
I20240503 02:03:47.533840 20828 raft_consensus.cc:686] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 1 LEADER]: Becoming Leader. State: Replica: b5c4d76c502647cbae2946723165732b, State: Running, Role: LEADER
I20240503 02:03:47.535319 20828 consensus_queue.cc:237] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.536301 20827 raft_consensus.cc:348] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.537393 20827 raft_consensus.cc:374] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.537693 20827 raft_consensus.cc:729] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.538630 20824 tablet_bootstrap.cc:654] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.538465 20827 consensus_queue.cc:260] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.541133 20827 ts_tablet_manager.cc:1398] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72: Time spent starting tablet: real 0.012s user 0.003s sys 0.002s
I20240503 02:03:47.541644 20825 tablet_bootstrap.cc:654] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b: Neither blocks nor log segments found. Creating new log.
W20240503 02:03:47.542063 20743 leader_election.cc:341] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:03:47.543020 20827 tablet_bootstrap.cc:492] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72: Bootstrap starting.
I20240503 02:03:47.543010 20743 leader_election.cc:302] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, 49d3adb9fdfc47a8a9760ac9be6f19b2; no voters:
I20240503 02:03:47.547516 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eabb3b19a3744efa5234554306453b9" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b"
I20240503 02:03:47.547834 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eabb3b19a3744efa5234554306453b9" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2"
I20240503 02:03:47.548146 20630 raft_consensus.cc:3037] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.548133 20854 raft_consensus.cc:2781] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:47.549064 20854 raft_consensus.cc:686] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Becoming Leader. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Running, Role: LEADER
W20240503 02:03:47.549252 20743 leader_election.cc:341] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:03:47.549964 20854 consensus_queue.cc:237] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.559209 20824 tablet_bootstrap.cc:492] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2: No bootstrap required, opened a new log
I20240503 02:03:47.559646 20824 ts_tablet_manager.cc:1367] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.011s
I20240503 02:03:47.562841 20827 tablet_bootstrap.cc:654] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.562624 20824 raft_consensus.cc:348] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.560148 20527 catalog_manager.cc:5484] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b reported cstate change: term changed from 0 to 1, leader changed from <none> to b5c4d76c502647cbae2946723165732b (127.19.126.1). New cstate: current_term: 1 leader_uuid: "b5c4d76c502647cbae2946723165732b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } health_report { overall_health: UNKNOWN } } }
W20240503 02:03:47.564889 20576 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:03:47.560148 20630 raft_consensus.cc:2454] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 1.
I20240503 02:03:47.563603 20824 raft_consensus.cc:374] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.566895 20824 raft_consensus.cc:729] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d3adb9fdfc47a8a9760ac9be6f19b2, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.568023 20743 leader_election.cc:302] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b; no voters: 49d3adb9fdfc47a8a9760ac9be6f19b2
I20240503 02:03:47.568089 20824 consensus_queue.cc:260] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.569373 20825 tablet_bootstrap.cc:492] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b: No bootstrap required, opened a new log
I20240503 02:03:47.569823 20825 ts_tablet_manager.cc:1367] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.002s
I20240503 02:03:47.570477 20824 ts_tablet_manager.cc:1398] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent starting tablet: real 0.011s user 0.002s sys 0.005s
I20240503 02:03:47.571388 20824 tablet_bootstrap.cc:492] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Bootstrap starting.
I20240503 02:03:47.572825 20825 raft_consensus.cc:348] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.573598 20825 raft_consensus.cc:374] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.573855 20825 raft_consensus.cc:729] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c4d76c502647cbae2946723165732b, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.574718 20854 raft_consensus.cc:2781] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:47.575177 20854 raft_consensus.cc:686] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Becoming Leader. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Running, Role: LEADER
I20240503 02:03:47.574407 20825 consensus_queue.cc:260] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.575619 20581 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:03:47.578745 20825 ts_tablet_manager.cc:1398] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240503 02:03:47.579955 20581 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:03:47.578564 20854 consensus_queue.cc:237] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.583107 20824 tablet_bootstrap.cc:654] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.588301 20824 tablet_bootstrap.cc:492] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2: No bootstrap required, opened a new log
I20240503 02:03:47.588670 20824 ts_tablet_manager.cc:1367] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.001s
I20240503 02:03:47.591611 20824 raft_consensus.cc:348] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.592358 20824 raft_consensus.cc:374] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.592658 20824 raft_consensus.cc:729] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d3adb9fdfc47a8a9760ac9be6f19b2, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.593221 20824 consensus_queue.cc:260] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.594206 20827 tablet_bootstrap.cc:492] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72: No bootstrap required, opened a new log
I20240503 02:03:47.594674 20827 ts_tablet_manager.cc:1367] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72: Time spent bootstrapping tablet: real 0.052s user 0.016s sys 0.006s
I20240503 02:03:47.595608 20824 ts_tablet_manager.cc:1398] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20240503 02:03:47.596488 20824 tablet_bootstrap.cc:492] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2: Bootstrap starting.
I20240503 02:03:47.597203 20827 raft_consensus.cc:348] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.597906 20827 raft_consensus.cc:374] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.598191 20827 raft_consensus.cc:729] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.598824 20827 consensus_queue.cc:260] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.598484 20526 catalog_manager.cc:5484] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3). New cstate: current_term: 1 leader_uuid: "1f03f282c3ee4622ab83463143d2bb72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } health_report { overall_health: HEALTHY } } }
I20240503 02:03:47.600977 20827 ts_tablet_manager.cc:1398] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:03:47.600375 20526 catalog_manager.cc:5484] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3). New cstate: current_term: 1 leader_uuid: "1f03f282c3ee4622ab83463143d2bb72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } health_report { overall_health: HEALTHY } } }
I20240503 02:03:47.603582 20824 tablet_bootstrap.cc:654] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.606355 20854 raft_consensus.cc:482] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.606746 20854 raft_consensus.cc:504] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.609028 20854 leader_election.cc:288] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:47.609546 20824 tablet_bootstrap.cc:492] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2: No bootstrap required, opened a new log
I20240503 02:03:47.610029 20824 ts_tablet_manager.cc:1367] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20240503 02:03:47.609974 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2d295a06734d2b8bd5bfae41bf51d3" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b" is_pre_election: true
I20240503 02:03:47.610374 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2d295a06734d2b8bd5bfae41bf51d3" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" is_pre_election: true
I20240503 02:03:47.610769 20630 raft_consensus.cc:2454] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 0.
I20240503 02:03:47.611004 20703 raft_consensus.cc:2454] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 0.
I20240503 02:03:47.612046 20743 leader_election.cc:302] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, 49d3adb9fdfc47a8a9760ac9be6f19b2; no voters:
I20240503 02:03:47.612632 20854 raft_consensus.cc:2781] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:03:47.613045 20854 raft_consensus.cc:482] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:03:47.612848 20824 raft_consensus.cc:348] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.613477 20854 raft_consensus.cc:3037] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.613665 20824 raft_consensus.cc:374] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.613957 20824 raft_consensus.cc:729] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d3adb9fdfc47a8a9760ac9be6f19b2, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.615126 20824 consensus_queue.cc:260] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.616708 20824 ts_tablet_manager.cc:1398] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:03:47.617717 20824 tablet_bootstrap.cc:492] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Bootstrap starting.
I20240503 02:03:47.619864 20854 raft_consensus.cc:504] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.622463 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2d295a06734d2b8bd5bfae41bf51d3" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b"
I20240503 02:03:47.622890 20630 raft_consensus.cc:3037] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.623558 20824 tablet_bootstrap.cc:654] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:47.628435 20630 raft_consensus.cc:2454] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 1.
I20240503 02:03:47.629506 20743 leader_election.cc:302] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b; no voters:
I20240503 02:03:47.630367 20832 raft_consensus.cc:2781] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:47.630475 20854 leader_election.cc:288] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 election: Requested vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:47.632647 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2d295a06734d2b8bd5bfae41bf51d3" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2"
I20240503 02:03:47.633212 20703 raft_consensus.cc:3037] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.635147 20824 tablet_bootstrap.cc:492] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2: No bootstrap required, opened a new log
I20240503 02:03:47.635502 20824 ts_tablet_manager.cc:1367] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20240503 02:03:47.637522 20824 raft_consensus.cc:348] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.638054 20703 raft_consensus.cc:2454] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 1.
I20240503 02:03:47.638404 20824 raft_consensus.cc:374] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:47.638797 20824 raft_consensus.cc:729] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d3adb9fdfc47a8a9760ac9be6f19b2, State: Initialized, Role: FOLLOWER
I20240503 02:03:47.638880 20832 raft_consensus.cc:686] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Becoming Leader. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Running, Role: LEADER
I20240503 02:03:47.639534 20824 consensus_queue.cc:260] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.639698 20832 consensus_queue.cc:237] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.641901 20824 ts_tablet_manager.cc:1398] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240503 02:03:47.647861 20833 raft_consensus.cc:482] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.648106 20527 catalog_manager.cc:5484] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3). New cstate: current_term: 1 leader_uuid: "1f03f282c3ee4622ab83463143d2bb72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } health_report { overall_health: HEALTHY } } }
I20240503 02:03:47.648818 20833 raft_consensus.cc:504] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.651839 20828 raft_consensus.cc:482] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.652257 20828 raft_consensus.cc:504] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.653702 20828 leader_election.cc:288] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755), 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097)
I20240503 02:03:47.654688 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908d384adc744a38a456ffbac58a0463" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" is_pre_election: true
I20240503 02:03:47.655295 20703 raft_consensus.cc:2454] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c4d76c502647cbae2946723165732b in term 0.
I20240503 02:03:47.655838 20779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908d384adc744a38a456ffbac58a0463" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f282c3ee4622ab83463143d2bb72" is_pre_election: true
I20240503 02:03:47.656381 20594 leader_election.cc:302] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d3adb9fdfc47a8a9760ac9be6f19b2, b5c4d76c502647cbae2946723165732b; no voters:
I20240503 02:03:47.656603 20779 raft_consensus.cc:2454] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c4d76c502647cbae2946723165732b in term 0.
I20240503 02:03:47.657846 20828 raft_consensus.cc:2781] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:03:47.658215 20828 raft_consensus.cc:482] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:03:47.658519 20828 raft_consensus.cc:3037] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.658758 20833 leader_election.cc:288] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097)
I20240503 02:03:47.663216 20828 raft_consensus.cc:504] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.665786 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908d384adc744a38a456ffbac58a0463" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2"
I20240503 02:03:47.666069 20779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908d384adc744a38a456ffbac58a0463" candidate_uuid: "b5c4d76c502647cbae2946723165732b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f282c3ee4622ab83463143d2bb72"
I20240503 02:03:47.666460 20703 raft_consensus.cc:3037] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.666656 20779 raft_consensus.cc:3037] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.670224 20828 leader_election.cc:288] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 election: Requested vote from peers 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755), 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097)
I20240503 02:03:47.673193 20779 raft_consensus.cc:2454] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c4d76c502647cbae2946723165732b in term 1.
I20240503 02:03:47.673944 20703 raft_consensus.cc:2454] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c4d76c502647cbae2946723165732b in term 1.
I20240503 02:03:47.674465 20596 leader_election.cc:302] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b; no voters:
I20240503 02:03:47.675676 20828 raft_consensus.cc:2781] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:47.676378 20828 raft_consensus.cc:686] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 1 LEADER]: Becoming Leader. State: Replica: b5c4d76c502647cbae2946723165732b, State: Running, Role: LEADER
I20240503 02:03:47.677158 20779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908d384adc744a38a456ffbac58a0463" candidate_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f282c3ee4622ab83463143d2bb72" is_pre_election: true
I20240503 02:03:47.677093 20828 consensus_queue.cc:237] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.678121 20779 raft_consensus.cc:2379] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49d3adb9fdfc47a8a9760ac9be6f19b2 in current term 1: Already voted for candidate b5c4d76c502647cbae2946723165732b in this term.
I20240503 02:03:47.678217 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908d384adc744a38a456ffbac58a0463" candidate_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b" is_pre_election: true
I20240503 02:03:47.680442 20669 leader_election.cc:302] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49d3adb9fdfc47a8a9760ac9be6f19b2; no voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b
I20240503 02:03:47.681242 20833 raft_consensus.cc:2726] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:03:47.686254 20527 catalog_manager.cc:5484] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b reported cstate change: term changed from 0 to 1, leader changed from <none> to b5c4d76c502647cbae2946723165732b (127.19.126.1). New cstate: current_term: 1 leader_uuid: "b5c4d76c502647cbae2946723165732b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } health_report { overall_health: UNKNOWN } } }
I20240503 02:03:47.786553 20832 raft_consensus.cc:482] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.787010 20832 raft_consensus.cc:504] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.788484 20832 leader_election.cc:288] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:47.789392 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe84da27587452aa191b1d7eafafad9" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b" is_pre_election: true
I20240503 02:03:47.789430 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe84da27587452aa191b1d7eafafad9" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" is_pre_election: true
I20240503 02:03:47.790004 20630 raft_consensus.cc:2454] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 0.
I20240503 02:03:47.790036 20703 raft_consensus.cc:2454] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 0.
I20240503 02:03:47.790930 20743 leader_election.cc:302] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b; no voters:
I20240503 02:03:47.791501 20832 raft_consensus.cc:2781] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:03:47.791771 20832 raft_consensus.cc:482] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:03:47.792022 20832 raft_consensus.cc:3037] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.796129 20832 raft_consensus.cc:504] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.797502 20832 leader_election.cc:288] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 election: Requested vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2:37755)
I20240503 02:03:47.798267 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe84da27587452aa191b1d7eafafad9" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b"
I20240503 02:03:47.798368 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe84da27587452aa191b1d7eafafad9" candidate_uuid: "1f03f282c3ee4622ab83463143d2bb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2"
I20240503 02:03:47.798700 20630 raft_consensus.cc:3037] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.798846 20703 raft_consensus.cc:3037] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.802754 20630 raft_consensus.cc:2454] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 1.
I20240503 02:03:47.802865 20703 raft_consensus.cc:2454] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f03f282c3ee4622ab83463143d2bb72 in term 1.
I20240503 02:03:47.803588 20743 leader_election.cc:302] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b; no voters:
I20240503 02:03:47.804184 20832 raft_consensus.cc:2781] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:47.804535 20832 raft_consensus.cc:686] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Becoming Leader. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Running, Role: LEADER
I20240503 02:03:47.805228 20832 consensus_queue.cc:237] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.811113 20527 catalog_manager.cc:5484] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3). New cstate: current_term: 1 leader_uuid: "1f03f282c3ee4622ab83463143d2bb72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } health_report { overall_health: HEALTHY } } }
I20240503 02:03:47.889973 20828 consensus_queue.cc:1035] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:47.900750 20856 consensus_queue.cc:1035] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:03:47.921900 20833 raft_consensus.cc:482] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.922309 20833 raft_consensus.cc:504] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.923789 20833 leader_election.cc:288] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097)
I20240503 02:03:47.924522 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f3f4ef8a70481592fc168f64dedaf8" candidate_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b" is_pre_election: true
I20240503 02:03:47.924688 20779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f3f4ef8a70481592fc168f64dedaf8" candidate_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f282c3ee4622ab83463143d2bb72" is_pre_election: true
I20240503 02:03:47.925401 20779 raft_consensus.cc:2379] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49d3adb9fdfc47a8a9760ac9be6f19b2 in current term 1: Already voted for candidate b5c4d76c502647cbae2946723165732b in this term.
I20240503 02:03:47.926745 20671 leader_election.cc:302] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49d3adb9fdfc47a8a9760ac9be6f19b2; no voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b
I20240503 02:03:47.927326 20833 raft_consensus.cc:3037] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.931721 20833 raft_consensus.cc:2726] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:03:47.932163 20833 raft_consensus.cc:482] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.932596 20833 raft_consensus.cc:504] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.933959 20833 leader_election.cc:288] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097)
I20240503 02:03:47.934724 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eabb3b19a3744efa5234554306453b9" candidate_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b" is_pre_election: true
I20240503 02:03:47.934809 20779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eabb3b19a3744efa5234554306453b9" candidate_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f282c3ee4622ab83463143d2bb72" is_pre_election: true
I20240503 02:03:47.935312 20630 raft_consensus.cc:2379] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49d3adb9fdfc47a8a9760ac9be6f19b2 in current term 1: Already voted for candidate 1f03f282c3ee4622ab83463143d2bb72 in this term.
I20240503 02:03:47.936161 20669 leader_election.cc:302] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49d3adb9fdfc47a8a9760ac9be6f19b2; no voters: 1f03f282c3ee4622ab83463143d2bb72, b5c4d76c502647cbae2946723165732b
I20240503 02:03:47.936709 20833 raft_consensus.cc:3037] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.941480 20833 raft_consensus.cc:2726] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:03:47.945240 20833 raft_consensus.cc:482] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:47.945585 20833 raft_consensus.cc:504] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.946885 20833 leader_election.cc:288] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097)
I20240503 02:03:47.947605 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866cc7b0167a4994bb6ff0a24fbadefb" candidate_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b" is_pre_election: true
I20240503 02:03:47.947835 20779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866cc7b0167a4994bb6ff0a24fbadefb" candidate_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f282c3ee4622ab83463143d2bb72" is_pre_election: true
I20240503 02:03:47.948093 20630 raft_consensus.cc:2454] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49d3adb9fdfc47a8a9760ac9be6f19b2 in term 0.
I20240503 02:03:47.948343 20779 raft_consensus.cc:2454] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49d3adb9fdfc47a8a9760ac9be6f19b2 in term 0.
I20240503 02:03:47.948935 20669 leader_election.cc:302] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d3adb9fdfc47a8a9760ac9be6f19b2, b5c4d76c502647cbae2946723165732b; no voters:
I20240503 02:03:47.949674 20833 raft_consensus.cc:2781] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:03:47.949918 20833 raft_consensus.cc:482] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:03:47.950163 20833 raft_consensus.cc:3037] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.954474 20833 raft_consensus.cc:504] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.955724 20833 leader_election.cc:288] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [CANDIDATE]: Term 1 election: Requested vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097)
I20240503 02:03:47.956380 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866cc7b0167a4994bb6ff0a24fbadefb" candidate_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c4d76c502647cbae2946723165732b"
I20240503 02:03:47.956542 20779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866cc7b0167a4994bb6ff0a24fbadefb" candidate_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f03f282c3ee4622ab83463143d2bb72"
I20240503 02:03:47.956887 20630 raft_consensus.cc:3037] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.957010 20779 raft_consensus.cc:3037] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:47.961259 20630 raft_consensus.cc:2454] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d3adb9fdfc47a8a9760ac9be6f19b2 in term 1.
I20240503 02:03:47.961417 20779 raft_consensus.cc:2454] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d3adb9fdfc47a8a9760ac9be6f19b2 in term 1.
I20240503 02:03:47.962270 20671 leader_election.cc:302] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f03f282c3ee4622ab83463143d2bb72, 49d3adb9fdfc47a8a9760ac9be6f19b2; no voters:
I20240503 02:03:47.962977 20833 raft_consensus.cc:2781] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:47.963322 20833 raft_consensus.cc:686] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 LEADER]: Becoming Leader. State: Replica: 49d3adb9fdfc47a8a9760ac9be6f19b2, State: Running, Role: LEADER
I20240503 02:03:47.963873 20833 consensus_queue.cc:237] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:47.970610 20526 catalog_manager.cc:5484] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2). New cstate: current_term: 1 leader_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } health_report { overall_health: UNKNOWN } } }
I20240503 02:03:48.009967 20856 consensus_queue.cc:1035] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:48.018841 20841 consensus_queue.cc:1035] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:48.030774 20832 consensus_queue.cc:1035] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:48.041028 20832 consensus_queue.cc:1035] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:48.051028 20832 consensus_queue.cc:1035] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:48.063482 20854 consensus_queue.cc:1035] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:48.154821 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:03:48.161324 20886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:48.162158 20887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:48.162861 20841 consensus_queue.cc:1035] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240503 02:03:48.163535 20889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:48.164121 20854 consensus_queue.cc:1035] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:48.165648 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:48.167023 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:48.167287 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:48.167546 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701828167528 us; error 0 us; skew 500 ppm
I20240503 02:03:48.168277 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:48.179826 20828 consensus_queue.cc:1035] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:03:48.188483 19960 webserver.cc:460] Webserver started at http://127.19.126.4:36523/ using document root <none> and password file <none>
I20240503 02:03:48.189141 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:48.189361 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:48.189656 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:48.190809 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-3-root/instance:
uuid: "205a3c6b3a8d4a209f66b86649a8ea7b"
format_stamp: "Formatted at 2024-05-03 02:03:48 on dist-test-slave-8rwr"
I20240503 02:03:48.192880 20854 consensus_queue.cc:1035] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:48.202517 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.008s sys 0.004s
I20240503 02:03:48.208892 20899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:48.209851 19960 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20240503 02:03:48.210326 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-3-root
uuid: "205a3c6b3a8d4a209f66b86649a8ea7b"
format_stamp: "Formatted at 2024-05-03 02:03:48 on dist-test-slave-8rwr"
I20240503 02:03:48.210847 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701820461004-19960-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:48.228983 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:48.230022 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:48.231402 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:03:48.233616 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:03:48.233819 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:48.234035 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:03:48.234182 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:48.276129 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.4:43569
I20240503 02:03:48.276208 20961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.4:43569 every 8 connection(s)
I20240503 02:03:48.289981 20962 heartbeater.cc:347] Connected to a master server at 127.19.126.62:34943
I20240503 02:03:48.290334 20962 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:48.290999 20962 heartbeater.cc:511] Master 127.19.126.62:34943 requested a full tablet report, sending...
I20240503 02:03:48.293493 20528 ts_manager.cc:194] Registered new tserver with Master: 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4:43569)
I20240503 02:03:48.294668 20528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49298
I20240503 02:03:48.350409 20854 consensus_queue.cc:1035] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:48.360805 20832 consensus_queue.cc:1035] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:48.493038 20833 consensus_queue.cc:1035] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:48.504410 20969 consensus_queue.cc:1035] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 02:03:49.296631 20962 heartbeater.cc:503] Master 127.19.126.62:34943 was elected leader, sending a full tablet report...
I20240503 02:03:49.580966 20581 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:03:49.596913 20779 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea2d295a06734d2b8bd5bfae41bf51d3"
dest_uuid: "1f03f282c3ee4622ab83463143d2bb72"
mode: GRACEFUL
new_leader_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2"
from {username='slave'} at 127.0.0.1:46174
I20240503 02:03:49.597556 20779 raft_consensus.cc:595] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Received request to transfer leadership to TS 49d3adb9fdfc47a8a9760ac9be6f19b2
I20240503 02:03:49.598681 20581 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 02:03:49.599160 20581 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:03:49.754290 20882 raft_consensus.cc:979] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72: : Instructing follower 49d3adb9fdfc47a8a9760ac9be6f19b2 to start an election
I20240503 02:03:49.754752 20974 raft_consensus.cc:1067] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Signalling peer 49d3adb9fdfc47a8a9760ac9be6f19b2 to start an election
I20240503 02:03:49.756134 20704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea2d295a06734d2b8bd5bfae41bf51d3"
dest_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2"
from {username='slave'} at 127.0.0.1:53446
I20240503 02:03:49.756707 20704 raft_consensus.cc:482] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:03:49.757066 20704 raft_consensus.cc:3037] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:03:49.761700 20704 raft_consensus.cc:504] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:49.763154 20704 leader_election.cc:288] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [CANDIDATE]: Term 2 election: Requested vote from peers b5c4d76c502647cbae2946723165732b (127.19.126.1:35423), 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3:38097)
I20240503 02:03:49.763955 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2d295a06734d2b8bd5bfae41bf51d3" candidate_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b5c4d76c502647cbae2946723165732b"
I20240503 02:03:49.764039 20779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2d295a06734d2b8bd5bfae41bf51d3" candidate_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1f03f282c3ee4622ab83463143d2bb72"
I20240503 02:03:49.764681 20630 raft_consensus.cc:3037] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:03:49.764745 20779 raft_consensus.cc:3032] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:03:49.765195 20779 raft_consensus.cc:729] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1f03f282c3ee4622ab83463143d2bb72, State: Running, Role: LEADER
I20240503 02:03:49.766033 20779 consensus_queue.cc:260] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:49.767293 20779 raft_consensus.cc:3037] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:03:49.771086 20630 raft_consensus.cc:2454] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d3adb9fdfc47a8a9760ac9be6f19b2 in term 2.
I20240503 02:03:49.772182 20669 leader_election.cc:302] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d3adb9fdfc47a8a9760ac9be6f19b2, b5c4d76c502647cbae2946723165732b; no voters:
I20240503 02:03:49.772823 20970 raft_consensus.cc:2781] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:03:49.772944 20779 raft_consensus.cc:2454] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d3adb9fdfc47a8a9760ac9be6f19b2 in term 2.
I20240503 02:03:49.773907 20970 raft_consensus.cc:686] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 2 LEADER]: Becoming Leader. State: Replica: 49d3adb9fdfc47a8a9760ac9be6f19b2, State: Running, Role: LEADER
I20240503 02:03:49.774636 20970 consensus_queue.cc:237] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } }
I20240503 02:03:49.781150 20528 catalog_manager.cc:5484] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 reported cstate change: term changed from 1 to 2, leader changed from 1f03f282c3ee4622ab83463143d2bb72 (127.19.126.3) to 49d3adb9fdfc47a8a9760ac9be6f19b2 (127.19.126.2). New cstate: current_term: 2 leader_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } health_report { overall_health: UNKNOWN } } }
I20240503 02:03:50.252285 20630 raft_consensus.cc:1261] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b [term 2 FOLLOWER]: Refusing update from remote peer 49d3adb9fdfc47a8a9760ac9be6f19b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:03:50.253433 20970 consensus_queue.cc:1035] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:03:50.263310 20779 raft_consensus.cc:1261] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 2 FOLLOWER]: Refusing update from remote peer 49d3adb9fdfc47a8a9760ac9be6f19b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:03:50.265107 20970 consensus_queue.cc:1035] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:03:51.600260 20581 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:03:51.606659 20581 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:03:51.607013 20581 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:03:53.594800 20779 consensus_queue.cc:237] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } attrs { replace: true } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } }
I20240503 02:03:53.602510 20704 raft_consensus.cc:1261] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Refusing update from remote peer 1f03f282c3ee4622ab83463143d2bb72: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:03:53.603461 20630 raft_consensus.cc:1261] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Refusing update from remote peer 1f03f282c3ee4622ab83463143d2bb72: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:03:53.604893 20989 consensus_queue.cc:1035] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:03:53.606103 20993 consensus_queue.cc:1035] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:03:53.608189 20581 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240503 02:03:53.616968 20742 consensus_peers.cc:485] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 -> Peer 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4:43569): Couldn't send request to peer 205a3c6b3a8d4a209f66b86649a8ea7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fe84da27587452aa191b1d7eafafad9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:03:53.618794 20993 raft_consensus.cc:2932] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } attrs { replace: true } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.619839 20704 raft_consensus.cc:2932] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } attrs { replace: true } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.621178 20630 raft_consensus.cc:2932] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } attrs { replace: true } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.624888 20581 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:03:53.625540 20581 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:03:53.633662 20528 catalog_manager.cc:5484] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 reported cstate change: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New cstate: current_term: 1 leader_uuid: "1f03f282c3ee4622ab83463143d2bb72" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } attrs { replace: true } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.634752 20779 consensus_queue.cc:237] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } attrs { replace: true } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } }
W20240503 02:03:53.642314 20742 consensus_peers.cc:485] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 -> Peer 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4:43569): Couldn't send request to peer 205a3c6b3a8d4a209f66b86649a8ea7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0eabb3b19a3744efa5234554306453b9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:03:53.642978 20630 raft_consensus.cc:1261] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Refusing update from remote peer 1f03f282c3ee4622ab83463143d2bb72: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:03:53.642973 20704 raft_consensus.cc:1261] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Refusing update from remote peer 1f03f282c3ee4622ab83463143d2bb72: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:03:53.644505 20993 consensus_queue.cc:1035] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:03:53.645381 20989 consensus_queue.cc:1035] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:03:53.654305 20989 raft_consensus.cc:2932] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } attrs { replace: true } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.655371 20630 raft_consensus.cc:2932] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } attrs { replace: true } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.659848 20704 raft_consensus.cc:2932] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } attrs { replace: true } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.672075 20526 catalog_manager.cc:5484] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b reported cstate change: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New cstate: current_term: 1 leader_uuid: "1f03f282c3ee4622ab83463143d2bb72" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } attrs { replace: true } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.679589 20630 consensus_queue.cc:237] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } attrs { replace: true } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } }
I20240503 02:03:53.686725 20704 raft_consensus.cc:1261] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Refusing update from remote peer b5c4d76c502647cbae2946723165732b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:03:53.688225 21009 consensus_queue.cc:1035] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:03:53.689064 20779 raft_consensus.cc:1261] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Refusing update from remote peer b5c4d76c502647cbae2946723165732b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:03:53.690204 21009 consensus_queue.cc:1035] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:03:53.696270 20828 raft_consensus.cc:2932] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } attrs { replace: true } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.697782 20704 raft_consensus.cc:2932] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } attrs { replace: true } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
W20240503 02:03:53.699374 20593 consensus_peers.cc:485] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b -> Peer 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4:43569): Couldn't send request to peer 205a3c6b3a8d4a209f66b86649a8ea7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3adc50685334bef9080b3ef04a50252. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:03:53.708982 20630 consensus_queue.cc:237] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } attrs { replace: true } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } }
I20240503 02:03:53.712206 20528 catalog_manager.cc:5484] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 reported cstate change: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New cstate: current_term: 1 leader_uuid: "b5c4d76c502647cbae2946723165732b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } attrs { replace: true } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.711892 20779 raft_consensus.cc:2932] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } attrs { replace: true } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
W20240503 02:03:53.741803 20593 consensus_peers.cc:485] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b -> Peer 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4:43569): Couldn't send request to peer 205a3c6b3a8d4a209f66b86649a8ea7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 908d384adc744a38a456ffbac58a0463. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:03:53.741813 20704 raft_consensus.cc:1261] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Refusing update from remote peer b5c4d76c502647cbae2946723165732b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:03:53.741916 20778 raft_consensus.cc:1261] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Refusing update from remote peer b5c4d76c502647cbae2946723165732b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:03:53.743520 21013 consensus_queue.cc:1035] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:03:53.744239 21009 consensus_queue.cc:1035] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:03:53.752947 20828 raft_consensus.cc:2932] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } attrs { replace: true } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.754192 20704 raft_consensus.cc:2932] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } attrs { replace: true } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.756060 20778 raft_consensus.cc:2932] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } attrs { replace: true } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.766973 20528 catalog_manager.cc:5484] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 reported cstate change: config changed from index -1 to 2, NON_VOTER 205a3c6b3a8d4a209f66b86649a8ea7b (127.19.126.4) added. New cstate: current_term: 1 leader_uuid: "b5c4d76c502647cbae2946723165732b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b5c4d76c502647cbae2946723165732b" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 35423 } } peers { permanent_uuid: "49d3adb9fdfc47a8a9760ac9be6f19b2" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 37755 } attrs { replace: true } } peers { permanent_uuid: "1f03f282c3ee4622ab83463143d2bb72" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 38097 } } peers { permanent_uuid: "205a3c6b3a8d4a209f66b86649a8ea7b" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43569 } attrs { promote: true } } }
I20240503 02:03:53.796185 19960 tablet_server.cc:178] TabletServer@127.19.126.1:0 shutting down...
W20240503 02:03:53.799826 20513 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20240503 02:03:53.803620 20576 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:03:53.803979 20576 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:03:53.814414 20576 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:03:53.814679 20576 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 02:03:53.824887 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:03:53.825657 19960 tablet_replica.cc:330] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b: stopping tablet replica
I20240503 02:03:53.826507 19960 raft_consensus.cc:2229] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:53.827054 19960 raft_consensus.cc:2258] T 3fe84da27587452aa191b1d7eafafad9 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.829422 19960 tablet_replica.cc:330] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b: stopping tablet replica
I20240503 02:03:53.830090 19960 raft_consensus.cc:2229] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:53.830868 19960 raft_consensus.cc:2258] T 908d384adc744a38a456ffbac58a0463 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.833273 19960 tablet_replica.cc:330] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b: stopping tablet replica
I20240503 02:03:53.833767 19960 raft_consensus.cc:2229] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:53.834203 19960 raft_consensus.cc:2258] T ea2d295a06734d2b8bd5bfae41bf51d3 P b5c4d76c502647cbae2946723165732b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.836372 19960 tablet_replica.cc:330] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b: stopping tablet replica
I20240503 02:03:53.836802 19960 raft_consensus.cc:2229] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:53.837179 19960 raft_consensus.cc:2258] T 0eabb3b19a3744efa5234554306453b9 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.839370 19960 tablet_replica.cc:330] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b: stopping tablet replica
I20240503 02:03:53.839788 19960 raft_consensus.cc:2229] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:53.840569 19960 raft_consensus.cc:2258] T 38f3f4ef8a70481592fc168f64dedaf8 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.842573 19960 tablet_replica.cc:330] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b: stopping tablet replica
I20240503 02:03:53.842983 19960 raft_consensus.cc:2229] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:53.843310 19960 raft_consensus.cc:2258] T 866cc7b0167a4994bb6ff0a24fbadefb P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.844760 19960 tablet_replica.cc:330] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b: stopping tablet replica
I20240503 02:03:53.845163 19960 raft_consensus.cc:2229] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:53.845575 19960 raft_consensus.cc:2258] T 70e5e4d17b9b47b485d6ee938059c45b P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.847138 19960 tablet_replica.cc:330] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b: stopping tablet replica
I20240503 02:03:53.847544 19960 raft_consensus.cc:2229] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:53.848295 19960 raft_consensus.cc:2258] T e3adc50685334bef9080b3ef04a50252 P b5c4d76c502647cbae2946723165732b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.877822 19960 tablet_server.cc:194] TabletServer@127.19.126.1:0 shutdown complete.
I20240503 02:03:53.899106 19960 tablet_server.cc:178] TabletServer@127.19.126.2:0 shutting down...
W20240503 02:03:53.922232 20743 consensus_peers.cc:485] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 -> Peer b5c4d76c502647cbae2946723165732b (127.19.126.1:35423): Couldn't send request to peer b5c4d76c502647cbae2946723165732b. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:35423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:03:53.937412 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:03:53.938180 19960 tablet_replica.cc:330] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2: stopping tablet replica
I20240503 02:03:53.939074 19960 raft_consensus.cc:2229] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:53.939499 19960 raft_consensus.cc:2258] T 908d384adc744a38a456ffbac58a0463 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.942380 19960 tablet_replica.cc:330] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2: stopping tablet replica
I20240503 02:03:53.942888 19960 raft_consensus.cc:2229] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:53.943533 19960 raft_consensus.cc:2258] T 866cc7b0167a4994bb6ff0a24fbadefb P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.945514 19960 tablet_replica.cc:330] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2: stopping tablet replica
I20240503 02:03:53.945981 19960 raft_consensus.cc:2229] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:53.946426 19960 raft_consensus.cc:2258] T 38f3f4ef8a70481592fc168f64dedaf8 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.948261 19960 tablet_replica.cc:330] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2: stopping tablet replica
I20240503 02:03:53.948697 19960 raft_consensus.cc:2229] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:53.949128 19960 raft_consensus.cc:2258] T 0eabb3b19a3744efa5234554306453b9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.951246 19960 tablet_replica.cc:330] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2: stopping tablet replica
I20240503 02:03:53.951833 19960 raft_consensus.cc:2229] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:53.952329 19960 raft_consensus.cc:2258] T 70e5e4d17b9b47b485d6ee938059c45b P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.954322 19960 tablet_replica.cc:330] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2: stopping tablet replica
I20240503 02:03:53.954908 19960 raft_consensus.cc:2229] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:03:53.955658 19960 raft_consensus.cc:2258] T ea2d295a06734d2b8bd5bfae41bf51d3 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.957422 19960 tablet_replica.cc:330] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2: stopping tablet replica
I20240503 02:03:53.957990 19960 raft_consensus.cc:2229] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:53.958423 19960 raft_consensus.cc:2258] T 3fe84da27587452aa191b1d7eafafad9 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.960470 19960 tablet_replica.cc:330] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2: stopping tablet replica
I20240503 02:03:53.961057 19960 raft_consensus.cc:2229] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:53.961503 19960 raft_consensus.cc:2258] T e3adc50685334bef9080b3ef04a50252 P 49d3adb9fdfc47a8a9760ac9be6f19b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:53.989862 19960 tablet_server.cc:194] TabletServer@127.19.126.2:0 shutdown complete.
I20240503 02:03:54.009253 19960 tablet_server.cc:178] TabletServer@127.19.126.3:0 shutting down...
W20240503 02:03:54.013803 20576 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:03:54.014045 20576 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:03:54.018497 20576 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:03:54.018726 20576 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 02:03:54.029630 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:03:54.030320 19960 tablet_replica.cc:330] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72: stopping tablet replica
I20240503 02:03:54.030915 19960 raft_consensus.cc:2229] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:54.031309 19960 raft_consensus.cc:2258] T ea2d295a06734d2b8bd5bfae41bf51d3 P 1f03f282c3ee4622ab83463143d2bb72 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:54.032920 19960 tablet_replica.cc:330] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72: stopping tablet replica
I20240503 02:03:54.033411 19960 raft_consensus.cc:2229] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:54.033758 19960 raft_consensus.cc:2258] T 866cc7b0167a4994bb6ff0a24fbadefb P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:54.035442 19960 tablet_replica.cc:330] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72: stopping tablet replica
I20240503 02:03:54.035902 19960 raft_consensus.cc:2229] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:54.036252 19960 raft_consensus.cc:2258] T 38f3f4ef8a70481592fc168f64dedaf8 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:54.037801 19960 tablet_replica.cc:330] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72: stopping tablet replica
I20240503 02:03:54.038267 19960 raft_consensus.cc:2229] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:54.039074 19960 raft_consensus.cc:2258] T 0eabb3b19a3744efa5234554306453b9 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:54.041587 19960 tablet_replica.cc:330] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72: stopping tablet replica
I20240503 02:03:54.042081 19960 raft_consensus.cc:2229] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:54.042724 19960 raft_consensus.cc:2258] T 70e5e4d17b9b47b485d6ee938059c45b P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:54.044253 19960 tablet_replica.cc:330] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72: stopping tablet replica
I20240503 02:03:54.044706 19960 raft_consensus.cc:2229] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:54.045533 19960 raft_consensus.cc:2258] T 3fe84da27587452aa191b1d7eafafad9 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:54.047381 19960 tablet_replica.cc:330] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72: stopping tablet replica
I20240503 02:03:54.047840 19960 raft_consensus.cc:2229] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:54.048238 19960 raft_consensus.cc:2258] T 908d384adc744a38a456ffbac58a0463 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:54.050259 19960 tablet_replica.cc:330] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72: stopping tablet replica
I20240503 02:03:54.050691 19960 raft_consensus.cc:2229] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:03:54.051090 19960 raft_consensus.cc:2258] T e3adc50685334bef9080b3ef04a50252 P 1f03f282c3ee4622ab83463143d2bb72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:54.076874 19960 tablet_server.cc:194] TabletServer@127.19.126.3:0 shutdown complete.
I20240503 02:03:54.091475 19960 tablet_server.cc:178] TabletServer@127.19.126.4:0 shutting down...
I20240503 02:03:54.109769 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:03:54.124886 19960 tablet_server.cc:194] TabletServer@127.19.126.4:0 shutdown complete.
I20240503 02:03:54.131862 19960 master.cc:537] Master@127.19.126.62:34943 shutting down...
I20240503 02:03:54.147526 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:54.147976 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:54.148267 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P b5a12468480143ea9365bbd30798bc57: stopping tablet replica
I20240503 02:03:54.167449 19960 master.cc:552] Master@127.19.126.62:34943 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (8763 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240503 02:03:54.204650 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:35085,127.19.126.61:39913,127.19.126.60:40399
I20240503 02:03:54.205866 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:03:54.211439 21019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:54.212112 21020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:54.212886 21022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:54.213629 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:54.214300 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:54.214480 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:54.214627 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701834214602 us; error 0 us; skew 500 ppm
I20240503 02:03:54.215133 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:54.217391 19960 webserver.cc:460] Webserver started at http://127.19.126.62:42849/ using document root <none> and password file <none>
I20240503 02:03:54.217847 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:54.218019 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:54.218274 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:54.219357 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "3d41f80c564a44b0ba1d2eda1fc4a013"
format_stamp: "Formatted at 2024-05-03 02:03:54 on dist-test-slave-8rwr"
I20240503 02:03:54.223537 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:03:54.226516 21027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:54.227161 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:03:54.227411 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "3d41f80c564a44b0ba1d2eda1fc4a013"
format_stamp: "Formatted at 2024-05-03 02:03:54 on dist-test-slave-8rwr"
I20240503 02:03:54.227685 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:54.235936 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:54.236985 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:54.271404 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:35085
I20240503 02:03:54.271495 21078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:35085 every 8 connection(s)
I20240503 02:03:54.274925 21079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:54.275187 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:03:54.280958 21081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:54.281787 21079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:03:54.282011 21082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:54.284505 19960 server_base.cc:1034] running on GCE node
W20240503 02:03:54.284731 21084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:54.285825 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:54.286161 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:54.286396 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701834286379 us; error 0 us; skew 500 ppm
I20240503 02:03:54.286888 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:54.289520 19960 webserver.cc:460] Webserver started at http://127.19.126.61:42775/ using document root <none> and password file <none>
I20240503 02:03:54.289938 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:54.290088 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:54.290329 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:54.291292 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-1-root/instance:
uuid: "77e48a58464e47b2b49dc989f0a976c5"
format_stamp: "Formatted at 2024-05-03 02:03:54 on dist-test-slave-8rwr"
I20240503 02:03:54.294963 21079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:03:54.296222 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:03:54.300194 21092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:54.301059 19960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
W20240503 02:03:54.301308 21079 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.19.126.61:39913: Network error: Client connection negotiation failed: client connection to 127.19.126.61:39913: connect: Connection refused (error 111)
I20240503 02:03:54.301365 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-1-root
uuid: "77e48a58464e47b2b49dc989f0a976c5"
format_stamp: "Formatted at 2024-05-03 02:03:54 on dist-test-slave-8rwr"
I20240503 02:03:54.301805 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:54.325577 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:54.326624 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:54.351969 21079 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } attempt: 1
W20240503 02:03:54.356325 21079 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.19.126.61:39913: Network error: Client connection negotiation failed: client connection to 127.19.126.61:39913: connect: Connection refused (error 111)
I20240503 02:03:54.362609 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.61:39913
I20240503 02:03:54.362725 21143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.61:39913 every 8 connection(s)
I20240503 02:03:54.366235 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:03:54.366341 21144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:03:54.372014 21146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:54.372925 21147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:54.374986 21149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:54.374946 21144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:03:54.376246 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:54.377316 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:54.377583 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:54.377737 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701834377721 us; error 0 us; skew 500 ppm
I20240503 02:03:54.378202 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:54.380833 19960 webserver.cc:460] Webserver started at http://127.19.126.60:39973/ using document root <none> and password file <none>
I20240503 02:03:54.381335 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:54.381510 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:54.381760 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:54.382862 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-2-root/instance:
uuid: "5c692f4431484e8ea037a15082290762"
format_stamp: "Formatted at 2024-05-03 02:03:54 on dist-test-slave-8rwr"
I20240503 02:03:54.385742 21144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:03:54.387578 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:03:54.391391 21156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:54.392313 19960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:03:54.392642 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-2-root
uuid: "5c692f4431484e8ea037a15082290762"
format_stamp: "Formatted at 2024-05-03 02:03:54 on dist-test-slave-8rwr"
I20240503 02:03:54.393000 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:54.397233 21144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:03:54.402758 21144 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.19.126.60:40399: Network error: Client connection negotiation failed: client connection to 127.19.126.60:40399: connect: Connection refused (error 111)
I20240503 02:03:54.413115 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:54.414265 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:54.419803 21079 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } attempt: 2
I20240503 02:03:54.430125 21079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } has no permanent_uuid. Determining permanent_uuid...
W20240503 02:03:54.435101 21079 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.19.126.60:40399: Network error: Client connection negotiation failed: client connection to 127.19.126.60:40399: connect: Connection refused (error 111)
I20240503 02:03:54.453509 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.60:40399
I20240503 02:03:54.453603 21208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.60:40399 every 8 connection(s)
I20240503 02:03:54.453769 21079 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } attempt: 1
I20240503 02:03:54.457263 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:03:54.458077 21210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:54.465004 21210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:03:54.466187 21144 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } attempt: 1
I20240503 02:03:54.472085 21079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013: Bootstrap starting.
I20240503 02:03:54.477850 21079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:54.481326 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5: Bootstrap starting.
I20240503 02:03:54.482298 21079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013: No bootstrap required, opened a new log
I20240503 02:03:54.483058 21210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:03:54.484876 21079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:03:54.485661 21079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:54.486063 21144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:54.485968 21079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d41f80c564a44b0ba1d2eda1fc4a013, State: Initialized, Role: FOLLOWER
I20240503 02:03:54.488788 21079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:03:54.490978 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5: No bootstrap required, opened a new log
I20240503 02:03:54.491019 21215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } } }
I20240503 02:03:54.491953 21215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:54.492717 21079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:03:54.493649 21144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:03:54.494447 21144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:54.494793 21144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e48a58464e47b2b49dc989f0a976c5, State: Initialized, Role: FOLLOWER
I20240503 02:03:54.495532 21210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } has no permanent_uuid. Determining permanent_uuid...
I20240503 02:03:54.495532 21144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:03:54.505827 21144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:03:54.506778 21218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } } }
I20240503 02:03:54.507616 21218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:54.519608 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762: Bootstrap starting.
I20240503 02:03:54.523278 21215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:03:54.523891 21215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:03:54.527570 21210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:54.529475 21119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e48a58464e47b2b49dc989f0a976c5" is_pre_election: true
I20240503 02:03:54.530539 21119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d41f80c564a44b0ba1d2eda1fc4a013 in term 0.
I20240503 02:03:54.532469 21031 leader_election.cc:302] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d41f80c564a44b0ba1d2eda1fc4a013, 77e48a58464e47b2b49dc989f0a976c5; no voters:
I20240503 02:03:54.533910 21215 leader_election.cc:288] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e48a58464e47b2b49dc989f0a976c5 (127.19.126.61:39913), 5c692f4431484e8ea037a15082290762 (127.19.126.60:40399)
I20240503 02:03:54.534696 21215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:03:54.535185 21215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:03:54.535929 21215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:54.540620 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762: No bootstrap required, opened a new log
I20240503 02:03:54.543939 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240503 02:03:54.545763 21215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:03:54.546145 21210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:03:54.547541 21210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:54.548038 21210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c692f4431484e8ea037a15082290762, State: Initialized, Role: FOLLOWER
I20240503 02:03:54.548609 21119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e48a58464e47b2b49dc989f0a976c5"
I20240503 02:03:54.549408 21119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:54.550309 21215 leader_election.cc:288] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [CANDIDATE]: Term 1 election: Requested vote from peers 77e48a58464e47b2b49dc989f0a976c5 (127.19.126.61:39913), 5c692f4431484e8ea037a15082290762 (127.19.126.60:40399)
I20240503 02:03:54.550073 21210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
W20240503 02:03:54.552867 21237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:03:54.553195 21237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:03:54.557672 21119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d41f80c564a44b0ba1d2eda1fc4a013 in term 1.
I20240503 02:03:54.558887 21031 leader_election.cc:302] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d41f80c564a44b0ba1d2eda1fc4a013, 77e48a58464e47b2b49dc989f0a976c5; no voters:
I20240503 02:03:54.559973 21215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:54.560678 21215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 1 LEADER]: Becoming Leader. State: Replica: 3d41f80c564a44b0ba1d2eda1fc4a013, State: Running, Role: LEADER
I20240503 02:03:54.560987 21210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:03:54.561419 21183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c692f4431484e8ea037a15082290762"
I20240503 02:03:54.561466 21215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:03:54.562172 21183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:54.566797 21241 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240503 02:03:54.567428 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 02:03:54.561084 21184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c692f4431484e8ea037a15082290762" is_pre_election: true
I20240503 02:03:54.569612 21238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d41f80c564a44b0ba1d2eda1fc4a013. Latest consensus state: current_term: 1 leader_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } } }
I20240503 02:03:54.570389 21238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [sys.catalog]: This master's current role is: LEADER
I20240503 02:03:54.570756 21183 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d41f80c564a44b0ba1d2eda1fc4a013 in term 1.
I20240503 02:03:54.572463 21239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } } }
I20240503 02:03:54.572979 21239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:54.574287 21244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:03:54.578943 21244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:03:54.585635 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:03:54.587445 21255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:03:54.587858 21255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:03:54.588099 21183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 1 FOLLOWER]: Refusing update from remote peer 3d41f80c564a44b0ba1d2eda1fc4a013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:03:54.588099 21119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 1 FOLLOWER]: Refusing update from remote peer 3d41f80c564a44b0ba1d2eda1fc4a013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:03:54.589584 21238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:03:54.590243 21215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:03:54.594653 21256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:54.596582 21257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:54.602880 21218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d41f80c564a44b0ba1d2eda1fc4a013. Latest consensus state: current_term: 1 leader_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } } }
I20240503 02:03:54.603566 21218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:54.603397 21239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d41f80c564a44b0ba1d2eda1fc4a013. Latest consensus state: current_term: 1 leader_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } } }
W20240503 02:03:54.604189 21259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:54.604211 21239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:54.607136 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:54.608497 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:54.608717 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:54.608911 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701834608892 us; error 0 us; skew 500 ppm
I20240503 02:03:54.609591 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:54.617821 21215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } } }
I20240503 02:03:54.618927 21215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [sys.catalog]: This master's current role is: LEADER
I20240503 02:03:54.619505 21218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } } }
I20240503 02:03:54.619896 21215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } } }
I20240503 02:03:54.620646 21215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [sys.catalog]: This master's current role is: LEADER
I20240503 02:03:54.620321 21218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:54.622586 21239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } } }
I20240503 02:03:54.623320 21239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 02:03:54.623797 21244 catalog_manager.cc:1347] Generated new cluster ID: e15c943a1b364ec995c8bbff575437b0
I20240503 02:03:54.624008 21244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:03:54.625833 19960 webserver.cc:460] Webserver started at http://127.19.126.1:34095/ using document root <none> and password file <none>
I20240503 02:03:54.626317 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:54.626502 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:54.626753 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:54.627892 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-0-root/instance:
uuid: "20175b1a16eb4fceb9a20b5bad66720a"
format_stamp: "Formatted at 2024-05-03 02:03:54 on dist-test-slave-8rwr"
I20240503 02:03:54.632797 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:03:54.636201 21272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:54.637027 19960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:03:54.637300 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "20175b1a16eb4fceb9a20b5bad66720a"
format_stamp: "Formatted at 2024-05-03 02:03:54 on dist-test-slave-8rwr"
I20240503 02:03:54.637563 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:54.649215 21244 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:03:54.650801 21244 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:03:54.653163 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:54.654402 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:54.655972 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:03:54.658716 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:03:54.658918 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:54.659148 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:03:54.659332 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:54.671972 21244 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013: Generated new TSK 0
I20240503 02:03:54.672770 21244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:03:54.707875 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:43345
I20240503 02:03:54.707955 21334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:43345 every 8 connection(s)
I20240503 02:03:54.716640 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:03:54.727186 21336 heartbeater.cc:347] Connected to a master server at 127.19.126.61:39913
I20240503 02:03:54.727586 21336 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:54.728339 21336 heartbeater.cc:511] Master 127.19.126.61:39913 requested a full tablet report, sending...
I20240503 02:03:54.729420 21335 heartbeater.cc:347] Connected to a master server at 127.19.126.60:40399
I20240503 02:03:54.729872 21335 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:54.730911 21335 heartbeater.cc:511] Master 127.19.126.60:40399 requested a full tablet report, sending...
I20240503 02:03:54.731444 21109 ts_manager.cc:194] Registered new tserver with Master: 20175b1a16eb4fceb9a20b5bad66720a (127.19.126.1:43345)
I20240503 02:03:54.734942 21174 ts_manager.cc:194] Registered new tserver with Master: 20175b1a16eb4fceb9a20b5bad66720a (127.19.126.1:43345)
I20240503 02:03:54.735703 21337 heartbeater.cc:347] Connected to a master server at 127.19.126.62:35085
I20240503 02:03:54.736084 21337 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:54.736979 21337 heartbeater.cc:511] Master 127.19.126.62:35085 requested a full tablet report, sending...
W20240503 02:03:54.737221 21342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:03:54.738523 21343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:54.740353 21044 ts_manager.cc:194] Registered new tserver with Master: 20175b1a16eb4fceb9a20b5bad66720a (127.19.126.1:43345)
W20240503 02:03:54.740710 21345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:54.741397 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:54.742173 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:03:54.742178 21044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50444
W20240503 02:03:54.742548 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:54.742725 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701834742711 us; error 0 us; skew 500 ppm
I20240503 02:03:54.743343 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:54.745718 19960 webserver.cc:460] Webserver started at http://127.19.126.2:40061/ using document root <none> and password file <none>
I20240503 02:03:54.746166 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:54.746341 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:54.746567 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:54.747866 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-1-root/instance:
uuid: "b29922cf0b9646b18cd743ada34008b0"
format_stamp: "Formatted at 2024-05-03 02:03:54 on dist-test-slave-8rwr"
I20240503 02:03:54.754001 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 02:03:54.757164 21350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:54.757870 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:03:54.758100 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "b29922cf0b9646b18cd743ada34008b0"
format_stamp: "Formatted at 2024-05-03 02:03:54 on dist-test-slave-8rwr"
I20240503 02:03:54.758327 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:54.780062 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:54.781095 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:54.782411 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:03:54.784508 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:03:54.784672 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:54.784894 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:03:54.785029 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:54.822407 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:43891
I20240503 02:03:54.822491 21412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:43891 every 8 connection(s)
I20240503 02:03:54.840273 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:03:54.841609 21414 heartbeater.cc:347] Connected to a master server at 127.19.126.61:39913
I20240503 02:03:54.842005 21414 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:54.842834 21414 heartbeater.cc:511] Master 127.19.126.61:39913 requested a full tablet report, sending...
I20240503 02:03:54.845611 21109 ts_manager.cc:194] Registered new tserver with Master: b29922cf0b9646b18cd743ada34008b0 (127.19.126.2:43891)
I20240503 02:03:54.851075 21413 heartbeater.cc:347] Connected to a master server at 127.19.126.60:40399
I20240503 02:03:54.851420 21413 heartbeater.cc:464] Registering TS with master...
W20240503 02:03:54.851589 21420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:54.852224 21413 heartbeater.cc:511] Master 127.19.126.60:40399 requested a full tablet report, sending...
I20240503 02:03:54.852659 21415 heartbeater.cc:347] Connected to a master server at 127.19.126.62:35085
I20240503 02:03:54.853089 21415 heartbeater.cc:464] Registering TS with master...
W20240503 02:03:54.852227 21421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:54.853945 21415 heartbeater.cc:511] Master 127.19.126.62:35085 requested a full tablet report, sending...
I20240503 02:03:54.855271 21174 ts_manager.cc:194] Registered new tserver with Master: b29922cf0b9646b18cd743ada34008b0 (127.19.126.2:43891)
W20240503 02:03:54.855701 21423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:03:54.856245 21044 ts_manager.cc:194] Registered new tserver with Master: b29922cf0b9646b18cd743ada34008b0 (127.19.126.2:43891)
I20240503 02:03:54.857765 19960 server_base.cc:1034] running on GCE node
I20240503 02:03:54.857851 21044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50450
I20240503 02:03:54.858491 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:03:54.858758 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:03:54.858991 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701834858974 us; error 0 us; skew 500 ppm
I20240503 02:03:54.859560 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:03:54.861889 19960 webserver.cc:460] Webserver started at http://127.19.126.3:34949/ using document root <none> and password file <none>
I20240503 02:03:54.862326 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:03:54.862507 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:03:54.862722 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:03:54.863672 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-2-root/instance:
uuid: "5f79c5fd2765431cb75f650f3ad5970a"
format_stamp: "Formatted at 2024-05-03 02:03:54 on dist-test-slave-8rwr"
I20240503 02:03:54.867657 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240503 02:03:54.870626 21428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:54.871323 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:03:54.871567 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "5f79c5fd2765431cb75f650f3ad5970a"
format_stamp: "Formatted at 2024-05-03 02:03:54 on dist-test-slave-8rwr"
I20240503 02:03:54.871796 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:03:54.890170 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:03:54.891289 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:03:54.892397 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:03:54.894726 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:03:54.894924 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:54.895135 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:03:54.895289 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:03:54.934134 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:46837
I20240503 02:03:54.934201 21490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:46837 every 8 connection(s)
I20240503 02:03:54.963963 21492 heartbeater.cc:347] Connected to a master server at 127.19.126.61:39913
I20240503 02:03:54.964447 21492 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:54.965278 21492 heartbeater.cc:511] Master 127.19.126.61:39913 requested a full tablet report, sending...
I20240503 02:03:54.965481 21491 heartbeater.cc:347] Connected to a master server at 127.19.126.60:40399
I20240503 02:03:54.965842 21491 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:54.966655 21491 heartbeater.cc:511] Master 127.19.126.60:40399 requested a full tablet report, sending...
I20240503 02:03:54.966856 21493 heartbeater.cc:347] Connected to a master server at 127.19.126.62:35085
I20240503 02:03:54.967192 21493 heartbeater.cc:464] Registering TS with master...
I20240503 02:03:54.967978 21493 heartbeater.cc:511] Master 127.19.126.62:35085 requested a full tablet report, sending...
I20240503 02:03:54.968081 21109 ts_manager.cc:194] Registered new tserver with Master: 5f79c5fd2765431cb75f650f3ad5970a (127.19.126.3:46837)
I20240503 02:03:54.970270 21044 ts_manager.cc:194] Registered new tserver with Master: 5f79c5fd2765431cb75f650f3ad5970a (127.19.126.3:46837)
I20240503 02:03:54.970379 21174 ts_manager.cc:194] Registered new tserver with Master: 5f79c5fd2765431cb75f650f3ad5970a (127.19.126.3:46837)
I20240503 02:03:54.971158 19960 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019122543s
I20240503 02:03:54.971638 21044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50466
I20240503 02:03:55.570892 21241 catalog_manager.cc:1259] Loaded cluster ID: e15c943a1b364ec995c8bbff575437b0
I20240503 02:03:55.571156 21241 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5: loading cluster ID for follower catalog manager: success
I20240503 02:03:55.575515 21241 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5: acquiring CA information for follower catalog manager: success
I20240503 02:03:55.579108 21241 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:03:55.591425 21255 catalog_manager.cc:1259] Loaded cluster ID: e15c943a1b364ec995c8bbff575437b0
I20240503 02:03:55.591675 21255 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762: loading cluster ID for follower catalog manager: success
I20240503 02:03:55.596148 21255 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762: acquiring CA information for follower catalog manager: success
I20240503 02:03:55.599241 21255 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 02:03:55.744745 21337 heartbeater.cc:503] Master 127.19.126.62:35085 was elected leader, sending a full tablet report...
I20240503 02:03:55.860153 21415 heartbeater.cc:503] Master 127.19.126.62:35085 was elected leader, sending a full tablet report...
I20240503 02:03:55.974157 21493 heartbeater.cc:503] Master 127.19.126.62:35085 was elected leader, sending a full tablet report...
I20240503 02:03:56.533228 21233 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:03:57.033339 19960 test_util.cc:268] Using random seed: 568614716
I20240503 02:03:57.080637 21044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58212:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:03:57.139240 21378 tablet_service.cc:1467] Processing CreateTablet for tablet 895d79ddf6e64a67a4d8ae6cbd91061c (DEFAULT_TABLE table=test-workload [id=142c1aef96d74692aaae604351ac282b]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 02:03:57.141139 21378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895d79ddf6e64a67a4d8ae6cbd91061c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:57.148932 21456 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd2ebe6dbc64e7d9f428299d7a58262 (DEFAULT_TABLE table=test-workload [id=142c1aef96d74692aaae604351ac282b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 02:03:57.150249 21456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dd2ebe6dbc64e7d9f428299d7a58262. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:57.150847 21300 tablet_service.cc:1467] Processing CreateTablet for tablet 3b97d3f0044b4589bbee70f92e548d42 (DEFAULT_TABLE table=test-workload [id=142c1aef96d74692aaae604351ac282b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 02:03:57.151932 21300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b97d3f0044b4589bbee70f92e548d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:03:57.158313 21526 tablet_bootstrap.cc:492] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0: Bootstrap starting.
I20240503 02:03:57.164564 21526 tablet_bootstrap.cc:654] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:57.165226 21527 tablet_bootstrap.cc:492] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a: Bootstrap starting.
I20240503 02:03:57.167133 21528 tablet_bootstrap.cc:492] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a: Bootstrap starting.
I20240503 02:03:57.170583 21527 tablet_bootstrap.cc:654] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:57.170600 21526 tablet_bootstrap.cc:492] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0: No bootstrap required, opened a new log
I20240503 02:03:57.171241 21526 ts_tablet_manager.cc:1367] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.002s
I20240503 02:03:57.173058 21528 tablet_bootstrap.cc:654] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a: Neither blocks nor log segments found. Creating new log.
I20240503 02:03:57.174031 21526 raft_consensus.cc:348] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29922cf0b9646b18cd743ada34008b0" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43891 } }
I20240503 02:03:57.174723 21526 raft_consensus.cc:374] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:57.175040 21526 raft_consensus.cc:729] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b29922cf0b9646b18cd743ada34008b0, State: Initialized, Role: FOLLOWER
I20240503 02:03:57.175729 21526 consensus_queue.cc:260] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29922cf0b9646b18cd743ada34008b0" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43891 } }
I20240503 02:03:57.176412 21526 raft_consensus.cc:388] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:03:57.176889 21526 raft_consensus.cc:482] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:03:57.177207 21526 raft_consensus.cc:3037] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:57.179719 21528 tablet_bootstrap.cc:492] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a: No bootstrap required, opened a new log
I20240503 02:03:57.180358 21528 ts_tablet_manager.cc:1367] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240503 02:03:57.181936 21526 raft_consensus.cc:504] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29922cf0b9646b18cd743ada34008b0" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43891 } }
I20240503 02:03:57.182512 21526 leader_election.cc:302] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b29922cf0b9646b18cd743ada34008b0; no voters:
I20240503 02:03:57.184017 21528 raft_consensus.cc:348] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f79c5fd2765431cb75f650f3ad5970a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 46837 } }
I20240503 02:03:57.185595 21528 raft_consensus.cc:374] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:57.185829 21526 leader_election.cc:288] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:03:57.185703 21532 raft_consensus.cc:2781] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:57.186852 21528 raft_consensus.cc:729] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f79c5fd2765431cb75f650f3ad5970a, State: Initialized, Role: FOLLOWER
I20240503 02:03:57.188863 21528 consensus_queue.cc:260] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f79c5fd2765431cb75f650f3ad5970a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 46837 } }
I20240503 02:03:57.189744 21528 raft_consensus.cc:388] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:03:57.190080 21528 raft_consensus.cc:482] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:03:57.190464 21528 raft_consensus.cc:3037] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:57.191329 21527 tablet_bootstrap.cc:492] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a: No bootstrap required, opened a new log
I20240503 02:03:57.191823 21527 ts_tablet_manager.cc:1367] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.012s
I20240503 02:03:57.193269 21532 raft_consensus.cc:686] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [term 1 LEADER]: Becoming Leader. State: Replica: b29922cf0b9646b18cd743ada34008b0, State: Running, Role: LEADER
I20240503 02:03:57.194366 21532 consensus_queue.cc:237] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29922cf0b9646b18cd743ada34008b0" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43891 } }
I20240503 02:03:57.194896 21527 raft_consensus.cc:348] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20175b1a16eb4fceb9a20b5bad66720a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 43345 } }
I20240503 02:03:57.195593 21527 raft_consensus.cc:374] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:03:57.195936 21527 raft_consensus.cc:729] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20175b1a16eb4fceb9a20b5bad66720a, State: Initialized, Role: FOLLOWER
I20240503 02:03:57.196628 21527 consensus_queue.cc:260] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20175b1a16eb4fceb9a20b5bad66720a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 43345 } }
I20240503 02:03:57.197096 21528 raft_consensus.cc:504] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f79c5fd2765431cb75f650f3ad5970a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 46837 } }
I20240503 02:03:57.197875 21528 leader_election.cc:302] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f79c5fd2765431cb75f650f3ad5970a; no voters:
I20240503 02:03:57.197343 21527 raft_consensus.cc:388] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:03:57.199347 21526 ts_tablet_manager.cc:1398] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0: Time spent starting tablet: real 0.028s user 0.017s sys 0.007s
I20240503 02:03:57.199559 21528 leader_election.cc:288] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:03:57.199368 21527 raft_consensus.cc:482] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:03:57.199968 21536 raft_consensus.cc:2781] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:57.201262 21527 raft_consensus.cc:3037] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:03:57.202667 21528 ts_tablet_manager.cc:1398] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a: Time spent starting tablet: real 0.022s user 0.023s sys 0.000s
I20240503 02:03:57.208416 21044 catalog_manager.cc:5484] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b29922cf0b9646b18cd743ada34008b0 (127.19.126.2). New cstate: current_term: 1 leader_uuid: "b29922cf0b9646b18cd743ada34008b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29922cf0b9646b18cd743ada34008b0" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43891 } health_report { overall_health: HEALTHY } } }
I20240503 02:03:57.210768 21527 raft_consensus.cc:504] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20175b1a16eb4fceb9a20b5bad66720a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 43345 } }
I20240503 02:03:57.211524 21527 leader_election.cc:302] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20175b1a16eb4fceb9a20b5bad66720a; no voters:
I20240503 02:03:57.240762 21536 raft_consensus.cc:686] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [term 1 LEADER]: Becoming Leader. State: Replica: 5f79c5fd2765431cb75f650f3ad5970a, State: Running, Role: LEADER
I20240503 02:03:57.241503 21527 leader_election.cc:288] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:03:57.241896 21538 raft_consensus.cc:2781] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:03:57.241593 21536 consensus_queue.cc:237] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f79c5fd2765431cb75f650f3ad5970a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 46837 } }
I20240503 02:03:57.244316 21527 ts_tablet_manager.cc:1398] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a: Time spent starting tablet: real 0.052s user 0.024s sys 0.012s
I20240503 02:03:57.246085 21538 raft_consensus.cc:686] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [term 1 LEADER]: Becoming Leader. State: Replica: 20175b1a16eb4fceb9a20b5bad66720a, State: Running, Role: LEADER
I20240503 02:03:57.246809 21538 consensus_queue.cc:237] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20175b1a16eb4fceb9a20b5bad66720a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 43345 } }
I20240503 02:03:57.255723 21044 catalog_manager.cc:5484] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f79c5fd2765431cb75f650f3ad5970a (127.19.126.3). New cstate: current_term: 1 leader_uuid: "5f79c5fd2765431cb75f650f3ad5970a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f79c5fd2765431cb75f650f3ad5970a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 46837 } health_report { overall_health: HEALTHY } } }
I20240503 02:03:57.268340 21043 catalog_manager.cc:5484] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a reported cstate change: term changed from 0 to 1, leader changed from <none> to 20175b1a16eb4fceb9a20b5bad66720a (127.19.126.1). New cstate: current_term: 1 leader_uuid: "20175b1a16eb4fceb9a20b5bad66720a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20175b1a16eb4fceb9a20b5bad66720a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 43345 } health_report { overall_health: HEALTHY } } }
I20240503 02:03:57.312000 19960 master.cc:537] Master@127.19.126.62:35085 shutting down...
I20240503 02:03:57.327075 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:03:57.327802 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:03:57.328088 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3d41f80c564a44b0ba1d2eda1fc4a013: stopping tablet replica
I20240503 02:03:57.348838 19960 master.cc:552] Master@127.19.126.62:35085 shutdown complete.
W20240503 02:03:58.242295 21415 heartbeater.cc:644] Failed to heartbeat to 127.19.126.62:35085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.126.62:35085: connect: Connection refused (error 111)
I20240503 02:03:58.786280 21550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d41f80c564a44b0ba1d2eda1fc4a013)
I20240503 02:03:58.786751 21550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:03:58.787389 21551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d41f80c564a44b0ba1d2eda1fc4a013)
I20240503 02:03:58.787760 21551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:03:58.788350 21550 leader_election.cc:288] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085), 5c692f4431484e8ea037a15082290762 (127.19.126.60:40399)
I20240503 02:03:58.789461 21551 leader_election.cc:288] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085), 77e48a58464e47b2b49dc989f0a976c5 (127.19.126.61:39913)
I20240503 02:03:58.790637 21184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77e48a58464e47b2b49dc989f0a976c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "5c692f4431484e8ea037a15082290762" is_pre_election: true
I20240503 02:03:58.791018 21119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c692f4431484e8ea037a15082290762" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "77e48a58464e47b2b49dc989f0a976c5" is_pre_election: true
I20240503 02:03:58.791296 21184 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e48a58464e47b2b49dc989f0a976c5 in term 1.
I20240503 02:03:58.791780 21119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c692f4431484e8ea037a15082290762 in term 1.
I20240503 02:03:58.793118 21160 leader_election.cc:302] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c692f4431484e8ea037a15082290762, 77e48a58464e47b2b49dc989f0a976c5; no voters:
I20240503 02:03:58.793802 21095 leader_election.cc:302] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c692f4431484e8ea037a15082290762, 77e48a58464e47b2b49dc989f0a976c5; no voters:
I20240503 02:03:58.793992 21551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:03:58.794484 21551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d41f80c564a44b0ba1d2eda1fc4a013)
I20240503 02:03:58.794625 21550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:03:58.795028 21550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d41f80c564a44b0ba1d2eda1fc4a013)
I20240503 02:03:58.795414 21550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:03:58.795819 21159 leader_election.cc:334] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085): Network error: Client connection negotiation failed: client connection to 127.19.126.62:35085: connect: Connection refused (error 111)
W20240503 02:03:58.796087 21095 leader_election.cc:334] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085): Network error: Client connection negotiation failed: client connection to 127.19.126.62:35085: connect: Connection refused (error 111)
I20240503 02:03:58.795017 21551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:03:58.800019 21550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:03:58.800644 21551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:03:58.801406 21550 leader_election.cc:288] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [CANDIDATE]: Term 2 election: Requested vote from peers 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085), 5c692f4431484e8ea037a15082290762 (127.19.126.60:40399)
I20240503 02:03:58.802814 21184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77e48a58464e47b2b49dc989f0a976c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "5c692f4431484e8ea037a15082290762"
I20240503 02:03:58.803359 21551 leader_election.cc:288] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [CANDIDATE]: Term 2 election: Requested vote from peers 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085), 77e48a58464e47b2b49dc989f0a976c5 (127.19.126.61:39913)
W20240503 02:03:58.803558 21159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.126.62:35085: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 02:03:58.804509 21119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c692f4431484e8ea037a15082290762" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "77e48a58464e47b2b49dc989f0a976c5"
I20240503 02:03:58.803558 21184 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e48a58464e47b2b49dc989f0a976c5 in current term 2: Already voted for candidate 5c692f4431484e8ea037a15082290762 in this term.
I20240503 02:03:58.805269 21119 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c692f4431484e8ea037a15082290762 in current term 2: Already voted for candidate 77e48a58464e47b2b49dc989f0a976c5 in this term.
W20240503 02:03:58.805646 21095 leader_election.cc:334] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085): Network error: Client connection negotiation failed: client connection to 127.19.126.62:35085: connect: Connection refused (error 111)
I20240503 02:03:58.806689 21095 leader_election.cc:302] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e48a58464e47b2b49dc989f0a976c5; no voters: 3d41f80c564a44b0ba1d2eda1fc4a013, 5c692f4431484e8ea037a15082290762
W20240503 02:03:58.806727 21159 leader_election.cc:334] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085): Network error: Client connection negotiation failed: client connection to 127.19.126.62:35085: connect: Connection refused (error 111)
I20240503 02:03:58.807425 21550 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:03:58.807900 21160 leader_election.cc:302] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c692f4431484e8ea037a15082290762; no voters: 3d41f80c564a44b0ba1d2eda1fc4a013, 77e48a58464e47b2b49dc989f0a976c5
I20240503 02:03:58.808442 21551 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:04:00.330128 21562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:00.330528 21562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:04:00.332136 21562 leader_election.cc:288] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085), 77e48a58464e47b2b49dc989f0a976c5 (127.19.126.61:39913)
I20240503 02:04:00.333323 21119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c692f4431484e8ea037a15082290762" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "77e48a58464e47b2b49dc989f0a976c5" is_pre_election: true
I20240503 02:04:00.334024 21119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c692f4431484e8ea037a15082290762 in term 2.
I20240503 02:04:00.335108 21160 leader_election.cc:302] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c692f4431484e8ea037a15082290762, 77e48a58464e47b2b49dc989f0a976c5; no voters:
I20240503 02:04:00.335819 21562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:04:00.336179 21562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:04:00.336445 21562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:04:00.337714 21159 leader_election.cc:334] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085): Network error: Client connection negotiation failed: client connection to 127.19.126.62:35085: connect: Connection refused (error 111)
I20240503 02:04:00.341367 21562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:04:00.342834 21562 leader_election.cc:288] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [CANDIDATE]: Term 3 election: Requested vote from peers 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085), 77e48a58464e47b2b49dc989f0a976c5 (127.19.126.61:39913)
I20240503 02:04:00.343725 21119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c692f4431484e8ea037a15082290762" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "77e48a58464e47b2b49dc989f0a976c5"
I20240503 02:04:00.344296 21119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:04:00.346004 21159 leader_election.cc:334] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085): Network error: Client connection negotiation failed: client connection to 127.19.126.62:35085: connect: Connection refused (error 111)
I20240503 02:04:00.348968 21119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c692f4431484e8ea037a15082290762 in term 3.
I20240503 02:04:00.349793 21160 leader_election.cc:302] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c692f4431484e8ea037a15082290762, 77e48a58464e47b2b49dc989f0a976c5; no voters: 3d41f80c564a44b0ba1d2eda1fc4a013
I20240503 02:04:00.350379 21562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:04:00.351297 21562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 3 LEADER]: Becoming Leader. State: Replica: 5c692f4431484e8ea037a15082290762, State: Running, Role: LEADER
I20240503 02:04:00.351933 21562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } }
I20240503 02:04:00.355199 21565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c692f4431484e8ea037a15082290762. Latest consensus state: current_term: 3 leader_uuid: "5c692f4431484e8ea037a15082290762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d41f80c564a44b0ba1d2eda1fc4a013" member_type: VOTER last_known_addr { host: "127.19.126.62" port: 35085 } } peers { permanent_uuid: "77e48a58464e47b2b49dc989f0a976c5" member_type: VOTER last_known_addr { host: "127.19.126.61" port: 39913 } } peers { permanent_uuid: "5c692f4431484e8ea037a15082290762" member_type: VOTER last_known_addr { host: "127.19.126.60" port: 40399 } } }
I20240503 02:04:00.355924 21565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [sys.catalog]: This master's current role is: LEADER
I20240503 02:04:00.357161 21567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:04:00.362269 21567 catalog_manager.cc:669] Loaded metadata for table test-workload [id=142c1aef96d74692aaae604351ac282b]
I20240503 02:04:00.367667 21567 tablet_loader.cc:96] loaded metadata for tablet 3b97d3f0044b4589bbee70f92e548d42 (table test-workload [id=142c1aef96d74692aaae604351ac282b])
I20240503 02:04:00.368559 21567 tablet_loader.cc:96] loaded metadata for tablet 3dd2ebe6dbc64e7d9f428299d7a58262 (table test-workload [id=142c1aef96d74692aaae604351ac282b])
I20240503 02:04:00.369544 21567 tablet_loader.cc:96] loaded metadata for tablet 895d79ddf6e64a67a4d8ae6cbd91061c (table test-workload [id=142c1aef96d74692aaae604351ac282b])
I20240503 02:04:00.370472 21567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:04:00.373713 21567 catalog_manager.cc:1259] Loaded cluster ID: e15c943a1b364ec995c8bbff575437b0
I20240503 02:04:00.373903 21567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:04:00.377323 21567 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:04:00.380296 21567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762: Loaded TSK: 0
I20240503 02:04:00.381059 21567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:04:00.585055 21253 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:00.585408 21253 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:00.653563 19960 tablet_server.cc:178] TabletServer@127.19.126.1:0 shutting down...
I20240503 02:04:00.672439 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:04:00.673027 19960 tablet_replica.cc:330] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a: stopping tablet replica
I20240503 02:04:00.673673 19960 raft_consensus.cc:2229] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:04:00.674073 19960 raft_consensus.cc:2258] T 3b97d3f0044b4589bbee70f92e548d42 P 20175b1a16eb4fceb9a20b5bad66720a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:00.722371 19960 tablet_server.cc:194] TabletServer@127.19.126.1:0 shutdown complete.
I20240503 02:04:00.733716 19960 tablet_server.cc:178] TabletServer@127.19.126.2:0 shutting down...
I20240503 02:04:00.760540 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:04:00.761157 19960 tablet_replica.cc:330] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0: stopping tablet replica
I20240503 02:04:00.761803 19960 raft_consensus.cc:2229] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:04:00.762491 19960 raft_consensus.cc:2258] T 895d79ddf6e64a67a4d8ae6cbd91061c P b29922cf0b9646b18cd743ada34008b0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:04:00.772364 21159 consensus_peers.cc:485] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 -> Peer 3d41f80c564a44b0ba1d2eda1fc4a013 (127.19.126.62:35085): Couldn't send request to peer 3d41f80c564a44b0ba1d2eda1fc4a013. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.62:35085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:00.781144 19960 tablet_server.cc:194] TabletServer@127.19.126.2:0 shutdown complete.
I20240503 02:04:00.790530 19960 tablet_server.cc:178] TabletServer@127.19.126.3:0 shutting down...
I20240503 02:04:00.809806 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:04:00.810425 19960 tablet_replica.cc:330] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a: stopping tablet replica
I20240503 02:04:00.810855 19960 raft_consensus.cc:2229] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:04:00.811307 19960 raft_consensus.cc:2258] T 3dd2ebe6dbc64e7d9f428299d7a58262 P 5f79c5fd2765431cb75f650f3ad5970a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:00.829686 19960 tablet_server.cc:194] TabletServer@127.19.126.3:0 shutdown complete.
I20240503 02:04:00.840826 19960 master.cc:537] Master@127.19.126.61:39913 shutting down...
I20240503 02:04:00.864467 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:04:00.865448 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:00.865955 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 77e48a58464e47b2b49dc989f0a976c5: stopping tablet replica
I20240503 02:04:00.885720 19960 master.cc:552] Master@127.19.126.61:39913 shutdown complete.
I20240503 02:04:00.898087 19960 master.cc:537] Master@127.19.126.60:40399 shutting down...
I20240503 02:04:00.916391 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 3 LEADER]: Raft consensus shutting down.
I20240503 02:04:00.917200 19960 pending_rounds.cc:62] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762: Trying to abort 1 pending ops.
I20240503 02:04:00.917440 19960 pending_rounds.cc:68] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762: Aborting op as it isn't in flight: id { term: 3 index: 10 } timestamp: 7023418738087841792 op_type: NO_OP noop_request { }
I20240503 02:04:00.917982 19960 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:04:00.918280 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:00.918547 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5c692f4431484e8ea037a15082290762: stopping tablet replica
I20240503 02:04:00.936468 19960 master.cc:552] Master@127.19.126.60:40399 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6764 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240503 02:04:00.969866 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:38485
I20240503 02:04:00.970937 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:00.976490 21569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:00.977223 21570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:00.977875 21572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:00.979051 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:00.979743 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:00.979933 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:00.980067 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701840980048 us; error 0 us; skew 500 ppm
I20240503 02:04:00.980546 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:00.982789 19960 webserver.cc:460] Webserver started at http://127.19.126.62:36865/ using document root <none> and password file <none>
I20240503 02:04:00.983229 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:00.983388 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:00.983644 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:00.984827 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "34f3aae138e643158a43dd63c81a8c34"
format_stamp: "Formatted at 2024-05-03 02:04:00 on dist-test-slave-8rwr"
I20240503 02:04:00.989466 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 02:04:00.992442 21577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:00.993091 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:04:00.993363 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "34f3aae138e643158a43dd63c81a8c34"
format_stamp: "Formatted at 2024-05-03 02:04:00 on dist-test-slave-8rwr"
I20240503 02:04:00.993614 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:01.020164 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:01.021279 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:01.056038 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:38485
I20240503 02:04:01.056123 21628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:38485 every 8 connection(s)
I20240503 02:04:01.059749 21629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:01.069406 21629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34: Bootstrap starting.
I20240503 02:04:01.073542 21629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:01.077471 21629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34: No bootstrap required, opened a new log
I20240503 02:04:01.079227 21629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f3aae138e643158a43dd63c81a8c34" member_type: VOTER }
I20240503 02:04:01.079651 21629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:01.079819 21629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f3aae138e643158a43dd63c81a8c34, State: Initialized, Role: FOLLOWER
I20240503 02:04:01.080317 21629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f3aae138e643158a43dd63c81a8c34" member_type: VOTER }
I20240503 02:04:01.080705 21629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:04:01.080888 21629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:04:01.081112 21629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:04:01.085098 21629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f3aae138e643158a43dd63c81a8c34" member_type: VOTER }
I20240503 02:04:01.085572 21629 leader_election.cc:302] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34f3aae138e643158a43dd63c81a8c34; no voters:
I20240503 02:04:01.086593 21629 leader_election.cc:288] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:04:01.086975 21632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:04:01.088168 21632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [term 1 LEADER]: Becoming Leader. State: Replica: 34f3aae138e643158a43dd63c81a8c34, State: Running, Role: LEADER
I20240503 02:04:01.088825 21632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f3aae138e643158a43dd63c81a8c34" member_type: VOTER }
I20240503 02:04:01.089262 21629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:04:01.096554 21634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34f3aae138e643158a43dd63c81a8c34. Latest consensus state: current_term: 1 leader_uuid: "34f3aae138e643158a43dd63c81a8c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f3aae138e643158a43dd63c81a8c34" member_type: VOTER } }
I20240503 02:04:01.097244 21634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [sys.catalog]: This master's current role is: LEADER
I20240503 02:04:01.097111 21633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34f3aae138e643158a43dd63c81a8c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f3aae138e643158a43dd63c81a8c34" member_type: VOTER } }
I20240503 02:04:01.097726 21633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [sys.catalog]: This master's current role is: LEADER
I20240503 02:04:01.099046 21641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:04:01.104903 21641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:04:01.105850 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:04:01.112797 21641 catalog_manager.cc:1347] Generated new cluster ID: 3a2ad66cceec45b09bdf458d959a0dc5
I20240503 02:04:01.113058 21641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:04:01.136533 21641 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:04:01.137648 21641 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:04:01.150700 21641 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34: Generated new TSK 0
I20240503 02:04:01.151192 21641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:04:01.172371 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:01.178445 21650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:01.180197 21651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:01.180932 21653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:01.181749 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:01.182417 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:01.182610 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:01.182755 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701841182740 us; error 0 us; skew 500 ppm
I20240503 02:04:01.183243 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:01.185562 19960 webserver.cc:460] Webserver started at http://127.19.126.1:38801/ using document root <none> and password file <none>
I20240503 02:04:01.186012 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:01.186197 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:01.186439 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:01.187476 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-0-root/instance:
uuid: "376d80ade31049f4ac418a45d08dc71a"
format_stamp: "Formatted at 2024-05-03 02:04:01 on dist-test-slave-8rwr"
I20240503 02:04:01.193535 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20240503 02:04:01.196568 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:01.197400 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:04:01.197661 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "376d80ade31049f4ac418a45d08dc71a"
format_stamp: "Formatted at 2024-05-03 02:04:01 on dist-test-slave-8rwr"
I20240503 02:04:01.197926 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:01.223865 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:01.224932 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:01.226336 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:04:01.228482 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:04:01.228665 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:01.228866 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:04:01.229019 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:01.266476 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:37159
I20240503 02:04:01.266564 21720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:37159 every 8 connection(s)
I20240503 02:04:01.271220 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:01.277740 21725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:01.281368 21726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:01.283257 19960 server_base.cc:1034] running on GCE node
W20240503 02:04:01.284055 21728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:01.284960 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:01.285164 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:01.285338 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701841285321 us; error 0 us; skew 500 ppm
I20240503 02:04:01.285642 21721 heartbeater.cc:347] Connected to a master server at 127.19.126.62:38485
I20240503 02:04:01.285893 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:01.285952 21721 heartbeater.cc:464] Registering TS with master...
I20240503 02:04:01.286842 21721 heartbeater.cc:511] Master 127.19.126.62:38485 requested a full tablet report, sending...
I20240503 02:04:01.288599 19960 webserver.cc:460] Webserver started at http://127.19.126.2:44069/ using document root <none> and password file <none>
I20240503 02:04:01.289229 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:01.289121 21594 ts_manager.cc:194] Registered new tserver with Master: 376d80ade31049f4ac418a45d08dc71a (127.19.126.1:37159)
I20240503 02:04:01.289564 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:01.289955 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:01.291397 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-1-root/instance:
uuid: "97ddfa4964c44e7ab343f2919dc087cc"
format_stamp: "Formatted at 2024-05-03 02:04:01 on dist-test-slave-8rwr"
I20240503 02:04:01.291661 21594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52410
I20240503 02:04:01.296566 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240503 02:04:01.299908 21733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:01.300596 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:04:01.300832 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "97ddfa4964c44e7ab343f2919dc087cc"
format_stamp: "Formatted at 2024-05-03 02:04:01 on dist-test-slave-8rwr"
I20240503 02:04:01.301069 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:01.312003 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:01.312963 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:01.314257 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:04:01.316298 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:04:01.316468 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:01.316704 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:04:01.316848 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:01.353678 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:44905
I20240503 02:04:01.353785 21795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:44905 every 8 connection(s)
I20240503 02:04:01.358064 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:01.364624 21799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:01.365619 21800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:01.366938 21802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:01.367843 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:01.368752 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:01.368984 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:01.369155 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701841369138 us; error 0 us; skew 500 ppm
I20240503 02:04:01.369771 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:01.369988 21796 heartbeater.cc:347] Connected to a master server at 127.19.126.62:38485
I20240503 02:04:01.370401 21796 heartbeater.cc:464] Registering TS with master...
I20240503 02:04:01.371088 21796 heartbeater.cc:511] Master 127.19.126.62:38485 requested a full tablet report, sending...
I20240503 02:04:01.372512 19960 webserver.cc:460] Webserver started at http://127.19.126.3:35223/ using document root <none> and password file <none>
I20240503 02:04:01.373113 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:01.373165 21594 ts_manager.cc:194] Registered new tserver with Master: 97ddfa4964c44e7ab343f2919dc087cc (127.19.126.2:44905)
I20240503 02:04:01.373445 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:01.373807 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:01.374949 21594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52420
I20240503 02:04:01.375283 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-2-root/instance:
uuid: "fe26bdc90e424830bcdb56c7771e1bcd"
format_stamp: "Formatted at 2024-05-03 02:04:01 on dist-test-slave-8rwr"
I20240503 02:04:01.380015 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:04:01.382968 21807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:01.383656 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:04:01.383920 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "fe26bdc90e424830bcdb56c7771e1bcd"
format_stamp: "Formatted at 2024-05-03 02:04:01 on dist-test-slave-8rwr"
I20240503 02:04:01.384184 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:01.392071 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:01.393033 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:01.394366 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:04:01.396498 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:04:01.396684 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:01.396896 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:04:01.397044 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:01.434968 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:33411
I20240503 02:04:01.435070 21869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:33411 every 8 connection(s)
I20240503 02:04:01.447338 21870 heartbeater.cc:347] Connected to a master server at 127.19.126.62:38485
I20240503 02:04:01.447664 21870 heartbeater.cc:464] Registering TS with master...
I20240503 02:04:01.448279 21870 heartbeater.cc:511] Master 127.19.126.62:38485 requested a full tablet report, sending...
I20240503 02:04:01.450062 21594 ts_manager.cc:194] Registered new tserver with Master: fe26bdc90e424830bcdb56c7771e1bcd (127.19.126.3:33411)
I20240503 02:04:01.450232 19960 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012385796s
I20240503 02:04:01.451821 21594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52430
I20240503 02:04:02.294231 21721 heartbeater.cc:503] Master 127.19.126.62:38485 was elected leader, sending a full tablet report...
I20240503 02:04:02.377514 21796 heartbeater.cc:503] Master 127.19.126.62:38485 was elected leader, sending a full tablet report...
I20240503 02:04:02.454408 21870 heartbeater.cc:503] Master 127.19.126.62:38485 was elected leader, sending a full tablet report...
I20240503 02:04:02.482446 19960 test_util.cc:268] Using random seed: 574063864
I20240503 02:04:02.510906 21594 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52442:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:04:02.512975 21594 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:04:02.568297 21835 tablet_service.cc:1467] Processing CreateTablet for tablet 24e820764537431a89c13520294ca9bf (DEFAULT_TABLE table=test-workload [id=ca11bb7db36e42318e7c65049f8d8636]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 02:04:02.568327 21834 tablet_service.cc:1467] Processing CreateTablet for tablet c148aff568cc48e9b2347176efc4199a (DEFAULT_TABLE table=test-workload [id=ca11bb7db36e42318e7c65049f8d8636]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 02:04:02.569895 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e820764537431a89c13520294ca9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:02.570714 21834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c148aff568cc48e9b2347176efc4199a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:02.580292 21686 tablet_service.cc:1467] Processing CreateTablet for tablet 24e820764537431a89c13520294ca9bf (DEFAULT_TABLE table=test-workload [id=ca11bb7db36e42318e7c65049f8d8636]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 02:04:02.580559 21685 tablet_service.cc:1467] Processing CreateTablet for tablet c148aff568cc48e9b2347176efc4199a (DEFAULT_TABLE table=test-workload [id=ca11bb7db36e42318e7c65049f8d8636]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 02:04:02.581665 21686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e820764537431a89c13520294ca9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:02.585332 21685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c148aff568cc48e9b2347176efc4199a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:02.585520 21761 tablet_service.cc:1467] Processing CreateTablet for tablet 24e820764537431a89c13520294ca9bf (DEFAULT_TABLE table=test-workload [id=ca11bb7db36e42318e7c65049f8d8636]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 02:04:02.586160 21760 tablet_service.cc:1467] Processing CreateTablet for tablet c148aff568cc48e9b2347176efc4199a (DEFAULT_TABLE table=test-workload [id=ca11bb7db36e42318e7c65049f8d8636]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 02:04:02.586817 21761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e820764537431a89c13520294ca9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:02.587551 21760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c148aff568cc48e9b2347176efc4199a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:02.607425 21890 tablet_bootstrap.cc:492] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd: Bootstrap starting.
I20240503 02:04:02.613795 21891 tablet_bootstrap.cc:492] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a: Bootstrap starting.
I20240503 02:04:02.614557 21890 tablet_bootstrap.cc:654] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:02.617902 21892 tablet_bootstrap.cc:492] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc: Bootstrap starting.
I20240503 02:04:02.620674 21891 tablet_bootstrap.cc:654] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:02.623503 21890 tablet_bootstrap.cc:492] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd: No bootstrap required, opened a new log
I20240503 02:04:02.623963 21890 ts_tablet_manager.cc:1367] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20240503 02:04:02.624408 21892 tablet_bootstrap.cc:654] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:02.625185 21891 tablet_bootstrap.cc:492] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a: No bootstrap required, opened a new log
I20240503 02:04:02.625588 21891 ts_tablet_manager.cc:1367] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240503 02:04:02.626713 21890 raft_consensus.cc:348] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } }
I20240503 02:04:02.627684 21890 raft_consensus.cc:374] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:02.627411 21891 raft_consensus.cc:348] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } }
I20240503 02:04:02.627964 21890 raft_consensus.cc:729] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26bdc90e424830bcdb56c7771e1bcd, State: Initialized, Role: FOLLOWER
I20240503 02:04:02.628245 21891 raft_consensus.cc:374] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:02.628541 21891 raft_consensus.cc:729] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376d80ade31049f4ac418a45d08dc71a, State: Initialized, Role: FOLLOWER
I20240503 02:04:02.628854 21890 consensus_queue.cc:260] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } }
I20240503 02:04:02.629094 21891 consensus_queue.cc:260] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } }
I20240503 02:04:02.629614 21892 tablet_bootstrap.cc:492] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc: No bootstrap required, opened a new log
I20240503 02:04:02.630095 21892 ts_tablet_manager.cc:1367] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240503 02:04:02.632779 21892 raft_consensus.cc:348] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } }
I20240503 02:04:02.633304 21890 ts_tablet_manager.cc:1398] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240503 02:04:02.633610 21892 raft_consensus.cc:374] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:02.633900 21892 raft_consensus.cc:729] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ddfa4964c44e7ab343f2919dc087cc, State: Initialized, Role: FOLLOWER
I20240503 02:04:02.634358 21890 tablet_bootstrap.cc:492] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd: Bootstrap starting.
I20240503 02:04:02.634675 21892 consensus_queue.cc:260] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } }
I20240503 02:04:02.644276 21890 tablet_bootstrap.cc:654] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:02.649470 21891 ts_tablet_manager.cc:1398] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a: Time spent starting tablet: real 0.024s user 0.020s sys 0.004s
I20240503 02:04:02.650882 21890 tablet_bootstrap.cc:492] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd: No bootstrap required, opened a new log
I20240503 02:04:02.651333 21890 ts_tablet_manager.cc:1367] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20240503 02:04:02.651363 21891 tablet_bootstrap.cc:492] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a: Bootstrap starting.
I20240503 02:04:02.654083 21890 raft_consensus.cc:348] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } }
I20240503 02:04:02.654624 21892 ts_tablet_manager.cc:1398] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc: Time spent starting tablet: real 0.024s user 0.010s sys 0.008s
I20240503 02:04:02.654919 21890 raft_consensus.cc:374] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:02.655297 21890 raft_consensus.cc:729] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26bdc90e424830bcdb56c7771e1bcd, State: Initialized, Role: FOLLOWER
I20240503 02:04:02.655695 21892 tablet_bootstrap.cc:492] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc: Bootstrap starting.
I20240503 02:04:02.656082 21890 consensus_queue.cc:260] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } }
I20240503 02:04:02.657229 21891 tablet_bootstrap.cc:654] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:02.661540 21892 tablet_bootstrap.cc:654] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:02.664366 21890 ts_tablet_manager.cc:1398] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240503 02:04:02.670307 21891 tablet_bootstrap.cc:492] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a: No bootstrap required, opened a new log
I20240503 02:04:02.671135 21891 ts_tablet_manager.cc:1367] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.008s
I20240503 02:04:02.671525 21892 tablet_bootstrap.cc:492] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc: No bootstrap required, opened a new log
I20240503 02:04:02.672057 21892 ts_tablet_manager.cc:1367] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20240503 02:04:02.674279 21891 raft_consensus.cc:348] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } }
I20240503 02:04:02.674458 21892 raft_consensus.cc:348] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } }
I20240503 02:04:02.674803 21891 raft_consensus.cc:374] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:02.675053 21892 raft_consensus.cc:374] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:02.675100 21891 raft_consensus.cc:729] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376d80ade31049f4ac418a45d08dc71a, State: Initialized, Role: FOLLOWER
I20240503 02:04:02.675441 21892 raft_consensus.cc:729] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ddfa4964c44e7ab343f2919dc087cc, State: Initialized, Role: FOLLOWER
I20240503 02:04:02.675814 21891 consensus_queue.cc:260] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } }
I20240503 02:04:02.676064 21892 consensus_queue.cc:260] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } }
I20240503 02:04:02.678120 21892 ts_tablet_manager.cc:1398] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240503 02:04:02.678699 21891 ts_tablet_manager.cc:1398] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:04:02.765432 21896 raft_consensus.cc:482] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:02.765866 21896 raft_consensus.cc:504] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } }
I20240503 02:04:02.767684 21896 leader_election.cc:288] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376d80ade31049f4ac418a45d08dc71a (127.19.126.1:37159), 97ddfa4964c44e7ab343f2919dc087cc (127.19.126.2:44905)
I20240503 02:04:02.778280 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c148aff568cc48e9b2347176efc4199a" candidate_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376d80ade31049f4ac418a45d08dc71a" is_pre_election: true
I20240503 02:04:02.778846 21771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c148aff568cc48e9b2347176efc4199a" candidate_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ddfa4964c44e7ab343f2919dc087cc" is_pre_election: true
I20240503 02:04:02.779155 21696 raft_consensus.cc:2454] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26bdc90e424830bcdb56c7771e1bcd in term 0.
I20240503 02:04:02.779449 21771 raft_consensus.cc:2454] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe26bdc90e424830bcdb56c7771e1bcd in term 0.
I20240503 02:04:02.780189 21811 leader_election.cc:302] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376d80ade31049f4ac418a45d08dc71a, fe26bdc90e424830bcdb56c7771e1bcd; no voters:
I20240503 02:04:02.780907 21896 raft_consensus.cc:2781] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:04:02.781198 21896 raft_consensus.cc:482] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:04:02.781474 21896 raft_consensus.cc:3037] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:04:02.817016 21896 raft_consensus.cc:504] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } }
I20240503 02:04:02.819625 21896 leader_election.cc:288] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [CANDIDATE]: Term 1 election: Requested vote from peers 376d80ade31049f4ac418a45d08dc71a (127.19.126.1:37159), 97ddfa4964c44e7ab343f2919dc087cc (127.19.126.2:44905)
I20240503 02:04:02.819943 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c148aff568cc48e9b2347176efc4199a" candidate_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376d80ade31049f4ac418a45d08dc71a"
I20240503 02:04:02.820251 21771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c148aff568cc48e9b2347176efc4199a" candidate_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ddfa4964c44e7ab343f2919dc087cc"
I20240503 02:04:02.820607 21696 raft_consensus.cc:3037] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:04:02.820772 21771 raft_consensus.cc:3037] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:04:02.879086 21897 raft_consensus.cc:482] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:02.879523 21897 raft_consensus.cc:504] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } }
I20240503 02:04:02.881450 21897 leader_election.cc:288] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe26bdc90e424830bcdb56c7771e1bcd (127.19.126.3:33411), 97ddfa4964c44e7ab343f2919dc087cc (127.19.126.2:44905)
I20240503 02:04:02.891366 21770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e820764537431a89c13520294ca9bf" candidate_uuid: "376d80ade31049f4ac418a45d08dc71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ddfa4964c44e7ab343f2919dc087cc" is_pre_election: true
I20240503 02:04:02.891382 21845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e820764537431a89c13520294ca9bf" candidate_uuid: "376d80ade31049f4ac418a45d08dc71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" is_pre_election: true
I20240503 02:04:02.892050 21770 raft_consensus.cc:2454] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 376d80ade31049f4ac418a45d08dc71a in term 0.
I20240503 02:04:02.892135 21845 raft_consensus.cc:2454] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 376d80ade31049f4ac418a45d08dc71a in term 0.
I20240503 02:04:02.892985 21660 leader_election.cc:302] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376d80ade31049f4ac418a45d08dc71a, 97ddfa4964c44e7ab343f2919dc087cc; no voters:
I20240503 02:04:02.893783 21897 raft_consensus.cc:2781] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:04:02.894065 21897 raft_consensus.cc:482] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:04:02.894296 21897 raft_consensus.cc:3037] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:04:02.917193 21696 raft_consensus.cc:2454] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26bdc90e424830bcdb56c7771e1bcd in term 1.
I20240503 02:04:02.917193 21771 raft_consensus.cc:2454] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe26bdc90e424830bcdb56c7771e1bcd in term 1.
I20240503 02:04:02.918910 21809 leader_election.cc:302] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97ddfa4964c44e7ab343f2919dc087cc, fe26bdc90e424830bcdb56c7771e1bcd; no voters:
I20240503 02:04:02.919708 21896 raft_consensus.cc:2781] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:04:02.921461 21896 raft_consensus.cc:686] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 1 LEADER]: Becoming Leader. State: Replica: fe26bdc90e424830bcdb56c7771e1bcd, State: Running, Role: LEADER
I20240503 02:04:02.922168 21896 consensus_queue.cc:237] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } }
I20240503 02:04:02.929906 21594 catalog_manager.cc:5484] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to fe26bdc90e424830bcdb56c7771e1bcd (127.19.126.3). New cstate: current_term: 1 leader_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } health_report { overall_health: HEALTHY } } }
I20240503 02:04:03.014405 21897 raft_consensus.cc:504] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } }
I20240503 02:04:03.016731 21897 leader_election.cc:288] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [CANDIDATE]: Term 1 election: Requested vote from peers fe26bdc90e424830bcdb56c7771e1bcd (127.19.126.3:33411), 97ddfa4964c44e7ab343f2919dc087cc (127.19.126.2:44905)
I20240503 02:04:03.017513 21845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e820764537431a89c13520294ca9bf" candidate_uuid: "376d80ade31049f4ac418a45d08dc71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26bdc90e424830bcdb56c7771e1bcd"
I20240503 02:04:03.018023 21845 raft_consensus.cc:3037] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:04:03.017856 21771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e820764537431a89c13520294ca9bf" candidate_uuid: "376d80ade31049f4ac418a45d08dc71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ddfa4964c44e7ab343f2919dc087cc"
I20240503 02:04:03.018537 21771 raft_consensus.cc:3037] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:04:03.102820 21642 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:04:03.104730 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:03.105371 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:03.114920 21845 raft_consensus.cc:2454] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376d80ade31049f4ac418a45d08dc71a in term 1.
I20240503 02:04:03.114897 21771 raft_consensus.cc:2454] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376d80ade31049f4ac418a45d08dc71a in term 1.
I20240503 02:04:03.116189 21659 leader_election.cc:302] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376d80ade31049f4ac418a45d08dc71a, fe26bdc90e424830bcdb56c7771e1bcd; no voters:
I20240503 02:04:03.116844 21897 raft_consensus.cc:2781] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:04:03.117859 21897 raft_consensus.cc:686] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 1 LEADER]: Becoming Leader. State: Replica: 376d80ade31049f4ac418a45d08dc71a, State: Running, Role: LEADER
I20240503 02:04:03.118579 21897 consensus_queue.cc:237] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } }
I20240503 02:04:03.124503 21592 catalog_manager.cc:5484] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a reported cstate change: term changed from 0 to 1, leader changed from <none> to 376d80ade31049f4ac418a45d08dc71a (127.19.126.1). New cstate: current_term: 1 leader_uuid: "376d80ade31049f4ac418a45d08dc71a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } health_report { overall_health: HEALTHY } } }
I20240503 02:04:03.308959 21896 consensus_queue.cc:1035] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:04:03.319957 21896 consensus_queue.cc:1035] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:04:03.575244 21897 consensus_queue.cc:1035] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:04:03.584486 21917 consensus_queue.cc:1035] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:04:05.106331 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:05.108248 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:05.108631 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:07.109578 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:07.111466 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:07.111748 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:09.112552 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:09.114341 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:09.114655 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 02:04:10.860536 21625 debug-util.cc:398] Leaking SignalData structure 0x7b08000378e0 after lost signal to thread 19963
W20240503 02:04:10.861761 21625 debug-util.cc:398] Leaking SignalData structure 0x7b08001e0800 after lost signal to thread 21628
W20240503 02:04:10.862833 21625 debug-util.cc:398] Leaking SignalData structure 0x7b080012e8a0 after lost signal to thread 21720
W20240503 02:04:10.863955 21625 debug-util.cc:398] Leaking SignalData structure 0x7b08000c0300 after lost signal to thread 21795
W20240503 02:04:10.864838 21625 debug-util.cc:398] Leaking SignalData structure 0x7b0800151400 after lost signal to thread 21869
I20240503 02:04:11.115758 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:11.117910 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:11.118208 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:13.119069 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:13.120867 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:13.121165 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:15.122054 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:15.123879 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:15.124140 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:15.236372 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:15.244490 21926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:15.245033 21927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:15.245803 21929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:15.246682 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:15.248812 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:15.249085 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:15.249307 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701855249267 us; error 0 us; skew 500 ppm
I20240503 02:04:15.249832 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:15.252480 19960 webserver.cc:460] Webserver started at http://127.19.126.4:36335/ using document root <none> and password file <none>
I20240503 02:04:15.252986 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:15.253201 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:15.253540 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:15.255234 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-3-root/instance:
uuid: "ac42ceb41cb54630832b81bad8fbe165"
format_stamp: "Formatted at 2024-05-03 02:04:15 on dist-test-slave-8rwr"
I20240503 02:04:15.260051 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240503 02:04:15.263236 21934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:15.263973 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:04:15.264282 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-3-root
uuid: "ac42ceb41cb54630832b81bad8fbe165"
format_stamp: "Formatted at 2024-05-03 02:04:15 on dist-test-slave-8rwr"
I20240503 02:04:15.264542 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:15.283380 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:15.284762 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:15.286311 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:04:15.288724 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:04:15.288908 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:15.289160 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:04:15.289371 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 02:04:15.328289 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.4:33137
I20240503 02:04:15.328369 21996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.4:33137 every 8 connection(s)
I20240503 02:04:15.342784 21997 heartbeater.cc:347] Connected to a master server at 127.19.126.62:38485
I20240503 02:04:15.343114 21997 heartbeater.cc:464] Registering TS with master...
I20240503 02:04:15.343788 21997 heartbeater.cc:511] Master 127.19.126.62:38485 requested a full tablet report, sending...
I20240503 02:04:15.345643 21594 ts_manager.cc:194] Registered new tserver with Master: ac42ceb41cb54630832b81bad8fbe165 (127.19.126.4:33137)
I20240503 02:04:15.346920 21594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51682
I20240503 02:04:16.185458 21696 consensus_queue.cc:237] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: true } }
I20240503 02:04:16.193666 21771 raft_consensus.cc:1261] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 1 FOLLOWER]: Refusing update from remote peer 376d80ade31049f4ac418a45d08dc71a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:04:16.196298 21925 consensus_queue.cc:1035] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:04:16.197443 21845 raft_consensus.cc:1261] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [term 1 FOLLOWER]: Refusing update from remote peer 376d80ade31049f4ac418a45d08dc71a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:04:16.198617 21925 consensus_queue.cc:1035] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:04:16.210072 21845 raft_consensus.cc:2932] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ac42ceb41cb54630832b81bad8fbe165 (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: true } } }
I20240503 02:04:16.209322 21925 raft_consensus.cc:2932] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ac42ceb41cb54630832b81bad8fbe165 (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: true } } }
I20240503 02:04:16.217000 21771 raft_consensus.cc:2932] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ac42ceb41cb54630832b81bad8fbe165 (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: true } } }
W20240503 02:04:16.220225 21661 consensus_peers.cc:485] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a -> Peer ac42ceb41cb54630832b81bad8fbe165 (127.19.126.4:33137): Couldn't send request to peer ac42ceb41cb54630832b81bad8fbe165. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24e820764537431a89c13520294ca9bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:04:16.232368 21594 catalog_manager.cc:5484] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd reported cstate change: config changed from index -1 to 2, NON_VOTER ac42ceb41cb54630832b81bad8fbe165 (127.19.126.4) added. New cstate: current_term: 1 leader_uuid: "376d80ade31049f4ac418a45d08dc71a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: true } } }
I20240503 02:04:16.349022 21997 heartbeater.cc:503] Master 127.19.126.62:38485 was elected leader, sending a full tablet report...
I20240503 02:04:16.704743 22015 ts_tablet_manager.cc:902] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165: Initiating tablet copy from peer 376d80ade31049f4ac418a45d08dc71a (127.19.126.1:37159)
I20240503 02:04:16.708747 22015 tablet_copy_client.cc:308] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165: tablet copy: Beginning tablet copy session from remote peer at address 127.19.126.1:37159
I20240503 02:04:16.721668 21706 tablet_copy_service.cc:140] P 376d80ade31049f4ac418a45d08dc71a: Received BeginTabletCopySession request for tablet 24e820764537431a89c13520294ca9bf from peer ac42ceb41cb54630832b81bad8fbe165 ({username='slave'} at 127.0.0.1:34608)
I20240503 02:04:16.722173 21706 tablet_copy_service.cc:161] P 376d80ade31049f4ac418a45d08dc71a: Beginning new tablet copy session on tablet 24e820764537431a89c13520294ca9bf from peer ac42ceb41cb54630832b81bad8fbe165 at {username='slave'} at 127.0.0.1:34608: session id = ac42ceb41cb54630832b81bad8fbe165-24e820764537431a89c13520294ca9bf
I20240503 02:04:16.729807 21706 tablet_copy_source_session.cc:205] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:04:16.733093 22015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e820764537431a89c13520294ca9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:16.746811 22015 tablet_copy_client.cc:783] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165: tablet copy: Starting download of 0 data blocks...
I20240503 02:04:16.747428 22015 tablet_copy_client.cc:647] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165: tablet copy: Starting download of 1 WAL segments...
I20240503 02:04:16.751032 22015 tablet_copy_client.cc:523] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:04:16.757588 22015 tablet_bootstrap.cc:492] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165: Bootstrap starting.
I20240503 02:04:16.773943 22015 tablet_bootstrap.cc:492] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:04:16.774717 22015 tablet_bootstrap.cc:492] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165: Bootstrap complete.
I20240503 02:04:16.775245 22015 ts_tablet_manager.cc:1367] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20240503 02:04:16.777429 22015 raft_consensus.cc:348] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: true } }
I20240503 02:04:16.778165 22015 raft_consensus.cc:729] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ac42ceb41cb54630832b81bad8fbe165, State: Initialized, Role: LEARNER
I20240503 02:04:16.778651 22015 consensus_queue.cc:260] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: true } }
I20240503 02:04:16.781157 22015 ts_tablet_manager.cc:1398] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240503 02:04:16.782724 21706 tablet_copy_service.cc:342] P 376d80ade31049f4ac418a45d08dc71a: Request end of tablet copy session ac42ceb41cb54630832b81bad8fbe165-24e820764537431a89c13520294ca9bf received from {username='slave'} at 127.0.0.1:34608
I20240503 02:04:16.783177 21706 tablet_copy_service.cc:433] P 376d80ade31049f4ac418a45d08dc71a: ending tablet copy session ac42ceb41cb54630832b81bad8fbe165-24e820764537431a89c13520294ca9bf on tablet 24e820764537431a89c13520294ca9bf with peer ac42ceb41cb54630832b81bad8fbe165
I20240503 02:04:17.124832 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:17.126605 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:17.126940 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:17.309834 21972 raft_consensus.cc:1203] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:04:17.832314 22025 raft_consensus.cc:1050] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a: attempting to promote NON_VOTER ac42ceb41cb54630832b81bad8fbe165 to VOTER
I20240503 02:04:17.834566 22025 consensus_queue.cc:237] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false } }
I20240503 02:04:17.840338 21972 raft_consensus.cc:1261] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165 [term 1 LEARNER]: Refusing update from remote peer 376d80ade31049f4ac418a45d08dc71a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:04:17.842237 21771 raft_consensus.cc:1261] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 1 FOLLOWER]: Refusing update from remote peer 376d80ade31049f4ac418a45d08dc71a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:04:17.843762 21845 raft_consensus.cc:1261] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [term 1 FOLLOWER]: Refusing update from remote peer 376d80ade31049f4ac418a45d08dc71a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:04:17.844209 22030 consensus_queue.cc:1035] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:04:17.845114 22025 consensus_queue.cc:1035] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:04:17.846295 22027 consensus_queue.cc:1035] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:04:17.859112 22030 raft_consensus.cc:2932] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ac42ceb41cb54630832b81bad8fbe165 (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false } } }
I20240503 02:04:17.861313 21845 raft_consensus.cc:2932] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ac42ceb41cb54630832b81bad8fbe165 (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false } } }
I20240503 02:04:17.861575 21972 raft_consensus.cc:2932] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ac42ceb41cb54630832b81bad8fbe165 (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false } } }
I20240503 02:04:17.871444 21771 raft_consensus.cc:2932] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ac42ceb41cb54630832b81bad8fbe165 (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false } } }
I20240503 02:04:17.871394 21592 catalog_manager.cc:5484] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd reported cstate change: config changed from index 2 to 3, ac42ceb41cb54630832b81bad8fbe165 (127.19.126.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "376d80ade31049f4ac418a45d08dc71a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe26bdc90e424830bcdb56c7771e1bcd" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 33411 } attrs { replace: true } } peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false } } }
I20240503 02:04:17.900136 21696 consensus_queue.cc:237] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false } }
I20240503 02:04:17.904922 21972 raft_consensus.cc:1261] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165 [term 1 FOLLOWER]: Refusing update from remote peer 376d80ade31049f4ac418a45d08dc71a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:04:17.904992 21771 raft_consensus.cc:1261] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 1 FOLLOWER]: Refusing update from remote peer 376d80ade31049f4ac418a45d08dc71a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:04:17.906266 22030 consensus_queue.cc:1035] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:04:17.907186 22025 consensus_queue.cc:1035] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:04:17.912743 22027 raft_consensus.cc:2932] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe26bdc90e424830bcdb56c7771e1bcd (127.19.126.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false } } }
I20240503 02:04:17.914806 21771 raft_consensus.cc:2932] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe26bdc90e424830bcdb56c7771e1bcd (127.19.126.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false } } }
I20240503 02:04:17.915025 21972 raft_consensus.cc:2932] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe26bdc90e424830bcdb56c7771e1bcd (127.19.126.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false } } }
I20240503 02:04:17.921931 21581 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 24e820764537431a89c13520294ca9bf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:04:17.924681 21594 catalog_manager.cc:5484] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc reported cstate change: config changed from index 3 to 4, VOTER fe26bdc90e424830bcdb56c7771e1bcd (127.19.126.3) evicted. New cstate: current_term: 1 leader_uuid: "376d80ade31049f4ac418a45d08dc71a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97ddfa4964c44e7ab343f2919dc087cc" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 44905 } } peers { permanent_uuid: "376d80ade31049f4ac418a45d08dc71a" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 37159 } } peers { permanent_uuid: "ac42ceb41cb54630832b81bad8fbe165" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 33137 } attrs { promote: false } } }
I20240503 02:04:17.933442 21834 tablet_service.cc:1514] Processing DeleteTablet for tablet 24e820764537431a89c13520294ca9bf with delete_type TABLET_DATA_TOMBSTONED (TS fe26bdc90e424830bcdb56c7771e1bcd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44620
I20240503 02:04:17.935631 22040 tablet_replica.cc:330] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd: stopping tablet replica
I20240503 02:04:17.936434 22040 raft_consensus.cc:2229] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:04:17.936940 22040 raft_consensus.cc:2258] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:17.939438 22040 ts_tablet_manager.cc:1875] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:04:17.948689 22040 ts_tablet_manager.cc:1888] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:04:17.948985 22040 log.cc:1196] T 24e820764537431a89c13520294ca9bf P fe26bdc90e424830bcdb56c7771e1bcd: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701820461004-19960-0/minicluster-data/ts-2-root/wals/24e820764537431a89c13520294ca9bf
I20240503 02:04:17.950277 21578 catalog_manager.cc:4830] TS fe26bdc90e424830bcdb56c7771e1bcd (127.19.126.3:33411): tablet 24e820764537431a89c13520294ca9bf (table test-workload [id=ca11bb7db36e42318e7c65049f8d8636]) successfully deleted
I20240503 02:04:19.127969 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:19.129874 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:19.130204 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:21.131338 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:21.133426 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:21.133760 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:23.134707 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:23.136569 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:23.136890 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:25.137847 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:25.139688 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:25.140013 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:27.140949 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:27.142809 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:27.143137 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:29.144002 21647 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:29.145948 21647 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:29.146257 21647 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:29.941543 19960 tablet_server.cc:178] TabletServer@127.19.126.1:0 shutting down...
I20240503 02:04:29.966428 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:04:29.967253 19960 tablet_replica.cc:330] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a: stopping tablet replica
I20240503 02:04:29.967916 19960 raft_consensus.cc:2229] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:04:29.968415 19960 raft_consensus.cc:2258] T c148aff568cc48e9b2347176efc4199a P 376d80ade31049f4ac418a45d08dc71a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:29.970562 19960 tablet_replica.cc:330] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a: stopping tablet replica
I20240503 02:04:29.970975 19960 raft_consensus.cc:2229] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:04:29.971733 19960 raft_consensus.cc:2258] T 24e820764537431a89c13520294ca9bf P 376d80ade31049f4ac418a45d08dc71a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:29.995529 19960 tablet_server.cc:194] TabletServer@127.19.126.1:0 shutdown complete.
I20240503 02:04:30.008018 19960 tablet_server.cc:178] TabletServer@127.19.126.2:0 shutting down...
I20240503 02:04:30.032748 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:04:30.033399 19960 tablet_replica.cc:330] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc: stopping tablet replica
I20240503 02:04:30.033970 19960 raft_consensus.cc:2229] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:04:30.034413 19960 raft_consensus.cc:2258] T c148aff568cc48e9b2347176efc4199a P 97ddfa4964c44e7ab343f2919dc087cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:30.036370 19960 tablet_replica.cc:330] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc: stopping tablet replica
I20240503 02:04:30.036820 19960 raft_consensus.cc:2229] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:04:30.037429 19960 raft_consensus.cc:2258] T 24e820764537431a89c13520294ca9bf P 97ddfa4964c44e7ab343f2919dc087cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:30.057495 19960 tablet_server.cc:194] TabletServer@127.19.126.2:0 shutdown complete.
I20240503 02:04:30.067715 19960 tablet_server.cc:178] TabletServer@127.19.126.3:0 shutting down...
I20240503 02:04:30.090149 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:04:30.090957 19960 tablet_replica.cc:330] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd: stopping tablet replica
I20240503 02:04:30.091512 19960 raft_consensus.cc:2229] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:04:30.092355 19960 raft_consensus.cc:2258] T c148aff568cc48e9b2347176efc4199a P fe26bdc90e424830bcdb56c7771e1bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:30.112308 19960 tablet_server.cc:194] TabletServer@127.19.126.3:0 shutdown complete.
I20240503 02:04:30.122588 19960 tablet_server.cc:178] TabletServer@127.19.126.4:0 shutting down...
I20240503 02:04:30.143878 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:04:30.144438 19960 tablet_replica.cc:330] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165: stopping tablet replica
I20240503 02:04:30.144942 19960 raft_consensus.cc:2229] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:04:30.145663 19960 raft_consensus.cc:2258] T 24e820764537431a89c13520294ca9bf P ac42ceb41cb54630832b81bad8fbe165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:30.163801 19960 tablet_server.cc:194] TabletServer@127.19.126.4:0 shutdown complete.
I20240503 02:04:30.172842 19960 master.cc:537] Master@127.19.126.62:38485 shutting down...
I20240503 02:04:30.193048 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:04:30.193635 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:30.193989 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 34f3aae138e643158a43dd63c81a8c34: stopping tablet replica
I20240503 02:04:30.227770 19960 master.cc:552] Master@127.19.126.62:38485 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (29293 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240503 02:04:30.263466 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:45003
I20240503 02:04:30.264642 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:30.271193 22043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:30.272074 22044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:30.272760 22046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:30.273669 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:30.274518 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:30.274703 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:30.274850 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701870274833 us; error 0 us; skew 500 ppm
I20240503 02:04:30.275346 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:30.282943 19960 webserver.cc:460] Webserver started at http://127.19.126.62:33779/ using document root <none> and password file <none>
I20240503 02:04:30.283393 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:30.283557 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:30.283818 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:30.284988 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "652e5cc181784194ba8f5f40b10b21ea"
format_stamp: "Formatted at 2024-05-03 02:04:30 on dist-test-slave-8rwr"
I20240503 02:04:30.289209 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:04:30.292291 22051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:30.292963 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:04:30.293205 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "652e5cc181784194ba8f5f40b10b21ea"
format_stamp: "Formatted at 2024-05-03 02:04:30 on dist-test-slave-8rwr"
I20240503 02:04:30.293483 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:30.308200 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:30.309253 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:30.344388 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:45003
I20240503 02:04:30.344467 22102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:45003 every 8 connection(s)
I20240503 02:04:30.348153 22103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:30.357847 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea: Bootstrap starting.
I20240503 02:04:30.361806 22103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:30.365633 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea: No bootstrap required, opened a new log
I20240503 02:04:30.367504 22103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652e5cc181784194ba8f5f40b10b21ea" member_type: VOTER }
I20240503 02:04:30.367985 22103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:30.368157 22103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652e5cc181784194ba8f5f40b10b21ea, State: Initialized, Role: FOLLOWER
I20240503 02:04:30.368609 22103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652e5cc181784194ba8f5f40b10b21ea" member_type: VOTER }
I20240503 02:04:30.369021 22103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:04:30.369199 22103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:04:30.369464 22103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:04:30.373301 22103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652e5cc181784194ba8f5f40b10b21ea" member_type: VOTER }
I20240503 02:04:30.373724 22103 leader_election.cc:302] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 652e5cc181784194ba8f5f40b10b21ea; no voters:
I20240503 02:04:30.374663 22103 leader_election.cc:288] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:04:30.375026 22106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:04:30.376294 22106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [term 1 LEADER]: Becoming Leader. State: Replica: 652e5cc181784194ba8f5f40b10b21ea, State: Running, Role: LEADER
I20240503 02:04:30.376904 22106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652e5cc181784194ba8f5f40b10b21ea" member_type: VOTER }
I20240503 02:04:30.377480 22103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:04:30.381335 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 652e5cc181784194ba8f5f40b10b21ea. Latest consensus state: current_term: 1 leader_uuid: "652e5cc181784194ba8f5f40b10b21ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652e5cc181784194ba8f5f40b10b21ea" member_type: VOTER } }
I20240503 02:04:30.381621 22107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "652e5cc181784194ba8f5f40b10b21ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652e5cc181784194ba8f5f40b10b21ea" member_type: VOTER } }
I20240503 02:04:30.382050 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [sys.catalog]: This master's current role is: LEADER
I20240503 02:04:30.382316 22107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [sys.catalog]: This master's current role is: LEADER
I20240503 02:04:30.384915 22112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:04:30.391395 22112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:04:30.393088 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:04:30.400048 22112 catalog_manager.cc:1347] Generated new cluster ID: 61bd58a34a32496da660547409545a0f
I20240503 02:04:30.400244 22112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:04:30.426201 22112 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:04:30.427326 22112 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:04:30.441165 22112 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea: Generated new TSK 0
I20240503 02:04:30.441710 22112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:04:30.458328 19960 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.000189671s
I20240503 02:04:42.551662 19960 master.cc:537] Master@127.19.126.62:45003 shutting down...
I20240503 02:04:42.575335 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:04:42.576076 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:42.576457 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 652e5cc181784194ba8f5f40b10b21ea: stopping tablet replica
I20240503 02:04:42.594751 19960 master.cc:552] Master@127.19.126.62:45003 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12361 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240503 02:04:42.624641 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:36201
I20240503 02:04:42.625648 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:42.631661 22127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:42.632722 22128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:42.633023 22130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:42.633805 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:42.634471 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:42.634657 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:42.634790 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701882634773 us; error 0 us; skew 500 ppm
I20240503 02:04:42.635273 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:42.637723 19960 webserver.cc:460] Webserver started at http://127.19.126.62:42999/ using document root <none> and password file <none>
I20240503 02:04:42.638170 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:42.638348 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:42.638593 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:42.639827 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "a6759c5ff9b94f81831416e3868f9125"
format_stamp: "Formatted at 2024-05-03 02:04:42 on dist-test-slave-8rwr"
I20240503 02:04:42.644218 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:04:42.647428 22135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:42.648052 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:04:42.648316 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "a6759c5ff9b94f81831416e3868f9125"
format_stamp: "Formatted at 2024-05-03 02:04:42 on dist-test-slave-8rwr"
I20240503 02:04:42.648562 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:42.660972 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:42.662087 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:42.697461 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:36201
I20240503 02:04:42.697530 22186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:36201 every 8 connection(s)
I20240503 02:04:42.701310 22187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:42.711751 22187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125: Bootstrap starting.
I20240503 02:04:42.715770 22187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:42.719568 22187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125: No bootstrap required, opened a new log
I20240503 02:04:42.721405 22187 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759c5ff9b94f81831416e3868f9125" member_type: VOTER }
I20240503 02:04:42.721848 22187 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:42.722015 22187 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6759c5ff9b94f81831416e3868f9125, State: Initialized, Role: FOLLOWER
I20240503 02:04:42.722460 22187 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759c5ff9b94f81831416e3868f9125" member_type: VOTER }
I20240503 02:04:42.722846 22187 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:04:42.723018 22187 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:04:42.723244 22187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:04:42.727295 22187 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759c5ff9b94f81831416e3868f9125" member_type: VOTER }
I20240503 02:04:42.727718 22187 leader_election.cc:302] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6759c5ff9b94f81831416e3868f9125; no voters:
I20240503 02:04:42.728662 22187 leader_election.cc:288] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:04:42.729060 22190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:04:42.730340 22190 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [term 1 LEADER]: Becoming Leader. State: Replica: a6759c5ff9b94f81831416e3868f9125, State: Running, Role: LEADER
I20240503 02:04:42.730999 22190 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759c5ff9b94f81831416e3868f9125" member_type: VOTER }
I20240503 02:04:42.731441 22187 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:04:42.734257 22192 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6759c5ff9b94f81831416e3868f9125. Latest consensus state: current_term: 1 leader_uuid: "a6759c5ff9b94f81831416e3868f9125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759c5ff9b94f81831416e3868f9125" member_type: VOTER } }
I20240503 02:04:42.734180 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6759c5ff9b94f81831416e3868f9125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759c5ff9b94f81831416e3868f9125" member_type: VOTER } }
I20240503 02:04:42.734984 22192 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [sys.catalog]: This master's current role is: LEADER
I20240503 02:04:42.735170 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [sys.catalog]: This master's current role is: LEADER
I20240503 02:04:42.738515 22196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:04:42.746589 22196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:04:42.747499 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:04:42.754964 22196 catalog_manager.cc:1347] Generated new cluster ID: 75fab5f0ae764b73b6fa98cd322aa235
I20240503 02:04:42.755246 22196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:04:42.777945 22196 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:04:42.779129 22196 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:04:42.794428 22196 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125: Generated new TSK 0
I20240503 02:04:42.794987 22196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:04:42.813457 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:42.819649 22208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:42.820603 22209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:42.821558 22211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:42.822350 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:42.822993 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:42.823160 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:42.823309 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701882823294 us; error 0 us; skew 500 ppm
I20240503 02:04:42.823809 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:42.825974 19960 webserver.cc:460] Webserver started at http://127.19.126.1:46517/ using document root <none> and password file <none>
I20240503 02:04:42.826407 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:42.826579 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:42.826822 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:42.827926 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-0-root/instance:
uuid: "bd086421228a4cafa137e90a156966f8"
format_stamp: "Formatted at 2024-05-03 02:04:42 on dist-test-slave-8rwr"
I20240503 02:04:42.832298 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:04:42.835465 22216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:42.836155 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:04:42.836407 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "bd086421228a4cafa137e90a156966f8"
format_stamp: "Formatted at 2024-05-03 02:04:42 on dist-test-slave-8rwr"
I20240503 02:04:42.836658 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:42.868789 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:42.869984 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:42.871457 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:04:42.873651 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:04:42.873816 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:42.874053 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:04:42.874219 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:42.911566 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:45677
I20240503 02:04:42.911659 22278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:45677 every 8 connection(s)
I20240503 02:04:42.916288 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:42.923897 22283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:42.924504 22284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:42.926726 22286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:42.927459 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:42.928310 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:42.928499 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:42.928648 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701882928631 us; error 0 us; skew 500 ppm
I20240503 02:04:42.929286 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:42.929250 22279 heartbeater.cc:347] Connected to a master server at 127.19.126.62:36201
I20240503 02:04:42.929780 22279 heartbeater.cc:464] Registering TS with master...
I20240503 02:04:42.930814 22279 heartbeater.cc:511] Master 127.19.126.62:36201 requested a full tablet report, sending...
I20240503 02:04:42.931977 19960 webserver.cc:460] Webserver started at http://127.19.126.2:34953/ using document root <none> and password file <none>
I20240503 02:04:42.932458 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:42.932637 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:42.932926 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:42.932880 22152 ts_manager.cc:194] Registered new tserver with Master: bd086421228a4cafa137e90a156966f8 (127.19.126.1:45677)
I20240503 02:04:42.934242 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-1-root/instance:
uuid: "4ab8789ea9294ff7bb7a5408dd6bb027"
format_stamp: "Formatted at 2024-05-03 02:04:42 on dist-test-slave-8rwr"
I20240503 02:04:42.934736 22152 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39402
I20240503 02:04:42.938974 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:04:42.943506 22291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:42.950894 19960 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20240503 02:04:42.951264 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "4ab8789ea9294ff7bb7a5408dd6bb027"
format_stamp: "Formatted at 2024-05-03 02:04:42 on dist-test-slave-8rwr"
I20240503 02:04:42.951597 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:42.966917 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:42.968034 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:42.969333 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:04:42.971418 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:04:42.971611 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:42.971820 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:04:42.971967 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:43.008888 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:46083
I20240503 02:04:43.008975 22353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:46083 every 8 connection(s)
I20240503 02:04:43.013419 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:43.020983 22357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:43.021730 22358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:43.023438 22360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:43.024474 22354 heartbeater.cc:347] Connected to a master server at 127.19.126.62:36201
I20240503 02:04:43.024808 22354 heartbeater.cc:464] Registering TS with master...
I20240503 02:04:43.025434 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:43.025672 22354 heartbeater.cc:511] Master 127.19.126.62:36201 requested a full tablet report, sending...
I20240503 02:04:43.026178 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:43.026381 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:43.026546 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701883026526 us; error 0 us; skew 500 ppm
I20240503 02:04:43.027254 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:43.027515 22152 ts_manager.cc:194] Registered new tserver with Master: 4ab8789ea9294ff7bb7a5408dd6bb027 (127.19.126.2:46083)
I20240503 02:04:43.029232 22152 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39404
I20240503 02:04:43.029896 19960 webserver.cc:460] Webserver started at http://127.19.126.3:41415/ using document root <none> and password file <none>
I20240503 02:04:43.030486 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:43.030725 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:43.031049 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:43.032397 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-2-root/instance:
uuid: "2f1716f02279452a8315c890726f9fa3"
format_stamp: "Formatted at 2024-05-03 02:04:43 on dist-test-slave-8rwr"
I20240503 02:04:43.036733 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:04:43.039861 22365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:43.040560 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:04:43.040844 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "2f1716f02279452a8315c890726f9fa3"
format_stamp: "Formatted at 2024-05-03 02:04:43 on dist-test-slave-8rwr"
I20240503 02:04:43.041097 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:43.060923 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:43.062036 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:43.063366 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:04:43.065498 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:04:43.065686 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:43.065898 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:04:43.066054 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:43.102748 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:33691
I20240503 02:04:43.102838 22427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:33691 every 8 connection(s)
I20240503 02:04:43.115150 22428 heartbeater.cc:347] Connected to a master server at 127.19.126.62:36201
I20240503 02:04:43.115491 22428 heartbeater.cc:464] Registering TS with master...
I20240503 02:04:43.116178 22428 heartbeater.cc:511] Master 127.19.126.62:36201 requested a full tablet report, sending...
I20240503 02:04:43.117863 22152 ts_manager.cc:194] Registered new tserver with Master: 2f1716f02279452a8315c890726f9fa3 (127.19.126.3:33691)
I20240503 02:04:43.118984 19960 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013275288s
I20240503 02:04:43.119205 22152 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39418
I20240503 02:04:43.937125 22279 heartbeater.cc:503] Master 127.19.126.62:36201 was elected leader, sending a full tablet report...
I20240503 02:04:44.031922 22354 heartbeater.cc:503] Master 127.19.126.62:36201 was elected leader, sending a full tablet report...
I20240503 02:04:44.121867 22428 heartbeater.cc:503] Master 127.19.126.62:36201 was elected leader, sending a full tablet report...
I20240503 02:04:44.745806 22205 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:46.746608 22205 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:48.747511 22205 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:50.748445 22205 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:52.749616 22205 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:54.750703 22205 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:55.211726 19960 tablet_server.cc:178] TabletServer@127.19.126.1:0 shutting down...
I20240503 02:04:55.236440 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:04:55.252835 19960 tablet_server.cc:194] TabletServer@127.19.126.1:0 shutdown complete.
I20240503 02:04:55.261858 19960 tablet_server.cc:178] TabletServer@127.19.126.2:0 shutting down...
I20240503 02:04:55.286309 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:04:55.301633 19960 tablet_server.cc:194] TabletServer@127.19.126.2:0 shutdown complete.
I20240503 02:04:55.309880 19960 tablet_server.cc:178] TabletServer@127.19.126.3:0 shutting down...
I20240503 02:04:55.329018 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:04:55.344547 19960 tablet_server.cc:194] TabletServer@127.19.126.3:0 shutdown complete.
I20240503 02:04:55.352571 19960 master.cc:537] Master@127.19.126.62:36201 shutting down...
I20240503 02:04:55.368954 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:04:55.369721 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:04:55.370074 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P a6759c5ff9b94f81831416e3868f9125: stopping tablet replica
I20240503 02:04:55.386776 19960 master.cc:552] Master@127.19.126.62:36201 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12790 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240503 02:04:55.415011 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:38891
I20240503 02:04:55.415971 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:55.421839 22433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:55.422571 22434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:55.423328 22436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:55.424096 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:55.424764 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:55.424939 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:55.425051 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701895425040 us; error 0 us; skew 500 ppm
I20240503 02:04:55.425516 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:55.427824 19960 webserver.cc:460] Webserver started at http://127.19.126.62:44611/ using document root <none> and password file <none>
I20240503 02:04:55.428220 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:55.428365 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:55.428583 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:55.429749 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "02fe891daee94769ac7ee550bb911f6d"
format_stamp: "Formatted at 2024-05-03 02:04:55 on dist-test-slave-8rwr"
I20240503 02:04:55.433780 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:04:55.436642 22441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.437364 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:04:55.437628 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "02fe891daee94769ac7ee550bb911f6d"
format_stamp: "Formatted at 2024-05-03 02:04:55 on dist-test-slave-8rwr"
I20240503 02:04:55.437868 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:55.450423 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:55.451507 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:55.486819 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:38891
I20240503 02:04:55.486914 22492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:38891 every 8 connection(s)
I20240503 02:04:55.490649 22493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:55.500695 22493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d: Bootstrap starting.
I20240503 02:04:55.504660 22493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:55.508400 22493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d: No bootstrap required, opened a new log
I20240503 02:04:55.510273 22493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fe891daee94769ac7ee550bb911f6d" member_type: VOTER }
I20240503 02:04:55.510704 22493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:55.510870 22493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe891daee94769ac7ee550bb911f6d, State: Initialized, Role: FOLLOWER
I20240503 02:04:55.511330 22493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fe891daee94769ac7ee550bb911f6d" member_type: VOTER }
I20240503 02:04:55.511721 22493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:04:55.511898 22493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:04:55.512107 22493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:04:55.515919 22493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fe891daee94769ac7ee550bb911f6d" member_type: VOTER }
I20240503 02:04:55.516368 22493 leader_election.cc:302] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02fe891daee94769ac7ee550bb911f6d; no voters:
I20240503 02:04:55.517392 22493 leader_election.cc:288] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:04:55.517745 22496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:04:55.519060 22496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [term 1 LEADER]: Becoming Leader. State: Replica: 02fe891daee94769ac7ee550bb911f6d, State: Running, Role: LEADER
I20240503 02:04:55.519608 22496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fe891daee94769ac7ee550bb911f6d" member_type: VOTER }
I20240503 02:04:55.520128 22493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:04:55.522569 22497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02fe891daee94769ac7ee550bb911f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fe891daee94769ac7ee550bb911f6d" member_type: VOTER } }
I20240503 02:04:55.522709 22498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02fe891daee94769ac7ee550bb911f6d. Latest consensus state: current_term: 1 leader_uuid: "02fe891daee94769ac7ee550bb911f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fe891daee94769ac7ee550bb911f6d" member_type: VOTER } }
I20240503 02:04:55.523325 22497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [sys.catalog]: This master's current role is: LEADER
I20240503 02:04:55.523721 22498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [sys.catalog]: This master's current role is: LEADER
I20240503 02:04:55.526067 22502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:04:55.532485 22502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:04:55.534178 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:04:55.540732 22502 catalog_manager.cc:1347] Generated new cluster ID: 0730116470e5441da151f06ba886b134
I20240503 02:04:55.541014 22502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:04:55.561738 22502 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:04:55.562937 22502 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:04:55.576335 22502 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d: Generated new TSK 0
I20240503 02:04:55.576818 22502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:04:55.599994 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:55.606099 22514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:55.607004 22515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:55.608227 22517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:55.609055 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:55.609853 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:55.610028 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:55.610181 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701895610163 us; error 0 us; skew 500 ppm
I20240503 02:04:55.610666 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:55.612840 19960 webserver.cc:460] Webserver started at http://127.19.126.1:46145/ using document root <none> and password file <none>
I20240503 02:04:55.613360 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:55.613633 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:55.613925 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:55.615020 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-0-root/instance:
uuid: "6a5fed9c46d849c8983a6403c3994622"
format_stamp: "Formatted at 2024-05-03 02:04:55 on dist-test-slave-8rwr"
I20240503 02:04:55.619128 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:04:55.622162 22522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.622870 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:04:55.623169 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "6a5fed9c46d849c8983a6403c3994622"
format_stamp: "Formatted at 2024-05-03 02:04:55 on dist-test-slave-8rwr"
I20240503 02:04:55.623401 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:55.653170 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:55.654232 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:55.655611 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:04:55.657829 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:04:55.658035 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.658255 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:04:55.658411 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.695492 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:45937
I20240503 02:04:55.695595 22584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:45937 every 8 connection(s)
I20240503 02:04:55.699904 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:55.708868 22590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:55.708161 22589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:55.710427 22592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:55.712077 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:55.712823 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:55.713050 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:55.713229 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701895713213 us; error 0 us; skew 500 ppm
I20240503 02:04:55.713433 22585 heartbeater.cc:347] Connected to a master server at 127.19.126.62:38891
I20240503 02:04:55.713783 22585 heartbeater.cc:464] Registering TS with master...
I20240503 02:04:55.713928 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:55.714650 22585 heartbeater.cc:511] Master 127.19.126.62:38891 requested a full tablet report, sending...
I20240503 02:04:55.716449 19960 webserver.cc:460] Webserver started at http://127.19.126.2:38767/ using document root <none> and password file <none>
I20240503 02:04:55.716823 22458 ts_manager.cc:194] Registered new tserver with Master: 6a5fed9c46d849c8983a6403c3994622 (127.19.126.1:45937)
I20240503 02:04:55.717034 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:55.717335 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:55.717674 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:55.719080 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-1-root/instance:
uuid: "df09306c76c74c4aa41fcae69b6df46a"
format_stamp: "Formatted at 2024-05-03 02:04:55 on dist-test-slave-8rwr"
I20240503 02:04:55.719192 22458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32996
I20240503 02:04:55.724057 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:04:55.727181 22597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.727902 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:04:55.728163 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "df09306c76c74c4aa41fcae69b6df46a"
format_stamp: "Formatted at 2024-05-03 02:04:55 on dist-test-slave-8rwr"
I20240503 02:04:55.728405 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:55.749524 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:55.750541 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:55.751845 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:04:55.753971 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:04:55.754155 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.754369 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:04:55.754513 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.791226 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:36779
I20240503 02:04:55.791316 22659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:36779 every 8 connection(s)
I20240503 02:04:55.795657 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:55.802935 22663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:55.804787 22664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:55.806167 22666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:55.806821 22660 heartbeater.cc:347] Connected to a master server at 127.19.126.62:38891
I20240503 02:04:55.807001 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:55.807188 22660 heartbeater.cc:464] Registering TS with master...
I20240503 02:04:55.807938 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:04:55.808085 22660 heartbeater.cc:511] Master 127.19.126.62:38891 requested a full tablet report, sending...
W20240503 02:04:55.808233 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:55.808459 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701895808442 us; error 0 us; skew 500 ppm
I20240503 02:04:55.809103 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:55.810180 22458 ts_manager.cc:194] Registered new tserver with Master: df09306c76c74c4aa41fcae69b6df46a (127.19.126.2:36779)
I20240503 02:04:55.811694 19960 webserver.cc:460] Webserver started at http://127.19.126.3:42555/ using document root <none> and password file <none>
I20240503 02:04:55.811703 22458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33010
I20240503 02:04:55.812283 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:55.812498 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:55.812810 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:55.813997 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-2-root/instance:
uuid: "890918ee0c894f3c96de2ef74d641d17"
format_stamp: "Formatted at 2024-05-03 02:04:55 on dist-test-slave-8rwr"
I20240503 02:04:55.817981 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:04:55.820923 22671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.821712 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:04:55.821975 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "890918ee0c894f3c96de2ef74d641d17"
format_stamp: "Formatted at 2024-05-03 02:04:55 on dist-test-slave-8rwr"
I20240503 02:04:55.822239 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:55.842731 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:55.843719 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:55.845028 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:04:55.846992 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:04:55.847182 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.847383 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:04:55.847525 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.884135 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:43077
I20240503 02:04:55.884219 22733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:43077 every 8 connection(s)
I20240503 02:04:55.888545 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:04:55.895675 22737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:04:55.897620 22738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:55.904403 22734 heartbeater.cc:347] Connected to a master server at 127.19.126.62:38891
I20240503 02:04:55.904738 22734 heartbeater.cc:464] Registering TS with master...
I20240503 02:04:55.905211 19960 server_base.cc:1034] running on GCE node
I20240503 02:04:55.905503 22734 heartbeater.cc:511] Master 127.19.126.62:38891 requested a full tablet report, sending...
W20240503 02:04:55.906127 22740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:04:55.907007 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:04:55.907276 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:04:55.907485 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701895907467 us; error 0 us; skew 500 ppm
I20240503 02:04:55.907713 22458 ts_manager.cc:194] Registered new tserver with Master: 890918ee0c894f3c96de2ef74d641d17 (127.19.126.3:43077)
I20240503 02:04:55.908252 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:04:55.909546 22458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33014
I20240503 02:04:55.911055 19960 webserver.cc:460] Webserver started at http://127.19.126.4:35995/ using document root <none> and password file <none>
I20240503 02:04:55.911528 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:04:55.911716 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:04:55.911952 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:04:55.913070 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-3-root/instance:
uuid: "934d69a9532a40839b10a5c2555fcddd"
format_stamp: "Formatted at 2024-05-03 02:04:55 on dist-test-slave-8rwr"
I20240503 02:04:55.917225 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:04:55.920147 22745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.920893 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:04:55.921164 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-3-root
uuid: "934d69a9532a40839b10a5c2555fcddd"
format_stamp: "Formatted at 2024-05-03 02:04:55 on dist-test-slave-8rwr"
I20240503 02:04:55.921461 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701820461004-19960-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:04:55.933878 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:04:55.934908 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:04:55.936379 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:04:55.938319 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:04:55.938527 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.938757 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:04:55.938890 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:04:55.976516 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.4:46065
I20240503 02:04:55.976600 22807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.4:46065 every 8 connection(s)
I20240503 02:04:55.988811 22808 heartbeater.cc:347] Connected to a master server at 127.19.126.62:38891
I20240503 02:04:55.989149 22808 heartbeater.cc:464] Registering TS with master...
I20240503 02:04:55.989885 22808 heartbeater.cc:511] Master 127.19.126.62:38891 requested a full tablet report, sending...
I20240503 02:04:55.991662 22458 ts_manager.cc:194] Registered new tserver with Master: 934d69a9532a40839b10a5c2555fcddd (127.19.126.4:46065)
I20240503 02:04:55.992743 19960 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.01339733s
I20240503 02:04:55.992914 22458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33022
I20240503 02:04:56.722015 22585 heartbeater.cc:503] Master 127.19.126.62:38891 was elected leader, sending a full tablet report...
I20240503 02:04:56.814016 22660 heartbeater.cc:503] Master 127.19.126.62:38891 was elected leader, sending a full tablet report...
I20240503 02:04:56.911927 22734 heartbeater.cc:503] Master 127.19.126.62:38891 was elected leader, sending a full tablet report...
I20240503 02:04:56.995508 22808 heartbeater.cc:503] Master 127.19.126.62:38891 was elected leader, sending a full tablet report...
I20240503 02:04:57.040580 19960 test_util.cc:268] Using random seed: 628622000
I20240503 02:04:57.060285 22458 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44174:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:04:57.099483 22550 tablet_service.cc:1467] Processing CreateTablet for tablet 3551fcf576d048528d3e9f9b0b8dabe4 (DEFAULT_TABLE table=test-workload [id=f45a94f6c1634f0db8009d07ce59543d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:04:57.100548 22550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3551fcf576d048528d3e9f9b0b8dabe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:57.103420 22699 tablet_service.cc:1467] Processing CreateTablet for tablet 3551fcf576d048528d3e9f9b0b8dabe4 (DEFAULT_TABLE table=test-workload [id=f45a94f6c1634f0db8009d07ce59543d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:04:57.104440 22773 tablet_service.cc:1467] Processing CreateTablet for tablet 3551fcf576d048528d3e9f9b0b8dabe4 (DEFAULT_TABLE table=test-workload [id=f45a94f6c1634f0db8009d07ce59543d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:04:57.104655 22699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3551fcf576d048528d3e9f9b0b8dabe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:57.105407 22773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3551fcf576d048528d3e9f9b0b8dabe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:04:57.119251 22829 tablet_bootstrap.cc:492] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd: Bootstrap starting.
I20240503 02:04:57.119940 22830 tablet_bootstrap.cc:492] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622: Bootstrap starting.
I20240503 02:04:57.121707 22831 tablet_bootstrap.cc:492] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17: Bootstrap starting.
I20240503 02:04:57.126221 22830 tablet_bootstrap.cc:654] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:57.126716 22831 tablet_bootstrap.cc:654] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:57.127209 22829 tablet_bootstrap.cc:654] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd: Neither blocks nor log segments found. Creating new log.
I20240503 02:04:57.131495 22830 tablet_bootstrap.cc:492] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622: No bootstrap required, opened a new log
I20240503 02:04:57.131824 22830 ts_tablet_manager.cc:1367] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 02:04:57.132385 22831 tablet_bootstrap.cc:492] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17: No bootstrap required, opened a new log
I20240503 02:04:57.132972 22831 ts_tablet_manager.cc:1367] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240503 02:04:57.133711 22830 raft_consensus.cc:348] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5fed9c46d849c8983a6403c3994622" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45937 } } peers { permanent_uuid: "890918ee0c894f3c96de2ef74d641d17" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43077 } } peers { permanent_uuid: "934d69a9532a40839b10a5c2555fcddd" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 46065 } }
I20240503 02:04:57.134402 22830 raft_consensus.cc:374] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:57.134690 22830 raft_consensus.cc:729] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a5fed9c46d849c8983a6403c3994622, State: Initialized, Role: FOLLOWER
I20240503 02:04:57.135426 22830 consensus_queue.cc:260] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5fed9c46d849c8983a6403c3994622" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45937 } } peers { permanent_uuid: "890918ee0c894f3c96de2ef74d641d17" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43077 } } peers { permanent_uuid: "934d69a9532a40839b10a5c2555fcddd" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 46065 } }
I20240503 02:04:57.136078 22831 raft_consensus.cc:348] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5fed9c46d849c8983a6403c3994622" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45937 } } peers { permanent_uuid: "890918ee0c894f3c96de2ef74d641d17" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43077 } } peers { permanent_uuid: "934d69a9532a40839b10a5c2555fcddd" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 46065 } }
I20240503 02:04:57.137109 22831 raft_consensus.cc:374] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:57.137548 22831 raft_consensus.cc:729] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890918ee0c894f3c96de2ef74d641d17, State: Initialized, Role: FOLLOWER
I20240503 02:04:57.138355 22830 ts_tablet_manager.cc:1398] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 02:04:57.138480 22831 consensus_queue.cc:260] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5fed9c46d849c8983a6403c3994622" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45937 } } peers { permanent_uuid: "890918ee0c894f3c96de2ef74d641d17" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43077 } } peers { permanent_uuid: "934d69a9532a40839b10a5c2555fcddd" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 46065 } }
I20240503 02:04:57.140132 22829 tablet_bootstrap.cc:492] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd: No bootstrap required, opened a new log
I20240503 02:04:57.140689 22829 ts_tablet_manager.cc:1367] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.009s
I20240503 02:04:57.141687 22831 ts_tablet_manager.cc:1398] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20240503 02:04:57.143549 22829 raft_consensus.cc:348] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5fed9c46d849c8983a6403c3994622" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45937 } } peers { permanent_uuid: "890918ee0c894f3c96de2ef74d641d17" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43077 } } peers { permanent_uuid: "934d69a9532a40839b10a5c2555fcddd" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 46065 } }
I20240503 02:04:57.144289 22829 raft_consensus.cc:374] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:04:57.144618 22829 raft_consensus.cc:729] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934d69a9532a40839b10a5c2555fcddd, State: Initialized, Role: FOLLOWER
I20240503 02:04:57.145411 22829 consensus_queue.cc:260] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5fed9c46d849c8983a6403c3994622" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45937 } } peers { permanent_uuid: "890918ee0c894f3c96de2ef74d641d17" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43077 } } peers { permanent_uuid: "934d69a9532a40839b10a5c2555fcddd" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 46065 } }
I20240503 02:04:57.150245 22829 ts_tablet_manager.cc:1398] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
W20240503 02:04:57.200599 22586 tablet.cc:2369] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:04:57.376346 22837 raft_consensus.cc:482] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:04:57.376799 22837 raft_consensus.cc:504] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5fed9c46d849c8983a6403c3994622" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45937 } } peers { permanent_uuid: "890918ee0c894f3c96de2ef74d641d17" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43077 } } peers { permanent_uuid: "934d69a9532a40839b10a5c2555fcddd" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 46065 } }
I20240503 02:04:57.378867 22837 leader_election.cc:288] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a5fed9c46d849c8983a6403c3994622 (127.19.126.1:45937), 890918ee0c894f3c96de2ef74d641d17 (127.19.126.3:43077)
I20240503 02:04:57.388322 22560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3551fcf576d048528d3e9f9b0b8dabe4" candidate_uuid: "934d69a9532a40839b10a5c2555fcddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a5fed9c46d849c8983a6403c3994622" is_pre_election: true
I20240503 02:04:57.388705 22709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3551fcf576d048528d3e9f9b0b8dabe4" candidate_uuid: "934d69a9532a40839b10a5c2555fcddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890918ee0c894f3c96de2ef74d641d17" is_pre_election: true
I20240503 02:04:57.389261 22560 raft_consensus.cc:2454] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934d69a9532a40839b10a5c2555fcddd in term 0.
I20240503 02:04:57.389428 22709 raft_consensus.cc:2454] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934d69a9532a40839b10a5c2555fcddd in term 0.
I20240503 02:04:57.390478 22749 leader_election.cc:302] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a5fed9c46d849c8983a6403c3994622, 934d69a9532a40839b10a5c2555fcddd; no voters:
I20240503 02:04:57.391088 22837 raft_consensus.cc:2781] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:04:57.391345 22837 raft_consensus.cc:482] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:04:57.391579 22837 raft_consensus.cc:3037] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:04:57.396153 22837 raft_consensus.cc:504] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5fed9c46d849c8983a6403c3994622" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45937 } } peers { permanent_uuid: "890918ee0c894f3c96de2ef74d641d17" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43077 } } peers { permanent_uuid: "934d69a9532a40839b10a5c2555fcddd" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 46065 } }
I20240503 02:04:57.398066 22837 leader_election.cc:288] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [CANDIDATE]: Term 1 election: Requested vote from peers 6a5fed9c46d849c8983a6403c3994622 (127.19.126.1:45937), 890918ee0c894f3c96de2ef74d641d17 (127.19.126.3:43077)
I20240503 02:04:57.398336 22560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3551fcf576d048528d3e9f9b0b8dabe4" candidate_uuid: "934d69a9532a40839b10a5c2555fcddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a5fed9c46d849c8983a6403c3994622"
I20240503 02:04:57.398589 22709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3551fcf576d048528d3e9f9b0b8dabe4" candidate_uuid: "934d69a9532a40839b10a5c2555fcddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890918ee0c894f3c96de2ef74d641d17"
I20240503 02:04:57.398782 22560 raft_consensus.cc:3037] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:04:57.399071 22709 raft_consensus.cc:3037] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:04:57.403194 22560 raft_consensus.cc:2454] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934d69a9532a40839b10a5c2555fcddd in term 1.
I20240503 02:04:57.403700 22709 raft_consensus.cc:2454] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934d69a9532a40839b10a5c2555fcddd in term 1.
I20240503 02:04:57.403997 22749 leader_election.cc:302] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a5fed9c46d849c8983a6403c3994622, 934d69a9532a40839b10a5c2555fcddd; no voters:
I20240503 02:04:57.404719 22837 raft_consensus.cc:2781] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:04:57.405705 22837 raft_consensus.cc:686] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 1 LEADER]: Becoming Leader. State: Replica: 934d69a9532a40839b10a5c2555fcddd, State: Running, Role: LEADER
I20240503 02:04:57.406407 22837 consensus_queue.cc:237] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5fed9c46d849c8983a6403c3994622" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45937 } } peers { permanent_uuid: "890918ee0c894f3c96de2ef74d641d17" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43077 } } peers { permanent_uuid: "934d69a9532a40839b10a5c2555fcddd" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 46065 } }
I20240503 02:04:57.412542 22458 catalog_manager.cc:5484] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 934d69a9532a40839b10a5c2555fcddd (127.19.126.4). New cstate: current_term: 1 leader_uuid: "934d69a9532a40839b10a5c2555fcddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5fed9c46d849c8983a6403c3994622" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "890918ee0c894f3c96de2ef74d641d17" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "934d69a9532a40839b10a5c2555fcddd" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 46065 } health_report { overall_health: HEALTHY } } }
I20240503 02:04:57.534227 22511 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:57.535542 22511 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:57.536041 22511 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:04:57.911545 22837 consensus_queue.cc:1035] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "890918ee0c894f3c96de2ef74d641d17" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:04:57.923507 22846 consensus_queue.cc:1035] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a5fed9c46d849c8983a6403c3994622" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:04:59.536916 22511 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:04:59.538271 22511 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:04:59.538569 22511 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:01.539634 22511 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:01.541203 22511 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:01.541499 22511 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:03.542419 22511 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:03.543570 22511 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:03.543833 22511 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:05.544683 22511 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:05.545960 22511 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:05.546236 22511 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:07.547070 22511 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:07.548365 22511 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:07.548748 22511 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:08.531188 19960 tablet_server.cc:178] TabletServer@127.19.126.1:0 shutting down...
I20240503 02:05:08.554476 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:08.555112 19960 tablet_replica.cc:330] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622: stopping tablet replica
I20240503 02:05:08.555619 19960 raft_consensus.cc:2229] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:08.556167 19960 raft_consensus.cc:2258] T 3551fcf576d048528d3e9f9b0b8dabe4 P 6a5fed9c46d849c8983a6403c3994622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:08.575415 19960 tablet_server.cc:194] TabletServer@127.19.126.1:0 shutdown complete.
I20240503 02:05:08.585369 19960 tablet_server.cc:178] TabletServer@127.19.126.2:0 shutting down...
I20240503 02:05:08.603153 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:08.618505 19960 tablet_server.cc:194] TabletServer@127.19.126.2:0 shutdown complete.
I20240503 02:05:08.627012 19960 tablet_server.cc:178] TabletServer@127.19.126.3:0 shutting down...
I20240503 02:05:08.655181 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:08.655786 19960 tablet_replica.cc:330] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17: stopping tablet replica
I20240503 02:05:08.656325 19960 raft_consensus.cc:2229] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:08.656801 19960 raft_consensus.cc:2258] T 3551fcf576d048528d3e9f9b0b8dabe4 P 890918ee0c894f3c96de2ef74d641d17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:08.675505 19960 tablet_server.cc:194] TabletServer@127.19.126.3:0 shutdown complete.
I20240503 02:05:08.684736 19960 tablet_server.cc:178] TabletServer@127.19.126.4:0 shutting down...
I20240503 02:05:08.704202 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:08.704784 19960 tablet_replica.cc:330] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd: stopping tablet replica
I20240503 02:05:08.705252 19960 raft_consensus.cc:2229] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:05:08.706112 19960 raft_consensus.cc:2258] T 3551fcf576d048528d3e9f9b0b8dabe4 P 934d69a9532a40839b10a5c2555fcddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:08.723989 19960 tablet_server.cc:194] TabletServer@127.19.126.4:0 shutdown complete.
I20240503 02:05:08.733196 19960 master.cc:537] Master@127.19.126.62:38891 shutting down...
I20240503 02:05:08.753728 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:05:08.754305 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:08.754607 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 02fe891daee94769ac7ee550bb911f6d: stopping tablet replica
I20240503 02:05:08.772058 19960 master.cc:552] Master@127.19.126.62:38891 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (13389 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240503 02:05:08.804863 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:44295
I20240503 02:05:08.805923 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:08.812101 22854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:08.812745 22855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:08.813560 22857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:08.814399 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:08.815184 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:08.815387 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:08.815522 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701908815506 us; error 0 us; skew 500 ppm
I20240503 02:05:08.815984 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:08.822793 19960 webserver.cc:460] Webserver started at http://127.19.126.62:42339/ using document root <none> and password file <none>
I20240503 02:05:08.823251 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:08.823433 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:08.823681 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:08.824817 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "63e9ee8a80864b19a1279e9fc747b168"
format_stamp: "Formatted at 2024-05-03 02:05:08 on dist-test-slave-8rwr"
I20240503 02:05:08.829053 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240503 02:05:08.832158 22862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:08.832863 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:05:08.833112 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "63e9ee8a80864b19a1279e9fc747b168"
format_stamp: "Formatted at 2024-05-03 02:05:08 on dist-test-slave-8rwr"
I20240503 02:05:08.833413 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:08.846621 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:08.847718 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:08.883208 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:44295
I20240503 02:05:08.883314 22913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:44295 every 8 connection(s)
I20240503 02:05:08.887022 22914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:08.897526 22914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168: Bootstrap starting.
I20240503 02:05:08.901495 22914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:08.905246 22914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168: No bootstrap required, opened a new log
I20240503 02:05:08.907160 22914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e9ee8a80864b19a1279e9fc747b168" member_type: VOTER }
I20240503 02:05:08.907614 22914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:08.907785 22914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e9ee8a80864b19a1279e9fc747b168, State: Initialized, Role: FOLLOWER
I20240503 02:05:08.908259 22914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e9ee8a80864b19a1279e9fc747b168" member_type: VOTER }
I20240503 02:05:08.908691 22914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:05:08.908879 22914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:05:08.909093 22914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:08.913305 22914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e9ee8a80864b19a1279e9fc747b168" member_type: VOTER }
I20240503 02:05:08.913774 22914 leader_election.cc:302] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e9ee8a80864b19a1279e9fc747b168; no voters:
I20240503 02:05:08.914808 22914 leader_election.cc:288] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:05:08.915189 22917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:08.916452 22917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [term 1 LEADER]: Becoming Leader. State: Replica: 63e9ee8a80864b19a1279e9fc747b168, State: Running, Role: LEADER
I20240503 02:05:08.917022 22917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e9ee8a80864b19a1279e9fc747b168" member_type: VOTER }
I20240503 02:05:08.917673 22914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:05:08.919812 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63e9ee8a80864b19a1279e9fc747b168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e9ee8a80864b19a1279e9fc747b168" member_type: VOTER } }
I20240503 02:05:08.919874 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e9ee8a80864b19a1279e9fc747b168. Latest consensus state: current_term: 1 leader_uuid: "63e9ee8a80864b19a1279e9fc747b168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e9ee8a80864b19a1279e9fc747b168" member_type: VOTER } }
I20240503 02:05:08.920493 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [sys.catalog]: This master's current role is: LEADER
I20240503 02:05:08.920644 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [sys.catalog]: This master's current role is: LEADER
I20240503 02:05:08.923409 22922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:05:08.931689 22922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:05:08.932600 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:05:08.940169 22922 catalog_manager.cc:1347] Generated new cluster ID: b143d544c4674bbdad0c8fd73301fa10
I20240503 02:05:08.940529 22922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:05:08.958480 22922 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:05:08.959625 22922 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:05:08.970592 22922 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168: Generated new TSK 0
I20240503 02:05:08.971088 22922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:05:08.998813 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:09.005182 22935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:09.005968 22936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:09.007081 22938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:09.007846 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:09.008599 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:09.008795 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:09.008960 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701909008940 us; error 0 us; skew 500 ppm
I20240503 02:05:09.009497 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:09.011806 19960 webserver.cc:460] Webserver started at http://127.19.126.1:41375/ using document root <none> and password file <none>
I20240503 02:05:09.012255 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:09.012435 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:09.012686 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:09.013809 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-0-root/instance:
uuid: "f4c34ed9259d4807b1b051ba7853b097"
format_stamp: "Formatted at 2024-05-03 02:05:09 on dist-test-slave-8rwr"
I20240503 02:05:09.018087 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240503 02:05:09.021219 22943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:09.022017 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:05:09.022294 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "f4c34ed9259d4807b1b051ba7853b097"
format_stamp: "Formatted at 2024-05-03 02:05:09 on dist-test-slave-8rwr"
I20240503 02:05:09.022562 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:09.037250 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:09.038290 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:09.039610 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:09.041728 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:09.041915 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:09.042138 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:09.042294 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:09.078843 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:40201
I20240503 02:05:09.078934 23005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:40201 every 8 connection(s)
I20240503 02:05:09.083205 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:09.091754 23010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:09.092938 23011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:09.094419 23013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:09.095247 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:09.095999 23006 heartbeater.cc:347] Connected to a master server at 127.19.126.62:44295
I20240503 02:05:09.096308 23006 heartbeater.cc:464] Registering TS with master...
I20240503 02:05:09.096308 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:09.096776 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:09.096958 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701909096939 us; error 0 us; skew 500 ppm
I20240503 02:05:09.097250 23006 heartbeater.cc:511] Master 127.19.126.62:44295 requested a full tablet report, sending...
I20240503 02:05:09.097566 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:09.099202 22879 ts_manager.cc:194] Registered new tserver with Master: f4c34ed9259d4807b1b051ba7853b097 (127.19.126.1:40201)
I20240503 02:05:09.100258 19960 webserver.cc:460] Webserver started at http://127.19.126.2:40341/ using document root <none> and password file <none>
I20240503 02:05:09.100737 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:09.100932 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:09.101181 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:09.101317 22879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34436
I20240503 02:05:09.102510 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-1-root/instance:
uuid: "02bb0a199f684d31b7cc2548b72ca9fd"
format_stamp: "Formatted at 2024-05-03 02:05:09 on dist-test-slave-8rwr"
I20240503 02:05:09.106681 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:05:09.109696 23018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:09.110350 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240503 02:05:09.110599 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "02bb0a199f684d31b7cc2548b72ca9fd"
format_stamp: "Formatted at 2024-05-03 02:05:09 on dist-test-slave-8rwr"
I20240503 02:05:09.110845 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:09.127512 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:09.128676 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:09.130074 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:09.132180 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:09.132378 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:09.132591 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:09.132740 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:09.169570 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:42705
I20240503 02:05:09.169660 23080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:42705 every 8 connection(s)
I20240503 02:05:09.174063 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:09.181236 23084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:09.183099 23085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:09.186005 23081 heartbeater.cc:347] Connected to a master server at 127.19.126.62:44295
I20240503 02:05:09.186473 23081 heartbeater.cc:464] Registering TS with master...
I20240503 02:05:09.187194 23081 heartbeater.cc:511] Master 127.19.126.62:44295 requested a full tablet report, sending...
W20240503 02:05:09.187783 23087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:09.188657 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:09.189396 22879 ts_manager.cc:194] Registered new tserver with Master: 02bb0a199f684d31b7cc2548b72ca9fd (127.19.126.2:42705)
I20240503 02:05:09.189498 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:09.189761 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:09.189908 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701909189894 us; error 0 us; skew 500 ppm
I20240503 02:05:09.190446 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:09.190825 22879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34442
I20240503 02:05:09.193203 19960 webserver.cc:460] Webserver started at http://127.19.126.3:45885/ using document root <none> and password file <none>
I20240503 02:05:09.193740 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:09.193909 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:09.194145 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:09.195264 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-2-root/instance:
uuid: "5afb86a2ada1445f8276f864e0eebea2"
format_stamp: "Formatted at 2024-05-03 02:05:09 on dist-test-slave-8rwr"
I20240503 02:05:09.199471 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:05:09.202347 23092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:09.203004 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:05:09.203261 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "5afb86a2ada1445f8276f864e0eebea2"
format_stamp: "Formatted at 2024-05-03 02:05:09 on dist-test-slave-8rwr"
I20240503 02:05:09.203514 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:09.215631 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:09.216691 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:09.218034 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:09.220120 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:09.220307 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:09.220528 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:09.220678 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:09.257915 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:36149
I20240503 02:05:09.258013 23154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:36149 every 8 connection(s)
I20240503 02:05:09.270469 23155 heartbeater.cc:347] Connected to a master server at 127.19.126.62:44295
I20240503 02:05:09.270808 23155 heartbeater.cc:464] Registering TS with master...
I20240503 02:05:09.271431 23155 heartbeater.cc:511] Master 127.19.126.62:44295 requested a full tablet report, sending...
I20240503 02:05:09.273193 22879 ts_manager.cc:194] Registered new tserver with Master: 5afb86a2ada1445f8276f864e0eebea2 (127.19.126.3:36149)
I20240503 02:05:09.273305 19960 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012309892s
I20240503 02:05:09.274861 22879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34450
I20240503 02:05:10.103650 23006 heartbeater.cc:503] Master 127.19.126.62:44295 was elected leader, sending a full tablet report...
I20240503 02:05:10.193586 23081 heartbeater.cc:503] Master 127.19.126.62:44295 was elected leader, sending a full tablet report...
I20240503 02:05:10.277045 23155 heartbeater.cc:503] Master 127.19.126.62:44295 was elected leader, sending a full tablet report...
I20240503 02:05:10.316887 19960 test_util.cc:268] Using random seed: 641898306
I20240503 02:05:10.338217 22879 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34460:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:05:10.340018 22879 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:05:10.380971 23046 tablet_service.cc:1467] Processing CreateTablet for tablet 39f53b819cc24cba801338acf1853254 (DEFAULT_TABLE table=test-workload [id=40047bac39b247ffa8df3f7abe29ed45]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:05:10.382323 23046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f53b819cc24cba801338acf1853254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:10.386927 22971 tablet_service.cc:1467] Processing CreateTablet for tablet 39f53b819cc24cba801338acf1853254 (DEFAULT_TABLE table=test-workload [id=40047bac39b247ffa8df3f7abe29ed45]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:05:10.388145 22971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f53b819cc24cba801338acf1853254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:10.388509 23120 tablet_service.cc:1467] Processing CreateTablet for tablet 39f53b819cc24cba801338acf1853254 (DEFAULT_TABLE table=test-workload [id=40047bac39b247ffa8df3f7abe29ed45]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:05:10.389408 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f53b819cc24cba801338acf1853254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:10.400725 23176 tablet_bootstrap.cc:492] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd: Bootstrap starting.
I20240503 02:05:10.403705 23177 tablet_bootstrap.cc:492] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2: Bootstrap starting.
I20240503 02:05:10.407109 23178 tablet_bootstrap.cc:492] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097: Bootstrap starting.
I20240503 02:05:10.407781 23176 tablet_bootstrap.cc:654] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:10.410004 23177 tablet_bootstrap.cc:654] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:10.413257 23176 tablet_bootstrap.cc:492] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd: No bootstrap required, opened a new log
I20240503 02:05:10.413748 23178 tablet_bootstrap.cc:654] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:10.413794 23176 ts_tablet_manager.cc:1367] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20240503 02:05:10.416934 23176 raft_consensus.cc:348] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 42705 } } peers { permanent_uuid: "f4c34ed9259d4807b1b051ba7853b097" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40201 } } peers { permanent_uuid: "5afb86a2ada1445f8276f864e0eebea2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 36149 } }
I20240503 02:05:10.417699 23176 raft_consensus.cc:374] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:10.417850 23177 tablet_bootstrap.cc:492] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2: No bootstrap required, opened a new log
I20240503 02:05:10.418030 23176 raft_consensus.cc:729] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bb0a199f684d31b7cc2548b72ca9fd, State: Initialized, Role: FOLLOWER
I20240503 02:05:10.418344 23177 ts_tablet_manager.cc:1367] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20240503 02:05:10.419245 23178 tablet_bootstrap.cc:492] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097: No bootstrap required, opened a new log
I20240503 02:05:10.419013 23176 consensus_queue.cc:260] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 42705 } } peers { permanent_uuid: "f4c34ed9259d4807b1b051ba7853b097" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40201 } } peers { permanent_uuid: "5afb86a2ada1445f8276f864e0eebea2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 36149 } }
I20240503 02:05:10.419709 23178 ts_tablet_manager.cc:1367] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240503 02:05:10.420526 23177 raft_consensus.cc:348] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 42705 } } peers { permanent_uuid: "f4c34ed9259d4807b1b051ba7853b097" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40201 } } peers { permanent_uuid: "5afb86a2ada1445f8276f864e0eebea2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 36149 } }
I20240503 02:05:10.421346 23177 raft_consensus.cc:374] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:10.421769 23177 raft_consensus.cc:729] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afb86a2ada1445f8276f864e0eebea2, State: Initialized, Role: FOLLOWER
I20240503 02:05:10.421936 23176 ts_tablet_manager.cc:1398] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20240503 02:05:10.422446 23178 raft_consensus.cc:348] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 42705 } } peers { permanent_uuid: "f4c34ed9259d4807b1b051ba7853b097" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40201 } } peers { permanent_uuid: "5afb86a2ada1445f8276f864e0eebea2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 36149 } }
I20240503 02:05:10.422624 23177 consensus_queue.cc:260] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 42705 } } peers { permanent_uuid: "f4c34ed9259d4807b1b051ba7853b097" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40201 } } peers { permanent_uuid: "5afb86a2ada1445f8276f864e0eebea2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 36149 } }
I20240503 02:05:10.423188 23178 raft_consensus.cc:374] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:10.423954 23178 raft_consensus.cc:729] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c34ed9259d4807b1b051ba7853b097, State: Initialized, Role: FOLLOWER
I20240503 02:05:10.424697 23178 consensus_queue.cc:260] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 42705 } } peers { permanent_uuid: "f4c34ed9259d4807b1b051ba7853b097" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40201 } } peers { permanent_uuid: "5afb86a2ada1445f8276f864e0eebea2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 36149 } }
I20240503 02:05:10.431922 23178 ts_tablet_manager.cc:1398] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097: Time spent starting tablet: real 0.012s user 0.005s sys 0.004s
I20240503 02:05:10.432914 23177 ts_tablet_manager.cc:1398] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2: Time spent starting tablet: real 0.014s user 0.003s sys 0.009s
I20240503 02:05:10.538556 23182 raft_consensus.cc:482] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:10.539036 23182 raft_consensus.cc:504] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 42705 } } peers { permanent_uuid: "f4c34ed9259d4807b1b051ba7853b097" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40201 } } peers { permanent_uuid: "5afb86a2ada1445f8276f864e0eebea2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 36149 } }
I20240503 02:05:10.540956 23182 leader_election.cc:288] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4c34ed9259d4807b1b051ba7853b097 (127.19.126.1:40201), 5afb86a2ada1445f8276f864e0eebea2 (127.19.126.3:36149)
I20240503 02:05:10.551448 22981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f53b819cc24cba801338acf1853254" candidate_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c34ed9259d4807b1b051ba7853b097" is_pre_election: true
I20240503 02:05:10.551448 23130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f53b819cc24cba801338acf1853254" candidate_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afb86a2ada1445f8276f864e0eebea2" is_pre_election: true
I20240503 02:05:10.552222 22981 raft_consensus.cc:2454] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bb0a199f684d31b7cc2548b72ca9fd in term 0.
I20240503 02:05:10.552323 23130 raft_consensus.cc:2454] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bb0a199f684d31b7cc2548b72ca9fd in term 0.
I20240503 02:05:10.553275 23022 leader_election.cc:302] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bb0a199f684d31b7cc2548b72ca9fd, f4c34ed9259d4807b1b051ba7853b097; no voters:
I20240503 02:05:10.553948 23182 raft_consensus.cc:2781] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:05:10.554277 23182 raft_consensus.cc:482] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:05:10.554533 23182 raft_consensus.cc:3037] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:10.559118 23182 raft_consensus.cc:504] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 42705 } } peers { permanent_uuid: "f4c34ed9259d4807b1b051ba7853b097" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40201 } } peers { permanent_uuid: "5afb86a2ada1445f8276f864e0eebea2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 36149 } }
I20240503 02:05:10.560611 23182 leader_election.cc:288] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [CANDIDATE]: Term 1 election: Requested vote from peers f4c34ed9259d4807b1b051ba7853b097 (127.19.126.1:40201), 5afb86a2ada1445f8276f864e0eebea2 (127.19.126.3:36149)
I20240503 02:05:10.561362 22981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f53b819cc24cba801338acf1853254" candidate_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c34ed9259d4807b1b051ba7853b097"
I20240503 02:05:10.561517 23130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f53b819cc24cba801338acf1853254" candidate_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afb86a2ada1445f8276f864e0eebea2"
I20240503 02:05:10.561803 22981 raft_consensus.cc:3037] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:10.562008 23130 raft_consensus.cc:3037] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:10.566416 22981 raft_consensus.cc:2454] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bb0a199f684d31b7cc2548b72ca9fd in term 1.
I20240503 02:05:10.566581 23130 raft_consensus.cc:2454] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bb0a199f684d31b7cc2548b72ca9fd in term 1.
I20240503 02:05:10.567341 23022 leader_election.cc:302] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bb0a199f684d31b7cc2548b72ca9fd, f4c34ed9259d4807b1b051ba7853b097; no voters:
I20240503 02:05:10.568055 23182 raft_consensus.cc:2781] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:10.569026 23182 raft_consensus.cc:686] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 1 LEADER]: Becoming Leader. State: Replica: 02bb0a199f684d31b7cc2548b72ca9fd, State: Running, Role: LEADER
I20240503 02:05:10.569743 23182 consensus_queue.cc:237] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 42705 } } peers { permanent_uuid: "f4c34ed9259d4807b1b051ba7853b097" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40201 } } peers { permanent_uuid: "5afb86a2ada1445f8276f864e0eebea2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 36149 } }
I20240503 02:05:10.575700 22878 catalog_manager.cc:5484] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 02bb0a199f684d31b7cc2548b72ca9fd (127.19.126.2). New cstate: current_term: 1 leader_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bb0a199f684d31b7cc2548b72ca9fd" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 42705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4c34ed9259d4807b1b051ba7853b097" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5afb86a2ada1445f8276f864e0eebea2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 36149 } health_report { overall_health: UNKNOWN } } }
I20240503 02:05:10.933631 22932 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:10.935691 22932 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:10.935981 22927 placement_policy_util.cc:407] tablet 39f53b819cc24cba801338acf1853254: detected majority of replicas (2 of 3) at location L0
I20240503 02:05:10.936028 22932 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:11.017841 23182 consensus_queue.cc:1035] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5afb86a2ada1445f8276f864e0eebea2" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 36149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:05:11.033159 23187 consensus_queue.cc:1035] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c34ed9259d4807b1b051ba7853b097" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:05:11.945227 22927 placement_policy_util.cc:407] tablet 39f53b819cc24cba801338acf1853254: detected majority of replicas (2 of 3) at location L0
I20240503 02:05:12.937392 22932 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:12.938656 22932 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:12.938952 22932 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:12.952461 22927 placement_policy_util.cc:407] tablet 39f53b819cc24cba801338acf1853254: detected majority of replicas (2 of 3) at location L0
I20240503 02:05:13.959928 22927 placement_policy_util.cc:407] tablet 39f53b819cc24cba801338acf1853254: detected majority of replicas (2 of 3) at location L0
I20240503 02:05:14.939795 22932 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:14.941737 22932 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:14.942168 22932 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:14.967468 22927 placement_policy_util.cc:407] tablet 39f53b819cc24cba801338acf1853254: detected majority of replicas (2 of 3) at location L0
I20240503 02:05:15.974983 22927 placement_policy_util.cc:407] tablet 39f53b819cc24cba801338acf1853254: detected majority of replicas (2 of 3) at location L0
I20240503 02:05:16.943264 22932 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:16.944973 22932 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:16.945245 22932 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:16.982626 22927 placement_policy_util.cc:407] tablet 39f53b819cc24cba801338acf1853254: detected majority of replicas (2 of 3) at location L0
I20240503 02:05:17.989945 22927 placement_policy_util.cc:407] tablet 39f53b819cc24cba801338acf1853254: detected majority of replicas (2 of 3) at location L0
I20240503 02:05:18.946352 22932 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:18.947986 22932 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:18.948380 22932 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:18.997412 22927 placement_policy_util.cc:407] tablet 39f53b819cc24cba801338acf1853254: detected majority of replicas (2 of 3) at location L0
I20240503 02:05:20.004056 22927 placement_policy_util.cc:407] tablet 39f53b819cc24cba801338acf1853254: detected majority of replicas (2 of 3) at location L0
I20240503 02:05:20.960090 22932 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:20.962550 22932 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:20.962823 22932 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:21.010620 22927 placement_policy_util.cc:407] tablet 39f53b819cc24cba801338acf1853254: detected majority of replicas (2 of 3) at location L0
I20240503 02:05:21.645172 19960 tablet_server.cc:178] TabletServer@127.19.126.1:0 shutting down...
I20240503 02:05:21.669414 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:21.670110 19960 tablet_replica.cc:330] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097: stopping tablet replica
I20240503 02:05:21.670619 19960 raft_consensus.cc:2229] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:21.671147 19960 raft_consensus.cc:2258] T 39f53b819cc24cba801338acf1853254 P f4c34ed9259d4807b1b051ba7853b097 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:21.691463 19960 tablet_server.cc:194] TabletServer@127.19.126.1:0 shutdown complete.
I20240503 02:05:21.703102 19960 tablet_server.cc:178] TabletServer@127.19.126.2:0 shutting down...
I20240503 02:05:21.721982 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:21.722657 19960 tablet_replica.cc:330] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd: stopping tablet replica
I20240503 02:05:21.723134 19960 raft_consensus.cc:2229] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:05:21.723953 19960 raft_consensus.cc:2258] T 39f53b819cc24cba801338acf1853254 P 02bb0a199f684d31b7cc2548b72ca9fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:21.742899 19960 tablet_server.cc:194] TabletServer@127.19.126.2:0 shutdown complete.
I20240503 02:05:21.752991 19960 tablet_server.cc:178] TabletServer@127.19.126.3:0 shutting down...
I20240503 02:05:21.773473 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:21.774106 19960 tablet_replica.cc:330] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2: stopping tablet replica
I20240503 02:05:21.774612 19960 raft_consensus.cc:2229] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:21.775043 19960 raft_consensus.cc:2258] T 39f53b819cc24cba801338acf1853254 P 5afb86a2ada1445f8276f864e0eebea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:21.793334 19960 tablet_server.cc:194] TabletServer@127.19.126.3:0 shutdown complete.
I20240503 02:05:21.802675 19960 master.cc:537] Master@127.19.126.62:44295 shutting down...
I20240503 02:05:21.820453 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:05:21.820974 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:21.821273 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 63e9ee8a80864b19a1279e9fc747b168: stopping tablet replica
I20240503 02:05:21.838771 19960 master.cc:552] Master@127.19.126.62:44295 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (13065 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:516: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] AutoRebalancerTest.TestMaxMovesPerServer (5 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240503 02:05:21.876513 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:46395
I20240503 02:05:21.877575 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:21.883924 23200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:21.884706 23201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:21.885450 23203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:21.886267 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:21.887010 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:21.887183 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:21.887291 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701921887281 us; error 0 us; skew 500 ppm
I20240503 02:05:21.887740 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:21.890045 19960 webserver.cc:460] Webserver started at http://127.19.126.62:35399/ using document root <none> and password file <none>
I20240503 02:05:21.890446 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:21.890599 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:21.890816 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:21.891898 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "bb660afc05844311b7394f1496ef1a22"
format_stamp: "Formatted at 2024-05-03 02:05:21 on dist-test-slave-8rwr"
I20240503 02:05:21.895998 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:05:21.898998 23208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:21.899715 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 02:05:21.899969 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "bb660afc05844311b7394f1496ef1a22"
format_stamp: "Formatted at 2024-05-03 02:05:21 on dist-test-slave-8rwr"
I20240503 02:05:21.900194 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:21.916173 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:21.917232 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:21.952010 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:46395
I20240503 02:05:21.952105 23259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:46395 every 8 connection(s)
I20240503 02:05:21.955886 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:21.966542 23260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22: Bootstrap starting.
I20240503 02:05:21.970669 23260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:21.974790 23260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22: No bootstrap required, opened a new log
I20240503 02:05:21.976760 23260 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb660afc05844311b7394f1496ef1a22" member_type: VOTER }
I20240503 02:05:21.977247 23260 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:21.977486 23260 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb660afc05844311b7394f1496ef1a22, State: Initialized, Role: FOLLOWER
I20240503 02:05:21.978061 23260 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb660afc05844311b7394f1496ef1a22" member_type: VOTER }
I20240503 02:05:21.978539 23260 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:05:21.978749 23260 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:05:21.979020 23260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:21.983247 23260 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb660afc05844311b7394f1496ef1a22" member_type: VOTER }
I20240503 02:05:21.983707 23260 leader_election.cc:302] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb660afc05844311b7394f1496ef1a22; no voters:
I20240503 02:05:21.984751 23260 leader_election.cc:288] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:05:21.985044 23263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:21.986390 23263 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [term 1 LEADER]: Becoming Leader. State: Replica: bb660afc05844311b7394f1496ef1a22, State: Running, Role: LEADER
I20240503 02:05:21.987071 23263 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb660afc05844311b7394f1496ef1a22" member_type: VOTER }
I20240503 02:05:21.987665 23260 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:05:21.990113 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb660afc05844311b7394f1496ef1a22. Latest consensus state: current_term: 1 leader_uuid: "bb660afc05844311b7394f1496ef1a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb660afc05844311b7394f1496ef1a22" member_type: VOTER } }
I20240503 02:05:21.990044 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb660afc05844311b7394f1496ef1a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb660afc05844311b7394f1496ef1a22" member_type: VOTER } }
I20240503 02:05:21.990847 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [sys.catalog]: This master's current role is: LEADER
I20240503 02:05:21.990958 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [sys.catalog]: This master's current role is: LEADER
I20240503 02:05:21.994448 23268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:05:22.002732 23268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:05:22.003661 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:05:22.011127 23268 catalog_manager.cc:1347] Generated new cluster ID: 9236e6c1668e4f06b4d7566a8c19c5c5
I20240503 02:05:22.011361 23268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:05:22.039439 23268 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:05:22.040694 23268 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:05:22.056779 23268 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22: Generated new TSK 0
I20240503 02:05:22.057375 23268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:05:22.070277 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:22.076570 23281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:22.077633 23282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:22.078642 23284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:22.079493 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:22.080204 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:22.080377 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:22.080540 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701922080523 us; error 0 us; skew 500 ppm
I20240503 02:05:22.081053 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:22.083264 19960 webserver.cc:460] Webserver started at http://127.19.126.1:46281/ using document root <none> and password file <none>
I20240503 02:05:22.083752 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:22.083921 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:22.084174 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:22.085187 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-0-root/instance:
uuid: "0196690be6f0425c8bdaa78a5aab10fb"
format_stamp: "Formatted at 2024-05-03 02:05:22 on dist-test-slave-8rwr"
I20240503 02:05:22.089471 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 02:05:22.092451 23289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:22.093233 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:05:22.093497 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "0196690be6f0425c8bdaa78a5aab10fb"
format_stamp: "Formatted at 2024-05-03 02:05:22 on dist-test-slave-8rwr"
I20240503 02:05:22.093762 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:22.105561 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:22.106604 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:22.107978 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:22.110296 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:22.110481 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:22.110718 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:22.110881 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:22.149266 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:34495
I20240503 02:05:22.149421 23351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:34495 every 8 connection(s)
I20240503 02:05:22.153738 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:22.162259 23356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:22.163110 23357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:22.164672 23359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:22.166046 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:22.166211 23352 heartbeater.cc:347] Connected to a master server at 127.19.126.62:46395
I20240503 02:05:22.166607 23352 heartbeater.cc:464] Registering TS with master...
I20240503 02:05:22.166993 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:22.167217 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:22.167285 23352 heartbeater.cc:511] Master 127.19.126.62:46395 requested a full tablet report, sending...
I20240503 02:05:22.167384 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701922167370 us; error 0 us; skew 500 ppm
I20240503 02:05:22.168049 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:22.169461 23225 ts_manager.cc:194] Registered new tserver with Master: 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495)
I20240503 02:05:22.170912 19960 webserver.cc:460] Webserver started at http://127.19.126.2:34439/ using document root <none> and password file <none>
I20240503 02:05:22.171404 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:22.171573 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:22.171586 23225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54300
I20240503 02:05:22.171900 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:22.173208 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-1-root/instance:
uuid: "033176be2d7d430e9348dd175dcb242c"
format_stamp: "Formatted at 2024-05-03 02:05:22 on dist-test-slave-8rwr"
I20240503 02:05:22.177467 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 02:05:22.180491 23364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:22.181200 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:05:22.181510 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "033176be2d7d430e9348dd175dcb242c"
format_stamp: "Formatted at 2024-05-03 02:05:22 on dist-test-slave-8rwr"
I20240503 02:05:22.181777 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:22.202701 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:22.203817 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:22.205176 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:22.207312 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:22.207504 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:22.207711 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:22.207861 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:22.244351 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:33021
I20240503 02:05:22.244438 23426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:33021 every 8 connection(s)
I20240503 02:05:22.248870 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:22.255995 23430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:22.257143 23431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:22.258114 23433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:22.259342 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:22.259847 23427 heartbeater.cc:347] Connected to a master server at 127.19.126.62:46395
I20240503 02:05:22.260179 23427 heartbeater.cc:464] Registering TS with master...
I20240503 02:05:22.260200 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:22.260540 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:22.260696 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701922260680 us; error 0 us; skew 500 ppm
I20240503 02:05:22.260987 23427 heartbeater.cc:511] Master 127.19.126.62:46395 requested a full tablet report, sending...
I20240503 02:05:22.261252 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:22.262812 23225 ts_manager.cc:194] Registered new tserver with Master: 033176be2d7d430e9348dd175dcb242c (127.19.126.2:33021)
I20240503 02:05:22.264008 19960 webserver.cc:460] Webserver started at http://127.19.126.3:41995/ using document root <none> and password file <none>
I20240503 02:05:22.264288 23225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54306
I20240503 02:05:22.264472 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:22.264715 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:22.264953 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:22.266191 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-2-root/instance:
uuid: "2fe3135122184018b5d42433a118ce05"
format_stamp: "Formatted at 2024-05-03 02:05:22 on dist-test-slave-8rwr"
I20240503 02:05:22.270270 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 02:05:22.273268 23438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:22.273998 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:05:22.274276 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "2fe3135122184018b5d42433a118ce05"
format_stamp: "Formatted at 2024-05-03 02:05:22 on dist-test-slave-8rwr"
I20240503 02:05:22.274515 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:22.296859 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:22.298034 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:22.299538 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:22.301657 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:22.301834 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:22.302062 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:22.302198 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:22.338819 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:35853
I20240503 02:05:22.338925 23500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:35853 every 8 connection(s)
I20240503 02:05:22.352061 23501 heartbeater.cc:347] Connected to a master server at 127.19.126.62:46395
I20240503 02:05:22.352401 23501 heartbeater.cc:464] Registering TS with master...
I20240503 02:05:22.353046 23501 heartbeater.cc:511] Master 127.19.126.62:46395 requested a full tablet report, sending...
I20240503 02:05:22.354830 23225 ts_manager.cc:194] Registered new tserver with Master: 2fe3135122184018b5d42433a118ce05 (127.19.126.3:35853)
I20240503 02:05:22.355523 19960 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013239316s
I20240503 02:05:22.356428 23225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54318
I20240503 02:05:23.174099 23352 heartbeater.cc:503] Master 127.19.126.62:46395 was elected leader, sending a full tablet report...
I20240503 02:05:23.266522 23427 heartbeater.cc:503] Master 127.19.126.62:46395 was elected leader, sending a full tablet report...
I20240503 02:05:23.359402 23501 heartbeater.cc:503] Master 127.19.126.62:46395 was elected leader, sending a full tablet report...
I20240503 02:05:23.388233 19960 test_util.cc:268] Using random seed: 654969653
I20240503 02:05:23.409652 23225 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54334:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:05:23.411703 23225 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:05:23.460093 23466 tablet_service.cc:1467] Processing CreateTablet for tablet 99a88e2ad895493394437673aead620c (DEFAULT_TABLE table=test-workload [id=10a0b799cdee487199f673f51f071764]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 02:05:23.460721 23465 tablet_service.cc:1467] Processing CreateTablet for tablet 926c9af7e67b48f88a9826a5430ddcb1 (DEFAULT_TABLE table=test-workload [id=10a0b799cdee487199f673f51f071764]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 02:05:23.461556 23466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a88e2ad895493394437673aead620c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:23.462013 23464 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb977b47e884f92a52721558b910903 (DEFAULT_TABLE table=test-workload [id=10a0b799cdee487199f673f51f071764]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 02:05:23.463099 23465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926c9af7e67b48f88a9826a5430ddcb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:23.464116 23464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb977b47e884f92a52721558b910903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:23.481308 23521 tablet_bootstrap.cc:492] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05: Bootstrap starting.
I20240503 02:05:23.482348 23392 tablet_service.cc:1467] Processing CreateTablet for tablet 99a88e2ad895493394437673aead620c (DEFAULT_TABLE table=test-workload [id=10a0b799cdee487199f673f51f071764]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 02:05:23.483668 23392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a88e2ad895493394437673aead620c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:23.484262 23317 tablet_service.cc:1467] Processing CreateTablet for tablet 99a88e2ad895493394437673aead620c (DEFAULT_TABLE table=test-workload [id=10a0b799cdee487199f673f51f071764]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 02:05:23.485574 23317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a88e2ad895493394437673aead620c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:23.486045 23391 tablet_service.cc:1467] Processing CreateTablet for tablet 926c9af7e67b48f88a9826a5430ddcb1 (DEFAULT_TABLE table=test-workload [id=10a0b799cdee487199f673f51f071764]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 02:05:23.486105 23315 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb977b47e884f92a52721558b910903 (DEFAULT_TABLE table=test-workload [id=10a0b799cdee487199f673f51f071764]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 02:05:23.487413 23391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926c9af7e67b48f88a9826a5430ddcb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:23.488135 23315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb977b47e884f92a52721558b910903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:23.487866 23316 tablet_service.cc:1467] Processing CreateTablet for tablet 926c9af7e67b48f88a9826a5430ddcb1 (DEFAULT_TABLE table=test-workload [id=10a0b799cdee487199f673f51f071764]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 02:05:23.489184 23316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926c9af7e67b48f88a9826a5430ddcb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:23.490841 23390 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb977b47e884f92a52721558b910903 (DEFAULT_TABLE table=test-workload [id=10a0b799cdee487199f673f51f071764]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 02:05:23.492097 23390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb977b47e884f92a52721558b910903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:23.487406 23521 tablet_bootstrap.cc:654] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:23.525033 23523 tablet_bootstrap.cc:492] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c: Bootstrap starting.
I20240503 02:05:23.525941 23524 tablet_bootstrap.cc:492] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb: Bootstrap starting.
I20240503 02:05:23.526836 23521 tablet_bootstrap.cc:492] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05: No bootstrap required, opened a new log
I20240503 02:05:23.527283 23521 ts_tablet_manager.cc:1367] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05: Time spent bootstrapping tablet: real 0.046s user 0.015s sys 0.000s
I20240503 02:05:23.529858 23521 raft_consensus.cc:348] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } }
I20240503 02:05:23.530617 23521 raft_consensus.cc:374] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:23.530905 23521 raft_consensus.cc:729] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe3135122184018b5d42433a118ce05, State: Initialized, Role: FOLLOWER
I20240503 02:05:23.531476 23524 tablet_bootstrap.cc:654] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:23.531575 23521 consensus_queue.cc:260] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } }
I20240503 02:05:23.532176 23523 tablet_bootstrap.cc:654] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:23.535760 23521 ts_tablet_manager.cc:1398] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240503 02:05:23.536370 23524 tablet_bootstrap.cc:492] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb: No bootstrap required, opened a new log
I20240503 02:05:23.536841 23521 tablet_bootstrap.cc:492] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05: Bootstrap starting.
I20240503 02:05:23.536854 23524 ts_tablet_manager.cc:1367] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240503 02:05:23.539757 23524 raft_consensus.cc:348] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } }
I20240503 02:05:23.540617 23524 raft_consensus.cc:374] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:23.540952 23524 raft_consensus.cc:729] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0196690be6f0425c8bdaa78a5aab10fb, State: Initialized, Role: FOLLOWER
I20240503 02:05:23.541767 23524 consensus_queue.cc:260] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } }
I20240503 02:05:23.544608 23521 tablet_bootstrap.cc:654] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:23.550935 23524 ts_tablet_manager.cc:1398] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20240503 02:05:23.551208 23521 tablet_bootstrap.cc:492] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05: No bootstrap required, opened a new log
I20240503 02:05:23.550951 23523 tablet_bootstrap.cc:492] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c: No bootstrap required, opened a new log
I20240503 02:05:23.551765 23521 ts_tablet_manager.cc:1367] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20240503 02:05:23.551817 23523 ts_tablet_manager.cc:1367] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.005s
I20240503 02:05:23.552485 23524 tablet_bootstrap.cc:492] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb: Bootstrap starting.
I20240503 02:05:23.554595 23523 raft_consensus.cc:348] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.554788 23521 raft_consensus.cc:348] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.555423 23523 raft_consensus.cc:374] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:23.555459 23521 raft_consensus.cc:374] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:23.556020 23521 raft_consensus.cc:729] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe3135122184018b5d42433a118ce05, State: Initialized, Role: FOLLOWER
I20240503 02:05:23.555958 23523 raft_consensus.cc:729] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033176be2d7d430e9348dd175dcb242c, State: Initialized, Role: FOLLOWER
I20240503 02:05:23.556913 23521 consensus_queue.cc:260] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.557145 23523 consensus_queue.cc:260] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.559177 23524 tablet_bootstrap.cc:654] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:23.560484 23521 ts_tablet_manager.cc:1398] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 02:05:23.561403 23521 tablet_bootstrap.cc:492] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05: Bootstrap starting.
I20240503 02:05:23.566679 23521 tablet_bootstrap.cc:654] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:23.566730 23523 ts_tablet_manager.cc:1398] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c: Time spent starting tablet: real 0.014s user 0.003s sys 0.004s
I20240503 02:05:23.567735 23523 tablet_bootstrap.cc:492] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c: Bootstrap starting.
I20240503 02:05:23.571484 23524 tablet_bootstrap.cc:492] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb: No bootstrap required, opened a new log
I20240503 02:05:23.571983 23524 ts_tablet_manager.cc:1367] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20240503 02:05:23.573462 23523 tablet_bootstrap.cc:654] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:23.574628 23524 raft_consensus.cc:348] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.575552 23524 raft_consensus.cc:374] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:23.575893 23524 raft_consensus.cc:729] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0196690be6f0425c8bdaa78a5aab10fb, State: Initialized, Role: FOLLOWER
I20240503 02:05:23.576987 23524 consensus_queue.cc:260] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.582262 23524 ts_tablet_manager.cc:1398] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240503 02:05:23.583770 23524 tablet_bootstrap.cc:492] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb: Bootstrap starting.
I20240503 02:05:23.590672 23524 tablet_bootstrap.cc:654] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:23.603992 23524 tablet_bootstrap.cc:492] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb: No bootstrap required, opened a new log
I20240503 02:05:23.604374 23524 ts_tablet_manager.cc:1367] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.004s
I20240503 02:05:23.604810 23521 tablet_bootstrap.cc:492] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05: No bootstrap required, opened a new log
I20240503 02:05:23.605201 23521 ts_tablet_manager.cc:1367] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.028s
I20240503 02:05:23.605855 23523 tablet_bootstrap.cc:492] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c: No bootstrap required, opened a new log
I20240503 02:05:23.606359 23523 ts_tablet_manager.cc:1367] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c: Time spent bootstrapping tablet: real 0.039s user 0.025s sys 0.012s
I20240503 02:05:23.607240 23521 raft_consensus.cc:348] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.607075 23524 raft_consensus.cc:348] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.607811 23521 raft_consensus.cc:374] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:23.607918 23524 raft_consensus.cc:374] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:23.608114 23521 raft_consensus.cc:729] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe3135122184018b5d42433a118ce05, State: Initialized, Role: FOLLOWER
I20240503 02:05:23.608295 23524 raft_consensus.cc:729] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0196690be6f0425c8bdaa78a5aab10fb, State: Initialized, Role: FOLLOWER
I20240503 02:05:23.608762 23521 consensus_queue.cc:260] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.608834 23523 raft_consensus.cc:348] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } }
I20240503 02:05:23.609072 23524 consensus_queue.cc:260] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.609699 23523 raft_consensus.cc:374] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:23.610098 23523 raft_consensus.cc:729] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033176be2d7d430e9348dd175dcb242c, State: Initialized, Role: FOLLOWER
I20240503 02:05:23.610673 23521 ts_tablet_manager.cc:1398] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:05:23.610831 23523 consensus_queue.cc:260] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } }
I20240503 02:05:23.613093 23524 ts_tablet_manager.cc:1398] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240503 02:05:23.614648 23523 ts_tablet_manager.cc:1398] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 02:05:23.615612 23523 tablet_bootstrap.cc:492] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c: Bootstrap starting.
I20240503 02:05:23.622547 23523 tablet_bootstrap.cc:654] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:23.626848 23523 tablet_bootstrap.cc:492] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c: No bootstrap required, opened a new log
I20240503 02:05:23.627228 23523 ts_tablet_manager.cc:1367] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240503 02:05:23.627761 23525 raft_consensus.cc:482] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:23.628134 23525 raft_consensus.cc:504] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } }
I20240503 02:05:23.629139 23523 raft_consensus.cc:348] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.629676 23523 raft_consensus.cc:374] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:23.629892 23523 raft_consensus.cc:729] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033176be2d7d430e9348dd175dcb242c, State: Initialized, Role: FOLLOWER
I20240503 02:05:23.630172 23525 leader_election.cc:288] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 033176be2d7d430e9348dd175dcb242c (127.19.126.2:33021), 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495)
I20240503 02:05:23.630712 23523 consensus_queue.cc:260] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.633728 23523 ts_tablet_manager.cc:1398] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:05:23.644301 23530 raft_consensus.cc:482] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:23.644433 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a88e2ad895493394437673aead620c" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "033176be2d7d430e9348dd175dcb242c" is_pre_election: true
I20240503 02:05:23.644863 23530 raft_consensus.cc:504] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } }
I20240503 02:05:23.645965 23402 raft_consensus.cc:2454] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 0.
I20240503 02:05:23.646384 23327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a88e2ad895493394437673aead620c" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb" is_pre_election: true
I20240503 02:05:23.647038 23327 raft_consensus.cc:2454] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 0.
I20240503 02:05:23.647241 23442 leader_election.cc:302] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 033176be2d7d430e9348dd175dcb242c, 2fe3135122184018b5d42433a118ce05; no voters:
I20240503 02:05:23.648239 23530 leader_election.cc:288] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fe3135122184018b5d42433a118ce05 (127.19.126.3:35853), 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495)
I20240503 02:05:23.648942 23525 raft_consensus.cc:2781] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:05:23.649451 23525 raft_consensus.cc:482] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:05:23.649883 23525 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:23.655937 23525 raft_consensus.cc:504] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } }
I20240503 02:05:23.658567 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a88e2ad895493394437673aead620c" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "033176be2d7d430e9348dd175dcb242c"
I20240503 02:05:23.659121 23402 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:23.660539 23476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a88e2ad895493394437673aead620c" candidate_uuid: "033176be2d7d430e9348dd175dcb242c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe3135122184018b5d42433a118ce05" is_pre_election: true
I20240503 02:05:23.661108 23476 raft_consensus.cc:2379] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 033176be2d7d430e9348dd175dcb242c in current term 1: Already voted for candidate 2fe3135122184018b5d42433a118ce05 in this term.
I20240503 02:05:23.662573 23327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a88e2ad895493394437673aead620c" candidate_uuid: "033176be2d7d430e9348dd175dcb242c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb" is_pre_election: true
I20240503 02:05:23.662854 23525 leader_election.cc:288] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 1 election: Requested vote from peers 033176be2d7d430e9348dd175dcb242c (127.19.126.2:33021), 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495)
I20240503 02:05:23.663182 23327 raft_consensus.cc:2454] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 033176be2d7d430e9348dd175dcb242c in term 0.
I20240503 02:05:23.664024 23326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a88e2ad895493394437673aead620c" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
I20240503 02:05:23.664551 23326 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:23.664296 23367 leader_election.cc:302] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0196690be6f0425c8bdaa78a5aab10fb, 033176be2d7d430e9348dd175dcb242c; no voters: 2fe3135122184018b5d42433a118ce05
I20240503 02:05:23.665437 23402 raft_consensus.cc:2454] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 1.
I20240503 02:05:23.665843 23530 raft_consensus.cc:2741] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 02:05:23.666281 23442 leader_election.cc:302] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 033176be2d7d430e9348dd175dcb242c, 2fe3135122184018b5d42433a118ce05; no voters:
I20240503 02:05:23.666934 23525 raft_consensus.cc:2781] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:23.668074 23525 raft_consensus.cc:686] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 1 LEADER]: Becoming Leader. State: Replica: 2fe3135122184018b5d42433a118ce05, State: Running, Role: LEADER
I20240503 02:05:23.668879 23525 consensus_queue.cc:237] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } }
I20240503 02:05:23.670948 23326 raft_consensus.cc:2454] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 1.
I20240503 02:05:23.676553 23224 catalog_manager.cc:5484] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fe3135122184018b5d42433a118ce05 (127.19.126.3). New cstate: current_term: 1 leader_uuid: "2fe3135122184018b5d42433a118ce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20240503 02:05:23.710001 23525 raft_consensus.cc:482] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:23.710388 23525 raft_consensus.cc:504] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.711933 23525 leader_election.cc:288] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495), 033176be2d7d430e9348dd175dcb242c (127.19.126.2:33021)
I20240503 02:05:23.712697 23326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb977b47e884f92a52721558b910903" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb" is_pre_election: true
I20240503 02:05:23.712872 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb977b47e884f92a52721558b910903" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "033176be2d7d430e9348dd175dcb242c" is_pre_election: true
I20240503 02:05:23.713243 23326 raft_consensus.cc:2454] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 0.
I20240503 02:05:23.713459 23402 raft_consensus.cc:2454] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 0.
I20240503 02:05:23.714298 23441 leader_election.cc:302] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196690be6f0425c8bdaa78a5aab10fb, 2fe3135122184018b5d42433a118ce05; no voters:
I20240503 02:05:23.714941 23525 raft_consensus.cc:2781] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:05:23.715313 23525 raft_consensus.cc:482] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:05:23.715562 23525 raft_consensus.cc:3037] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:23.719833 23525 raft_consensus.cc:504] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.721222 23525 leader_election.cc:288] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 1 election: Requested vote from peers 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495), 033176be2d7d430e9348dd175dcb242c (127.19.126.2:33021)
I20240503 02:05:23.721859 23326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb977b47e884f92a52721558b910903" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
I20240503 02:05:23.722178 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb977b47e884f92a52721558b910903" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "033176be2d7d430e9348dd175dcb242c"
I20240503 02:05:23.722297 23326 raft_consensus.cc:3037] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:23.722669 23402 raft_consensus.cc:3037] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:23.726661 23326 raft_consensus.cc:2454] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 1.
I20240503 02:05:23.726727 23402 raft_consensus.cc:2454] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 1.
I20240503 02:05:23.727599 23441 leader_election.cc:302] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196690be6f0425c8bdaa78a5aab10fb, 2fe3135122184018b5d42433a118ce05; no voters:
I20240503 02:05:23.728200 23525 raft_consensus.cc:2781] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:23.728585 23525 raft_consensus.cc:686] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 1 LEADER]: Becoming Leader. State: Replica: 2fe3135122184018b5d42433a118ce05, State: Running, Role: LEADER
I20240503 02:05:23.729275 23525 consensus_queue.cc:237] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.735456 23225 catalog_manager.cc:5484] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fe3135122184018b5d42433a118ce05 (127.19.126.3). New cstate: current_term: 1 leader_uuid: "2fe3135122184018b5d42433a118ce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } health_report { overall_health: HEALTHY } } }
I20240503 02:05:23.771633 23525 raft_consensus.cc:482] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:23.772138 23525 raft_consensus.cc:504] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.773756 23525 leader_election.cc:288] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495), 033176be2d7d430e9348dd175dcb242c (127.19.126.2:33021)
I20240503 02:05:23.774470 23326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb" is_pre_election: true
I20240503 02:05:23.774708 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "033176be2d7d430e9348dd175dcb242c" is_pre_election: true
I20240503 02:05:23.775095 23326 raft_consensus.cc:2454] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 0.
I20240503 02:05:23.775287 23402 raft_consensus.cc:2454] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 0.
I20240503 02:05:23.776252 23441 leader_election.cc:302] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196690be6f0425c8bdaa78a5aab10fb, 2fe3135122184018b5d42433a118ce05; no voters:
I20240503 02:05:23.776891 23525 raft_consensus.cc:2781] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:05:23.777213 23525 raft_consensus.cc:482] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:05:23.777496 23525 raft_consensus.cc:3037] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:23.781769 23525 raft_consensus.cc:504] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.783236 23525 leader_election.cc:288] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 1 election: Requested vote from peers 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495), 033176be2d7d430e9348dd175dcb242c (127.19.126.2:33021)
I20240503 02:05:23.783866 23326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
I20240503 02:05:23.784309 23326 raft_consensus.cc:3037] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:23.784404 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "033176be2d7d430e9348dd175dcb242c"
I20240503 02:05:23.784933 23402 raft_consensus.cc:3037] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:23.788519 23326 raft_consensus.cc:2454] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 1.
I20240503 02:05:23.789116 23402 raft_consensus.cc:2454] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 1.
I20240503 02:05:23.789523 23441 leader_election.cc:302] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196690be6f0425c8bdaa78a5aab10fb, 2fe3135122184018b5d42433a118ce05; no voters:
I20240503 02:05:23.790163 23525 raft_consensus.cc:2781] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:23.790549 23525 raft_consensus.cc:686] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 1 LEADER]: Becoming Leader. State: Replica: 2fe3135122184018b5d42433a118ce05, State: Running, Role: LEADER
I20240503 02:05:23.791213 23525 consensus_queue.cc:237] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:23.797119 23225 catalog_manager.cc:5484] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fe3135122184018b5d42433a118ce05 (127.19.126.3). New cstate: current_term: 1 leader_uuid: "2fe3135122184018b5d42433a118ce05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } health_report { overall_health: HEALTHY } } }
I20240503 02:05:23.865830 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:23.872133 23544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:23.872957 23545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:23.873932 23547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:23.875043 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:23.876190 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:23.876395 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:23.876552 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701923876536 us; error 0 us; skew 500 ppm
I20240503 02:05:23.877038 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:23.879252 19960 webserver.cc:460] Webserver started at http://127.19.126.4:35613/ using document root <none> and password file <none>
I20240503 02:05:23.879684 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:23.879858 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:23.880095 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:23.881108 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-3-root/instance:
uuid: "a1260b4816b6471cacfda910361d86ed"
format_stamp: "Formatted at 2024-05-03 02:05:23 on dist-test-slave-8rwr"
I20240503 02:05:23.885507 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240503 02:05:23.888700 23552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:23.889477 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:05:23.889703 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-3-root
uuid: "a1260b4816b6471cacfda910361d86ed"
format_stamp: "Formatted at 2024-05-03 02:05:23 on dist-test-slave-8rwr"
I20240503 02:05:23.889911 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:23.907553 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:23.908522 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:23.909644 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:23.911526 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:23.911712 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:23.911931 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:23.912073 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:23.948369 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.4:40255
I20240503 02:05:23.948447 23614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.4:40255 every 8 connection(s)
I20240503 02:05:23.961192 23615 heartbeater.cc:347] Connected to a master server at 127.19.126.62:46395
I20240503 02:05:23.961577 23615 heartbeater.cc:464] Registering TS with master...
I20240503 02:05:23.962247 23615 heartbeater.cc:511] Master 127.19.126.62:46395 requested a full tablet report, sending...
I20240503 02:05:23.963968 23225 ts_manager.cc:194] Registered new tserver with Master: a1260b4816b6471cacfda910361d86ed (127.19.126.4:40255)
I20240503 02:05:23.965204 23225 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54350
I20240503 02:05:23.981092 23525 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:05:23.986677 23525 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:05:23.987920 23525 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240503 02:05:23.988209 23525 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240503 02:05:24.002939 23278 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:24.017005 23476 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1"
dest_uuid: "2fe3135122184018b5d42433a118ce05"
mode: GRACEFUL
new_leader_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
from {username='slave'} at 127.0.0.1:39276
I20240503 02:05:24.017774 23476 raft_consensus.cc:595] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 1 LEADER]: Received request to transfer leadership to TS 0196690be6f0425c8bdaa78a5aab10fb
I20240503 02:05:24.020058 23476 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99a88e2ad895493394437673aead620c"
dest_uuid: "2fe3135122184018b5d42433a118ce05"
mode: GRACEFUL
new_leader_uuid: "033176be2d7d430e9348dd175dcb242c"
from {username='slave'} at 127.0.0.1:39276
I20240503 02:05:24.020234 23475 consensus_queue.cc:237] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:24.023324 23476 raft_consensus.cc:595] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 1 LEADER]: Received request to transfer leadership to TS 033176be2d7d430e9348dd175dcb242c
I20240503 02:05:24.026553 23326 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 1 FOLLOWER]: Refusing update from remote peer 2fe3135122184018b5d42433a118ce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:05:24.028244 23620 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:05:24.028846 23278 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 02:05:24.029007 23402 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 1 FOLLOWER]: Refusing update from remote peer 2fe3135122184018b5d42433a118ce05: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:05:24.029484 23278 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:24.031738 23540 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:05:24.032399 23620 consensus_queue.cc:1035] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:05:24.037050 23540 consensus_queue.cc:1035] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:05:24.042239 23326 raft_consensus.cc:2932] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a1260b4816b6471cacfda910361d86ed (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } } }
I20240503 02:05:24.042429 23400 raft_consensus.cc:2932] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a1260b4816b6471cacfda910361d86ed (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } } }
I20240503 02:05:24.058071 23225 catalog_manager.cc:5484] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb reported cstate change: config changed from index -1 to 2, NON_VOTER a1260b4816b6471cacfda910361d86ed (127.19.126.4) added. New cstate: current_term: 1 leader_uuid: "2fe3135122184018b5d42433a118ce05" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } } }
W20240503 02:05:24.061113 23439 consensus_peers.cc:485] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 -> Peer a1260b4816b6471cacfda910361d86ed (127.19.126.4:40255): Couldn't send request to peer a1260b4816b6471cacfda910361d86ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a88e2ad895493394437673aead620c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:24.114198 23540 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:05:24.161772 23540 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:05:24.166577 23627 raft_consensus.cc:979] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05: : Instructing follower 0196690be6f0425c8bdaa78a5aab10fb to start an election
I20240503 02:05:24.167312 23540 raft_consensus.cc:1067] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 1 LEADER]: Signalling peer 0196690be6f0425c8bdaa78a5aab10fb to start an election
I20240503 02:05:24.168648 23326 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1"
dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
from {username='slave'} at 127.0.0.1:41908
I20240503 02:05:24.169066 23326 raft_consensus.cc:482] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:05:24.169327 23326 raft_consensus.cc:3037] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:05:24.173473 23326 raft_consensus.cc:504] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:24.175148 23326 leader_election.cc:288] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [CANDIDATE]: Term 2 election: Requested vote from peers 033176be2d7d430e9348dd175dcb242c (127.19.126.2:33021), 2fe3135122184018b5d42433a118ce05 (127.19.126.3:35853)
I20240503 02:05:24.185854 23475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1" candidate_uuid: "0196690be6f0425c8bdaa78a5aab10fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2fe3135122184018b5d42433a118ce05"
I20240503 02:05:24.185871 23401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1" candidate_uuid: "0196690be6f0425c8bdaa78a5aab10fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "033176be2d7d430e9348dd175dcb242c"
I20240503 02:05:24.186425 23475 raft_consensus.cc:3032] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:05:24.186511 23401 raft_consensus.cc:3037] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:05:24.186708 23475 raft_consensus.cc:729] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2fe3135122184018b5d42433a118ce05, State: Running, Role: LEADER
I20240503 02:05:24.187484 23475 consensus_queue.cc:260] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:24.188462 23475 raft_consensus.cc:3037] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:05:24.191409 23401 raft_consensus.cc:2454] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0196690be6f0425c8bdaa78a5aab10fb in term 2.
I20240503 02:05:24.192286 23293 leader_election.cc:302] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196690be6f0425c8bdaa78a5aab10fb, 033176be2d7d430e9348dd175dcb242c; no voters:
I20240503 02:05:24.193195 23528 raft_consensus.cc:2781] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:05:24.193329 23475 raft_consensus.cc:2454] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0196690be6f0425c8bdaa78a5aab10fb in term 2.
I20240503 02:05:24.194520 23528 raft_consensus.cc:686] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 2 LEADER]: Becoming Leader. State: Replica: 0196690be6f0425c8bdaa78a5aab10fb, State: Running, Role: LEADER
I20240503 02:05:24.195266 23528 consensus_queue.cc:237] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } }
I20240503 02:05:24.300413 23224 catalog_manager.cc:5484] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb reported cstate change: term changed from 1 to 2, leader changed from 2fe3135122184018b5d42433a118ce05 (127.19.126.3) to 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1). New cstate: current_term: 2 leader_uuid: "0196690be6f0425c8bdaa78a5aab10fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } health_report { overall_health: UNKNOWN } } }
I20240503 02:05:24.314114 23639 ts_tablet_manager.cc:902] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed: Initiating tablet copy from peer 2fe3135122184018b5d42433a118ce05 (127.19.126.3:35853)
I20240503 02:05:24.317577 23639 tablet_copy_client.cc:308] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed: tablet copy: Beginning tablet copy session from remote peer at address 127.19.126.3:35853
I20240503 02:05:24.324898 23486 tablet_copy_service.cc:140] P 2fe3135122184018b5d42433a118ce05: Received BeginTabletCopySession request for tablet 99a88e2ad895493394437673aead620c from peer a1260b4816b6471cacfda910361d86ed ({username='slave'} at 127.0.0.1:39300)
I20240503 02:05:24.325351 23486 tablet_copy_service.cc:161] P 2fe3135122184018b5d42433a118ce05: Beginning new tablet copy session on tablet 99a88e2ad895493394437673aead620c from peer a1260b4816b6471cacfda910361d86ed at {username='slave'} at 127.0.0.1:39300: session id = a1260b4816b6471cacfda910361d86ed-99a88e2ad895493394437673aead620c
I20240503 02:05:24.329914 23486 tablet_copy_source_session.cc:205] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:05:24.332260 23639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a88e2ad895493394437673aead620c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:24.342072 23639 tablet_copy_client.cc:783] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed: tablet copy: Starting download of 0 data blocks...
I20240503 02:05:24.342458 23639 tablet_copy_client.cc:647] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed: tablet copy: Starting download of 1 WAL segments...
I20240503 02:05:24.345137 23639 tablet_copy_client.cc:523] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:05:24.350450 23639 tablet_bootstrap.cc:492] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed: Bootstrap starting.
I20240503 02:05:24.362504 23639 tablet_bootstrap.cc:492] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:05:24.363162 23639 tablet_bootstrap.cc:492] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed: Bootstrap complete.
I20240503 02:05:24.363662 23639 ts_tablet_manager.cc:1367] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 02:05:24.365473 23639 raft_consensus.cc:348] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } }
I20240503 02:05:24.366703 23639 raft_consensus.cc:729] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a1260b4816b6471cacfda910361d86ed, State: Initialized, Role: LEARNER
I20240503 02:05:24.367305 23639 consensus_queue.cc:260] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:24.369515 23615 heartbeater.cc:503] Master 127.19.126.62:46395 was elected leader, sending a full tablet report...
I20240503 02:05:24.369740 23639 ts_tablet_manager.cc:1398] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20240503 02:05:24.371317 23486 tablet_copy_service.cc:342] P 2fe3135122184018b5d42433a118ce05: Request end of tablet copy session a1260b4816b6471cacfda910361d86ed-99a88e2ad895493394437673aead620c received from {username='slave'} at 127.0.0.1:39300
I20240503 02:05:24.371693 23486 tablet_copy_service.cc:433] P 2fe3135122184018b5d42433a118ce05: ending tablet copy session a1260b4816b6471cacfda910361d86ed-99a88e2ad895493394437673aead620c on tablet 99a88e2ad895493394437673aead620c with peer a1260b4816b6471cacfda910361d86ed
I20240503 02:05:24.441308 23475 raft_consensus.cc:1261] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 2 FOLLOWER]: Refusing update from remote peer 0196690be6f0425c8bdaa78a5aab10fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:05:24.442449 23528 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:05:24.561362 23401 raft_consensus.cc:1261] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 2 FOLLOWER]: Refusing update from remote peer 0196690be6f0425c8bdaa78a5aab10fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:05:24.562553 23528 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:05:24.572212 23590 raft_consensus.cc:1203] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:05:24.573038 23590 raft_consensus.cc:2932] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 1 LEARNER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a1260b4816b6471cacfda910361d86ed (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } } }
I20240503 02:05:24.631758 23525 raft_consensus.cc:979] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05: : Instructing follower 033176be2d7d430e9348dd175dcb242c to start an election
I20240503 02:05:24.632388 23634 raft_consensus.cc:1067] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 1 LEADER]: Signalling peer 033176be2d7d430e9348dd175dcb242c to start an election
I20240503 02:05:24.633857 23401 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99a88e2ad895493394437673aead620c"
dest_uuid: "033176be2d7d430e9348dd175dcb242c"
from {username='slave'} at 127.0.0.1:36174
I20240503 02:05:24.634305 23401 raft_consensus.cc:482] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:05:24.634562 23401 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:05:24.638796 23401 raft_consensus.cc:504] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:24.640367 23401 leader_election.cc:288] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [CANDIDATE]: Term 2 election: Requested vote from peers 2fe3135122184018b5d42433a118ce05 (127.19.126.3:35853), 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495)
I20240503 02:05:24.641176 23475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a88e2ad895493394437673aead620c" candidate_uuid: "033176be2d7d430e9348dd175dcb242c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "2fe3135122184018b5d42433a118ce05"
I20240503 02:05:24.641322 23326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a88e2ad895493394437673aead620c" candidate_uuid: "033176be2d7d430e9348dd175dcb242c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
I20240503 02:05:24.641680 23475 raft_consensus.cc:3032] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:05:24.641913 23326 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:05:24.642024 23475 raft_consensus.cc:729] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2fe3135122184018b5d42433a118ce05, State: Running, Role: LEADER
I20240503 02:05:24.642705 23475 consensus_queue.cc:260] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:24.643836 23475 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:05:24.646410 23326 raft_consensus.cc:2454] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 033176be2d7d430e9348dd175dcb242c in term 2.
I20240503 02:05:24.647305 23367 leader_election.cc:302] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196690be6f0425c8bdaa78a5aab10fb, 033176be2d7d430e9348dd175dcb242c; no voters:
I20240503 02:05:24.648049 23530 raft_consensus.cc:2781] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:05:24.648178 23475 raft_consensus.cc:2454] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 033176be2d7d430e9348dd175dcb242c in term 2.
I20240503 02:05:24.649071 23530 raft_consensus.cc:686] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 2 LEADER]: Becoming Leader. State: Replica: 033176be2d7d430e9348dd175dcb242c, State: Running, Role: LEADER
I20240503 02:05:24.649832 23530 consensus_queue.cc:237] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:24.655800 23225 catalog_manager.cc:5484] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c reported cstate change: term changed from 1 to 2, leader changed from 2fe3135122184018b5d42433a118ce05 (127.19.126.3) to 033176be2d7d430e9348dd175dcb242c (127.19.126.2). New cstate: current_term: 2 leader_uuid: "033176be2d7d430e9348dd175dcb242c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:05:24.913185 23475 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 2 FOLLOWER]: Refusing update from remote peer 033176be2d7d430e9348dd175dcb242c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:05:24.914450 23530 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:05:24.916646 23475 raft_consensus.cc:2932] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 2 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a1260b4816b6471cacfda910361d86ed (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } } }
I20240503 02:05:24.937124 23326 consensus_queue.cc:237] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:24.944501 23401 raft_consensus.cc:1261] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 2 FOLLOWER]: Refusing update from remote peer 0196690be6f0425c8bdaa78a5aab10fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:05:24.944732 23475 raft_consensus.cc:1261] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 2 FOLLOWER]: Refusing update from remote peer 0196690be6f0425c8bdaa78a5aab10fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:05:24.947314 23649 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:05:24.948473 23528 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:05:24.957320 23401 raft_consensus.cc:2932] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a1260b4816b6471cacfda910361d86ed (127.19.126.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } } }
W20240503 02:05:24.958845 23290 consensus_peers.cc:485] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb -> Peer a1260b4816b6471cacfda910361d86ed (127.19.126.4:40255): Couldn't send request to peer a1260b4816b6471cacfda910361d86ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 926c9af7e67b48f88a9826a5430ddcb1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:24.957547 23475 raft_consensus.cc:2932] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a1260b4816b6471cacfda910361d86ed (127.19.126.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } } }
I20240503 02:05:24.960307 23326 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 2 FOLLOWER]: Refusing update from remote peer 033176be2d7d430e9348dd175dcb242c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:05:24.961305 23530 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:05:24.971526 23225 catalog_manager.cc:5484] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c reported cstate change: config changed from index -1 to 3, NON_VOTER a1260b4816b6471cacfda910361d86ed (127.19.126.4) added. New cstate: current_term: 2 leader_uuid: "0196690be6f0425c8bdaa78a5aab10fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } } }
I20240503 02:05:24.981276 23590 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 1 LEARNER]: Advancing to term 2
I20240503 02:05:24.985960 23590 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 2 LEARNER]: Refusing update from remote peer 033176be2d7d430e9348dd175dcb242c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:05:24.986981 23530 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:05:25.161733 23528 raft_consensus.cc:2932] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a1260b4816b6471cacfda910361d86ed (127.19.126.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } } }
I20240503 02:05:25.173455 23401 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99a88e2ad895493394437673aead620c"
dest_uuid: "033176be2d7d430e9348dd175dcb242c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36162
I20240503 02:05:25.174021 23401 raft_consensus.cc:595] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 2 LEADER]: Received request to transfer leadership
I20240503 02:05:25.178475 23326 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1"
dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41902
I20240503 02:05:25.178926 23326 raft_consensus.cc:595] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 2 LEADER]: Received request to transfer leadership
I20240503 02:05:25.183334 23401 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99a88e2ad895493394437673aead620c"
dest_uuid: "033176be2d7d430e9348dd175dcb242c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36162
I20240503 02:05:25.183797 23401 raft_consensus.cc:595] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 2 LEADER]: Received request to transfer leadership
W20240503 02:05:25.184608 23212 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 99a88e2ad895493394437673aead620c already in progress [suppressed 3 similar messages]
I20240503 02:05:25.186688 23401 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99a88e2ad895493394437673aead620c"
dest_uuid: "033176be2d7d430e9348dd175dcb242c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36162
I20240503 02:05:25.187114 23401 raft_consensus.cc:595] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 2 LEADER]: Received request to transfer leadership
W20240503 02:05:25.188046 23273 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 99a88e2ad895493394437673aead620c already in progress
W20240503 02:05:25.188278 23273 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 99a88e2ad895493394437673aead620c already in progress
I20240503 02:05:25.191954 23326 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1"
dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41902
I20240503 02:05:25.192452 23326 raft_consensus.cc:595] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 2 LEADER]: Received request to transfer leadership
I20240503 02:05:25.194700 23326 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1"
dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41902
I20240503 02:05:25.195215 23326 raft_consensus.cc:595] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 2 LEADER]: Received request to transfer leadership
W20240503 02:05:25.196483 23273 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 926c9af7e67b48f88a9826a5430ddcb1 already in progress
W20240503 02:05:25.196811 23273 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 926c9af7e67b48f88a9826a5430ddcb1 already in progress
I20240503 02:05:25.225081 23647 raft_consensus.cc:1050] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c: attempting to promote NON_VOTER a1260b4816b6471cacfda910361d86ed to VOTER
W20240503 02:05:25.225507 23647 raft_consensus.cc:1054] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c: Unable to promote non-voter a1260b4816b6471cacfda910361d86ed: Service unavailable: leader transfer in progress
I20240503 02:05:25.239463 23659 ts_tablet_manager.cc:902] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed: Initiating tablet copy from peer 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495)
I20240503 02:05:25.242398 23659 tablet_copy_client.cc:308] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed: tablet copy: Beginning tablet copy session from remote peer at address 127.19.126.1:34495
I20240503 02:05:25.251318 23337 tablet_copy_service.cc:140] P 0196690be6f0425c8bdaa78a5aab10fb: Received BeginTabletCopySession request for tablet 926c9af7e67b48f88a9826a5430ddcb1 from peer a1260b4816b6471cacfda910361d86ed ({username='slave'} at 127.0.0.1:41936)
I20240503 02:05:25.251729 23337 tablet_copy_service.cc:161] P 0196690be6f0425c8bdaa78a5aab10fb: Beginning new tablet copy session on tablet 926c9af7e67b48f88a9826a5430ddcb1 from peer a1260b4816b6471cacfda910361d86ed at {username='slave'} at 127.0.0.1:41936: session id = a1260b4816b6471cacfda910361d86ed-926c9af7e67b48f88a9826a5430ddcb1
I20240503 02:05:25.256580 23337 tablet_copy_source_session.cc:205] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:05:25.258994 23659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926c9af7e67b48f88a9826a5430ddcb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:25.269161 23659 tablet_copy_client.cc:783] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed: tablet copy: Starting download of 0 data blocks...
I20240503 02:05:25.269651 23659 tablet_copy_client.cc:647] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed: tablet copy: Starting download of 1 WAL segments...
I20240503 02:05:25.272462 23659 tablet_copy_client.cc:523] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:05:25.277568 23659 tablet_bootstrap.cc:492] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed: Bootstrap starting.
I20240503 02:05:25.291630 23659 tablet_bootstrap.cc:492] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:05:25.292274 23659 tablet_bootstrap.cc:492] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed: Bootstrap complete.
I20240503 02:05:25.292788 23659 ts_tablet_manager.cc:1367] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240503 02:05:25.294629 23659 raft_consensus.cc:348] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:25.295239 23659 raft_consensus.cc:729] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a1260b4816b6471cacfda910361d86ed, State: Initialized, Role: LEARNER
I20240503 02:05:25.295718 23659 consensus_queue.cc:260] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:25.297551 23659 ts_tablet_manager.cc:1398] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20240503 02:05:25.299139 23337 tablet_copy_service.cc:342] P 0196690be6f0425c8bdaa78a5aab10fb: Request end of tablet copy session a1260b4816b6471cacfda910361d86ed-926c9af7e67b48f88a9826a5430ddcb1 received from {username='slave'} at 127.0.0.1:41936
I20240503 02:05:25.299443 23337 tablet_copy_service.cc:433] P 0196690be6f0425c8bdaa78a5aab10fb: ending tablet copy session a1260b4816b6471cacfda910361d86ed-926c9af7e67b48f88a9826a5430ddcb1 on tablet 926c9af7e67b48f88a9826a5430ddcb1 with peer a1260b4816b6471cacfda910361d86ed
I20240503 02:05:25.464035 23530 raft_consensus.cc:979] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c: : Instructing follower 2fe3135122184018b5d42433a118ce05 to start an election
I20240503 02:05:25.464533 23647 raft_consensus.cc:1067] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 2 LEADER]: Signalling peer 2fe3135122184018b5d42433a118ce05 to start an election
I20240503 02:05:25.466027 23475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99a88e2ad895493394437673aead620c"
dest_uuid: "2fe3135122184018b5d42433a118ce05"
from {username='slave'} at 127.0.0.1:39270
I20240503 02:05:25.466475 23475 raft_consensus.cc:482] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:05:25.466785 23475 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:05:25.471205 23475 raft_consensus.cc:504] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:25.472764 23475 leader_election.cc:288] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 3 election: Requested vote from peers 033176be2d7d430e9348dd175dcb242c (127.19.126.2:33021), 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495)
I20240503 02:05:25.473562 23401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a88e2ad895493394437673aead620c" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "033176be2d7d430e9348dd175dcb242c"
I20240503 02:05:25.473883 23326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a88e2ad895493394437673aead620c" candidate_uuid: "2fe3135122184018b5d42433a118ce05" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
I20240503 02:05:25.474133 23401 raft_consensus.cc:3032] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:05:25.474398 23326 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:05:25.474439 23401 raft_consensus.cc:729] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 033176be2d7d430e9348dd175dcb242c, State: Running, Role: LEADER
I20240503 02:05:25.475339 23401 consensus_queue.cc:260] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:25.476639 23401 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:05:25.479035 23326 raft_consensus.cc:2454] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 3.
I20240503 02:05:25.480000 23441 leader_election.cc:302] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196690be6f0425c8bdaa78a5aab10fb, 2fe3135122184018b5d42433a118ce05; no voters:
I20240503 02:05:25.480794 23525 raft_consensus.cc:2781] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:05:25.481235 23525 raft_consensus.cc:686] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 3 LEADER]: Becoming Leader. State: Replica: 2fe3135122184018b5d42433a118ce05, State: Running, Role: LEADER
I20240503 02:05:25.481412 23401 raft_consensus.cc:2454] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe3135122184018b5d42433a118ce05 in term 3.
I20240503 02:05:25.482197 23525 consensus_queue.cc:237] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:25.488639 23224 catalog_manager.cc:5484] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 reported cstate change: term changed from 2 to 3, leader changed from 033176be2d7d430e9348dd175dcb242c (127.19.126.2) to 2fe3135122184018b5d42433a118ce05 (127.19.126.3). New cstate: current_term: 3 leader_uuid: "2fe3135122184018b5d42433a118ce05" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:05:25.489931 23652 raft_consensus.cc:979] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb: : Instructing follower 033176be2d7d430e9348dd175dcb242c to start an election
I20240503 02:05:25.490504 23638 raft_consensus.cc:1067] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 2 LEADER]: Signalling peer 033176be2d7d430e9348dd175dcb242c to start an election
I20240503 02:05:25.491969 23401 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1"
dest_uuid: "033176be2d7d430e9348dd175dcb242c"
from {username='slave'} at 127.0.0.1:36178
I20240503 02:05:25.492545 23401 raft_consensus.cc:482] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:05:25.493031 23401 raft_consensus.cc:3037] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:05:25.498221 23401 raft_consensus.cc:504] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:25.499647 23401 leader_election.cc:288] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [CANDIDATE]: Term 3 election: Requested vote from peers 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495), 2fe3135122184018b5d42433a118ce05 (127.19.126.3:35853)
I20240503 02:05:25.500353 23326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1" candidate_uuid: "033176be2d7d430e9348dd175dcb242c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
I20240503 02:05:25.500564 23475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926c9af7e67b48f88a9826a5430ddcb1" candidate_uuid: "033176be2d7d430e9348dd175dcb242c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "2fe3135122184018b5d42433a118ce05"
I20240503 02:05:25.500969 23326 raft_consensus.cc:3032] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:05:25.501191 23475 raft_consensus.cc:3037] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:05:25.501343 23326 raft_consensus.cc:729] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0196690be6f0425c8bdaa78a5aab10fb, State: Running, Role: LEADER
I20240503 02:05:25.502125 23326 consensus_queue.cc:260] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:25.503304 23326 raft_consensus.cc:3037] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:05:25.505980 23475 raft_consensus.cc:2454] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 033176be2d7d430e9348dd175dcb242c in term 3.
I20240503 02:05:25.507177 23365 leader_election.cc:302] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 033176be2d7d430e9348dd175dcb242c, 2fe3135122184018b5d42433a118ce05; no voters:
I20240503 02:05:25.507834 23647 raft_consensus.cc:2781] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:05:25.508227 23326 raft_consensus.cc:2454] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 033176be2d7d430e9348dd175dcb242c in term 3.
I20240503 02:05:25.508356 23647 raft_consensus.cc:686] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 3 LEADER]: Becoming Leader. State: Replica: 033176be2d7d430e9348dd175dcb242c, State: Running, Role: LEADER
I20240503 02:05:25.509267 23647 consensus_queue.cc:237] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:25.515810 23222 catalog_manager.cc:5484] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c reported cstate change: term changed from 2 to 3, leader changed from 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1) to 033176be2d7d430e9348dd175dcb242c (127.19.126.2). New cstate: current_term: 3 leader_uuid: "033176be2d7d430e9348dd175dcb242c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:05:25.748818 23401 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 3 FOLLOWER]: Refusing update from remote peer 2fe3135122184018b5d42433a118ce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 02:05:25.749964 23525 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:25.774775 23590 raft_consensus.cc:3037] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed [term 2 LEARNER]: Advancing to term 3
I20240503 02:05:25.777078 23326 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 3 FOLLOWER]: Refusing update from remote peer 2fe3135122184018b5d42433a118ce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 02:05:25.778378 23525 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:25.781363 23590 raft_consensus.cc:1261] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed [term 3 LEARNER]: Refusing update from remote peer 033176be2d7d430e9348dd175dcb242c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 02:05:25.783016 23530 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:25.789959 23475 raft_consensus.cc:1261] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 3 FOLLOWER]: Refusing update from remote peer 033176be2d7d430e9348dd175dcb242c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 02:05:25.791250 23647 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:25.837050 23326 raft_consensus.cc:1261] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 3 FOLLOWER]: Refusing update from remote peer 033176be2d7d430e9348dd175dcb242c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 02:05:25.838116 23647 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:25.840440 23590 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 2 LEARNER]: Advancing to term 3
I20240503 02:05:25.845798 23590 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 3 LEARNER]: Refusing update from remote peer 2fe3135122184018b5d42433a118ce05: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 02:05:25.846892 23525 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:25.875080 23647 raft_consensus.cc:1050] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c: attempting to promote NON_VOTER a1260b4816b6471cacfda910361d86ed to VOTER
I20240503 02:05:25.876899 23647 consensus_queue.cc:237] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } }
I20240503 02:05:25.882411 23590 raft_consensus.cc:1261] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed [term 3 LEARNER]: Refusing update from remote peer 033176be2d7d430e9348dd175dcb242c: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:05:25.882628 23326 raft_consensus.cc:1261] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 3 FOLLOWER]: Refusing update from remote peer 033176be2d7d430e9348dd175dcb242c: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:05:25.882665 23475 raft_consensus.cc:1261] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 3 FOLLOWER]: Refusing update from remote peer 033176be2d7d430e9348dd175dcb242c: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:05:25.883756 23674 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:25.884404 23647 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:05:25.884961 23657 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:05:25.892632 23326 raft_consensus.cc:2932] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, a1260b4816b6471cacfda910361d86ed (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:25.892980 23590 raft_consensus.cc:2932] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, a1260b4816b6471cacfda910361d86ed (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:25.904593 23224 catalog_manager.cc:5484] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb reported cstate change: config changed from index 3 to 5, a1260b4816b6471cacfda910361d86ed (127.19.126.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "033176be2d7d430e9348dd175dcb242c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:25.979521 23475 raft_consensus.cc:2932] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, a1260b4816b6471cacfda910361d86ed (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:26.030550 23278 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:26.034799 23475 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99a88e2ad895493394437673aead620c"
dest_uuid: "2fe3135122184018b5d42433a118ce05"
mode: GRACEFUL
new_leader_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
from {username='slave'} at 127.0.0.1:39276
I20240503 02:05:26.035398 23475 raft_consensus.cc:595] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 3 LEADER]: Received request to transfer leadership to TS 0196690be6f0425c8bdaa78a5aab10fb
I20240503 02:05:26.036321 23278 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 02:05:26.036693 23278 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:26.139771 23634 raft_consensus.cc:1050] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05: attempting to promote NON_VOTER a1260b4816b6471cacfda910361d86ed to VOTER
W20240503 02:05:26.140205 23634 raft_consensus.cc:1054] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05: Unable to promote non-voter a1260b4816b6471cacfda910361d86ed: Service unavailable: leader transfer in progress
W20240503 02:05:26.205458 23273 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
W20240503 02:05:26.208511 23273 auto_rebalancer.cc:663] Could not move replica: Illegal state: 99a88e2ad895493394437673aead620c: source replica 2fe3135122184018b5d42433a118ce05 does not have REPLACE attribute set
W20240503 02:05:26.208748 23273 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: 99a88e2ad895493394437673aead620c: source replica 2fe3135122184018b5d42433a118ce05 does not have REPLACE attribute set
I20240503 02:05:26.482694 23530 raft_consensus.cc:2932] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, a1260b4816b6471cacfda910361d86ed (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } attrs { replace: true } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:26.496672 23401 consensus_queue.cc:237] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } }
I20240503 02:05:26.501013 23590 raft_consensus.cc:1261] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed [term 3 FOLLOWER]: Refusing update from remote peer 033176be2d7d430e9348dd175dcb242c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 02:05:26.501122 23475 raft_consensus.cc:1261] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 3 FOLLOWER]: Refusing update from remote peer 033176be2d7d430e9348dd175dcb242c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 02:05:26.502326 23676 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:05:26.502893 23647 consensus_queue.cc:1035] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:05:26.510507 23590 raft_consensus.cc:2932] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:26.511181 23475 raft_consensus.cc:2932] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:26.520561 23225 catalog_manager.cc:5484] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed reported cstate change: config changed from index 5 to 6, VOTER 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1) evicted. New cstate: current_term: 3 leader_uuid: "033176be2d7d430e9348dd175dcb242c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:26.522812 23675 raft_consensus.cc:979] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05: : Instructing follower 0196690be6f0425c8bdaa78a5aab10fb to start an election
I20240503 02:05:26.523223 23525 raft_consensus.cc:1067] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 3 LEADER]: Signalling peer 0196690be6f0425c8bdaa78a5aab10fb to start an election
I20240503 02:05:26.524634 23326 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99a88e2ad895493394437673aead620c"
dest_uuid: "0196690be6f0425c8bdaa78a5aab10fb"
from {username='slave'} at 127.0.0.1:41908
I20240503 02:05:26.525080 23326 raft_consensus.cc:482] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:05:26.525367 23326 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:05:26.529596 23326 raft_consensus.cc:504] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:26.531021 23326 leader_election.cc:288] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [CANDIDATE]: Term 4 election: Requested vote from peers 2fe3135122184018b5d42433a118ce05 (127.19.126.3:35853), 033176be2d7d430e9348dd175dcb242c (127.19.126.2:33021)
I20240503 02:05:26.531805 23475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a88e2ad895493394437673aead620c" candidate_uuid: "0196690be6f0425c8bdaa78a5aab10fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2fe3135122184018b5d42433a118ce05"
I20240503 02:05:26.531922 23401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a88e2ad895493394437673aead620c" candidate_uuid: "0196690be6f0425c8bdaa78a5aab10fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "033176be2d7d430e9348dd175dcb242c"
I20240503 02:05:26.532300 23475 raft_consensus.cc:3032] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:05:26.532505 23401 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:05:26.532555 23475 raft_consensus.cc:729] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2fe3135122184018b5d42433a118ce05, State: Running, Role: LEADER
I20240503 02:05:26.533445 23475 consensus_queue.cc:260] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:26.534543 23475 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:05:26.536792 23401 raft_consensus.cc:2454] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0196690be6f0425c8bdaa78a5aab10fb in term 4.
I20240503 02:05:26.537765 23293 leader_election.cc:302] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0196690be6f0425c8bdaa78a5aab10fb, 033176be2d7d430e9348dd175dcb242c; no voters:
I20240503 02:05:26.539093 23679 raft_consensus.cc:2781] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:05:26.539244 23475 raft_consensus.cc:2454] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0196690be6f0425c8bdaa78a5aab10fb in term 4.
I20240503 02:05:26.540150 23679 raft_consensus.cc:686] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 4 LEADER]: Becoming Leader. State: Replica: 0196690be6f0425c8bdaa78a5aab10fb, State: Running, Role: LEADER
I20240503 02:05:26.540948 23679 consensus_queue.cc:237] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } }
I20240503 02:05:26.546797 23224 catalog_manager.cc:5484] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb reported cstate change: term changed from 3 to 4, leader changed from 2fe3135122184018b5d42433a118ce05 (127.19.126.3) to 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1). New cstate: current_term: 4 leader_uuid: "0196690be6f0425c8bdaa78a5aab10fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:05:26.744894 23315 tablet_service.cc:1514] Processing DeleteTablet for tablet 926c9af7e67b48f88a9826a5430ddcb1 with delete_type TABLET_DATA_TOMBSTONED (TS 0196690be6f0425c8bdaa78a5aab10fb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41902
I20240503 02:05:26.746923 23682 tablet_replica.cc:330] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb: stopping tablet replica
I20240503 02:05:26.747607 23682 raft_consensus.cc:2229] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:26.748175 23682 raft_consensus.cc:2258] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:26.750720 23682 ts_tablet_manager.cc:1875] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:05:26.760798 23682 ts_tablet_manager.cc:1888] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20240503 02:05:26.761109 23682 log.cc:1196] T 926c9af7e67b48f88a9826a5430ddcb1 P 0196690be6f0425c8bdaa78a5aab10fb: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-0-root/wals/926c9af7e67b48f88a9826a5430ddcb1
I20240503 02:05:26.762324 23211 catalog_manager.cc:4830] TS 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1:34495): tablet 926c9af7e67b48f88a9826a5430ddcb1 (table test-workload [id=10a0b799cdee487199f673f51f071764]) successfully deleted
I20240503 02:05:26.848038 23590 raft_consensus.cc:3037] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 3 LEARNER]: Advancing to term 4
I20240503 02:05:26.853041 23590 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 4 LEARNER]: Refusing update from remote peer 0196690be6f0425c8bdaa78a5aab10fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 02:05:26.854234 23679 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:05:26.876950 23679 raft_consensus.cc:1050] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb: attempting to promote NON_VOTER a1260b4816b6471cacfda910361d86ed to VOTER
W20240503 02:05:26.877368 23679 raft_consensus.cc:1054] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb: Unable to promote non-voter a1260b4816b6471cacfda910361d86ed: Illegal state: Leader has not yet committed an operation in its own term
I20240503 02:05:26.879763 23401 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 4 FOLLOWER]: Refusing update from remote peer 0196690be6f0425c8bdaa78a5aab10fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 02:05:26.880878 23679 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:05:26.881490 23475 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 4 FOLLOWER]: Refusing update from remote peer 0196690be6f0425c8bdaa78a5aab10fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 02:05:26.882611 23679 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:05:26.968587 23676 raft_consensus.cc:2932] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 0196690be6f0425c8bdaa78a5aab10fb (127.19.126.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } } peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:26.975069 23212 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 926c9af7e67b48f88a9826a5430ddcb1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 02:05:27.217725 23273 auto_rebalancer.cc:254] failed to send replica move request: Invalid argument: must modify a field when calling MODIFY_PEER
I20240503 02:05:27.286355 23680 raft_consensus.cc:1050] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb: attempting to promote NON_VOTER a1260b4816b6471cacfda910361d86ed to VOTER
I20240503 02:05:27.288286 23680 consensus_queue.cc:237] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } }
I20240503 02:05:27.293190 23401 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 4 FOLLOWER]: Refusing update from remote peer 0196690be6f0425c8bdaa78a5aab10fb: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240503 02:05:27.293817 23590 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 4 LEARNER]: Refusing update from remote peer 0196690be6f0425c8bdaa78a5aab10fb: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240503 02:05:27.295027 23688 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 02:05:27.296272 23690 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:05:27.298650 23475 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 4 FOLLOWER]: Refusing update from remote peer 0196690be6f0425c8bdaa78a5aab10fb: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240503 02:05:27.299983 23680 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 02:05:27.305227 23590 raft_consensus.cc:2932] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 2 to 6, a1260b4816b6471cacfda910361d86ed (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:27.309211 23475 raft_consensus.cc:2932] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 2 to 6, a1260b4816b6471cacfda910361d86ed (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:27.317809 23225 catalog_manager.cc:5484] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed reported cstate change: config changed from index 2 to 6, a1260b4816b6471cacfda910361d86ed (127.19.126.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "0196690be6f0425c8bdaa78a5aab10fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:27.539634 23680 raft_consensus.cc:1013] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 4 LEADER]: attempt to promote peer a1260b4816b6471cacfda910361d86ed: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 02:05:27.626884 23679 raft_consensus.cc:2932] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 2 to 6, a1260b4816b6471cacfda910361d86ed (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:27.634800 23401 raft_consensus.cc:2932] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 2 to 6, a1260b4816b6471cacfda910361d86ed (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "033176be2d7d430e9348dd175dcb242c" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:27.641479 23326 consensus_queue.cc:237] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } }
I20240503 02:05:27.646977 23590 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 4 FOLLOWER]: Refusing update from remote peer 0196690be6f0425c8bdaa78a5aab10fb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240503 02:05:27.648257 23680 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:05:27.651870 23475 raft_consensus.cc:1261] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 4 FOLLOWER]: Refusing update from remote peer 0196690be6f0425c8bdaa78a5aab10fb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240503 02:05:27.653841 23680 consensus_queue.cc:1035] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:05:27.656926 23590 raft_consensus.cc:2932] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 033176be2d7d430e9348dd175dcb242c (127.19.126.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:27.661115 23475 raft_consensus.cc:2932] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 033176be2d7d430e9348dd175dcb242c (127.19.126.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:27.667662 23222 catalog_manager.cc:5484] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed reported cstate change: config changed from index 6 to 7, VOTER 033176be2d7d430e9348dd175dcb242c (127.19.126.2) evicted. New cstate: current_term: 4 leader_uuid: "0196690be6f0425c8bdaa78a5aab10fb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:27.681421 23390 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a88e2ad895493394437673aead620c with delete_type TABLET_DATA_TOMBSTONED (TS 033176be2d7d430e9348dd175dcb242c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36162
I20240503 02:05:27.684017 23693 tablet_replica.cc:330] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c: stopping tablet replica
I20240503 02:05:27.684733 23693 raft_consensus.cc:2229] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:27.685494 23693 raft_consensus.cc:2258] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:27.688120 23693 ts_tablet_manager.cc:1875] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:05:27.699098 23693 ts_tablet_manager.cc:1888] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.6
I20240503 02:05:27.699383 23693 log.cc:1196] T 99a88e2ad895493394437673aead620c P 033176be2d7d430e9348dd175dcb242c: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714701820461004-19960-0/minicluster-data/ts-1-root/wals/99a88e2ad895493394437673aead620c
I20240503 02:05:27.700551 23212 catalog_manager.cc:4830] TS 033176be2d7d430e9348dd175dcb242c (127.19.126.2:33021): tablet 99a88e2ad895493394437673aead620c (table test-workload [id=10a0b799cdee487199f673f51f071764]) successfully deleted
I20240503 02:05:28.037598 23278 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:28.040834 23278 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:28.041376 23278 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:28.336812 23679 raft_consensus.cc:2932] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 4 LEADER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 033176be2d7d430e9348dd175dcb242c (127.19.126.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2fe3135122184018b5d42433a118ce05" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35853 } } peers { permanent_uuid: "0196690be6f0425c8bdaa78a5aab10fb" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 34495 } } peers { permanent_uuid: "a1260b4816b6471cacfda910361d86ed" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 40255 } attrs { promote: false } } }
I20240503 02:05:28.342877 23211 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99a88e2ad895493394437673aead620c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:05:30.042304 23278 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:30.044668 23278 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:30.044958 23278 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:32.045931 23278 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:32.048511 23278 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:32.048851 23278 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:33.013093 19960 tablet_server.cc:178] TabletServer@127.19.126.1:0 shutting down...
I20240503 02:05:33.037977 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:33.038807 19960 tablet_replica.cc:330] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb: stopping tablet replica
I20240503 02:05:33.039475 19960 raft_consensus.cc:2229] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:33.040182 19960 raft_consensus.cc:2258] T 6eb977b47e884f92a52721558b910903 P 0196690be6f0425c8bdaa78a5aab10fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:33.042536 19960 tablet_replica.cc:330] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb: stopping tablet replica
I20240503 02:05:33.042995 19960 raft_consensus.cc:2229] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 4 LEADER]: Raft consensus shutting down.
I20240503 02:05:33.043829 19960 raft_consensus.cc:2258] T 99a88e2ad895493394437673aead620c P 0196690be6f0425c8bdaa78a5aab10fb [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:33.067831 19960 tablet_server.cc:194] TabletServer@127.19.126.1:0 shutdown complete.
I20240503 02:05:33.083534 19960 tablet_server.cc:178] TabletServer@127.19.126.2:0 shutting down...
I20240503 02:05:33.107347 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:33.108245 19960 tablet_replica.cc:330] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c: stopping tablet replica
I20240503 02:05:33.108837 19960 raft_consensus.cc:2229] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:33.109323 19960 raft_consensus.cc:2258] T 6eb977b47e884f92a52721558b910903 P 033176be2d7d430e9348dd175dcb242c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:33.111366 19960 tablet_replica.cc:330] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c: stopping tablet replica
I20240503 02:05:33.111829 19960 raft_consensus.cc:2229] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 3 LEADER]: Raft consensus shutting down.
I20240503 02:05:33.112632 19960 raft_consensus.cc:2258] T 926c9af7e67b48f88a9826a5430ddcb1 P 033176be2d7d430e9348dd175dcb242c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:33.134815 19960 tablet_server.cc:194] TabletServer@127.19.126.2:0 shutdown complete.
I20240503 02:05:33.148159 19960 tablet_server.cc:178] TabletServer@127.19.126.3:0 shutting down...
I20240503 02:05:33.168407 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:33.169054 19960 tablet_replica.cc:330] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05: stopping tablet replica
I20240503 02:05:33.169694 19960 raft_consensus.cc:2229] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:33.170578 19960 raft_consensus.cc:2258] T 926c9af7e67b48f88a9826a5430ddcb1 P 2fe3135122184018b5d42433a118ce05 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:33.172596 19960 tablet_replica.cc:330] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05: stopping tablet replica
I20240503 02:05:33.173061 19960 raft_consensus.cc:2229] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:05:33.173892 19960 raft_consensus.cc:2258] T 6eb977b47e884f92a52721558b910903 P 2fe3135122184018b5d42433a118ce05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:33.175885 19960 tablet_replica.cc:330] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05: stopping tablet replica
I20240503 02:05:33.176375 19960 raft_consensus.cc:2229] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:33.177074 19960 raft_consensus.cc:2258] T 99a88e2ad895493394437673aead620c P 2fe3135122184018b5d42433a118ce05 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:33.199199 19960 tablet_server.cc:194] TabletServer@127.19.126.3:0 shutdown complete.
I20240503 02:05:33.213382 19960 tablet_server.cc:178] TabletServer@127.19.126.4:0 shutting down...
I20240503 02:05:33.237680 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:33.238292 19960 tablet_replica.cc:330] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed: stopping tablet replica
I20240503 02:05:33.238812 19960 raft_consensus.cc:2229] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:33.239488 19960 raft_consensus.cc:2258] T 926c9af7e67b48f88a9826a5430ddcb1 P a1260b4816b6471cacfda910361d86ed [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:33.241514 19960 tablet_replica.cc:330] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed: stopping tablet replica
I20240503 02:05:33.241935 19960 raft_consensus.cc:2229] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:33.242499 19960 raft_consensus.cc:2258] T 99a88e2ad895493394437673aead620c P a1260b4816b6471cacfda910361d86ed [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:33.262266 19960 tablet_server.cc:194] TabletServer@127.19.126.4:0 shutdown complete.
I20240503 02:05:33.272572 19960 master.cc:537] Master@127.19.126.62:46395 shutting down...
I20240503 02:05:33.289848 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:05:33.290355 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:33.290660 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P bb660afc05844311b7394f1496ef1a22: stopping tablet replica
I20240503 02:05:33.307799 19960 master.cc:552] Master@127.19.126.62:46395 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (11467 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20240503 02:05:33.344480 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:40421
I20240503 02:05:33.345506 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:33.351598 23696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:33.352232 23697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:33.352993 23699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:33.353750 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:33.354529 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:33.354702 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:33.354815 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701933354804 us; error 0 us; skew 500 ppm
I20240503 02:05:33.355237 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:33.362972 19960 webserver.cc:460] Webserver started at http://127.19.126.62:46285/ using document root <none> and password file <none>
I20240503 02:05:33.363380 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:33.363531 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:33.363745 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:33.364854 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "b093f679c7534d44a735d9e1541be3c8"
format_stamp: "Formatted at 2024-05-03 02:05:33 on dist-test-slave-8rwr"
I20240503 02:05:33.369098 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20240503 02:05:33.372179 23704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:33.372874 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:05:33.373139 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "b093f679c7534d44a735d9e1541be3c8"
format_stamp: "Formatted at 2024-05-03 02:05:33 on dist-test-slave-8rwr"
I20240503 02:05:33.373426 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:33.398458 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:33.399645 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:33.435300 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:40421
I20240503 02:05:33.435386 23755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:40421 every 8 connection(s)
I20240503 02:05:33.438891 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:33.448623 23756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8: Bootstrap starting.
I20240503 02:05:33.452724 23756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:33.456653 23756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8: No bootstrap required, opened a new log
I20240503 02:05:33.458515 23756 raft_consensus.cc:348] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b093f679c7534d44a735d9e1541be3c8" member_type: VOTER }
I20240503 02:05:33.458957 23756 raft_consensus.cc:374] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:33.459168 23756 raft_consensus.cc:729] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b093f679c7534d44a735d9e1541be3c8, State: Initialized, Role: FOLLOWER
I20240503 02:05:33.459733 23756 consensus_queue.cc:260] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b093f679c7534d44a735d9e1541be3c8" member_type: VOTER }
I20240503 02:05:33.460170 23756 raft_consensus.cc:388] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:05:33.460377 23756 raft_consensus.cc:482] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:05:33.460621 23756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:33.464520 23756 raft_consensus.cc:504] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b093f679c7534d44a735d9e1541be3c8" member_type: VOTER }
I20240503 02:05:33.464967 23756 leader_election.cc:302] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b093f679c7534d44a735d9e1541be3c8; no voters:
I20240503 02:05:33.466009 23756 leader_election.cc:288] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:05:33.466413 23759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:33.467640 23759 raft_consensus.cc:686] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [term 1 LEADER]: Becoming Leader. State: Replica: b093f679c7534d44a735d9e1541be3c8, State: Running, Role: LEADER
I20240503 02:05:33.468187 23759 consensus_queue.cc:237] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b093f679c7534d44a735d9e1541be3c8" member_type: VOTER }
I20240503 02:05:33.468762 23756 sys_catalog.cc:564] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:05:33.471105 23760 sys_catalog.cc:455] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b093f679c7534d44a735d9e1541be3c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b093f679c7534d44a735d9e1541be3c8" member_type: VOTER } }
I20240503 02:05:33.471246 23761 sys_catalog.cc:455] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b093f679c7534d44a735d9e1541be3c8. Latest consensus state: current_term: 1 leader_uuid: "b093f679c7534d44a735d9e1541be3c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b093f679c7534d44a735d9e1541be3c8" member_type: VOTER } }
I20240503 02:05:33.471901 23760 sys_catalog.cc:458] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [sys.catalog]: This master's current role is: LEADER
I20240503 02:05:33.471925 23761 sys_catalog.cc:458] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [sys.catalog]: This master's current role is: LEADER
I20240503 02:05:33.474236 23765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:05:33.480897 23765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:05:33.482692 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:05:33.489327 23765 catalog_manager.cc:1347] Generated new cluster ID: 85cebf0411d14108ac98f225c834292a
I20240503 02:05:33.489562 23765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:05:33.513242 23765 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:05:33.514469 23765 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:05:33.526261 23765 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8: Generated new TSK 0
I20240503 02:05:33.526794 23765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:05:33.548539 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:33.554745 23777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:33.555828 23778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:33.556872 23780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:33.557706 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:33.558394 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:33.558579 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:33.558724 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701933558706 us; error 0 us; skew 500 ppm
I20240503 02:05:33.559197 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:33.561349 19960 webserver.cc:460] Webserver started at http://127.19.126.1:37939/ using document root <none> and password file <none>
I20240503 02:05:33.561793 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:33.561983 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:33.562235 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:33.563328 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-0-root/instance:
uuid: "5e600e5f270c4edb98b2f160c5c5d0b9"
format_stamp: "Formatted at 2024-05-03 02:05:33 on dist-test-slave-8rwr"
I20240503 02:05:33.567250 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:05:33.570379 23785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:33.571039 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:05:33.571291 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "5e600e5f270c4edb98b2f160c5c5d0b9"
format_stamp: "Formatted at 2024-05-03 02:05:33 on dist-test-slave-8rwr"
I20240503 02:05:33.571545 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:33.586621 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:33.587644 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:33.588961 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:33.591071 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:33.591249 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:33.591480 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:33.591637 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:33.628634 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:33641
I20240503 02:05:33.628732 23847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:33641 every 8 connection(s)
I20240503 02:05:33.632997 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:33.641669 23852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:33.642154 23853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:33.644158 23855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:33.645004 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:33.646021 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:05:33.646106 23848 heartbeater.cc:347] Connected to a master server at 127.19.126.62:40421
W20240503 02:05:33.646234 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:33.646495 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701933646476 us; error 0 us; skew 500 ppm
I20240503 02:05:33.646543 23848 heartbeater.cc:464] Registering TS with master...
I20240503 02:05:33.647051 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:33.647271 23848 heartbeater.cc:511] Master 127.19.126.62:40421 requested a full tablet report, sending...
I20240503 02:05:33.649358 23721 ts_manager.cc:194] Registered new tserver with Master: 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1:33641)
I20240503 02:05:33.649832 19960 webserver.cc:460] Webserver started at http://127.19.126.2:40145/ using document root <none> and password file <none>
I20240503 02:05:33.650424 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:33.650663 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:33.650986 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:33.651734 23721 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38712
I20240503 02:05:33.652532 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-1-root/instance:
uuid: "5361ef00fb4c4c8fbdd115783ec378e8"
format_stamp: "Formatted at 2024-05-03 02:05:33 on dist-test-slave-8rwr"
I20240503 02:05:33.657213 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240503 02:05:33.660205 23860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:33.660899 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:05:33.661150 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "5361ef00fb4c4c8fbdd115783ec378e8"
format_stamp: "Formatted at 2024-05-03 02:05:33 on dist-test-slave-8rwr"
I20240503 02:05:33.661439 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:33.691002 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:33.692116 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:33.693470 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:33.695545 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:33.695741 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:33.695957 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:33.696107 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:33.732291 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:36633
I20240503 02:05:33.732388 23922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:36633 every 8 connection(s)
I20240503 02:05:33.736672 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:33.743893 23926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:33.745590 23927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:33.748983 23923 heartbeater.cc:347] Connected to a master server at 127.19.126.62:40421
I20240503 02:05:33.749382 23923 heartbeater.cc:464] Registering TS with master...
W20240503 02:05:33.749462 23930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:33.750308 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:33.750330 23923 heartbeater.cc:511] Master 127.19.126.62:40421 requested a full tablet report, sending...
I20240503 02:05:33.751138 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:33.751350 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:33.751509 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701933751493 us; error 0 us; skew 500 ppm
I20240503 02:05:33.752040 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:33.752355 23721 ts_manager.cc:194] Registered new tserver with Master: 5361ef00fb4c4c8fbdd115783ec378e8 (127.19.126.2:36633)
I20240503 02:05:33.754163 23721 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38720
I20240503 02:05:33.754760 19960 webserver.cc:460] Webserver started at http://127.19.126.3:39547/ using document root <none> and password file <none>
I20240503 02:05:33.755367 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:33.755554 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:33.755824 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:33.756970 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-2-root/instance:
uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c"
format_stamp: "Formatted at 2024-05-03 02:05:33 on dist-test-slave-8rwr"
I20240503 02:05:33.761114 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:05:33.764216 23934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:33.764927 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:05:33.765184 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c"
format_stamp: "Formatted at 2024-05-03 02:05:33 on dist-test-slave-8rwr"
I20240503 02:05:33.765476 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:33.776139 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:33.777206 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:33.778589 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:33.780607 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:33.780793 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:33.781006 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:33.781155 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:33.817211 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:44603
I20240503 02:05:33.817341 23996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:44603 every 8 connection(s)
I20240503 02:05:33.829488 23997 heartbeater.cc:347] Connected to a master server at 127.19.126.62:40421
I20240503 02:05:33.829890 23997 heartbeater.cc:464] Registering TS with master...
I20240503 02:05:33.830638 23997 heartbeater.cc:511] Master 127.19.126.62:40421 requested a full tablet report, sending...
I20240503 02:05:33.832325 23721 ts_manager.cc:194] Registered new tserver with Master: b63d7be3a0e94ff6ad3ec8f8b2704a4c (127.19.126.3:44603)
I20240503 02:05:33.833393 19960 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013193063s
I20240503 02:05:33.833581 23721 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38722
I20240503 02:05:34.654780 23848 heartbeater.cc:503] Master 127.19.126.62:40421 was elected leader, sending a full tablet report...
I20240503 02:05:34.756611 23923 heartbeater.cc:503] Master 127.19.126.62:40421 was elected leader, sending a full tablet report...
I20240503 02:05:34.835856 23997 heartbeater.cc:503] Master 127.19.126.62:40421 was elected leader, sending a full tablet report...
I20240503 02:05:34.864866 19960 test_util.cc:268] Using random seed: 666446283
I20240503 02:05:34.886241 23721 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38734:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:05:34.888079 23721 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:05:34.926093 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 310df7d1142b4cacb9a72fb8b410be0d (DEFAULT_TABLE table=test-workload [id=932e85492c2f4453b861a8f0f8bb0435]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:05:34.927244 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310df7d1142b4cacb9a72fb8b410be0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:34.929195 23962 tablet_service.cc:1467] Processing CreateTablet for tablet 310df7d1142b4cacb9a72fb8b410be0d (DEFAULT_TABLE table=test-workload [id=932e85492c2f4453b861a8f0f8bb0435]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:05:34.930349 23962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310df7d1142b4cacb9a72fb8b410be0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:34.934758 23813 tablet_service.cc:1467] Processing CreateTablet for tablet 310df7d1142b4cacb9a72fb8b410be0d (DEFAULT_TABLE table=test-workload [id=932e85492c2f4453b861a8f0f8bb0435]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:05:34.935954 23813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310df7d1142b4cacb9a72fb8b410be0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:34.950369 24018 tablet_bootstrap.cc:492] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c: Bootstrap starting.
I20240503 02:05:34.952850 24017 tablet_bootstrap.cc:492] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8: Bootstrap starting.
I20240503 02:05:34.957386 24019 tablet_bootstrap.cc:492] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9: Bootstrap starting.
I20240503 02:05:34.957777 24018 tablet_bootstrap.cc:654] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:34.959574 24017 tablet_bootstrap.cc:654] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:34.962218 24019 tablet_bootstrap.cc:654] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:34.963445 24018 tablet_bootstrap.cc:492] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c: No bootstrap required, opened a new log
I20240503 02:05:34.964002 24018 ts_tablet_manager.cc:1367] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240503 02:05:34.966536 24017 tablet_bootstrap.cc:492] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8: No bootstrap required, opened a new log
I20240503 02:05:34.966990 24017 ts_tablet_manager.cc:1367] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20240503 02:05:34.966688 24018 raft_consensus.cc:348] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:34.967517 24019 tablet_bootstrap.cc:492] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9: No bootstrap required, opened a new log
I20240503 02:05:34.967419 24018 raft_consensus.cc:374] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:34.967854 24018 raft_consensus.cc:729] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63d7be3a0e94ff6ad3ec8f8b2704a4c, State: Initialized, Role: FOLLOWER
I20240503 02:05:34.967938 24019 ts_tablet_manager.cc:1367] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 02:05:34.968680 24018 consensus_queue.cc:260] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:34.969156 24017 raft_consensus.cc:348] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:34.969833 24017 raft_consensus.cc:374] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:34.970086 24017 raft_consensus.cc:729] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5361ef00fb4c4c8fbdd115783ec378e8, State: Initialized, Role: FOLLOWER
I20240503 02:05:34.970849 24017 consensus_queue.cc:260] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:34.971756 24018 ts_tablet_manager.cc:1398] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 02:05:34.970782 24019 raft_consensus.cc:348] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:34.972527 24019 raft_consensus.cc:374] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:34.972836 24019 raft_consensus.cc:729] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e600e5f270c4edb98b2f160c5c5d0b9, State: Initialized, Role: FOLLOWER
I20240503 02:05:34.973441 24019 consensus_queue.cc:260] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:34.977768 24017 ts_tablet_manager.cc:1398] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s
I20240503 02:05:34.979797 24019 ts_tablet_manager.cc:1398] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20240503 02:05:34.987134 23924 tablet.cc:2369] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:05:35.050593 24023 raft_consensus.cc:482] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:35.051306 24023 raft_consensus.cc:504] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:35.055261 24023 leader_election.cc:288] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5361ef00fb4c4c8fbdd115783ec378e8 (127.19.126.2:36633), 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1:33641)
W20240503 02:05:35.585328 23936 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.000s sys 0.011s
W20240503 02:05:35.585754 23936 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.530s user 0.000s sys 0.011s
I20240503 02:05:35.613000 24028 raft_consensus.cc:482] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:35.613844 24028 raft_consensus.cc:504] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:35.616204 24028 leader_election.cc:288] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5361ef00fb4c4c8fbdd115783ec378e8 (127.19.126.2:36633), b63d7be3a0e94ff6ad3ec8f8b2704a4c (127.19.126.3:44603)
W20240503 02:05:35.618767 23769 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:05:35.630860 23774 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:35.631688 23774 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:35.634178 23972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310df7d1142b4cacb9a72fb8b410be0d" candidate_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" is_pre_election: true
I20240503 02:05:35.634831 23972 raft_consensus.cc:2454] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e600e5f270c4edb98b2f160c5c5d0b9 in term 0.
I20240503 02:05:35.635859 23787 leader_election.cc:302] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e600e5f270c4edb98b2f160c5c5d0b9, b63d7be3a0e94ff6ad3ec8f8b2704a4c; no voters:
I20240503 02:05:35.636654 24028 raft_consensus.cc:2781] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:05:35.637034 24028 raft_consensus.cc:482] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:05:35.637394 24028 raft_consensus.cc:3037] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:35.642249 24031 raft_consensus.cc:482] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:35.642753 24031 raft_consensus.cc:504] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:35.644172 24028 raft_consensus.cc:504] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:35.646606 24028 leader_election.cc:288] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [CANDIDATE]: Term 1 election: Requested vote from peers 5361ef00fb4c4c8fbdd115783ec378e8 (127.19.126.2:36633), b63d7be3a0e94ff6ad3ec8f8b2704a4c (127.19.126.3:44603)
I20240503 02:05:35.646644 24031 leader_election.cc:288] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b63d7be3a0e94ff6ad3ec8f8b2704a4c (127.19.126.3:44603), 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1:33641)
I20240503 02:05:35.665742 23972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310df7d1142b4cacb9a72fb8b410be0d" candidate_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c"
I20240503 02:05:35.666687 23972 raft_consensus.cc:3037] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:35.684288 23897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310df7d1142b4cacb9a72fb8b410be0d" candidate_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361ef00fb4c4c8fbdd115783ec378e8"
I20240503 02:05:35.685212 23897 raft_consensus.cc:3037] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:35.684237 23898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310df7d1142b4cacb9a72fb8b410be0d" candidate_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" is_pre_election: true
I20240503 02:05:35.693184 23897 raft_consensus.cc:2454] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e600e5f270c4edb98b2f160c5c5d0b9 in term 1.
I20240503 02:05:35.694730 23788 leader_election.cc:302] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361ef00fb4c4c8fbdd115783ec378e8, 5e600e5f270c4edb98b2f160c5c5d0b9; no voters:
I20240503 02:05:35.695910 24028 raft_consensus.cc:2781] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:35.702704 23897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310df7d1142b4cacb9a72fb8b410be0d" candidate_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" is_pre_election: true
I20240503 02:05:35.703627 23897 raft_consensus.cc:2379] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b63d7be3a0e94ff6ad3ec8f8b2704a4c in current term 1: Already voted for candidate 5e600e5f270c4edb98b2f160c5c5d0b9 in this term.
I20240503 02:05:35.707839 24028 raft_consensus.cc:686] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 1 LEADER]: Becoming Leader. State: Replica: 5e600e5f270c4edb98b2f160c5c5d0b9, State: Running, Role: LEADER
I20240503 02:05:35.708912 24028 consensus_queue.cc:237] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:35.731081 23823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310df7d1142b4cacb9a72fb8b410be0d" candidate_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" is_pre_election: true
I20240503 02:05:35.732640 23936 leader_election.cc:302] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b63d7be3a0e94ff6ad3ec8f8b2704a4c; no voters: 5361ef00fb4c4c8fbdd115783ec378e8, 5e600e5f270c4edb98b2f160c5c5d0b9
I20240503 02:05:35.733623 23719 catalog_manager.cc:5484] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1). New cstate: current_term: 1 leader_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } health_report { overall_health: HEALTHY } } }
I20240503 02:05:35.735767 23972 raft_consensus.cc:2454] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e600e5f270c4edb98b2f160c5c5d0b9 in term 1.
I20240503 02:05:35.739183 24040 raft_consensus.cc:2726] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:05:35.740415 23972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310df7d1142b4cacb9a72fb8b410be0d" candidate_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" is_pre_election: true
I20240503 02:05:35.740870 23823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310df7d1142b4cacb9a72fb8b410be0d" candidate_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" is_pre_election: true
I20240503 02:05:35.741209 23972 raft_consensus.cc:2379] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5361ef00fb4c4c8fbdd115783ec378e8 in current term 1: Already voted for candidate 5e600e5f270c4edb98b2f160c5c5d0b9 in this term.
I20240503 02:05:35.742543 23862 leader_election.cc:302] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5361ef00fb4c4c8fbdd115783ec378e8; no voters: 5e600e5f270c4edb98b2f160c5c5d0b9, b63d7be3a0e94ff6ad3ec8f8b2704a4c
I20240503 02:05:35.743242 24031 raft_consensus.cc:2726] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:05:35.771059 19960 tablet_server.cc:178] TabletServer@127.19.126.1:0 shutting down...
I20240503 02:05:35.789145 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:35.789812 19960 tablet_replica.cc:330] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9: stopping tablet replica
I20240503 02:05:35.790370 19960 raft_consensus.cc:2229] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:05:35.790931 19960 pending_rounds.cc:62] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9: Trying to abort 1 pending ops.
I20240503 02:05:35.791141 19960 pending_rounds.cc:68] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023419128673771520 op_type: NO_OP noop_request { }
I20240503 02:05:35.791450 19960 raft_consensus.cc:2866] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:05:35.791709 19960 raft_consensus.cc:2258] T 310df7d1142b4cacb9a72fb8b410be0d P 5e600e5f270c4edb98b2f160c5c5d0b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:35.810739 19960 tablet_server.cc:194] TabletServer@127.19.126.1:0 shutdown complete.
W20240503 02:05:36.702399 23919 debug-util.cc:398] Leaking SignalData structure 0x7b0800196b00 after lost signal to thread 19963
W20240503 02:05:36.703281 23919 debug-util.cc:398] Leaking SignalData structure 0x7b0800195420 after lost signal to thread 23755
W20240503 02:05:36.704880 23919 debug-util.cc:398] Leaking SignalData structure 0x7b08000660c0 after lost signal to thread 23922
W20240503 02:05:36.705732 23919 debug-util.cc:398] Leaking SignalData structure 0x7b08001e6820 after lost signal to thread 23996
I20240503 02:05:37.632561 23774 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:37.633870 23774 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:37.634166 23774 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:37.788556 24044 raft_consensus.cc:482] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:37.789021 24044 raft_consensus.cc:504] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:37.790575 24044 leader_election.cc:288] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5361ef00fb4c4c8fbdd115783ec378e8 (127.19.126.2:36633), 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1:33641)
I20240503 02:05:37.791734 23897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310df7d1142b4cacb9a72fb8b410be0d" candidate_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" is_pre_election: true
I20240503 02:05:37.792444 23897 raft_consensus.cc:2454] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63d7be3a0e94ff6ad3ec8f8b2704a4c in term 1.
W20240503 02:05:37.793068 23936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.126.1:33641: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20240503 02:05:37.793413 23937 leader_election.cc:302] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5361ef00fb4c4c8fbdd115783ec378e8, b63d7be3a0e94ff6ad3ec8f8b2704a4c; no voters:
I20240503 02:05:37.794365 24044 raft_consensus.cc:2781] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:05:37.794710 24044 raft_consensus.cc:482] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:05:37.794986 24044 raft_consensus.cc:3037] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:05:37.796658 23936 leader_election.cc:334] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1:33641): Network error: Client connection negotiation failed: client connection to 127.19.126.1:33641: connect: Connection refused (error 111)
I20240503 02:05:37.800173 24044 raft_consensus.cc:504] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:37.801591 24044 leader_election.cc:288] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [CANDIDATE]: Term 2 election: Requested vote from peers 5361ef00fb4c4c8fbdd115783ec378e8 (127.19.126.2:36633), 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1:33641)
I20240503 02:05:37.802474 23897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310df7d1142b4cacb9a72fb8b410be0d" candidate_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5361ef00fb4c4c8fbdd115783ec378e8"
I20240503 02:05:37.803048 23897 raft_consensus.cc:3037] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:05:37.805533 23936 leader_election.cc:334] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1:33641): Network error: Client connection negotiation failed: client connection to 127.19.126.1:33641: connect: Connection refused (error 111)
I20240503 02:05:37.807860 23897 raft_consensus.cc:2454] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63d7be3a0e94ff6ad3ec8f8b2704a4c in term 2.
I20240503 02:05:37.808682 23937 leader_election.cc:302] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5361ef00fb4c4c8fbdd115783ec378e8, b63d7be3a0e94ff6ad3ec8f8b2704a4c; no voters: 5e600e5f270c4edb98b2f160c5c5d0b9
I20240503 02:05:37.809270 24044 raft_consensus.cc:2781] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:05:37.810391 24044 raft_consensus.cc:686] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 2 LEADER]: Becoming Leader. State: Replica: b63d7be3a0e94ff6ad3ec8f8b2704a4c, State: Running, Role: LEADER
I20240503 02:05:37.811461 24044 consensus_queue.cc:237] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } }
I20240503 02:05:37.818488 23721 catalog_manager.cc:5484] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c reported cstate change: term changed from 1 to 2, leader changed from 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1) to b63d7be3a0e94ff6ad3ec8f8b2704a4c (127.19.126.3). New cstate: current_term: 2 leader_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63d7be3a0e94ff6ad3ec8f8b2704a4c" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 44603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e600e5f270c4edb98b2f160c5c5d0b9" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 33641 } health_report { overall_health: UNKNOWN } } }
W20240503 02:05:38.255911 23936 consensus_peers.cc:485] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c -> Peer 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1:33641): Couldn't send request to peer 5e600e5f270c4edb98b2f160c5c5d0b9. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:33641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:38.304675 23897 raft_consensus.cc:1261] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 2 FOLLOWER]: Refusing update from remote peer b63d7be3a0e94ff6ad3ec8f8b2704a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 02:05:38.306051 24044 consensus_queue.cc:1035] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5361ef00fb4c4c8fbdd115783ec378e8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 36633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:05:39.634929 23774 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:39.636096 23774 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:39.636384 23774 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 02:05:40.645499 23936 consensus_peers.cc:485] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c -> Peer 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1:33641): Couldn't send request to peer 5e600e5f270c4edb98b2f160c5c5d0b9. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:33641: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:05:41.637495 23774 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:41.639075 23774 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:41.639392 23774 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 02:05:43.230733 23936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.126.1:33641: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 02:05:43.233573 23936 consensus_peers.cc:485] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c -> Peer 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1:33641): Couldn't send request to peer 5e600e5f270c4edb98b2f160c5c5d0b9. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:33641: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:05:43.640249 23774 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:43.641577 23774 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:43.641861 23774 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:45.642690 23774 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:45.643888 23774 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:45.644158 23774 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 02:05:45.860642 23936 consensus_peers.cc:485] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c -> Peer 5e600e5f270c4edb98b2f160c5c5d0b9 (127.19.126.1:33641): Couldn't send request to peer 5e600e5f270c4edb98b2f160c5c5d0b9. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:33641: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:05:47.644995 23774 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:05:47.646200 23774 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:05:47.646450 23774 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:05:47.879806 19960 tablet_server.cc:178] TabletServer@127.19.126.2:0 shutting down...
I20240503 02:05:47.902539 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:47.903189 19960 tablet_replica.cc:330] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8: stopping tablet replica
I20240503 02:05:47.903739 19960 raft_consensus.cc:2229] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:47.904335 19960 raft_consensus.cc:2258] T 310df7d1142b4cacb9a72fb8b410be0d P 5361ef00fb4c4c8fbdd115783ec378e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:47.942836 19960 tablet_server.cc:194] TabletServer@127.19.126.2:0 shutdown complete.
I20240503 02:05:47.955493 19960 tablet_server.cc:178] TabletServer@127.19.126.3:0 shutting down...
I20240503 02:05:47.976289 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:47.976999 19960 tablet_replica.cc:330] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c: stopping tablet replica
I20240503 02:05:47.977612 19960 raft_consensus.cc:2229] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:05:47.978478 19960 raft_consensus.cc:2258] T 310df7d1142b4cacb9a72fb8b410be0d P b63d7be3a0e94ff6ad3ec8f8b2704a4c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:47.997443 19960 tablet_server.cc:194] TabletServer@127.19.126.3:0 shutdown complete.
I20240503 02:05:48.008522 19960 master.cc:537] Master@127.19.126.62:40421 shutting down...
I20240503 02:05:48.027943 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:05:48.028517 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:48.028833 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P b093f679c7534d44a735d9e1541be3c8: stopping tablet replica
I20240503 02:05:48.046780 19960 master.cc:552] Master@127.19.126.62:40421 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (14735 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20240503 02:05:48.080672 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:40583
I20240503 02:05:48.081758 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:48.087803 24078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:48.088654 24079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:48.089563 24081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:48.090081 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:48.090858 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:48.091039 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:48.091166 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701948091154 us; error 0 us; skew 500 ppm
I20240503 02:05:48.091574 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:48.098654 19960 webserver.cc:460] Webserver started at http://127.19.126.62:40927/ using document root <none> and password file <none>
I20240503 02:05:48.099083 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:48.099227 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:48.099445 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:48.100661 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "843ea11dd71c4fe59021a9cc07ad8be6"
format_stamp: "Formatted at 2024-05-03 02:05:48 on dist-test-slave-8rwr"
I20240503 02:05:48.104836 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240503 02:05:48.107959 24086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:48.108664 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:05:48.108916 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "843ea11dd71c4fe59021a9cc07ad8be6"
format_stamp: "Formatted at 2024-05-03 02:05:48 on dist-test-slave-8rwr"
I20240503 02:05:48.109187 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:48.135478 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:48.136641 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:48.173063 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:40583
I20240503 02:05:48.173172 24137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:40583 every 8 connection(s)
I20240503 02:05:48.176966 24138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:48.188107 24138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6: Bootstrap starting.
I20240503 02:05:48.192044 24138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:48.196228 24138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6: No bootstrap required, opened a new log
I20240503 02:05:48.198184 24138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843ea11dd71c4fe59021a9cc07ad8be6" member_type: VOTER }
I20240503 02:05:48.198617 24138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:48.198781 24138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 843ea11dd71c4fe59021a9cc07ad8be6, State: Initialized, Role: FOLLOWER
I20240503 02:05:48.199226 24138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843ea11dd71c4fe59021a9cc07ad8be6" member_type: VOTER }
I20240503 02:05:48.199620 24138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:05:48.199811 24138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:05:48.200009 24138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:48.204195 24138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843ea11dd71c4fe59021a9cc07ad8be6" member_type: VOTER }
I20240503 02:05:48.204645 24138 leader_election.cc:302] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 843ea11dd71c4fe59021a9cc07ad8be6; no voters:
I20240503 02:05:48.205622 24138 leader_election.cc:288] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:05:48.205981 24141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:48.207769 24141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [term 1 LEADER]: Becoming Leader. State: Replica: 843ea11dd71c4fe59021a9cc07ad8be6, State: Running, Role: LEADER
I20240503 02:05:48.208469 24141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843ea11dd71c4fe59021a9cc07ad8be6" member_type: VOTER }
I20240503 02:05:48.209259 24138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:05:48.215510 24143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 843ea11dd71c4fe59021a9cc07ad8be6. Latest consensus state: current_term: 1 leader_uuid: "843ea11dd71c4fe59021a9cc07ad8be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843ea11dd71c4fe59021a9cc07ad8be6" member_type: VOTER } }
I20240503 02:05:48.216012 24142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "843ea11dd71c4fe59021a9cc07ad8be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843ea11dd71c4fe59021a9cc07ad8be6" member_type: VOTER } }
I20240503 02:05:48.216289 24143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [sys.catalog]: This master's current role is: LEADER
I20240503 02:05:48.216495 24142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [sys.catalog]: This master's current role is: LEADER
I20240503 02:05:48.218322 24150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:05:48.223028 24150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:05:48.226780 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:05:48.231069 24150 catalog_manager.cc:1347] Generated new cluster ID: 3ec47b091ad943f98f5241c435a7102f
I20240503 02:05:48.231261 24150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:05:48.262962 24150 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:05:48.264106 24150 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:05:48.285912 24150 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6: Generated new TSK 0
I20240503 02:05:48.286404 24150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:05:48.292488 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:48.298429 24159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:48.299427 24160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:48.300165 24162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:48.300976 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:48.301796 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:48.302016 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:48.302165 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701948302145 us; error 0 us; skew 500 ppm
I20240503 02:05:48.302690 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:48.304843 19960 webserver.cc:460] Webserver started at http://127.19.126.1:34605/ using document root <none> and password file <none>
I20240503 02:05:48.305267 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:48.305473 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:48.305697 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:48.306689 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-0-root/instance:
uuid: "1f78e25b61ff47d28f6f7119dc404175"
format_stamp: "Formatted at 2024-05-03 02:05:48 on dist-test-slave-8rwr"
I20240503 02:05:48.311049 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240503 02:05:48.314023 24167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:48.314750 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:05:48.314999 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "1f78e25b61ff47d28f6f7119dc404175"
format_stamp: "Formatted at 2024-05-03 02:05:48 on dist-test-slave-8rwr"
I20240503 02:05:48.315254 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:48.324571 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:48.325609 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:48.326910 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:48.328958 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:48.329141 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:48.329380 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:48.329547 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:48.367239 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:45449
I20240503 02:05:48.367324 24229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:45449 every 8 connection(s)
I20240503 02:05:48.372145 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:48.379573 24234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:48.380961 24235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:48.384150 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:48.384614 24230 heartbeater.cc:347] Connected to a master server at 127.19.126.62:40583
I20240503 02:05:48.384920 24230 heartbeater.cc:464] Registering TS with master...
W20240503 02:05:48.385483 24237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:48.385613 24230 heartbeater.cc:511] Master 127.19.126.62:40583 requested a full tablet report, sending...
I20240503 02:05:48.386305 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:48.386502 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:48.386646 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701948386631 us; error 0 us; skew 500 ppm
I20240503 02:05:48.387318 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:48.387580 24103 ts_manager.cc:194] Registered new tserver with Master: 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449)
I20240503 02:05:48.389636 24103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60364
I20240503 02:05:48.389909 19960 webserver.cc:460] Webserver started at http://127.19.126.2:35331/ using document root <none> and password file <none>
I20240503 02:05:48.390384 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:48.390566 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:48.390801 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:48.391933 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-1-root/instance:
uuid: "a844e765195247f1913fd1c68e443511"
format_stamp: "Formatted at 2024-05-03 02:05:48 on dist-test-slave-8rwr"
I20240503 02:05:48.396152 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240503 02:05:48.399386 24242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:48.400185 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:05:48.400446 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "a844e765195247f1913fd1c68e443511"
format_stamp: "Formatted at 2024-05-03 02:05:48 on dist-test-slave-8rwr"
I20240503 02:05:48.400709 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:48.423130 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:48.424223 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:48.425539 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:48.427623 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:48.427809 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:48.428014 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:48.428157 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:48.466603 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:38555
I20240503 02:05:48.466689 24304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:38555 every 8 connection(s)
I20240503 02:05:48.471180 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:48.478312 24308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:48.479267 24309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:48.481534 19960 server_base.cc:1034] running on GCE node
W20240503 02:05:48.483599 24311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:48.484337 24305 heartbeater.cc:347] Connected to a master server at 127.19.126.62:40583
I20240503 02:05:48.484549 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:05:48.484669 24305 heartbeater.cc:464] Registering TS with master...
W20240503 02:05:48.484822 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:48.485028 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701948485009 us; error 0 us; skew 500 ppm
I20240503 02:05:48.485540 24305 heartbeater.cc:511] Master 127.19.126.62:40583 requested a full tablet report, sending...
I20240503 02:05:48.485651 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:48.487787 24103 ts_manager.cc:194] Registered new tserver with Master: a844e765195247f1913fd1c68e443511 (127.19.126.2:38555)
I20240503 02:05:48.488575 19960 webserver.cc:460] Webserver started at http://127.19.126.3:42395/ using document root <none> and password file <none>
I20240503 02:05:48.489204 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:48.489457 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:48.489679 24103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60370
I20240503 02:05:48.489774 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:48.491477 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-2-root/instance:
uuid: "235799307cad4d30bce77aba9a4d5c52"
format_stamp: "Formatted at 2024-05-03 02:05:48 on dist-test-slave-8rwr"
I20240503 02:05:48.496043 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:05:48.499250 24316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:48.499967 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:05:48.500214 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "235799307cad4d30bce77aba9a4d5c52"
format_stamp: "Formatted at 2024-05-03 02:05:48 on dist-test-slave-8rwr"
I20240503 02:05:48.500447 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:48.511377 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:48.512463 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:48.513804 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:48.515955 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:48.516148 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:48.516394 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:48.516531 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:48.553712 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:42913
I20240503 02:05:48.553802 24378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:42913 every 8 connection(s)
I20240503 02:05:48.566926 24379 heartbeater.cc:347] Connected to a master server at 127.19.126.62:40583
I20240503 02:05:48.567240 24379 heartbeater.cc:464] Registering TS with master...
I20240503 02:05:48.567984 24379 heartbeater.cc:511] Master 127.19.126.62:40583 requested a full tablet report, sending...
I20240503 02:05:48.569630 24103 ts_manager.cc:194] Registered new tserver with Master: 235799307cad4d30bce77aba9a4d5c52 (127.19.126.3:42913)
I20240503 02:05:48.570102 19960 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013366511s
I20240503 02:05:48.570883 24103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60382
I20240503 02:05:49.392140 24230 heartbeater.cc:503] Master 127.19.126.62:40583 was elected leader, sending a full tablet report...
I20240503 02:05:49.492533 24305 heartbeater.cc:503] Master 127.19.126.62:40583 was elected leader, sending a full tablet report...
I20240503 02:05:49.573379 24379 heartbeater.cc:503] Master 127.19.126.62:40583 was elected leader, sending a full tablet report...
I20240503 02:05:49.617832 19960 test_util.cc:268] Using random seed: 681199251
I20240503 02:05:49.639837 24103 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60392:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:05:49.642063 24103 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:05:49.691571 24270 tablet_service.cc:1467] Processing CreateTablet for tablet 41db5c769a474264805464c3418e346c (DEFAULT_TABLE table=test-workload [id=f459271f1bb24e7fafc82b47255d0806]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:05:49.693466 24270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41db5c769a474264805464c3418e346c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:49.698705 24269 tablet_service.cc:1467] Processing CreateTablet for tablet e1e04891c2b04ec288f8fe077265c900 (DEFAULT_TABLE table=test-workload [id=f459271f1bb24e7fafc82b47255d0806]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:05:49.699939 24269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e04891c2b04ec288f8fe077265c900. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:49.706215 24268 tablet_service.cc:1467] Processing CreateTablet for tablet 18443df96de94fdc930859b89a06e13e (DEFAULT_TABLE table=test-workload [id=f459271f1bb24e7fafc82b47255d0806]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:05:49.707371 24268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18443df96de94fdc930859b89a06e13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:49.715258 24195 tablet_service.cc:1467] Processing CreateTablet for tablet 41db5c769a474264805464c3418e346c (DEFAULT_TABLE table=test-workload [id=f459271f1bb24e7fafc82b47255d0806]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:05:49.715680 24194 tablet_service.cc:1467] Processing CreateTablet for tablet e1e04891c2b04ec288f8fe077265c900 (DEFAULT_TABLE table=test-workload [id=f459271f1bb24e7fafc82b47255d0806]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:05:49.716259 24193 tablet_service.cc:1467] Processing CreateTablet for tablet 18443df96de94fdc930859b89a06e13e (DEFAULT_TABLE table=test-workload [id=f459271f1bb24e7fafc82b47255d0806]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:05:49.716670 24195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41db5c769a474264805464c3418e346c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:49.717396 24193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18443df96de94fdc930859b89a06e13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:49.729773 24194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e04891c2b04ec288f8fe077265c900. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:49.740435 24344 tablet_service.cc:1467] Processing CreateTablet for tablet 41db5c769a474264805464c3418e346c (DEFAULT_TABLE table=test-workload [id=f459271f1bb24e7fafc82b47255d0806]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:05:49.741837 24344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41db5c769a474264805464c3418e346c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:49.747516 24343 tablet_service.cc:1467] Processing CreateTablet for tablet e1e04891c2b04ec288f8fe077265c900 (DEFAULT_TABLE table=test-workload [id=f459271f1bb24e7fafc82b47255d0806]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:05:49.748399 24195 tablet_service.cc:1467] Processing CreateTablet for tablet b1d852fa67354e9696275e159f12b5ce (DEFAULT_TABLE table=test-workload [id=f459271f1bb24e7fafc82b47255d0806]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:05:49.748795 24343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e04891c2b04ec288f8fe077265c900. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:49.749189 24195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1d852fa67354e9696275e159f12b5ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:05:49.751051 24134 debug-util.cc:398] Leaking SignalData structure 0x7b08000a3e20 after lost signal to thread 19963
I20240503 02:05:49.758412 24268 tablet_service.cc:1467] Processing CreateTablet for tablet b1d852fa67354e9696275e159f12b5ce (DEFAULT_TABLE table=test-workload [id=f459271f1bb24e7fafc82b47255d0806]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:05:49.758872 24342 tablet_service.cc:1467] Processing CreateTablet for tablet 18443df96de94fdc930859b89a06e13e (DEFAULT_TABLE table=test-workload [id=f459271f1bb24e7fafc82b47255d0806]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:05:49.759641 24268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1d852fa67354e9696275e159f12b5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:49.761514 24342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18443df96de94fdc930859b89a06e13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:49.761785 24399 tablet_bootstrap.cc:492] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511: Bootstrap starting.
I20240503 02:05:49.763535 24341 tablet_service.cc:1467] Processing CreateTablet for tablet b1d852fa67354e9696275e159f12b5ce (DEFAULT_TABLE table=test-workload [id=f459271f1bb24e7fafc82b47255d0806]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:05:49.764694 24341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1d852fa67354e9696275e159f12b5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:49.770511 24399 tablet_bootstrap.cc:654] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:49.774660 24400 tablet_bootstrap.cc:492] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175: Bootstrap starting.
I20240503 02:05:49.786413 24401 tablet_bootstrap.cc:492] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52: Bootstrap starting.
I20240503 02:05:49.795373 24401 tablet_bootstrap.cc:654] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:49.797860 24400 tablet_bootstrap.cc:654] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:49.797973 24399 tablet_bootstrap.cc:492] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511: No bootstrap required, opened a new log
I20240503 02:05:49.798544 24399 ts_tablet_manager.cc:1367] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.014s
I20240503 02:05:49.799834 24401 tablet_bootstrap.cc:492] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52: No bootstrap required, opened a new log
I20240503 02:05:49.800201 24401 ts_tablet_manager.cc:1367] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240503 02:05:49.801586 24399 raft_consensus.cc:348] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.802469 24399 raft_consensus.cc:374] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:49.802624 24401 raft_consensus.cc:348] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.802820 24399 raft_consensus.cc:729] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a844e765195247f1913fd1c68e443511, State: Initialized, Role: FOLLOWER
I20240503 02:05:49.803254 24401 raft_consensus.cc:374] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:49.803359 24400 tablet_bootstrap.cc:492] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175: No bootstrap required, opened a new log
I20240503 02:05:49.803536 24401 raft_consensus.cc:729] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235799307cad4d30bce77aba9a4d5c52, State: Initialized, Role: FOLLOWER
I20240503 02:05:49.803839 24400 ts_tablet_manager.cc:1367] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.000s
I20240503 02:05:49.803751 24399 consensus_queue.cc:260] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.804162 24401 consensus_queue.cc:260] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.806712 24400 raft_consensus.cc:348] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.807463 24399 ts_tablet_manager.cc:1398] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20240503 02:05:49.808493 24401 ts_tablet_manager.cc:1398] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240503 02:05:49.809397 24401 tablet_bootstrap.cc:492] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52: Bootstrap starting.
I20240503 02:05:49.809543 24405 raft_consensus.cc:482] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:49.808611 24399 tablet_bootstrap.cc:492] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511: Bootstrap starting.
I20240503 02:05:49.810060 24405 raft_consensus.cc:504] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.816498 24401 tablet_bootstrap.cc:654] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:49.817996 24399 tablet_bootstrap.cc:654] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:49.807507 24400 raft_consensus.cc:374] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:49.822831 24400 raft_consensus.cc:729] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f78e25b61ff47d28f6f7119dc404175, State: Initialized, Role: FOLLOWER
I20240503 02:05:49.827728 24405 leader_election.cc:288] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a844e765195247f1913fd1c68e443511 (127.19.126.2:38555), 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449)
I20240503 02:05:49.837719 24400 consensus_queue.cc:260] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.840709 24401 tablet_bootstrap.cc:492] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52: No bootstrap required, opened a new log
I20240503 02:05:49.840745 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41db5c769a474264805464c3418e346c" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f78e25b61ff47d28f6f7119dc404175" is_pre_election: true
I20240503 02:05:49.841367 24401 ts_tablet_manager.cc:1367] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.003s
W20240503 02:05:49.842849 24320 leader_election.cc:341] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:05:49.844028 24401 raft_consensus.cc:348] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.844758 24401 raft_consensus.cc:374] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:49.845162 24400 ts_tablet_manager.cc:1398] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175: Time spent starting tablet: real 0.041s user 0.000s sys 0.012s
I20240503 02:05:49.845714 24399 tablet_bootstrap.cc:492] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511: No bootstrap required, opened a new log
I20240503 02:05:49.846067 24400 tablet_bootstrap.cc:492] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175: Bootstrap starting.
I20240503 02:05:49.846230 24399 ts_tablet_manager.cc:1367] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511: Time spent bootstrapping tablet: real 0.038s user 0.021s sys 0.008s
I20240503 02:05:49.845036 24401 raft_consensus.cc:729] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235799307cad4d30bce77aba9a4d5c52, State: Initialized, Role: FOLLOWER
I20240503 02:05:49.847506 24401 consensus_queue.cc:260] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.849057 24399 raft_consensus.cc:348] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.849855 24399 raft_consensus.cc:374] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:49.850277 24399 raft_consensus.cc:729] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a844e765195247f1913fd1c68e443511, State: Initialized, Role: FOLLOWER
I20240503 02:05:49.851029 24399 consensus_queue.cc:260] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.851652 24401 ts_tablet_manager.cc:1398] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 02:05:49.851979 24400 tablet_bootstrap.cc:654] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:49.852459 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41db5c769a474264805464c3418e346c" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a844e765195247f1913fd1c68e443511" is_pre_election: true
I20240503 02:05:49.853212 24280 raft_consensus.cc:2454] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 0.
I20240503 02:05:49.853725 24399 ts_tablet_manager.cc:1398] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 02:05:49.854679 24399 tablet_bootstrap.cc:492] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511: Bootstrap starting.
I20240503 02:05:49.855754 24401 tablet_bootstrap.cc:492] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52: Bootstrap starting.
I20240503 02:05:49.856194 24319 leader_election.cc:302] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 235799307cad4d30bce77aba9a4d5c52, a844e765195247f1913fd1c68e443511; no voters: 1f78e25b61ff47d28f6f7119dc404175
I20240503 02:05:49.857415 24405 raft_consensus.cc:2781] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:05:49.857785 24405 raft_consensus.cc:482] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:05:49.858121 24405 raft_consensus.cc:3037] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:49.860615 24399 tablet_bootstrap.cc:654] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:49.861652 24401 tablet_bootstrap.cc:654] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:49.879923 24400 tablet_bootstrap.cc:492] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175: No bootstrap required, opened a new log
I20240503 02:05:49.880396 24400 ts_tablet_manager.cc:1367] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.005s
I20240503 02:05:49.883004 24400 raft_consensus.cc:348] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.883709 24400 raft_consensus.cc:374] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:49.883982 24400 raft_consensus.cc:729] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f78e25b61ff47d28f6f7119dc404175, State: Initialized, Role: FOLLOWER
I20240503 02:05:49.884622 24400 consensus_queue.cc:260] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.887372 24405 raft_consensus.cc:504] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.890756 24399 tablet_bootstrap.cc:492] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511: No bootstrap required, opened a new log
I20240503 02:05:49.894701 24405 leader_election.cc:288] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 election: Requested vote from peers a844e765195247f1913fd1c68e443511 (127.19.126.2:38555), 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449)
I20240503 02:05:49.895699 24400 ts_tablet_manager.cc:1398] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20240503 02:05:49.896205 24399 ts_tablet_manager.cc:1367] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511: Time spent bootstrapping tablet: real 0.042s user 0.017s sys 0.017s
I20240503 02:05:49.896695 24400 tablet_bootstrap.cc:492] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175: Bootstrap starting.
I20240503 02:05:49.896906 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41db5c769a474264805464c3418e346c" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a844e765195247f1913fd1c68e443511"
I20240503 02:05:49.901098 24280 raft_consensus.cc:3037] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:49.902813 24400 tablet_bootstrap.cc:654] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:49.897642 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41db5c769a474264805464c3418e346c" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f78e25b61ff47d28f6f7119dc404175"
I20240503 02:05:49.903358 24205 raft_consensus.cc:3037] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:49.908421 24400 tablet_bootstrap.cc:492] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175: No bootstrap required, opened a new log
I20240503 02:05:49.910485 24205 raft_consensus.cc:2454] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 1.
I20240503 02:05:49.911620 24320 leader_election.cc:302] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f78e25b61ff47d28f6f7119dc404175, 235799307cad4d30bce77aba9a4d5c52; no voters:
I20240503 02:05:49.912451 24405 raft_consensus.cc:2781] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:49.912894 24280 raft_consensus.cc:2454] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 1.
I20240503 02:05:49.899153 24399 raft_consensus.cc:348] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.913970 24399 raft_consensus.cc:374] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:49.914322 24399 raft_consensus.cc:729] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a844e765195247f1913fd1c68e443511, State: Initialized, Role: FOLLOWER
I20240503 02:05:49.915421 24400 ts_tablet_manager.cc:1367] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.000s
I20240503 02:05:49.891510 24401 tablet_bootstrap.cc:492] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52: No bootstrap required, opened a new log
I20240503 02:05:49.916164 24401 ts_tablet_manager.cc:1367] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52: Time spent bootstrapping tablet: real 0.061s user 0.018s sys 0.017s
I20240503 02:05:49.915287 24399 consensus_queue.cc:260] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.918314 24400 raft_consensus.cc:348] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.919780 24400 raft_consensus.cc:374] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:49.920070 24400 raft_consensus.cc:729] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f78e25b61ff47d28f6f7119dc404175, State: Initialized, Role: FOLLOWER
I20240503 02:05:49.919118 24405 raft_consensus.cc:686] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Becoming Leader. State: Replica: 235799307cad4d30bce77aba9a4d5c52, State: Running, Role: LEADER
I20240503 02:05:49.921156 24399 ts_tablet_manager.cc:1398] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511: Time spent starting tablet: real 0.025s user 0.005s sys 0.001s
I20240503 02:05:49.922187 24399 tablet_bootstrap.cc:492] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511: Bootstrap starting.
I20240503 02:05:49.922878 24401 raft_consensus.cc:348] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.923710 24401 raft_consensus.cc:374] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:49.924046 24401 raft_consensus.cc:729] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235799307cad4d30bce77aba9a4d5c52, State: Initialized, Role: FOLLOWER
I20240503 02:05:49.924923 24401 consensus_queue.cc:260] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.927963 24399 tablet_bootstrap.cc:654] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:49.921095 24405 consensus_queue.cc:237] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.933477 24411 raft_consensus.cc:482] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:49.934110 24411 raft_consensus.cc:504] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.935175 24401 ts_tablet_manager.cc:1398] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52: Time spent starting tablet: real 0.019s user 0.009s sys 0.005s
I20240503 02:05:49.936838 24411 leader_election.cc:288] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a844e765195247f1913fd1c68e443511 (127.19.126.2:38555), 235799307cad4d30bce77aba9a4d5c52 (127.19.126.3:42913)
I20240503 02:05:49.939903 24400 consensus_queue.cc:260] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.941001 24401 tablet_bootstrap.cc:492] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52: Bootstrap starting.
I20240503 02:05:49.948239 24399 tablet_bootstrap.cc:492] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511: No bootstrap required, opened a new log
I20240503 02:05:49.948732 24399 ts_tablet_manager.cc:1367] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.008s
I20240503 02:05:49.949486 24400 ts_tablet_manager.cc:1398] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20240503 02:05:49.953261 24401 tablet_bootstrap.cc:654] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:49.957094 24400 tablet_bootstrap.cc:492] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175: Bootstrap starting.
I20240503 02:05:49.954227 24399 raft_consensus.cc:348] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.960860 24399 raft_consensus.cc:374] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:49.961175 24399 raft_consensus.cc:729] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a844e765195247f1913fd1c68e443511, State: Initialized, Role: FOLLOWER
I20240503 02:05:49.962548 24399 consensus_queue.cc:260] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.965081 24400 tablet_bootstrap.cc:654] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175: Neither blocks nor log segments found. Creating new log.
I20240503 02:05:49.972424 24401 tablet_bootstrap.cc:492] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52: No bootstrap required, opened a new log
I20240503 02:05:49.972950 24401 ts_tablet_manager.cc:1367] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.003s
I20240503 02:05:49.975878 24401 raft_consensus.cc:348] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.979271 24401 raft_consensus.cc:374] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:49.982221 24354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18443df96de94fdc930859b89a06e13e" candidate_uuid: "1f78e25b61ff47d28f6f7119dc404175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235799307cad4d30bce77aba9a4d5c52" is_pre_election: true
I20240503 02:05:49.982971 24354 raft_consensus.cc:2454] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f78e25b61ff47d28f6f7119dc404175 in term 0.
I20240503 02:05:49.984426 24169 leader_election.cc:302] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f78e25b61ff47d28f6f7119dc404175, 235799307cad4d30bce77aba9a4d5c52; no voters:
I20240503 02:05:49.984843 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18443df96de94fdc930859b89a06e13e" candidate_uuid: "1f78e25b61ff47d28f6f7119dc404175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a844e765195247f1913fd1c68e443511" is_pre_election: true
I20240503 02:05:49.985332 24411 raft_consensus.cc:2781] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:05:49.985605 24280 raft_consensus.cc:2454] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f78e25b61ff47d28f6f7119dc404175 in term 0.
I20240503 02:05:49.977208 24399 ts_tablet_manager.cc:1398] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20240503 02:05:49.985797 24411 raft_consensus.cc:482] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:05:49.985455 24401 raft_consensus.cc:729] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235799307cad4d30bce77aba9a4d5c52, State: Initialized, Role: FOLLOWER
I20240503 02:05:49.986282 24411 raft_consensus.cc:3037] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:49.987471 24401 consensus_queue.cc:260] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.989938 24401 ts_tablet_manager.cc:1398] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240503 02:05:49.993860 24411 raft_consensus.cc:504] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:49.996177 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18443df96de94fdc930859b89a06e13e" candidate_uuid: "1f78e25b61ff47d28f6f7119dc404175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a844e765195247f1913fd1c68e443511"
I20240503 02:05:49.996387 24354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18443df96de94fdc930859b89a06e13e" candidate_uuid: "1f78e25b61ff47d28f6f7119dc404175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235799307cad4d30bce77aba9a4d5c52"
I20240503 02:05:49.996762 24280 raft_consensus.cc:3037] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:49.997098 24354 raft_consensus.cc:3037] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:49.996364 24103 catalog_manager.cc:5484] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 235799307cad4d30bce77aba9a4d5c52 (127.19.126.3). New cstate: current_term: 1 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240503 02:05:50.000829 24411 leader_election.cc:288] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [CANDIDATE]: Term 1 election: Requested vote from peers a844e765195247f1913fd1c68e443511 (127.19.126.2:38555), 235799307cad4d30bce77aba9a4d5c52 (127.19.126.3:42913)
I20240503 02:05:50.002621 24400 tablet_bootstrap.cc:492] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175: No bootstrap required, opened a new log
I20240503 02:05:50.003127 24400 ts_tablet_manager.cc:1367] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.001s
I20240503 02:05:50.006515 24354 raft_consensus.cc:2454] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f78e25b61ff47d28f6f7119dc404175 in term 1.
I20240503 02:05:50.006713 24400 raft_consensus.cc:348] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:50.007488 24400 raft_consensus.cc:374] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:05:50.007591 24169 leader_election.cc:302] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f78e25b61ff47d28f6f7119dc404175, 235799307cad4d30bce77aba9a4d5c52; no voters:
I20240503 02:05:50.007807 24400 raft_consensus.cc:729] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f78e25b61ff47d28f6f7119dc404175, State: Initialized, Role: FOLLOWER
I20240503 02:05:50.008423 24411 raft_consensus.cc:2781] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:50.008560 24400 consensus_queue.cc:260] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:50.014875 24411 raft_consensus.cc:686] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 1 LEADER]: Becoming Leader. State: Replica: 1f78e25b61ff47d28f6f7119dc404175, State: Running, Role: LEADER
I20240503 02:05:50.015784 24411 consensus_queue.cc:237] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:50.003747 24280 raft_consensus.cc:2454] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f78e25b61ff47d28f6f7119dc404175 in term 1.
I20240503 02:05:50.021756 24400 ts_tablet_manager.cc:1398] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175: Time spent starting tablet: real 0.017s user 0.005s sys 0.012s
I20240503 02:05:50.024528 24103 catalog_manager.cc:5484] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1). New cstate: current_term: 1 leader_uuid: "1f78e25b61ff47d28f6f7119dc404175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } health_report { overall_health: UNKNOWN } } }
I20240503 02:05:50.053864 24405 raft_consensus.cc:482] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:50.054438 24405 raft_consensus.cc:504] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:50.056605 24405 leader_election.cc:288] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449), a844e765195247f1913fd1c68e443511 (127.19.126.2:38555)
I20240503 02:05:50.057346 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d852fa67354e9696275e159f12b5ce" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f78e25b61ff47d28f6f7119dc404175" is_pre_election: true
I20240503 02:05:50.057624 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d852fa67354e9696275e159f12b5ce" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a844e765195247f1913fd1c68e443511" is_pre_election: true
I20240503 02:05:50.057863 24205 raft_consensus.cc:2454] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 0.
I20240503 02:05:50.058353 24280 raft_consensus.cc:2454] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 0.
I20240503 02:05:50.058808 24320 leader_election.cc:302] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f78e25b61ff47d28f6f7119dc404175, 235799307cad4d30bce77aba9a4d5c52; no voters:
I20240503 02:05:50.059510 24405 raft_consensus.cc:2781] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:05:50.059847 24405 raft_consensus.cc:482] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:05:50.060086 24405 raft_consensus.cc:3037] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:50.064267 24405 raft_consensus.cc:504] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:50.065706 24405 leader_election.cc:288] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 election: Requested vote from peers 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449), a844e765195247f1913fd1c68e443511 (127.19.126.2:38555)
I20240503 02:05:50.066426 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d852fa67354e9696275e159f12b5ce" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f78e25b61ff47d28f6f7119dc404175"
I20240503 02:05:50.066881 24205 raft_consensus.cc:3037] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:50.066776 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d852fa67354e9696275e159f12b5ce" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a844e765195247f1913fd1c68e443511"
I20240503 02:05:50.067446 24280 raft_consensus.cc:3037] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:50.069193 24405 raft_consensus.cc:482] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:50.069759 24405 raft_consensus.cc:504] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:50.072477 24205 raft_consensus.cc:2454] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 1.
I20240503 02:05:50.072654 24204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e04891c2b04ec288f8fe077265c900" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f78e25b61ff47d28f6f7119dc404175" is_pre_election: true
I20240503 02:05:50.073343 24279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e04891c2b04ec288f8fe077265c900" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a844e765195247f1913fd1c68e443511" is_pre_election: true
I20240503 02:05:50.073448 24204 raft_consensus.cc:2454] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 0.
I20240503 02:05:50.073592 24320 leader_election.cc:302] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f78e25b61ff47d28f6f7119dc404175, 235799307cad4d30bce77aba9a4d5c52; no voters:
I20240503 02:05:50.074016 24279 raft_consensus.cc:2454] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 0.
I20240503 02:05:50.074594 24417 raft_consensus.cc:2781] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:50.075043 24417 raft_consensus.cc:686] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Becoming Leader. State: Replica: 235799307cad4d30bce77aba9a4d5c52, State: Running, Role: LEADER
I20240503 02:05:50.076064 24319 leader_election.cc:302] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235799307cad4d30bce77aba9a4d5c52, a844e765195247f1913fd1c68e443511; no voters:
I20240503 02:05:50.075804 24417 consensus_queue.cc:237] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:50.077075 24416 raft_consensus.cc:2781] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:05:50.077466 24416 raft_consensus.cc:482] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:05:50.077646 24280 raft_consensus.cc:2454] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 1.
I20240503 02:05:50.077776 24416 raft_consensus.cc:3037] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:50.078761 24405 leader_election.cc:288] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449), a844e765195247f1913fd1c68e443511 (127.19.126.2:38555)
I20240503 02:05:50.086385 24416 raft_consensus.cc:504] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:50.087035 24103 catalog_manager.cc:5484] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 235799307cad4d30bce77aba9a4d5c52 (127.19.126.3). New cstate: current_term: 1 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240503 02:05:50.088325 24416 leader_election.cc:288] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 election: Requested vote from peers 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449), a844e765195247f1913fd1c68e443511 (127.19.126.2:38555)
I20240503 02:05:50.089121 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e04891c2b04ec288f8fe077265c900" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f78e25b61ff47d28f6f7119dc404175"
I20240503 02:05:50.089270 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e04891c2b04ec288f8fe077265c900" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a844e765195247f1913fd1c68e443511"
I20240503 02:05:50.089789 24205 raft_consensus.cc:3037] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:50.089833 24280 raft_consensus.cc:3037] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:05:50.096899 24280 raft_consensus.cc:2454] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 1.
I20240503 02:05:50.097220 24205 raft_consensus.cc:2454] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 1.
I20240503 02:05:50.097947 24319 leader_election.cc:302] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235799307cad4d30bce77aba9a4d5c52, a844e765195247f1913fd1c68e443511; no voters:
I20240503 02:05:50.098855 24416 raft_consensus.cc:2781] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:05:50.099296 24416 raft_consensus.cc:686] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Becoming Leader. State: Replica: 235799307cad4d30bce77aba9a4d5c52, State: Running, Role: LEADER
I20240503 02:05:50.100035 24416 consensus_queue.cc:237] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:50.107143 24103 catalog_manager.cc:5484] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 235799307cad4d30bce77aba9a4d5c52 (127.19.126.3). New cstate: current_term: 1 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240503 02:05:50.172881 19960 tablet_server.cc:178] TabletServer@127.19.126.1:0 shutting down...
I20240503 02:05:50.194000 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:50.195088 19960 tablet_replica.cc:330] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175: stopping tablet replica
I20240503 02:05:50.195600 19960 raft_consensus.cc:2229] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:50.195951 19960 raft_consensus.cc:2258] T b1d852fa67354e9696275e159f12b5ce P 1f78e25b61ff47d28f6f7119dc404175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:50.197887 19960 tablet_replica.cc:330] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175: stopping tablet replica
I20240503 02:05:50.198348 19960 raft_consensus.cc:2229] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:50.198668 19960 raft_consensus.cc:2258] T e1e04891c2b04ec288f8fe077265c900 P 1f78e25b61ff47d28f6f7119dc404175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:50.200178 19960 tablet_replica.cc:330] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175: stopping tablet replica
I20240503 02:05:50.200645 19960 raft_consensus.cc:2229] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:50.200986 19960 raft_consensus.cc:2258] T 41db5c769a474264805464c3418e346c P 1f78e25b61ff47d28f6f7119dc404175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:50.203050 19960 tablet_replica.cc:330] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175: stopping tablet replica
I20240503 02:05:50.203497 19960 raft_consensus.cc:2229] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:05:50.203970 19960 pending_rounds.cc:62] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175: Trying to abort 1 pending ops.
I20240503 02:05:50.204154 19960 pending_rounds.cc:68] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023419187273904128 op_type: NO_OP noop_request { }
I20240503 02:05:50.204460 19960 raft_consensus.cc:2866] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:05:50.204725 19960 raft_consensus.cc:2258] T 18443df96de94fdc930859b89a06e13e P 1f78e25b61ff47d28f6f7119dc404175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:50.226981 19960 tablet_server.cc:194] TabletServer@127.19.126.1:0 shutdown complete.
W20240503 02:05:50.363010 24320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:05:50.365953 24320 consensus_peers.cc:485] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 -> Peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Couldn't send request to peer 1f78e25b61ff47d28f6f7119dc404175. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:50.557507 24416 consensus_queue.cc:1035] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240503 02:05:50.583271 24320 consensus_peers.cc:485] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 -> Peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Couldn't send request to peer 1f78e25b61ff47d28f6f7119dc404175. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:05:50.614317 24320 consensus_peers.cc:485] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 -> Peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Couldn't send request to peer 1f78e25b61ff47d28f6f7119dc404175. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:50.614869 24416 consensus_queue.cc:1035] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:05:50.651804 24405 consensus_queue.cc:1035] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:05:50.938436 24417 consensus_queue.cc:579] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Leader has been unable to successfully communicate with peer 1f78e25b61ff47d28f6f7119dc404175 for more than 1 seconds (1.000s)
W20240503 02:05:50.946636 24101 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 41db5c769a474264805464c3418e346c with cas_config_opid_index -1: no extra replica candidate found for tablet 41db5c769a474264805464c3418e346c (table test-workload [id=f459271f1bb24e7fafc82b47255d0806]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20240503 02:05:51.107553 24417 consensus_queue.cc:579] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Leader has been unable to successfully communicate with peer 1f78e25b61ff47d28f6f7119dc404175 for more than 1 seconds (1.030s)
W20240503 02:05:51.231554 24151 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 1f78e25b61ff47d28f6f7119dc404175 not available for placement
I20240503 02:05:51.239095 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:05:51.247325 24439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:51.250396 24440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:05:51.252027 24442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:05:51.253003 19960 server_base.cc:1034] running on GCE node
I20240503 02:05:51.254029 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:05:51.254238 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:05:51.254406 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701951254388 us; error 0 us; skew 500 ppm
I20240503 02:05:51.254961 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:05:51.258301 19960 webserver.cc:460] Webserver started at http://127.19.126.4:36497/ using document root <none> and password file <none>
I20240503 02:05:51.258808 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:05:51.259003 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:05:51.259332 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:05:51.260816 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-3-root/instance:
uuid: "d46e2c66a7c84c609e592bfd5b8777db"
format_stamp: "Formatted at 2024-05-03 02:05:51 on dist-test-slave-8rwr"
I20240503 02:05:51.265750 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240503 02:05:51.269219 24447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:51.270117 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:05:51.270376 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-3-root
uuid: "d46e2c66a7c84c609e592bfd5b8777db"
format_stamp: "Formatted at 2024-05-03 02:05:51 on dist-test-slave-8rwr"
I20240503 02:05:51.270615 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714701820461004-19960-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:05:51.282783 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:05:51.283987 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:05:51.285640 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:05:51.288677 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:05:51.288874 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:51.289094 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:05:51.289253 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:05:51.346963 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.4:34669
I20240503 02:05:51.347143 24509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.4:34669 every 8 connection(s)
I20240503 02:05:51.373133 24510 heartbeater.cc:347] Connected to a master server at 127.19.126.62:40583
I20240503 02:05:51.373720 24510 heartbeater.cc:464] Registering TS with master...
I20240503 02:05:51.374715 24510 heartbeater.cc:511] Master 127.19.126.62:40583 requested a full tablet report, sending...
I20240503 02:05:51.377063 24101 ts_manager.cc:194] Registered new tserver with Master: d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4:34669)
I20240503 02:05:51.379029 24101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60398
I20240503 02:05:51.408954 24417 consensus_queue.cc:579] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Leader has been unable to successfully communicate with peer 1f78e25b61ff47d28f6f7119dc404175 for more than 1 seconds (1.308s)
I20240503 02:05:51.420768 24354 consensus_queue.cc:237] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } }
I20240503 02:05:51.429821 24280 raft_consensus.cc:1261] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:05:51.431437 24437 consensus_queue.cc:1035] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240503 02:05:51.441392 24320 consensus_peers.cc:485] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 -> Peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Couldn't send request to peer 1f78e25b61ff47d28f6f7119dc404175. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:51.440917 24437 raft_consensus.cc:2932] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } } }
I20240503 02:05:51.442751 24354 consensus_queue.cc:237] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } }
W20240503 02:05:51.446115 24319 consensus_peers.cc:485] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 -> Peer d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4:34669): Couldn't send request to peer d46e2c66a7c84c609e592bfd5b8777db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1e04891c2b04ec288f8fe077265c900. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:51.445411 24280 raft_consensus.cc:2932] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } } }
W20240503 02:05:51.455178 24319 consensus_peers.cc:485] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 -> Peer d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4:34669): Couldn't send request to peer d46e2c66a7c84c609e592bfd5b8777db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1d852fa67354e9696275e159f12b5ce. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:51.455384 24279 raft_consensus.cc:1261] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:05:51.457242 24417 consensus_queue.cc:1035] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240503 02:05:51.473804 24320 consensus_peers.cc:485] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 -> Peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Couldn't send request to peer 1f78e25b61ff47d28f6f7119dc404175. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:51.472402 24417 raft_consensus.cc:2932] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } } }
I20240503 02:05:51.475279 24280 raft_consensus.cc:2932] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } } }
I20240503 02:05:51.479213 24101 catalog_manager.cc:5484] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 reported cstate change: config changed from index -1 to 2, NON_VOTER d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) added. New cstate: current_term: 1 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } } }
I20240503 02:05:51.491007 24088 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b1d852fa67354e9696275e159f12b5ce with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20240503 02:05:51.495172 24088 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e1e04891c2b04ec288f8fe077265c900 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:05:51.511152 24103 catalog_manager.cc:5484] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 reported cstate change: config changed from index -1 to 2, NON_VOTER d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) added. New cstate: current_term: 1 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:05:51.606434 24354 consensus_queue.cc:237] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } }
W20240503 02:05:51.613639 24319 consensus_peers.cc:485] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 -> Peer d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4:34669): Couldn't send request to peer d46e2c66a7c84c609e592bfd5b8777db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db5c769a474264805464c3418e346c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:51.614415 24280 raft_consensus.cc:1261] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:05:51.615577 24320 consensus_peers.cc:485] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 -> Peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Couldn't send request to peer 1f78e25b61ff47d28f6f7119dc404175. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:51.615823 24417 consensus_queue.cc:1035] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:05:51.623180 24516 raft_consensus.cc:2932] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } } }
I20240503 02:05:51.624380 24280 raft_consensus.cc:2932] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } } }
I20240503 02:05:51.633775 24088 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 41db5c769a474264805464c3418e346c with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20240503 02:05:51.635710 24101 catalog_manager.cc:5484] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 reported cstate change: config changed from index -1 to 2, NON_VOTER d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) added. New cstate: current_term: 1 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } } }
I20240503 02:05:51.764430 24437 raft_consensus.cc:482] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:05:51.764966 24437 raft_consensus.cc:504] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:51.766630 24437 leader_election.cc:288] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449), a844e765195247f1913fd1c68e443511 (127.19.126.2:38555)
I20240503 02:05:51.767864 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18443df96de94fdc930859b89a06e13e" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a844e765195247f1913fd1c68e443511" is_pre_election: true
I20240503 02:05:51.768919 24280 raft_consensus.cc:2454] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 1.
I20240503 02:05:51.770079 24319 leader_election.cc:302] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235799307cad4d30bce77aba9a4d5c52, a844e765195247f1913fd1c68e443511; no voters:
W20240503 02:05:51.770596 24320 leader_election.cc:334] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111)
I20240503 02:05:51.770740 24437 raft_consensus.cc:2781] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:05:51.771188 24437 raft_consensus.cc:482] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:05:51.771533 24437 raft_consensus.cc:3037] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:05:51.778764 24437 raft_consensus.cc:504] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:51.780711 24437 leader_election.cc:288] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 2 election: Requested vote from peers 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449), a844e765195247f1913fd1c68e443511 (127.19.126.2:38555)
I20240503 02:05:51.781807 24280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18443df96de94fdc930859b89a06e13e" candidate_uuid: "235799307cad4d30bce77aba9a4d5c52" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a844e765195247f1913fd1c68e443511"
I20240503 02:05:51.782490 24280 raft_consensus.cc:3037] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:05:51.783774 24320 leader_election.cc:334] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111)
I20240503 02:05:51.789433 24280 raft_consensus.cc:2454] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235799307cad4d30bce77aba9a4d5c52 in term 2.
I20240503 02:05:51.790550 24319 leader_election.cc:302] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 235799307cad4d30bce77aba9a4d5c52, a844e765195247f1913fd1c68e443511; no voters: 1f78e25b61ff47d28f6f7119dc404175
I20240503 02:05:51.791399 24437 raft_consensus.cc:2781] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:05:51.791764 24437 raft_consensus.cc:686] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 2 LEADER]: Becoming Leader. State: Replica: 235799307cad4d30bce77aba9a4d5c52, State: Running, Role: LEADER
I20240503 02:05:51.792546 24437 consensus_queue.cc:237] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } }
I20240503 02:05:51.799348 24103 catalog_manager.cc:5484] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 reported cstate change: term changed from 1 to 2, leader changed from 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) to 235799307cad4d30bce77aba9a4d5c52 (127.19.126.3). New cstate: current_term: 2 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } health_report { overall_health: HEALTHY } } }
I20240503 02:05:51.873654 24525 ts_tablet_manager.cc:902] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db: Initiating tablet copy from peer 235799307cad4d30bce77aba9a4d5c52 (127.19.126.3:42913)
I20240503 02:05:51.877507 24525 tablet_copy_client.cc:308] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Beginning tablet copy session from remote peer at address 127.19.126.3:42913
I20240503 02:05:51.888823 24364 tablet_copy_service.cc:140] P 235799307cad4d30bce77aba9a4d5c52: Received BeginTabletCopySession request for tablet b1d852fa67354e9696275e159f12b5ce from peer d46e2c66a7c84c609e592bfd5b8777db ({username='slave'} at 127.0.0.1:56324)
I20240503 02:05:51.889464 24364 tablet_copy_service.cc:161] P 235799307cad4d30bce77aba9a4d5c52: Beginning new tablet copy session on tablet b1d852fa67354e9696275e159f12b5ce from peer d46e2c66a7c84c609e592bfd5b8777db at {username='slave'} at 127.0.0.1:56324: session id = d46e2c66a7c84c609e592bfd5b8777db-b1d852fa67354e9696275e159f12b5ce
I20240503 02:05:51.895927 24364 tablet_copy_source_session.cc:205] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:05:51.898351 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1d852fa67354e9696275e159f12b5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:51.908797 24525 tablet_copy_client.cc:783] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Starting download of 0 data blocks...
I20240503 02:05:51.909415 24525 tablet_copy_client.cc:647] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Starting download of 1 WAL segments...
I20240503 02:05:51.912639 24525 tablet_copy_client.cc:523] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:05:51.917970 24525 tablet_bootstrap.cc:492] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db: Bootstrap starting.
I20240503 02:05:51.931803 24525 tablet_bootstrap.cc:492] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:05:51.932442 24525 tablet_bootstrap.cc:492] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db: Bootstrap complete.
I20240503 02:05:51.932956 24525 ts_tablet_manager.cc:1367] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.002s
I20240503 02:05:51.934866 24525 raft_consensus.cc:348] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } }
I20240503 02:05:51.935451 24525 raft_consensus.cc:729] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d46e2c66a7c84c609e592bfd5b8777db, State: Initialized, Role: LEARNER
I20240503 02:05:51.935927 24525 consensus_queue.cc:260] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } }
I20240503 02:05:51.938220 24510 heartbeater.cc:503] Master 127.19.126.62:40583 was elected leader, sending a full tablet report...
I20240503 02:05:51.938469 24525 ts_tablet_manager.cc:1398] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db: Time spent starting tablet: real 0.005s user 0.003s sys 0.005s
I20240503 02:05:51.940047 24364 tablet_copy_service.cc:342] P 235799307cad4d30bce77aba9a4d5c52: Request end of tablet copy session d46e2c66a7c84c609e592bfd5b8777db-b1d852fa67354e9696275e159f12b5ce received from {username='slave'} at 127.0.0.1:56324
I20240503 02:05:51.940411 24364 tablet_copy_service.cc:433] P 235799307cad4d30bce77aba9a4d5c52: ending tablet copy session d46e2c66a7c84c609e592bfd5b8777db-b1d852fa67354e9696275e159f12b5ce on tablet b1d852fa67354e9696275e159f12b5ce with peer d46e2c66a7c84c609e592bfd5b8777db
I20240503 02:05:51.997325 24525 ts_tablet_manager.cc:902] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db: Initiating tablet copy from peer 235799307cad4d30bce77aba9a4d5c52 (127.19.126.3:42913)
I20240503 02:05:52.001188 24525 tablet_copy_client.cc:308] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Beginning tablet copy session from remote peer at address 127.19.126.3:42913
I20240503 02:05:52.002756 24364 tablet_copy_service.cc:140] P 235799307cad4d30bce77aba9a4d5c52: Received BeginTabletCopySession request for tablet e1e04891c2b04ec288f8fe077265c900 from peer d46e2c66a7c84c609e592bfd5b8777db ({username='slave'} at 127.0.0.1:56324)
I20240503 02:05:52.003310 24364 tablet_copy_service.cc:161] P 235799307cad4d30bce77aba9a4d5c52: Beginning new tablet copy session on tablet e1e04891c2b04ec288f8fe077265c900 from peer d46e2c66a7c84c609e592bfd5b8777db at {username='slave'} at 127.0.0.1:56324: session id = d46e2c66a7c84c609e592bfd5b8777db-e1e04891c2b04ec288f8fe077265c900
I20240503 02:05:52.011715 24364 tablet_copy_source_session.cc:205] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:05:52.014636 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e04891c2b04ec288f8fe077265c900. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:52.024237 24525 tablet_copy_client.cc:783] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Starting download of 0 data blocks...
I20240503 02:05:52.024686 24525 tablet_copy_client.cc:647] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Starting download of 1 WAL segments...
I20240503 02:05:52.027853 24525 tablet_copy_client.cc:523] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:05:52.033164 24525 tablet_bootstrap.cc:492] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db: Bootstrap starting.
I20240503 02:05:52.046583 24525 tablet_bootstrap.cc:492] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:05:52.047226 24525 tablet_bootstrap.cc:492] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db: Bootstrap complete.
I20240503 02:05:52.047741 24525 ts_tablet_manager.cc:1367] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.006s
I20240503 02:05:52.049645 24525 raft_consensus.cc:348] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } }
I20240503 02:05:52.050231 24525 raft_consensus.cc:729] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d46e2c66a7c84c609e592bfd5b8777db, State: Initialized, Role: LEARNER
I20240503 02:05:52.050727 24525 consensus_queue.cc:260] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } }
I20240503 02:05:52.052524 24525 ts_tablet_manager.cc:1398] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240503 02:05:52.054167 24364 tablet_copy_service.cc:342] P 235799307cad4d30bce77aba9a4d5c52: Request end of tablet copy session d46e2c66a7c84c609e592bfd5b8777db-e1e04891c2b04ec288f8fe077265c900 received from {username='slave'} at 127.0.0.1:56324
I20240503 02:05:52.054476 24364 tablet_copy_service.cc:433] P 235799307cad4d30bce77aba9a4d5c52: ending tablet copy session d46e2c66a7c84c609e592bfd5b8777db-e1e04891c2b04ec288f8fe077265c900 on tablet e1e04891c2b04ec288f8fe077265c900 with peer d46e2c66a7c84c609e592bfd5b8777db
I20240503 02:05:52.144462 24525 ts_tablet_manager.cc:902] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db: Initiating tablet copy from peer 235799307cad4d30bce77aba9a4d5c52 (127.19.126.3:42913)
I20240503 02:05:52.148265 24525 tablet_copy_client.cc:308] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Beginning tablet copy session from remote peer at address 127.19.126.3:42913
I20240503 02:05:52.149617 24364 tablet_copy_service.cc:140] P 235799307cad4d30bce77aba9a4d5c52: Received BeginTabletCopySession request for tablet 41db5c769a474264805464c3418e346c from peer d46e2c66a7c84c609e592bfd5b8777db ({username='slave'} at 127.0.0.1:56324)
I20240503 02:05:52.150053 24364 tablet_copy_service.cc:161] P 235799307cad4d30bce77aba9a4d5c52: Beginning new tablet copy session on tablet 41db5c769a474264805464c3418e346c from peer d46e2c66a7c84c609e592bfd5b8777db at {username='slave'} at 127.0.0.1:56324: session id = d46e2c66a7c84c609e592bfd5b8777db-41db5c769a474264805464c3418e346c
I20240503 02:05:52.155244 24364 tablet_copy_source_session.cc:205] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:05:52.157518 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41db5c769a474264805464c3418e346c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:52.166728 24525 tablet_copy_client.cc:783] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Starting download of 0 data blocks...
I20240503 02:05:52.167188 24525 tablet_copy_client.cc:647] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Starting download of 1 WAL segments...
I20240503 02:05:52.169845 24525 tablet_copy_client.cc:523] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:05:52.175391 24525 tablet_bootstrap.cc:492] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db: Bootstrap starting.
I20240503 02:05:52.186939 24525 tablet_bootstrap.cc:492] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:05:52.187530 24525 tablet_bootstrap.cc:492] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db: Bootstrap complete.
I20240503 02:05:52.187964 24525 ts_tablet_manager.cc:1367] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.001s
I20240503 02:05:52.189613 24525 raft_consensus.cc:348] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } }
I20240503 02:05:52.190127 24525 raft_consensus.cc:729] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d46e2c66a7c84c609e592bfd5b8777db, State: Initialized, Role: LEARNER
I20240503 02:05:52.190551 24525 consensus_queue.cc:260] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } }
I20240503 02:05:52.192137 24525 ts_tablet_manager.cc:1398] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:05:52.193807 24364 tablet_copy_service.cc:342] P 235799307cad4d30bce77aba9a4d5c52: Request end of tablet copy session d46e2c66a7c84c609e592bfd5b8777db-41db5c769a474264805464c3418e346c received from {username='slave'} at 127.0.0.1:56324
I20240503 02:05:52.194236 24364 tablet_copy_service.cc:433] P 235799307cad4d30bce77aba9a4d5c52: ending tablet copy session d46e2c66a7c84c609e592bfd5b8777db-41db5c769a474264805464c3418e346c on tablet 41db5c769a474264805464c3418e346c with peer d46e2c66a7c84c609e592bfd5b8777db
W20240503 02:05:52.234336 24151 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 1f78e25b61ff47d28f6f7119dc404175 not available for placement
I20240503 02:05:52.278959 24280 raft_consensus.cc:1261] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 2 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 02:05:52.280140 24417 consensus_queue.cc:1035] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:05:52.305853 24320 consensus_peers.cc:485] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 -> Peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Couldn't send request to peer 1f78e25b61ff47d28f6f7119dc404175. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:52.328042 24485 raft_consensus.cc:1203] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:05:52.468463 24548 consensus_queue.cc:579] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Leader has been unable to successfully communicate with peer 1f78e25b61ff47d28f6f7119dc404175 for more than 1 seconds (1.046s)
I20240503 02:05:52.515375 24485 raft_consensus.cc:1203] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:05:52.735256 24485 raft_consensus.cc:1203] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:05:52.911170 24548 consensus_queue.cc:579] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Leader has been unable to successfully communicate with peer 1f78e25b61ff47d28f6f7119dc404175 for more than 1 seconds (1.118s)
I20240503 02:05:52.919871 24354 consensus_queue.cc:237] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } }
W20240503 02:05:52.924677 24319 consensus_peers.cc:485] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 -> Peer d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4:34669): Couldn't send request to peer d46e2c66a7c84c609e592bfd5b8777db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18443df96de94fdc930859b89a06e13e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:52.925062 24280 raft_consensus.cc:1261] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 2 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:05:52.926324 24548 consensus_queue.cc:1035] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:05:52.927093 24320 consensus_peers.cc:485] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 -> Peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Couldn't send request to peer 1f78e25b61ff47d28f6f7119dc404175. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:52.930838 24417 raft_consensus.cc:2932] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } } }
I20240503 02:05:52.932188 24280 raft_consensus.cc:2932] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } } }
I20240503 02:05:52.937382 24549 raft_consensus.cc:1050] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52: attempting to promote NON_VOTER d46e2c66a7c84c609e592bfd5b8777db to VOTER
I20240503 02:05:52.940223 24549 consensus_queue.cc:237] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } }
I20240503 02:05:52.944983 24088 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 18443df96de94fdc930859b89a06e13e with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:05:52.948587 24101 catalog_manager.cc:5484] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 reported cstate change: config changed from index -1 to 2, NON_VOTER d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) added. New cstate: current_term: 2 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } } }
I20240503 02:05:52.953646 24485 raft_consensus.cc:1261] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db [term 1 LEARNER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:05:52.955219 24553 consensus_queue.cc:1035] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:05:52.956359 24320 consensus_peers.cc:485] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 -> Peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Couldn't send request to peer 1f78e25b61ff47d28f6f7119dc404175. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:52.953646 24280 raft_consensus.cc:1261] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:05:52.958367 24553 consensus_queue.cc:1035] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:05:52.965978 24548 raft_consensus.cc:2932] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:52.970373 24280 raft_consensus.cc:2932] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:52.976250 24485 raft_consensus.cc:2932] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:52.980978 24103 catalog_manager.cc:5484] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 reported cstate change: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:05:52.992061 24354 consensus_queue.cc:237] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } }
I20240503 02:05:52.996706 24485 raft_consensus.cc:1261] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:05:52.997465 24280 raft_consensus.cc:1261] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:05:52.997975 24553 consensus_queue.cc:1035] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:52.998731 24548 consensus_queue.cc:1035] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:53.004444 24417 raft_consensus.cc:2932] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.005661 24280 raft_consensus.cc:2932] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.006145 24485 raft_consensus.cc:2932] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.014312 24088 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1d852fa67354e9696275e159f12b5ce with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:05:53.015525 24100 catalog_manager.cc:5484] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 reported cstate change: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New cstate: current_term: 1 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
W20240503 02:05:53.029124 24090 catalog_manager.cc:4573] TS 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b1d852fa67354e9696275e159f12b5ce: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111)
I20240503 02:05:53.066536 24417 raft_consensus.cc:1050] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52: attempting to promote NON_VOTER d46e2c66a7c84c609e592bfd5b8777db to VOTER
I20240503 02:05:53.068193 24417 consensus_queue.cc:237] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } }
I20240503 02:05:53.073143 24549 consensus_queue.cc:579] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Leader has been unable to successfully communicate with peer 1f78e25b61ff47d28f6f7119dc404175 for more than 1 seconds (1.465s)
I20240503 02:05:53.074237 24280 raft_consensus.cc:1261] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:05:53.073917 24485 raft_consensus.cc:1261] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db [term 1 LEARNER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:05:53.075551 24417 consensus_queue.cc:1035] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:05:53.076843 24320 consensus_peers.cc:485] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 -> Peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Couldn't send request to peer 1f78e25b61ff47d28f6f7119dc404175. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:53.077252 24549 consensus_queue.cc:1035] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:05:53.084090 24417 raft_consensus.cc:2932] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.085460 24485 raft_consensus.cc:2932] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.091593 24280 raft_consensus.cc:2932] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.095278 24101 catalog_manager.cc:5484] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 reported cstate change: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:05:53.107231 24354 consensus_queue.cc:237] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } }
I20240503 02:05:53.111699 24280 raft_consensus.cc:1261] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:05:53.112013 24485 raft_consensus.cc:1261] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:05:53.113118 24417 consensus_queue.cc:1035] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:53.113837 24553 consensus_queue.cc:1035] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:53.119506 24548 raft_consensus.cc:2932] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.121044 24485 raft_consensus.cc:2932] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.125582 24088 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e1e04891c2b04ec288f8fe077265c900 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:05:53.126121 24280 raft_consensus.cc:2932] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.130847 24101 catalog_manager.cc:5484] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db reported cstate change: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New cstate: current_term: 1 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.215695 24548 raft_consensus.cc:1050] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52: attempting to promote NON_VOTER d46e2c66a7c84c609e592bfd5b8777db to VOTER
I20240503 02:05:53.217384 24548 consensus_queue.cc:237] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } }
I20240503 02:05:53.222733 24485 raft_consensus.cc:1261] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db [term 1 LEARNER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:05:53.222750 24280 raft_consensus.cc:1261] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:05:53.224383 24417 consensus_queue.cc:1035] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:05:53.225107 24553 consensus_queue.cc:1035] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:05:53.225777 24320 consensus_peers.cc:485] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 -> Peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Couldn't send request to peer 1f78e25b61ff47d28f6f7119dc404175. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:53.231565 24417 raft_consensus.cc:2932] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.232915 24485 raft_consensus.cc:2932] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
W20240503 02:05:53.236753 24151 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 1f78e25b61ff47d28f6f7119dc404175 not available for placement
I20240503 02:05:53.243304 24280 raft_consensus.cc:2932] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.243845 24100 catalog_manager.cc:5484] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 reported cstate change: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:05:53.258499 24354 consensus_queue.cc:237] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } }
I20240503 02:05:53.262423 24485 raft_consensus.cc:1261] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:05:53.262636 24280 raft_consensus.cc:1261] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:05:53.263568 24548 consensus_queue.cc:1035] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:53.264883 24417 consensus_queue.cc:1035] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:53.269018 24548 raft_consensus.cc:2932] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.270567 24485 raft_consensus.cc:2932] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.271193 24280 raft_consensus.cc:2932] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.279086 24088 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 41db5c769a474264805464c3418e346c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:05:53.280589 24101 catalog_manager.cc:5484] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 reported cstate change: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New cstate: current_term: 1 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:53.484423 24568 ts_tablet_manager.cc:902] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db: Initiating tablet copy from peer 235799307cad4d30bce77aba9a4d5c52 (127.19.126.3:42913)
I20240503 02:05:53.487596 24568 tablet_copy_client.cc:308] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Beginning tablet copy session from remote peer at address 127.19.126.3:42913
I20240503 02:05:53.488972 24364 tablet_copy_service.cc:140] P 235799307cad4d30bce77aba9a4d5c52: Received BeginTabletCopySession request for tablet 18443df96de94fdc930859b89a06e13e from peer d46e2c66a7c84c609e592bfd5b8777db ({username='slave'} at 127.0.0.1:56324)
I20240503 02:05:53.489424 24364 tablet_copy_service.cc:161] P 235799307cad4d30bce77aba9a4d5c52: Beginning new tablet copy session on tablet 18443df96de94fdc930859b89a06e13e from peer d46e2c66a7c84c609e592bfd5b8777db at {username='slave'} at 127.0.0.1:56324: session id = d46e2c66a7c84c609e592bfd5b8777db-18443df96de94fdc930859b89a06e13e
I20240503 02:05:53.494239 24364 tablet_copy_source_session.cc:205] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:05:53.496721 24568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18443df96de94fdc930859b89a06e13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:05:53.506130 24568 tablet_copy_client.cc:783] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Starting download of 0 data blocks...
I20240503 02:05:53.506642 24568 tablet_copy_client.cc:647] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Starting download of 1 WAL segments...
I20240503 02:05:53.509956 24568 tablet_copy_client.cc:523] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:05:53.515331 24568 tablet_bootstrap.cc:492] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db: Bootstrap starting.
I20240503 02:05:53.529249 24568 tablet_bootstrap.cc:492] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:05:53.530112 24568 tablet_bootstrap.cc:492] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db: Bootstrap complete.
I20240503 02:05:53.530802 24568 ts_tablet_manager.cc:1367] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.001s
I20240503 02:05:53.533114 24568 raft_consensus.cc:348] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } }
I20240503 02:05:53.533787 24568 raft_consensus.cc:729] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d46e2c66a7c84c609e592bfd5b8777db, State: Initialized, Role: LEARNER
I20240503 02:05:53.534493 24568 consensus_queue.cc:260] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: true } }
I20240503 02:05:53.536638 24568 ts_tablet_manager.cc:1398] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:05:53.538024 24364 tablet_copy_service.cc:342] P 235799307cad4d30bce77aba9a4d5c52: Request end of tablet copy session d46e2c66a7c84c609e592bfd5b8777db-18443df96de94fdc930859b89a06e13e received from {username='slave'} at 127.0.0.1:56324
I20240503 02:05:53.538408 24364 tablet_copy_service.cc:433] P 235799307cad4d30bce77aba9a4d5c52: ending tablet copy session d46e2c66a7c84c609e592bfd5b8777db-18443df96de94fdc930859b89a06e13e on tablet 18443df96de94fdc930859b89a06e13e with peer d46e2c66a7c84c609e592bfd5b8777db
I20240503 02:05:53.899463 24485 raft_consensus.cc:1203] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
W20240503 02:05:54.205792 24090 catalog_manager.cc:4573] TS 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b1d852fa67354e9696275e159f12b5ce: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111)
W20240503 02:05:54.239923 24151 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 1f78e25b61ff47d28f6f7119dc404175 not available for placement
I20240503 02:05:54.247051 24577 consensus_queue.cc:579] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Leader has been unable to successfully communicate with peer 1f78e25b61ff47d28f6f7119dc404175 for more than 1 seconds (1.326s)
I20240503 02:05:54.356758 24577 raft_consensus.cc:1050] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52: attempting to promote NON_VOTER d46e2c66a7c84c609e592bfd5b8777db to VOTER
I20240503 02:05:54.358533 24577 consensus_queue.cc:237] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } }
I20240503 02:05:54.364248 24280 raft_consensus.cc:1261] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 2 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:05:54.363605 24485 raft_consensus.cc:1261] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db [term 2 LEARNER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:05:54.365335 24579 consensus_queue.cc:1035] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:05:54.366297 24577 consensus_queue.cc:1035] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:05:54.366897 24320 consensus_peers.cc:485] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 -> Peer 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): Couldn't send request to peer 1f78e25b61ff47d28f6f7119dc404175. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:54.372707 24579 raft_consensus.cc:2932] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:54.374205 24485 raft_consensus.cc:2932] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:54.381685 24280 raft_consensus.cc:2932] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:54.384447 24103 catalog_manager.cc:5484] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 reported cstate change: config changed from index 2 to 3, d46e2c66a7c84c609e592bfd5b8777db (127.19.126.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f78e25b61ff47d28f6f7119dc404175" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 45449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:05:54.398931 24354 consensus_queue.cc:237] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } }
I20240503 02:05:54.403478 24485 raft_consensus.cc:1261] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db [term 2 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:05:54.403632 24280 raft_consensus.cc:1261] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 2 FOLLOWER]: Refusing update from remote peer 235799307cad4d30bce77aba9a4d5c52: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:05:54.404554 24579 consensus_queue.cc:1035] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:54.405241 24553 consensus_queue.cc:1035] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:05:54.410557 24577 raft_consensus.cc:2932] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:54.411684 24280 raft_consensus.cc:2932] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:54.412715 24485 raft_consensus.cc:2932] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:54.422056 24100 catalog_manager.cc:5484] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 reported cstate change: config changed from index 3 to 4, VOTER 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1) evicted. New cstate: current_term: 2 leader_uuid: "235799307cad4d30bce77aba9a4d5c52" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a844e765195247f1913fd1c68e443511" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38555 } } peers { permanent_uuid: "235799307cad4d30bce77aba9a4d5c52" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 42913 } } peers { permanent_uuid: "d46e2c66a7c84c609e592bfd5b8777db" member_type: VOTER last_known_addr { host: "127.19.126.4" port: 34669 } attrs { promote: false } } }
I20240503 02:05:54.425089 24088 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 18443df96de94fdc930859b89a06e13e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 02:05:55.260231 24090 catalog_manager.cc:4573] TS 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b1d852fa67354e9696275e159f12b5ce: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111)
W20240503 02:05:55.496474 24090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:05:56.629945 24090 catalog_manager.cc:4573] TS 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 18443df96de94fdc930859b89a06e13e: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111)
I20240503 02:05:58.387094 19960 tablet_server.cc:178] TabletServer@127.19.126.2:0 shutting down...
I20240503 02:05:58.407878 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:58.408605 19960 tablet_replica.cc:330] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511: stopping tablet replica
I20240503 02:05:58.409298 19960 raft_consensus.cc:2229] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:58.409857 19960 raft_consensus.cc:2258] T b1d852fa67354e9696275e159f12b5ce P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:58.413223 19960 tablet_replica.cc:330] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511: stopping tablet replica
I20240503 02:05:58.413760 19960 raft_consensus.cc:2229] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:58.414294 19960 raft_consensus.cc:2258] T 18443df96de94fdc930859b89a06e13e P a844e765195247f1913fd1c68e443511 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:58.415969 19960 tablet_replica.cc:330] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511: stopping tablet replica
I20240503 02:05:58.416527 19960 raft_consensus.cc:2229] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:58.417059 19960 raft_consensus.cc:2258] T e1e04891c2b04ec288f8fe077265c900 P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:58.418998 19960 tablet_replica.cc:330] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511: stopping tablet replica
I20240503 02:05:58.419459 19960 raft_consensus.cc:2229] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:58.420101 19960 raft_consensus.cc:2258] T 41db5c769a474264805464c3418e346c P a844e765195247f1913fd1c68e443511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:58.442970 19960 tablet_server.cc:194] TabletServer@127.19.126.2:0 shutdown complete.
I20240503 02:05:58.457196 19960 tablet_server.cc:178] TabletServer@127.19.126.3:0 shutting down...
W20240503 02:05:58.469390 24319 consensus_peers.cc:485] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 -> Peer a844e765195247f1913fd1c68e443511 (127.19.126.2:38555): Couldn't send request to peer a844e765195247f1913fd1c68e443511. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.2:38555: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:05:58.483232 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:58.483950 19960 tablet_replica.cc:330] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52: stopping tablet replica
I20240503 02:05:58.484514 19960 raft_consensus.cc:2229] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:05:58.485394 19960 raft_consensus.cc:2258] T b1d852fa67354e9696275e159f12b5ce P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:58.487532 19960 tablet_replica.cc:330] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52: stopping tablet replica
I20240503 02:05:58.488173 19960 raft_consensus.cc:2229] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:05:58.488830 19960 raft_consensus.cc:2258] T 18443df96de94fdc930859b89a06e13e P 235799307cad4d30bce77aba9a4d5c52 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:58.490518 19960 tablet_replica.cc:330] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52: stopping tablet replica
I20240503 02:05:58.490957 19960 raft_consensus.cc:2229] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:05:58.491608 19960 raft_consensus.cc:2258] T e1e04891c2b04ec288f8fe077265c900 P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:58.493378 19960 tablet_replica.cc:330] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52: stopping tablet replica
I20240503 02:05:58.493774 19960 raft_consensus.cc:2229] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:05:58.494393 19960 raft_consensus.cc:2258] T 41db5c769a474264805464c3418e346c P 235799307cad4d30bce77aba9a4d5c52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:58.515795 19960 tablet_server.cc:194] TabletServer@127.19.126.3:0 shutdown complete.
I20240503 02:05:58.528790 19960 tablet_server.cc:178] TabletServer@127.19.126.4:0 shutting down...
I20240503 02:05:58.549224 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:05:58.549837 19960 tablet_replica.cc:330] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db: stopping tablet replica
I20240503 02:05:58.550338 19960 raft_consensus.cc:2229] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:58.550827 19960 raft_consensus.cc:2258] T 18443df96de94fdc930859b89a06e13e P d46e2c66a7c84c609e592bfd5b8777db [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:58.552537 19960 tablet_replica.cc:330] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db: stopping tablet replica
I20240503 02:05:58.552992 19960 raft_consensus.cc:2229] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:58.553480 19960 raft_consensus.cc:2258] T 41db5c769a474264805464c3418e346c P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:58.555047 19960 tablet_replica.cc:330] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db: stopping tablet replica
I20240503 02:05:58.555498 19960 raft_consensus.cc:2229] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:58.555932 19960 raft_consensus.cc:2258] T e1e04891c2b04ec288f8fe077265c900 P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:58.557475 19960 tablet_replica.cc:330] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db: stopping tablet replica
I20240503 02:05:58.557878 19960 raft_consensus.cc:2229] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:05:58.558298 19960 raft_consensus.cc:2258] T b1d852fa67354e9696275e159f12b5ce P d46e2c66a7c84c609e592bfd5b8777db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:05:58.579027 19960 tablet_server.cc:194] TabletServer@127.19.126.4:0 shutdown complete.
I20240503 02:05:58.589140 19960 master.cc:537] Master@127.19.126.62:40583 shutting down...
W20240503 02:05:58.711424 24090 catalog_manager.cc:4573] TS 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 18443df96de94fdc930859b89a06e13e: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111)
W20240503 02:06:01.499653 24090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240503 02:06:01.502935 24090 catalog_manager.cc:4573] TS 1f78e25b61ff47d28f6f7119dc404175 (127.19.126.1:45449): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b1d852fa67354e9696275e159f12b5ce: Network error: Client connection negotiation failed: client connection to 127.19.126.1:45449: connect: Connection refused (error 111)
I20240503 02:06:02.197710 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:02.198722 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:02.199098 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 843ea11dd71c4fe59021a9cc07ad8be6: stopping tablet replica
I20240503 02:06:02.239326 19960 master.cc:552] Master@127.19.126.62:40583 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (14204 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20240503 02:06:02.285794 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:37105
I20240503 02:06:02.286772 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:02.292603 24599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:02.293323 24600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:02.294071 24602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:02.294484 19960 server_base.cc:1034] running on GCE node
I20240503 02:06:02.295248 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:02.295433 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:02.295553 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701962295542 us; error 0 us; skew 500 ppm
I20240503 02:06:02.295965 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:02.298264 19960 webserver.cc:460] Webserver started at http://127.19.126.62:35909/ using document root <none> and password file <none>
I20240503 02:06:02.298660 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:02.298800 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:02.299003 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:02.300161 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "babf6a18bafb42e4a4d5d5e9583e65a8"
format_stamp: "Formatted at 2024-05-03 02:06:02 on dist-test-slave-8rwr"
I20240503 02:06:02.304364 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:06:02.307478 24607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:02.308187 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:06:02.308409 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "babf6a18bafb42e4a4d5d5e9583e65a8"
format_stamp: "Formatted at 2024-05-03 02:06:02 on dist-test-slave-8rwr"
I20240503 02:06:02.308691 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:02.321979 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:02.323055 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:02.359154 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:37105
I20240503 02:06:02.359325 24658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:37105 every 8 connection(s)
I20240503 02:06:02.363301 24659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:02.374611 24659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8: Bootstrap starting.
I20240503 02:06:02.378916 24659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:02.382944 24659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8: No bootstrap required, opened a new log
I20240503 02:06:02.384932 24659 raft_consensus.cc:348] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babf6a18bafb42e4a4d5d5e9583e65a8" member_type: VOTER }
I20240503 02:06:02.385478 24659 raft_consensus.cc:374] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:02.385717 24659 raft_consensus.cc:729] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babf6a18bafb42e4a4d5d5e9583e65a8, State: Initialized, Role: FOLLOWER
I20240503 02:06:02.386250 24659 consensus_queue.cc:260] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babf6a18bafb42e4a4d5d5e9583e65a8" member_type: VOTER }
I20240503 02:06:02.386699 24659 raft_consensus.cc:388] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:06:02.386914 24659 raft_consensus.cc:482] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:06:02.387168 24659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:02.391532 24659 raft_consensus.cc:504] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babf6a18bafb42e4a4d5d5e9583e65a8" member_type: VOTER }
I20240503 02:06:02.392005 24659 leader_election.cc:302] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: babf6a18bafb42e4a4d5d5e9583e65a8; no voters:
I20240503 02:06:02.393078 24659 leader_election.cc:288] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:06:02.393472 24662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:02.394773 24662 raft_consensus.cc:686] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [term 1 LEADER]: Becoming Leader. State: Replica: babf6a18bafb42e4a4d5d5e9583e65a8, State: Running, Role: LEADER
I20240503 02:06:02.395346 24662 consensus_queue.cc:237] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babf6a18bafb42e4a4d5d5e9583e65a8" member_type: VOTER }
I20240503 02:06:02.395942 24659 sys_catalog.cc:564] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:06:02.401443 24663 sys_catalog.cc:455] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "babf6a18bafb42e4a4d5d5e9583e65a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babf6a18bafb42e4a4d5d5e9583e65a8" member_type: VOTER } }
I20240503 02:06:02.401716 24664 sys_catalog.cc:455] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader babf6a18bafb42e4a4d5d5e9583e65a8. Latest consensus state: current_term: 1 leader_uuid: "babf6a18bafb42e4a4d5d5e9583e65a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babf6a18bafb42e4a4d5d5e9583e65a8" member_type: VOTER } }
I20240503 02:06:02.402215 24664 sys_catalog.cc:458] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [sys.catalog]: This master's current role is: LEADER
I20240503 02:06:02.402207 24663 sys_catalog.cc:458] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [sys.catalog]: This master's current role is: LEADER
I20240503 02:06:02.408283 24670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:06:02.416608 24670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:06:02.417626 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:06:02.425102 24670 catalog_manager.cc:1347] Generated new cluster ID: 52ed644a47cf4a3cb194eb80705c85ec
I20240503 02:06:02.425375 24670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:06:02.446827 24670 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:06:02.448022 24670 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:06:02.457764 24670 catalog_manager.cc:5857] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8: Generated new TSK 0
I20240503 02:06:02.458278 24670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:06:02.484308 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:02.490466 24680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:02.492275 24681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:02.493460 24683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:02.494231 19960 server_base.cc:1034] running on GCE node
I20240503 02:06:02.495005 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:02.495198 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:02.495321 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701962495309 us; error 0 us; skew 500 ppm
I20240503 02:06:02.495847 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:02.498261 19960 webserver.cc:460] Webserver started at http://127.19.126.1:41909/ using document root <none> and password file <none>
I20240503 02:06:02.498703 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:02.498893 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:02.499212 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:02.500730 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-0-root/instance:
uuid: "5b5743a96978464389348dd954b48c96"
format_stamp: "Formatted at 2024-05-03 02:06:02 on dist-test-slave-8rwr"
I20240503 02:06:02.505702 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 02:06:02.508836 24688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:02.509527 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:06:02.509840 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "5b5743a96978464389348dd954b48c96"
format_stamp: "Formatted at 2024-05-03 02:06:02 on dist-test-slave-8rwr"
I20240503 02:06:02.510107 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:02.526372 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:02.527431 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:02.528782 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:02.531080 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:02.531328 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:02.531596 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:02.531744 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:02.568490 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:40993
I20240503 02:06:02.568583 24750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:40993 every 8 connection(s)
I20240503 02:06:02.573084 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:02.580421 24755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:02.581502 24756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:02.585644 24758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:02.586122 24751 heartbeater.cc:347] Connected to a master server at 127.19.126.62:37105
I20240503 02:06:02.586457 24751 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:02.586942 19960 server_base.cc:1034] running on GCE node
I20240503 02:06:02.587414 24751 heartbeater.cc:511] Master 127.19.126.62:37105 requested a full tablet report, sending...
I20240503 02:06:02.587718 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:02.587934 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:02.588086 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701962588070 us; error 0 us; skew 500 ppm
I20240503 02:06:02.588668 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:02.589601 24624 ts_manager.cc:194] Registered new tserver with Master: 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993)
I20240503 02:06:02.591434 19960 webserver.cc:460] Webserver started at http://127.19.126.2:36517/ using document root <none> and password file <none>
I20240503 02:06:02.592056 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:02.592290 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:02.592252 24624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39202
I20240503 02:06:02.592759 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:02.594157 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-1-root/instance:
uuid: "665eb3bd61e54938b4e1674df80f52a8"
format_stamp: "Formatted at 2024-05-03 02:06:02 on dist-test-slave-8rwr"
I20240503 02:06:02.598331 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240503 02:06:02.601464 24763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:02.602195 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:06:02.602461 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "665eb3bd61e54938b4e1674df80f52a8"
format_stamp: "Formatted at 2024-05-03 02:06:02 on dist-test-slave-8rwr"
I20240503 02:06:02.602712 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:02.626855 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:02.627977 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:02.629309 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:02.631366 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:02.631554 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:02.631754 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:02.631894 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:02.668680 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:43059
I20240503 02:06:02.668773 24825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:43059 every 8 connection(s)
I20240503 02:06:02.673450 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:02.680671 24829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:02.681782 24830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:02.683286 24832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:02.684521 24826 heartbeater.cc:347] Connected to a master server at 127.19.126.62:37105
I20240503 02:06:02.684944 24826 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:02.685629 24826 heartbeater.cc:511] Master 127.19.126.62:37105 requested a full tablet report, sending...
I20240503 02:06:02.686328 19960 server_base.cc:1034] running on GCE node
I20240503 02:06:02.687038 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:02.687243 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:02.687414 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701962687396 us; error 0 us; skew 500 ppm
I20240503 02:06:02.687602 24624 ts_manager.cc:194] Registered new tserver with Master: 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059)
I20240503 02:06:02.687947 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:02.689055 24624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39212
I20240503 02:06:02.690687 19960 webserver.cc:460] Webserver started at http://127.19.126.3:40387/ using document root <none> and password file <none>
I20240503 02:06:02.691187 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:02.691371 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:02.691612 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:02.692761 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-2-root/instance:
uuid: "fa8da70b70624f94b60257932988054a"
format_stamp: "Formatted at 2024-05-03 02:06:02 on dist-test-slave-8rwr"
I20240503 02:06:02.697037 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:06:02.700206 24837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:02.700904 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:06:02.701262 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "fa8da70b70624f94b60257932988054a"
format_stamp: "Formatted at 2024-05-03 02:06:02 on dist-test-slave-8rwr"
I20240503 02:06:02.701613 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:02.722679 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:02.723798 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:02.725209 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:02.727409 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:02.727613 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:02.727825 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:02.727970 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:02.765379 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:43765
I20240503 02:06:02.765461 24899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:43765 every 8 connection(s)
I20240503 02:06:02.777804 24900 heartbeater.cc:347] Connected to a master server at 127.19.126.62:37105
I20240503 02:06:02.778149 24900 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:02.778851 24900 heartbeater.cc:511] Master 127.19.126.62:37105 requested a full tablet report, sending...
I20240503 02:06:02.780570 24624 ts_manager.cc:194] Registered new tserver with Master: fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:02.780697 19960 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012314986s
I20240503 02:06:02.781877 24624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39224
I20240503 02:06:03.595017 24751 heartbeater.cc:503] Master 127.19.126.62:37105 was elected leader, sending a full tablet report...
I20240503 02:06:03.691535 24826 heartbeater.cc:503] Master 127.19.126.62:37105 was elected leader, sending a full tablet report...
I20240503 02:06:03.784552 24900 heartbeater.cc:503] Master 127.19.126.62:37105 was elected leader, sending a full tablet report...
I20240503 02:06:03.812741 19960 test_util.cc:268] Using random seed: 695394159
I20240503 02:06:03.833506 24624 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39230:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:06:03.835583 24624 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:06:03.890888 24791 tablet_service.cc:1467] Processing CreateTablet for tablet 0e73a484b6e9487ebd53af9983a857b0 (DEFAULT_TABLE table=test-workload [id=7ef0a3018e0c4cf5bdd74658499b4245]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:06:03.891569 24789 tablet_service.cc:1467] Processing CreateTablet for tablet 4db6be3ede84405686653fa4e6d04075 (DEFAULT_TABLE table=test-workload [id=7ef0a3018e0c4cf5bdd74658499b4245]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:06:03.892321 24791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e73a484b6e9487ebd53af9983a857b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:03.892963 24789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db6be3ede84405686653fa4e6d04075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:03.893066 24788 tablet_service.cc:1467] Processing CreateTablet for tablet 300c03aa7f9e45cabea2330f78343504 (DEFAULT_TABLE table=test-workload [id=7ef0a3018e0c4cf5bdd74658499b4245]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:06:03.893919 24790 tablet_service.cc:1467] Processing CreateTablet for tablet ef0a9169ad3c42a380f97acda754f322 (DEFAULT_TABLE table=test-workload [id=7ef0a3018e0c4cf5bdd74658499b4245]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:06:03.894404 24788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300c03aa7f9e45cabea2330f78343504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:03.895232 24790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef0a9169ad3c42a380f97acda754f322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:03.896199 24716 tablet_service.cc:1467] Processing CreateTablet for tablet 0e73a484b6e9487ebd53af9983a857b0 (DEFAULT_TABLE table=test-workload [id=7ef0a3018e0c4cf5bdd74658499b4245]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:06:03.896689 24713 tablet_service.cc:1467] Processing CreateTablet for tablet 300c03aa7f9e45cabea2330f78343504 (DEFAULT_TABLE table=test-workload [id=7ef0a3018e0c4cf5bdd74658499b4245]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:06:03.897662 24716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e73a484b6e9487ebd53af9983a857b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:03.898744 24714 tablet_service.cc:1467] Processing CreateTablet for tablet 4db6be3ede84405686653fa4e6d04075 (DEFAULT_TABLE table=test-workload [id=7ef0a3018e0c4cf5bdd74658499b4245]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:06:03.900051 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db6be3ede84405686653fa4e6d04075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:03.902405 24715 tablet_service.cc:1467] Processing CreateTablet for tablet ef0a9169ad3c42a380f97acda754f322 (DEFAULT_TABLE table=test-workload [id=7ef0a3018e0c4cf5bdd74658499b4245]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:06:03.903647 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef0a9169ad3c42a380f97acda754f322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:03.905858 24713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300c03aa7f9e45cabea2330f78343504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:03.930351 24920 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap starting.
I20240503 02:06:03.932796 24921 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: Bootstrap starting.
I20240503 02:06:03.942191 24920 tablet_bootstrap.cc:654] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:03.945732 24921 tablet_bootstrap.cc:654] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:03.952546 24865 tablet_service.cc:1467] Processing CreateTablet for tablet 0e73a484b6e9487ebd53af9983a857b0 (DEFAULT_TABLE table=test-workload [id=7ef0a3018e0c4cf5bdd74658499b4245]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:06:03.954363 24865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e73a484b6e9487ebd53af9983a857b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:03.959957 24864 tablet_service.cc:1467] Processing CreateTablet for tablet ef0a9169ad3c42a380f97acda754f322 (DEFAULT_TABLE table=test-workload [id=7ef0a3018e0c4cf5bdd74658499b4245]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:06:03.961722 24864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef0a9169ad3c42a380f97acda754f322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:03.979308 24863 tablet_service.cc:1467] Processing CreateTablet for tablet 4db6be3ede84405686653fa4e6d04075 (DEFAULT_TABLE table=test-workload [id=7ef0a3018e0c4cf5bdd74658499b4245]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:06:03.981040 24863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db6be3ede84405686653fa4e6d04075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:03.983601 24921 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: No bootstrap required, opened a new log
I20240503 02:06:03.984061 24921 ts_tablet_manager.cc:1367] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: Time spent bootstrapping tablet: real 0.052s user 0.014s sys 0.026s
I20240503 02:06:03.987236 24921 raft_consensus.cc:348] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:03.987176 24862 tablet_service.cc:1467] Processing CreateTablet for tablet 300c03aa7f9e45cabea2330f78343504 (DEFAULT_TABLE table=test-workload [id=7ef0a3018e0c4cf5bdd74658499b4245]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:06:03.987931 24921 raft_consensus.cc:374] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:03.988171 24921 raft_consensus.cc:729] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5743a96978464389348dd954b48c96, State: Initialized, Role: FOLLOWER
I20240503 02:06:03.988651 24862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300c03aa7f9e45cabea2330f78343504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:03.988699 24921 consensus_queue.cc:260] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:03.991238 24920 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8: No bootstrap required, opened a new log
I20240503 02:06:03.991904 24920 ts_tablet_manager.cc:1367] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8: Time spent bootstrapping tablet: real 0.062s user 0.013s sys 0.037s
I20240503 02:06:04.002956 24920 raft_consensus.cc:348] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.004436 24920 raft_consensus.cc:374] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:04.004864 24920 raft_consensus.cc:729] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665eb3bd61e54938b4e1674df80f52a8, State: Initialized, Role: FOLLOWER
I20240503 02:06:04.005852 24920 consensus_queue.cc:260] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.011140 24925 raft_consensus.cc:482] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:04.011719 24925 raft_consensus.cc:504] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.025861 24921 ts_tablet_manager.cc:1398] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: Time spent starting tablet: real 0.042s user 0.019s sys 0.016s
I20240503 02:06:04.027025 24925 leader_election.cc:288] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:04.027223 24921 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96: Bootstrap starting.
I20240503 02:06:04.028585 24926 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a: Bootstrap starting.
I20240503 02:06:04.034761 24921 tablet_bootstrap.cc:654] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:04.037225 24920 ts_tablet_manager.cc:1398] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8: Time spent starting tablet: real 0.045s user 0.021s sys 0.015s
I20240503 02:06:04.038290 24920 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap starting.
I20240503 02:06:04.035485 24926 tablet_bootstrap.cc:654] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:04.044309 24920 tablet_bootstrap.cc:654] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:04.049862 24801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73a484b6e9487ebd53af9983a857b0" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8" is_pre_election: true
I20240503 02:06:04.050487 24921 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96: No bootstrap required, opened a new log
I20240503 02:06:04.050971 24921 ts_tablet_manager.cc:1367] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.001s
W20240503 02:06:04.051648 24691 leader_election.cc:341] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:04.051891 24920 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: No bootstrap required, opened a new log
I20240503 02:06:04.052402 24920 ts_tablet_manager.cc:1367] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240503 02:06:04.054715 24921 raft_consensus.cc:348] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.055493 24921 raft_consensus.cc:374] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:04.055759 24921 raft_consensus.cc:729] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5743a96978464389348dd954b48c96, State: Initialized, Role: FOLLOWER
I20240503 02:06:04.056123 24920 raft_consensus.cc:348] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.056825 24920 raft_consensus.cc:374] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:04.056434 24921 consensus_queue.cc:260] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.057114 24920 raft_consensus.cc:729] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665eb3bd61e54938b4e1674df80f52a8, State: Initialized, Role: FOLLOWER
I20240503 02:06:04.057988 24920 consensus_queue.cc:260] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.060487 24921 ts_tablet_manager.cc:1398] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 02:06:04.061522 24921 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: Bootstrap starting.
I20240503 02:06:04.061676 24920 ts_tablet_manager.cc:1398] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240503 02:06:04.062623 24920 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap starting.
I20240503 02:06:04.063549 24875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73a484b6e9487ebd53af9983a857b0" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a" is_pre_election: true
W20240503 02:06:04.065114 24690 leader_election.cc:341] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa8da70b70624f94b60257932988054a (127.19.126.3:43765): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:04.065583 24690 leader_election.cc:302] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b5743a96978464389348dd954b48c96; no voters: 665eb3bd61e54938b4e1674df80f52a8, fa8da70b70624f94b60257932988054a
I20240503 02:06:04.066864 24925 raft_consensus.cc:2726] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:06:04.067122 24920 tablet_bootstrap.cc:654] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:04.067348 24921 tablet_bootstrap.cc:654] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:04.075738 24920 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: No bootstrap required, opened a new log
I20240503 02:06:04.076093 24920 ts_tablet_manager.cc:1367] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.002s
I20240503 02:06:04.059186 24926 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a: No bootstrap required, opened a new log
I20240503 02:06:04.077176 24926 ts_tablet_manager.cc:1367] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.000s
I20240503 02:06:04.078454 24920 raft_consensus.cc:348] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.078987 24920 raft_consensus.cc:374] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:04.079196 24920 raft_consensus.cc:729] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665eb3bd61e54938b4e1674df80f52a8, State: Initialized, Role: FOLLOWER
I20240503 02:06:04.079465 24921 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: No bootstrap required, opened a new log
I20240503 02:06:04.079950 24921 ts_tablet_manager.cc:1367] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20240503 02:06:04.079785 24920 consensus_queue.cc:260] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.079639 24926 raft_consensus.cc:348] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.080658 24926 raft_consensus.cc:374] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:04.081040 24926 raft_consensus.cc:729] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8da70b70624f94b60257932988054a, State: Initialized, Role: FOLLOWER
I20240503 02:06:04.082087 24920 ts_tablet_manager.cc:1398] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240503 02:06:04.081956 24926 consensus_queue.cc:260] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.083114 24920 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap starting.
I20240503 02:06:04.082685 24921 raft_consensus.cc:348] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.083649 24921 raft_consensus.cc:374] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:04.084074 24921 raft_consensus.cc:729] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5743a96978464389348dd954b48c96, State: Initialized, Role: FOLLOWER
I20240503 02:06:04.084935 24921 consensus_queue.cc:260] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.087751 24921 ts_tablet_manager.cc:1398] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240503 02:06:04.088946 24921 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96: Bootstrap starting.
I20240503 02:06:04.089077 24920 tablet_bootstrap.cc:654] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:04.095139 24921 tablet_bootstrap.cc:654] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:04.096702 24926 ts_tablet_manager.cc:1398] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a: Time spent starting tablet: real 0.019s user 0.008s sys 0.004s
I20240503 02:06:04.097733 24926 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Bootstrap starting.
I20240503 02:06:04.104640 24926 tablet_bootstrap.cc:654] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:04.105878 24920 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8: No bootstrap required, opened a new log
I20240503 02:06:04.106245 24921 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96: No bootstrap required, opened a new log
I20240503 02:06:04.106468 24920 ts_tablet_manager.cc:1367] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.007s
I20240503 02:06:04.106789 24921 ts_tablet_manager.cc:1367] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20240503 02:06:04.109359 24920 raft_consensus.cc:348] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.109676 24921 raft_consensus.cc:348] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.110222 24920 raft_consensus.cc:374] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:04.110293 24926 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: No bootstrap required, opened a new log
I20240503 02:06:04.110324 24921 raft_consensus.cc:374] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:04.110630 24920 raft_consensus.cc:729] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665eb3bd61e54938b4e1674df80f52a8, State: Initialized, Role: FOLLOWER
I20240503 02:06:04.110937 24921 raft_consensus.cc:729] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5743a96978464389348dd954b48c96, State: Initialized, Role: FOLLOWER
I20240503 02:06:04.111039 24926 ts_tablet_manager.cc:1367] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20240503 02:06:04.111444 24920 consensus_queue.cc:260] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.111738 24921 consensus_queue.cc:260] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.113646 24921 ts_tablet_manager.cc:1398] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240503 02:06:04.115516 24920 ts_tablet_manager.cc:1398] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8: Time spent starting tablet: real 0.009s user 0.001s sys 0.004s
I20240503 02:06:04.116307 24926 raft_consensus.cc:348] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.116806 24926 raft_consensus.cc:374] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:04.117022 24926 raft_consensus.cc:729] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8da70b70624f94b60257932988054a, State: Initialized, Role: FOLLOWER
I20240503 02:06:04.117620 24926 consensus_queue.cc:260] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.119753 24926 ts_tablet_manager.cc:1398] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240503 02:06:04.120602 24926 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: Bootstrap starting.
I20240503 02:06:04.125856 24926 tablet_bootstrap.cc:654] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:04.130355 24926 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: No bootstrap required, opened a new log
I20240503 02:06:04.130720 24926 ts_tablet_manager.cc:1367] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20240503 02:06:04.132478 24926 raft_consensus.cc:348] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.133011 24926 raft_consensus.cc:374] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:04.133214 24926 raft_consensus.cc:729] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8da70b70624f94b60257932988054a, State: Initialized, Role: FOLLOWER
I20240503 02:06:04.133811 24926 consensus_queue.cc:260] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.135516 24926 ts_tablet_manager.cc:1398] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240503 02:06:04.136276 24926 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a: Bootstrap starting.
I20240503 02:06:04.140467 24926 tablet_bootstrap.cc:654] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:04.145129 24926 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a: No bootstrap required, opened a new log
I20240503 02:06:04.145514 24926 ts_tablet_manager.cc:1367] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240503 02:06:04.147228 24926 raft_consensus.cc:348] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.147725 24926 raft_consensus.cc:374] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:04.147931 24926 raft_consensus.cc:729] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8da70b70624f94b60257932988054a, State: Initialized, Role: FOLLOWER
I20240503 02:06:04.148492 24926 consensus_queue.cc:260] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.150028 24926 ts_tablet_manager.cc:1398] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:06:04.275897 24927 raft_consensus.cc:482] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:04.276340 24927 raft_consensus.cc:504] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.278208 24927 leader_election.cc:288] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:04.287751 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a9169ad3c42a380f97acda754f322" candidate_uuid: "665eb3bd61e54938b4e1674df80f52a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5743a96978464389348dd954b48c96" is_pre_election: true
I20240503 02:06:04.288050 24875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a9169ad3c42a380f97acda754f322" candidate_uuid: "665eb3bd61e54938b4e1674df80f52a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a" is_pre_election: true
I20240503 02:06:04.288497 24726 raft_consensus.cc:2454] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 665eb3bd61e54938b4e1674df80f52a8 in term 0.
I20240503 02:06:04.288636 24875 raft_consensus.cc:2454] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 665eb3bd61e54938b4e1674df80f52a8 in term 0.
I20240503 02:06:04.289644 24764 leader_election.cc:302] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b5743a96978464389348dd954b48c96, 665eb3bd61e54938b4e1674df80f52a8; no voters:
I20240503 02:06:04.290262 24927 raft_consensus.cc:2781] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:04.290550 24927 raft_consensus.cc:482] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:04.290767 24927 raft_consensus.cc:3037] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:04.295488 24927 raft_consensus.cc:504] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.296895 24927 leader_election.cc:288] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [CANDIDATE]: Term 1 election: Requested vote from peers 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:04.297583 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a9169ad3c42a380f97acda754f322" candidate_uuid: "665eb3bd61e54938b4e1674df80f52a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5743a96978464389348dd954b48c96"
I20240503 02:06:04.297814 24875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a9169ad3c42a380f97acda754f322" candidate_uuid: "665eb3bd61e54938b4e1674df80f52a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a"
I20240503 02:06:04.298049 24726 raft_consensus.cc:3037] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:04.298341 24875 raft_consensus.cc:3037] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:04.302578 24726 raft_consensus.cc:2454] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 665eb3bd61e54938b4e1674df80f52a8 in term 1.
I20240503 02:06:04.302737 24875 raft_consensus.cc:2454] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 665eb3bd61e54938b4e1674df80f52a8 in term 1.
I20240503 02:06:04.303429 24764 leader_election.cc:302] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b5743a96978464389348dd954b48c96, 665eb3bd61e54938b4e1674df80f52a8; no voters:
I20240503 02:06:04.304185 24927 raft_consensus.cc:2781] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:04.305235 24927 raft_consensus.cc:686] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 LEADER]: Becoming Leader. State: Replica: 665eb3bd61e54938b4e1674df80f52a8, State: Running, Role: LEADER
I20240503 02:06:04.305905 24927 consensus_queue.cc:237] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.311700 24927 raft_consensus.cc:482] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:04.312067 24927 raft_consensus.cc:504] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.312546 24622 catalog_manager.cc:5484] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2). New cstate: current_term: 1 leader_uuid: "665eb3bd61e54938b4e1674df80f52a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:04.313789 24927 leader_election.cc:288] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:04.314354 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300c03aa7f9e45cabea2330f78343504" candidate_uuid: "665eb3bd61e54938b4e1674df80f52a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5743a96978464389348dd954b48c96" is_pre_election: true
I20240503 02:06:04.314736 24875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300c03aa7f9e45cabea2330f78343504" candidate_uuid: "665eb3bd61e54938b4e1674df80f52a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a" is_pre_election: true
I20240503 02:06:04.314980 24726 raft_consensus.cc:2454] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 665eb3bd61e54938b4e1674df80f52a8 in term 0.
I20240503 02:06:04.315353 24875 raft_consensus.cc:2454] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 665eb3bd61e54938b4e1674df80f52a8 in term 0.
I20240503 02:06:04.315887 24764 leader_election.cc:302] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b5743a96978464389348dd954b48c96, 665eb3bd61e54938b4e1674df80f52a8; no voters:
I20240503 02:06:04.316504 24927 raft_consensus.cc:2781] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:04.316773 24927 raft_consensus.cc:482] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:04.316957 24927 raft_consensus.cc:3037] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:04.321974 24927 raft_consensus.cc:504] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.323424 24927 leader_election.cc:288] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [CANDIDATE]: Term 1 election: Requested vote from peers 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:04.324079 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300c03aa7f9e45cabea2330f78343504" candidate_uuid: "665eb3bd61e54938b4e1674df80f52a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5743a96978464389348dd954b48c96"
I20240503 02:06:04.324285 24875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300c03aa7f9e45cabea2330f78343504" candidate_uuid: "665eb3bd61e54938b4e1674df80f52a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a"
I20240503 02:06:04.324532 24726 raft_consensus.cc:3037] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:04.324815 24875 raft_consensus.cc:3037] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:04.329763 24875 raft_consensus.cc:2454] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 665eb3bd61e54938b4e1674df80f52a8 in term 1.
I20240503 02:06:04.330580 24726 raft_consensus.cc:2454] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 665eb3bd61e54938b4e1674df80f52a8 in term 1.
I20240503 02:06:04.330670 24765 leader_election.cc:302] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 665eb3bd61e54938b4e1674df80f52a8, fa8da70b70624f94b60257932988054a; no voters:
I20240503 02:06:04.331375 24927 raft_consensus.cc:2781] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:04.331734 24927 raft_consensus.cc:686] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 LEADER]: Becoming Leader. State: Replica: 665eb3bd61e54938b4e1674df80f52a8, State: Running, Role: LEADER
I20240503 02:06:04.332762 24927 consensus_queue.cc:237] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.339092 24622 catalog_manager.cc:5484] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2). New cstate: current_term: 1 leader_uuid: "665eb3bd61e54938b4e1674df80f52a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:04.373023 24935 raft_consensus.cc:482] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:04.373446 24935 raft_consensus.cc:504] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.375456 24935 leader_election.cc:288] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993), 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059)
I20240503 02:06:04.384207 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db6be3ede84405686653fa4e6d04075" candidate_uuid: "fa8da70b70624f94b60257932988054a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5743a96978464389348dd954b48c96" is_pre_election: true
I20240503 02:06:04.384595 24801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db6be3ede84405686653fa4e6d04075" candidate_uuid: "fa8da70b70624f94b60257932988054a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8" is_pre_election: true
I20240503 02:06:04.384932 24726 raft_consensus.cc:2454] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa8da70b70624f94b60257932988054a in term 0.
I20240503 02:06:04.385252 24801 raft_consensus.cc:2454] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa8da70b70624f94b60257932988054a in term 0.
I20240503 02:06:04.386060 24838 leader_election.cc:302] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b5743a96978464389348dd954b48c96, fa8da70b70624f94b60257932988054a; no voters:
I20240503 02:06:04.386780 24935 raft_consensus.cc:2781] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:04.387113 24935 raft_consensus.cc:482] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:04.387339 24935 raft_consensus.cc:3037] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:04.391860 24935 raft_consensus.cc:504] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.393431 24935 leader_election.cc:288] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [CANDIDATE]: Term 1 election: Requested vote from peers 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993), 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059)
I20240503 02:06:04.394070 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db6be3ede84405686653fa4e6d04075" candidate_uuid: "fa8da70b70624f94b60257932988054a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5743a96978464389348dd954b48c96"
I20240503 02:06:04.394492 24726 raft_consensus.cc:3037] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:04.394506 24801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db6be3ede84405686653fa4e6d04075" candidate_uuid: "fa8da70b70624f94b60257932988054a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8"
I20240503 02:06:04.395053 24801 raft_consensus.cc:3037] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:04.398818 24726 raft_consensus.cc:2454] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa8da70b70624f94b60257932988054a in term 1.
I20240503 02:06:04.399276 24801 raft_consensus.cc:2454] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa8da70b70624f94b60257932988054a in term 1.
I20240503 02:06:04.399601 24838 leader_election.cc:302] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b5743a96978464389348dd954b48c96, fa8da70b70624f94b60257932988054a; no voters:
I20240503 02:06:04.400308 24935 raft_consensus.cc:2781] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:04.401077 24935 raft_consensus.cc:686] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 1 LEADER]: Becoming Leader. State: Replica: fa8da70b70624f94b60257932988054a, State: Running, Role: LEADER
I20240503 02:06:04.401703 24935 consensus_queue.cc:237] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
W20240503 02:06:04.408106 24672 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 02:06:04.407919 24622 catalog_manager.cc:5484] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a reported cstate change: term changed from 0 to 1, leader changed from <none> to fa8da70b70624f94b60257932988054a (127.19.126.3). New cstate: current_term: 1 leader_uuid: "fa8da70b70624f94b60257932988054a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:04.411907 24935 raft_consensus.cc:482] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:04.412397 24935 raft_consensus.cc:504] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.414135 24935 leader_election.cc:288] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059), 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993)
I20240503 02:06:04.414878 24801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73a484b6e9487ebd53af9983a857b0" candidate_uuid: "fa8da70b70624f94b60257932988054a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8" is_pre_election: true
I20240503 02:06:04.415427 24801 raft_consensus.cc:2454] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa8da70b70624f94b60257932988054a in term 0.
I20240503 02:06:04.416177 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73a484b6e9487ebd53af9983a857b0" candidate_uuid: "fa8da70b70624f94b60257932988054a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5743a96978464389348dd954b48c96" is_pre_election: true
I20240503 02:06:04.416324 24840 leader_election.cc:302] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 665eb3bd61e54938b4e1674df80f52a8, fa8da70b70624f94b60257932988054a; no voters:
I20240503 02:06:04.416643 24726 raft_consensus.cc:2454] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa8da70b70624f94b60257932988054a in term 0.
I20240503 02:06:04.416996 24935 raft_consensus.cc:2781] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:04.417271 24935 raft_consensus.cc:482] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:04.417562 24935 raft_consensus.cc:3037] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:04.421648 24935 raft_consensus.cc:504] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.423055 24935 leader_election.cc:288] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [CANDIDATE]: Term 1 election: Requested vote from peers 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059), 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993)
I20240503 02:06:04.423681 24801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73a484b6e9487ebd53af9983a857b0" candidate_uuid: "fa8da70b70624f94b60257932988054a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8"
I20240503 02:06:04.424279 24801 raft_consensus.cc:3037] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:04.424331 24726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73a484b6e9487ebd53af9983a857b0" candidate_uuid: "fa8da70b70624f94b60257932988054a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5743a96978464389348dd954b48c96"
I20240503 02:06:04.424861 24726 raft_consensus.cc:3037] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:04.428864 24726 raft_consensus.cc:2454] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa8da70b70624f94b60257932988054a in term 1.
I20240503 02:06:04.429049 24801 raft_consensus.cc:2454] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa8da70b70624f94b60257932988054a in term 1.
I20240503 02:06:04.429636 24838 leader_election.cc:302] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b5743a96978464389348dd954b48c96, fa8da70b70624f94b60257932988054a; no voters:
I20240503 02:06:04.430274 24935 raft_consensus.cc:2781] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:04.430611 24935 raft_consensus.cc:686] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 1 LEADER]: Becoming Leader. State: Replica: fa8da70b70624f94b60257932988054a, State: Running, Role: LEADER
I20240503 02:06:04.431284 24935 consensus_queue.cc:237] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:04.437812 24622 catalog_manager.cc:5484] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a reported cstate change: term changed from 0 to 1, leader changed from <none> to fa8da70b70624f94b60257932988054a (127.19.126.3). New cstate: current_term: 1 leader_uuid: "fa8da70b70624f94b60257932988054a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:04.473155 19960 tablet_server.cc:178] TabletServer@127.19.126.1:0 shutting down...
I20240503 02:06:04.491614 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:04.492370 19960 tablet_replica.cc:330] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96: stopping tablet replica
I20240503 02:06:04.492919 19960 raft_consensus.cc:2229] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:04.493357 19960 raft_consensus.cc:2258] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:04.495098 19960 tablet_replica.cc:330] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: stopping tablet replica
I20240503 02:06:04.495561 19960 raft_consensus.cc:2229] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:04.495895 19960 raft_consensus.cc:2258] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:04.497648 19960 tablet_replica.cc:330] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96: stopping tablet replica
I20240503 02:06:04.498085 19960 raft_consensus.cc:2229] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:04.498390 19960 raft_consensus.cc:2258] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:04.499964 19960 tablet_replica.cc:330] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: stopping tablet replica
I20240503 02:06:04.500370 19960 raft_consensus.cc:2229] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:04.500675 19960 raft_consensus.cc:2258] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:04.523882 19960 tablet_server.cc:194] TabletServer@127.19.126.1:0 shutdown complete.
I20240503 02:06:04.533511 19960 tablet_server.cc:178] TabletServer@127.19.126.2:0 shutting down...
I20240503 02:06:04.548487 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:04.549240 19960 tablet_replica.cc:330] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: stopping tablet replica
I20240503 02:06:04.549750 19960 raft_consensus.cc:2229] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:04.550223 19960 pending_rounds.cc:62] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: Trying to abort 1 pending ops.
I20240503 02:06:04.550379 19960 pending_rounds.cc:68] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023419245803773952 op_type: NO_OP noop_request { }
I20240503 02:06:04.550630 19960 raft_consensus.cc:2866] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:06:04.550824 19960 raft_consensus.cc:2258] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:04.552912 19960 tablet_replica.cc:330] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8: stopping tablet replica
I20240503 02:06:04.553488 19960 raft_consensus.cc:2229] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:04.553897 19960 raft_consensus.cc:2258] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:04.555475 19960 tablet_replica.cc:330] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: stopping tablet replica
I20240503 02:06:04.555920 19960 raft_consensus.cc:2229] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:04.556434 19960 pending_rounds.cc:62] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Trying to abort 1 pending ops.
I20240503 02:06:04.556583 19960 pending_rounds.cc:68] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023419245913051136 op_type: NO_OP noop_request { }
I20240503 02:06:04.556833 19960 raft_consensus.cc:2866] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:06:04.557080 19960 raft_consensus.cc:2258] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:04.559270 19960 tablet_replica.cc:330] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8: stopping tablet replica
I20240503 02:06:04.559677 19960 raft_consensus.cc:2229] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:04.559989 19960 raft_consensus.cc:2258] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:04.580443 19960 tablet_server.cc:194] TabletServer@127.19.126.2:0 shutdown complete.
I20240503 02:06:04.589751 19960 tablet_server.cc:178] TabletServer@127.19.126.3:0 shutting down...
I20240503 02:06:04.604974 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:04.605724 19960 tablet_replica.cc:330] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a: stopping tablet replica
I20240503 02:06:04.606398 19960 raft_consensus.cc:2229] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:04.606775 19960 raft_consensus.cc:2258] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:04.608510 19960 tablet_replica.cc:330] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: stopping tablet replica
I20240503 02:06:04.608995 19960 raft_consensus.cc:2229] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:04.609584 19960 pending_rounds.cc:62] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: Trying to abort 1 pending ops.
I20240503 02:06:04.609761 19960 pending_rounds.cc:68] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023419246195765248 op_type: NO_OP noop_request { }
I20240503 02:06:04.610018 19960 raft_consensus.cc:2866] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:06:04.610262 19960 raft_consensus.cc:2258] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:04.612646 19960 tablet_replica.cc:330] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: stopping tablet replica
I20240503 02:06:04.613137 19960 raft_consensus.cc:2229] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:04.613715 19960 pending_rounds.cc:62] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Trying to abort 1 pending ops.
I20240503 02:06:04.613860 19960 pending_rounds.cc:68] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023419246317412352 op_type: NO_OP noop_request { }
I20240503 02:06:04.614127 19960 raft_consensus.cc:2866] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:06:04.614408 19960 raft_consensus.cc:2258] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:04.616158 19960 tablet_replica.cc:330] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a: stopping tablet replica
I20240503 02:06:04.616554 19960 raft_consensus.cc:2229] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:04.616869 19960 raft_consensus.cc:2258] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:04.637233 19960 tablet_server.cc:194] TabletServer@127.19.126.3:0 shutdown complete.
I20240503 02:06:04.647607 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:04.652614 24951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:04.653749 24952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:04.654345 24954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:04.655019 19960 server_base.cc:1034] running on GCE node
I20240503 02:06:04.655932 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:04.656208 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:04.656447 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701964656434 us; error 0 us; skew 500 ppm
I20240503 02:06:04.656980 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:04.659237 19960 webserver.cc:460] Webserver started at http://127.19.126.4:34921/ using document root <none> and password file <none>
I20240503 02:06:04.659844 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:04.660112 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:04.660440 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:04.661593 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-3-root/instance:
uuid: "623fa4c949e845aea8dcd2238e37c99a"
format_stamp: "Formatted at 2024-05-03 02:06:04 on dist-test-slave-8rwr"
I20240503 02:06:04.665747 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 02:06:04.668694 24959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:04.669461 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:06:04.669821 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-3-root
uuid: "623fa4c949e845aea8dcd2238e37c99a"
format_stamp: "Formatted at 2024-05-03 02:06:04 on dist-test-slave-8rwr"
I20240503 02:06:04.670181 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:04.697548 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:04.698520 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:04.699928 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:04.702029 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:04.702303 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:04.702620 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:04.702877 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:04.738139 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.4:34713
I20240503 02:06:04.738220 25021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.4:34713 every 8 connection(s)
I20240503 02:06:04.751580 25022 heartbeater.cc:347] Connected to a master server at 127.19.126.62:37105
I20240503 02:06:04.751996 25022 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:04.752696 25022 heartbeater.cc:511] Master 127.19.126.62:37105 requested a full tablet report, sending...
I20240503 02:06:04.754436 24622 ts_manager.cc:194] Registered new tserver with Master: 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4:34713)
I20240503 02:06:04.755771 24622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39242
W20240503 02:06:05.419404 24672 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:05.425135 24672 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:05.425565 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:05.429347 24672 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:05.429692 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:05.433372 24672 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:05.433722 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
I20240503 02:06:05.757963 25022 heartbeater.cc:503] Master 127.19.126.62:37105 was elected leader, sending a full tablet report...
W20240503 02:06:06.462765 24672 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:06.471498 24672 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:06.471753 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:06.475477 24672 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:06.475714 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:06.479400 24672 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:06.479645 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:07.490891 24672 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:07.493716 24610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240503 02:06:07.496273 24672 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:07.496495 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:07.500589 24672 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:07.500804 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:07.504539 24672 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:07.504750 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:08.515956 24672 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:08.521575 24672 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:08.521895 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.126.2:43059: connect: Connection refused (error 111)
W20240503 02:06:08.525583 24672 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:08.525765 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:08.529268 24672 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:08.529492 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.126.3:43765: connect: Connection refused (error 111)
W20240503 02:06:09.531430 24672 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 665eb3bd61e54938b4e1674df80f52a8 not available for placement
W20240503 02:06:10.533329 24672 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 665eb3bd61e54938b4e1674df80f52a8 not available for placement
I20240503 02:06:10.776764 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:10.782878 25038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:10.783792 25039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:10.784768 25041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:10.788127 19960 server_base.cc:1034] running on GCE node
I20240503 02:06:10.790725 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:10.790904 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:10.791057 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701970791042 us; error 0 us; skew 500 ppm
I20240503 02:06:10.791564 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:10.799221 19960 webserver.cc:460] Webserver started at http://127.19.126.1:41909/ using document root <none> and password file <none>
I20240503 02:06:10.799649 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:10.799796 19960 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 02:06:10.803329 19960 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:06:10.806258 25046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:10.806982 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:06:10.807334 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "5b5743a96978464389348dd954b48c96"
format_stamp: "Formatted at 2024-05-03 02:06:02 on dist-test-slave-8rwr"
I20240503 02:06:10.807621 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:10.838836 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:10.839825 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:10.841187 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:10.844313 25053 ts_tablet_manager.cc:522] Loading tablet metadata (0/4 complete)
I20240503 02:06:10.863041 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240503 02:06:10.863317 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.020s user 0.002s sys 0.000s
I20240503 02:06:10.863565 19960 ts_tablet_manager.cc:574] Registering tablets (0/4 complete)
I20240503 02:06:10.867434 25053 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96: Bootstrap starting.
I20240503 02:06:10.876974 25053 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:10.877211 19960 ts_tablet_manager.cc:590] Registered 4 tablets
I20240503 02:06:10.877486 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.014s user 0.014s sys 0.000s
I20240503 02:06:10.877650 25053 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96: Bootstrap complete.
I20240503 02:06:10.878218 25053 ts_tablet_manager.cc:1367] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240503 02:06:10.879989 25053 raft_consensus.cc:348] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:10.880537 25053 raft_consensus.cc:729] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5743a96978464389348dd954b48c96, State: Initialized, Role: FOLLOWER
I20240503 02:06:10.881101 25053 consensus_queue.cc:260] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:10.883213 25053 ts_tablet_manager.cc:1398] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:06:10.883823 25053 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: Bootstrap starting.
I20240503 02:06:10.893762 25053 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:10.894400 25053 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: Bootstrap complete.
I20240503 02:06:10.895042 25053 ts_tablet_manager.cc:1367] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 02:06:10.897745 25053 raft_consensus.cc:348] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:10.898442 25053 raft_consensus.cc:729] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5743a96978464389348dd954b48c96, State: Initialized, Role: FOLLOWER
I20240503 02:06:10.899044 25053 consensus_queue.cc:260] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:10.900566 25053 ts_tablet_manager.cc:1398] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:06:10.901335 25053 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: Bootstrap starting.
I20240503 02:06:10.912694 25053 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:10.913401 25053 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: Bootstrap complete.
I20240503 02:06:10.914014 25053 ts_tablet_manager.cc:1367] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20240503 02:06:10.915913 25053 raft_consensus.cc:348] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:10.916450 25053 raft_consensus.cc:729] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5743a96978464389348dd954b48c96, State: Initialized, Role: FOLLOWER
I20240503 02:06:10.917145 25053 consensus_queue.cc:260] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:10.918797 25053 ts_tablet_manager.cc:1398] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:06:10.919348 25053 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96: Bootstrap starting.
I20240503 02:06:10.931454 25053 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:10.932088 25053 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96: Bootstrap complete.
I20240503 02:06:10.932349 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:40993
I20240503 02:06:10.932452 25114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:40993 every 8 connection(s)
I20240503 02:06:10.932618 25053 ts_tablet_manager.cc:1367] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 02:06:10.934475 25053 raft_consensus.cc:348] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:10.934974 25053 raft_consensus.cc:729] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5743a96978464389348dd954b48c96, State: Initialized, Role: FOLLOWER
I20240503 02:06:10.935592 25053 consensus_queue.cc:260] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
W20240503 02:06:10.936182 25116 tablet.cc:2369] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:06:10.937453 25053 ts_tablet_manager.cc:1398] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96: Time spent starting tablet: real 0.004s user 0.006s sys 0.001s
I20240503 02:06:10.936795 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:10.948549 25119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:10.953840 25120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:10.956208 19960 server_base.cc:1034] running on GCE node
W20240503 02:06:10.957159 25122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:10.958096 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:10.958314 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:10.958489 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701970958472 us; error 0 us; skew 500 ppm
I20240503 02:06:10.959113 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:10.960216 25115 heartbeater.cc:347] Connected to a master server at 127.19.126.62:37105
I20240503 02:06:10.960647 25115 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:10.961341 25115 heartbeater.cc:511] Master 127.19.126.62:37105 requested a full tablet report, sending...
I20240503 02:06:10.961755 19960 webserver.cc:460] Webserver started at http://127.19.126.2:36517/ using document root <none> and password file <none>
I20240503 02:06:10.962234 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:10.962400 19960 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 02:06:10.965072 24622 ts_manager.cc:194] Re-registered known tserver with Master: 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993)
I20240503 02:06:10.966670 19960 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:06:10.970641 25127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:10.971457 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240503 02:06:10.971457 24622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54248
I20240503 02:06:10.972028 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "665eb3bd61e54938b4e1674df80f52a8"
format_stamp: "Formatted at 2024-05-03 02:06:02 on dist-test-slave-8rwr"
I20240503 02:06:10.972421 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:10.974102 25115 heartbeater.cc:503] Master 127.19.126.62:37105 was elected leader, sending a full tablet report...
I20240503 02:06:10.985586 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:10.986651 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:10.987947 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:11.010643 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240503 02:06:11.010919 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.021s user 0.001s sys 0.000s
I20240503 02:06:11.015038 25134 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap starting.
I20240503 02:06:11.024860 19960 ts_tablet_manager.cc:590] Registered 4 tablets
I20240503 02:06:11.025089 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.014s user 0.011s sys 0.000s
I20240503 02:06:11.025543 25134 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:06:11.026371 25134 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap complete.
I20240503 02:06:11.026880 25134 ts_tablet_manager.cc:1367] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240503 02:06:11.028681 25134 raft_consensus.cc:348] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.029414 25134 raft_consensus.cc:729] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665eb3bd61e54938b4e1674df80f52a8, State: Initialized, Role: FOLLOWER
I20240503 02:06:11.030082 25134 consensus_queue.cc:260] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.032786 25134 ts_tablet_manager.cc:1398] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:06:11.033506 25134 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap starting.
I20240503 02:06:11.044135 25134 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:11.045116 25134 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap complete.
I20240503 02:06:11.045912 25134 ts_tablet_manager.cc:1367] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.011s
I20240503 02:06:11.048691 25134 raft_consensus.cc:348] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.049616 25134 raft_consensus.cc:729] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665eb3bd61e54938b4e1674df80f52a8, State: Initialized, Role: FOLLOWER
I20240503 02:06:11.050465 25134 consensus_queue.cc:260] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.052945 25134 ts_tablet_manager.cc:1398] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:06:11.053902 25134 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap starting.
I20240503 02:06:11.068761 25134 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:11.069748 25134 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap complete.
I20240503 02:06:11.070555 25134 ts_tablet_manager.cc:1367] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20240503 02:06:11.072137 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:43059
I20240503 02:06:11.072264 25194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:43059 every 8 connection(s)
I20240503 02:06:11.073268 25134 raft_consensus.cc:348] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.074033 25134 raft_consensus.cc:729] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665eb3bd61e54938b4e1674df80f52a8, State: Initialized, Role: FOLLOWER
I20240503 02:06:11.074713 25134 consensus_queue.cc:260] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.076862 25134 ts_tablet_manager.cc:1398] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20240503 02:06:11.077188 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:06:11.077689 25134 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap starting.
W20240503 02:06:11.087960 25199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:11.089583 25195 heartbeater.cc:347] Connected to a master server at 127.19.126.62:37105
I20240503 02:06:11.090143 25195 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:11.091255 25195 heartbeater.cc:511] Master 127.19.126.62:37105 requested a full tablet report, sending...
I20240503 02:06:11.096741 24622 ts_manager.cc:194] Re-registered known tserver with Master: 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059)
W20240503 02:06:11.086005 25198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:11.098794 19960 server_base.cc:1034] running on GCE node
W20240503 02:06:11.100531 25201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:11.100665 25134 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:06:11.101552 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:06:11.101763 25134 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: Bootstrap complete.
W20240503 02:06:11.101773 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:11.102275 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701971102258 us; error 0 us; skew 500 ppm
I20240503 02:06:11.102593 25134 ts_tablet_manager.cc:1367] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.006s
I20240503 02:06:11.102867 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:11.103124 24622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54260
I20240503 02:06:11.105034 25134 raft_consensus.cc:348] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.105846 25134 raft_consensus.cc:729] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665eb3bd61e54938b4e1674df80f52a8, State: Initialized, Role: FOLLOWER
I20240503 02:06:11.106439 25195 heartbeater.cc:503] Master 127.19.126.62:37105 was elected leader, sending a full tablet report...
I20240503 02:06:11.106504 25134 consensus_queue.cc:260] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.108049 19960 webserver.cc:460] Webserver started at http://127.19.126.3:40387/ using document root <none> and password file <none>
I20240503 02:06:11.108541 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:11.108713 19960 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 02:06:11.109091 25134 ts_tablet_manager.cc:1398] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240503 02:06:11.113261 19960 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 02:06:11.118448 25208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:11.146601 19960 fs_manager.cc:725] Time spent opening block manager: real 0.031s user 0.005s sys 0.000s
I20240503 02:06:11.147398 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "fa8da70b70624f94b60257932988054a"
format_stamp: "Formatted at 2024-05-03 02:06:02 on dist-test-slave-8rwr"
I20240503 02:06:11.147987 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:11.163204 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:11.164605 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:11.166059 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:11.191650 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240503 02:06:11.191965 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.023s user 0.001s sys 0.000s
I20240503 02:06:11.197726 25215 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a: Bootstrap starting.
I20240503 02:06:11.207204 19960 ts_tablet_manager.cc:590] Registered 4 tablets
I20240503 02:06:11.207547 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.015s user 0.004s sys 0.009s
I20240503 02:06:11.209847 25215 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:11.210522 25215 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a: Bootstrap complete.
I20240503 02:06:11.211158 25215 ts_tablet_manager.cc:1367] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20240503 02:06:11.213567 25215 raft_consensus.cc:348] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.214527 25215 raft_consensus.cc:729] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8da70b70624f94b60257932988054a, State: Initialized, Role: FOLLOWER
I20240503 02:06:11.215314 25215 consensus_queue.cc:260] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.219040 25215 ts_tablet_manager.cc:1398] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240503 02:06:11.219746 25215 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: Bootstrap starting.
I20240503 02:06:11.232053 25215 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:06:11.232740 25215 tablet_bootstrap.cc:492] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: Bootstrap complete.
I20240503 02:06:11.233354 25215 ts_tablet_manager.cc:1367] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240503 02:06:11.235396 25215 raft_consensus.cc:348] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.236181 25215 raft_consensus.cc:729] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8da70b70624f94b60257932988054a, State: Initialized, Role: FOLLOWER
I20240503 02:06:11.236915 25215 consensus_queue.cc:260] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.238656 25215 ts_tablet_manager.cc:1398] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:06:11.239233 25215 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Bootstrap starting.
I20240503 02:06:11.251322 25215 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:06:11.252321 25215 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Bootstrap complete.
I20240503 02:06:11.252990 25215 ts_tablet_manager.cc:1367] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20240503 02:06:11.254972 25215 raft_consensus.cc:348] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.255748 25215 raft_consensus.cc:729] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8da70b70624f94b60257932988054a, State: Initialized, Role: FOLLOWER
I20240503 02:06:11.256408 25215 consensus_queue.cc:260] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.258081 25215 ts_tablet_manager.cc:1398] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:06:11.258725 25215 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a: Bootstrap starting.
I20240503 02:06:11.267050 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:43765
I20240503 02:06:11.267292 25275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:43765 every 8 connection(s)
I20240503 02:06:11.269578 25215 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:11.270195 25215 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a: Bootstrap complete.
I20240503 02:06:11.270750 25215 ts_tablet_manager.cc:1367] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 02:06:11.274739 25215 raft_consensus.cc:348] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.275683 25215 raft_consensus.cc:729] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8da70b70624f94b60257932988054a, State: Initialized, Role: FOLLOWER
I20240503 02:06:11.276615 25215 consensus_queue.cc:260] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:11.278864 25215 ts_tablet_manager.cc:1398] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 02:06:11.283762 25277 heartbeater.cc:347] Connected to a master server at 127.19.126.62:37105
I20240503 02:06:11.284202 25277 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:11.284933 25277 heartbeater.cc:511] Master 127.19.126.62:37105 requested a full tablet report, sending...
I20240503 02:06:11.288332 24622 ts_manager.cc:194] Re-registered known tserver with Master: fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:11.293094 24622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54272
I20240503 02:06:11.295400 25277 heartbeater.cc:503] Master 127.19.126.62:37105 was elected leader, sending a full tablet report...
W20240503 02:06:11.553983 24672 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica fa8da70b70624f94b60257932988054a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } }
W20240503 02:06:11.557586 24672 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0e73a484b6e9487ebd53af9983a857b0, TS 5b5743a96978464389348dd954b48c96 -> TS 623fa4c949e845aea8dcd2238e37c99a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
W20240503 02:06:11.558010 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0e73a484b6e9487ebd53af9983a857b0, TS 5b5743a96978464389348dd954b48c96 -> TS 623fa4c949e845aea8dcd2238e37c99a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
W20240503 02:06:11.561051 24672 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4db6be3ede84405686653fa4e6d04075, TS 665eb3bd61e54938b4e1674df80f52a8 -> TS 623fa4c949e845aea8dcd2238e37c99a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
W20240503 02:06:11.561477 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4db6be3ede84405686653fa4e6d04075, TS 665eb3bd61e54938b4e1674df80f52a8 -> TS 623fa4c949e845aea8dcd2238e37c99a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
W20240503 02:06:11.573941 24672 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ef0a9169ad3c42a380f97acda754f322, TS fa8da70b70624f94b60257932988054a -> TS 623fa4c949e845aea8dcd2238e37c99a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
W20240503 02:06:11.574337 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ef0a9169ad3c42a380f97acda754f322, TS fa8da70b70624f94b60257932988054a -> TS 623fa4c949e845aea8dcd2238e37c99a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.104856 25283 raft_consensus.cc:482] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:12.105417 25283 raft_consensus.cc:504] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.107439 25283 leader_election.cc:288] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:12.127002 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db6be3ede84405686653fa4e6d04075" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a" is_pre_election: true
I20240503 02:06:12.127158 25169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db6be3ede84405686653fa4e6d04075" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8" is_pre_election: true
I20240503 02:06:12.127908 25250 raft_consensus.cc:2396] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b5743a96978464389348dd954b48c96 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 02:06:12.127846 25169 raft_consensus.cc:2454] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b5743a96978464389348dd954b48c96 in term 1.
I20240503 02:06:12.128993 25049 leader_election.cc:302] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b5743a96978464389348dd954b48c96, 665eb3bd61e54938b4e1674df80f52a8; no voters: fa8da70b70624f94b60257932988054a
I20240503 02:06:12.129621 25283 raft_consensus.cc:2781] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:06:12.129990 25283 raft_consensus.cc:482] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:12.130249 25283 raft_consensus.cc:3037] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:12.135453 25283 raft_consensus.cc:504] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.136838 25283 leader_election.cc:288] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 2 election: Requested vote from peers 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:12.137805 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db6be3ede84405686653fa4e6d04075" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a"
I20240503 02:06:12.137914 25169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db6be3ede84405686653fa4e6d04075" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8"
I20240503 02:06:12.138275 25250 raft_consensus.cc:3037] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:12.138412 25169 raft_consensus.cc:3037] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:12.143620 25250 raft_consensus.cc:2396] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5b5743a96978464389348dd954b48c96 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 02:06:12.143731 25169 raft_consensus.cc:2454] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b5743a96978464389348dd954b48c96 in term 2.
I20240503 02:06:12.144644 25049 leader_election.cc:302] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b5743a96978464389348dd954b48c96, 665eb3bd61e54938b4e1674df80f52a8; no voters: fa8da70b70624f94b60257932988054a
I20240503 02:06:12.145236 25283 raft_consensus.cc:2781] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:12.146287 25283 raft_consensus.cc:686] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 2 LEADER]: Becoming Leader. State: Replica: 5b5743a96978464389348dd954b48c96, State: Running, Role: LEADER
I20240503 02:06:12.147233 25283 consensus_queue.cc:237] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.154454 24623 catalog_manager.cc:5484] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 reported cstate change: term changed from 1 to 2, leader changed from fa8da70b70624f94b60257932988054a (127.19.126.3) to 5b5743a96978464389348dd954b48c96 (127.19.126.1). New cstate: current_term: 2 leader_uuid: "5b5743a96978464389348dd954b48c96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:12.189231 25283 raft_consensus.cc:482] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:12.189718 25283 raft_consensus.cc:504] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.191361 25283 leader_election.cc:288] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:12.192096 25169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300c03aa7f9e45cabea2330f78343504" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8" is_pre_election: true
I20240503 02:06:12.192276 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300c03aa7f9e45cabea2330f78343504" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a" is_pre_election: true
I20240503 02:06:12.192873 25169 raft_consensus.cc:2396] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b5743a96978464389348dd954b48c96 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 02:06:12.192835 25250 raft_consensus.cc:2454] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b5743a96978464389348dd954b48c96 in term 1.
I20240503 02:06:12.193857 25048 leader_election.cc:302] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b5743a96978464389348dd954b48c96, fa8da70b70624f94b60257932988054a; no voters:
I20240503 02:06:12.194489 25283 raft_consensus.cc:2781] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:06:12.194733 25283 raft_consensus.cc:482] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:12.194938 25283 raft_consensus.cc:3037] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:12.199748 25283 raft_consensus.cc:504] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.201159 25283 leader_election.cc:288] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 2 election: Requested vote from peers 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:12.201793 25169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300c03aa7f9e45cabea2330f78343504" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8"
I20240503 02:06:12.202054 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300c03aa7f9e45cabea2330f78343504" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a"
I20240503 02:06:12.202239 25169 raft_consensus.cc:3037] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:12.202533 25250 raft_consensus.cc:3037] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:12.206876 25250 raft_consensus.cc:2454] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b5743a96978464389348dd954b48c96 in term 2.
I20240503 02:06:12.207525 25169 raft_consensus.cc:2396] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5b5743a96978464389348dd954b48c96 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 02:06:12.207701 25048 leader_election.cc:302] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b5743a96978464389348dd954b48c96, fa8da70b70624f94b60257932988054a; no voters:
I20240503 02:06:12.208426 25283 raft_consensus.cc:2781] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:12.208739 25283 raft_consensus.cc:686] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 2 LEADER]: Becoming Leader. State: Replica: 5b5743a96978464389348dd954b48c96, State: Running, Role: LEADER
I20240503 02:06:12.209462 25283 consensus_queue.cc:237] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.216097 24623 catalog_manager.cc:5484] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 reported cstate change: term changed from 1 to 2, leader changed from 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2) to 5b5743a96978464389348dd954b48c96 (127.19.126.1). New cstate: current_term: 2 leader_uuid: "5b5743a96978464389348dd954b48c96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:12.239265 25295 raft_consensus.cc:482] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:12.239683 25295 raft_consensus.cc:504] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.241726 25295 leader_election.cc:288] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:12.254688 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a9169ad3c42a380f97acda754f322" candidate_uuid: "665eb3bd61e54938b4e1674df80f52a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5b5743a96978464389348dd954b48c96" is_pre_election: true
I20240503 02:06:12.254659 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a9169ad3c42a380f97acda754f322" candidate_uuid: "665eb3bd61e54938b4e1674df80f52a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a" is_pre_election: true
I20240503 02:06:12.255594 25088 raft_consensus.cc:2454] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 665eb3bd61e54938b4e1674df80f52a8 in term 1.
I20240503 02:06:12.255611 25250 raft_consensus.cc:2454] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 665eb3bd61e54938b4e1674df80f52a8 in term 1.
I20240503 02:06:12.256855 25129 leader_election.cc:302] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 665eb3bd61e54938b4e1674df80f52a8, fa8da70b70624f94b60257932988054a; no voters:
I20240503 02:06:12.257550 25295 raft_consensus.cc:2781] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:06:12.257853 25295 raft_consensus.cc:482] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:12.258087 25295 raft_consensus.cc:3037] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:12.262845 25295 raft_consensus.cc:504] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.264214 25295 leader_election.cc:288] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [CANDIDATE]: Term 2 election: Requested vote from peers 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:12.265041 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a9169ad3c42a380f97acda754f322" candidate_uuid: "665eb3bd61e54938b4e1674df80f52a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5b5743a96978464389348dd954b48c96"
I20240503 02:06:12.265115 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a9169ad3c42a380f97acda754f322" candidate_uuid: "665eb3bd61e54938b4e1674df80f52a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a"
I20240503 02:06:12.265678 25088 raft_consensus.cc:3037] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:12.265769 25250 raft_consensus.cc:3037] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:12.270000 25250 raft_consensus.cc:2454] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 665eb3bd61e54938b4e1674df80f52a8 in term 2.
I20240503 02:06:12.270452 25088 raft_consensus.cc:2454] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 665eb3bd61e54938b4e1674df80f52a8 in term 2.
I20240503 02:06:12.270843 25129 leader_election.cc:302] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 665eb3bd61e54938b4e1674df80f52a8, fa8da70b70624f94b60257932988054a; no voters:
I20240503 02:06:12.271510 25295 raft_consensus.cc:2781] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:12.272665 25295 raft_consensus.cc:686] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 LEADER]: Becoming Leader. State: Replica: 665eb3bd61e54938b4e1674df80f52a8, State: Running, Role: LEADER
I20240503 02:06:12.273396 25295 consensus_queue.cc:237] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.279795 24623 catalog_manager.cc:5484] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "665eb3bd61e54938b4e1674df80f52a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:12.316978 25283 raft_consensus.cc:482] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:12.317463 25283 raft_consensus.cc:504] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.319090 25283 leader_election.cc:288] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:12.319726 25169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73a484b6e9487ebd53af9983a857b0" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8" is_pre_election: true
I20240503 02:06:12.320003 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73a484b6e9487ebd53af9983a857b0" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a" is_pre_election: true
I20240503 02:06:12.320286 25169 raft_consensus.cc:2454] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b5743a96978464389348dd954b48c96 in term 1.
I20240503 02:06:12.320722 25250 raft_consensus.cc:2396] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b5743a96978464389348dd954b48c96 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 02:06:12.321138 25049 leader_election.cc:302] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b5743a96978464389348dd954b48c96, 665eb3bd61e54938b4e1674df80f52a8; no voters:
I20240503 02:06:12.321874 25283 raft_consensus.cc:2781] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:06:12.322193 25283 raft_consensus.cc:482] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:12.322402 25283 raft_consensus.cc:3037] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:12.326540 25283 raft_consensus.cc:504] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.327919 25283 leader_election.cc:288] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 2 election: Requested vote from peers 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059), fa8da70b70624f94b60257932988054a (127.19.126.3:43765)
I20240503 02:06:12.328549 25169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73a484b6e9487ebd53af9983a857b0" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8"
I20240503 02:06:12.329089 25169 raft_consensus.cc:3037] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:12.329126 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73a484b6e9487ebd53af9983a857b0" candidate_uuid: "5b5743a96978464389348dd954b48c96" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa8da70b70624f94b60257932988054a"
I20240503 02:06:12.329710 25250 raft_consensus.cc:3037] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:12.333441 25169 raft_consensus.cc:2454] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b5743a96978464389348dd954b48c96 in term 2.
I20240503 02:06:12.334301 25250 raft_consensus.cc:2396] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5b5743a96978464389348dd954b48c96 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 02:06:12.334429 25049 leader_election.cc:302] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b5743a96978464389348dd954b48c96, 665eb3bd61e54938b4e1674df80f52a8; no voters:
I20240503 02:06:12.335244 25283 raft_consensus.cc:2781] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:12.335716 25283 raft_consensus.cc:686] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 2 LEADER]: Becoming Leader. State: Replica: 5b5743a96978464389348dd954b48c96, State: Running, Role: LEADER
I20240503 02:06:12.336385 25283 consensus_queue.cc:237] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.342967 24623 catalog_manager.cc:5484] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 reported cstate change: term changed from 1 to 2, leader changed from fa8da70b70624f94b60257932988054a (127.19.126.3) to 5b5743a96978464389348dd954b48c96 (127.19.126.1). New cstate: current_term: 2 leader_uuid: "5b5743a96978464389348dd954b48c96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:12.407430 25302 raft_consensus.cc:482] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:12.407969 25302 raft_consensus.cc:504] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.410185 25302 leader_election.cc:288] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993), 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059)
I20240503 02:06:12.420231 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db6be3ede84405686653fa4e6d04075" candidate_uuid: "fa8da70b70624f94b60257932988054a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5b5743a96978464389348dd954b48c96" is_pre_election: true
I20240503 02:06:12.420744 25169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db6be3ede84405686653fa4e6d04075" candidate_uuid: "fa8da70b70624f94b60257932988054a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8" is_pre_election: true
I20240503 02:06:12.421425 25169 raft_consensus.cc:2454] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa8da70b70624f94b60257932988054a in term 2.
I20240503 02:06:12.422338 25211 leader_election.cc:302] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 665eb3bd61e54938b4e1674df80f52a8, fa8da70b70624f94b60257932988054a; no voters: 5b5743a96978464389348dd954b48c96
I20240503 02:06:12.422969 25302 raft_consensus.cc:2781] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:06:12.423235 25302 raft_consensus.cc:482] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:12.423480 25302 raft_consensus.cc:3037] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:12.428229 25302 raft_consensus.cc:504] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.429587 25302 leader_election.cc:288] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [CANDIDATE]: Term 3 election: Requested vote from peers 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993), 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059)
I20240503 02:06:12.430469 25169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db6be3ede84405686653fa4e6d04075" candidate_uuid: "fa8da70b70624f94b60257932988054a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "665eb3bd61e54938b4e1674df80f52a8"
I20240503 02:06:12.430428 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db6be3ede84405686653fa4e6d04075" candidate_uuid: "fa8da70b70624f94b60257932988054a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5b5743a96978464389348dd954b48c96"
I20240503 02:06:12.431233 25169 raft_consensus.cc:3037] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:12.435421 25169 raft_consensus.cc:2454] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa8da70b70624f94b60257932988054a in term 3.
I20240503 02:06:12.436162 25211 leader_election.cc:302] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 665eb3bd61e54938b4e1674df80f52a8, fa8da70b70624f94b60257932988054a; no voters: 5b5743a96978464389348dd954b48c96
I20240503 02:06:12.436751 25302 raft_consensus.cc:2781] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:06:12.437785 25302 raft_consensus.cc:686] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 3 LEADER]: Becoming Leader. State: Replica: fa8da70b70624f94b60257932988054a, State: Running, Role: LEADER
I20240503 02:06:12.438351 25302 consensus_queue.cc:237] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.444339 24623 catalog_manager.cc:5484] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a reported cstate change: term changed from 2 to 3, leader changed from 5b5743a96978464389348dd954b48c96 (127.19.126.1) to fa8da70b70624f94b60257932988054a (127.19.126.3). New cstate: current_term: 3 leader_uuid: "fa8da70b70624f94b60257932988054a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:12.555857 25169 raft_consensus.cc:1226] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 3 FOLLOWER]: Rejecting Update request from peer 5b5743a96978464389348dd954b48c96 for earlier term 2. Current term is 3. Ops: []
I20240503 02:06:12.557107 25283 consensus_queue.cc:1046] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:12.558029 25288 raft_consensus.cc:3032] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:06:12.558346 25288 raft_consensus.cc:729] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5b5743a96978464389348dd954b48c96, State: Running, Role: LEADER
I20240503 02:06:12.558941 25288 consensus_queue.cc:260] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.559863 25288 raft_consensus.cc:3037] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:06:12.591490 24672 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240503 02:06:12.594789 24672 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 300c03aa7f9e45cabea2330f78343504, TS fa8da70b70624f94b60257932988054a -> TS 623fa4c949e845aea8dcd2238e37c99a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
W20240503 02:06:12.594992 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 300c03aa7f9e45cabea2330f78343504, TS fa8da70b70624f94b60257932988054a -> TS 623fa4c949e845aea8dcd2238e37c99a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
W20240503 02:06:12.597872 24672 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4db6be3ede84405686653fa4e6d04075, TS 5b5743a96978464389348dd954b48c96 -> TS 623fa4c949e845aea8dcd2238e37c99a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
W20240503 02:06:12.598083 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4db6be3ede84405686653fa4e6d04075, TS 5b5743a96978464389348dd954b48c96 -> TS 623fa4c949e845aea8dcd2238e37c99a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
W20240503 02:06:12.600816 24672 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0e73a484b6e9487ebd53af9983a857b0, TS 665eb3bd61e54938b4e1674df80f52a8 -> TS 623fa4c949e845aea8dcd2238e37c99a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
W20240503 02:06:12.601025 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0e73a484b6e9487ebd53af9983a857b0, TS 665eb3bd61e54938b4e1674df80f52a8 -> TS 623fa4c949e845aea8dcd2238e37c99a move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } }
I20240503 02:06:12.652437 25169 raft_consensus.cc:1261] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Refusing update from remote peer 5b5743a96978464389348dd954b48c96: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240503 02:06:12.652797 25169 pending_rounds.cc:76] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Aborting all ops after (but not including) 0
I20240503 02:06:12.653020 25169 pending_rounds.cc:97] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240503 02:06:12.653331 25169 raft_consensus.cc:2866] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:06:12.654549 25288 consensus_queue.cc:1035] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:12.665035 25250 raft_consensus.cc:1261] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Refusing update from remote peer 5b5743a96978464389348dd954b48c96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 02:06:12.666764 25310 consensus_queue.cc:1035] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:06:12.720973 25088 raft_consensus.cc:1261] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Refusing update from remote peer 665eb3bd61e54938b4e1674df80f52a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:12.722293 25295 consensus_queue.cc:1035] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:12.733336 25250 raft_consensus.cc:1261] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Refusing update from remote peer 665eb3bd61e54938b4e1674df80f52a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:12.735635 25315 consensus_queue.cc:1035] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:06:12.767705 25169 raft_consensus.cc:1261] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Refusing update from remote peer 5b5743a96978464389348dd954b48c96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 02:06:12.768805 25288 consensus_queue.cc:1035] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:12.777652 25250 raft_consensus.cc:1261] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Refusing update from remote peer 5b5743a96978464389348dd954b48c96: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240503 02:06:12.778043 25250 pending_rounds.cc:76] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Aborting all ops after (but not including) 0
I20240503 02:06:12.778276 25250 pending_rounds.cc:97] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240503 02:06:12.778518 25250 raft_consensus.cc:2866] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:06:12.781121 25288 consensus_queue.cc:1035] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:12.859616 25088 raft_consensus.cc:1261] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 3 FOLLOWER]: Refusing update from remote peer fa8da70b70624f94b60257932988054a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 02:06:12.860956 25302 consensus_queue.cc:1035] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:12.861610 25302 consensus_queue.cc:1225] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [LEADER]: Peer 5b5743a96978464389348dd954b48c96 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 02:06:12.863826 25088 pending_rounds.cc:76] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: Aborting all ops after (but not including) 0
I20240503 02:06:12.864058 25088 pending_rounds.cc:97] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240503 02:06:12.864289 25088 raft_consensus.cc:2866] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 02:06:12.870957 25169 raft_consensus.cc:1261] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 3 FOLLOWER]: Refusing update from remote peer fa8da70b70624f94b60257932988054a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 02:06:12.872185 25302 consensus_queue.cc:1035] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [LEADER]: Connected to new peer: Peer: permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:13.610775 25088 consensus_queue.cc:237] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } attrs { replace: true } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } }
I20240503 02:06:13.616492 25169 raft_consensus.cc:1261] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Refusing update from remote peer 5b5743a96978464389348dd954b48c96: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:13.616647 25250 raft_consensus.cc:1261] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Refusing update from remote peer 5b5743a96978464389348dd954b48c96: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:13.618716 25283 consensus_queue.cc:1035] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:06:13.620409 25310 consensus_queue.cc:1035] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:13.625598 25283 raft_consensus.cc:2932] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } attrs { replace: true } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } } }
I20240503 02:06:13.627226 25250 raft_consensus.cc:2932] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } attrs { replace: true } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } } }
I20240503 02:06:13.631912 25169 raft_consensus.cc:2932] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } attrs { replace: true } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } } }
I20240503 02:06:13.644605 25088 consensus_queue.cc:237] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } attrs { replace: true } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } }
I20240503 02:06:13.646319 24623 catalog_manager.cc:5484] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 reported cstate change: config changed from index -1 to 2, NON_VOTER 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4) added. New cstate: current_term: 2 leader_uuid: "5b5743a96978464389348dd954b48c96" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:06:13.648295 25050 consensus_peers.cc:485] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 -> Peer 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4:34713): Couldn't send request to peer 623fa4c949e845aea8dcd2238e37c99a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 300c03aa7f9e45cabea2330f78343504. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:13.651501 25169 raft_consensus.cc:1261] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Refusing update from remote peer 5b5743a96978464389348dd954b48c96: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:13.651583 25249 raft_consensus.cc:1261] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Refusing update from remote peer 5b5743a96978464389348dd954b48c96: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:13.653012 25283 consensus_queue.cc:1035] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:13.655892 25310 consensus_queue.cc:1035] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:06:13.660559 25050 consensus_peers.cc:485] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 -> Peer 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4:34713): Couldn't send request to peer 623fa4c949e845aea8dcd2238e37c99a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e73a484b6e9487ebd53af9983a857b0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:13.664063 25310 raft_consensus.cc:2932] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } attrs { replace: true } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } } }
I20240503 02:06:13.666419 25169 raft_consensus.cc:2932] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } attrs { replace: true } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } } }
I20240503 02:06:13.671429 25250 raft_consensus.cc:2932] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } attrs { replace: true } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } } }
I20240503 02:06:13.678930 24623 catalog_manager.cc:5484] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 reported cstate change: config changed from index -1 to 2, NON_VOTER 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4) added. New cstate: current_term: 2 leader_uuid: "5b5743a96978464389348dd954b48c96" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } attrs { replace: true } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } } }
I20240503 02:06:13.680922 25169 consensus_queue.cc:237] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } attrs { replace: true } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } }
I20240503 02:06:13.689085 25088 raft_consensus.cc:1261] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Refusing update from remote peer 665eb3bd61e54938b4e1674df80f52a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:06:13.689468 25250 raft_consensus.cc:1261] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Refusing update from remote peer 665eb3bd61e54938b4e1674df80f52a8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:06:13.690822 25321 consensus_queue.cc:1035] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:06:13.691846 25322 consensus_queue.cc:1035] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:06:13.700024 25321 raft_consensus.cc:2932] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } attrs { replace: true } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } } }
I20240503 02:06:13.702607 25249 raft_consensus.cc:2932] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } attrs { replace: true } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } } }
I20240503 02:06:13.704239 25088 raft_consensus.cc:2932] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } attrs { replace: true } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } } }
W20240503 02:06:13.707140 25131 consensus_peers.cc:485] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 -> Peer 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4:34713): Couldn't send request to peer 623fa4c949e845aea8dcd2238e37c99a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef0a9169ad3c42a380f97acda754f322. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:13.719004 24622 catalog_manager.cc:5484] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a reported cstate change: config changed from index -1 to 3, NON_VOTER 623fa4c949e845aea8dcd2238e37c99a (127.19.126.4) added. New cstate: current_term: 2 leader_uuid: "665eb3bd61e54938b4e1674df80f52a8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } attrs { replace: true } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } } }
I20240503 02:06:14.100072 25330 ts_tablet_manager.cc:902] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a: Initiating tablet copy from peer 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993)
I20240503 02:06:14.104350 25330 tablet_copy_client.cc:308] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.126.1:40993
I20240503 02:06:14.119989 25335 ts_tablet_manager.cc:902] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a: Initiating tablet copy from peer 665eb3bd61e54938b4e1674df80f52a8 (127.19.126.2:43059)
I20240503 02:06:14.131920 25335 tablet_copy_client.cc:308] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.126.2:43059
I20240503 02:06:14.131834 25099 tablet_copy_service.cc:140] P 5b5743a96978464389348dd954b48c96: Received BeginTabletCopySession request for tablet 0e73a484b6e9487ebd53af9983a857b0 from peer 623fa4c949e845aea8dcd2238e37c99a ({username='slave'} at 127.0.0.1:40548)
I20240503 02:06:14.132772 25099 tablet_copy_service.cc:161] P 5b5743a96978464389348dd954b48c96: Beginning new tablet copy session on tablet 0e73a484b6e9487ebd53af9983a857b0 from peer 623fa4c949e845aea8dcd2238e37c99a at {username='slave'} at 127.0.0.1:40548: session id = 623fa4c949e845aea8dcd2238e37c99a-0e73a484b6e9487ebd53af9983a857b0
I20240503 02:06:14.140532 25099 tablet_copy_source_session.cc:205] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:06:14.142349 25341 ts_tablet_manager.cc:902] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a: Initiating tablet copy from peer 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993)
I20240503 02:06:14.144032 25330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e73a484b6e9487ebd53af9983a857b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:14.168807 25179 tablet_copy_service.cc:140] P 665eb3bd61e54938b4e1674df80f52a8: Received BeginTabletCopySession request for tablet ef0a9169ad3c42a380f97acda754f322 from peer 623fa4c949e845aea8dcd2238e37c99a ({username='slave'} at 127.0.0.1:34038)
I20240503 02:06:14.169418 25179 tablet_copy_service.cc:161] P 665eb3bd61e54938b4e1674df80f52a8: Beginning new tablet copy session on tablet ef0a9169ad3c42a380f97acda754f322 from peer 623fa4c949e845aea8dcd2238e37c99a at {username='slave'} at 127.0.0.1:34038: session id = 623fa4c949e845aea8dcd2238e37c99a-ef0a9169ad3c42a380f97acda754f322
I20240503 02:06:14.170783 25330 tablet_copy_client.cc:783] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a: tablet copy: Starting download of 0 data blocks...
I20240503 02:06:14.171859 25330 tablet_copy_client.cc:647] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:14.177605 25179 tablet_copy_source_session.cc:205] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:06:14.176862 25330 tablet_copy_client.cc:523] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:14.182600 25335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef0a9169ad3c42a380f97acda754f322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:14.187294 25330 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a: Bootstrap starting.
I20240503 02:06:14.193982 25341 tablet_copy_client.cc:308] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.126.1:40993
I20240503 02:06:14.195722 25099 tablet_copy_service.cc:140] P 5b5743a96978464389348dd954b48c96: Received BeginTabletCopySession request for tablet 300c03aa7f9e45cabea2330f78343504 from peer 623fa4c949e845aea8dcd2238e37c99a ({username='slave'} at 127.0.0.1:40548)
I20240503 02:06:14.196214 25099 tablet_copy_service.cc:161] P 5b5743a96978464389348dd954b48c96: Beginning new tablet copy session on tablet 300c03aa7f9e45cabea2330f78343504 from peer 623fa4c949e845aea8dcd2238e37c99a at {username='slave'} at 127.0.0.1:40548: session id = 623fa4c949e845aea8dcd2238e37c99a-300c03aa7f9e45cabea2330f78343504
I20240503 02:06:14.198275 25335 tablet_copy_client.cc:783] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a: tablet copy: Starting download of 0 data blocks...
I20240503 02:06:14.198891 25335 tablet_copy_client.cc:647] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:14.203433 25335 tablet_copy_client.cc:523] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:14.203775 25099 tablet_copy_source_session.cc:205] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:06:14.207253 25341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300c03aa7f9e45cabea2330f78343504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:14.213232 25335 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a: Bootstrap starting.
I20240503 02:06:14.230774 25341 tablet_copy_client.cc:783] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a: tablet copy: Starting download of 0 data blocks...
I20240503 02:06:14.231467 25341 tablet_copy_client.cc:647] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:14.235317 25330 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:14.236250 25330 tablet_bootstrap.cc:492] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a: Bootstrap complete.
I20240503 02:06:14.236994 25330 ts_tablet_manager.cc:1367] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a: Time spent bootstrapping tablet: real 0.050s user 0.019s sys 0.028s
I20240503 02:06:14.238152 25335 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:14.239106 25335 tablet_bootstrap.cc:492] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a: Bootstrap complete.
I20240503 02:06:14.239310 25341 tablet_copy_client.cc:523] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:14.239763 25335 ts_tablet_manager.cc:1367] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a: Time spent bootstrapping tablet: real 0.027s user 0.025s sys 0.000s
I20240503 02:06:14.239950 25330 raft_consensus.cc:348] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } attrs { replace: true } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } }
I20240503 02:06:14.240964 25330 raft_consensus.cc:729] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 623fa4c949e845aea8dcd2238e37c99a, State: Initialized, Role: LEARNER
I20240503 02:06:14.241672 25330 consensus_queue.cc:260] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } attrs { replace: true } } peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } }
I20240503 02:06:14.242533 25335 raft_consensus.cc:348] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } attrs { replace: true } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } }
I20240503 02:06:14.243456 25335 raft_consensus.cc:729] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 623fa4c949e845aea8dcd2238e37c99a, State: Initialized, Role: LEARNER
I20240503 02:06:14.244122 25335 consensus_queue.cc:260] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } attrs { replace: true } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } }
I20240503 02:06:14.249879 25341 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a: Bootstrap starting.
I20240503 02:06:14.250648 25330 ts_tablet_manager.cc:1398] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20240503 02:06:14.262368 25099 tablet_copy_service.cc:342] P 5b5743a96978464389348dd954b48c96: Request end of tablet copy session 623fa4c949e845aea8dcd2238e37c99a-0e73a484b6e9487ebd53af9983a857b0 received from {username='slave'} at 127.0.0.1:40548
I20240503 02:06:14.263034 25099 tablet_copy_service.cc:433] P 5b5743a96978464389348dd954b48c96: ending tablet copy session 623fa4c949e845aea8dcd2238e37c99a-0e73a484b6e9487ebd53af9983a857b0 on tablet 0e73a484b6e9487ebd53af9983a857b0 with peer 623fa4c949e845aea8dcd2238e37c99a
I20240503 02:06:14.269024 25335 ts_tablet_manager.cc:1398] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a: Time spent starting tablet: real 0.029s user 0.023s sys 0.001s
I20240503 02:06:14.270680 25179 tablet_copy_service.cc:342] P 665eb3bd61e54938b4e1674df80f52a8: Request end of tablet copy session 623fa4c949e845aea8dcd2238e37c99a-ef0a9169ad3c42a380f97acda754f322 received from {username='slave'} at 127.0.0.1:34038
I20240503 02:06:14.271066 25179 tablet_copy_service.cc:433] P 665eb3bd61e54938b4e1674df80f52a8: ending tablet copy session 623fa4c949e845aea8dcd2238e37c99a-ef0a9169ad3c42a380f97acda754f322 on tablet ef0a9169ad3c42a380f97acda754f322 with peer 623fa4c949e845aea8dcd2238e37c99a
I20240503 02:06:14.274091 25341 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:14.274960 25341 tablet_bootstrap.cc:492] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a: Bootstrap complete.
I20240503 02:06:14.275487 25341 ts_tablet_manager.cc:1367] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.000s
I20240503 02:06:14.277910 25341 raft_consensus.cc:348] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } attrs { replace: true } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } }
I20240503 02:06:14.278535 25341 raft_consensus.cc:729] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 623fa4c949e845aea8dcd2238e37c99a, State: Initialized, Role: LEARNER
I20240503 02:06:14.278981 25341 consensus_queue.cc:260] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b5743a96978464389348dd954b48c96" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 40993 } } peers { permanent_uuid: "665eb3bd61e54938b4e1674df80f52a8" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 43059 } } peers { permanent_uuid: "fa8da70b70624f94b60257932988054a" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 43765 } attrs { replace: true } } peers { permanent_uuid: "623fa4c949e845aea8dcd2238e37c99a" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 34713 } attrs { promote: true } }
I20240503 02:06:14.281109 25341 ts_tablet_manager.cc:1398] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 02:06:14.282586 25099 tablet_copy_service.cc:342] P 5b5743a96978464389348dd954b48c96: Request end of tablet copy session 623fa4c949e845aea8dcd2238e37c99a-300c03aa7f9e45cabea2330f78343504 received from {username='slave'} at 127.0.0.1:40548
I20240503 02:06:14.282876 25099 tablet_copy_service.cc:433] P 5b5743a96978464389348dd954b48c96: ending tablet copy session 623fa4c949e845aea8dcd2238e37c99a-300c03aa7f9e45cabea2330f78343504 on tablet 300c03aa7f9e45cabea2330f78343504 with peer 623fa4c949e845aea8dcd2238e37c99a
I20240503 02:06:14.522707 24997 raft_consensus.cc:1203] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240503 02:06:14.530026 24997 raft_consensus.cc:1203] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240503 02:06:14.566929 24997 raft_consensus.cc:1203] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 02:06:14.789680 19960 tablet_server.cc:178] TabletServer@127.19.126.1:40993 shutting down...
W20240503 02:06:14.796577 24666 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20240503 02:06:14.802147 24672 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:06:14.802457 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:06:14.808225 24672 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:06:14.809912 25128 consensus_peers.cc:485] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 -> Peer 5b5743a96978464389348dd954b48c96 (127.19.126.1:40993): Couldn't send request to peer 5b5743a96978464389348dd954b48c96. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:06:14.810235 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 02:06:14.821069 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:14.821903 19960 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:14.822525 19960 raft_consensus.cc:2229] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:14.822955 19960 raft_consensus.cc:2258] T ef0a9169ad3c42a380f97acda754f322 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:14.825176 19960 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:14.825649 19960 raft_consensus.cc:2229] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:14.826020 19960 raft_consensus.cc:2258] T 4db6be3ede84405686653fa4e6d04075 P 5b5743a96978464389348dd954b48c96 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:14.828107 19960 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:14.828534 19960 raft_consensus.cc:2229] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:06:14.829474 19960 raft_consensus.cc:2258] T 0e73a484b6e9487ebd53af9983a857b0 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:14.831399 19960 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:14.831805 19960 raft_consensus.cc:2229] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:06:14.832940 19960 raft_consensus.cc:2258] T 300c03aa7f9e45cabea2330f78343504 P 5b5743a96978464389348dd954b48c96 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:14.859042 19960 tablet_server.cc:194] TabletServer@127.19.126.1:40993 shutdown complete.
I20240503 02:06:14.873979 19960 tablet_server.cc:178] TabletServer@127.19.126.2:43059 shutting down...
W20240503 02:06:14.878351 24672 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 02:06:14.878669 24672 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 02:06:14.896764 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:14.897594 19960 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:14.898123 19960 raft_consensus.cc:2229] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:06:14.899227 19960 raft_consensus.cc:2258] T ef0a9169ad3c42a380f97acda754f322 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:14.901119 19960 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:14.901608 19960 raft_consensus.cc:2229] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:14.901957 19960 raft_consensus.cc:2258] T 4db6be3ede84405686653fa4e6d04075 P 665eb3bd61e54938b4e1674df80f52a8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:14.903535 19960 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:14.903921 19960 raft_consensus.cc:2229] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:14.904278 19960 raft_consensus.cc:2258] T 0e73a484b6e9487ebd53af9983a857b0 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:14.905997 19960 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:14.906421 19960 raft_consensus.cc:2229] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:14.906821 19960 raft_consensus.cc:2258] T 300c03aa7f9e45cabea2330f78343504 P 665eb3bd61e54938b4e1674df80f52a8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:14.929117 19960 tablet_server.cc:194] TabletServer@127.19.126.2:43059 shutdown complete.
I20240503 02:06:14.941640 19960 tablet_server.cc:178] TabletServer@127.19.126.3:43765 shutting down...
I20240503 02:06:14.958729 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:14.959231 19960 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:14.959702 19960 raft_consensus.cc:2229] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:14.960090 19960 raft_consensus.cc:2258] T ef0a9169ad3c42a380f97acda754f322 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:14.961743 19960 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:14.962208 19960 raft_consensus.cc:2229] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 3 LEADER]: Raft consensus shutting down.
I20240503 02:06:14.962863 19960 raft_consensus.cc:2258] T 4db6be3ede84405686653fa4e6d04075 P fa8da70b70624f94b60257932988054a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:14.964432 19960 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:14.964883 19960 raft_consensus.cc:2229] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:14.965246 19960 raft_consensus.cc:2258] T 0e73a484b6e9487ebd53af9983a857b0 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:14.966728 19960 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:14.967106 19960 raft_consensus.cc:2229] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:14.967447 19960 raft_consensus.cc:2258] T 300c03aa7f9e45cabea2330f78343504 P fa8da70b70624f94b60257932988054a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:14.987924 19960 tablet_server.cc:194] TabletServer@127.19.126.3:43765 shutdown complete.
I20240503 02:06:14.998920 19960 tablet_server.cc:178] TabletServer@127.19.126.4:0 shutting down...
I20240503 02:06:15.023226 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:15.023780 19960 tablet_replica.cc:330] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a: stopping tablet replica
I20240503 02:06:15.024256 19960 raft_consensus.cc:2229] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Raft consensus shutting down.
I20240503 02:06:15.024601 19960 raft_consensus.cc:2258] T ef0a9169ad3c42a380f97acda754f322 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Raft consensus is shut down!
I20240503 02:06:15.026374 19960 tablet_replica.cc:330] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a: stopping tablet replica
I20240503 02:06:15.026789 19960 raft_consensus.cc:2229] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Raft consensus shutting down.
I20240503 02:06:15.027078 19960 raft_consensus.cc:2258] T 300c03aa7f9e45cabea2330f78343504 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Raft consensus is shut down!
I20240503 02:06:15.028470 19960 tablet_replica.cc:330] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a: stopping tablet replica
I20240503 02:06:15.028842 19960 raft_consensus.cc:2229] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Raft consensus shutting down.
I20240503 02:06:15.029120 19960 raft_consensus.cc:2258] T 0e73a484b6e9487ebd53af9983a857b0 P 623fa4c949e845aea8dcd2238e37c99a [term 2 LEARNER]: Raft consensus is shut down!
I20240503 02:06:15.047669 19960 tablet_server.cc:194] TabletServer@127.19.126.4:0 shutdown complete.
I20240503 02:06:15.058259 19960 master.cc:537] Master@127.19.126.62:37105 shutting down...
I20240503 02:06:15.073166 19960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:15.073662 19960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:15.073983 19960 tablet_replica.cc:330] T 00000000000000000000000000000000 P babf6a18bafb42e4a4d5d5e9583e65a8: stopping tablet replica
I20240503 02:06:15.091009 19960 master.cc:552] Master@127.19.126.62:37105 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (12840 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20240503 02:06:15.126116 19960 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.126.62:44719
I20240503 02:06:15.127064 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:15.132345 25360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:15.133194 25361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:15.133910 25363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:15.134881 19960 server_base.cc:1034] running on GCE node
I20240503 02:06:15.136512 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:15.136723 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:15.136833 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701975136822 us; error 0 us; skew 500 ppm
I20240503 02:06:15.137241 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:15.139559 19960 webserver.cc:460] Webserver started at http://127.19.126.62:35995/ using document root <none> and password file <none>
I20240503 02:06:15.139947 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:15.140098 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:15.140311 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:15.141320 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/master-0-root/instance:
uuid: "f057e928aab0437586b59e032600170c"
format_stamp: "Formatted at 2024-05-03 02:06:15 on dist-test-slave-8rwr"
I20240503 02:06:15.145522 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 02:06:15.148573 25368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:15.149329 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 02:06:15.149596 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/master-0-root
uuid: "f057e928aab0437586b59e032600170c"
format_stamp: "Formatted at 2024-05-03 02:06:15 on dist-test-slave-8rwr"
I20240503 02:06:15.149873 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:15.181077 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:15.182247 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:15.217271 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.62:44719
I20240503 02:06:15.217388 25419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.62:44719 every 8 connection(s)
I20240503 02:06:15.220832 25420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:15.230183 25420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c: Bootstrap starting.
I20240503 02:06:15.234043 25420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:15.237852 25420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c: No bootstrap required, opened a new log
I20240503 02:06:15.239589 25420 raft_consensus.cc:348] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f057e928aab0437586b59e032600170c" member_type: VOTER }
I20240503 02:06:15.239938 25420 raft_consensus.cc:374] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:15.240119 25420 raft_consensus.cc:729] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f057e928aab0437586b59e032600170c, State: Initialized, Role: FOLLOWER
I20240503 02:06:15.240569 25420 consensus_queue.cc:260] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f057e928aab0437586b59e032600170c" member_type: VOTER }
I20240503 02:06:15.241003 25420 raft_consensus.cc:388] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:06:15.241180 25420 raft_consensus.cc:482] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:06:15.241417 25420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:15.245347 25420 raft_consensus.cc:504] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f057e928aab0437586b59e032600170c" member_type: VOTER }
I20240503 02:06:15.245792 25420 leader_election.cc:302] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f057e928aab0437586b59e032600170c; no voters:
I20240503 02:06:15.246800 25420 leader_election.cc:288] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:06:15.247186 25423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:15.248462 25423 raft_consensus.cc:686] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [term 1 LEADER]: Becoming Leader. State: Replica: f057e928aab0437586b59e032600170c, State: Running, Role: LEADER
I20240503 02:06:15.248975 25423 consensus_queue.cc:237] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f057e928aab0437586b59e032600170c" member_type: VOTER }
I20240503 02:06:15.249498 25420 sys_catalog.cc:564] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:06:15.251701 25425 sys_catalog.cc:455] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f057e928aab0437586b59e032600170c. Latest consensus state: current_term: 1 leader_uuid: "f057e928aab0437586b59e032600170c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f057e928aab0437586b59e032600170c" member_type: VOTER } }
I20240503 02:06:15.251858 25424 sys_catalog.cc:455] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f057e928aab0437586b59e032600170c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f057e928aab0437586b59e032600170c" member_type: VOTER } }
I20240503 02:06:15.252310 25424 sys_catalog.cc:458] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [sys.catalog]: This master's current role is: LEADER
I20240503 02:06:15.252281 25425 sys_catalog.cc:458] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c [sys.catalog]: This master's current role is: LEADER
I20240503 02:06:15.255815 25428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:06:15.263810 25428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:06:15.264688 19960 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:06:15.271582 25428 catalog_manager.cc:1347] Generated new cluster ID: 28dec01861354f99b2a1487b42bc2070
I20240503 02:06:15.271833 25428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:06:15.285929 25428 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:06:15.287119 25428 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:06:15.297493 25428 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c: Generated new TSK 0
I20240503 02:06:15.298014 25428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:06:15.330861 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:15.336741 25441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:15.338115 25442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:15.338825 25444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:15.339712 19960 server_base.cc:1034] running on GCE node
I20240503 02:06:15.341194 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:15.341450 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:15.341603 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701975341585 us; error 0 us; skew 500 ppm
I20240503 02:06:15.342094 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:15.344326 19960 webserver.cc:460] Webserver started at http://127.19.126.1:42207/ using document root <none> and password file <none>
I20240503 02:06:15.344839 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:15.345045 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:15.345440 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:15.346544 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root/instance:
uuid: "698037111e64432092d44f7db2edfa08"
format_stamp: "Formatted at 2024-05-03 02:06:15 on dist-test-slave-8rwr"
I20240503 02:06:15.350759 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:06:15.353932 25449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:15.354629 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:06:15.354882 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "698037111e64432092d44f7db2edfa08"
format_stamp: "Formatted at 2024-05-03 02:06:15 on dist-test-slave-8rwr"
I20240503 02:06:15.355151 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:15.394661 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:15.395704 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:15.397053 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:15.399287 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:15.399469 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:15.399684 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:15.399827 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:15.436911 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:46025
I20240503 02:06:15.437019 25511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:46025 every 8 connection(s)
I20240503 02:06:15.441515 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:15.448845 25516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:15.449236 25517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:15.452888 25512 heartbeater.cc:347] Connected to a master server at 127.19.126.62:44719
I20240503 02:06:15.453253 25512 heartbeater.cc:464] Registering TS with master...
W20240503 02:06:15.453327 25519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:15.454165 25512 heartbeater.cc:511] Master 127.19.126.62:44719 requested a full tablet report, sending...
I20240503 02:06:15.454360 19960 server_base.cc:1034] running on GCE node
I20240503 02:06:15.455155 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:15.455430 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:15.455636 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701975455617 us; error 0 us; skew 500 ppm
I20240503 02:06:15.456285 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:15.456378 25385 ts_manager.cc:194] Registered new tserver with Master: 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025)
I20240503 02:06:15.458829 25385 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42960
I20240503 02:06:15.459180 19960 webserver.cc:460] Webserver started at http://127.19.126.2:37347/ using document root <none> and password file <none>
I20240503 02:06:15.459794 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:15.460027 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:15.460335 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:15.461656 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-1-root/instance:
uuid: "a491687290b64a83b750b5506c9ba041"
format_stamp: "Formatted at 2024-05-03 02:06:15 on dist-test-slave-8rwr"
I20240503 02:06:15.465626 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:06:15.468518 25524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:15.469224 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:06:15.469532 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-1-root
uuid: "a491687290b64a83b750b5506c9ba041"
format_stamp: "Formatted at 2024-05-03 02:06:15 on dist-test-slave-8rwr"
I20240503 02:06:15.469767 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:15.485630 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:15.486668 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:15.488029 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:15.490206 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:15.490372 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:15.490602 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:15.490737 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:15.526741 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.2:38473
I20240503 02:06:15.526831 25586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.2:38473 every 8 connection(s)
I20240503 02:06:15.531160 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:15.537928 25590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:15.540536 25591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:15.542738 19960 server_base.cc:1034] running on GCE node
W20240503 02:06:15.543586 25593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:15.544461 25587 heartbeater.cc:347] Connected to a master server at 127.19.126.62:44719
I20240503 02:06:15.544661 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:06:15.544883 25587 heartbeater.cc:464] Registering TS with master...
W20240503 02:06:15.544991 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:15.545336 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701975545316 us; error 0 us; skew 500 ppm
I20240503 02:06:15.545768 25587 heartbeater.cc:511] Master 127.19.126.62:44719 requested a full tablet report, sending...
I20240503 02:06:15.545924 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:15.547698 25385 ts_manager.cc:194] Registered new tserver with Master: a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473)
I20240503 02:06:15.548756 19960 webserver.cc:460] Webserver started at http://127.19.126.3:33023/ using document root <none> and password file <none>
I20240503 02:06:15.549393 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:15.549616 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:15.549649 25385 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42976
I20240503 02:06:15.550052 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:15.551241 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-2-root/instance:
uuid: "1d724b88dae941358c33ea9e56a022db"
format_stamp: "Formatted at 2024-05-03 02:06:15 on dist-test-slave-8rwr"
I20240503 02:06:15.555292 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:06:15.558077 25598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:15.558785 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:06:15.559037 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-2-root
uuid: "1d724b88dae941358c33ea9e56a022db"
format_stamp: "Formatted at 2024-05-03 02:06:15 on dist-test-slave-8rwr"
I20240503 02:06:15.559252 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:15.567795 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:15.568755 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:15.570127 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:15.572286 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:15.572456 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:15.572688 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:15.572826 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:15.607793 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.3:35187
I20240503 02:06:15.607879 25660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.3:35187 every 8 connection(s)
I20240503 02:06:15.619910 25661 heartbeater.cc:347] Connected to a master server at 127.19.126.62:44719
I20240503 02:06:15.620255 25661 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:15.620877 25661 heartbeater.cc:511] Master 127.19.126.62:44719 requested a full tablet report, sending...
I20240503 02:06:15.622630 25385 ts_manager.cc:194] Registered new tserver with Master: 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:15.622735 19960 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012249718s
I20240503 02:06:15.623906 25385 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42988
I20240503 02:06:16.461593 25512 heartbeater.cc:503] Master 127.19.126.62:44719 was elected leader, sending a full tablet report...
I20240503 02:06:16.552136 25587 heartbeater.cc:503] Master 127.19.126.62:44719 was elected leader, sending a full tablet report...
I20240503 02:06:16.626427 25661 heartbeater.cc:503] Master 127.19.126.62:44719 was elected leader, sending a full tablet report...
I20240503 02:06:16.654413 19960 test_util.cc:268] Using random seed: 708235832
I20240503 02:06:16.679713 25385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42998:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:06:16.681998 25385 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:06:16.740512 25477 tablet_service.cc:1467] Processing CreateTablet for tablet ce3b95aac25b414e8054bdc8f1c8ce5e (DEFAULT_TABLE table=test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:06:16.741838 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3b95aac25b414e8054bdc8f1c8ce5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:16.745409 25474 tablet_service.cc:1467] Processing CreateTablet for tablet 520c0c559a714c588dc6b6a34533ce23 (DEFAULT_TABLE table=test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:06:16.746685 25474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520c0c559a714c588dc6b6a34533ce23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:16.750401 25475 tablet_service.cc:1467] Processing CreateTablet for tablet 6e7274d6c77140dbb8e72c15e56e0fff (DEFAULT_TABLE table=test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:06:16.751664 25475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7274d6c77140dbb8e72c15e56e0fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:16.752050 25550 tablet_service.cc:1467] Processing CreateTablet for tablet 6e7274d6c77140dbb8e72c15e56e0fff (DEFAULT_TABLE table=test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:06:16.752465 25476 tablet_service.cc:1467] Processing CreateTablet for tablet 39d38439e4f746659d32731924b4c05b (DEFAULT_TABLE table=test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:06:16.753329 25550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7274d6c77140dbb8e72c15e56e0fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:16.753749 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d38439e4f746659d32731924b4c05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:16.753579 25552 tablet_service.cc:1467] Processing CreateTablet for tablet ce3b95aac25b414e8054bdc8f1c8ce5e (DEFAULT_TABLE table=test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:06:16.754853 25552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3b95aac25b414e8054bdc8f1c8ce5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:16.755618 25549 tablet_service.cc:1467] Processing CreateTablet for tablet 520c0c559a714c588dc6b6a34533ce23 (DEFAULT_TABLE table=test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:06:16.756778 25549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520c0c559a714c588dc6b6a34533ce23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:16.756686 25551 tablet_service.cc:1467] Processing CreateTablet for tablet 39d38439e4f746659d32731924b4c05b (DEFAULT_TABLE table=test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:06:16.757912 25551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d38439e4f746659d32731924b4c05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:16.767597 25681 tablet_bootstrap.cc:492] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:16.774232 25681 tablet_bootstrap.cc:654] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:16.792897 25626 tablet_service.cc:1467] Processing CreateTablet for tablet ce3b95aac25b414e8054bdc8f1c8ce5e (DEFAULT_TABLE table=test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:06:16.794232 25626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3b95aac25b414e8054bdc8f1c8ce5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:16.798333 25625 tablet_service.cc:1467] Processing CreateTablet for tablet 39d38439e4f746659d32731924b4c05b (DEFAULT_TABLE table=test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:06:16.799592 25625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d38439e4f746659d32731924b4c05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:16.801116 25623 tablet_service.cc:1467] Processing CreateTablet for tablet 520c0c559a714c588dc6b6a34533ce23 (DEFAULT_TABLE table=test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:06:16.802350 25623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520c0c559a714c588dc6b6a34533ce23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:16.806408 25624 tablet_service.cc:1467] Processing CreateTablet for tablet 6e7274d6c77140dbb8e72c15e56e0fff (DEFAULT_TABLE table=test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:06:16.807621 25624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7274d6c77140dbb8e72c15e56e0fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:16.832604 25681 tablet_bootstrap.cc:492] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08: No bootstrap required, opened a new log
I20240503 02:06:16.833204 25681 ts_tablet_manager.cc:1367] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.054s
I20240503 02:06:16.835822 25681 raft_consensus.cc:348] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:16.836660 25681 raft_consensus.cc:374] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:16.836995 25681 raft_consensus.cc:729] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:16.837774 25681 consensus_queue.cc:260] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:16.838307 25684 tablet_bootstrap.cc:492] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db: Bootstrap starting.
I20240503 02:06:16.838574 25683 tablet_bootstrap.cc:492] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041: Bootstrap starting.
I20240503 02:06:16.844128 25684 tablet_bootstrap.cc:654] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:16.846273 25681 ts_tablet_manager.cc:1398] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.013s user 0.003s sys 0.007s
I20240503 02:06:16.847353 25683 tablet_bootstrap.cc:654] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:16.847412 25681 tablet_bootstrap.cc:492] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:16.854876 25681 tablet_bootstrap.cc:654] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:16.859102 25683 tablet_bootstrap.cc:492] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041: No bootstrap required, opened a new log
I20240503 02:06:16.859592 25683 ts_tablet_manager.cc:1367] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.008s
I20240503 02:06:16.860299 25684 tablet_bootstrap.cc:492] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db: No bootstrap required, opened a new log
I20240503 02:06:16.860870 25684 ts_tablet_manager.cc:1367] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.018s
I20240503 02:06:16.862380 25683 raft_consensus.cc:348] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.863234 25683 raft_consensus.cc:374] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:16.863570 25683 raft_consensus.cc:729] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a491687290b64a83b750b5506c9ba041, State: Initialized, Role: FOLLOWER
I20240503 02:06:16.863531 25684 raft_consensus.cc:348] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.864486 25684 raft_consensus.cc:374] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:16.864902 25684 raft_consensus.cc:729] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d724b88dae941358c33ea9e56a022db, State: Initialized, Role: FOLLOWER
I20240503 02:06:16.864540 25683 consensus_queue.cc:260] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.866297 25681 tablet_bootstrap.cc:492] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08: No bootstrap required, opened a new log
I20240503 02:06:16.866858 25681 ts_tablet_manager.cc:1367] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240503 02:06:16.866212 25684 consensus_queue.cc:260] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.869114 25681 raft_consensus.cc:348] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.869635 25681 raft_consensus.cc:374] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:16.869868 25681 raft_consensus.cc:729] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:16.870445 25681 consensus_queue.cc:260] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.872305 25681 ts_tablet_manager.cc:1398] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:06:16.873242 25681 tablet_bootstrap.cc:492] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:16.875910 25684 ts_tablet_manager.cc:1398] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db: Time spent starting tablet: real 0.015s user 0.005s sys 0.002s
I20240503 02:06:16.878311 25681 tablet_bootstrap.cc:654] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:16.882354 25684 tablet_bootstrap.cc:492] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db: Bootstrap starting.
I20240503 02:06:16.891639 25683 ts_tablet_manager.cc:1398] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041: Time spent starting tablet: real 0.032s user 0.019s sys 0.011s
I20240503 02:06:16.892699 25683 tablet_bootstrap.cc:492] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041: Bootstrap starting.
I20240503 02:06:16.892983 25681 tablet_bootstrap.cc:492] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08: No bootstrap required, opened a new log
I20240503 02:06:16.893486 25681 ts_tablet_manager.cc:1367] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.010s
I20240503 02:06:16.895380 25684 tablet_bootstrap.cc:654] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:16.895439 25689 raft_consensus.cc:482] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:16.895989 25689 raft_consensus.cc:504] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.896092 25681 raft_consensus.cc:348] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.896971 25681 raft_consensus.cc:374] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:16.897338 25681 raft_consensus.cc:729] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:16.898250 25683 tablet_bootstrap.cc:654] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:16.898062 25681 consensus_queue.cc:260] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.903434 25681 ts_tablet_manager.cc:1398] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240503 02:06:16.904727 25681 tablet_bootstrap.cc:492] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:16.907424 25685 raft_consensus.cc:482] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:16.907907 25685 raft_consensus.cc:504] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.910564 25685 leader_election.cc:288] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:16.910789 25681 tablet_bootstrap.cc:654] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:16.911542 25689 leader_election.cc:288] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:16.934024 25681 tablet_bootstrap.cc:492] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08: No bootstrap required, opened a new log
I20240503 02:06:16.935540 25684 tablet_bootstrap.cc:492] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db: No bootstrap required, opened a new log
I20240503 02:06:16.936288 25684 ts_tablet_manager.cc:1367] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.011s
I20240503 02:06:16.938928 25683 tablet_bootstrap.cc:492] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041: No bootstrap required, opened a new log
I20240503 02:06:16.941155 25683 ts_tablet_manager.cc:1367] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.011s
I20240503 02:06:16.939181 25684 raft_consensus.cc:348] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.940192 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7274d6c77140dbb8e72c15e56e0fff" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08" is_pre_election: true
I20240503 02:06:16.942390 25487 raft_consensus.cc:2454] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 0.
I20240503 02:06:16.943615 25525 leader_election.cc:302] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:16.943979 25683 raft_consensus.cc:348] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:16.940515 25681 ts_tablet_manager.cc:1367] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.004s
I20240503 02:06:16.944602 25689 raft_consensus.cc:2781] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:16.944806 25683 raft_consensus.cc:374] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:16.945053 25689 raft_consensus.cc:482] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:16.945423 25684 raft_consensus.cc:374] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:16.945482 25689 raft_consensus.cc:3037] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:16.945804 25684 raft_consensus.cc:729] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d724b88dae941358c33ea9e56a022db, State: Initialized, Role: FOLLOWER
I20240503 02:06:16.946695 25685 raft_consensus.cc:482] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:16.947111 25685 raft_consensus.cc:504] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:16.948082 25684 consensus_queue.cc:260] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.949277 25685 leader_election.cc:288] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187), a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473)
I20240503 02:06:16.947098 25681 raft_consensus.cc:348] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.950052 25681 raft_consensus.cc:374] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:16.950353 25681 raft_consensus.cc:729] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:16.950990 25681 consensus_queue.cc:260] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.951901 25684 ts_tablet_manager.cc:1398] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20240503 02:06:16.952845 25684 tablet_bootstrap.cc:492] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db: Bootstrap starting.
I20240503 02:06:16.953138 25689 raft_consensus.cc:504] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.953784 25681 ts_tablet_manager.cc:1398] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240503 02:06:16.945231 25683 raft_consensus.cc:729] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a491687290b64a83b750b5506c9ba041, State: Initialized, Role: FOLLOWER
I20240503 02:06:16.955219 25683 consensus_queue.cc:260] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:16.958019 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7274d6c77140dbb8e72c15e56e0fff" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08"
I20240503 02:06:16.958613 25487 raft_consensus.cc:3037] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:16.958662 25684 tablet_bootstrap.cc:654] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:16.965505 25487 raft_consensus.cc:2454] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 1.
I20240503 02:06:16.966573 25525 leader_election.cc:302] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:16.973304 25689 leader_election.cc:288] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 election: Requested vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:16.979288 25689 raft_consensus.cc:2781] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:16.980232 25689 raft_consensus.cc:686] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Becoming Leader. State: Replica: a491687290b64a83b750b5506c9ba041, State: Running, Role: LEADER
I20240503 02:06:16.981532 25689 consensus_queue.cc:237] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:16.990340 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7274d6c77140dbb8e72c15e56e0fff" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041" is_pre_election: true
I20240503 02:06:16.991865 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b95aac25b414e8054bdc8f1c8ce5e" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041" is_pre_election: true
I20240503 02:06:16.992668 25561 raft_consensus.cc:2454] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 698037111e64432092d44f7db2edfa08 in term 0.
I20240503 02:06:16.994165 25451 leader_election.cc:302] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:16.994407 25683 ts_tablet_manager.cc:1398] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041: Time spent starting tablet: real 0.053s user 0.023s sys 0.019s
I20240503 02:06:16.995535 25685 raft_consensus.cc:2781] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:16.995882 25685 raft_consensus.cc:482] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:16.996218 25685 raft_consensus.cc:3037] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:16.996229 25683 tablet_bootstrap.cc:492] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041: Bootstrap starting.
I20240503 02:06:16.999526 25684 tablet_bootstrap.cc:492] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db: No bootstrap required, opened a new log
I20240503 02:06:17.000031 25684 ts_tablet_manager.cc:1367] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db: Time spent bootstrapping tablet: real 0.047s user 0.018s sys 0.022s
I20240503 02:06:17.000981 25385 catalog_manager.cc:5484] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 reported cstate change: term changed from 0 to 1, leader changed from <none> to a491687290b64a83b750b5506c9ba041 (127.19.126.2). New cstate: current_term: 1 leader_uuid: "a491687290b64a83b750b5506c9ba041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:17.003290 25685 raft_consensus.cc:504] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:17.004053 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7274d6c77140dbb8e72c15e56e0fff" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:17.004750 25636 raft_consensus.cc:2454] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 0.
I20240503 02:06:17.005627 25683 tablet_bootstrap.cc:654] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.005872 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7274d6c77140dbb8e72c15e56e0fff" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db"
I20240503 02:06:17.006525 25635 raft_consensus.cc:3037] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.006625 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b95aac25b414e8054bdc8f1c8ce5e" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041"
I20240503 02:06:17.007194 25561 raft_consensus.cc:3037] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.012853 25684 raft_consensus.cc:348] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.013701 25633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b95aac25b414e8054bdc8f1c8ce5e" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db"
I20240503 02:06:17.013276 25635 raft_consensus.cc:2454] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 1.
I20240503 02:06:17.014060 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b95aac25b414e8054bdc8f1c8ce5e" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:17.014997 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7274d6c77140dbb8e72c15e56e0fff" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:17.018187 25636 raft_consensus.cc:2379] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 698037111e64432092d44f7db2edfa08 in current term 1: Already voted for candidate a491687290b64a83b750b5506c9ba041 in this term.
I20240503 02:06:17.014127 25685 leader_election.cc:288] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 election: Requested vote from peers 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187), a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473)
I20240503 02:06:17.013877 25684 raft_consensus.cc:374] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.019433 25684 raft_consensus.cc:729] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d724b88dae941358c33ea9e56a022db, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.020231 25561 raft_consensus.cc:2454] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 698037111e64432092d44f7db2edfa08 in term 1.
I20240503 02:06:17.020087 25684 consensus_queue.cc:260] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
W20240503 02:06:17.021960 25452 leader_election.cc:341] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:06:17.022614 25452 leader_election.cc:341] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:17.023610 25451 leader_election.cc:302] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters: 1d724b88dae941358c33ea9e56a022db
I20240503 02:06:17.022612 25684 ts_tablet_manager.cc:1398] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20240503 02:06:17.023571 25452 leader_election.cc:302] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 698037111e64432092d44f7db2edfa08; no voters: 1d724b88dae941358c33ea9e56a022db, a491687290b64a83b750b5506c9ba041
I20240503 02:06:17.024824 25685 raft_consensus.cc:2781] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:17.025197 25684 tablet_bootstrap.cc:492] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db: Bootstrap starting.
I20240503 02:06:17.026137 25683 tablet_bootstrap.cc:492] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041: No bootstrap required, opened a new log
I20240503 02:06:17.026609 25683 ts_tablet_manager.cc:1367] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.011s
I20240503 02:06:17.029561 25683 raft_consensus.cc:348] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.030261 25683 raft_consensus.cc:374] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.030534 25683 raft_consensus.cc:729] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a491687290b64a83b750b5506c9ba041, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.031347 25684 tablet_bootstrap.cc:654] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.031164 25683 consensus_queue.cc:260] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.033602 25683 ts_tablet_manager.cc:1398] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240503 02:06:17.034641 25683 tablet_bootstrap.cc:492] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041: Bootstrap starting.
I20240503 02:06:17.041515 25683 tablet_bootstrap.cc:654] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.061735 25705 raft_consensus.cc:2726] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:06:17.065255 25685 raft_consensus.cc:686] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 1 LEADER]: Becoming Leader. State: Replica: 698037111e64432092d44f7db2edfa08, State: Running, Role: LEADER
I20240503 02:06:17.066211 25685 consensus_queue.cc:237] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:17.067203 25684 tablet_bootstrap.cc:492] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db: No bootstrap required, opened a new log
I20240503 02:06:17.067214 25706 raft_consensus.cc:482] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:17.067792 25684 ts_tablet_manager.cc:1367] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db: Time spent bootstrapping tablet: real 0.043s user 0.022s sys 0.017s
I20240503 02:06:17.068126 25706 raft_consensus.cc:504] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.070889 25683 tablet_bootstrap.cc:492] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041: No bootstrap required, opened a new log
I20240503 02:06:17.071056 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041" is_pre_election: true
I20240503 02:06:17.071363 25683 ts_tablet_manager.cc:1367] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.023s
I20240503 02:06:17.071246 25684 raft_consensus.cc:348] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:17.072403 25684 raft_consensus.cc:374] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:06:17.072708 25451 leader_election.cc:341] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:06:17.072825 25684 raft_consensus.cc:729] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d724b88dae941358c33ea9e56a022db, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.074198 25683 raft_consensus.cc:348] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.074656 25706 leader_election.cc:288] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.075042 25683 raft_consensus.cc:374] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.074920 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:17.075414 25683 raft_consensus.cc:729] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a491687290b64a83b750b5506c9ba041, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.075755 25636 raft_consensus.cc:2454] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 698037111e64432092d44f7db2edfa08 in term 0.
I20240503 02:06:17.076089 25683 consensus_queue.cc:260] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.074043 25684 consensus_queue.cc:260] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:17.077023 25452 leader_election.cc:302] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d724b88dae941358c33ea9e56a022db, 698037111e64432092d44f7db2edfa08; no voters: a491687290b64a83b750b5506c9ba041
I20240503 02:06:17.078320 25685 raft_consensus.cc:2781] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:17.078539 25684 ts_tablet_manager.cc:1398] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20240503 02:06:17.078826 25685 raft_consensus.cc:482] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:17.079212 25685 raft_consensus.cc:3037] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.080920 25683 ts_tablet_manager.cc:1398] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240503 02:06:17.082701 25385 catalog_manager.cc:5484] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 698037111e64432092d44f7db2edfa08 (127.19.126.1). New cstate: current_term: 1 leader_uuid: "698037111e64432092d44f7db2edfa08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:17.085491 25685 raft_consensus.cc:504] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.087985 25685 leader_election.cc:288] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 election: Requested vote from peers a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.088097 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041"
I20240503 02:06:17.088675 25561 raft_consensus.cc:3037] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.089471 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db"
I20240503 02:06:17.090023 25636 raft_consensus.cc:3037] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.095918 25561 raft_consensus.cc:2454] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 698037111e64432092d44f7db2edfa08 in term 1.
I20240503 02:06:17.096865 25451 leader_election.cc:302] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:17.097604 25685 raft_consensus.cc:2781] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:17.098043 25685 raft_consensus.cc:686] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 1 LEADER]: Becoming Leader. State: Replica: 698037111e64432092d44f7db2edfa08, State: Running, Role: LEADER
I20240503 02:06:17.098099 25636 raft_consensus.cc:2454] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 698037111e64432092d44f7db2edfa08 in term 1.
I20240503 02:06:17.098937 25685 consensus_queue.cc:237] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.104712 25385 catalog_manager.cc:5484] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 698037111e64432092d44f7db2edfa08 (127.19.126.1). New cstate: current_term: 1 leader_uuid: "698037111e64432092d44f7db2edfa08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:17.179041 25689 raft_consensus.cc:482] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:17.179464 25689 raft_consensus.cc:504] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.181030 25689 leader_election.cc:288] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.181849 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d38439e4f746659d32731924b4c05b" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08" is_pre_election: true
I20240503 02:06:17.182018 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d38439e4f746659d32731924b4c05b" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:17.182374 25487 raft_consensus.cc:2454] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 0.
I20240503 02:06:17.182596 25636 raft_consensus.cc:2454] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 0.
I20240503 02:06:17.183297 25525 leader_election.cc:302] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:17.183950 25689 raft_consensus.cc:2781] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:17.184296 25689 raft_consensus.cc:482] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:17.184532 25689 raft_consensus.cc:3037] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.188886 25689 raft_consensus.cc:504] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.190322 25689 leader_election.cc:288] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 election: Requested vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.191021 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d38439e4f746659d32731924b4c05b" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08"
I20240503 02:06:17.191466 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d38439e4f746659d32731924b4c05b" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db"
I20240503 02:06:17.191545 25487 raft_consensus.cc:3037] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.191906 25636 raft_consensus.cc:3037] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.195791 25487 raft_consensus.cc:2454] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 1.
I20240503 02:06:17.196630 25525 leader_election.cc:302] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:17.196808 25636 raft_consensus.cc:2454] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 1.
I20240503 02:06:17.197415 25689 raft_consensus.cc:2781] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:17.197885 25689 raft_consensus.cc:686] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Becoming Leader. State: Replica: a491687290b64a83b750b5506c9ba041, State: Running, Role: LEADER
I20240503 02:06:17.198544 25689 consensus_queue.cc:237] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.204849 25385 catalog_manager.cc:5484] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 reported cstate change: term changed from 0 to 1, leader changed from <none> to a491687290b64a83b750b5506c9ba041 (127.19.126.2). New cstate: current_term: 1 leader_uuid: "a491687290b64a83b750b5506c9ba041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:17.231202 19960 test_util.cc:268] Using random seed: 708812618
I20240503 02:06:17.250142 25385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43888:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:06:17.252211 25385 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:06:17.278800 25474 tablet_service.cc:1467] Processing CreateTablet for tablet 83218d07a65646f4896c9baa9f87ea6a (DEFAULT_TABLE table=dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:06:17.279419 25550 tablet_service.cc:1467] Processing CreateTablet for tablet 83218d07a65646f4896c9baa9f87ea6a (DEFAULT_TABLE table=dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:06:17.280009 25474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83218d07a65646f4896c9baa9f87ea6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:17.280122 25626 tablet_service.cc:1467] Processing CreateTablet for tablet 83218d07a65646f4896c9baa9f87ea6a (DEFAULT_TABLE table=dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 02:06:17.280612 25550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83218d07a65646f4896c9baa9f87ea6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:17.281348 25626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83218d07a65646f4896c9baa9f87ea6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:17.282964 25551 tablet_service.cc:1467] Processing CreateTablet for tablet e2d5910838654d63839e42f51aa40753 (DEFAULT_TABLE table=dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:06:17.284127 25551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d5910838654d63839e42f51aa40753. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:17.290011 25625 tablet_service.cc:1467] Processing CreateTablet for tablet e2d5910838654d63839e42f51aa40753 (DEFAULT_TABLE table=dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:06:17.290163 25476 tablet_service.cc:1467] Processing CreateTablet for tablet 78942ce15f0e4594930b3d4fbeb57c02 (DEFAULT_TABLE table=dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:06:17.291208 25625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d5910838654d63839e42f51aa40753. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:17.291375 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78942ce15f0e4594930b3d4fbeb57c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:17.294050 25477 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb3d1846ab542f69676509de46edba9 (DEFAULT_TABLE table=dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:06:17.295152 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb3d1846ab542f69676509de46edba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:17.297755 25681 tablet_bootstrap.cc:492] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:17.299923 25624 tablet_service.cc:1467] Processing CreateTablet for tablet 78942ce15f0e4594930b3d4fbeb57c02 (DEFAULT_TABLE table=dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:06:17.301123 25624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78942ce15f0e4594930b3d4fbeb57c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:17.302894 25549 tablet_service.cc:1467] Processing CreateTablet for tablet 78942ce15f0e4594930b3d4fbeb57c02 (DEFAULT_TABLE table=dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 02:06:17.304073 25549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78942ce15f0e4594930b3d4fbeb57c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:17.304225 25623 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb3d1846ab542f69676509de46edba9 (DEFAULT_TABLE table=dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:06:17.305344 25623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb3d1846ab542f69676509de46edba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:17.306216 25552 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb3d1846ab542f69676509de46edba9 (DEFAULT_TABLE table=dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 02:06:17.307343 25552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb3d1846ab542f69676509de46edba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:17.309068 25683 tablet_bootstrap.cc:492] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041: Bootstrap starting.
I20240503 02:06:17.306236 25684 tablet_bootstrap.cc:492] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db: Bootstrap starting.
I20240503 02:06:17.286717 25475 tablet_service.cc:1467] Processing CreateTablet for tablet e2d5910838654d63839e42f51aa40753 (DEFAULT_TABLE table=dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 02:06:17.314430 25475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d5910838654d63839e42f51aa40753. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:17.314527 25683 tablet_bootstrap.cc:654] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.322533 25681 tablet_bootstrap.cc:654] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.332890 25684 tablet_bootstrap.cc:654] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.345715 25684 tablet_bootstrap.cc:492] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db: No bootstrap required, opened a new log
I20240503 02:06:17.346210 25684 ts_tablet_manager.cc:1367] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.008s
I20240503 02:06:17.348796 25684 raft_consensus.cc:348] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.349531 25684 raft_consensus.cc:374] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.349807 25684 raft_consensus.cc:729] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d724b88dae941358c33ea9e56a022db, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.352685 25681 tablet_bootstrap.cc:492] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: No bootstrap required, opened a new log
I20240503 02:06:17.353216 25681 ts_tablet_manager.cc:1367] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.005s
I20240503 02:06:17.352864 25684 consensus_queue.cc:260] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.356102 25681 raft_consensus.cc:348] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.356869 25681 raft_consensus.cc:374] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.357167 25681 raft_consensus.cc:729] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.357971 25681 consensus_queue.cc:260] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.360160 25684 ts_tablet_manager.cc:1398] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240503 02:06:17.362092 25684 tablet_bootstrap.cc:492] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db: Bootstrap starting.
I20240503 02:06:17.363423 25681 ts_tablet_manager.cc:1398] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240503 02:06:17.366604 25681 tablet_bootstrap.cc:492] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:17.371232 25684 tablet_bootstrap.cc:654] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.372100 25681 tablet_bootstrap.cc:654] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.376957 25683 tablet_bootstrap.cc:492] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041: No bootstrap required, opened a new log
I20240503 02:06:17.377354 25683 ts_tablet_manager.cc:1367] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041: Time spent bootstrapping tablet: real 0.068s user 0.031s sys 0.012s
I20240503 02:06:17.378160 25681 tablet_bootstrap.cc:492] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: No bootstrap required, opened a new log
I20240503 02:06:17.378526 25681 ts_tablet_manager.cc:1367] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20240503 02:06:17.378924 25684 tablet_bootstrap.cc:492] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db: No bootstrap required, opened a new log
I20240503 02:06:17.379348 25684 ts_tablet_manager.cc:1367] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240503 02:06:17.379863 25683 raft_consensus.cc:348] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.380578 25683 raft_consensus.cc:374] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.381018 25683 raft_consensus.cc:729] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a491687290b64a83b750b5506c9ba041, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.380615 25681 raft_consensus.cc:348] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.381511 25681 raft_consensus.cc:374] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.381778 25681 raft_consensus.cc:729] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.381752 25683 consensus_queue.cc:260] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.382552 25681 consensus_queue.cc:260] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.382581 25684 raft_consensus.cc:348] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.383428 25684 raft_consensus.cc:374] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.383723 25684 raft_consensus.cc:729] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d724b88dae941358c33ea9e56a022db, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.384171 25683 ts_tablet_manager.cc:1398] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:06:17.384788 25681 ts_tablet_manager.cc:1398] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:06:17.384395 25684 consensus_queue.cc:260] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.385103 25683 tablet_bootstrap.cc:492] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041: Bootstrap starting.
I20240503 02:06:17.385723 25681 tablet_bootstrap.cc:492] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:17.388734 25684 ts_tablet_manager.cc:1398] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20240503 02:06:17.389709 25684 tablet_bootstrap.cc:492] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db: Bootstrap starting.
I20240503 02:06:17.390761 25683 tablet_bootstrap.cc:654] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.390862 25681 tablet_bootstrap.cc:654] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.395517 25684 tablet_bootstrap.cc:654] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.401952 25683 tablet_bootstrap.cc:492] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041: No bootstrap required, opened a new log
I20240503 02:06:17.402438 25683 ts_tablet_manager.cc:1367] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.001s
I20240503 02:06:17.404147 25681 tablet_bootstrap.cc:492] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: No bootstrap required, opened a new log
I20240503 02:06:17.404489 25684 tablet_bootstrap.cc:492] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db: No bootstrap required, opened a new log
I20240503 02:06:17.404596 25681 ts_tablet_manager.cc:1367] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.002s
I20240503 02:06:17.405007 25684 ts_tablet_manager.cc:1367] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.002s
I20240503 02:06:17.404979 25683 raft_consensus.cc:348] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.405691 25683 raft_consensus.cc:374] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.405982 25683 raft_consensus.cc:729] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a491687290b64a83b750b5506c9ba041, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.406642 25683 consensus_queue.cc:260] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.407272 25681 raft_consensus.cc:348] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.407967 25681 raft_consensus.cc:374] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.408414 25681 raft_consensus.cc:729] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.408782 25683 ts_tablet_manager.cc:1398] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20240503 02:06:17.409138 25681 consensus_queue.cc:260] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.409739 25683 tablet_bootstrap.cc:492] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041: Bootstrap starting.
I20240503 02:06:17.407922 25684 raft_consensus.cc:348] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.410904 25684 raft_consensus.cc:374] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.411199 25684 raft_consensus.cc:729] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d724b88dae941358c33ea9e56a022db, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.411296 25681 ts_tablet_manager.cc:1398] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 02:06:17.412326 25681 tablet_bootstrap.cc:492] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:17.411959 25684 consensus_queue.cc:260] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.414692 25684 ts_tablet_manager.cc:1398] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240503 02:06:17.415529 25683 tablet_bootstrap.cc:654] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.415613 25684 tablet_bootstrap.cc:492] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db: Bootstrap starting.
I20240503 02:06:17.418148 25681 tablet_bootstrap.cc:654] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.422619 25684 tablet_bootstrap.cc:654] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.428789 25683 tablet_bootstrap.cc:492] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041: No bootstrap required, opened a new log
I20240503 02:06:17.429435 25683 ts_tablet_manager.cc:1367] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.012s
I20240503 02:06:17.431799 25681 tablet_bootstrap.cc:492] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: No bootstrap required, opened a new log
I20240503 02:06:17.432214 25684 tablet_bootstrap.cc:492] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db: No bootstrap required, opened a new log
I20240503 02:06:17.432281 25681 ts_tablet_manager.cc:1367] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20240503 02:06:17.432926 25684 ts_tablet_manager.cc:1367] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20240503 02:06:17.432066 25683 raft_consensus.cc:348] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.433605 25683 raft_consensus.cc:374] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.433907 25683 raft_consensus.cc:729] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a491687290b64a83b750b5506c9ba041, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.434677 25683 consensus_queue.cc:260] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.434980 25681 raft_consensus.cc:348] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.435599 25681 raft_consensus.cc:374] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.435861 25681 raft_consensus.cc:729] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.436012 25684 raft_consensus.cc:348] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.436888 25684 raft_consensus.cc:374] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.437238 25684 raft_consensus.cc:729] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d724b88dae941358c33ea9e56a022db, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.437984 25684 consensus_queue.cc:260] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.439631 25683 ts_tablet_manager.cc:1398] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20240503 02:06:17.439738 25684 ts_tablet_manager.cc:1398] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:06:17.436456 25681 consensus_queue.cc:260] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.440666 25683 tablet_bootstrap.cc:492] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041: Bootstrap starting.
I20240503 02:06:17.441802 25681 ts_tablet_manager.cc:1398] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240503 02:06:17.445377 25690 raft_consensus.cc:482] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:17.445816 25690 raft_consensus.cc:504] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:17.448011 25690 leader_election.cc:288] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473)
I20240503 02:06:17.451412 25689 raft_consensus.cc:482] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:17.452039 25689 raft_consensus.cc:504] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.454788 25689 leader_election.cc:288] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.456681 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d5910838654d63839e42f51aa40753" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08" is_pre_election: true
I20240503 02:06:17.460600 25487 raft_consensus.cc:2454] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 0.
I20240503 02:06:17.460958 25683 tablet_bootstrap.cc:654] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041: Neither blocks nor log segments found. Creating new log.
I20240503 02:06:17.462358 25525 leader_election.cc:302] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:17.463436 25685 raft_consensus.cc:482] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:17.464036 25685 raft_consensus.cc:504] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.467525 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78942ce15f0e4594930b3d4fbeb57c02" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041" is_pre_election: true
I20240503 02:06:17.468299 25561 raft_consensus.cc:2454] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 698037111e64432092d44f7db2edfa08 in term 0.
I20240503 02:06:17.469839 25451 leader_election.cc:302] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:17.470907 25706 raft_consensus.cc:2781] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:17.470976 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d5910838654d63839e42f51aa40753" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:17.471418 25706 raft_consensus.cc:482] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:17.471694 25636 raft_consensus.cc:2454] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 0.
I20240503 02:06:17.471899 25706 raft_consensus.cc:3037] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.474481 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b95aac25b414e8054bdc8f1c8ce5e" candidate_uuid: "1d724b88dae941358c33ea9e56a022db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08" is_pre_election: true
I20240503 02:06:17.476912 25683 tablet_bootstrap.cc:492] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041: No bootstrap required, opened a new log
I20240503 02:06:17.477624 25683 ts_tablet_manager.cc:1367] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.002s
I20240503 02:06:17.482232 25689 raft_consensus.cc:2781] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:17.482656 25689 raft_consensus.cc:482] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:17.483052 25689 raft_consensus.cc:3037] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.488318 25689 raft_consensus.cc:504] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.488970 25685 leader_election.cc:288] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.490792 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d5910838654d63839e42f51aa40753" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08"
I20240503 02:06:17.491740 25487 raft_consensus.cc:3037] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.490602 25683 raft_consensus.cc:348] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.492307 25683 raft_consensus.cc:374] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:06:17.492666 25683 raft_consensus.cc:729] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a491687290b64a83b750b5506c9ba041, State: Initialized, Role: FOLLOWER
I20240503 02:06:17.493604 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78942ce15f0e4594930b3d4fbeb57c02" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:17.493427 25683 consensus_queue.cc:260] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.494318 25636 raft_consensus.cc:2454] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 698037111e64432092d44f7db2edfa08 in term 0.
I20240503 02:06:17.493480 25706 raft_consensus.cc:504] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.495903 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b95aac25b414e8054bdc8f1c8ce5e" candidate_uuid: "1d724b88dae941358c33ea9e56a022db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041" is_pre_election: true
I20240503 02:06:17.496353 25706 leader_election.cc:288] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 election: Requested vote from peers a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.496603 25561 raft_consensus.cc:2379] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1d724b88dae941358c33ea9e56a022db in current term 1: Already voted for candidate 698037111e64432092d44f7db2edfa08 in this term.
I20240503 02:06:17.497818 25683 ts_tablet_manager.cc:1398] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041: Time spent starting tablet: real 0.020s user 0.009s sys 0.008s
I20240503 02:06:17.498593 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78942ce15f0e4594930b3d4fbeb57c02" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041"
I20240503 02:06:17.498966 25600 leader_election.cc:302] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d724b88dae941358c33ea9e56a022db; no voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041
I20240503 02:06:17.499112 25561 raft_consensus.cc:3037] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.499758 25690 raft_consensus.cc:3037] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.504619 25689 leader_election.cc:288] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 election: Requested vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.504796 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78942ce15f0e4594930b3d4fbeb57c02" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db"
I20240503 02:06:17.505376 25636 raft_consensus.cc:3037] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.507215 25690 raft_consensus.cc:2726] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:06:17.509495 25561 raft_consensus.cc:2454] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 698037111e64432092d44f7db2edfa08 in term 1.
I20240503 02:06:17.510524 25487 raft_consensus.cc:2454] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 1.
I20240503 02:06:17.510627 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d5910838654d63839e42f51aa40753" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db"
I20240503 02:06:17.511497 25634 raft_consensus.cc:3037] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.512050 25636 raft_consensus.cc:2454] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 698037111e64432092d44f7db2edfa08 in term 1.
I20240503 02:06:17.512351 25451 leader_election.cc:302] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:17.513126 25706 raft_consensus.cc:2781] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:17.513667 25525 leader_election.cc:302] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:17.514379 25706 raft_consensus.cc:686] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 1 LEADER]: Becoming Leader. State: Replica: 698037111e64432092d44f7db2edfa08, State: Running, Role: LEADER
I20240503 02:06:17.514576 25689 raft_consensus.cc:2781] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:17.515185 25689 raft_consensus.cc:686] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Becoming Leader. State: Replica: a491687290b64a83b750b5506c9ba041, State: Running, Role: LEADER
I20240503 02:06:17.515254 25706 consensus_queue.cc:237] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.516124 25689 consensus_queue.cc:237] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.518278 25634 raft_consensus.cc:2454] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 1.
I20240503 02:06:17.524873 25385 catalog_manager.cc:5484] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 698037111e64432092d44f7db2edfa08 (127.19.126.1). New cstate: current_term: 1 leader_uuid: "698037111e64432092d44f7db2edfa08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:17.526726 25384 catalog_manager.cc:5484] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 reported cstate change: term changed from 0 to 1, leader changed from <none> to a491687290b64a83b750b5506c9ba041 (127.19.126.2). New cstate: current_term: 1 leader_uuid: "a491687290b64a83b750b5506c9ba041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:17.559717 25689 consensus_queue.cc:1035] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:17.562758 25706 consensus_queue.cc:1035] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:17.581239 25689 raft_consensus.cc:482] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:17.584801 25736 consensus_queue.cc:1035] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:17.582298 25689 raft_consensus.cc:504] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.589946 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb3d1846ab542f69676509de46edba9" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08" is_pre_election: true
I20240503 02:06:17.590595 25486 raft_consensus.cc:2454] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 0.
I20240503 02:06:17.591292 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb3d1846ab542f69676509de46edba9" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:17.592335 25634 raft_consensus.cc:2454] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 0.
I20240503 02:06:17.593715 25525 leader_election.cc:302] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:17.593922 25689 leader_election.cc:288] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.594728 25736 raft_consensus.cc:2781] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:17.596830 25736 raft_consensus.cc:482] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:17.597177 25736 raft_consensus.cc:3037] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.597832 25743 raft_consensus.cc:482] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:17.598271 25743 raft_consensus.cc:504] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.600370 25743 leader_election.cc:288] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.601197 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb3d1846ab542f69676509de46edba9" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:17.601922 25634 raft_consensus.cc:2454] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 698037111e64432092d44f7db2edfa08 in term 0.
I20240503 02:06:17.602784 25452 leader_election.cc:302] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d724b88dae941358c33ea9e56a022db, 698037111e64432092d44f7db2edfa08; no voters:
I20240503 02:06:17.603201 25736 raft_consensus.cc:504] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.603581 25706 raft_consensus.cc:2781] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:17.604123 25706 raft_consensus.cc:482] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:17.604542 25706 raft_consensus.cc:3037] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.605379 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb3d1846ab542f69676509de46edba9" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08"
I20240503 02:06:17.607719 25736 leader_election.cc:288] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 election: Requested vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.608927 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb3d1846ab542f69676509de46edba9" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db"
I20240503 02:06:17.609679 25634 raft_consensus.cc:3037] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.612339 25706 raft_consensus.cc:504] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.613843 25486 raft_consensus.cc:2379] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a491687290b64a83b750b5506c9ba041 in current term 1: Already voted for candidate 698037111e64432092d44f7db2edfa08 in this term.
I20240503 02:06:17.614670 25706 leader_election.cc:288] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 election: Requested vote from peers a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.616137 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb3d1846ab542f69676509de46edba9" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041" is_pre_election: true
I20240503 02:06:17.616853 25561 raft_consensus.cc:2379] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 698037111e64432092d44f7db2edfa08 in current term 1: Already voted for candidate a491687290b64a83b750b5506c9ba041 in this term.
I20240503 02:06:17.619222 25560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb3d1846ab542f69676509de46edba9" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041"
I20240503 02:06:17.619992 25560 raft_consensus.cc:2379] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 698037111e64432092d44f7db2edfa08 in current term 1: Already voted for candidate a491687290b64a83b750b5506c9ba041 in this term.
I20240503 02:06:17.621783 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb3d1846ab542f69676509de46edba9" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db"
I20240503 02:06:17.624231 25452 leader_election.cc:302] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 698037111e64432092d44f7db2edfa08; no voters: 1d724b88dae941358c33ea9e56a022db, a491687290b64a83b750b5506c9ba041
I20240503 02:06:17.625384 25742 raft_consensus.cc:2726] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:06:17.626060 25742 consensus_queue.cc:1035] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:17.626684 25634 raft_consensus.cc:2454] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 1.
I20240503 02:06:17.628947 25527 leader_election.cc:302] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d724b88dae941358c33ea9e56a022db, a491687290b64a83b750b5506c9ba041; no voters: 698037111e64432092d44f7db2edfa08
I20240503 02:06:17.630542 25736 raft_consensus.cc:2781] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:17.631320 25736 raft_consensus.cc:686] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Becoming Leader. State: Replica: a491687290b64a83b750b5506c9ba041, State: Running, Role: LEADER
I20240503 02:06:17.632159 25736 consensus_queue.cc:237] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.641840 25384 catalog_manager.cc:5484] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 reported cstate change: term changed from 0 to 1, leader changed from <none> to a491687290b64a83b750b5506c9ba041 (127.19.126.2). New cstate: current_term: 1 leader_uuid: "a491687290b64a83b750b5506c9ba041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:17.647719 25736 raft_consensus.cc:482] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:17.648226 25736 raft_consensus.cc:504] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.650792 25736 leader_election.cc:288] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.651995 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83218d07a65646f4896c9baa9f87ea6a" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:17.652556 25634 raft_consensus.cc:2454] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 0.
I20240503 02:06:17.653360 25527 leader_election.cc:302] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d724b88dae941358c33ea9e56a022db, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:17.654079 25736 raft_consensus.cc:2781] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:06:17.653975 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83218d07a65646f4896c9baa9f87ea6a" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08" is_pre_election: true
I20240503 02:06:17.654390 25736 raft_consensus.cc:482] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:17.654623 25487 raft_consensus.cc:2454] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 0.
I20240503 02:06:17.654724 25736 raft_consensus.cc:3037] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.659408 25736 raft_consensus.cc:504] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.660753 25736 leader_election.cc:288] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 election: Requested vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:17.661530 25487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83218d07a65646f4896c9baa9f87ea6a" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08"
I20240503 02:06:17.661681 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83218d07a65646f4896c9baa9f87ea6a" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db"
I20240503 02:06:17.662124 25487 raft_consensus.cc:3037] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.662210 25634 raft_consensus.cc:3037] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:06:17.668476 25487 raft_consensus.cc:2454] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 1.
I20240503 02:06:17.668469 25634 raft_consensus.cc:2454] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 1.
I20240503 02:06:17.669584 25527 leader_election.cc:302] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d724b88dae941358c33ea9e56a022db, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:17.670166 25736 raft_consensus.cc:2781] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:06:17.670573 25736 raft_consensus.cc:686] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Becoming Leader. State: Replica: a491687290b64a83b750b5506c9ba041, State: Running, Role: LEADER
I20240503 02:06:17.671205 25736 consensus_queue.cc:237] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:17.678192 25383 catalog_manager.cc:5484] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 reported cstate change: term changed from 0 to 1, leader changed from <none> to a491687290b64a83b750b5506c9ba041 (127.19.126.2). New cstate: current_term: 1 leader_uuid: "a491687290b64a83b750b5506c9ba041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:17.691102 25742 consensus_queue.cc:1035] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:17.700146 25685 consensus_queue.cc:1035] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:17.738425 19960 tablet_server.cc:178] TabletServer@127.19.126.1:0 shutting down...
W20240503 02:06:17.744853 25525 consensus_peers.cc:485] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 -> Peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Couldn't send request to peer 698037111e64432092d44f7db2edfa08. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:17.758263 25736 consensus_queue.cc:1035] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:17.761910 19960 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:06:17.763072 19960 tablet_replica.cc:330] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: stopping tablet replica
I20240503 02:06:17.763621 19960 raft_consensus.cc:2229] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:17.763979 19960 raft_consensus.cc:2258] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:17.766628 19960 tablet_replica.cc:330] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: stopping tablet replica
I20240503 02:06:17.767390 19960 raft_consensus.cc:2229] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:17.768149 19960 pending_rounds.cc:62] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Trying to abort 1 pending ops.
I20240503 02:06:17.768374 19960 pending_rounds.cc:68] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023419299912286208 op_type: NO_OP noop_request { }
I20240503 02:06:17.768776 19960 raft_consensus.cc:2866] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:06:17.769098 19960 raft_consensus.cc:2258] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:17.771962 19960 tablet_replica.cc:330] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: stopping tablet replica
I20240503 02:06:17.772742 19960 raft_consensus.cc:2229] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:17.773195 19960 raft_consensus.cc:2258] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:17.774976 19960 tablet_replica.cc:330] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: stopping tablet replica
I20240503 02:06:17.775429 19960 raft_consensus.cc:2229] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:17.775751 19960 raft_consensus.cc:2258] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:17.777344 19960 tablet_replica.cc:330] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08: stopping tablet replica
I20240503 02:06:17.777786 19960 raft_consensus.cc:2229] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:17.778442 19960 raft_consensus.cc:2258] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:17.780246 19960 tablet_replica.cc:330] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08: stopping tablet replica
I20240503 02:06:17.780686 19960 raft_consensus.cc:2229] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:17.781036 19960 raft_consensus.cc:2258] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:17.782922 19960 tablet_replica.cc:330] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08: stopping tablet replica
I20240503 02:06:17.783386 19960 raft_consensus.cc:2229] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:17.783694 19960 raft_consensus.cc:2258] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:17.785027 19960 tablet_replica.cc:330] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08: stopping tablet replica
I20240503 02:06:17.785430 19960 raft_consensus.cc:2229] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:17.786058 19960 raft_consensus.cc:2258] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:17.811283 19960 tablet_server.cc:194] TabletServer@127.19.126.1:0 shutdown complete.
I20240503 02:06:18.026305 25741 consensus_queue.cc:1035] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:06:18.039109 25525 consensus_peers.cc:485] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 -> Peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Couldn't send request to peer 698037111e64432092d44f7db2edfa08. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:18.070241 25689 consensus_queue.cc:1035] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:06:18.081583 25525 consensus_peers.cc:485] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 -> Peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Couldn't send request to peer 698037111e64432092d44f7db2edfa08. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:06:18.091310 25525 consensus_peers.cc:485] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 -> Peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Couldn't send request to peer 698037111e64432092d44f7db2edfa08. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:06:18.120173 25525 consensus_peers.cc:485] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 -> Peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Couldn't send request to peer 698037111e64432092d44f7db2edfa08. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:18.227139 25689 consensus_queue.cc:1035] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:18.572832 25689 consensus_queue.cc:579] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [LEADER]: Leader has been unable to successfully communicate with peer 698037111e64432092d44f7db2edfa08 for more than 1 seconds (1.055s)
I20240503 02:06:18.643702 25689 consensus_queue.cc:579] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [LEADER]: Leader has been unable to successfully communicate with peer 698037111e64432092d44f7db2edfa08 for more than 1 seconds (1.010s)
I20240503 02:06:18.695057 25689 consensus_queue.cc:579] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [LEADER]: Leader has been unable to successfully communicate with peer 698037111e64432092d44f7db2edfa08 for more than 1 seconds (1.023s)
I20240503 02:06:18.837525 25384 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42998:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240503 02:06:18.838027 25384 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42998:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240503 02:06:18.852023 25384 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c: Sending DeleteTablet for 3 replicas of tablet 83218d07a65646f4896c9baa9f87ea6a
I20240503 02:06:18.853503 25384 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c: Sending DeleteTablet for 3 replicas of tablet e2d5910838654d63839e42f51aa40753
I20240503 02:06:18.853948 25552 tablet_service.cc:1514] Processing DeleteTablet for tablet 83218d07a65646f4896c9baa9f87ea6a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:06:18 UTC) from {username='slave'} at 127.0.0.1:50798
I20240503 02:06:18.855172 25384 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c: Sending DeleteTablet for 3 replicas of tablet 78942ce15f0e4594930b3d4fbeb57c02
I20240503 02:06:18.855933 25550 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d5910838654d63839e42f51aa40753 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:06:18 UTC) from {username='slave'} at 127.0.0.1:50798
I20240503 02:06:18.856446 25624 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d5910838654d63839e42f51aa40753 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:06:18 UTC) from {username='slave'} at 127.0.0.1:39890
I20240503 02:06:18.856506 25623 tablet_service.cc:1514] Processing DeleteTablet for tablet 83218d07a65646f4896c9baa9f87ea6a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:06:18 UTC) from {username='slave'} at 127.0.0.1:39890
I20240503 02:06:18.857326 25764 tablet_replica.cc:330] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041: stopping tablet replica
I20240503 02:06:18.858088 25764 raft_consensus.cc:2229] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:18.858330 25765 tablet_replica.cc:330] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db: stopping tablet replica
I20240503 02:06:18.859079 25764 raft_consensus.cc:2258] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:18.859334 25765 raft_consensus.cc:2229] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:18.859963 25552 tablet_service.cc:1514] Processing DeleteTablet for tablet 78942ce15f0e4594930b3d4fbeb57c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:06:18 UTC) from {username='slave'} at 127.0.0.1:50798
I20240503 02:06:18.860018 25765 raft_consensus.cc:2258] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:18.861483 25384 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f057e928aab0437586b59e032600170c: Sending DeleteTablet for 3 replicas of tablet 9bb3d1846ab542f69676509de46edba9
I20240503 02:06:18.861994 25624 tablet_service.cc:1514] Processing DeleteTablet for tablet 78942ce15f0e4594930b3d4fbeb57c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:06:18 UTC) from {username='slave'} at 127.0.0.1:39890
I20240503 02:06:18.863317 25552 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb3d1846ab542f69676509de46edba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:06:18 UTC) from {username='slave'} at 127.0.0.1:50798
W20240503 02:06:18.863396 25369 catalog_manager.cc:4573] TS 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83218d07a65646f4896c9baa9f87ea6a: Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111)
I20240503 02:06:18.864449 25624 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb3d1846ab542f69676509de46edba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:06:18 UTC) from {username='slave'} at 127.0.0.1:39890
I20240503 02:06:18.863413 25765 ts_tablet_manager.cc:1875] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:06:18.865391 25764 ts_tablet_manager.cc:1875] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:06:18.867120 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:06:18.874313 25767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:18.877519 25765 ts_tablet_manager.cc:1888] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:06:18.877940 25765 log.cc:1196] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-2-root/wals/e2d5910838654d63839e42f51aa40753
I20240503 02:06:18.878748 25765 ts_tablet_manager.cc:1909] T e2d5910838654d63839e42f51aa40753 P 1d724b88dae941358c33ea9e56a022db: Deleting consensus metadata
I20240503 02:06:18.881814 25765 tablet_replica.cc:330] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db: stopping tablet replica
I20240503 02:06:18.881898 25371 catalog_manager.cc:4830] TS 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187): tablet e2d5910838654d63839e42f51aa40753 (table dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]) successfully deleted
W20240503 02:06:18.881856 25768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:18.883448 25764 ts_tablet_manager.cc:1888] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:06:18.883916 25764 log.cc:1196] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-1-root/wals/83218d07a65646f4896c9baa9f87ea6a
I20240503 02:06:18.883852 25765 raft_consensus.cc:2229] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:18.884946 25765 raft_consensus.cc:2258] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:18.885135 25764 ts_tablet_manager.cc:1909] T 83218d07a65646f4896c9baa9f87ea6a P a491687290b64a83b750b5506c9ba041: Deleting consensus metadata
I20240503 02:06:18.889431 25765 ts_tablet_manager.cc:1875] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:06:18.889261 25370 catalog_manager.cc:4830] TS a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473): tablet 83218d07a65646f4896c9baa9f87ea6a (table dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]) successfully deleted
I20240503 02:06:18.890223 25764 tablet_replica.cc:330] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041: stopping tablet replica
I20240503 02:06:18.891480 25764 raft_consensus.cc:2229] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:18.892858 25764 raft_consensus.cc:2258] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:18.897817 25764 ts_tablet_manager.cc:1875] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:06:18.898608 19960 server_base.cc:1034] running on GCE node
W20240503 02:06:18.899864 25770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:18.900890 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:18.901166 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:18.901398 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701978901380 us; error 0 us; skew 500 ppm
I20240503 02:06:18.902123 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:18.905336 19960 webserver.cc:460] Webserver started at http://127.19.126.4:32963/ using document root <none> and password file <none>
I20240503 02:06:18.905501 25765 ts_tablet_manager.cc:1888] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:06:18.905894 25765 log.cc:1196] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-2-root/wals/83218d07a65646f4896c9baa9f87ea6a
I20240503 02:06:18.905967 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:18.906286 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:06:18.906491 25765 ts_tablet_manager.cc:1909] T 83218d07a65646f4896c9baa9f87ea6a P 1d724b88dae941358c33ea9e56a022db: Deleting consensus metadata
I20240503 02:06:18.906599 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:06:18.908038 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-3-root/instance:
uuid: "dfd1304d660543258f12e618eaf8e401"
format_stamp: "Formatted at 2024-05-03 02:06:18 on dist-test-slave-8rwr"
I20240503 02:06:18.908723 25765 tablet_replica.cc:330] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db: stopping tablet replica
I20240503 02:06:18.908922 25371 catalog_manager.cc:4830] TS 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187): tablet 83218d07a65646f4896c9baa9f87ea6a (table dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]) successfully deleted
I20240503 02:06:18.909651 25765 raft_consensus.cc:2229] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:18.910161 25765 raft_consensus.cc:2258] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:18.912472 25765 ts_tablet_manager.cc:1875] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:06:18.913203 25764 ts_tablet_manager.cc:1888] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:06:18.913654 25764 log.cc:1196] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-1-root/wals/e2d5910838654d63839e42f51aa40753
I20240503 02:06:18.914505 25764 ts_tablet_manager.cc:1909] T e2d5910838654d63839e42f51aa40753 P a491687290b64a83b750b5506c9ba041: Deleting consensus metadata
I20240503 02:06:18.918164 25370 catalog_manager.cc:4830] TS a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473): tablet e2d5910838654d63839e42f51aa40753 (table dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]) successfully deleted
I20240503 02:06:18.919466 25764 tablet_replica.cc:330] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041: stopping tablet replica
I20240503 02:06:18.920521 25764 raft_consensus.cc:2229] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:18.921013 25764 raft_consensus.cc:2258] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:18.921190 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.005s
I20240503 02:06:18.923408 25764 ts_tablet_manager.cc:1875] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:06:18.925532 25775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:18.925678 25765 ts_tablet_manager.cc:1888] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:06:18.926055 25765 log.cc:1196] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-2-root/wals/78942ce15f0e4594930b3d4fbeb57c02
I20240503 02:06:18.926549 19960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:06:18.926647 25765 ts_tablet_manager.cc:1909] T 78942ce15f0e4594930b3d4fbeb57c02 P 1d724b88dae941358c33ea9e56a022db: Deleting consensus metadata
I20240503 02:06:18.926988 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-3-root
uuid: "dfd1304d660543258f12e618eaf8e401"
format_stamp: "Formatted at 2024-05-03 02:06:18 on dist-test-slave-8rwr"
I20240503 02:06:18.927388 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:18.928900 25765 tablet_replica.cc:330] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db: stopping tablet replica
I20240503 02:06:18.929057 25371 catalog_manager.cc:4830] TS 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187): tablet 78942ce15f0e4594930b3d4fbeb57c02 (table dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]) successfully deleted
I20240503 02:06:18.929792 25765 raft_consensus.cc:2229] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:18.930342 25765 raft_consensus.cc:2258] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:18.933169 25765 ts_tablet_manager.cc:1875] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:06:18.937894 25764 ts_tablet_manager.cc:1888] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:06:18.938315 25764 log.cc:1196] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-1-root/wals/78942ce15f0e4594930b3d4fbeb57c02
I20240503 02:06:18.938977 25764 ts_tablet_manager.cc:1909] T 78942ce15f0e4594930b3d4fbeb57c02 P a491687290b64a83b750b5506c9ba041: Deleting consensus metadata
I20240503 02:06:18.941817 25370 catalog_manager.cc:4830] TS a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473): tablet 78942ce15f0e4594930b3d4fbeb57c02 (table dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]) successfully deleted
I20240503 02:06:18.942778 25764 tablet_replica.cc:330] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041: stopping tablet replica
I20240503 02:06:18.943686 25764 raft_consensus.cc:2229] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:06:18.944689 25764 raft_consensus.cc:2258] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:18.948279 25765 ts_tablet_manager.cc:1888] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:06:18.948813 25765 log.cc:1196] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-2-root/wals/9bb3d1846ab542f69676509de46edba9
I20240503 02:06:18.949102 25764 ts_tablet_manager.cc:1875] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:06:18.949822 25765 ts_tablet_manager.cc:1909] T 9bb3d1846ab542f69676509de46edba9 P 1d724b88dae941358c33ea9e56a022db: Deleting consensus metadata
I20240503 02:06:18.952770 25371 catalog_manager.cc:4830] TS 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187): tablet 9bb3d1846ab542f69676509de46edba9 (table dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]) successfully deleted
I20240503 02:06:18.961305 25764 ts_tablet_manager.cc:1888] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:06:18.961697 25764 log.cc:1196] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-1-root/wals/9bb3d1846ab542f69676509de46edba9
I20240503 02:06:18.962436 25764 ts_tablet_manager.cc:1909] T 9bb3d1846ab542f69676509de46edba9 P a491687290b64a83b750b5506c9ba041: Deleting consensus metadata
I20240503 02:06:18.964991 25370 catalog_manager.cc:4830] TS a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473): tablet 9bb3d1846ab542f69676509de46edba9 (table dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]) successfully deleted
I20240503 02:06:18.971735 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:18.972793 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:18.974823 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:06:18.977015 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:06:18.977182 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:18.977396 19960 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:06:18.977533 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:19.016899 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.4:43149
I20240503 02:06:19.016983 25837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.4:43149 every 8 connection(s)
I20240503 02:06:19.020900 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:06:19.027009 25689 consensus_queue.cc:579] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [LEADER]: Leader has been unable to successfully communicate with peer 698037111e64432092d44f7db2edfa08 for more than 1 seconds (1.282s)
W20240503 02:06:19.027861 25841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:06:19.029955 25842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:19.038082 25838 heartbeater.cc:347] Connected to a master server at 127.19.126.62:44719
I20240503 02:06:19.038547 25838 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:19.039340 25838 heartbeater.cc:511] Master 127.19.126.62:44719 requested a full tablet report, sending...
W20240503 02:06:19.040356 25844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:06:19.041217 25384 ts_manager.cc:194] Registered new tserver with Master: dfd1304d660543258f12e618eaf8e401 (127.19.126.4:43149)
I20240503 02:06:19.042598 19960 server_base.cc:1034] running on GCE node
I20240503 02:06:19.042863 25384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43904
I20240503 02:06:19.043685 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:06:19.043959 19960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:06:19.044179 19960 hybrid_clock.cc:648] HybridClock initialized: now 1714701979044160 us; error 0 us; skew 500 ppm
I20240503 02:06:19.044728 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:06:19.047003 19960 webserver.cc:460] Webserver started at http://127.19.126.1:42207/ using document root <none> and password file <none>
I20240503 02:06:19.047430 19960 fs_manager.cc:359] Metadata directory not provided
I20240503 02:06:19.047595 19960 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 02:06:19.051179 19960 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:06:19.054594 25849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:06:19.055413 19960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:06:19.055691 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root
uuid: "698037111e64432092d44f7db2edfa08"
format_stamp: "Formatted at 2024-05-03 02:06:15 on dist-test-slave-8rwr"
I20240503 02:06:19.056016 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:06:19.075521 25560 consensus_queue.cc:237] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } }
I20240503 02:06:19.082038 25634 raft_consensus.cc:1261] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Refusing update from remote peer a491687290b64a83b750b5506c9ba041: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:06:19.083439 25689 consensus_queue.cc:1035] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:19.086151 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:06:19.087684 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:06:19.096168 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
W20240503 02:06:19.099848 25525 consensus_peers.cc:485] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 -> Peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Couldn't send request to peer 698037111e64432092d44f7db2edfa08. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:19.103459 25860 ts_tablet_manager.cc:522] Loading tablet metadata (0/8 complete)
I20240503 02:06:19.112996 25736 raft_consensus.cc:2932] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
I20240503 02:06:19.114898 25634 raft_consensus.cc:2932] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
I20240503 02:06:19.119231 25689 raft_consensus.cc:482] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 698037111e64432092d44f7db2edfa08)
I20240503 02:06:19.119765 25689 raft_consensus.cc:504] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:19.121763 25689 leader_election.cc:288] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:19.123247 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b95aac25b414e8054bdc8f1c8ce5e" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:19.123397 25370 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 39d38439e4f746659d32731924b4c05b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 2)
W20240503 02:06:19.125841 25525 leader_election.cc:334] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111)
I20240503 02:06:19.126317 25525 leader_election.cc:302] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a491687290b64a83b750b5506c9ba041; no voters: 1d724b88dae941358c33ea9e56a022db, 698037111e64432092d44f7db2edfa08
W20240503 02:06:19.128818 25525 consensus_peers.cc:485] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 -> Peer dfd1304d660543258f12e618eaf8e401 (127.19.126.4:43149): Couldn't send request to peer dfd1304d660543258f12e618eaf8e401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39d38439e4f746659d32731924b4c05b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:19.129559 25736 raft_consensus.cc:2726] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:06:19.132737 25384 catalog_manager.cc:5484] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 reported cstate change: config changed from index -1 to 2, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New cstate: current_term: 1 leader_uuid: "a491687290b64a83b750b5506c9ba041" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:19.146533 25862 raft_consensus.cc:482] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 698037111e64432092d44f7db2edfa08)
I20240503 02:06:19.147038 25862 raft_consensus.cc:504] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:19.150931 25862 leader_election.cc:288] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473)
I20240503 02:06:19.152397 25560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b95aac25b414e8054bdc8f1c8ce5e" candidate_uuid: "1d724b88dae941358c33ea9e56a022db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041" is_pre_election: true
I20240503 02:06:19.154021 25560 raft_consensus.cc:2454] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d724b88dae941358c33ea9e56a022db in term 1.
I20240503 02:06:19.155606 25600 leader_election.cc:302] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d724b88dae941358c33ea9e56a022db, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:19.156412 25862 raft_consensus.cc:2781] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:06:19.156870 25862 raft_consensus.cc:482] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Starting leader election (detected failure of leader 698037111e64432092d44f7db2edfa08)
I20240503 02:06:19.157222 25862 raft_consensus.cc:3037] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:06:19.166184 25599 leader_election.cc:334] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111)
I20240503 02:06:19.167742 25862 raft_consensus.cc:504] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:19.169663 25862 leader_election.cc:288] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 2 election: Requested vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473)
I20240503 02:06:19.170647 25560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b95aac25b414e8054bdc8f1c8ce5e" candidate_uuid: "1d724b88dae941358c33ea9e56a022db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041"
I20240503 02:06:19.171423 25560 raft_consensus.cc:3037] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:06:19.173508 25599 leader_election.cc:334] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111)
I20240503 02:06:19.176052 19960 ts_tablet_manager.cc:559] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20240503 02:06:19.176292 19960 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.074s user 0.003s sys 0.000s
I20240503 02:06:19.176371 25560 raft_consensus.cc:2454] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d724b88dae941358c33ea9e56a022db in term 2.
I20240503 02:06:19.176553 19960 ts_tablet_manager.cc:574] Registering tablets (0/8 complete)
I20240503 02:06:19.177268 25600 leader_election.cc:302] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d724b88dae941358c33ea9e56a022db, a491687290b64a83b750b5506c9ba041; no voters: 698037111e64432092d44f7db2edfa08
I20240503 02:06:19.177919 25862 raft_consensus.cc:2781] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:06:19.178346 25862 raft_consensus.cc:686] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 2 LEADER]: Becoming Leader. State: Replica: 1d724b88dae941358c33ea9e56a022db, State: Running, Role: LEADER
I20240503 02:06:19.179057 25862 consensus_queue.cc:237] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:19.180954 25860 tablet_bootstrap.cc:492] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:19.185256 25384 catalog_manager.cc:5484] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db reported cstate change: term changed from 1 to 2, leader changed from 698037111e64432092d44f7db2edfa08 (127.19.126.1) to 1d724b88dae941358c33ea9e56a022db (127.19.126.3). New cstate: current_term: 2 leader_uuid: "1d724b88dae941358c33ea9e56a022db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:19.197657 25860 tablet_bootstrap.cc:492] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:19.198532 25860 tablet_bootstrap.cc:492] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08: Bootstrap complete.
I20240503 02:06:19.199132 25860 ts_tablet_manager.cc:1367] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20240503 02:06:19.199460 25736 raft_consensus.cc:482] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 698037111e64432092d44f7db2edfa08)
I20240503 02:06:19.199910 25736 raft_consensus.cc:504] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.201481 25736 leader_election.cc:288] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:19.201382 25860 raft_consensus.cc:348] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.202178 25860 raft_consensus.cc:729] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:19.203228 25862 raft_consensus.cc:482] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 698037111e64432092d44f7db2edfa08)
I20240503 02:06:19.203402 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:19.203747 25862 raft_consensus.cc:504] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.202924 25860 consensus_queue.cc:260] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.205587 25634 raft_consensus.cc:2454] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 1.
W20240503 02:06:19.207235 25525 leader_election.cc:334] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111)
I20240503 02:06:19.207778 25527 leader_election.cc:302] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d724b88dae941358c33ea9e56a022db, a491687290b64a83b750b5506c9ba041; no voters: 698037111e64432092d44f7db2edfa08
I20240503 02:06:19.208534 25862 leader_election.cc:288] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473)
I20240503 02:06:19.210027 25560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "1d724b88dae941358c33ea9e56a022db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041" is_pre_election: true
I20240503 02:06:19.210670 25560 raft_consensus.cc:2454] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d724b88dae941358c33ea9e56a022db in term 1.
I20240503 02:06:19.211478 25689 raft_consensus.cc:2781] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:06:19.211819 25689 raft_consensus.cc:482] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 698037111e64432092d44f7db2edfa08)
I20240503 02:06:19.211895 25600 leader_election.cc:302] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d724b88dae941358c33ea9e56a022db, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:19.212240 25689 raft_consensus.cc:3037] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:06:19.214577 25599 leader_election.cc:334] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111)
I20240503 02:06:19.214747 25862 raft_consensus.cc:2781] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:06:19.215070 25862 raft_consensus.cc:482] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Starting leader election (detected failure of leader 698037111e64432092d44f7db2edfa08)
I20240503 02:06:19.215390 25862 raft_consensus.cc:3037] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:19.220052 25689 raft_consensus.cc:504] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.221657 25862 raft_consensus.cc:504] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.224049 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db"
I20240503 02:06:19.224792 25689 leader_election.cc:288] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 2 election: Requested vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:19.225807 25634 raft_consensus.cc:2379] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a491687290b64a83b750b5506c9ba041 in current term 2: Already voted for candidate 1d724b88dae941358c33ea9e56a022db in this term.
W20240503 02:06:19.226253 25525 leader_election.cc:334] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111)
I20240503 02:06:19.226939 25860 ts_tablet_manager.cc:1398] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.027s user 0.020s sys 0.008s
I20240503 02:06:19.227459 25527 leader_election.cc:302] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a491687290b64a83b750b5506c9ba041; no voters: 1d724b88dae941358c33ea9e56a022db, 698037111e64432092d44f7db2edfa08
I20240503 02:06:19.227777 25860 tablet_bootstrap.cc:492] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:19.228578 25689 raft_consensus.cc:2726] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:06:19.230978 25560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "1d724b88dae941358c33ea9e56a022db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041"
I20240503 02:06:19.231577 19960 ts_tablet_manager.cc:590] Registered 8 tablets
I20240503 02:06:19.231748 25560 raft_consensus.cc:2379] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1d724b88dae941358c33ea9e56a022db in current term 2: Already voted for candidate a491687290b64a83b750b5506c9ba041 in this term.
I20240503 02:06:19.231860 19960 ts_tablet_manager.cc:569] Time spent register tablets: real 0.055s user 0.045s sys 0.003s
I20240503 02:06:19.230998 25862 leader_election.cc:288] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 2 election: Requested vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473)
W20240503 02:06:19.233717 25599 leader_election.cc:334] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111)
I20240503 02:06:19.234241 25599 leader_election.cc:302] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d724b88dae941358c33ea9e56a022db; no voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041
I20240503 02:06:19.235383 25862 raft_consensus.cc:2726] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:06:19.236505 25560 consensus_queue.cc:237] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } }
I20240503 02:06:19.242709 25634 raft_consensus.cc:1261] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Refusing update from remote peer a491687290b64a83b750b5506c9ba041: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:06:19.243958 25689 consensus_queue.cc:1035] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:06:19.246521 25860 tablet_bootstrap.cc:492] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240503 02:06:19.246484 25525 consensus_peers.cc:485] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 -> Peer dfd1304d660543258f12e618eaf8e401 (127.19.126.4:43149): Couldn't send request to peer dfd1304d660543258f12e618eaf8e401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e7274d6c77140dbb8e72c15e56e0fff. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:19.247969 25860 tablet_bootstrap.cc:492] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08: Bootstrap complete.
I20240503 02:06:19.248777 25860 ts_tablet_manager.cc:1367] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.000s
W20240503 02:06:19.251842 25525 consensus_peers.cc:485] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 -> Peer 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): Couldn't send request to peer 698037111e64432092d44f7db2edfa08. Status: Network error: Client connection negotiation failed: client connection to 127.19.126.1:46025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:19.251829 25860 raft_consensus.cc:348] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:19.253496 25860 raft_consensus.cc:729] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:19.254416 25860 consensus_queue.cc:260] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } }
I20240503 02:06:19.256412 25736 raft_consensus.cc:2932] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
I20240503 02:06:19.257512 25860 ts_tablet_manager.cc:1398] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240503 02:06:19.258558 25860 tablet_bootstrap.cc:492] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:19.261883 25634 raft_consensus.cc:2932] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
I20240503 02:06:19.266562 25370 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6e7274d6c77140dbb8e72c15e56e0fff with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20240503 02:06:19.272320 25384 catalog_manager.cc:5484] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 reported cstate change: config changed from index -1 to 2, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New cstate: current_term: 1 leader_uuid: "a491687290b64a83b750b5506c9ba041" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:06:19.279434 25433 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 698037111e64432092d44f7db2edfa08 not available for placement
I20240503 02:06:19.285997 25860 tablet_bootstrap.cc:492] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:19.287041 25860 tablet_bootstrap.cc:492] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: Bootstrap complete.
I20240503 02:06:19.287914 25860 ts_tablet_manager.cc:1367] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.008s
I20240503 02:06:19.291069 25860 raft_consensus.cc:348] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.292074 25860 raft_consensus.cc:729] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:19.292913 25860 consensus_queue.cc:260] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.295492 25860 ts_tablet_manager.cc:1398] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20240503 02:06:19.296427 25860 tablet_bootstrap.cc:492] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:19.312297 25860 tablet_bootstrap.cc:492] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:06:19.313359 25860 tablet_bootstrap.cc:492] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Bootstrap complete.
I20240503 02:06:19.314070 25860 ts_tablet_manager.cc:1367] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20240503 02:06:19.316548 25860 raft_consensus.cc:348] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.317445 25860 raft_consensus.cc:729] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:19.318266 25860 consensus_queue.cc:260] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.320137 19960 rpc_server.cc:307] RPC server started. Bound to: 127.19.126.1:46025
I20240503 02:06:19.320209 25928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.126.1:46025 every 8 connection(s)
I20240503 02:06:19.320453 25860 ts_tablet_manager.cc:1398] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:06:19.321221 25860 tablet_bootstrap.cc:492] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
W20240503 02:06:19.326347 25419 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.19.126.62:44719: Invalid argument: socket address is ambiguous
I20240503 02:06:19.331667 25860 tablet_bootstrap.cc:492] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:19.332280 25860 tablet_bootstrap.cc:492] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: Bootstrap complete.
I20240503 02:06:19.333031 25860 ts_tablet_manager.cc:1367] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240503 02:06:19.334911 25860 raft_consensus.cc:348] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.335510 25860 raft_consensus.cc:729] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:19.335806 25929 heartbeater.cc:347] Connected to a master server at 127.19.126.62:44719
I20240503 02:06:19.336160 25929 heartbeater.cc:464] Registering TS with master...
I20240503 02:06:19.336081 25860 consensus_queue.cc:260] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.336839 25929 heartbeater.cc:511] Master 127.19.126.62:44719 requested a full tablet report, sending...
I20240503 02:06:19.337568 25860 ts_tablet_manager.cc:1398] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 02:06:19.338135 25860 tablet_bootstrap.cc:492] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:19.341531 25383 ts_manager.cc:194] Re-registered known tserver with Master: 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025)
I20240503 02:06:19.348387 25860 tablet_bootstrap.cc:492] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:19.349782 25860 tablet_bootstrap.cc:492] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: Bootstrap complete.
I20240503 02:06:19.350458 25860 ts_tablet_manager.cc:1367] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20240503 02:06:19.352516 25383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43912
I20240503 02:06:19.352610 25860 raft_consensus.cc:348] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.353361 25860 raft_consensus.cc:729] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:19.354018 25860 consensus_queue.cc:260] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.355302 25929 heartbeater.cc:503] Master 127.19.126.62:44719 was elected leader, sending a full tablet report...
I20240503 02:06:19.355379 25860 ts_tablet_manager.cc:1398] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240503 02:06:19.356034 25860 tablet_bootstrap.cc:492] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:19.368762 25860 tablet_bootstrap.cc:492] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:19.369774 25860 tablet_bootstrap.cc:492] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08: Bootstrap complete.
I20240503 02:06:19.370379 25860 ts_tablet_manager.cc:1367] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240503 02:06:19.372201 25860 raft_consensus.cc:348] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.372654 25860 raft_consensus.cc:729] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:19.373167 25860 consensus_queue.cc:260] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.375028 25860 ts_tablet_manager.cc:1398] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:06:19.375733 25860 tablet_bootstrap.cc:492] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08: Bootstrap starting.
I20240503 02:06:19.385915 25860 tablet_bootstrap.cc:492] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:19.386673 25860 tablet_bootstrap.cc:492] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08: Bootstrap complete.
I20240503 02:06:19.387296 25860 ts_tablet_manager.cc:1367] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240503 02:06:19.388844 25860 raft_consensus.cc:348] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.389330 25860 raft_consensus.cc:729] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698037111e64432092d44f7db2edfa08, State: Initialized, Role: FOLLOWER
I20240503 02:06:19.389809 25860 consensus_queue.cc:260] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:19.391469 25860 ts_tablet_manager.cc:1398] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:06:19.462647 25893 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d5910838654d63839e42f51aa40753 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:06:18 UTC) from {username='slave'} at 127.0.0.1:35868
I20240503 02:06:19.463930 25937 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:19.464566 25937 raft_consensus.cc:2229] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:19.464978 25937 raft_consensus.cc:2258] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:19.466920 25937 ts_tablet_manager.cc:1875] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:06:19.476927 25937 ts_tablet_manager.cc:1888] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:06:19.477219 25937 log.cc:1196] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root/wals/e2d5910838654d63839e42f51aa40753
I20240503 02:06:19.477756 25937 ts_tablet_manager.cc:1909] T e2d5910838654d63839e42f51aa40753 P 698037111e64432092d44f7db2edfa08: Deleting consensus metadata
I20240503 02:06:19.479880 25369 catalog_manager.cc:4830] TS 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): tablet e2d5910838654d63839e42f51aa40753 (table dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]) successfully deleted
I20240503 02:06:19.486686 25893 tablet_service.cc:1514] Processing DeleteTablet for tablet 83218d07a65646f4896c9baa9f87ea6a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:06:18 UTC) from {username='slave'} at 127.0.0.1:35868
I20240503 02:06:19.487766 25937 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:19.488399 25937 raft_consensus.cc:2229] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:19.488797 25937 raft_consensus.cc:2258] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:19.490725 25937 ts_tablet_manager.cc:1875] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:06:19.494577 25893 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb3d1846ab542f69676509de46edba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:06:18 UTC) from {username='slave'} at 127.0.0.1:35868
I20240503 02:06:19.501551 25937 ts_tablet_manager.cc:1888] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:06:19.501914 25937 log.cc:1196] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root/wals/83218d07a65646f4896c9baa9f87ea6a
I20240503 02:06:19.502420 25937 ts_tablet_manager.cc:1909] T 83218d07a65646f4896c9baa9f87ea6a P 698037111e64432092d44f7db2edfa08: Deleting consensus metadata
I20240503 02:06:19.504493 25937 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:19.504511 25369 catalog_manager.cc:4830] TS 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): tablet 83218d07a65646f4896c9baa9f87ea6a (table dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]) successfully deleted
I20240503 02:06:19.505410 25937 raft_consensus.cc:2229] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:19.505821 25937 raft_consensus.cc:2258] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:19.507793 25937 ts_tablet_manager.cc:1875] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:06:19.517741 25937 ts_tablet_manager.cc:1888] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 02:06:19.518052 25937 log.cc:1196] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root/wals/9bb3d1846ab542f69676509de46edba9
I20240503 02:06:19.518565 25937 ts_tablet_manager.cc:1909] T 9bb3d1846ab542f69676509de46edba9 P 698037111e64432092d44f7db2edfa08: Deleting consensus metadata
I20240503 02:06:19.520607 25369 catalog_manager.cc:4830] TS 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): tablet 9bb3d1846ab542f69676509de46edba9 (table dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]) successfully deleted
I20240503 02:06:19.566776 25938 ts_tablet_manager.cc:902] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: Initiating tablet copy from peer a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473)
I20240503 02:06:19.570082 25938 tablet_copy_client.cc:308] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: tablet copy: Beginning tablet copy session from remote peer at address 127.19.126.2:38473
I20240503 02:06:19.573468 25893 tablet_service.cc:1514] Processing DeleteTablet for tablet 78942ce15f0e4594930b3d4fbeb57c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:06:18 UTC) from {username='slave'} at 127.0.0.1:35868
I20240503 02:06:19.574179 25937 tablet_replica.cc:330] stopping tablet replica
I20240503 02:06:19.575031 25937 raft_consensus.cc:2229] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:06:19.575384 25937 pending_rounds.cc:62] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Trying to abort 1 pending ops.
I20240503 02:06:19.575634 25937 pending_rounds.cc:68] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023419299912286208 op_type: NO_OP noop_request { }
I20240503 02:06:19.576050 25937 raft_consensus.cc:2866] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:06:19.576398 25937 raft_consensus.cc:2258] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:06:19.578944 25937 ts_tablet_manager.cc:1875] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:06:19.581821 25572 tablet_copy_service.cc:140] P a491687290b64a83b750b5506c9ba041: Received BeginTabletCopySession request for tablet 39d38439e4f746659d32731924b4c05b from peer dfd1304d660543258f12e618eaf8e401 ({username='slave'} at 127.0.0.1:50836)
I20240503 02:06:19.582363 25572 tablet_copy_service.cc:161] P a491687290b64a83b750b5506c9ba041: Beginning new tablet copy session on tablet 39d38439e4f746659d32731924b4c05b from peer dfd1304d660543258f12e618eaf8e401 at {username='slave'} at 127.0.0.1:50836: session id = dfd1304d660543258f12e618eaf8e401-39d38439e4f746659d32731924b4c05b
I20240503 02:06:19.589767 25572 tablet_copy_source_session.cc:205] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:06:19.592301 25938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d38439e4f746659d32731924b4c05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:19.592753 25937 ts_tablet_manager.cc:1888] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 02:06:19.593088 25937 log.cc:1196] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-0-root/wals/78942ce15f0e4594930b3d4fbeb57c02
I20240503 02:06:19.593644 25937 ts_tablet_manager.cc:1909] T 78942ce15f0e4594930b3d4fbeb57c02 P 698037111e64432092d44f7db2edfa08: Deleting consensus metadata
I20240503 02:06:19.595677 25369 catalog_manager.cc:4830] TS 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025): tablet 78942ce15f0e4594930b3d4fbeb57c02 (table dugtrio [id=326206fb5b1f438baa4052d2ce1669b6]) successfully deleted
I20240503 02:06:19.602193 25938 tablet_copy_client.cc:783] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: tablet copy: Starting download of 0 data blocks...
I20240503 02:06:19.602686 25938 tablet_copy_client.cc:647] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:19.605533 25938 tablet_copy_client.cc:523] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:19.611157 25938 tablet_bootstrap.cc:492] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: Bootstrap starting.
I20240503 02:06:19.624497 25938 tablet_bootstrap.cc:492] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:19.625239 25938 tablet_bootstrap.cc:492] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: Bootstrap complete.
I20240503 02:06:19.626230 25938 ts_tablet_manager.cc:1367] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: Time spent bootstrapping tablet: real 0.015s user 0.015s sys 0.001s
I20240503 02:06:19.628314 25938 raft_consensus.cc:348] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } }
I20240503 02:06:19.628928 25938 raft_consensus.cc:729] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfd1304d660543258f12e618eaf8e401, State: Initialized, Role: LEARNER
I20240503 02:06:19.629452 25938 consensus_queue.cc:260] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } }
I20240503 02:06:19.631498 25838 heartbeater.cc:503] Master 127.19.126.62:44719 was elected leader, sending a full tablet report...
I20240503 02:06:19.631641 25938 ts_tablet_manager.cc:1398] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 02:06:19.633075 25572 tablet_copy_service.cc:342] P a491687290b64a83b750b5506c9ba041: Request end of tablet copy session dfd1304d660543258f12e618eaf8e401-39d38439e4f746659d32731924b4c05b received from {username='slave'} at 127.0.0.1:50836
I20240503 02:06:19.633932 25572 tablet_copy_service.cc:433] P a491687290b64a83b750b5506c9ba041: ending tablet copy session dfd1304d660543258f12e618eaf8e401-39d38439e4f746659d32731924b4c05b on tablet 39d38439e4f746659d32731924b4c05b with peer dfd1304d660543258f12e618eaf8e401
I20240503 02:06:19.654081 25903 raft_consensus.cc:3037] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:19.659191 25903 raft_consensus.cc:1261] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 2 FOLLOWER]: Refusing update from remote peer 1d724b88dae941358c33ea9e56a022db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:19.660349 25862 consensus_queue.cc:1035] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [LEADER]: Connected to new peer: Peer: permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:19.670879 25560 raft_consensus.cc:1261] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Refusing update from remote peer 1d724b88dae941358c33ea9e56a022db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:06:19.672732 25861 consensus_queue.cc:1035] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:06:19.688237 25736 consensus_queue.cc:1037] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:06:19.692991 25903 raft_consensus.cc:2932] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
I20240503 02:06:19.707469 25938 ts_tablet_manager.cc:902] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: Initiating tablet copy from peer a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473)
I20240503 02:06:19.711043 25938 tablet_copy_client.cc:308] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: tablet copy: Beginning tablet copy session from remote peer at address 127.19.126.2:38473
I20240503 02:06:19.712252 25572 tablet_copy_service.cc:140] P a491687290b64a83b750b5506c9ba041: Received BeginTabletCopySession request for tablet 6e7274d6c77140dbb8e72c15e56e0fff from peer dfd1304d660543258f12e618eaf8e401 ({username='slave'} at 127.0.0.1:50836)
I20240503 02:06:19.712663 25572 tablet_copy_service.cc:161] P a491687290b64a83b750b5506c9ba041: Beginning new tablet copy session on tablet 6e7274d6c77140dbb8e72c15e56e0fff from peer dfd1304d660543258f12e618eaf8e401 at {username='slave'} at 127.0.0.1:50836: session id = dfd1304d660543258f12e618eaf8e401-6e7274d6c77140dbb8e72c15e56e0fff
I20240503 02:06:19.717154 25572 tablet_copy_source_session.cc:205] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:06:19.719760 25938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7274d6c77140dbb8e72c15e56e0fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:19.729022 25938 tablet_copy_client.cc:783] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: tablet copy: Starting download of 0 data blocks...
I20240503 02:06:19.729454 25938 tablet_copy_client.cc:647] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:19.732594 25938 tablet_copy_client.cc:523] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:19.737931 25938 tablet_bootstrap.cc:492] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: Bootstrap starting.
I20240503 02:06:19.752553 25938 tablet_bootstrap.cc:492] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:19.753175 25938 tablet_bootstrap.cc:492] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: Bootstrap complete.
I20240503 02:06:19.753684 25938 ts_tablet_manager.cc:1367] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20240503 02:06:19.755513 25938 raft_consensus.cc:348] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } }
I20240503 02:06:19.756171 25938 raft_consensus.cc:729] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfd1304d660543258f12e618eaf8e401, State: Initialized, Role: LEARNER
I20240503 02:06:19.756748 25938 consensus_queue.cc:260] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } }
I20240503 02:06:19.758651 25938 ts_tablet_manager.cc:1398] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:06:19.760048 25572 tablet_copy_service.cc:342] P a491687290b64a83b750b5506c9ba041: Request end of tablet copy session dfd1304d660543258f12e618eaf8e401-6e7274d6c77140dbb8e72c15e56e0fff received from {username='slave'} at 127.0.0.1:50836
I20240503 02:06:19.760361 25572 tablet_copy_service.cc:433] P a491687290b64a83b750b5506c9ba041: ending tablet copy session dfd1304d660543258f12e618eaf8e401-6e7274d6c77140dbb8e72c15e56e0fff on tablet 6e7274d6c77140dbb8e72c15e56e0fff with peer dfd1304d660543258f12e618eaf8e401
I20240503 02:06:19.795881 25903 raft_consensus.cc:2932] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
I20240503 02:06:20.015811 25813 raft_consensus.cc:1203] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:06:20.152251 25560 consensus_queue.cc:237] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:20.157510 25634 raft_consensus.cc:1261] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Refusing update from remote peer a491687290b64a83b750b5506c9ba041: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:06:20.157657 25903 raft_consensus.cc:1261] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Refusing update from remote peer a491687290b64a83b750b5506c9ba041: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:06:20.158929 25741 consensus_queue.cc:1035] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:20.159574 25960 consensus_queue.cc:1035] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:20.165016 25741 raft_consensus.cc:2932] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } }
I20240503 02:06:20.166517 25903 raft_consensus.cc:2932] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } }
I20240503 02:06:20.171777 25634 raft_consensus.cc:2932] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } }
I20240503 02:06:20.173564 25370 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39d38439e4f746659d32731924b4c05b with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:06:20.177997 25383 catalog_manager.cc:5484] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 reported cstate change: config changed from index 2 to 3, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) evicted. New cstate: current_term: 1 leader_uuid: "a491687290b64a83b750b5506c9ba041" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:20.199584 25803 tablet_service.cc:1514] Processing DeleteTablet for tablet 39d38439e4f746659d32731924b4c05b with delete_type TABLET_DATA_TOMBSTONED (TS dfd1304d660543258f12e618eaf8e401 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44366
I20240503 02:06:20.200881 25968 tablet_replica.cc:330] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: stopping tablet replica
I20240503 02:06:20.201521 25968 raft_consensus.cc:2229] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 02:06:20.201958 25968 raft_consensus.cc:2258] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 02:06:20.203699 25968 ts_tablet_manager.cc:1875] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:06:20.213194 25968 ts_tablet_manager.cc:1888] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:06:20.213529 25968 log.cc:1196] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-3-root/wals/39d38439e4f746659d32731924b4c05b
I20240503 02:06:20.214722 25369 catalog_manager.cc:4830] TS dfd1304d660543258f12e618eaf8e401 (127.19.126.4:43149): tablet 39d38439e4f746659d32731924b4c05b (table test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]) successfully deleted
I20240503 02:06:20.298998 25560 consensus_queue.cc:237] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } attrs { replace: true } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } }
I20240503 02:06:20.303602 25634 raft_consensus.cc:1261] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Refusing update from remote peer a491687290b64a83b750b5506c9ba041: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:06:20.304147 25525 consensus_peers.cc:485] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 -> Peer dfd1304d660543258f12e618eaf8e401 (127.19.126.4:43149): Couldn't send request to peer dfd1304d660543258f12e618eaf8e401. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:20.304139 25903 raft_consensus.cc:1261] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Refusing update from remote peer a491687290b64a83b750b5506c9ba041: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:06:20.304797 25741 consensus_queue.cc:1035] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:20.305718 25736 consensus_queue.cc:1035] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:06:20.311184 25741 raft_consensus.cc:2932] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } attrs { replace: true } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
I20240503 02:06:20.313119 25634 raft_consensus.cc:2932] T 39d38439e4f746659d32731924b4c05b P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } attrs { replace: true } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
I20240503 02:06:20.314106 25903 raft_consensus.cc:2932] T 39d38439e4f746659d32731924b4c05b P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } attrs { replace: true } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
I20240503 02:06:20.323148 25385 catalog_manager.cc:5484] T 39d38439e4f746659d32731924b4c05b P a491687290b64a83b750b5506c9ba041 reported cstate change: config changed from index 3 to 4, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New cstate: current_term: 1 leader_uuid: "a491687290b64a83b750b5506c9ba041" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:06:20.330305 25813 raft_consensus.cc:1203] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:06:20.338531 25634 consensus_queue.cc:237] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } attrs { replace: true } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } }
I20240503 02:06:20.344071 25560 raft_consensus.cc:1261] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Refusing update from remote peer 1d724b88dae941358c33ea9e56a022db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:06:20.345562 25970 consensus_queue.cc:1035] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:20.346216 25903 raft_consensus.cc:1261] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 2 FOLLOWER]: Refusing update from remote peer 1d724b88dae941358c33ea9e56a022db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:06:20.348639 25970 consensus_queue.cc:1035] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [LEADER]: Connected to new peer: Peer: permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:20.351907 25861 raft_consensus.cc:2932] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } attrs { replace: true } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
I20240503 02:06:20.356426 25903 raft_consensus.cc:2932] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } attrs { replace: true } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
W20240503 02:06:20.359184 25599 consensus_peers.cc:485] T ce3b95aac25b414e8054bdc8f1c8ce5e P 1d724b88dae941358c33ea9e56a022db -> Peer dfd1304d660543258f12e618eaf8e401 (127.19.126.4:43149): Couldn't send request to peer dfd1304d660543258f12e618eaf8e401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce3b95aac25b414e8054bdc8f1c8ce5e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:20.358366 25560 raft_consensus.cc:2932] T ce3b95aac25b414e8054bdc8f1c8ce5e P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } attrs { replace: true } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
I20240503 02:06:20.369993 25384 catalog_manager.cc:5484] T ce3b95aac25b414e8054bdc8f1c8ce5e P 698037111e64432092d44f7db2edfa08 reported cstate change: config changed from index -1 to 3, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) added. New cstate: current_term: 2 leader_uuid: "1d724b88dae941358c33ea9e56a022db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } attrs { replace: true } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } } }
W20240503 02:06:20.373454 25433 auto_rebalancer.cc:254] failed to send replica move request: Invalid argument: Server with UUID dfd1304d660543258f12e618eaf8e401 is already a member of the config. RaftConfig: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } peers { permanent_uuid: "dfd1304d660543258f12e618eaf8e401" member_type: NON_VOTER last_known_addr { host: "127.19.126.4" port: 43149 } attrs { promote: true } }
I20240503 02:06:20.378168 25560 consensus_queue.cc:237] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:20.383821 25634 raft_consensus.cc:1261] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Refusing update from remote peer a491687290b64a83b750b5506c9ba041: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:06:20.383951 25903 raft_consensus.cc:1261] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Refusing update from remote peer a491687290b64a83b750b5506c9ba041: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:06:20.385437 25960 consensus_queue.cc:1035] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:06:20.386050 25736 consensus_queue.cc:1035] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:06:20.391160 25433 auto_rebalancer.cc:663] Could not move replica: Illegal state: 6e7274d6c77140dbb8e72c15e56e0fff: source replica 698037111e64432092d44f7db2edfa08 does not have REPLACE attribute set
W20240503 02:06:20.391494 25433 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: 6e7274d6c77140dbb8e72c15e56e0fff: source replica 698037111e64432092d44f7db2edfa08 does not have REPLACE attribute set
I20240503 02:06:20.392477 25960 raft_consensus.cc:2932] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } }
I20240503 02:06:20.394798 25903 raft_consensus.cc:2932] T 6e7274d6c77140dbb8e72c15e56e0fff P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } }
I20240503 02:06:20.401916 25370 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e7274d6c77140dbb8e72c15e56e0fff with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:06:20.403918 25635 raft_consensus.cc:2932] T 6e7274d6c77140dbb8e72c15e56e0fff P 1d724b88dae941358c33ea9e56a022db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } } }
I20240503 02:06:20.405265 25385 catalog_manager.cc:5484] T 6e7274d6c77140dbb8e72c15e56e0fff P a491687290b64a83b750b5506c9ba041 reported cstate change: config changed from index 2 to 3, NON_VOTER dfd1304d660543258f12e618eaf8e401 (127.19.126.4) evicted. New cstate: current_term: 1 leader_uuid: "a491687290b64a83b750b5506c9ba041" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: HEALTHY } } }
I20240503 02:06:20.415952 25803 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e7274d6c77140dbb8e72c15e56e0fff with delete_type TABLET_DATA_TOMBSTONED (TS dfd1304d660543258f12e618eaf8e401 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44366
I20240503 02:06:20.416821 25968 tablet_replica.cc:330] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: stopping tablet replica
I20240503 02:06:20.417560 25968 raft_consensus.cc:2229] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 02:06:20.418043 25968 raft_consensus.cc:2258] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 02:06:20.421113 25968 ts_tablet_manager.cc:1875] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:06:20.434374 25968 ts_tablet_manager.cc:1888] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:06:20.434700 25968 log.cc:1196] T 6e7274d6c77140dbb8e72c15e56e0fff P dfd1304d660543258f12e618eaf8e401: Deleting WAL directory at /tmp/dist-test-task0XqAtE/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714701820461004-19960-0/minicluster-data/ts-3-root/wals/6e7274d6c77140dbb8e72c15e56e0fff
I20240503 02:06:20.435964 25369 catalog_manager.cc:4830] TS dfd1304d660543258f12e618eaf8e401 (127.19.126.4:43149): tablet 6e7274d6c77140dbb8e72c15e56e0fff (table test-workload [id=bb93520bb1ae4392a32bdcbd9145ad35]) successfully deleted
I20240503 02:06:20.618404 25867 raft_consensus.cc:482] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:20.618865 25867 raft_consensus.cc:504] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:20.621078 25867 leader_election.cc:288] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:20.632136 25560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a491687290b64a83b750b5506c9ba041" is_pre_election: true
I20240503 02:06:20.632969 25560 raft_consensus.cc:2379] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 698037111e64432092d44f7db2edfa08 in current term 2: Already voted for candidate a491687290b64a83b750b5506c9ba041 in this term.
I20240503 02:06:20.633427 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "698037111e64432092d44f7db2edfa08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:20.634178 25635 raft_consensus.cc:2379] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 698037111e64432092d44f7db2edfa08 in current term 2: Already voted for candidate 1d724b88dae941358c33ea9e56a022db in this term.
I20240503 02:06:20.635205 25852 leader_election.cc:302] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 698037111e64432092d44f7db2edfa08; no voters: 1d724b88dae941358c33ea9e56a022db, a491687290b64a83b750b5506c9ba041
I20240503 02:06:20.636334 25867 raft_consensus.cc:3037] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:06:20.641968 25867 raft_consensus.cc:2726] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:06:20.835273 25979 ts_tablet_manager.cc:902] T ce3b95aac25b414e8054bdc8f1c8ce5e P dfd1304d660543258f12e618eaf8e401: Initiating tablet copy from peer 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
F20240503 02:06:20.836144 25979 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0e73a484b6e9487ebd53af9983a857b0.queue_time_us, tablet-download-pool-0e73a484b6e9487ebd53af9983a857b0.queue_time_us) (tablet-download-pool-0e73a484b6e9487ebd53af9983a857b0.run_cpu_time_us, tablet-download-pool-0e73a484b6e9487ebd53af9983a857b0.run_cpu_time_us) (tablet-download-pool-0e73a484b6e9487ebd53af9983a857b0.run_wall_time_us, tablet-download-pool-0e73a484b6e9487ebd53af9983a857b0.run_wall_time_us) (tablet-download-pool-18443df96de94fdc930859b89a06e13e.queue_time_us, tablet-download-pool-18443df96de94fdc930859b89a06e13e.queue_time_us) (tablet-download-pool-18443df96de94fdc930859b89a06e13e.run_cpu_time_us, tablet-download-pool-18443df96de94fdc930859b89a06e13e.run_cpu_time_us) (tablet-download-pool-18443df96de94fdc930859b89a06e13e.run_wall_time_us, tablet-download-pool-18443df96de94fdc930859b89a06e13e.run_wall_time_us) (tablet-download-pool-24e820764537431a89c13520294ca9bf.queue_time_us, tablet-download-pool-24e820764537431a89c13520294ca9bf.queue_time_us) (tablet-download-pool-24e820764537431a89c13520294ca9bf.run_cpu_time_us, tablet-download-pool-24e820764537431a89c13520294ca9bf.run_cpu_time_us) (tablet-download-pool-24e820764537431a89c13520294ca9bf.run_wall_time_us, tablet-download-pool-24e820764537431a89c13520294ca9bf.run_wall_time_us) (tablet-download-pool-300c03aa7f9e45cabea2330f78343504.queue_time_us, tablet-download-pool-300c03aa7f9e45cabea2330f78343504.queue_time_us) (tablet-download-pool-300c03aa7f9e45cabea2330f78343504.run_cpu_time_us, tablet-download-pool-300c03aa7f9e45cabea2330f78343504.run_cpu_time_us) (tablet-download-pool-300c03aa7f9e45cabea2330f78343504.run_wall_time_us, tablet-download-pool-300c03aa7f9e45cabea2330f78343504.run_wall_time_us) (tablet-download-pool-39d38439e4f746659d32731924b4c05b.queue_time_us, tablet-download-pool-39d38439e4f746659d32731924b4c05b.queue_time_us) (tablet-download-pool-39d38439e4f746659d32731924b4c05b.run_cpu_time_us, tablet-download-pool-39d38439e4f746659d32731924b4c05b.run_cpu_time_us) (tablet-download-pool-39d38439e4f746659d32731924b4c05b.run_wall_time_us, tablet-download-pool-39d38439e4f746659d32731924b4c05b.run_wall_time_us) (tablet-download-pool-41db5c769a474264805464c3418e346c.queue_time_us, tablet-download-pool-41db5c769a474264805464c3418e346c.queue_time_us) (tablet-download-pool-41db5c769a474264805464c3418e346c.run_cpu_time_us, tablet-download-pool-41db5c769a474264805464c3418e346c.run_cpu_time_us) (tablet-download-pool-41db5c769a474264805464c3418e346c.run_wall_time_us, tablet-download-pool-41db5c769a474264805464c3418e346c.run_wall_time_us) (tablet-download-pool-6e7274d6c77140dbb8e72c15e56e0fff.queue_time_us, tablet-download-pool-6e7274d6c77140dbb8e72c15e56e0fff.queue_time_us) (tablet-download-pool-6e7274d6c77140dbb8e72c15e56e0fff.run_cpu_time_us, tablet-download-pool-6e7274d6c77140dbb8e72c15e56e0fff.run_cpu_time_us) (tablet-download-pool-6e7274d6c77140dbb8e72c15e56e0fff.run_wall_time_us, tablet-download-pool-6e7274d6c77140dbb8e72c15e56e0fff.run_wall_time_us) (tablet-download-pool-926c9af7e67b48f88a9826a5430ddcb1.queue_time_us, tablet-download-pool-926c9af7e67b48f88a9826a5430ddcb1.queue_time_us) (tablet-download-pool-926c9af7e67b48f88a9826a5430ddcb1.run_cpu_time_us, tablet-download-pool-926c9af7e67b48f88a9826a5430ddcb1.run_cpu_time_us) (tablet-download-pool-926c9af7e67b48f88a9826a5430ddcb1.run_wall_time_us, tablet-download-pool-926c9af7e67b48f88a9826a5430ddcb1.run_wall_time_us) (tablet-download-pool-99a88e2ad895493394437673aead620c.queue_time_us, tablet-download-pool-99a88e2ad895493394437673aead620c.queue_time_us) (tablet-download-pool-99a88e2ad895493394437673aead620c.run_cpu_time_us, tablet-download-pool-99a88e2ad895493394437673aead620c.run_cpu_time_us) (tablet-download-pool-99a88e2ad895493394437673aead620c.run_wall_time_us, tablet-download-pool-99a88e2ad895493394437673aead620c.run_wall_time_us) (tablet-download-pool-b1d852fa67354e9696275e159f12b5ce.queue_time_us, tablet-download-pool-b1d852fa67354e9696275e159f12b5ce.queue_time_us) (tablet-download-pool-b1d852fa67354e9696275e159f12b5ce.run_cpu_time_us, tablet-download-pool-b1d852fa67354e9696275e159f12b5ce.run_cpu_time_us) (tablet-download-pool-b1d852fa67354e9696275e159f12b5ce.run_wall_time_us, tablet-download-pool-b1d852fa67354e9696275e159f12b5ce.run_wall_time_us) (tablet-download-pool-ce3b95aac25b414e8054bdc8f1c8ce5e.queue_time_us, tablet-download-pool-ce3b95aac25b414e8054bdc8f1c8ce5e.queue_time_us) (tablet-download-pool-e1e04891c2b04ec288f8fe077265c900.queue_time_us, tablet-download-pool-e1e04891c2b04ec288f8fe077265c900.queue_time_us) (tablet-download-pool-e1e04891c2b04ec288f8fe077265c900.run_cpu_time_us, tablet-download-pool-e1e04891c2b04ec288f8fe077265c900.run_cpu_time_us) (tablet-download-pool-e1e04891c2b04ec288f8fe077265c900.run_wall_time_us, tablet-download-pool-e1e04891c2b04ec288f8fe077265c900.run_wall_time_us) (tablet-download-pool-ef0a9169ad3c42a380f97acda754f322.queue_time_us, tablet-download-pool-ef0a9169ad3c42a380f97acda754f322.queue_time_us) (tablet-download-pool-ef0a9169ad3c42a380f97acda754f322.run_cpu_time_us, tablet-download-pool-ef0a9169ad3c42a380f97acda754f322.run_cpu_time_us) (tablet-download-pool-ef0a9169ad3c42a380f97acda754f322.run_wall_time_us, tablet-download-pool-ef0a9169ad3c42a380f97acda754f322.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714701980 (unix time) try "date -d @1714701980" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800004df8) received by PID 19960 (TID 0x7fd1b1ef5700) from PID 19960; stack trace: ***
@ 0x2e41a0 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e6683 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7fd25a897980 (unknown) at ??:0
I20240503 02:06:20.918877 25981 ts_tablet_manager.cc:902] T 39d38439e4f746659d32731924b4c05b P dfd1304d660543258f12e618eaf8e401: Initiating tablet copy from peer a491687290b64a83b750b5506c9ba041 (127.19.126.2:38473)
@ 0x7fd253e0bfb7 gsignal at ??:0
@ 0x7fd253e0d921 abort at ??:0
@ 0x2ea196 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7fd256c01da4 google::LogMessage::SendToLog() at ??:0
@ 0x7fd256c02950 google::LogMessage::Flush() at ??:0
@ 0x7fd256c07a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd25758f3c7 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fd25756594f kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fd2575630a5 kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 02:06:21.086334 25960 raft_consensus.cc:482] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:06:21.086926 25960 raft_consensus.cc:504] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:21.088971 25960 leader_election.cc:288] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:21.089835 25903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08" is_pre_election: true
I20240503 02:06:21.090636 25903 raft_consensus.cc:2454] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 2.
I20240503 02:06:21.090899 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db" is_pre_election: true
I20240503 02:06:21.091670 25634 raft_consensus.cc:2454] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 2.
I20240503 02:06:21.091872 25525 leader_election.cc:302] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698037111e64432092d44f7db2edfa08, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:21.093042 25960 raft_consensus.cc:2781] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:06:21.093402 25960 raft_consensus.cc:482] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:06:21.093724 25960 raft_consensus.cc:3037] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:21.099084 25960 raft_consensus.cc:504] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
I20240503 02:06:21.101454 25903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "698037111e64432092d44f7db2edfa08"
I20240503 02:06:21.101583 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520c0c559a714c588dc6b6a34533ce23" candidate_uuid: "a491687290b64a83b750b5506c9ba041" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d724b88dae941358c33ea9e56a022db"
I20240503 02:06:21.102052 25903 raft_consensus.cc:3037] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:21.102133 25635 raft_consensus.cc:3037] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:06:21.105465 25960 leader_election.cc:288] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 3 election: Requested vote from peers 698037111e64432092d44f7db2edfa08 (127.19.126.1:46025), 1d724b88dae941358c33ea9e56a022db (127.19.126.3:35187)
I20240503 02:06:21.106774 25635 raft_consensus.cc:2454] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 3.
I20240503 02:06:21.107764 25527 leader_election.cc:302] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d724b88dae941358c33ea9e56a022db, a491687290b64a83b750b5506c9ba041; no voters:
I20240503 02:06:21.108563 25960 raft_consensus.cc:2781] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:06:21.110162 25903 raft_consensus.cc:2454] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a491687290b64a83b750b5506c9ba041 in term 3.
I20240503 02:06:21.112258 25960 raft_consensus.cc:686] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [term 3 LEADER]: Becoming Leader. State: Replica: a491687290b64a83b750b5506c9ba041, State: Running, Role: LEADER
I20240503 02:06:21.113044 25960 consensus_queue.cc:237] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } }
@ 0x7fd25f383422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 02:06:21.126236 25383 catalog_manager.cc:5484] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 reported cstate change: term changed from 1 to 3, leader changed from 698037111e64432092d44f7db2edfa08 (127.19.126.1) to a491687290b64a83b750b5506c9ba041 (127.19.126.2). New cstate: current_term: 3 leader_uuid: "a491687290b64a83b750b5506c9ba041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a491687290b64a83b750b5506c9ba041" member_type: VOTER last_known_addr { host: "127.19.126.2" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 } health_report { overall_health: UNKNOWN } } }
@ 0x7fd25f389c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fd25f41f4ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fd25f4503cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fd25f4361ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fd25f43617a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
@ 0x7fd25f43610a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7fd25f4360d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fd25f4352fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x7fd2607dbda5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7fd2607dbbda std::__1::function<>::operator()() at ??:0
@ 0x7fd257567237 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd25756a692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd25756a64a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7fd25756a5da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7fd25756a5a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fd25756989e std::__1::__function::__func<>::operator()() at ??:0
I20240503 02:06:21.576630 25903 raft_consensus.cc:1261] T 520c0c559a714c588dc6b6a34533ce23 P 698037111e64432092d44f7db2edfa08 [term 3 FOLLOWER]: Refusing update from remote peer a491687290b64a83b750b5506c9ba041: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 02:06:21.578620 25960 consensus_queue.cc:1035] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "698037111e64432092d44f7db2edfa08" member_type: VOTER last_known_addr { host: "127.19.126.1" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
@ 0x7fd2607dbda5 std::__1::__function::__value_func<>::operator()() at ??:0
I20240503 02:06:21.607725 25634 raft_consensus.cc:1261] T 520c0c559a714c588dc6b6a34533ce23 P 1d724b88dae941358c33ea9e56a022db [term 3 FOLLOWER]: Refusing update from remote peer a491687290b64a83b750b5506c9ba041: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 02:06:21.609982 25960 consensus_queue.cc:1035] T 520c0c559a714c588dc6b6a34533ce23 P a491687290b64a83b750b5506c9ba041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d724b88dae941358c33ea9e56a022db" member_type: VOTER last_known_addr { host: "127.19.126.3" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
@ 0x7fd2607dbbda std::__1::function<>::operator()() at ??:0
@ 0x7fd257549937 kudu::Thread::SuperviseThread() at ??:0