Out of memory when using mergekit

#10

by kongzym - opened 10 days ago

Hey, great work @mlabonne

I am currently using mergkit to merge two 70B models on a node with 8*A800, but got out of memory issues.
Do you know mergekit support loading a model on different GPUs? I checked the code, but not find the related part.

Thanks in advance!

mlabonne

Owner 9 days ago

Hey, you shouldn't need any GPU to perform the merge. You might require additional RAM perhaps?

kongzym

9 days ago

Thanks for your reply! I am using merge-evolve, in which a CMA-ES was implemented. This requires GPUs, I guess.

mlabonne

Owner 9 days ago

Oh alright, sorry I'm not familiar with this technique

kongzym

9 days ago

It's okay. Thank you!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment