GPU Memory increase after each prompt

by thanhpt93 - opened Apr 5

Hi, I loaded model with 15Gb ram, but after each prompt, gpu memory increase. How to release memory? thank you very much!

Tamnemtf

Apr 6

•

edited Apr 6

I suggest you calling torch.cuda.empty_cache() after every 2-3 prompts. It may not be the most effective solution, especially if the questions are not connected and the model needs to maintain context. Using a stronger GPU like the A100, A40, or L40 is a much better approach.

thanhpt93

Apr 7

@Tamnemtf Thank for your solution, it work

thanhpt93 changed discussion status to closed Apr 7

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment