the int8 speed are very slow

by lucasjin - opened Nov 28, 2023

Nov 28, 2023

why

YShow

Apr 30, 2024

It may be related to your hardware, and in addition, you can try the following inference framework for acceleration, such as vllm

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment