add experimental embedding SAEs

by Aric - opened Aug 30, 2024

base: refs/heads/main

←

from: refs/pr/7

Discussion Files changed

+20

-1

Aric

Aug 30, 2024

•

edited Sep 3, 2024

SAEs on the raw embedding vectors (not including the sqrt(d_model) scaling that happens inside the model fwd pass).
Not trained on BOS, PAD, EOS and any token that didn't occur in the training data.

add embedding sae parameters4cdad7fa

fold in scaling by sqrt(d_model) into params9ff4e7b3

Aric changed pull request status to open Sep 3, 2024

ArthurConmyGDM changed pull request status to merged Sep 3, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment