@Jaward on Hugging Face: "Lightweight (nanoGPT) implementation of hybrid norm - an intuitive…"

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Jaward

posted an update Mar 9

Post

2591

Lightweight (nanoGPT) implementation of hybrid norm - an intuitive normalization method that combines the strength of both pre-norm (i.e QKV-norm in MHA) and post-norm in the feed-forward network.
Code: https://github.com/Jaykef/ai-algorithms/blob/main/hybrid_normalization.ipynb

In this post

Jaward Jaward Sesay