🔄 In a Training Loop

Dean Byrne PRO

Quazim0t0

·

dean-byrne-02a28b191

AI & ML interests

DaisyChainAI🌼 / SmallLM's / San Francisco / Open Source

Recent Activity

reacted to satgeze's post with 🤗 about 8 hours ago

First GGUF quants of Tencent's Hy3 (299B MoE), built before official llama.cpp support exists. Hy3 dropped ~30 hours ago with only MLX and MXFP4 quants, both datacenter-sized. So I converted it myself using a community llama.cpp fork that implements the hy_v3 architecture. What's in the repo: - IQ1_M (62GB, fits a 128GB MacBook), IQ2_M (90GB), Q2_K (101GB), all with 1M context baked in via YaRN - IQ quants are importance-matrix: bootstrap style. The static Q2_K ran RAM-resident to compute the imatrix, then IQ1_M and IQ2_M were requantized from the archived f16 with it - Fixed chat template (the stock one uses .format() calls llama.cpp's Jinja rejects) - Build instructions for the fork, including the two gotchas that cost me three build attempts Honesty section, because that is how these repos work: this is EXPERIMENTAL. Not needle-certified yet (1M is baked but unverified, certification ladder will be published either way). MTP layer exists in the checkpoint but no llama.cpp build can run hy_v3 MTP inference yet, so it is not included. Real gate outputs are on the card, misses and all, judge for yourself. https://huggingface.co/satgeze/Hy3-1M-GGUF Full quant ladder (Q3 through Q8) is mirroring to ModelScope for bigger hardware.

repliedto their post about 8 hours ago

Created research language model whose channel-mixing block is not an MLP. It is a differentiable Neighbour-Sensing fungal-colony-growth model: each token is expanded into a colony of hyphal tips that grow in a bounded latent region, sense a shared density field, and steer their own growth — the "MLP" is replaced by a few differentiable steps of colony growth, read back out into the hidden state. https://huggingface.co/Quazim0t0/Mycel-LM-79M Also the original SpikeWhale project — the one that sparked all the other SpikeWhale related projects. Every spiking primitive here is hand-written in plain PyTorch: the leaky integrate-and-fire (LIF) neuron dynamics, the fast-sigmoid surrogate gradient, and the backprop-through-time training loop. No snntorch, no spikingjelly, no norse, no bindsnet — the network is a genuine from-scratch SNN. https://huggingface.co/Quazim0t0/SpikeWhale-SNN-216M

repliedto their post about 9 hours ago

Created research language model whose channel-mixing block is not an MLP. It is a differentiable Neighbour-Sensing fungal-colony-growth model: each token is expanded into a colony of hyphal tips that grow in a bounded latent region, sense a shared density field, and steer their own growth — the "MLP" is replaced by a few differentiable steps of colony growth, read back out into the hidden state. https://huggingface.co/Quazim0t0/Mycel-LM-79M Also the original SpikeWhale project — the one that sparked all the other SpikeWhale related projects. Every spiking primitive here is hand-written in plain PyTorch: the leaky integrate-and-fire (LIF) neuron dynamics, the fast-sigmoid surrogate gradient, and the backprop-through-time training loop. No snntorch, no spikingjelly, no norse, no bindsnet — the network is a genuine from-scratch SNN. https://huggingface.co/Quazim0t0/SpikeWhale-SNN-216M

View all activity

Organizations

published an article 3 days ago

Article

Some notes on swapping out the "important" parts of neural nets

Quazim0t0

•

3 days ago

• 1