Question

#1
by WesPro - opened

What do you mean by no GPU needed? Are you saying that you can run the model on 64gb RAM with just CPU?

No i believe it means that 64gb of ram was used to create the model. Feel free to correct me if im wrong @Kquant03

@rombodawg Yeah ok that makes more sense because only CPU for such a big model would be crazy...

Replete-AI org

yeah exactly, created it using only my CPU and RAM

Replete-AI org
β€’
edited Mar 24

What do you mean by no GPU needed? Are you saying that you can run the model on 64gb RAM with just CPU?

You need at least 1.6TB of ram/vram to run it

Replete-AI org

Basically you CAN run it on cpu if you have enough ram, but you either need something like nvidia EOS or alien tech to run it at comfortable speeds on cpu

Walmart-the-bag changed discussion status to closed

Game
How many org members can we get into 1 community post lol

Basically you CAN run it on cpu if you have enough ram, but you either need something like nvidia EOS or alien tech to run it at comfortable speeds on cpu

Are Nvidia Eos not GPU's? Is alien tech a joke or is that actually a name for something?

Replete-AI org

EOS is a supercomputer with like 6000 h100s

"alien tech" I meant in the literal sense of computers from some extraterrestrial race that has achieved computers far faster than we have

Among many new records and milestones, one in generative AI stands out: NVIDIA Eos β€” an AI supercomputer powered by a whopping 10,752 NVIDIA H100 Tensor Core GPUs and NVIDIA Quantum-2 InfiniBand networking β€” completed a training benchmark based on a GPT-3 model with 175 billion parameters trained on one billion tokens in just 3.9 minutes.

You'd need uh $721,803,023.91 USD to create your own EOS
PS. You'd need ten to to match the fastest supercomputer in the world though #^#

Replete-AI org

Among many new records and milestones, one in generative AI stands out: NVIDIA Eos β€” an AI supercomputer powered by a whopping 10,752 NVIDIA H100 Tensor Core GPUs and NVIDIA Quantum-2 InfiniBand networking β€” completed a training benchmark based on a GPT-3 model with 175 billion parameters trained on one billion tokens in just 3.9 minutes.

You'd need uh $721,803,023.91 USD to create your own EOS
PS. You'd need ten to to match the fastest supercomputer in the world though #^#

image.png

what about 115x the strength of EOS?

Kquant03 changed discussion status to open
Replete-AI org

thats gonna make me insane

Going off consumer cost that is roughly 1.4 million H100's
100,000,000,000 / 721,803,023 = 138.5
138.5 x 10752 = 1,489,152
Power usage?
1,489,152 x 350w (H100 tdp) = 521,203,200 Watts of tpd in the gpus alone
VRAM?
119.132 Petabytes

Replete-AI org

119.132 Petabytes of VRAM...wow lol

I'm just wondering if there will be smaller llama 3 versions, all the recent releases have been getting bigger and bigger. My 8gbs of vram is screaming for help.
But companies see
More parameter + More benchmark points
= More investors

Replete-AI org

Among many new records and milestones, one in generative AI stands out: NVIDIA Eos β€” an AI supercomputer powered by a whopping 10,752 NVIDIA H100 Tensor Core GPUs and NVIDIA Quantum-2 InfiniBand networking β€” completed a training benchmark based on a GPT-3 model with 175 billion parameters trained on one billion tokens in just 3.9 minutes.

You'd need uh $721,803,023.91 USD to create your own EOS
PS. You'd need ten to to match the fastest supercomputer in the world though #^#

Hold on let me check my coat pockets

Will a few strands of cat hair and dryer lint suffice?

Replete-AI org

Among many new records and milestones, one in generative AI stands out: NVIDIA Eos β€” an AI supercomputer powered by a whopping 10,752 NVIDIA H100 Tensor Core GPUs and NVIDIA Quantum-2 InfiniBand networking β€” completed a training benchmark based on a GPT-3 model with 175 billion parameters trained on one billion tokens in just 3.9 minutes.

You'd need uh $721,803,023.91 USD to create your own EOS
PS. You'd need ten to to match the fastest supercomputer in the world though #^#

Hold on let me check my coat pockets

Will a few strands of cat hair and dryer lint suffice?

We can tie it together and compress it until it becomes so dense that it starts creating energy which we can use for our machines

Replete-AI org

I'm just wondering if there will be smaller llama 3 versions, all the recent releases have been getting bigger and bigger. My 8gbs of vram is screaming for help.
But companies see
More parameter + More benchmark points
= More investors

microsoft with phi-2 tho :)

I think the companies will be tired of spending millions of dollars every time they want a model...I think the most important thing that will change in the next ten years is how we process and transfer information.

has anybody run inference with this model?

Replete-AI org

has anybody run inference with this model?

Honestly, this model is just an experiment to see what I could do locally with passthrough. I had a script to interleave every single layer of Phi-2-Super in order to double its size while retaining 98.6% performance...so I copy pasted it 115 times to see if my pc would break...it didn't...it just took 3 days to merge it and 4 days to upload it lol.

I don't know anyone who would be capable of inferencing with this model to be honest. No one in our team has done it.

Walmart-the-bag changed discussion status to closed

Sign up or log in to comment