Rendered at 14:08:18 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
srcreigh 17 hours ago [-]
Makes you realize how insane the M5 Ultra Mac Studio is. 1.2TB/s bandwidth 512GB memory. Its rated max power draw is just 480W. And it also has amazing M-series CPUs. It costs less than just one of these GPUs which each take 700W to run.
segmondy 16 hours ago [-]
If you want to see how impressive Nvidia is, serve 32 concurrent request on it and compare the same with the mac.
No comparison.
None.
srcreigh 15 hours ago [-]
There is comparison actually. I spent all day researching this a few days ago.
Memory-wise, the RTX PRO 6000 can barely hold two 1M context Qwen 3.8 27B models at 8 bit quantization at the same time. The 512GB M5 Ultra Mac Studio could hold around 14.
At such high concurrency, batch performance is usually limited more by memory bandwidth than compute. The RTX PRO 6000's memory bandwidth is just 50% faster than the M5 Ultra.
So yeah, I think if we are talking about many short context requests, sure. But if you are chewing through a backlog of coding tasks with Qwen overnight, they might actually be comparable.
Im sure in Nov when the M5 Ultra comes out we'll see a lot of interesting benchmarks.
16 hours ago [-]
qeternity 16 hours ago [-]
*TB/s
teaearlgraycold 17 hours ago [-]
These GPUs are extremely inflated in price because Nvidia effectively has a monopoly on hardware that is used to train models. Apple Silicon tends to have good inference software available but as soon as you want to train even a YOLO model bits and pieces fall back to software implementations. Try to train an LLM and it'll get even worse.
The M3 Ultra's GPU performance is around a 4070 Ti. The M5 Ultra more like a 5080. They're both amazing deals compared to Nvidia for local inference because of their massive pool of high bandwidth memory. But a single RTX PRO 6000 should be 2 or 3x the compute of an M5 Ultra.
segmondy 16 hours ago [-]
I can't stand Nvidia, but these GPUs are not inflated because of monopoly. They are expensive because demand exceeds supply. That's it. The world wants compute and we don't have enough of it!
teaearlgraycold 16 hours ago [-]
I said inflated. They would always be expensive. But they charge more than their competitors per flop and per byte because they’re the only manufacturer that can run CUDA. If they lost that moat their prices would go down a notch.
schaefer 18 hours ago [-]
> We currently have 14x nodes of CG480-S6053 ready to ship.
Oh, okay, so this is an ad.
I do still think it's well written and interesting... But if anything, it's just making me more curious about the newest generation of M5 Ultra. (and less and less interested in PCI-E Gen 5 anything)
baron3dl 18 hours ago [-]
You must be the only one 'round here without a stack of RTX PRO 6000s, 8 high, that you're not sure how to use.
schaefer 18 hours ago [-]
what I have is a DGX Spark to play and learn on, and an OpenRouter account for actual work.
I mostly rent my tokens.
baron3dl 17 hours ago [-]
that was an attempt at monocle and top hat irony.
edit: how do you feel about paying 5% on every token's cost to stripe? it made me cancel, and sign up directly with a few providers.
schaefer 17 hours ago [-]
> attempt at ... irony
For the record, I did laugh, I just didn't type it out.
> how do you feel about paying 5% on every token's cost to stripe?
Life is compromise. In a perfect world, I'd love to buy a linux native system comparable to the Mac Studio that could run something in the ballpark of Deepseek V4 flash "fast" at home.
Not only does that not exist, but even if it did I couldn't justify the cost. Between work and home my annual token spend is maybe 3k.
So... paying a 5% fee on a service that allows me to not shell out 10k on hardware at home still seems like a pretty sweet deal.
--
Plus, I still am learning a ton and playing lots. It's hard to imagine anything beating openrouter for that. So much exposure to all the latest LLMs...
RachelF 18 hours ago [-]
For those who can't afford RTX 6000's you can unlock around 20% increased card to card speed on consumer GPUs using this library:
The hardware supports it, but Nvidia disabled it if the driver detects cheaper cards.
jimmoores 17 hours ago [-]
These people have zero idea what they're doing. Not a single mention of pipeline parallelism that would actually make the setup useful to run a big model.
itkovian_ 17 hours ago [-]
I can’t stand it. Very engineering-y over specified formal language around a complete lack of core understanding. Is damaging other people read this and try to learn things from it.
schaefer 17 hours ago [-]
can you point to a write up that discusses what you're talking about?
because I would read it.
kelmoran 17 hours ago [-]
I feel like you would want to run 8 smaller models separately for quantity of raw output. 1 big model is slow and isnt guaranteed to make no mistakes.
monster_truck 17 hours ago [-]
That's not quite how it works. Throwing Deepseek V4 Flash on 4 of these would net you something like >200tk/s for 16 concurrent requests, that's 600 million _output_ tokens a month. Guess what happens when you use 8
teaearlgraycold 17 hours ago [-]
Doesn't really mean anything without a specific use case to guide model selection.
The thing is Qwen 3.8 27B can be ran on far far cheaper hardware. If you're spending the big bucks on these rigs you probably made the wrong choice if you aren't using models that require all of that VRAM.
kmike84 17 hours ago [-]
Pass. When articles keep mentioning models like DeepSeek R1, or Llama 3.1, or Qwen3 32B, it is a pretty robust indicator of AI slop. LLMs love to suggest DeepSeek R1, etc. - training data cut-off?
No person with real practical experience and real use cases will be using these ancient models as examples, when talking about local LLMs.
proxysna 17 hours ago [-]
Utterly awful article. R1? Llama3.1? Not being able to serve larger llms on 8(!) RTX pro’s? You can literally run open weight SOTA models with relative ease. Even 4 GPUs get you there with a bit of elbow grease and compression. Pure slop.
Avlin67 15 hours ago [-]
did you try p2p enabled driver and proper nccl env vars ?
xyst 17 hours ago [-]
SLI is relevant again
CamperBob2 17 hours ago [-]
4x RTX 6000 Blackwell cards is a good place to be if you can't swing 8 of them, or if you don't have the power or cooling to run that many. A system based on 4x RTX6K can run GLM 5.3 at NVFP4 precision [1] from a US-standard 120V 20A circuit when derated to 300W, and give you a better pelican than Fable 5.1 [2]. What's not to like?
(Edit: I'm mistaken here, the pelican didn't come from Flash on 4 cards but from the full GLM 5.3 model on 8. But the Flash model is still crazy good for its size.)
> A system based on 4x RTX6K can run GLM 5.3 at NVFP4 precision
It actually runs fine at FP8 on this hardware too, with the full 1M context.
CamperBob2 14 hours ago [-]
Flash will run on 4x, but at the time I ran that test there were no 4-card quants for the full 744B-A40B 5.3 model. There are now, though, with KLD figures close to the FP8 level. I need to do some more benchmarking to see if they live up to the hype.
nojs 12 hours ago [-]
Oh right, I was referring to flash. I haven’t tried these either, but the ones for 5.2 looked interesting.
christkv 18 hours ago [-]
600W * 8 just for the GPUs when maxed out (besides the cost). Def nothing for my home lab.
touisteur 17 hours ago [-]
I'm curious whether actual inference workloads actually push to 600W (and not 350W) and what the last 250W get you. Rare is the (generic gpu) workload where I get >5%, some rare light inference benchmarks up to 10%...
CamperBob2 14 hours ago [-]
They don't usually max out for single prompts, but concurrency in the form of multiple users and/or multiple agents will make them draw full power.
robotnikman 17 hours ago [-]
Now if only I could afford 8 RTX PRO 6000's
CamperBob2 17 hours ago [-]
Start with 4, see my other comment. The recent GLM, Qwen and DeepSeek releases are amazingly promising.
jplusequalt 17 hours ago [-]
>Start with 4
Sure, let me just buy $60,000 worth of GPUs to run a *quantized non-frontier model*.
For that price you could:
- put a down payment on a home in a large % of the US
- buy a brand new car in cash (possibly two!)
- take a long sabbatical and travel the world
- pay all 4 years or your child's college tuition
theodric 15 hours ago [-]
Yeah, but this is HN, and none of those things are gonna 10x your B2B SaaS startup, or help you maximize shareholder value for your employer
CamperBob2 14 hours ago [-]
I mean, I can't afford a new F80, but you don't see me going out of my way to post on /r/Ferrari about it. I seriously do not understand the point of these posts.
The topic of the story is running 8x RTX PRO 6000s, so whining about how much they cost or what you could/should buy instead is completely off-topic.
12 hours ago [-]
estebarb 17 hours ago [-]
Now, if I could afford 4...
varispeed 18 hours ago [-]
> (~9.2 million tokens node-wide at 4k context).
stopped reading after that. What 4k context would be usable for?
monster_truck 17 hours ago [-]
So much more than you'd realize!
That's a solid 2200 words to spend on operating parameters and conveying state, leaving a generous 700 word window for them to decide and respond in.
When the bonsai/prism 1bit models dropped and I saw how many prompts a minute I could get from a dusty m2 mini I started hooking it up to all sorts of shit, like a traffic simulator that translates the car state/surroundings/immediate goal into text, it responds with a seqeunce of actions defined in the system prompt, which then get translated back into NPC input.
What I was hoping for here was that it would result in fucking chaos, all sorts of stupid decisions and epic car accidents. I cannot overstate my disappointment (and terror) when they were perfectly reasonable, safe drivers. I had to cut the tire grip by 75% without telling them and make them control twice as many cars to delay their ability to respond before I saw anything resembling an enjoyable traffic accident.
erdaltoprak 17 hours ago [-]
Not long-horizon coding but for a lot of other things
like batch processes with structured outputs, quick checks/fixes, making sense of unstructured data etc..
inventor7777 17 hours ago [-]
This will really help when my 8 RTX PRO 6000s ship. /s
No comparison.
None.
Memory-wise, the RTX PRO 6000 can barely hold two 1M context Qwen 3.8 27B models at 8 bit quantization at the same time. The 512GB M5 Ultra Mac Studio could hold around 14.
At such high concurrency, batch performance is usually limited more by memory bandwidth than compute. The RTX PRO 6000's memory bandwidth is just 50% faster than the M5 Ultra.
So yeah, I think if we are talking about many short context requests, sure. But if you are chewing through a backlog of coding tasks with Qwen overnight, they might actually be comparable.
Im sure in Nov when the M5 Ultra comes out we'll see a lot of interesting benchmarks.
The M3 Ultra's GPU performance is around a 4070 Ti. The M5 Ultra more like a 5080. They're both amazing deals compared to Nvidia for local inference because of their massive pool of high bandwidth memory. But a single RTX PRO 6000 should be 2 or 3x the compute of an M5 Ultra.
Oh, okay, so this is an ad.
I do still think it's well written and interesting... But if anything, it's just making me more curious about the newest generation of M5 Ultra. (and less and less interested in PCI-E Gen 5 anything)
I mostly rent my tokens.
edit: how do you feel about paying 5% on every token's cost to stripe? it made me cancel, and sign up directly with a few providers.
For the record, I did laugh, I just didn't type it out.
> how do you feel about paying 5% on every token's cost to stripe?
Life is compromise. In a perfect world, I'd love to buy a linux native system comparable to the Mac Studio that could run something in the ballpark of Deepseek V4 flash "fast" at home.
Not only does that not exist, but even if it did I couldn't justify the cost. Between work and home my annual token spend is maybe 3k.
So... paying a 5% fee on a service that allows me to not shell out 10k on hardware at home still seems like a pretty sweet deal.
--
Plus, I still am learning a ton and playing lots. It's hard to imagine anything beating openrouter for that. So much exposure to all the latest LLMs...
https://github.com/aikitoria/open-gpu-kernel-modules
The hardware supports it, but Nvidia disabled it if the driver detects cheaper cards.
because I would read it.
The thing is Qwen 3.8 27B can be ran on far far cheaper hardware. If you're spending the big bucks on these rigs you probably made the wrong choice if you aren't using models that require all of that VRAM.
No person with real practical experience and real use cases will be using these ancient models as examples, when talking about local LLMs.
(Edit: I'm mistaken here, the pelican didn't come from Flash on 4 cards but from the full GLM 5.3 model on 8. But the Flash model is still crazy good for its size.)
1: https://huggingface.co/local-inference-lab/GLM-5.3-NVFP4
2: https://crimson-jeri-74.tiiny.site/
It actually runs fine at FP8 on this hardware too, with the full 1M context.
Sure, let me just buy $60,000 worth of GPUs to run a *quantized non-frontier model*.
For that price you could:
- put a down payment on a home in a large % of the US
- buy a brand new car in cash (possibly two!)
- take a long sabbatical and travel the world
- pay all 4 years or your child's college tuition
The topic of the story is running 8x RTX PRO 6000s, so whining about how much they cost or what you could/should buy instead is completely off-topic.
stopped reading after that. What 4k context would be usable for?
That's a solid 2200 words to spend on operating parameters and conveying state, leaving a generous 700 word window for them to decide and respond in.
When the bonsai/prism 1bit models dropped and I saw how many prompts a minute I could get from a dusty m2 mini I started hooking it up to all sorts of shit, like a traffic simulator that translates the car state/surroundings/immediate goal into text, it responds with a seqeunce of actions defined in the system prompt, which then get translated back into NPC input.
What I was hoping for here was that it would result in fucking chaos, all sorts of stupid decisions and epic car accidents. I cannot overstate my disappointment (and terror) when they were perfectly reasonable, safe drivers. I had to cut the tire grip by 75% without telling them and make them control twice as many cars to delay their ability to respond before I saw anything resembling an enjoyable traffic accident.