news.ycombinator.com
Ask HN: MacBook vs. Dedicated GPU for LLM
For those who are using llms on macbook, Want to understand how macbook is different than dedicated GPU in running those models? and how to know how much a macbook is capable of running a model?
Comments
5 preview comments · loading full threadLog in to h4cker, then connect Hacker News to publish comments.
I asked a few of my friends that are ML engineers this question and all of them said to run the LLMs in the cloud with their infrastructure because it was going to be way faster. If you just want to tinker around I would look at @JSR_FDD's comment.
If you want a massive MacBook anyway then it's great. They are decent for local LLMs, awesome for local image models and it's a MacBook so AppleCare+ has your back. IMO it's a no brainer if you wanted a MacBook anyway but it's a poor choice if your reason to buy it is to run LLMs.
MacBooks with their unified memory behave like a slow GPU with enormous amount of video RAM. So you can run large smart models slowly. Dedicated GPUs have less video RAM so can run smaller less smart models quickly.
How bad is the token-per-second drop-off on the Mac once you scale up the context window? Does it hit a wall at 32K or 64k tokens?
My opinion is that you should wait for 6-12 months before making a purchase either way. Open weight models are getting good. With GLM 5.2 now chasing Opus, I'm very excited to see a smaller model's distillation. Plus, the OLED MacBook Pro should be released by then.