news.ycombinator.com
Untitled
Realistically, you're looking at least 2x DGX sparks to run this at a 2 bit quant, but quantization really lobotomizes models so it's just better to run DSv4 flash at full precision. 4x DGX sparks should let you run this at 4 bit at least and there are some folks who ran GLM 5.2 on this configuration in r/LocalLlama
Comments
3 preview comments · loading full threadLog in to h4cker, then connect Hacker News to publish comments.
For Flash there are some excellent Q2/Q4 hybrids. I know that model was QAT so it handles Q4 better but the meta on quantization seems to be shifting a little bit to be more intelligent about what exactly gets quantized.
How fast are 2x or 4x DGX? I only have one and am wondering what the benefits are of getting another. I feel I will be disappointed…
i run flash v4 at 2bit, its pretty great and on my tests against full model It didn't lose any capabilities. It just was thinking more. So you don't have the same efficiency.