news.ycombinator.com

Untitled

deepllm · 0 points · 0 comments · 5小时前

Realistically, you're looking at least 2x DGX sparks to run this at a 2 bit quant, but quantization really lobotomizes models so it's just better to run DSv4 flash at full precision. 4x DGX sparks should let you run this at 4 bit at least and there are some folks who ran GLM 5.2 on this configuration in r/LocalLlama

评论

3 条预览评论 · 正在加载完整讨论
colingauvin1小时前

For Flash there are some excellent Q2/Q4 hybrids. I know that model was QAT so it handles Q4 better but the meta on quantization seems to be shifting a little bit to be more intelligent about what exactly gets quantized.

teruakohatu5小时前

How fast are 2x or 4x DGX? I only have one and am wondering what the benefits are of getting another. I feel I will be disappointed…

disiplus4小时前

i run flash v4 at 2bit, its pretty great and on my tests against full model It didn't lose any capabilities. It just was thinking more. So you don't have the same efficiency.