news.ycombinator.com

Untitled

deepllm · 0 points · 0 comments · 5 godzin temu

Realistically, you're looking at least 2x DGX sparks to run this at a 2 bit quant, but quantization really lobotomizes models so it's just better to run DSv4 flash at full precision. 4x DGX sparks should let you run this at 4 bit at least and there are some folks who ran GLM 5.2 on this configuration in r/LocalLlama

Comments

3 preview comments · loading full thread

colingauvin1 godzinę temu

For Flash there are some excellent Q2/Q4 hybrids. I know that model was QAT so it handles Q4 better but the meta on quantization seems to be shifting a little bit to be more intelligent about what exactly gets quantized.

teruakohatu5 godzin temu

How fast are 2x or 4x DGX? I only have one and am wondering what the benefits are of getting another. I feel I will be disappointed…

disiplus4 godziny temu

i run flash v4 at 2bit, its pretty great and on my tests against full model It didn't lose any capabilities. It just was thinking more. So you don't have the same efficiency.