2·
4 months agoI’m surprised that the model’s benchmark performance isn’t that much worse than the base given that it’s running with 1-bit quantization. Impressive.
Irdial
I make computers
- 0 Posts
- 2 Comments
Joined 8 months ago
Cake day: March 22nd, 2026
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
I’ve got an M4 Mac mini with 32 GB of unified memory, so I can run MoE models in the <40B parameter range quite comfortably. I use LM Studio as the backend with Hermes as the frontend. Works great for everyday chat, web search, and RAG.