- Deployment
- Quantized Local Apps
Deployment
Quantized Local Apps
Use quantized variants of North Mini Code in local runtimes like llama.cpp, Ollama, LM Studio, Jan, and MLX.
If you want a smaller local footprint or a GUI-first workflow, start from the Hugging Face quantization browser instead of the full base weights.
The Hugging Face quantization list changes quickly. Treat the filtered browser page as the source of truth, and use the examples below as starting points rather than a fixed catalog.
Quantization browser
Filtered list of quantized North Mini Code variants on Hugging Face.
Local apps guide
Hugging Face overview for running Hub models through local apps.
GGUF guide
Background on GGUF-based local runtimes such as llama.cpp.
Ollama guide
Run GGUF checkpoints directly from Hugging Face with Ollama.
MLX guide
Apple silicon path for MLX-based local workflows.
llama.cpp-filtered view
Quantized variants already tagged for llama.cpp-compatible flows.
Example Variants Currently Listed
FP8
Cohere-maintained FP8 variant for reduced-memory deployment.
GGUF
Community GGUF route for llama.cpp-style local runtimes.
AWQ INT4
Community INT4 quantization for smaller GPU footprints.
MLX 4-bit
Apple silicon-oriented quantized path.
Inference from Hugging Face’s app and quantization pages: GGUF variants are the best starting point for llama.cpp, Ollama, LM Studio, and similar local apps; FP8/AWQ variants are more likely to fit GPU-oriented runtimes such as vLLM; MLX variants are the natural path on Apple silicon.