Voxta ExLlamaV2
Local LLM inference for GPTQ and EXL2 quantized models on consumer GPUs.
ExLlamaV2 is a Python inference library for running LLMs locally on consumer-grade GPUs, with strong support for GPTQ and EXL2 quantized models. Voxta installs the runtime automatically on first use.
Being removed. ExLlamaV2 is no longer developed, and Voxta drops it in the release after 1.11. Move to ExLlamaV3, which covers what it did.
Setup
Install the service in Voxta
Manage Modules → Add Modules → ExLlamaV2 → Install. Voxta installs the Python module and dependencies on first run (watch the Terminal for progress).
Pick a model
In the ExLlamaV2 config:
- Model — full path to a model file, or HuggingFace model name (e.g.
Loyal-Macaroni-Maid-7B-GPTQ). - Models Directory — where ExLlamaV2 stores downloaded models. Default
Data/HuggingFace.
(Optional) Tune presets and prompt formatting
- Preset for text generation — default sampling settings for replies.
- Preset for action inference — favors reliability over creativity.
- Preset for summarization — same.
- Prompt Formatting Template — auto-detect or pick manually.
(Optional) Label
If you'll run multiple ExLlamaV2 instances with different models, add a Label to distinguish them.