Private Business LLM on a Single GPU

Team track record (prior role) · AI / Team track record

A secure, business-oriented local LLM that keeps company data on premise.

From our team's track record: our AI lead contributed to this as part of an LLM team in a previous role. This was not a NezT Digital client engagement.

What our ai lead worked on

Data sanitisation and supervised fine-tuning.

GGUF packaging, metadata correction and local secure serving.

Benchmarking local inference and integrating a FastAPI proxy with a Next.js client.

Tech stack

Qwen2 base, QLoRA, Unsloth, GGUF, Ollama, FastAPI, Next.js

Impact

Benchmarked at about 196 tokens per second with 0.68s average latency on a single RTX 4070, using roughly 2.8 GB of VRAM. Business data stays on premise.