Offline Enterprise LLM on Edge Hardware

Team track record (prior role) · AI / Team track record

Private, on-premise AI for organisations that cannot send sensitive data to the cloud.

From our team's track record: contributed by our AI lead in a previous role, as part of a product team working with an overseas engineering team. This was not a NezT Digital client engagement.

What was built

An offline RAG system able to run on edge hardware with no cloud dependency, part of a wider private AI platform.

Reranker integration for better answer quality.

What our ai lead owned

Optimised inference on NVIDIA Jetson Orin NX using multi-threading and llama.cpp-based parallelism.

Converted Gemma 2B to GGUF and validated results; experimented with Llama 3.1 8B.

Tech stack

llama.cpp, GGUF, Gemma 2B, Llama 3.1 8B, NVIDIA Jetson Orin NX, RAG, Rerankers

Impact

RAG response time reduced from roughly 6 seconds to nearly 2 seconds on edge hardware. Sensitive data stays on premise.