Air-Gapped LLM Deployment: Transfer, Sizing and Checklist
An air-gapped LLM is a language model that runs on a network with no connection to the internet: no outbound traffic, no package registries, no license server, no telemetry. Everything it needs, from the model weights to the inference server to the operating system packages, has to be carried in through a controlled transfer process and verified on arrival.
The model itself does not care that it is offline. Once the weights are in GPU memory, inference is local computation. The hard part is everything around it: moving tens of gigabytes across the gap, proving they are the files you meant to bring, and keeping the system current without ever opening a connection.
This guide covers who needs that, the problems you will hit, how to size the hardware, and an ordered checklist for the deployment.
What air-gapped means for an LLM
NIST's glossary describes an air gap as an interface between two systems that "are not connected physically" and where "any logical connection is not automated" (NIST CSRC glossary). Data crosses only by hand, under human control.
For an LLM, that rules out defaults most tools assume:
pip installanddocker pullreaching public registries- an inference server fetching model weights from a public hub on first launch
- usage statistics sent to the tool's maintainers
- software licenses validated against a vendor server
- automatic updates
Some teams say "air-gapped" when they mean a connected network with outbound traffic blocked at the firewall. That is restricted egress, and it is easier, because you can open a controlled path for updates. A true air gap has no path at all. This guide assumes the strict version.
Who needs an air-gapped LLM
Defence and national security. Classified networks are physically separate by design. If analysts want a model on that network, it has to run there.
Government agencies that hold citizen records, case files or internal deliberations under a policy that the data never touches an internet-connected system.
Regulated industries and critical infrastructure. Trading networks, hospital clinical networks, energy and industrial control environments already isolate their core systems, and they will not make an exception for an AI tool.
IP-heavy research teams in areas like chip design or drug discovery, where a leak costs more than the operational pain of isolation.
In each case a hosted API is out regardless of the provider's contract, because the network cannot reach it. Keeping data inside your environment supports your compliance obligations, but meeting them remains your organization's responsibility. This post is not legal advice.
The practical problems
Getting software and weights across the gap
Everything arrives through a transfer process. The usual shape is a connected staging machine outside, approved removable media or a one-way data diode, and a receiving host inside that scans and stages files before anything is installed.
Plan for volume. A 70B model at 4-bit is about 42 GB (70B x ~0.6 bytes per weight), and about 70 GB at 8-bit. Bring two quantization variants each of three models and you are moving several hundred gigabytes, often with a manual approval at each hop.
Software needs the same treatment, with every dependency resolved in advance. On the connected staging machine, collect pinned artifacts:
docker pull registry.example.com/inference-server:1.4.2
docker save -o inference-server-1.4.2.tar registry.example.com/inference-server:1.4.2
pip download -r requirements.txt -d ./wheels
hf download <org>/<model> --revision <commit-hash> --local-dir ./models/<model>
Inside the gap, install them with no network:
docker load -i inference-server-1.4.2.tar
pip install --no-index --find-links ./wheels -r requirements.txt
docker save writes images to a tar archive that docker load restores (Docker docs). The pip documentation describes pip download output as something you later pass to pip install --find-links for "offline or locked down package installation" (pip docs). On Hugging Face, hf download --revision accepts a commit hash (Hugging Face CLI guide). Use the hash, not a branch name, so the files you approved are the files you fetched.
GPU drivers are the item teams forget. Build the driver and CUDA or ROCm packages for the exact OS release on the inside, and test the install on a disconnected machine before the real transfer.
Checksums and verification
A file that crossed the gap is only trustworthy if you can prove it is the one you meant to bring. Verify twice.
First, on the connected side, check the download against its source. hf cache verify <repo> --local-dir <dir> compares local files with the checksums on the Hub and exits non-zero on a mismatch. Second, generate your own manifest before transfer:
cd models && find . -type f -print0 | xargs -0 sha256sum > ../MANIFEST.sha256
Then check it on the receiving host inside:
cd models && sha256sum -c ../MANIFEST.sha256
Sign the manifest with a key your security team already manages, so the receiving side can tell it came from your staging process and not from whoever handled the media.
Prefer safetensors and GGUF files over pickle-based checkpoints (.bin, .pt). Hugging Face's own security documentation warns that loading a pickle file can run arbitrary code (Hugging Face pickle scanning). An isolated network is a bad place to discover that.
Updates
Nothing updates itself, which is half the point. It also turns every patch into a small project. Set a cadence before go-live: for example, monthly for OS and container security patches and quarterly for model upgrades, with an emergency path for critical vulnerabilities. Keep the previous version of every component on disk so rollback is a local operation, not another transfer.
Treat a new model version as a release. Run it on a staging GPU inside the gap against a fixed set of real tasks, compare outputs with the current version, then promote it.
Telemetry, and proving there is none
Several common tools send usage data by default. vLLM's documentation says it "collects anonymous usage data by default" and lists VLLM_NO_USAGE_STATS=1 or DO_NOT_TRACK=1 to opt out (vLLM usage stats). Hugging Face libraries disable telemetry with HF_HUB_DISABLE_TELEMETRY=1, and HF_HUB_OFFLINE=1 stops all HTTP calls to the Hub (Hugging Face environment variables).
On a real air gap those calls fail anyway, but failing calls cause slow startups, timeouts and noisy logs. Set the flags, then run a packet capture during acceptance testing to confirm nothing tries to leave.
Licensing checks offline
There are two kinds. Model licenses come with open-weight models and some restrict use, so approve each license before transfer and store the license file next to the weights, where an auditor inside the gap can find it. Software licenses are the trap: any commercial product that validates its license against a vendor server will stop working. Ask every vendor how offline license validation works before you buy, and test it on a disconnected machine.
Sizing an air-gapped LLM
The math is the same as for any self-hosted model, with one difference: you cannot burst to the cloud, and new hardware takes procurement plus the transfer process. Size for peak load, not average.
GPU memory = weights + KV cache + ~10% overhead
Llama 3.1 8B at 8-bit for 20 analysts at once, each with 8K tokens of context:
weights: 8B x 1.06 bytes = 8.5 GB
KV cache: 20 users x 1 GB (8K) = 20.0 GB
overhead: 10% of 28.5 GB = 2.9 GB
total ~ 31.4 GB -> a 48 GB card
The 1 GB per user comes from the KV cache formula (2 x 32 layers x 8 KV heads x 128 head size x 2 bytes = 128 KB per token, times 8,192 tokens). Our guide to GPU memory for LLM inference walks through it, and LLM quantization explained covers which bit width to pick.
Disk needs its own budget. Keep the current and previous version of each model, plus a staging copy: three models averaging 40 GB x 2 versions x 2 copies is 480 GB before container images and wheels. Plan for at least 1 TB of fast local storage, and a second GPU host for staging if you can afford it.
Air-gapped LLM deployment checklist
- Write down the use case, user count, data classification and who approves transfers.
- Agree the transfer process with security: media or diode, scanning, approvals, manifest signing.
- Shortlist models by task and license. Approve the licenses.
- Size GPUs, RAM and disk for peak load and order the hardware.
- Build OS, GPU driver and runtime packages for the exact target OS release.
- Collect pinned container images, wheels and model weights, and verify them against their sources.
- Generate and sign a SHA-256 manifest, then transfer.
- Inside, verify the manifest, scan, load images and install.
- Set offline and telemetry flags, and confirm with a packet capture that nothing egresses.
- Deploy the inference server behind internal TLS and authentication, exposing an OpenAI-compatible API so internal apps can use standard SDKs.
- Add access control, an audit log and monitoring before opening it to users.
- Test with real tasks, and write the install into a runbook that rebuilds the system from scratch.
- Schedule the update cadence and rehearse a rollback.
The self-hosted LLM guide covers steps 3, 10 and 11 in more depth for connected environments.
Where AI Refinery fits, and where it does not
AI Refinery, a self-hosted LLM platform, installs inside your own infrastructure, including a fully air-gapped network, and nothing phones home. The install is reproducible from the operations runbook, which covers checklist step 12. Its catalog models are GGUF files in several quantization variants, each benchmarked before it reaches the catalog, with size on disk, GPU memory needed and measured tokens per second, plus security scoring and license documentation. That is the information you would otherwise gather by hand before deciding what to carry across.
Inside the network, you pick a model and quantization level and launch an inference server with one click. Each deployment exposes an OpenAI-compatible /v1/chat/completions endpoint, and RBAC, SSO, approval workflows, a full audit log and per-server monitoring of requests, latency, tokens and errors cover steps 10 and 11. Your transfer process still governs how files cross the gap. Ask [email protected] how that works for your environment.
It is not the right fit if you need a chat UI or document search out of the box: you would pair it with an open-source chat UI such as Open WebUI and a vector database of your choice, and both also have to cross the gap. It does not supply hardware, so you need GPUs inside the network. And if your network only restricts egress rather than being fully disconnected, a simpler setup with a controlled update path may be enough.
Public-sector teams can read the related guide on an LLM for government.