LLM Governance: A Practical Program for Enterprise Teams

9 min read

LLM governance is the set of rules and controls that decide which language models your company runs, who can use them, with what data, and how you prove all of that afterwards. A working program is mostly plumbing, not a policy PDF: an inventory, an approval step, access control tied to your identity provider, an audit log, and a schedule for reading what the logs say.

Most companies build it in the wrong order: usage spreads first and governance arrives later as a cleanup project. This guide lays out a program you can put in place early, or retrofit. It maps to the NIST AI Risk Management Framework and notes where the EU AI Act applies.

What an LLM governance program has to answer

Every control below answers one of five questions an auditor, a customer or your security team will eventually ask:

  1. Which models are running, and who approved each one?
  2. Who can use which model, and how is that enforced?
  3. Who sent which request to which model, and when?
  4. What data is allowed to go into which model?
  5. When did a model change, and who signed off?

If you can answer all five from records rather than memory, you have a program. If any answer is "ask the team that set it up", you have a gap.

The frameworks worth anchoring to

NIST released the AI RMF 1.0 in January 2023. It is voluntary and organizes the work into four functions: Govern, Map, Measure and Manage. In July 2024 NIST added a Generative AI Profile (NIST AI 600-1) that lists risks specific to generative models. Among them are confabulation (confidently stated false output), data privacy, information security, and value chain and component integration: the risk that comes from third-party models and datasets you did not build.

The EU AI Act (Regulation (EU) 2024/1689) is law. For a company that deploys rather than builds models, two parts matter most. Since 2 February 2025, Article 4 requires deployers to take measures to build AI literacy among staff who use AI systems. And a high-risk use case, such as decisions about employment or education, brings extra duties, including keeping the system's automatically generated logs for at least six months.

After the AI Omnibus amendments, high-risk obligations for sensitive use cases apply from 2 December 2027. The timeline has already moved once, so check the Commission's AI Act page before planning around a date.

An internal summary assistant is very unlikely to be high-risk. A model that screens job applicants may well be. Classify each use case, not each model.

Step 1: Keep a model inventory and an approval gate

The inventory is a table with one row per model version you run. At minimum:

Field Example
Model, version and quantization A Qwen 14B instruct release at Q4_K_M
Source and file hash Download source, SHA-256 of the file
License Name, link, any commercial or usage restrictions
Approved use cases Internal summarization, code review
Data classes allowed Public and internal, not customer PII
Owner A named person, not a team alias
Approval Approver and date
Evaluation Link to the test set and scores used for approval

The approval gate is the rule that nothing reaches production without a row. Keep it light: one reviewer from security or risk, one from the business owner, and a written checklist. A monthly committee teaches teams to route around it.

The reviewer checks that the license allows your use, the source is trusted and the hash matches, the model passed your evaluation set, and known security issues for that family have been reviewed. The license check catches more problems than people expect, because some popular open-weight families carry use restrictions or user-count thresholds. Our guide to choosing an open-source LLM for enterprise use covers how to read them.

Step 2: Decide who can use which model

Access control for LLMs has three layers.

Identity. People sign in through SSO (single sign-on) with your existing identity provider, so leaving the company removes access everywhere at once. Applications use API tokens: one per application, never shared, each with an owner and an expiry date.

Roles. RBAC (role-based access control) maps groups to permissions: who can deploy a model, who can approve one, who can only call the endpoint, and who can read the audit log. Keep the people who approve models separate from the people who deploy them.

Size tiers. Not every team needs a 70B model. A size-tier rule such as "support can use models up to 14B, research up to 70B" controls GPU spend and narrows the set of models each team can reach.

Step 3: Decide what the audit log records

An audit log is useful only if it answers the five questions above without a forensic project. Record two kinds of events.

Administrative events: a model downloaded, deployed, stopped, upgraded or deleted; role and permission changes; tokens created, rotated or revoked; approvals granted or refused. Each entry needs who, what, when, and the state before and after.

Inference events: timestamp, caller (user or token), model and version, prompt and response size in tokens, latency, and status or error code.

Storing full prompts and responses is a separate decision, and the default should be no: full text is the most sensitive data in the system. If you need it for debugging or quality review, store it apart from the metadata log, with a short retention period and tighter access.

Then set retention. Six months is the floor the EU AI Act sets for logs of high-risk systems. Sector rules, contracts and your security team may require longer for metadata. For prompt text, shorter is safer.

The audit log records what happened; monitoring shows what is happening now. You need both, and the LLM observability guide covers what to measure.

Step 4: Pin versions and control changes

A model that changes without notice breaks the evaluation you approved it on. Pin the exact version and quantization in configuration, and treat an upgrade like any other production change:

  1. Open a change request naming the old and new version.
  2. Run the evaluation set used at approval and record both sets of scores side by side.
  3. Get sign-off from the model owner.
  4. Send a share of traffic to the new version, watch error rates and output quality, then complete the rollout.
  5. Keep the previous version deployable until the new one has proven itself.

With a hosted API you can usually pin a dated model snapshot, but the provider decides when it is retired. With a self-hosted model, the file on disk does not change until you replace it, so change control is a process you own end to end. The self-hosted LLM guide explains that trade.

Applications should call models through aliases such as summarizer or code-assistant, mapped to a pinned version in one place. An upgrade is then one config change and one audit entry. A central self-hosted LLM gateway is the usual place to keep that mapping.

Step 5: Write data handling rules people can follow

Start from your existing data classification and write one table: for each data class, which deployment types may receive it. An example:

Data class Hosted public API Self-hosted model in your network
Public Allowed Allowed
Internal Allowed under enterprise terms Allowed
Confidential or customer data Only after contract review Allowed for approved use cases
Regulated (health, payment, special category) Usually not Allowed after legal and security review

Add two rules: output that contains customer data inherits that data's classification, and prompts may be stored only where, and for as long as, the table says.

Step 6: Publish an acceptable-use policy

Keep it to one page. It should say which AI tools are approved and that unapproved tools are not used with company data. It should point to the data-class table above. It should require a human to review model output before it reaches a customer, a contract or a production system. And it should explain how to report a problem.

Pair it with short training on what the models do well and badly, which also covers much of the AI literacy measure the EU AI Act asks of deployers. A policy works only if there is an approved tool people prefer; how to replace shadow AI covers detection and rollout.

Step 7: Set a review cadence

Controls decay without a schedule. A cadence that suits most companies:

  • Monthly: usage by model and team, error trends, tokens with no recent use (revoke them), and access changes.
  • Quarterly: recertify roles and size tiers with each team lead, re-check licenses for models whose terms changed, and close stale change requests.
  • Twice a year: re-run evaluations on every production model and retire models with no owner or no traffic.
  • On event: a new model family, a new high-risk use case, a security incident or a regulation change triggers an extra review.

LLM governance checklist

  • Every production model has an inventory row with owner, license, hash and approval
  • Nothing deploys without approval, and approvers are not the deployers
  • SSO for people; owned, expiring tokens for applications
  • RBAC roles defined and size tiers set per team
  • Audit log records admin and inference events, with retention set
  • Full prompt storage off by default, or kept separately with short retention
  • Versions pinned; upgrades go through change control and re-evaluation
  • Data-class table published and linked from the acceptable-use policy
  • Use cases classified against the EU AI Act where it applies
  • Review cadence on the calendar, with named owners

Where AI Refinery fits

AI Refinery, a self-hosted LLM platform, puts several of these controls in one place inside your own infrastructure: your cloud, your VPC, on premises or an air-gapped network. It provides RBAC, SSO, size-tier controls, approval workflows and a full audit log. Catalog models come with license documentation and security scoring, useful inputs to the approval review in step 1. Applications call models through an OpenAI-compatible endpoint with API tokens managed in the console, and the observability dashboard shows request volume, latency, token usage and errors for each inference server.

It does not write your policies, classify your use cases or run your review meetings. Steps 5 to 7 stay with your team.

It is not the right fit if all of your LLM use runs through one hosted provider and you have no plans to self-host. In that case the governance work lives in your identity provider, your API gateway and your contract with that provider. It is also not a fit if you have no GPUs and no one to run them, because you provide the hardware.

The same controls matter for agencies; see LLM for government.