LLM for Financial Services: Use Cases, Risk and Rollout
An LLM for financial services earns its place in the work that eats analyst hours: reading long documents, pulling facts out of case files, answering questions about internal policy, and helping developers with old code. It should not be making credit, claims or investment decisions on its own, and most firms that get value from one keep a person in the loop for anything that reaches a customer or a regulator.
The hard part is rarely the model. It is the constraints around it: client data that cannot leave the firm, model risk teams that need to validate what you deploy, and third-party risk rules that cover every outside service you depend on. Running an open-weight model on your own infrastructure answers several of those constraints at once, which is why this guide ends with a self-hosted, phased rollout.
This post is general information, not legal or regulatory advice. Check every requirement with your compliance and legal teams.
Realistic use cases in banks, insurers and asset managers
The use cases that hold up share three traits: the input is text the firm already has, the output is a draft a qualified person checks, and a mistake is caught before it matters.
| Use case | Who uses it | What the model does | Who checks it |
|---|---|---|---|
| Document summarization | Credit, research, claims | Condenses credit files, earnings call transcripts, claims files or prospectuses into a structured summary | The analyst who owns the file |
| KYC/AML analyst support | Financial crime operations | Summarizes an alert's case file, lists the transactions and entities involved, drafts a first narrative | The investigator, who makes the decision |
| Policy Q&A | Operations, front office, HR | Answers "what does our procedure say about X" from internal policy documents, with the source section quoted | The user, by reading the quoted source |
| Code assistance | Engineering | Explains legacy code, writes tests, drafts migration scripts | Code review, as for any change |
Document summarization
This is the safest place to start. A credit analyst reading a 60-page file, a claims handler going through medical reports and adjuster notes, or a research associate condensing ten earnings calls all do the same thing: read and extract. Ask the model for a fixed structure (parties, amounts, dates, risks, open questions) rather than free prose, so reviewers can check each field against the source.
KYC and AML analyst support
Know-your-customer (KYC) and anti-money-laundering (AML) teams spend much of their time assembling information before they can judge it. A model can gather the alert details, transaction history and adverse media already in the case file into one readable summary, and draft a narrative the investigator edits. The investigator still decides whether activity is suspicious. Keep that line explicit in your procedures, because "the model said it was fine" is not an answer an examiner will accept.
Policy and procedure Q&A
Large firms have thousands of pages of procedures that staff search badly. Retrieval-augmented generation (RAG), where the system finds the relevant passages and gives them to the model with the question, lets staff ask in plain language and get an answer that quotes the source section. Our guide to RAG with a local LLM walks through building that pipeline.
Code assistance
Banks and insurers run a lot of old code. A code model can explain a COBOL program, suggest unit tests or draft a SQL migration. Source code is often as sensitive as customer data here, because it reveals how controls work, which is another reason to keep the model in-house.
The constraints on an LLM for financial services
Data residency and confidentiality
Customer records, account data and transaction histories are covered by banking secrecy, privacy law and client contracts, and many firms also have rules about which countries data may be processed in. Sending that data to a public AI API means it is processed on someone else's servers, often in a region you do not choose, under terms that can change. For many firms that alone rules out a public API for anything beyond non-confidential work.
Model risk management in the US
For years, US banks applied the Federal Reserve's SR 11-7 model risk guidance to any new modeling technology. On April 17, 2026, the Federal Reserve, OCC and FDIC replaced it with SR 26-2, Revised Guidance on Model Risk Management, which supersedes both SR 11-7 and SR 21-8. The letter says it is expected to be most relevant to banking organizations with over $30 billion in total assets.
Two points in the guidance text matter for LLM projects:
- Generative AI is outside its formal scope. A footnote says generative AI and agentic AI models "are not within the scope of this guidance", but adds that a bank's risk management and governance practices should still guide the controls for tools the guidance does not cover. In practice, expect your model risk and technology risk teams to apply their own framework to an LLM rather than wave it through.
- Vendor models still need validation. The section on vendor and third-party products says sound practice includes understanding a vendor model's design, development data and performance, plus ongoing monitoring of whether it stays fit for purpose. That is hard when the model behind an API can change without notice.
Broker-dealers have a parallel reminder: FINRA Regulatory Notice 24-09 says FINRA's rules are technology neutral and apply whether a firm builds its own generative AI tool or uses a third party's.
Third-party and outsourcing risk in the EU
In the EU, the Digital Operational Resilience Act (DORA) has applied since 17 January 2025 to banks, insurers, investment firms and other financial entities. According to EIOPA's DORA overview, it requires firms to manage the risk from their ICT third-party providers, sets key contractual provisions for those arrangements, and creates EU-level oversight of critical ICT providers because of the concentration risk of many firms relying on a few of them. A hosted AI API is an ICT third-party service like any other, with all the contract, register and exit-plan work that implies.
Why self-hosting suits financial firms
A self-hosted model runs on infrastructure you control: your data center, your cloud account or your VPC. That changes several of the constraints above.
- Data stays inside your environment. Prompts, documents and outputs never leave your network, which supports your residency and confidentiality obligations. It does not meet them for you: compliance remains your firm's responsibility.
- The validated model stays the validated model. An open-weight model file does not change unless you replace it. Your model risk team can test a specific version and quantization and know that is what production runs.
- You own the evidence. Every request can pass through your own audit log and monitoring, so "who used which model, on what, when" has an answer. See LLM governance for enterprises for the controls to put in place.
- The third-party question changes shape. You still have suppliers: the software, the model's publisher and your hardware or cloud provider. But no outside party processes your prompts, and the platform keeps running if a vendor changes its terms. Firms with the strictest requirements can go further with an air-gapped LLM deployment.
The cost side is easier to reason about than it looks. Take 200 analysts making 40 requests a day at about 3,000 tokens each: 200 x 40 x 3,000 = 24 million tokens a day. Whether that is cheaper on your own GPUs depends on hardware and utilization, which our private LLM vs OpenAI comparison works through.
A phased rollout
Move from low-risk to higher-risk work, and do not skip the review step between phases.
- Phase 0, groundwork. Classify which data each use case touches. Agree with model risk, compliance and information security how LLM tools will be inventoried, tested and approved. Decide which roles may use which models.
- Phase 1, internal and low-risk. Code assistance and Q&A over non-confidential policy documents. Measure accuracy on a test set of 50-100 real questions with known answers before wider release.
- Phase 2, confidential documents with review. Summarization of credit files, claims files or research inputs, where the owner of the file checks every summary. Track how often reviewers correct the output.
- Phase 3, analyst support in regulated workflows. KYC/AML case summaries and draft narratives, with the investigator's decision and edits recorded. Keep the model's output and the final human version, so you can show the difference.
- Ongoing. Re-test whenever you change the model, the quantization or the prompts, and review usage logs for drift into use cases nobody approved.
For hardware, start small. A 14B-parameter model at 4-bit quantization needs about 14 x 0.6 = 8.4 GB for the weights, plus memory for the KV cache (the model's working memory per conversation), so it fits on a single data center GPU. Many summarization and Q&A tasks work well at that size. Larger models for harder reasoning tasks can follow once Phase 1 shows where the smaller one falls short.
Where AI Refinery fits
AI Refinery, a self-hosted LLM platform, installs inside your own infrastructure: your cloud, your VPC, on-premise or a fully air-gapped network, and nothing phones home. You pick open-source models from a curated catalog where every quantization variant is benchmarked for size on disk, GPU memory and tokens per second, and each model has security scoring and license documentation, which gives model risk reviewers a starting point. Every deployed model exposes an OpenAI-compatible endpoint, and governance comes built in: RBAC, SSO, size-tier controls, approval workflows and a full audit log, plus monitoring of request volume, latency, token usage and errors.
It is not a compliance product. It holds no financial-sector certifications, and it does not replace your model validation, third-party risk process or legal review. It also does not include a chat interface or a vector database: you pair it with the tools your firm already approves for those.
It is not the right fit if you have no GPUs and no plan to get them, if you need the single most capable frontier model for a task, or if your use case only touches public information. In those cases a hosted API under a well-negotiated contract may serve you better.