Every few weeks someone asks us whether they should run their own on premise LLM because the API bill keeps growing. Usually the honest answer is no. For general knowledge work like drafting, coding help, research and summarizing, frontier models are better than anything you can host yourself, and per-seat enterprise plans or plain API access cost less than a GPU cluster plus the people to run it.
There is a real set of cases where a private LLM is the right call, though, and in those cases it can beat a frontier API on cost, latency and control. This guide covers both sides: when not to self-host, the four situations where it pays off, the costs people leave out of the spreadsheet, and what building one actually involves.
Should most companies run an on premise LLM?
No. If the goal is to give employees a capable assistant, buy seats or use an API. Here’s why:
- Quality. The frontier models are ahead of the open models you can fit on a single server, especially on long, open-ended work.
- Utilization. Chat traffic is bursty. People use it during work hours and barely at all overnight. A GPU you own costs the same whether it’s busy or idle, while an API charges only for what you use.
- Upgrades are free. When a frontier lab ships a better model, you switch a model ID. When you self-host, every upgrade is a project.
- Privacy is often already handled. Enterprise plans and cloud-hosted versions of frontier models come with contractual commitments not to train on your data. More on that below.
So if the reason for self-hosting is “chat is getting expensive,” look first at seat plans, usage caps, routing simple requests to a cheaper model and caching. Those usually fix the bill without buying hardware.
Is there a middle option between a public API and on-prem?
Yes, and it’s the one we recommend checking first. All three big clouds sell frontier and open models that run inside your own cloud account, under the same contracts, logging and network controls as the rest of your infrastructure.
- Amazon Bedrock. AWS says your inputs and outputs are not shared with model providers and are not used to train base models. Models run in AWS-operated deployment accounts that the model providers can’t access, and you can reach Bedrock over AWS PrivateLink without crossing the public internet. AWS lists Bedrock as HIPAA eligible and in scope for FedRAMP Moderate, and it’s also available in AWS GovCloud (US), where AWS publishes FedRAMP and DoD IL4/IL5 status model by model.
- Azure OpenAI and other models sold by Azure. Microsoft says your prompts, completions and training data are not available to OpenAI or other model providers and are not used to train foundation models without your permission. Standard deployments process data within the geography you choose (Global and DataZone deployments can process it more widely). Microsoft also offers these models in Azure Government.
- Google Cloud Vertex AI (Google’s docs now call it Gemini Enterprise Agent Platform). Google says it won’t use your data to train or fine-tune models without your permission, and documents how to reach zero data retention.
For a lot of PII and HIPAA workloads, and for some government work, one of these clears the compliance review without anyone racking a GPU. Check the specific model and region against your requirements, because coverage varies by model.
When does self-hosting an LLM actually make sense?
In our experience it comes down to four situations. If none of them describes you, stay on an API.
1. The data can’t leave your network
Some data can’t go to a cloud API, even one in your own tenancy:
- Air-gapped networks. Classified enclaves, some OT networks and isolated labs have no route to a cloud at all. An air gapped LLM is the only kind that works there.
- CUI and ITAR work. Defense primes and suppliers handling Controlled Unclassified Information have to meet NIST SP 800-171, and export-controlled technical data comes with its own rules. A government cloud region may satisfy those rules for some programs and not for others. When it doesn’t, the model has to run on hardware you control.
- Some PII and health data. When a contract, a regulator or your own risk team says the data stays in your data center, that settles it.
2. High-volume, narrow tasks inside a product or pipeline
This is where self-hosting can win on cost and on speed. The task is the same thing millions of times: classify this transaction, extract these fields from this document, route this claim, label this ticket. A huge general-purpose model is overkill for that. A small open model fine-tuned on your own data can match or beat it, at lower latency, on hardware you can keep busy around the clock.
Coinbase recently published a detailed example. In Part 3 of its Owning Intelligence series (Oct 8, 2026), its team post-trained Qwen3.5-9B, an open-weight model with nine billion parameters, on its proprietary fraud data for one narrow task: classifying transaction risk for Coinbase Onramp. They used reinforcement learning with rewards scored against known fraud outcomes and LoRA adapters, and trained on a single NVIDIA RTX PRO 6000 96GB GPU for a total cost of under $100. On Coinbase’s own benchmark, the post-trained model beat Claude Opus 4.5 on all four detection metrics: 7.5 higher precision, 12.0 higher recall and 9.6 higher F1 (the post’s chart labels these as percentage-point differences), plus 35.4 higher dollar-weighted recall. Served with vLLM and Ray Serve LLM on L40S GPUs, scaling between 2 and 8 replicas, its median end-to-end latency in production was 0.683 seconds against Opus 4.5’s 1.515 seconds, and P99 was 1.036 seconds against 3.312.
Three things in that write-up are worth noticing. The unadapted Qwen model performed poorly, so the gain came from training on Coinbase’s data, not from the open model itself. Coinbase built a domain benchmark first. And Coinbase says owning the model means owning its lifecycle: fraud patterns change, so the model needs ongoing evaluation and retraining.
The $100 figure covers the training run. It doesn’t include the people, the dataset, the benchmark or the serving cluster. Even so, it shows how cheap fine-tuning has become for a well-defined task.
3. Edge and offline use
Drones, vehicles, factory lines, pump stations and remote sites often have weak or no connectivity, tight latency budgets, or both. A model that has to call a cloud API can’t run there. Small models running on edge hardware like NVIDIA Jetson can handle tasks like summarizing a maintenance log, reading a work order or interpreting sensor alarms on site. This is the same work we do in edge AI and computer vision: sizing the model to the compute and power you actually have.
4. A model that won’t change under you
Hosted models get retired on a schedule. Microsoft’s Foundry lifecycle policy describes a standard 18-month lifecycle for generally available models, with models closed to new customers at 12 months. AWS says most Bedrock models get a 6-month Legacy period before end of life, and some get 45 days. Google lists retirement dates for its models, too. That’s reasonable for most software. But if a model sits inside a validated workflow, such as a regulated process, a certified system or a pipeline with an audit trail, every forced upgrade means revalidating. With open weights you hosted yourself, the version you validated keeps running until you decide to change it.
Need help setting up a local LLM? Axio Intelligence can help. We scope it, build it and run it, from the eval set and fine-tune to serving and monitoring. See our data and AI platform work or book a scoping call.
What does self-hosting cost that people forget?
The GPU invoice is the visible part. These are the costs that usually decide whether a private LLM is worth it:
- Idle capacity. You pay for a GPU 24 hours a day. If it’s busy for eight of them, your real cost per request is three times what the benchmark suggested. Pipelines that run all day fit this model well. Chat usually doesn’t.
- Evals. You need a test set that reflects your actual task and gets rerun on every model, prompt and data change. Without one, you can’t tell whether the self-hosted model is good enough, or whether it got worse.
- Monitoring. Track quality, latency, throughput and drift in production, with alerts and a tested rollback.
- Upgrades. New open models come out constantly. Deciding whether to move, retraining adapters and revalidating all take engineering time.
- Security patching. The serving stack, drivers, container images and model files all need patching and access controls like any other production system.
- People on call. Someone has to answer when the inference server falls over at 2 a.m. If no one on your team owns that, you don’t have a production system yet.
A worked example (assumptions, not quotes)
Here’s how we frame the cost question with a client. The numbers below are made up to show the math. Plug in your own.
- Assume a self-hosted setup costs $20,000 a month all-in: hardware amortized over three years, power, hosting, and a share of an engineer’s time for operations.
- Assume the API you’d use instead costs $5 per million tokens, blended across input and output.
- Break-even is $20,000 ÷ $5 = 4,000 million tokens, or 4 billion tokens a month.
A 200-person team where each person uses 100,000 tokens per working day, over 22 working days, comes to 440 million tokens a month, about a ninth of break-even. That’s why we tell most companies to stay on the API for chat. A document pipeline handling a million documents a month at 5,000 tokens each comes to 5 billion tokens, and now the self-hosted option deserves a closer look, especially if a smaller fine-tuned model needs fewer tokens per document than a frontier one.
The real decision usually turns on quality, latency and data control, not cost alone. But if the volume isn’t anywhere near break-even, cost won’t be the reason to self-host.
How do you decide? A quick checklist
| Question | If yes | If no |
|---|---|---|
| Is this general knowledge work (writing, coding, research, chat)? | Use seats or an API | Keep going |
| Must the data stay off any cloud, including your own tenancy? | Self-host | Try Bedrock, Azure OpenAI or Vertex AI first |
| Is it one narrow task run at high volume, all day? | A fine-tuned small model is worth testing | An API is probably cheaper |
| Does it run where connectivity is unreliable or latency is tight? | Edge or on-prem model | API is fine |
| Does a model change force revalidation of a regulated workflow? | Pin a self-hosted version | Hosted lifecycle is fine |
| Do you have someone to own evals, monitoring and on-call? | You’re ready | Plan for that before you build |
If you answered yes to the first row and no to the rest, you don’t need an on-prem model.
What does building a private LLM involve?
If you do need one, this is the order we work in. It’s the same order whether you build it yourself, hire LLM developers, or bring in a team for custom AI model development.
- Build the eval set first. Collect a few hundred to a few thousand real examples with correct answers, and score the frontier API against them. That score is the bar to beat, and it’s what tells you whether the project is worth doing.
- Prepare the data. Clean, label and version the training examples, and keep test examples out of training. Most of the work in any effort to fine-tune an LLM on your own data is here.
- Pick a base model and fine-tune. Start with the smallest open model that might do the job, then use LoRA or a similar method to train it on your data. Read the model card from the publisher for license terms and intended use.
- Serve it. Use an inference server such as vLLM on hardware sized to your traffic, with autoscaling if the load varies. For edge, that means quantizing and optimizing the model for the target device.
- Govern it. Version the weights, prompts and decision logic together, log what the model does, and set a review process for changes. The NIST AI Risk Management Framework is a useful structure for this.
- Operate it. Monitor quality and latency, retrain when the data shifts, and keep a rollback ready.
Good LLM fine-tuning services spend most of their time on steps 1, 2 and 6. If a proposal is mostly about GPUs and model selection, ask where the eval set is.
FAQ
Is an on-premise LLM cheaper than using an API?
For general chat and knowledge work, usually not, because owned GPUs sit idle much of the day and the frontier models are better. For a narrow task running at high volume, a small fine-tuned model on hardware you keep busy can cost less per request than an API. Do the math with your own volume.
What’s the difference between a private LLM and an air-gapped LLM?
A private LLM runs where you control the data, which can be your own cloud account or your own data center. An air-gapped LLM runs on a network with no internet connection at all. Every air-gapped LLM is private, but most private deployments aren’t air-gapped.
Can Amazon Bedrock or Azure OpenAI meet HIPAA or government requirements?
Often, yes. AWS lists Bedrock as HIPAA eligible and in scope for FedRAMP Moderate, and offers it in GovCloud with IL4/IL5 status published per model. Microsoft offers its models in Azure Government. Coverage varies by model and region, so check the exact model you plan to use.
How much data do we need to fine-tune an LLM on our own data?
It depends on the task. For a narrow classification or extraction task, a few thousand good examples is often a reasonable place to start, and the eval set tells you whether you need more. Quality and correct labels matter more than volume.
Will a small open model be as good as a frontier model?
On broad, open-ended work, generally not. On one narrow task with good training data, it can be. Coinbase’s post-trained 9B model beat Claude Opus 4.5 on its fraud benchmark, while the same model without training performed poorly.
Should we hire LLM developers or bring in an outside team?
If you’ll run several models for years, build an internal team. If you need one model built, validated and handed over, an outside team can get you there faster, as long as it leaves you with the eval set, the training pipeline and the documentation to run it.
Where do we fit?
We’ll tell you honestly whether you need your own model. Often the answer is a cloud-hosted frontier model with the right contract and controls. When you do need one, we build it: the eval set, the data pipeline, fine-tuning, serving on-prem or at the edge, and the monitoring to keep it accurate. If you’re weighing an on-prem LLM, book a call and we’ll walk through your use case.
