Short answer: Estimate enterprise AI application cost as two separate budgets: a one-time cost to design, build, integrate, secure, test and deploy, and a recurring cost to run it in production. Start from workload assumptions (users, tasks, tokens, retrievals, tool calls, retries, human review), turn each into a billing or labor unit, and judge the result by cost per successful task and 12-month total cost of ownership (TCO).
Key takeaways
- Model API spend is only one line. In the three scenarios below it is between 0.2% and 3.0% of monthly operating cost.
- Budget build and run costs separately so nothing is counted twice.
- Use cost per successful task, not cost per request, as the decision metric.
- Treat every price and rate as an assumption with a date, and refresh it before approval.
Pricing reference date: October 16, 2026. Worked examples use USD and exclude taxes, negotiated discounts, and regional premiums. Labor rates, tool costs, and infrastructure figures are planning assumptions, not LuMay pricing.
Start with two cost buckets: build and run
A common budgeting mistake is counting only model API spend. Keep two buckets.
One-time implementation costs
- Discovery and solution architecture
- Data assessment, cleanup, and permissions mapping
- Model evaluation and selection
- RAG design, ingestion, and indexing
- Enterprise integrations and tool orchestration
- Web, mobile, or conversational interfaces
- Evaluation datasets and test automation
- Security, access controls, and audit logging
- Deployment, environments, and CI/CD
- Documentation, training, and project management
The NIST AI Risk Management Framework treats governance, measurement, and risk management as lifecycle activities. Give evaluation and security their own engineering budget instead of hiding them inside "development".
Recurring operating costs
- Model inference and embeddings
- Vector or search storage and retrieval
- Application compute, databases, and networking
- Third-party API and tool calls
- Observability and log retention
- Human review and escalation
- Support, maintenance, and model or prompt upkeep
- RAG content refresh and regression testing
- Incident handling and security operations
Build the workload model before looking at prices
Begin with business activity, not tokens. "Monthly users" alone is not enough, because two users can generate very different token, retrieval, and tool volumes.
| Workload driver | What to estimate |
|---|---|
| Active users and tasks | Tasks per user per month |
| Model executions | Calls per task, including routing, validation, and retries |
| Input tokens | System prompt, conversation, retrieved context, tool results |
| Output tokens | Average generated response |
| Retrievals | Search or vector requests per task |
| Tool calls | CRM, ERP, database, email, payment, or other API calls |
| Retry rate | Repeated model or tool calls after failures |
| Concurrency | Peak simultaneous requests |
| Cache usage | Share of input tokens served from reusable cached context |
| Success rate | Share of tasks that reach the required outcome |
| Growth | Expected monthly increase in users, data, or requests |
For agentic workflows, one user request is not one model request. A single task may involve routing, retrieval, generation, tool execution, validation, and a retry, which multiplies inference and API cost.
Use transparent formulas
Monthly operating cost = inference + embeddings + retrieval and storage + tool APIs + infrastructure + observability + human review + maintenance and support + incident allowance
Inference cost = (uncached input tokens / 1,000,000 x input rate) + (cached input tokens / 1,000,000 x cached rate) + (cache-write tokens / 1,000,000 x cache-write rate) + (output tokens / 1,000,000 x output rate)
Successful tasks = initiated tasks x success rate
Cost per successful task = monthly operating cost / successful tasks
12-month TCO = one-time implementation + implementation contingency + 12 months of operating cost
If maintenance engineering is already in the monthly cost, do not add a second annual maintenance charge. Do not count implementation contingency again as an incident reserve. If volume will grow, calculate each month separately instead of multiplying month one by 12.
Reference prices used in the examples
These are published list prices checked on October 16, 2026. They are examples, not recommendations, and not LuMay pricing.
- Model API: OpenAI lists GPT-5.6 Terra standard short-context pricing at $2.00 per 1M input tokens, $0.20 per 1M cached input tokens, and $12.00 per 1M output tokens. Cache writes are billed at 1.25 times the uncached input rate ($2.50 per 1M). Prompts above 272K input tokens are priced higher.
- Embeddings: text-embedding-3-small is listed at $0.02 per 1M input tokens.
- Vector database: Pinecone lists a $50 per month minimum for Standard and $500 per month for Enterprise. Real cost is metered usage, so replace these minimums with measured figures after load testing.
Three illustrative scenarios
All three use the prices above plus planning assumptions: $0.02 per paid tool call, a 100-token query embedding per retrieval, and flat workload for TCO.
Scenario 1: Internal knowledge assistant pilot
Workload: 100 users x 10 tasks = 1,000 tasks per month. 5% retries give 1,050 executions at 2,000 input and 400 output tokens each. No prompt caching. 1 retrieval per task, 0.1 paid tool calls per task. 90% success.
- Inference: 2.1M input x $2 + 0.42M output x $12 = $4.20 + $5.04 = $9.24
- Embeddings: 100,000 tokens x $0.02 per 1M = $0.002
- Vector database $50, tool APIs $2, infrastructure $300, observability $150
- Human review: 10 hours x $60 = $600
- Maintenance: 20 hours x $125 = $2,500. Incident allowance: 3 hours x $125 = $375
Monthly operating cost: $3,986.24. At 900 successful tasks, cost per successful task is about $4.43.
Implementation: 400 hours x $125 = $50,000, plus 15% contingency = $57,500. 12-month TCO: about $105,335.
Scenario 2: Departmental production assistant
Workload: 500 users x 20 tasks = 10,000 tasks per month. 10% retries give 11,000 executions at 4,000 input and 800 output tokens. 20% of input is cached, with 120,000 cache-write tokens per month. 1.2 retrievals and 0.5 tool calls per task. 93% success.
- Inference: 35.2M uncached x $2 + 8.8M cached x $0.20 + 0.12M cache-write x $2.50 + 8.8M output x $12 = $178.06
- Embeddings $0.024, Pinecone Standard floor $50, tool APIs (5,000 calls) $100
- Infrastructure $1,500, observability $600
- Human review: 40 hours x $60 = $2,400
- Maintenance: 60 hours x $140 = $8,400. Incident allowance: 10 hours x $140 = $1,400
Monthly operating cost: $14,628.08. At 9,300 successful tasks, cost per successful task is about $1.57.
Implementation: 1,200 hours x $140 = $168,000, plus 20% contingency = $201,600. 12-month TCO: about $377,137.
Scenario 3: Enterprise workflow agent
Workload: 2,000 users x 30 tasks = 60,000 tasks per month. 15% retries give 69,000 executions at 8,000 input and 1,200 output tokens. 35% of input is cached, with 1M cache-write tokens per month. 1.5 retrievals and 1.5 paid tool calls per task. 95% success.
- Inference: 358.8M uncached x $2 + 193.2M cached x $0.20 + 1M cache-write x $2.50 + 82.8M output x $12 = $1,752.34
- Embeddings $0.18, Pinecone Enterprise floor $500, tool APIs (90,000 calls) $1,800
- Infrastructure $6,000, observability $2,000
- Human review: 200 hours x $70 = $14,000
- Maintenance: 160 hours x $160 = $25,600. Incident allowance: 40 hours x $160 = $6,400
Monthly operating cost: $58,052.52. At 57,000 successful tasks, cost per successful task is about $1.02.
Implementation: 2,600 hours x $160 = $416,000, plus 20% contingency = $499,200. 12-month TCO: about $1.196 million.
Scenario summary
| Scenario 1 | Scenario 2 | Scenario 3 | |
|---|---|---|---|
| Tasks per month | 1,000 | 10,000 | 60,000 |
| Monthly operating cost | $3,986 | $14,628 | $58,053 |
| Inference share of operating cost | 0.2% | 1.2% | 3.0% |
| Cost per successful task | $4.43 | $1.57 | $1.02 |
| Implementation (with contingency) | $57,500 | $201,600 | $499,200 |
| 12-month TCO | about $105,335 | about $377,137 | about $1.196M |
The pattern is the point: engineering, review, support, and operations cost far more than raw tokens here. That can change for very high-volume or long-context workloads, so replace assumptions with production telemetry as early as possible.
Run a sensitivity analysis before approving the budget
Change one input at a time against Scenario 2 and see what moves the monthly bill.
| Change | Approximate monthly impact |
|---|---|
| Output tokens increase 50% | +$52.80 |
| Cache hit rate falls from 20% to 0% | +$15.54 |
| Human review doubles from 40 to 80 hours | +$2,400 |
| Maintenance increases 25% | +$2,100 |
| Tool API price rises from $0.02 to $0.10 | +$400 |
Optimize the largest driver in your own architecture, not automatically the token price. Also test request volume (minus 50% to plus 100%), retry rate, task-success rate, and growth in indexed documents.
Managed APIs versus self-hosted models
| Decision | Managed API | Self-hosted model |
|---|---|---|
| Upfront effort | Lower | Higher: GPUs, serving stack, optimization, security |
| Cost shape | Token or request based | Capacity based, including idle time |
| Scaling | Provider manages most of it | Your team provisions and benchmarks |
| Control | Limited by provider models, regions, quotas | More control over model, network, and data |
| Best fit | Fast delivery, variable load, early production | Steady high utilization or strict deployment limits |
Do not justify self-hosting by the absence of token invoices alone. Include idle capacity, redundancy, MLOps, patching, observability, and on-call coverage.
RAG versus fine-tuning
RAG adds recurring embedding, indexing, storage, retrieval, and context-token costs. It fits applications that must use proprietary, changing, or permission-controlled information.
Fine-tuning adds dataset preparation, training, evaluation, versioning, and regression testing. It shapes style and behavior but is not a substitute for current enterprise knowledge, and model support varies. For example, OpenAI's GPT-5.6 Terra page lists fine-tuning as not supported, so a fine-tuning requirement affects model choice before it affects budget.
Hidden costs finance teams should check
- Data cleaning, redaction, and permissions mapping
- Evaluation-set creation and ongoing regression testing
- Security reviews, adversarial testing, and compliance or legal review
- Non-production environments and test data
- Retry storms, agent loops, and duplicate tool calls
- Re-indexing, model migration, and prompt changes
- Network egress, private endpoints, and data residency
- Log retention, backup, and disaster recovery
- On-call coverage and incident response
- Training, adoption support, and process redesign
Practical ways to control cost
- Measure outcomes. Track cost per successful task, not just cost per token.
- Route by task. Send simple classification and extraction to a lower-cost model, and reserve capable models for work that evaluations show needs them.
- Limit context. Long histories, excess retrieval, and oversized tool responses raise cost and latency.
- Cache deliberately. Stable prompt prefixes can cut input cost, but cache writes are billed, so monitor reads per write.
- Cap retries and loops. Set execution limits, failure policies, and escalation thresholds.
- Improve retrieval first. Better chunking, metadata filters, and reranking reduce wasted context.
- Budget human review explicitly. Do not bury it inside "support".
- Load-test before procurement. Measure tokens, retrievals, latency, concurrency, and success rates on a representative workload.
- Set budgets and alerts. Track spend by workflow, model, team, and environment.
- Re-estimate quarterly. Prices, traffic, and models change faster than traditional infrastructure.
A better business case
The lowest token price does not give the lowest application cost. A cheaper model that causes more retries, failed tasks, and human corrections can raise cost per successful task, while a pricier model that finishes workflows in fewer calls can lower it. Put the financial model next to the evaluation framework: engineering measures quality and reliability, finance measures TCO, and the business owner measures whether successful tasks create enough value to justify both.
Plan the architecture and the economics together
Budgeting works best when architecture, evaluation, security, and cost modeling happen together. LuMay's enterprise AI services include generative AI development, RAG development, AI agent development, automation, and integrations. Use them to scope the architecture, integrations, and operating model before you set a production budget.
