Insights · 12 min read
LLM integration explained: the four patterns for putting a large language model behind your existing software, what each costs to build and run, and the parts most teams get wrong.
By GGP Editorial
Most businesses that want AI already have the software. They have a support desk, a document store, an admin panel, a CRM full of customer notes. The question they bring us is not how to build an AI startup. It is how to put a large language model behind a system that is already running, without breaking it and without spending a year on it.
I write from the vendor side. GlobeSoft is a software company founded in 2018 with 40-plus engineers and more than 300 delivered projects, and a growing share of that work is wiring LLMs into products that were never designed for them. This guide covers what LLM integration actually means, the four patterns you can choose from, the parts that cost real money, and what a realistic budget and timeline look like.
If you have not settled on whether AI is worth the investment, start with how much does AI development cost. If you are still deciding between adding AI to existing software or building something new, read how to add AI to existing business software first. This article assumes that decision is made and the work is on the table.
LLM integration is putting a language model behind your product so it can read, write, or act on your data. It is not the same as opening a chatbot account and pasting in a prompt. It is also not the same as building a whole AI product.
Three examples make the difference concrete.
A support tool that drafts a reply to a ticket, with the model reading the customer's history from your database. A document assistant that answers questions against your internal policies and cites the page. A finance admin panel where a user types "show me all overdue invoices for our German clients" and the model turns that sentence into a database query and runs it.
In each case the LLM is one component inside software you already own. The work is in the plumbing around it: where the model sits, how your data reaches it, and how its answers are checked before they reach a user.
There are four ways to put an LLM behind an existing system, on a spectrum from cheap and fast to expensive and bespoke.
| Pattern | What it is | Best for | Rough build effort |
|---|---|---|---|
| Prompted API | Your code calls a hosted model with a prompt and gets text back | Drafts, summaries, classification, extraction | Low |
| Retrieval (RAG) | The model answers using your documents, fetched at query time | Q&A over policies, docs, knowledge bases | Medium |
| Tool calling | The model calls your APIs and acts on the result | "Do this in my system" requests, agents | Medium to high |
| Fine-tuned model | A model trained further on your data and tasks | Narrow, high-volume, repetitive tasks | High |
Most projects start in the first two rows and stay there. The prompted API is the easiest place to begin. Retrieval is where most of the real business value sits, because it lets the model answer from your data instead of its training data. Tool calling is where things get useful but also risky, because the model is now doing things, not just writing things. Fine-tuning is the rarest path and the one most teams reach for too early.
The retrieval-versus-fine-tuning decision is covered in detail in RAG vs fine-tuning. The short version: retrieval first, fine-tuning only for a specific, repetitive task with enough clean examples.
The fastest way to scope an integration is one question: does the answer come from your data or from the model's reasoning?
If the model is summarizing a document you give it, or classifying a ticket, or extracting fields from an email, a prompted API is enough. You send text, you get text back, you are done.
If the answer should come from your documents, policies, or product catalog, you need retrieval. This is the "chat with our docs" use case, and it is the most common integration we build.
If the model needs to do something in your system, like update a record, book a slot, or run a report, you need tool calling. This is where the agent conversation starts, and where you need the most guardrails.
| Request | Pattern |
|---|---|
| Draft replies to customer emails | Prompted API |
| Answer questions from our policy documents | Retrieval |
| Turn "show overdue invoices" into a query | Tool calling |
| Classify incoming support tickets | Prompted API |
| Extract line items from supplier invoices | Prompted API |
| A copilot that updates CRM fields from chat | Tool calling |
The model is the easy part. The four things that decide whether an integration ships, and whether it stays live, sit around the model.
Evaluation. You cannot tell whether the model is doing a good job by reading three sample answers. You need a set of test questions with known-good answers, and you run the model against them every time you change a prompt, a model, or a retrieval setup. Without this, every tweak is a guess.
Guardrails. A model that drafts an email is low risk. A model that can run a query or send a message is a different animal. Decide what the model is allowed to do, and check every output before it acts. For anything that touches customer data or money, apply the same controls we describe in security requirements for FinTech platforms.
Cost control. The per-token price of a hosted model is small and still falling, but it compounds. A retrieval system that pulls five documents and feeds them to the model on every question spends more than one that pulls two. The engineering that controls token spend usually matters more than the model price itself.
Latency. A support agent will wait two seconds for a drafted reply. A search box will not. Where the model sits in your user's flow decides whether you need streaming, a faster model, or a cached answer, and each choice has a cost.
Data. Where does your data go when it leaves your system and hits a model API? The answer differs by provider and contract. If your data is regulated, or your customers expect it to stay with you, this question comes before the build, not after. Some teams route only non-sensitive text to a hosted model and keep the rest on an open model they run themselves.
LLM integration follows the same cost logic as any software project, with one extra line item: the model itself.
A simple prompted-API integration, one use case, one model, wired into one system, usually lands between $10,000 and $30,000. A retrieval system over your documents, with a vector store, an ingestion pipeline, evaluation, and a UI, runs $30,000 to $80,000. A tool-calling agent that acts on your systems, with guardrails, logging, and a review flow, starts around $80,000 and goes up from there. These are ranges, not quotes. The only number that matters comes from a scoped proposal, which I wrote about in how to estimate a custom software project.
The running cost is smaller than most people expect and bigger than they plan for. Hosted model APIs charge per token, roughly a few dollars per million input tokens and more per million output tokens, depending on the model. For most internal tools the model bill is a few hundred dollars a month. What people miss is the engineering cost: the model needs monitoring, the prompts need updating as the model changes, and the evaluation set needs to keep growing. That ongoing work is a real budget line.
The cost drivers, roughly in order of impact:
A prompted-API integration can go live in two to four weeks. A retrieval system usually takes six to ten weeks. A tool-calling agent with guardrails and a review flow runs three to five months. The schedule is set by your data and your integrations, not by the model, which is available in a day.
Plan the first week around the evaluation set and the prompts, not the code. If you cannot write down twenty questions the model should answer correctly, you are not ready to build.
Demoing before defining. A chat window that answers three canned questions looks great in a demo. That is not a product. Define the real user task before you write code.
Treating the model as the product. The model is a component. The product is the workflow around it: the data, the checks, the UI, the logging.
Skipping evaluation. Without a test set, every change is a gamble. Build the test set first.
No human review for actions. If the model can write to your system, a person should approve anything risky until the guardrails have proven themselves. The moment you trust them is a decision, not a default.
Underestimating the data work. Retrieval is a data engineering problem wearing an AI costume. If your documents are a mess, the model answers from a mess.
Choosing the model before the task. People ask which model first. The question that matters is what the task is and what correct looks like. The model follows from that.
Not every integration needs an outside team. A single prompted-API integration, one use case, is something a competent in-house engineer can ship in a couple of weeks. Retrieval and tool calling are different. They need data work, evaluation, guardrails, and a sense of what breaks at volume, and they are the projects that go over budget when they are underestimated.
If you have the team and the time, the decision is simple. If you do not, the question is whether to hire or bring in a partner. I covered the trade-off in how to choose an AI development company, and the build-versus-buy logic in build vs buy AI software. If this is one step in a larger program, the pilot-to-production path in enterprise AI development is worth reading.
It is wiring a large language model into software you already run so the model can read your data and act on it where needed. The model is one component; the integration is the plumbing, data, and checks around it.
A simple prompted-API integration runs roughly $10,000 to $30,000. A retrieval system runs $30,000 to $80,000. A tool-calling agent starts around $80,000. Running costs are mostly engineering and monitoring; the model API itself is usually a few hundred dollars a month for internal tools.
Retrieval first. It lets the model answer from your documents without retraining, and it is cheaper to build and update. Fine-tuning is for a narrow, repetitive, high-volume task with clean training data. Details in RAG vs fine-tuning.
No. Retrieval feeds your data to the model at query time, which covers most use cases. Fine-tuning helps only for a specific style or task at high volume.
It depends on the provider and your contract. For regulated or sensitive data, check where the data goes, whether it is used for training, and whether you need to run an open model in-house instead. Treat this as a scoping decision, not an afterthought.
Two to four weeks for a prompted API, six to ten weeks for retrieval, and three to five months for a tool-calling agent with guardrails.
Yes, with guardrails. Define what the model can do, check outputs before they act, and keep a human in the loop for anything risky until the guardrails have proven themselves.
GlobeSoft is a software development company founded in 2018, with 40-plus engineers and more than 300 delivered projects for over 100 clients across the US, Brazil, South Africa, and Singapore. We build custom software, AI products, SaaS platforms, FinTech systems, and ERP and CRM on a stack that includes Java, Spring Boot, Go, Python, React, Vue, and Node. LLM integration sits inside the AI development services we already ship, from retrieval systems to tool-calling agents to full AI products.
If you are planning to put an LLM behind an existing system, talk to us about your project. Tell us what the system does, what the model should do, and where the data lives, and we will help you pick the pattern, scope the guardrails, and price it realistically before you spend anything.
Tell us what you are building and where you are today. We typically reply within 24 hours.
