Building AI agents with small language models, embedding models, and retrieval-augmented generation.
Start with a job worth delegating
An impressive conversation is easy to mistake for a useful agent. The more demanding test is whether the system can complete a defined piece of work with the right information and within the right authority.
Our applied AI work brings together small language models, embedding models, and retrieval-augmented generation to build agents. Each component has a different responsibility. The language model interprets and generates. The embedding model helps represent information for retrieval. RAG brings retrieved material into the generation process.
The task should determine how those pieces are assembled. A narrowly bounded classification or extraction task has different needs from an assistant searching a changing collection of operating documents.
Choose a model against the task
A smaller model can be a useful choice when deployment constraints, latency, and cost matter. The Phi-3 technical report is one published example of research into capable small language models. It is evidence for the category, not a guarantee about performance on a particular business workflow.
Model choice needs a representative evaluation set. The relevant question is whether the model succeeds on the intended job, including difficult examples and ambiguous inputs. Parameter count alone cannot answer that.
Sometimes the right architecture gives a small model a limited responsibility and escalates uncertain cases. A clear boundary can be more valuable than asking one model to perform every part of the process.
Retrieval must preserve the right context
The original RAG research combines language generation with retrieved external information. In a business application, the quality of that information path matters as much as the fluency of the answer.
Documents need sensible boundaries, useful metadata, and a version history. Retrieval needs to respect access permissions. A semantically similar passage from the wrong tenant, obsolete procedure, or unrelated product can be a worse input than no passage at all.
Embeddings help locate related material, but similarity is not proof that a passage answers the question. The agent still needs to identify what supports its response and recognise when the available context is insufficient.
Separate knowing from doing
Answering a question and changing a system are different responsibilities. An agent may be able to retrieve an operating procedure without being authorised to alter a production setting.
Tools should have narrow contracts, explicit permissions, and observable results. A model’s proposed action can be checked against those contracts before it reaches another system. High-consequence actions can require a person to review the actual proposed change.
The record should connect the request, retrieved evidence, chosen action, and tool result. Without that chain, a failure becomes difficult to reproduce and a success becomes difficult to trust.
Evaluate the whole loop
A useful evaluation separates retrieval quality, answer quality, and task completion. It also considers permission boundaries, outdated documents, missing evidence, and tool failures. A high-quality answer on an easy question does not demonstrate that an agent can recover from a broken dependency.
Measure what the workflow needs: whether the correct evidence was found, whether the response stayed within it, whether the action was authorised, and whether the result actually completed the task.
The strongest agent architecture gives each component a job it can perform reliably, then makes the handoffs visible. That is how a collection of models becomes a tool people can use in real work.
An agent is a system of responsibilities, not just a model with a prompt.
Working through a similar problem?
Bring us the operating context, the constraints, and the part that needs to work better.
Talk to Wireone Labs ↗