Principles of Building AI Agents

Understanding Large Language Models (LLMs)

Large Language Models (LLMs) like OpenAI's GPT series, Anthropic's Claude, Google's Gemini, and Meta's LLaMA serve as the brains behind modern AI agents. They are capable of complex reasoning, comprehensive answering, and even executing intricate tasks when wrapped inside agent frameworks. As their token windows significantly expand (e.g., 2 million tokens in Google Gemini Flash 1.5 Pro), agents gain the remarkable ability to process vast contextual input in a single pass, thereby reducing or even eliminating reliance on retrieval mechanisms for contextual information. This larger context allows for more nuanced and complete understanding of user queries and historical data.

Source: interconnctd

Choosing the Right Model

• Hosted vs Open-Source: For rapid prototyping and ease of use, starting with hosted models (like those from OpenAI, Anthropic, or Google) is often recommended as it abstracts away infrastructure management. For greater flexibility, cost-efficiency in the long run, and control over data, transitioning to open-source models (such as LLaMA, Mistral, or DeepSeek) is a viable path. Model routing libraries can facilitate switching between providers and models without significant code changes.
• Model Size: Larger models generally offer superior reasoning capabilities and higher accuracy, making them ideal for complex tasks. However, this comes with inherent tradeoffs in terms of increased latency and higher operational costs. Smaller models, conversely, are faster and more economical but may exhibit less sophisticated reasoning. A common strategy is to prototype with more capable, albeit more expensive, models and then optimize for cost and speed by switching to smaller models or fine-tuning as the application matures.
• Context Window: The size of an LLM's context window (the amount of text it can process at one time) is a critical factor. Larger context windows simplify memory management for agents and greatly support long-form reasoning by allowing the agent to hold and process more information about the current task and past interactions without losing coherence.
• Specialization: Consider models optimized for specific tasks. For instance, Claude models are highly regarded for their reasoning abilities, while models like Codex (or similar specialized models) excel in code generation and understanding. Mixtral, an open-source model, is often chosen for its balance of speed and performance. Understanding the strengths of different models allows developers to select the best fit for their agent's primary functions.

Prompt Engineering

Prompts are the primary mechanism through which developers shape an agent's behavior, personality, and response format. Here's how to use them effectively:

• Zero/One/Few-shot: These techniques involve providing examples to the LLM to guide its output.
  • **Zero-shot prompting** involves asking a direct question without any examples.
  • **Single-shot prompting** provides one example of input and its desired output.
  • **Few-shot prompting** includes multiple input/output examples, offering more precise control over the model's response structure, tone, and content. More examples generally lead to better guidance but can increase token usage and processing time.
• System Prompts: Utilizing the API's dedicated system prompt feature allows you to define an agent's overarching personality, style of response, and general guidelines. This ensures consistency in how the agent interacts and presents information.
• Seed Crystals: When unsure how to craft the most effective prompt, the "seed crystal" approach involves asking the LLM itself to generate a better prompt based on your initial, less refined query. This iterative process can significantly improve prompt quality and agent performance.
• Formatting: LLMs are highly sensitive to the formatting of prompts. Techniques such as using capitalization for emphasis, employing XML-like tags, or structured markdown can help models follow instructions more precisely and generate responses in the desired format (e.g., JSON). Experimentation and evaluation are crucial to optimizing prompt effectiveness for your specific use case.

const prompt = `You are a helpful assistant. Answer in JSON.`;
const res = await llm.chat(prompt);

Building AI Agents

AI agents are sophisticated constructs that wrap around core LLMs, augmenting them with crucial functionalities such as access to external tools, persistent memory, defined roles, and complex workflows. This transformation allows them to act more like autonomous "employees": persistent, stateful entities responsible for maintaining context, executing tasks, and achieving specific goals. They go beyond simple chatbot interactions to become proactive problem-solvers.

Source: vectorize.io

Tool Calling Example

Tools are essentially functions that agents can invoke to perform specific actions that an LLM cannot do on its own, such as fetching real-time data, interacting with external APIs, querying databases, or executing calculations. Effective tool usage is fundamental to an agent's utility and relies heavily on clear communication with the model about the tool's purpose and how to use it.

Best practices for tool design include providing detailed descriptions in tool definitions and system prompts, utilizing specific input/output schemas, and employing semantic naming that clearly matches the tool's function. By breaking down complex problems into clear, reusable operations and implementing each as a dedicated tool, agents become significantly more capable and reliable.

defineTool({
  name: "getWeather",
  description: "Get weather for a city",
  parameters: { city: "string" },
  execute: async ({ city }) => await weatherAPI(city),
});

Memory Architecture

Memory is a critical component that allows agents to maintain context across interactions and provide continuity in their conversations and tasks. Since LLMs have limited context windows, explicit memory management is essential for long-term and complex interactions.

• Working Memory: This type of memory stores immediate session context, such as the user's current intent, names mentioned in the conversation, active preferences, and the most recent few turns of dialogue. It's akin to short-term memory, holding information relevant to the ongoing interaction.
• Long-Term Memory: This involves archiving historical interactions, user-specific metadata, or frequently accessed knowledge. It allows agents to retrieve and apply relevant past experiences or information, even from much older conversations, enhancing personalization and problem-solving. This often involves vector databases for semantic search and retrieval-augmented generation (RAG).
• Processors: Memory processors act as a middleware for memory. They control token limits by strategically truncating or summarizing older messages to prevent context window overflow. They can also filter memory types (e.g., distinguishing between conversational turns and internal thoughts), and prioritize relevance, ensuring that only the most pertinent information is fed to the LLM at any given time.

Memory Example

const memory = createMemory({
  summarizer: true,
  store: vectorDB,
  maxTokens: 2000,
});

Dynamic Agents & Middleware

Dynamic agents are those whose properties (such as their instructions, the specific LLM they use, or the tools available to them) can be determined and altered at runtime based on real-time factors like user input, environmental context, or user permissions. This offers significantly greater flexibility compared to static agents. Middleware plays a crucial role in enabling this adaptability by providing a layer for implementing various runtime functionalities.

• Model Routing: Directs requests to different LLMs based on criteria like user tier, task complexity, or cost optimization.
• User Authentication & Authorization: Manages which users can interact with an agent and what resources the agent can access on behalf of a user, ensuring secure and compliant operations.
• Tool Assignment: Dynamically assigns specific tools to an agent based on the current task or user context, ensuring the agent has only the necessary capabilities.
• Guardrails: Crucial for sanitizing input and output to prevent prompt injection attacks, filter requests for Personally Identifiable Information (PII), or steer conversations away from off-topic or unsafe content.

createAgent((ctx) => ({
  model: ctx.user.paid ? "gpt-4" : "gpt-3.5",
  prompt: `You're helping a user in ${ctx.language}`,
  tools: getUserTools(ctx.user),
}));

Model Context Protocol (MCP)

The Model Context Protocol (MCP) is emerging as a universal protocol for exposing tools and functionalities to AI agents, akin to how USB standardized connectivity for computer peripherals. Proposed by Anthropic and gaining rapid adoption, MCP enables seamless inter-agent communication and modular integration across different models and services. It functions as a standard for remote code execution, similar in concept to OpenAPI or RPC.

The MCP ecosystem includes "Servers" that wrap sets of MCP tools and communicate with "Clients" (models or agents) over HTTP. This standardization means vendors can ship MCP servers for their APIs, developers can create custom servers for specific functionalities, and registries can catalog the growing number of available servers. Frameworks like Mastra offer built-in abstractions for MCP server and client implementation, simplifying development.

If your application roadmap includes numerous third-party service integrations, building an MCP client can significantly streamline development. Conversely, if you are developing a tool or API intended for use by other agents, shipping an MCP server is advisable to ensure broad compatibility and ease of adoption. Challenges within the MCP ecosystem currently include fragmented discovery mechanisms, the need for standardized quality assurance, and variations in configuration and client implementations.

Graph-Based Workflows

While individual agents offer immense flexibility, graph-based workflows provide a powerful technique for building LLM applications that require more predictable and structured output. Instead of chaining prompts linearly, these workflows allow for the definition of complex branching logic, resilient retries, necessary suspensions, and parallel execution of tasks. They excel at breaking down large problems, defining explicit decision trees, and having agents make smaller, precise binary decisions rather than single, large, ambiguous ones.

• Core Operations: Workflows leverage fundamental operations like:
  • **Branching:** Triggers multiple LLM calls on the same input, allowing for parallel execution of tasks (e.g., checking multiple conditions or generating diverse responses simultaneously).
  • **Chaining:** Sequences operations where each step waits for the preceding one to complete and has full access to its results, forming a clear execution path.
  • **Merging:** Combines results from divergent branched paths, consolidating information or outcomes before proceeding to the next stage.
  • **Conditions:** Allows workflows to make dynamic decisions based on intermediate results, with conditional path execution defined on child steps. This enables intelligent routing and adaptive behavior.
• Suspend and Resume: Workflows often need to pause execution while awaiting external input, such as confirmation from a human user or data from an external system. The ability to persist the workflow's state and precisely resume execution later (`.suspend()` and `.resume()` functions) is crucial for handling such arbitrary delays and building human-in-the-loop systems.
• Best Practices: To maximize the effectiveness of graph-based workflows, it's advisable to compose steps such that their input and output are meaningful for tracing and debugging. Additionally, decompose complex operations into granular steps, ensuring that each LLM call performs a single, well-defined task. Complex functionalities like loops and sophisticated retry mechanisms can be robustly built by combining these fundamental workflow primitives.

workflow
  .step("loadUser")
  .then("summarize")
  .if("needsApproval")
  .suspend()
  .resume("notifyUser")

Streaming UX

To enhance the user experience and reduce perceived latency in LLM applications, streaming outputs token-by-token is essential. Instead of waiting for a complete response, users receive real-time updates as the agent processes information, making the application feel fast and responsive. Furthermore, logging and displaying intermediate reasoning steps significantly boosts transparency and trust. Users can see the agent's thought process, understand how it arrived at its conclusion, and gain confidence in its capabilities, even during complex, multi-step operations. This provides reassurance and a sense of progress, transforming what could be a frustrating wait into an engaging experience.

Conclusion

Building truly great AI agents transcends simple prompt engineering; it demands a holistic approach to designing adaptive, modular, and intelligent systems. These systems must be capable of observing their environment, learning from interactions, and scaling efficiently to meet evolving demands. The journey of agent development, though dynamic, benefits from a foundational principle: start simple. Begin with one model, integrate one essential tool, and establish a basic memory system. From this solid base, you can progressively layer on complexity, introducing more sophisticated tools, advanced memory architectures, dynamic behaviors, and intricate graph-based workflows as your agent's capabilities grow. This iterative approach ensures a robust and maintainable path to building powerful AI agents.