Part 2: Conversational AI Agent Techniques

Building a conversational AI agent isn't structurally difficult—you take a user's text, send it to an API, and return the response. But building one that actually works, that remembers who you are, costs a reasonable amount to run, and doesn't feel like talking to a rigid corporate form, is an entirely different engineering challenge.

This post is a deep dive into the conversational techniques and memory systems behind Aki, the AI companion I built on Telegram. I realized that a good companion needs to handle a few specific problems: overcoming LLM statelessness, bridging short-term exchanges with a long-term sense of identity, managing context window limits, and utilizing structured schemas to make the AI output feel more human.

The source code is on GitHub, found here. The project landing page is at aki-landing.vercel.app. The first part of this series covering the Telegram infrastructure and my personal journey building this is here.

When I started designing the agent's core "brain," I knew I didn't want to tightly couple my codebase directly to the OpenAI or Anthropic SDKs. Models change too rapidly, and I needed the flexibility to benchmark different providers against each other without rewriting my logic.

I integrated LiteLLM as a universal adapter. It proxies requests and normalizes the input/output schemas so I can treat entirely different LLM providers the exact same way:

# utils/llm_client.py
response = await litellm.acompletion(
    model=model,          # "claude-haiku-4-5-20251001" or "gpt-4o" — same API
    messages=messages,
    temperature=temperature,
    max_tokens=max_tokens,
)

This abstraction proved essential. It gave me the freedom to swap models on the fly by simply changing an environment variable. If Anthropic experiences an outage, I can instantly route traffic to OpenAI.

With LiteLLM abstracting the APIs, I quickly realized that treating every LLM task equally was a massive waste of resources. I didn't need the heaviest, most expensive frontier model to summarize a chat log.

I created a model-routing configuration that splits workload across different LLMs based on task complexity:

# Configuration Router
MODEL_CONVERSATION  = "claude-haiku-4-5-20251001"   # Fast, cheap. Runs for every message.
MODEL_MEMORY        = "claude-haiku-4-5-20251001"   # Background task. Latency doesn't matter here.
MODEL_INSIGHTS      = "claude-sonnet-4-5-20250929"  # Expensive but rare. Used for weekly features.
MODEL_DAILY_MESSAGE = "claude-haiku-4-5-20251001"   # Simple generation. Haiku is more than enough.

The golden rule here became: use the cheapest model that consistently clears the quality threshold. I reserved the heavy, expensive models exclusively for high-value, low-frequency tasks (like generating deep weekly insights about the user), while keeping the moment-to-moment chat relegated to faster, cheaper variants.

Early on, I tried prompting the LLM with generic instructions like "respond naturally." It was incredibly inconsistent. LLMs, left to their own devices, will often default back to "As an AI..." safety guardrails or adopt an overly formal, robotic tone.

To fix this, I abandoned free-text output and enforced a strict XML schema for every single response. I chose XML over JSON because I found LLMs are far less likely to break XML formatting when generating creative text, and it's infinitely easier to cleanly parse using basic regex tools.

<?xml version="1.0"?>
<message>
  <thinking>
    Internal reasoning. Never shown to user.
    What's happening? Did they ask a question or make a statement?
    Should I match their energy, react, or ask a follow-up?
  </thinking>
  <emoji>❤️</emoji>
  <response>
    YESS okay that's so exciting[BREAK]wait tell me everything
  </response>
</message>

In the backend, I built a cascading parser for Aki. It first looks for the strict <?xml version="1.0"?> wrapper. If the model hallucinations and breaks the root tag, it falls back to regex-extracting standalone <thinking> and <response> blocks. If the model forgets XML entirely, it defaults to breaking the output along standard line breaks or [BREAK] tags. This multi-layered parsing ensures Aki never crashes even if the AI output gets messy.

A major revelation was realizing that the instructions for how to format the XML and the personality of the agent are fundamentally two different things.

I split my system implementation into two halves: the System Frame (the structural scaffolding that demands the <thinking> and <emoji> blocks) and the Persona (a swappable text block that dictates character).

# system_frame.py
SYSTEM_STATIC = """
Your name is Aki. You exist inside the user's phone.
{persona}
---
FORMAT:
Respond with valid XML. Start with the XML declaration...
"""

By separating these, the behavior instructions stay perfectly intact while the persona itself can be hot-swapped effortlessly. The persona prompt itself heavily discourages "therapy speak" and explicitly authorizes the LLM to be distracted, dry, or even slightly off. Sometimes the right response really is just "that's wild" instead of a five-sentence psychological breakdown.

The <thinking> block was the single most impactful change to the conversational quality. I realized that the LLM needed a dedicated "warm up" workspace before outputting the visible message.

When the model is forced to output the final message immediately, it often grasps for the most obvious, generic response. By forcing it to first write a private <thinking> block, it gets the space to reason through complex emotional subtext before committing to an answer. It qualitatively improved the agent's empathy. By saving these <thinking> blocks into the PostgreSQL memory database right next to the message, it becomes trivial to debug why the agent hallucinated or reacted poorly from the dashboard.

You'll notice an <emoji> block in the XML schema above. I built a dedicated engine strictly for Telegram reactions and stickers to make the agent feel less sterile and robotic.

Instead of sending the LLM's chosen emoji as standard text padding, my backend intercepts it. If the emoji matches a predefined set of native Telegram reactions (like 👍, 🔥, or 🤯), the API directly applies it to the user's previous message bubble.

Even better, I built a sticker translation system. The LLM has no idea what a Telegram sticker pack is, but it knows how to use basic Unicode emojis. I created a stickers.json registry that maps hundreds of generic emojis to specific Telegram Sticker File IDs. If the LLM generates 😭, the engine randomly picks from an array of hand-picked crying meme stickers and fires it off using native Telegram payload syntax. To prevent this from becoming overwhelming, these sticker interactions are gated behind randomized counters—the bot might react on message #3, but wait until message #8 to drop a sticker.

A jarring flaw in many chatbots is time-blindness. If you say "I'm going to sleep," and the bot doesn't know it's 3:00 AM your time, its response will feel hollow.

Every system prompt I generate injects the user's localized time of day. I store the user's IANA timezone upon registration (e.g. America/New_York) and convert all UTC timestamps in the prompt to their local variant using pytz. Furthermore, the LLM uses dateparser internally any time it tries to schedule a reminder or proactive reach-out, ensuring "tomorrow morning" means their morning.

Since LLMs have no persistent state, they suffer from amnesia. Every API call is a completely blank slate. I had to build a "mind" in my PostgreSQL database to manually reconstruct context for every interaction.

I structured the database to follow how a human actually processes and retains information:

User
├── id, telegram_id, name, timezone
├── reach_out settings

Conversation (the raw log)
├── user_id, role, message
├── thinking (LLM internal reasoning)
└── timestamp

DiaryEntry (the processed mind)
├── entry_type: summary, memory, or insight
├── content, importance
├── exchange_start, exchange_end
└── timestamp

I found that I needed to conceptually differentiate between remembering an event and understanding a trait.

1. compact_summary (Events): This records standard factual history. For example: "On March 5, they talked about their job interview. They felt nervous but thought they answered the technical questions well."
2. conversation_memory (Traits): This records deeper psychological or behavioral reflections. For example: "They tend to mask anxiety with dark humor. They care deeply about their career trajectory, even if they pretend they don't."

To manage cost and prevent the database from exploding, I tied the generation of both memories strictly to message intervals defined in my environment settings (COMPACT_INTERVAL and MEMORY_ENTRY_INTERVAL, both defaulting to 30 messages).

When the chat hits this 30-message integer limit, the orchestration script intercepts the flow and uses asyncio.create_task to fire off two parallel, non-blocking background workers. Worker A asks a cheap, fast model (like Haiku) to compress the actual chat log into a compact_summary. Worker B feeds that identical chat log into a heavier, smarter model and asks it to psychoanalyze the exchange to produce a conversation_memory trait.

By storing both types of records, the agent can reference a past event while also understanding the emotional weight behind it, without adding a single millisecond of latency to the user's active front-end experience.

A companion shouldn't just sit statically waiting for you to query it. Real friends text first. They reach out when they haven't heard from you.

I built an asynchronous background task utilizing the python-telegram-bot strictly native JobQueue module (run_repeating(..., interval=3600)). Every 60 minutes, the orchestrator executes a wide asyncpg SQL sweep across the Neon PostgreSQL database, specifically looking for users who have not interacted with Aki in over 12 hours (the min_silence_hours threshold). When the scanner flags an inactive user, it gathers their recent compact_summary entries and passes them through a specialized proactive prompt.

Instead of generating a generic "checking in like a wellness app" message ( "How are you feeling today?" ), the prompt forces Aki to scan the memory exclusively for unfinished threads.

The result is a notification that feels organic: "Hey, did you ever finish writing that python script you were stressing over yesterday?"

Aki uses these reach-outs to prove it's actively "thinking" about the user even when the chat window is closed. Naturally, users can configure the frequency, quiet-hours, or entirely disable these reach-outs via native Telegram commands (such as /reachout_settings which invokes an inline UI menu).

The key takeaway from building the LLM layer of this agent was learning how to separate the frame from the personality.

The XML structure, the memory database triggers, the thinking blocks, and the time injection represent the frame. They dictate how the agent thinks. The actual system prompt describing its attitude represents the personality. By keeping these decoupled, I can entirely rewrite the agent's character just by swapping out a single text string, without ever touching the complex reasoning architecture beneath it.

In Part 3, I'll dive into the raw performance architecture: how I made this entire system run concurrently using Python's asyncio, how I handle debouncing, and how I manage the financial costs of rate limits and prompt caching.