Agent Memory: Short-Term vs Long-Term

By Kokil Thapa | Last reviewed: September 2026

Agent memory: short-term vs long-term is the design choice that separates a chatbot that forgets after refresh from a system that recalls a client's case file, preferences, and prior decisions. Short-term memory lives inside the model's active context. Long-term memory lives in databases, vector indexes, and application records you control. If you are building AI agents for production web apps, you need both layers mapped clearly before you wire tools, APIs, or billing.

What is agent memory and why does short-term vs long-term matter?

Agent memory is any mechanism that lets an autonomous or semi-autonomous AI system retain and reuse information across steps, turns, or sessions. Without memory, every request starts cold. The model only sees what you pass in that single call.

Short-term memory is bounded by the model's context window. It is fast, implicit, and expensive at scale because you pay tokens for every byte you resend. Long-term memory is external storage you query selectively. It is cheaper per fact over time but adds latency, retrieval quality risk, and data-governance work.

On production Laravel applications I have extended with LLM APIs, the split is rarely theoretical. A legal-tech portal needs short-term memory to track the current intake form. It needs long-term memory to recall uploaded documents and prior consultation notes. Mixing those roles in one blob of context is how teams blow budgets and leak data across tenants.

Think of memory types the way operating systems do. RAM is short-term. Disk is long-term. You do not load an entire hard drive into RAM for every keystroke. You load the pages you need. Agent design follows the same rule.

How does short-term memory work in AI agents?

Short-term memory is everything the model can read on the current inference call without an external lookup. That includes system instructions, recent chat turns, tool results, and any scratchpad text your orchestrator injects.

Context window as the hard ceiling

Every major provider publishes context limits. Models advertised as "long context" still charge per token and slow down as the window fills. Strategies from long-context LLM limits apply directly here: summarisation, sliding windows, and hierarchical compression.

A practical sliding-window buffer keeps the last N user and assistant messages plus a compressed summary of older turns:

// Laravel service sketch — short-term conversation buffer
final class ShortTermMemory
{
    public function __construct(
        private readonly int $maxTurns = 12,
    ) {}

    public function append(int $sessionId, string $role, string $content): void
    {
        ConversationTurn::create([
            'session_id' => $sessionId,
            'role'       => $role,
            'content'    => $content,
        ]);

        $overflow = ConversationTurn::where('session_id', $sessionId)
            ->orderByDesc('id')
            ->skip($this->maxTurns)
            ->pluck('id');

        ConversationTurn::whereIn('id', $overflow)->delete();
    }

    public function toMessages(int $sessionId): array
    {
        return ConversationTurn::where('session_id', $sessionId)
            ->orderBy('id')
            ->get(['role', 'content'])
            ->toArray();
    }
}

Scratchpads and tool output

When an agent calls a CRM API or runs SQL, the raw JSON often exceeds what you want permanently stored. Short-term memory holds that payload for the current reasoning chain. After the task completes, persist only the distilled outcome to long-term storage.

I follow the same pattern described in building agents with tool use: tools return structured data, the orchestrator trims it, and the model sees a bounded excerpt. Unbounded tool dumps are a top cause of context overflow and hallucinated field names.

When short-term memory is enough

• Single-session workflows: form filling, code review on one PR, one support ticket.
• Tasks where full history fits comfortably under roughly 30–50% of the context budget.
• Stateless API endpoints that receive all required facts in the request body.
• High-stakes turns where you want zero retrieval noise from old embeddings.

How do you implement long-term memory for AI agents?

Long-term memory stores facts outside the model weights. Retrieval happens at runtime through search, SQL filters, or hybrid pipelines. The model never " remembers" permanently unless you write data back through your application.

Memory types worth modelling explicitly

1. Episodic memory — timestamped events: "User uploaded passport scan on 2082-04-12."
2. Semantic memory — stable facts: "Client prefers email over SMS."
3. Procedural memory — how-to rules: "Always verify PAN before generating invoice."

Store episodic data in relational tables with tenant IDs and audit columns. Store semantic chunks in a vector index after embedding. Keep procedural rules in version-controlled prompt templates or policy tables, not buried inside chat logs.

Vector retrieval pattern

The standard long-term pipeline embeds text, stores vectors with metadata, and retrieves top-k chunks at query time. Official guidance from OpenAI's embeddings documentation and patterns in the LangChain memory concepts guide align on the same shape even if your stack is PHP and Laravel rather than Python.

// Retrieval step before each agent turn
$embedding = $openAi->embeddings($userMessage);

$chunks = DocumentChunk::query()
    ->where('tenant_id', $tenantId)
    ->nearestNeighbors('embedding', $embedding, 8)
    ->get();

$memoryBlock = $chunks->map(fn ($c) => $c->content)->implode("\n---\n");

$messages = [
    ['role' => 'system', 'content' => $systemPrompt],
    ['role' => 'system', 'content' => "Relevant memory:\n{$memoryBlock}"],
    ...$shortTerm->toMessages($sessionId),
    ['role' => 'user', 'content' => $userMessage],
];

Application-owned records beat raw chat logs

For client portals such as those in our Mijar Law Associates portfolio, the source of truth is structured case data, not whatever the model said last Tuesday. Long-term memory should index authoritative rows: matter status, document titles, fee agreements. Chat transcripts are supplementary evidence, not primary storage.

Use JSON formatting tools during development to inspect tool payloads before you decide what gets embedded. Garbage in the vector index is harder to debug than garbage in one request.

Write path: when to persist

Do not embed every assistant reply automatically. That creates stale, contradictory memories. Persist when:

• A user explicitly confirms a preference or fact.
• A tool successfully commits business data to your database.
• A summarisation job extracts durable entities from a closed session.
• Compliance requires an immutable audit trail regardless of model output.

Agent memory short-term vs long-term: which should you use when?

The answer is almost always both, but with strict boundaries. Short-term carries working state. Long-term carries durable knowledge. Treat retrieval as a deliberate act, not a default dump of everything you have stored.

Verdict: Use short-term memory for anything the agent must reason about right now. Use long-term memory for anything that must survive browser refresh, handoff to a human, or a return visit months later. Never substitute a bigger context window for a data model you should have built anyway.

Cost control ties directly to this split. AI rate limits and cost optimisation become manageable when you stop re-sending 80 KB of history on every turn. Fetch five relevant chunks instead.

What are common mistakes when designing agent memory?

Teams new to multi-agent patterns often collapse memory layers because prototypes feel easier that way. Production pain follows quickly.

Treating RAG as perfect memory

Retrieval-augmented generation is a search shortcut, not guaranteed recall. Similar embeddings do not mean semantically correct for your business rule. Pair vector search with metadata filters: tenant_id, matter_id, doc_status=verified.

Skipping tenant isolation

One shared index without strict filters is a data breach waiting for a weird cosine score. Scope every read and write by tenant and user role. This mirrors standard API development practice: authorisation happens in your code, not in the model's good intentions.

Embedding unverified model outputs

If the assistant hallucinates a court fee and you embed it, you have poisoned long-term memory. Only persist facts that passed validation, user confirmation, or a trusted tool response. Aligns with themes in AI governance basics.

No expiration or correction path

Memories go stale: prices change, laws update, staff leave. Build update and tombstone operations. For Nepal-facing legal content, pair automated recall with human-reviewed source documents rather than model paraphrases alone.

Confusing PHP memory limits with agent memory

Server-side OOM errors from unbounded arrays are a different problem, though the symptom looks similar. See PHP memory limits and leak patterns for FPM tuning. Agent memory is an architecture concern, not a memory_limit ini tweak.

How do you combine short-term and long-term memory in production?

Production agents use a memory orchestration layer that runs before every model call. The sequence is predictable and testable.

1. Authenticate the user and resolve tenant_id.
2. Load short-term buffer for the session from Redis or SQL.
3. Embed the latest user message and retrieve long-term chunks with filters.
4. Fetch structured facts from your app DB (orders, bookings, matter status).
5. Merge into a token-budgeted prompt; drop lowest-score chunks first.
6. Run inference; execute tools if requested.
7. Append trimmed turns to short-term storage.
8. Queue async jobs to summarise or embed only validated new facts.

On booking systems like Adventure Third Pole Trek, long-term memory might recall a trekker's dietary notes while short-term memory tracks the current availability check across three tool calls. The user sees one coherent reply. Under the hood, two memory systems cooperated.

For enterprise deployments, expose memory operations through your existing enterprise application layer: admin screens to delete user memory, export GDPR packages, and re-index documents after template changes. Operators need buttons, not SSH and prayer.

Observability matters. Log retrieval IDs, chunk scores, token counts, and which memories were injected. When the agent says something wrong, you want to know whether bad retrieval or bad reasoning caused it. AI-assisted debugging workflows start with those logs.

Rate limiting still applies at the memory boundary. Bulk re-embedding after a document upload spike can hit the same provider quotas as chat. Use queues and backoff consistent with API rate limiting patterns.

If you are outsourcing the build, specify memory requirements in the brief: retention period, deletion SLA, tenant model, and which data is authoritative. Vague "make it remember users" requests become expensive rewrites. Our AI integration and automation service typically scopes memory architecture before any prompt engineering.

Human handoff is another integration point. When a support agent opens the same ticket, they should see structured memory, not a raw 40-turn chat export. Short-term dialogue summarises into a long-term case note. That pattern works for Notary Nepal-style service portals where staff continuity beats model verbosity.

Security review checklist:

• Encrypt embeddings at rest if they represent sensitive text.
• Never pass one user's retrieved chunks into another user's session.
• Redact PAN, passport numbers, and payment tokens before embedding.
• Version procedural prompts; do not rely on the model to recall policy changes.
• Test deletion: when a user requests erasure, vectors and rows must both go.

Reference architectures from Anthropic's published research emphasise tool-grounded answers over bloated context. That aligns with practical engineering: smaller short-term windows plus precise long-term retrieval often beat brute-force million-token prompts on cost, speed, and accuracy.

Key Takeaways

• Short-term memory is the active context window plus trimmed tool output; it dies with the session unless you promote facts outward.
• Long-term memory lives in your databases and vector indexes; treat the model as stateless between calls.
• Always filter retrieval by tenant and role; never trust embedding similarity alone for authorization.
• Persist only validated or tool-confirmed facts — never auto-embed raw assistant prose.
• Budget tokens deliberately: retrieve top-k long-term chunks instead of resending full chat history.
• Give operators admin tools to inspect, correct, and delete memory — compliance depends on it.

People Also Ask

What is the difference between short-term and long-term memory in AI agents?

Short-term memory is information loaded into the model's current prompt, usually recent messages and tool results bounded by the context window. Long-term memory is stored externally in databases or vector indexes and retrieved when relevant. Short-term handles the live task; long-term preserves user history and business facts across sessions.

Do AI agents remember conversations permanently?

Not by default. Commercial LLMs do not retain your chat after the API call unless your application writes data somewhere. Permanent recall requires you to implement long-term storage, retrieval, and retention policies. The model weights themselves are not your product memory.

Is RAG the same as long-term agent memory?

RAG is one implementation pattern for long-term memory, typically vector search over document chunks at query time. Full long-term memory can also include SQL records, key-value preferences, and procedural rules in prompts. RAG alone does not cover structured transactional data or confirmed user settings unless you design for those separately.

How much does agent memory cost to run?

Short-term memory costs scale with input tokens on every turn; long chats get expensive fast. Long-term memory adds storage, embedding fees, and search infrastructure — often Rs 2,000–15,000/month (~USD 15–110) for small SaaS workloads depending on volume. Hybrid designs with summarisation and selective retrieval usually cost less than stuffing entire histories into context.

Ship agents that remember the right things

Agent memory: short-term vs long-term is not a vendor feature you toggle on. It is application architecture: buffers, indexes, policies, and deletion hooks you own. Start with a clear data model, add retrieval second, and keep the context window for reasoning — not archival storage. If you want help scoping memory for a Laravel portal, booking platform, or internal copilot, review our portfolio and background, then contact us with your retention and compliance requirements spelled out.