I Wrote a Playwright Script to Test LLM Long-Term Memory - and Found 3 Critical Bugs

Deconstructing the problem: why manual testing fails for LLM long-term memory

We were building an LLM-powered customer support assistant, with LangChain’s ConversationSummaryBufferMemory handling long-term storage on the backend. The dream was simple: the user’s name, requests, and preferences should be remembered across sessions. The reality? Memory is probabilistic - the model forgets sometimes, the framework messes up sometimes, and edge cases (like concurrent sessions) can cause memory to leak from one user to another.

Manual testing feels impossible because:

• Repetition burns you out: Every regression means holding dozens of conversations again. No human can keep up.
• No precise assertions: Scanning a response and thinking “hmm, seems like it remembers” is way too subjective. Get one order number wrong and it explodes.
• Cross-session state is a nightmare to simulate: Cookie, localStorage, and session ID isolation - you just can’t reliably reconstruct that by hand.

Regular unit tests can verify the memory component in isolation, but the true end-to-end memory flow - from user input to memory summarization to cross-session retrieval - only reveals itself through real browser interaction. That’s when I turned to Playwright: it can click pages, log in, operate multiple windows just like a real user, and reliably extract the LLM’s reply text from the DOM for assertions.

Architecture decisions: why not Selenium or call the API directly?

A lot of people asked me: “Why do you need a browser to verify memory? Can’t you just call the /chat endpoint?” If we were only checking model outputs, sure. But the full long-term memory chain involves frontend session management and browser-side state retention. For example:

• The session ID might be generated by the frontend and stored in sessionStorage. The backend only recognizes that ID.
• Some memory cleanup logic is tightly coupled with the creation of a new chat session, so you have to simulate a “click new session” action.

So browser testing is unavoidable.

Next came tooling:

• Selenium is too heavy, its async wait mechanism feels dated, and it forces you to write piles of explicit waits for modern SPAs - every time I used it I wanted to smash my keyboard.
• Puppeteer is great, but my team works primarily in Python. Maintaining a cross-language test suite is a net loss.
• Playwright for Python offers native async/await (with a synchronous API too), auto-waiting for elements, network interception, and the ability to open multiple browser contexts in a single script to simulate different users. Perfect.

For architecture, we used pytest to drive Playwright. Each test case simulates a full memory flow:

• Create a user (log in, initialize a session).
• Inject a memory in session A (e.g., “My name is Wang Xiaoming, employee ID 9812”).
• Open an entirely new session B (simulating a visit the next day).
• Ask a memory-related question, extract the LLM response, and assert that key information is present.

This approach keeps every test case independent and parallelizable.

Core implementation: building a reusable memory verification script step by step

1. Shared fixtures: browser instance and page

This code solves the “start/stop browser for every test” problem. We use a session-scoped fixture to launch Chromium only once.

# conftest.py
import pytest
from playwright.sync_api import sync_playwright

@pytest.fixture(scope="session")
def browser():
    with sync_playwright() as p:
        # headless=False 可以在调试时看到浏览器操作，CI 里设 True
        browser = p.chromium.launch(headless=False, slow_mo=100)
        yield browser
        browser.close()

@pytest.fixture
def page(browser):
    context = browser.new_context()  # 隔离的浏览器上下文，相当于一个全新用户
    page = context.new_page()
    yield page
    context.close()

2. The simplest memory test: store in one session, retrieve in another

This test validates the most basic long-term memory correctness - that something you put in can be pulled out on a different day.

# test_memory.py
import re
import pytest

def test_basic_long_term_memory(page):
    page.goto("http://localhost:3000/login")
    # 登录（假设一个简单的表单）
    page.fill("input[name='username']", "alice")
    page.fill("input[name='password']", "secret")
    page.click("button:has-text('登录')")
    page.wait_for_selector(".chat-container")  # 等待主聊天界面出现

    # --- 会话 A：注入记忆 ---
    page.fill("textarea.chat-input", "我叫王小明，工号 9812，负责华东区售后。")
    page.click("button:has-text('发送')")
    # 等待 LLM 回复出现在最后一条消息中（必须确定有文字，不能只是 DOM 挂载）
    page.wait_for_selector(".message.assistant:last-child :text-is('好的')", timeout=10000)

    # 新建会话，模拟跨天
    page.click("button:has-text('新会话')")
    page.wait_for_selector(".chat-container")

    # --- 会话 B：验证记忆 ---
    page.fill("textarea.chat-input", "请问我的工号是多少？")
    page.click("button:has-text('发送')")
    # 关键点：不用 sleep，用文本断言等待
    page.wait_for_selector(".message.assistant:last-child", timeout=10000)