Chat sessions and history

The three references

ChatSession::create([
    'tenant_type' => 'organization', 'tenant_id' => $org->id,
    'actor_type'  => 'user',         'actor_id'  => $user->id,
    'sessionable_type' => 'matter',  'sessionable_id' => $matter->id,
]);

tenant and sessionable are not the same axis, and conflating them is what makes a chat impossible to move to the next application. A background job holding only a session id still has to know who to bill, and the package cannot walk from a matter to the firm that owns it.

The subject is the one people get wrong. In a legal product it is the case the whole conversation lives inside, and it is real. In a real estate one there is usually no subject at all, because a chat ranges over many properties and a property is data the tools fetch, not a frame. If you find yourself wanting a session per record, that record is data, not a subject.

A null actor is what keeps personal memories out of an autonomous run: with nobody to scope to, nothing personal applies.

Replaying the conversation

Every turn replays the whole conversation, so a chat that goes well gets more expensive per turn until it hits the context limit and stops working. HistoryCompressor trades the older turns for a summary, once, and reuses it from then on.

$messages = $compressor->buildHistory($session, $systemPrompt);

'chat' => [
    'max_history_tokens' => 6000,
    'recent_messages_min' => 4,
],

Under the budget, everything goes through verbatim. Over it, the older turns are summarised by a cheap model, the summary is stored on the session with a watermark, and from then on only what came after the watermark needs folding in.

Two things it does not do, on purpose.

It never compresses the most recent messages. Whatever the budget says, the last few turns go through verbatim, because that is where the thread of the exchange lives and a summary of "what we just said" is where these systems get vague.

It never touches tool calls or their results. Those belong to the turn that produced them. Replaying them across turns is how a loop starts re-reading its own plumbing, and they crowd out the part worth reading.

The token count is a heuristic, four characters to a token, not a real tokeniser. It decides whether to compress, and being ten per cent out means compressing a turn early or late. That is not worth a dependency.

If compression fails it returns the previous summary rather than throwing. Losing the summary would silently drop everything the conversation established; keeping a slightly stale one costs a little context and nothing else.

new HistoryCompressor($client, model: 'gpt-4o-mini', language: 'Spanish');

Crossing the budget writes memories

Compressing means a substantial amount has been said, which is exactly when there is something worth remembering. So compression dispatches the distiller:

'memory' => ['distil_on_compress' => true],

That is the main path into memory. The scheduled sweep only catches conversations that stopped before ever getting here.

Messages

$session->messages;      // user, assistant, and tool, in order
$session->total_tokens;
$session->addTokens($response->totalTokens());

laragents_chat_messages holds the tool plumbing too: the assistant's tool calls with their ids, and each tool result with its tool_call_id. buildHistory() filters those out when replaying, but they are there, which is what makes a conversation debuggable six weeks later.

Per-conversation privacy

$session->anonymize_pii = false;

The second of the two switches redaction needs. The application-wide one is off by default; this one is on by default, so a product that redacts does it unless this user asked otherwise in this conversation.