Data Ownership
Where every piece of backend-owned data lives: Supabase for relational, Qdrant for vectors, TypeSense for BM25, Redis for caches and locks, the local filesystem for system prompt backups.
The backend owns a small, well-defined set of data. The rest of the system lives in Supabase and is read or written by the frontend. This page lists exactly what the backend creates, where it stores it, and how it is keyed.
Tables
| Data | Store | Key |
|---|---|---|
| Books | Supabase books table | book_id |
| Characters | Supabase characters table | character_id |
| Chat sessions | Supabase chat_sessions table | session_id |
| Chat messages | Supabase messages table | session_id + created_at |
| System prompts (backup) | Backend filesystem | /prompts/{book_id}/{safe_name}.md |
The frontend creates books and chat sessions. The backend writes characters and chat messages, and updates books.processing_status after ingestion completes.
Indexes
| Data | Store | Key |
|---|---|---|
| Book text (vectors) | Qdrant collection {book_id} | chunk_index |
| Book text (BM25) | TypeSense collection {book_id} | {book_id}:{chunk_index} |
Qdrant collections use cosine distance with EMBEDDING_DIMENSION dimensionality (driven by the configured embedding provider). TypeSense collection names match the book_id exactly.
Redis Keys
| Data | Key | Value |
|---|---|---|
| Session context cache | chat:sess:{session_id} | Last 10 messages + summary |
| Character cache | chat:char:{character_id} | Full character row |
| Book cache | chat:book:{book_id} | Book + category data |
| Query-rewrite cache | chat:qr:{character_id}:{hash} | Narrative + keyword queries |
| Session write lock | chat:lock:{session_id} | Hex token, 30s TTL |
| Job queue | processing_queue (list) | Job dicts |
The job queue is a single Redis FIFO list shared by all backend instances. RPUSH enqueues, LPOP dequeues. There is no separate dead-letter queue; failed jobs are logged with their job id.
Filesystem
System prompts are written to SYSTEM_PROMPT_STORAGE_PATH/{book_id}/{safe_name}.md as a backup. The same text is also stored in the Supabase characters table. The filesystem copy is the source of truth only if the database row is ever lost.
Lifecycle
- Books are created by the frontend, owned by the frontend, and read by the backend for validation. The backend updates
processing_statusbut no other field. - Characters are created by the backend during ingestion. The frontend reads them to render the book-detail page and the chat UI.
- Chat sessions are created by the frontend. The backend reads them and updates
message_countandpreviewafter each chat. - Chat messages are written by the backend only. The frontend reads them to render the conversation.
- Qdrant and TypeSense collections are created by the backend during ingestion and never deleted. The frontend does not read from them directly.
- Redis caches are best-effort. They are repopulated from Supabase on miss and expire after 3600s.
- Redis locks have a 30-second TTL. The lock is released earlier via a Lua CAS script when work completes normally.
- System prompt files are written once during ingestion. They are never deleted or rotated by the backend.
Chat Flow
End-to-end walkthrough of POST /character/chat. Per-session lock, query rewrite, hybrid retrieval across Qdrant and TypeSense, rerank, stream, persist.
Concurrency Model
BookNLP singleton, JobManager capacity gate, per-session chat locks, Redis connection pool, and per-request async clients. How the backend keeps many concurrent requests from corrupting shared state.