feat: migrate from PocketBase to PostgreSQL with Drizzle ORM
- Add @project-e/db package with Drizzle schema and migrations - Replace PocketBase client with PostgreSQL-based database client - Migrate auth from custom to NextAuth.js - Add Docker Compose with PostgreSQL container - Update worker to use new database client - Remove PocketBase-specific files and migrations - Add drizzle config and initial migration
This commit is contained in:
+63
-461
@@ -1,505 +1,107 @@
|
||||
# Architecture Documentation
|
||||
|
||||
Project E is a monorepo application built on Next.js with PocketBase as the data layer. This document explains the system design, data flow, and key architectural decisions.
|
||||
Project E is a Next.js monorepo that stores application data in PostgreSQL 16 through Drizzle ORM. NextAuth credentials authentication manages user sessions. This document describes the implemented system.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [System Overview](#system-overview)
|
||||
- [Three-Layer Architecture](#three-layer-architecture)
|
||||
- [Data Flow](#data-flow)
|
||||
- [Realtime Architecture](#realtime-architecture)
|
||||
- [Worker Architecture](#worker-architecture)
|
||||
- [MCP Server Architecture](#mcp-server-architecture)
|
||||
- [Security](#security)
|
||||
- [Performance](#performance)
|
||||
- [Accessibility](#accessibility)
|
||||
|
||||
## System Overview
|
||||
## System overview
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ Client (Browser) │
|
||||
│ Next.js (React 19) · Zustand · SSE Client · PocketBase SDK │
|
||||
└──────────────────────────────┬───────────────────────────────────────┘
|
||||
│
|
||||
┌──────────┴──────────┐
|
||||
│ │
|
||||
REST API SSE /api/realtime
|
||||
/api/* (PocketBase events)
|
||||
│ │
|
||||
┌───────────────────▼─────────────────────▼────────────────────────────┐
|
||||
│ Next.js Server (apps/web) │
|
||||
│ │
|
||||
│ ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ ┌───────────┐ │
|
||||
│ │ API Routes │ │ MCP Server │ │ Auth │ │ Middleware │ │
|
||||
│ │ (CRUD ops) │ │ (61 tools) │ │ (cookies) │ │ (routing) │ │
|
||||
│ └──────┬──────┘ └──────┬───────┘ └──────┬──────┘ └───────────┘ │
|
||||
│ │ │ │ │
|
||||
│ ┌──────▼────────────────▼──────────────────▼──────────────────────┐ │
|
||||
│ │ Service Layer │ │
|
||||
│ │ task-service · habit-service · note-service · report-service │ │
|
||||
│ │ webhook-service · agent-mention-service · project-service │ │
|
||||
│ └──────────────────────────┬──────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────────────────▼──────────────────────────────────────┐ │
|
||||
│ │ PocketBase Client │ │
|
||||
│ │ createPocketBaseClient() · createAdminClient() │ │
|
||||
│ └──────────────────────────┬──────────────────────────────────────┘ │
|
||||
└─────────────────────────────┼────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────────▼────────────────────────────────────────┐
|
||||
│ PocketBase (SQLite) │
|
||||
│ Auth · Collections · Realtime · File Storage · Admin UI │
|
||||
│ 30+ collections: tasks, habits, projects, notes, reports, ... │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ Background Worker │
|
||||
│ Polls queue_jobs collection · Exponential backoff · 5 job types │
|
||||
│ webhook_delivery · agent_mention · report_generation │
|
||||
│ recurring_task · cleanup │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
Browser
|
||||
│
|
||||
├── REST requests and Server-Sent Events
|
||||
▼
|
||||
Next.js application (apps/web)
|
||||
├── App Router pages and API routes
|
||||
├── NextAuth credentials provider and JWT sessions
|
||||
├── Services, validation, and collection adapter
|
||||
├── MCP server
|
||||
└── Drizzle ORM
|
||||
│
|
||||
▼
|
||||
PostgreSQL 16
|
||||
├── users
|
||||
├── records (JSONB application data)
|
||||
└── project_e_events notifications
|
||||
│
|
||||
▼
|
||||
Background worker
|
||||
└── Polls queue_jobs records and processes asynchronous work
|
||||
```
|
||||
|
||||
## Three-Layer Architecture
|
||||
## Application layers
|
||||
|
||||
The application separates concerns into three layers.
|
||||
### Presentation
|
||||
|
||||
### Presentation Layer
|
||||
`apps/web/app/` contains App Router pages and API routes. `apps/web/components/` contains UI components. Zustand stores and hooks manage client-side state and consume API and SSE updates.
|
||||
|
||||
**Location:** `apps/web/app/` (pages) and `apps/web/components/` (UI)
|
||||
### Business logic
|
||||
|
||||
The presentation layer handles rendering, user interaction, and client-side state.
|
||||
`apps/web/lib/services/` contains entity behavior such as task, habit, note, report, project, webhook, and agent-mention services. `packages/shared/` provides Zod schemas and TypeScript types for validation across the app and worker.
|
||||
|
||||
**Key components:**
|
||||
### Persistence
|
||||
|
||||
- **Pages**: Next.js App Router pages organized by route groups: `(auth)` for login/signup, `(dashboard)` for the main application
|
||||
- **UI Components**: shadcn/ui primitives in `components/ui/`, feature components in `components/`
|
||||
- **Client State**: Zustand stores in `lib/stores/` manage UI state (sidebar, filters, theme, timer)
|
||||
- **Hooks**: Custom hooks in `hooks/` encapsulate reusable logic
|
||||
`packages/db/src/schema.ts` defines Drizzle's PostgreSQL schema. `packages/db/src/index.ts` creates a Drizzle client from `DATABASE_URL`.
|
||||
|
||||
The presentation layer communicates with the business logic layer through:
|
||||
1. REST API calls (fetch to `/api/*`)
|
||||
2. Zustand store actions (which call the API)
|
||||
3. SSE events from the realtime endpoint
|
||||
The `users` table stores user identity, email addresses, and bcrypt password hashes. The `records` table stores collection data in JSONB with a collection name and timestamps. `apps/web/lib/database.ts` exposes collection operations over those records and emits PostgreSQL notifications after creates, updates, and deletes.
|
||||
|
||||
### Business Logic Layer
|
||||
## Authentication and authorization
|
||||
|
||||
**Location:** `apps/web/lib/services/` and `packages/shared/`
|
||||
NextAuth uses the credentials provider and JWT sessions. A user signs in with an email address and password. The authorization flow normalizes the email address, reads the user from PostgreSQL, and compares the password against its bcrypt hash.
|
||||
|
||||
The business logic layer handles validation, data transformation, and orchestration.
|
||||
On an empty `users` table, a sign-in matching `INITIAL_ADMIN_EMAIL` and `INITIAL_ADMIN_PASSWORD` creates the first account. Set `NEXTAUTH_SECRET` for each environment and keep it stable to preserve active sessions.
|
||||
|
||||
**Services:**
|
||||
API routes require an authenticated user through the application's authentication middleware. MCP requests use agent API keys and permission tiers. The `read_only` tier limits agents to read operations.
|
||||
|
||||
| Service | Responsibility |
|
||||
|---------|---------------|
|
||||
| `task-service` | Task CRUD, recurring task spawning, dependency resolution |
|
||||
| `habit-service` | Habit tracking, streak calculation, completion scoring |
|
||||
| `note-service` | Note CRUD, wikilink parsing, word count |
|
||||
| `report-service` | Report generation, template rendering |
|
||||
| `project-service` | Project progress calculation, milestone tracking |
|
||||
| `webhook-service` | Webhook event dispatch, delivery tracking |
|
||||
| `agent-mention-service` | Agent @mention parsing, task dispatch |
|
||||
## Request and data flow
|
||||
|
||||
**Shared package (`@project-e/shared`):**
|
||||
|
||||
Contains Zod schemas and TypeScript types shared between the web app, worker, and potentially other consumers. Every entity has:
|
||||
- A Zod schema for runtime validation
|
||||
- Inferred TypeScript types for compile-time safety
|
||||
- Create and update schema variants (omit server-generated fields)
|
||||
|
||||
### Data Access Layer
|
||||
|
||||
**Location:** `apps/web/lib/pocketbase.ts` and `apps/web/app/api/`
|
||||
|
||||
The data access layer handles communication with PocketBase.
|
||||
|
||||
**PocketBase client (`lib/pocketbase.ts`):**
|
||||
|
||||
Provides two factory functions:
|
||||
- `createPocketBaseClient(token?)`: Creates a client with optional user auth
|
||||
- `createAdminClient()`: Creates a client with admin privileges (uses `POCKETBASE_ADMIN_TOKEN`)
|
||||
|
||||
Also provides generic helpers: `getRecord`, `listRecords`, `createRecord`, `updateRecord`, `deleteRecord`.
|
||||
|
||||
**API routes (`app/api/`):**
|
||||
|
||||
Each entity has a directory under `app/api/` with a `route.ts` file implementing GET (list) and POST (create). Dynamic routes use `[id]/route.ts` for GET (single), PATCH (update), and DELETE.
|
||||
|
||||
All API routes use the `withAuth` middleware to enforce authentication.
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Creating a Task
|
||||
### Create a task
|
||||
|
||||
```
|
||||
User fills form → React component
|
||||
│
|
||||
▼
|
||||
POST /api/tasks (JSON body)
|
||||
│
|
||||
▼
|
||||
withAuth middleware validates session
|
||||
│
|
||||
▼
|
||||
Zod schema validates body (createTaskSchema)
|
||||
│
|
||||
▼
|
||||
PocketBase client creates record
|
||||
│
|
||||
▼
|
||||
PocketBase triggers realtime event
|
||||
│
|
||||
├──▶ SSE /api/realtime → Browser updates UI
|
||||
│
|
||||
└──▶ Worker picks up job (if webhook subscribed)
|
||||
Browser → POST /api/tasks → authenticated API route
|
||||
→ Zod validation → service and collection adapter
|
||||
→ Drizzle write to PostgreSQL
|
||||
→ pg_notify('project_e_events')
|
||||
→ SSE clients receive the update
|
||||
```
|
||||
|
||||
### Realtime Update Flow
|
||||
### Realtime updates
|
||||
|
||||
```
|
||||
User A updates a task
|
||||
│
|
||||
▼
|
||||
PATCH /api/tasks/abc123
|
||||
│
|
||||
▼
|
||||
PocketBase updates SQLite record
|
||||
│
|
||||
▼
|
||||
PocketBase emits realtime event
|
||||
│
|
||||
├──▶ User A's SSE connection receives event → UI updates
|
||||
│
|
||||
├──▶ User B's SSE connection receives event → UI updates
|
||||
│
|
||||
└──▶ Webhook delivery queued (if subscribed)
|
||||
```
|
||||
The `/api/realtime` endpoint authenticates the request, opens a PostgreSQL listener for `project_e_events`, and streams matching collection events through Server-Sent Events. Clients can specify a comma-separated `collections` query parameter. The endpoint sends a keepalive comment every 30 seconds and closes its PostgreSQL listener when the client disconnects.
|
||||
|
||||
### Agent Tool Call Flow
|
||||
The reverse proxy must not buffer SSE responses.
|
||||
|
||||
```
|
||||
AI agent sends MCP request
|
||||
│
|
||||
▼
|
||||
GET/POST /api/mcp (Authorization: Bearer <api_key>)
|
||||
│
|
||||
▼
|
||||
Authenticate: look up agent by API key
|
||||
│
|
||||
▼
|
||||
MCP server routes to tool handler
|
||||
│
|
||||
▼
|
||||
Tool handler calls PocketBase (admin client)
|
||||
│
|
||||
▼
|
||||
Return JSON result to agent
|
||||
│
|
||||
▼
|
||||
Agent activity logged to agent_activity collection
|
||||
```
|
||||
## Drizzle migrations and Docker
|
||||
|
||||
## Realtime Architecture
|
||||
Drizzle Kit reads `drizzle.config.ts`, which points to `packages/db/src/schema.ts` and writes generated SQL to `drizzle/`. `DATABASE_URL` supplies the PostgreSQL connection string.
|
||||
|
||||
Project E uses Server-Sent Events (SSE) to push realtime updates from PocketBase to the browser.
|
||||
Docker Compose runs PostgreSQL 16 and mounts `drizzle/0000_first_mauler.sql` as an initialization script. PostgreSQL runs initialization scripts only for a new data volume. Apply later reviewed migrations during deployment; restarting an existing database container does not run them.
|
||||
|
||||
### Why SSE Instead of WebSockets
|
||||
## Background worker
|
||||
|
||||
1. **Simpler infrastructure**: SSE works over standard HTTP, no special proxy configuration needed
|
||||
2. **Unidirectional**: The server pushes events; the client sends commands through REST. This matches the data flow.
|
||||
3. **Auto-reconnect**: The browser's `EventSource` API handles reconnection automatically
|
||||
4. **PocketBase native**: PocketBase has built-in realtime subscriptions. SSE is a natural proxy.
|
||||
The worker polls `queue_jobs` records for due work. It marks a job `in_progress`, runs the handler, then records `completed` or schedules a retry after an error. Its polling interval starts at five seconds, grows to 60 seconds when no work is available, and resets when it finds jobs.
|
||||
|
||||
### SSE Proxy Architecture
|
||||
Supported job types include webhook delivery, agent mentions, report generation, recurring tasks, and cleanup. Failed jobs retry with exponential backoff up to their configured retry limit.
|
||||
|
||||
PocketBase's realtime uses WebSockets internally. The Next.js app acts as a proxy:
|
||||
## MCP server
|
||||
|
||||
```
|
||||
Browser ──SSE──▶ /api/realtime ──WebSocket──▶ PocketBase
|
||||
```
|
||||
The MCP server runs at `/api/mcp` over Streamable HTTP. It establishes sessions with `GET`, accepts JSON-RPC requests with `POST`, and ends sessions with `DELETE`. The app keeps active transports in memory.
|
||||
|
||||
The proxy:
|
||||
1. Opens a PocketBase WebSocket connection for each subscribed collection
|
||||
2. Translates PocketBase events into SSE `data:` frames
|
||||
3. Sends a `:ping` comment every 30 seconds to keep the connection alive
|
||||
4. Cleans up all subscriptions when the client disconnects
|
||||
|
||||
### Subscription Management
|
||||
|
||||
Clients specify which collections to subscribe to via query parameters:
|
||||
|
||||
```
|
||||
GET /api/realtime?collections=tasks,habits,projects
|
||||
```
|
||||
|
||||
Default subscriptions (if no parameter): `tasks`, `habits`, `projects`, `notes`, `reports`, `milestones`, `notifications`.
|
||||
|
||||
### Event Format
|
||||
|
||||
```
|
||||
data: {"type":"connected","collections":["tasks","habits"]}
|
||||
|
||||
data: {"type":"create","collection":"tasks","record":{"id":"abc","title":"New task"}}
|
||||
|
||||
data: {"type":"update","collection":"tasks","record":{"id":"abc","status":"done"}}
|
||||
|
||||
data: {"type":"delete","collection":"tasks","record":{"id":"abc"}}
|
||||
|
||||
:ping
|
||||
```
|
||||
|
||||
### Client-Side Handling
|
||||
|
||||
The Zustand stores subscribe to SSE events and update local state:
|
||||
|
||||
```typescript
|
||||
const eventSource = new EventSource("/api/realtime?collections=tasks");
|
||||
|
||||
eventSource.onmessage = (event) => {
|
||||
const data = JSON.parse(event.data);
|
||||
if (data.type === "update" && data.collection === "tasks") {
|
||||
useTaskStore.getState().updateTask(data.record);
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## Worker Architecture
|
||||
|
||||
The background worker processes asynchronous jobs stored in the `queue_jobs` PocketBase collection.
|
||||
|
||||
### Job Lifecycle
|
||||
|
||||
```
|
||||
API creates job → status: "pending"
|
||||
│
|
||||
▼
|
||||
Worker polls for pending jobs (every 5s base)
|
||||
│
|
||||
▼
|
||||
Worker marks job as "in_progress"
|
||||
│
|
||||
├──▶ Success → status: "completed"
|
||||
│
|
||||
└──▶ Failure → retry_count < max_retries?
|
||||
│
|
||||
├──▶ Yes → status: "pending", scheduled_at: now + backoff
|
||||
│
|
||||
└──▶ No → status: "failed", error: message
|
||||
```
|
||||
|
||||
### Polling Strategy
|
||||
|
||||
The worker uses exponential backoff to reduce database load when idle:
|
||||
|
||||
| Condition | Poll Interval |
|
||||
|-----------|--------------|
|
||||
| Jobs found | Reset to 5 seconds |
|
||||
| No jobs | Multiply by 1.5 (max 60 seconds) |
|
||||
| Error | Keep current interval |
|
||||
|
||||
### Job Types
|
||||
|
||||
| Type | Handler | Description |
|
||||
|------|---------|-------------|
|
||||
| `webhook_delivery` | `handleWebhookDelivery` | POST payload to webhook URL with HMAC signature |
|
||||
| `agent_mention` | `handleAgentMention` | Dispatch task to agent webhook |
|
||||
| `report_generation` | `handleReportGeneration` | Collect data and populate report content |
|
||||
| `recurring_task` | `handleRecurringTask` | Compute next due date from RRULE and spawn task |
|
||||
| `cleanup` | `handleCleanup` | Purge old webhook deliveries and error logs |
|
||||
|
||||
### Retry Strategy
|
||||
|
||||
Failed jobs retry with exponential backoff:
|
||||
|
||||
```
|
||||
Retry 1: 5 seconds
|
||||
Retry 2: 10 seconds
|
||||
Retry 3: 20 seconds
|
||||
...
|
||||
Max backoff: 5 minutes
|
||||
Max retries: 3 (configurable per job)
|
||||
```
|
||||
|
||||
### Cleanup
|
||||
|
||||
The worker schedules a daily cleanup job that:
|
||||
- Deletes webhook deliveries older than 90 days
|
||||
- Deletes error logs older than 30 days
|
||||
|
||||
## MCP Server Architecture
|
||||
|
||||
The MCP server runs inside the Next.js API at `/api/mcp`. It uses the `@modelcontextprotocol/sdk` package.
|
||||
|
||||
### Transport
|
||||
|
||||
The server uses **Streamable HTTP** transport:
|
||||
|
||||
1. `GET /api/mcp`: Establish a session, returns `mcp-session-id` header
|
||||
2. `POST /api/mcp`: Send JSON-RPC requests with `mcp-session-id` header
|
||||
3. `DELETE /api/mcp`: End the session
|
||||
|
||||
Sessions are stored in an in-memory `Map<string, Transport>`.
|
||||
|
||||
### Authentication
|
||||
|
||||
Every request validates the `Authorization: Bearer <api_key>` header against the `agents` collection. Only active agents can connect.
|
||||
|
||||
### Tool Organization
|
||||
|
||||
Tools are organized by entity in `apps/web/lib/mcp/tools/`:
|
||||
|
||||
```
|
||||
lib/mcp/
|
||||
├── server.ts : Creates McpServer, registers all tools
|
||||
└── tools/
|
||||
├── tasks.ts : 8 tools
|
||||
├── habits.ts : 7 tools
|
||||
├── projects.ts : 6 tools
|
||||
├── notes.ts : 6 tools
|
||||
├── reports.ts : 5 tools
|
||||
├── milestones.ts : 5 tools
|
||||
├── domains.ts : 5 tools
|
||||
├── tags.ts : 5 tools
|
||||
├── agents.ts : 5 tools
|
||||
├── webhooks.ts : 5 tools
|
||||
└── analytics.ts : 4 tools
|
||||
```
|
||||
|
||||
Each file exports a `register*Tools(server)` function that adds tools to the server.
|
||||
|
||||
### Tool Execution
|
||||
|
||||
Tools use the admin PocketBase client (`createAdminClient()`) to bypass row-level security. This is safe because:
|
||||
1. The MCP endpoint requires a valid agent API key
|
||||
2. Agent permission tiers control which tools are available
|
||||
3. All tool calls are logged to `agent_activity`
|
||||
Tools live in `apps/web/lib/mcp/tools/` and register by entity. Tool handlers use the database-backed collection adapter. Agent API keys identify callers, permission tiers restrict access, and the app records agent activity.
|
||||
|
||||
## Security
|
||||
|
||||
### Authentication
|
||||
- PostgreSQL credentials come from `DATABASE_URL` and `POSTGRES_PASSWORD`.
|
||||
- NextAuth signs JWT sessions with `NEXTAUTH_SECRET`.
|
||||
- The app stores password hashes with bcrypt, never plaintext passwords.
|
||||
- Cookies use `httpOnly`, `sameSite: lax`, and `secure` settings appropriate to the deployment.
|
||||
- Zod schemas validate API request bodies.
|
||||
- Reverse proxies should enforce TLS, host restrictions, and rate limits.
|
||||
|
||||
**User authentication** uses PocketBase's built-in auth system:
|
||||
- Email/password login
|
||||
- JWT tokens stored in `httpOnly` cookies
|
||||
- 7-day token expiry with refresh endpoint
|
||||
- Tokens validated on every API request via `withAuth` middleware
|
||||
## Performance and operations
|
||||
|
||||
**Agent authentication** uses API keys:
|
||||
- Each agent has a unique API key (UUID)
|
||||
- Keys validated against the `agents` collection
|
||||
- Disabled agents cannot authenticate
|
||||
- Used for both MCP server and agent webhook callbacks
|
||||
PostgreSQL stores indexed collection and creation-time metadata while retaining flexible collection fields in JSONB. API routes paginate list responses. The Drizzle client uses a connection pool with a maximum of 10 connections.
|
||||
|
||||
### Authorization
|
||||
|
||||
**API routes** use the `withAuth` middleware. All routes require a valid user session. PocketBase row-level security provides additional protection at the database level.
|
||||
|
||||
**MCP tools** use agent permission tiers. The `read_only` tier can only call `get_*` and `list_*` tools. The server enforces this at the tool registration level.
|
||||
|
||||
### Data Protection
|
||||
|
||||
- **Passwords**: Hashed by PocketBase using bcrypt
|
||||
- **API keys**: Stored as plain text (UUIDs). Treat them like passwords.
|
||||
- **Webhook secrets**: Used for HMAC-SHA256 payload signing
|
||||
- **Admin token**: Stored in environment variable, not in the database
|
||||
- **Cookies**: `httpOnly`, `secure` (production), `sameSite: lax`
|
||||
|
||||
### Input Validation
|
||||
|
||||
All API endpoints validate request bodies with Zod schemas from `@project-e/shared`. Invalid input returns a `400 VALIDATION_ERROR` with details about which fields failed.
|
||||
|
||||
### CORS
|
||||
|
||||
Configure CORS in `next.config.ts` for cross-origin requests. By default, Next.js allows same-origin requests only.
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
The application does not enforce rate limiting. Configure it at the reverse proxy or infrastructure layer. See [Deployment Guide](DEPLOYMENT.md#rate-limiting) for recommendations.
|
||||
|
||||
## Performance
|
||||
|
||||
### Database
|
||||
|
||||
PocketBase uses SQLite, which handles read-heavy workloads well.
|
||||
|
||||
**Optimizations:**
|
||||
- PocketBase enables WAL mode by default for concurrent reads
|
||||
- Indexes on frequently queried fields (status, domain, project_id)
|
||||
- Pagination limits prevent large result sets
|
||||
|
||||
**Limits:**
|
||||
- SQLite handles thousands of concurrent reads
|
||||
- Write throughput is limited by single-writer model
|
||||
- For most personal/team use cases, this is not a bottleneck
|
||||
|
||||
### API Responses
|
||||
|
||||
**Caching:**
|
||||
- List endpoints set `Cache-Control: private, max-age=60, stale-while-revalidate=300`
|
||||
- This allows the browser to use stale data while revalidating in the background
|
||||
|
||||
**Compression:**
|
||||
- Next.js compresses responses automatically (gzip/brotli)
|
||||
- The reverse proxy should also enable compression
|
||||
|
||||
### Realtime
|
||||
|
||||
**Connection management:**
|
||||
- Each SSE connection holds a PocketBase WebSocket subscription
|
||||
- The proxy cleans up subscriptions on client disconnect
|
||||
- Keepalive pings prevent idle connection timeouts
|
||||
|
||||
**Scaling limits:**
|
||||
- Each SSE connection uses a server-side WebSocket to PocketBase
|
||||
- For hundreds of concurrent users, consider connection pooling or a dedicated realtime service
|
||||
|
||||
### Frontend
|
||||
|
||||
**Next.js optimizations:**
|
||||
- App Router with React Server Components for initial page loads
|
||||
- Client components only where interactivity is needed
|
||||
- Automatic code splitting per route
|
||||
- Image optimization via `next/image`
|
||||
|
||||
**Bundle size:**
|
||||
- shadcn/ui components are tree-shaken (only imported components are included)
|
||||
- Zustand stores are lightweight (no boilerplate)
|
||||
- Tiptap editor loads lazily
|
||||
Each realtime client holds a PostgreSQL notification listener. Monitor long-lived SSE connections and proxy timeouts as concurrency grows. The worker uses database-backed job records, so multiple workers can process queued work when the job-claiming behavior supports the workload.
|
||||
|
||||
## Accessibility
|
||||
|
||||
Project E follows WCAG 2.1 AA guidelines.
|
||||
|
||||
### Keyboard Navigation
|
||||
|
||||
- All interactive elements are reachable via keyboard
|
||||
- Focus order follows visual layout
|
||||
- Focus indicators are visible (Tailwind `focus-visible:ring-2`)
|
||||
- Modal dialogs trap focus and return it on close
|
||||
|
||||
### ARIA Labels
|
||||
|
||||
- Icon-only buttons include `aria-label` attributes
|
||||
- Form fields have associated `<label>` elements
|
||||
- Dynamic content uses `aria-live` regions
|
||||
- Dialogs use `role="dialog"` and `aria-modal="true"`
|
||||
|
||||
### Color and Contrast
|
||||
|
||||
- Text meets 4.5:1 contrast ratio against backgrounds
|
||||
- Status indicators use both color and text/icons
|
||||
- Domain colors are customizable but default to accessible palette
|
||||
|
||||
### Screen Readers
|
||||
|
||||
- Semantic HTML elements (`<nav>`, `<main>`, `<section>`, `<article>`)
|
||||
- Headings follow proper hierarchy (h1 → h2 → h3)
|
||||
- Tables use `<th>` with `scope` attributes
|
||||
- Toast notifications announce via `aria-live="polite"`
|
||||
|
||||
### Motion
|
||||
|
||||
- Animations respect `prefers-reduced-motion`
|
||||
- No auto-playing content
|
||||
- Transitions are subtle and brief
|
||||
Project E targets WCAG 2.1 AA. Interactive elements support keyboard navigation, icon-only controls include accessible names, form fields use labels, dialogs manage focus, and dynamic content uses live regions. The UI respects `prefers-reduced-motion` and pairs status colors with text or icons.
|
||||
|
||||
+66
-547
@@ -1,404 +1,126 @@
|
||||
# Deployment Guide
|
||||
|
||||
Deploy Project E with Docker Compose and Nginx Proxy Manager. Three containers run the application: web (Next.js), db (PocketBase), and worker (background jobs).
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Quick Deploy](#quick-deploy)
|
||||
- [Environment Configuration](#environment-configuration)
|
||||
- [Nginx Proxy Manager Setup](#nginx-proxy-manager-setup)
|
||||
- [PocketBase Setup](#pocketbase-setup)
|
||||
- [Backups](#backups)
|
||||
- [Monitoring](#monitoring)
|
||||
- [Scaling](#scaling)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
Deploy Project E with Docker Compose and a reverse proxy such as Nginx Proxy Manager. Compose runs three services: the Next.js web app, PostgreSQL 16, and the background worker.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Docker** 24.0 or later
|
||||
- **Docker Compose** 2.20 or later
|
||||
- **Nginx Proxy Manager** installed and running
|
||||
- **At least 1GB RAM** and 10GB disk space
|
||||
- Docker 24.0 or later
|
||||
- Docker Compose 2.20 or later
|
||||
- A reverse proxy for TLS and public routing
|
||||
- At least 1 GB RAM and 10 GB disk space
|
||||
|
||||
## Quick Deploy
|
||||
## Deploy Project E
|
||||
|
||||
1. **Clone the repository on your server**
|
||||
1. Clone the repository on the server.
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd ProjectE
|
||||
```
|
||||
|
||||
2. **Create the environment file**
|
||||
2. Create the environment file.
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Edit `.env` and set the required variables:
|
||||
3. Set the required values in `.env`.
|
||||
|
||||
```bash
|
||||
POCKETBASE_ADMIN_TOKEN=your-secure-random-token
|
||||
PUBLIC_URL=http://project-e.local
|
||||
COOKIE_SECURE=false
|
||||
ALLOWED_HOSTS=project-e.local,localhost
|
||||
POSTGRES_PASSWORD=your_postgres_password
|
||||
DATABASE_URL=postgresql://project_e:your_postgres_password@localhost:5432/project_e
|
||||
NEXTAUTH_SECRET=your_long_random_secret
|
||||
INITIAL_ADMIN_EMAIL=admin@example.com
|
||||
INITIAL_ADMIN_PASSWORD=your_initial_admin_password
|
||||
PUBLIC_URL=https://project-e.example.com
|
||||
COOKIE_SECURE=true
|
||||
ALLOWED_HOSTS=project-e.example.com
|
||||
```
|
||||
|
||||
Generate a secure token:
|
||||
Generate secrets with `openssl rand -base64 32`. Use the same `POSTGRES_PASSWORD` in `DATABASE_URL`. The web and worker containers use an internal database URL that Compose builds from `POSTGRES_PASSWORD`.
|
||||
|
||||
```bash
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
3. **Build and start containers**
|
||||
4. Build and start the services.
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
4. **Verify all containers are running**
|
||||
5. Confirm that the services are healthy.
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
You should see three containers with status `Up (healthy)`:
|
||||
- `project-e-web` (internal only, no exposed ports)
|
||||
- `project-e-db` (internal only, no exposed ports)
|
||||
- `project-e-worker` running in the background
|
||||
6. Route your domain to `project-e-web:3000` through your reverse proxy.
|
||||
|
||||
5. **Configure Nginx Proxy Manager** (see below)
|
||||
7. Open the app and sign in with `INITIAL_ADMIN_EMAIL` and `INITIAL_ADMIN_PASSWORD`. Project E creates that account only when the database contains no users.
|
||||
|
||||
6. **Create your admin account**
|
||||
## Environment variables
|
||||
|
||||
Once NPM is configured, access Project E through your domain. Create your first user account.
|
||||
| Variable | Required | Description |
|
||||
|----------|----------|-------------|
|
||||
| `DATABASE_URL` | Yes | PostgreSQL connection string for local tools and processes outside Compose. |
|
||||
| `POSTGRES_PASSWORD` | Yes | Password for the `project_e` PostgreSQL user. |
|
||||
| `NEXTAUTH_SECRET` | Yes | Secret used to sign NextAuth JWT sessions. |
|
||||
| `INITIAL_ADMIN_EMAIL` | Yes | Email address for the first account. |
|
||||
| `INITIAL_ADMIN_PASSWORD` | Yes | Password for the first account. |
|
||||
| `PUBLIC_URL` | No | Public application URL. Defaults to `http://localhost:3000`. |
|
||||
| `COOKIE_SECURE` | No | Set to `true` behind HTTPS. Defaults to `false`. |
|
||||
| `ALLOWED_HOSTS` | No | Comma-separated allowed hostnames. |
|
||||
|
||||
## Environment Configuration
|
||||
Keep `.env` out of version control. Rotate `NEXTAUTH_SECRET` only when you intend to end active sessions.
|
||||
|
||||
### Required Variables
|
||||
## PostgreSQL and Drizzle
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `POCKETBASE_ADMIN_TOKEN` | Admin token from PocketBase. Required for the worker and server-side API operations. |
|
||||
The `db` service runs `postgres:16-alpine`, stores its data in the `project-e-pg-data` Docker volume, and exposes port 5432. The web and worker services connect to `db:5432` over the Compose network.
|
||||
|
||||
### Optional Variables
|
||||
Docker mounts `drizzle/0000_first_mauler.sql` into PostgreSQL's initialization directory. PostgreSQL runs that file only while it initializes an empty data volume. For a later Drizzle migration, apply the reviewed SQL as part of your release process. Do not expect a container restart to apply a new migration to an existing volume.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `POCKETBASE_URL` | `http://db:8090` | PocketBase URL (internal Docker network) |
|
||||
| `NODE_ENV` | `production` | Node environment |
|
||||
| `PUBLIC_URL` | `http://localhost:3000` | Public URL where users access Project E (used for generating absolute URLs) |
|
||||
| `COOKIE_SECURE` | `false` | Set to `true` if using HTTPS through NPM, `false` for HTTP-only LAN access |
|
||||
| `ALLOWED_HOSTS` | `localhost` | Comma-separated list of domains that can access the app |
|
||||
|
||||
### Setting Variables
|
||||
|
||||
Create a `.env` file in the project root:
|
||||
To apply a migration from the host, run:
|
||||
|
||||
```bash
|
||||
POCKETBASE_ADMIN_TOKEN=abc123def456...
|
||||
POCKETBASE_URL=http://db:8090
|
||||
PUBLIC_URL=http://project-e.local
|
||||
COOKIE_SECURE=false
|
||||
ALLOWED_HOSTS=project-e.local,localhost
|
||||
docker compose exec -T db psql -U project_e -d project_e < drizzle/<migration>.sql
|
||||
```
|
||||
|
||||
Docker Compose reads this file automatically.
|
||||
Back up the database before applying schema changes.
|
||||
|
||||
## Nginx Proxy Manager Setup
|
||||
## Nginx Proxy Manager
|
||||
|
||||
### Add Proxy Host
|
||||
|
||||
1. Open Nginx Proxy Manager admin interface
|
||||
2. Go to **Hosts** → **Proxy Hosts** → **Add Proxy Host**
|
||||
|
||||
3. Configure the following:
|
||||
|
||||
**Details Tab:**
|
||||
- **Domain Names:** `project-e.local` (or your chosen domain)
|
||||
- **Scheme:** `http`
|
||||
- **Forward Hostname/IP:** `project-e-web` (the Docker container name)
|
||||
- **Forward Port:** `3000`
|
||||
|
||||
**SSL Tab:**
|
||||
- If using HTTPS: Select your SSL certificate
|
||||
- If HTTP-only on LAN: Leave SSL disabled
|
||||
|
||||
**Advanced Tab:**
|
||||
Add these custom Nginx configuration lines:
|
||||
1. In Nginx Proxy Manager, open **Hosts** > **Proxy Hosts** > **Add Proxy Host**.
|
||||
2. Set the domain name, choose `http`, set the forward hostname to `project-e-web`, and set the forward port to `3000`.
|
||||
3. Select a certificate and force SSL for HTTPS deployments.
|
||||
4. Add the following advanced configuration to support Server-Sent Events:
|
||||
|
||||
```nginx
|
||||
# WebSocket support for realtime features
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
|
||||
# Forward real client information
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Host $host;
|
||||
|
||||
# SSE support for realtime endpoint
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
proxy_read_timeout 300s;
|
||||
|
||||
# Increase timeout for long-running requests
|
||||
proxy_connect_timeout 300s;
|
||||
proxy_send_timeout 300s;
|
||||
```
|
||||
|
||||
4. Click **Save**
|
||||
5. Save the host and open the configured domain. You should reach the Project E sign-in page.
|
||||
|
||||
### Testing the Connection
|
||||
## Back up and restore PostgreSQL
|
||||
|
||||
1. From a device on your LAN, open a browser
|
||||
2. Navigate to `http://project-e.local` (or your configured domain)
|
||||
3. You should see the Project E login page
|
||||
|
||||
### Optional: PocketBase Admin Access
|
||||
|
||||
If you need direct access to the PocketBase admin UI:
|
||||
|
||||
1. In NPM, add another Proxy Host:
|
||||
- **Domain Names:** `pb.project-e.local`
|
||||
- **Scheme:** `http`
|
||||
- **Forward Hostname/IP:** `project-e-db`
|
||||
- **Forward Port:** `8090`
|
||||
|
||||
2. Or temporarily expose the port in `docker-compose.yml`:
|
||||
```yaml
|
||||
db:
|
||||
ports:
|
||||
- "8090:8090"
|
||||
```
|
||||
|
||||
## PocketBase Setup
|
||||
|
||||
### Initial Configuration
|
||||
|
||||
PocketBase runs as a standalone container. After starting it for the first time:
|
||||
|
||||
1. Access the admin UI at `http://your-server:8090/_/`
|
||||
2. Create your admin account
|
||||
3. Configure authentication settings under **Settings > Auth**
|
||||
4. Enable the auth methods you need (email/password is enabled by default)
|
||||
|
||||
### Migrations
|
||||
|
||||
Database migrations are in `pocketbase/pb_migrations/`. They run automatically when the PocketBase container starts.
|
||||
|
||||
To add a new migration:
|
||||
|
||||
1. Create a file in `pocketbase/pb_migrations/` with the naming convention `YYYYMMDDHHMMSS_description.js`
|
||||
2. Restart the PocketBase container: `docker compose restart db`
|
||||
|
||||
### Data Directory
|
||||
|
||||
PocketBase stores all data (SQLite database, uploads, logs) in the `/pb_data` volume. This volume persists across container restarts.
|
||||
|
||||
## Reverse Proxy
|
||||
|
||||
Put a reverse proxy in front of the application to handle SSL, compression, and routing.
|
||||
|
||||
### Caddy (Recommended)
|
||||
|
||||
Caddy handles SSL automatically.
|
||||
|
||||
Create a `Caddyfile`:
|
||||
|
||||
```
|
||||
your-domain.com {
|
||||
reverse_proxy localhost:3000
|
||||
|
||||
header {
|
||||
Strict-Transport-Security "max-age=31536000; includeSubDomains"
|
||||
X-Frame-Options "DENY"
|
||||
X-Content-Type-Options "nosniff"
|
||||
}
|
||||
}
|
||||
|
||||
pb.your-domain.com {
|
||||
reverse_proxy localhost:8090
|
||||
}
|
||||
```
|
||||
|
||||
Install and run Caddy:
|
||||
Create a logical backup without stopping the database:
|
||||
|
||||
```bash
|
||||
# Install Caddy
|
||||
sudo apt install caddy # Debian/Ubuntu
|
||||
# or
|
||||
brew install caddy # macOS
|
||||
|
||||
# Start Caddy
|
||||
caddy start
|
||||
mkdir -p backups
|
||||
```
|
||||
|
||||
### Traefik
|
||||
|
||||
Create a `traefik.yml`:
|
||||
|
||||
```yaml
|
||||
entryPoints:
|
||||
web:
|
||||
address: ":80"
|
||||
http:
|
||||
redirections:
|
||||
entryPoint:
|
||||
to: websecure
|
||||
scheme: https
|
||||
websecure:
|
||||
address: ":443"
|
||||
|
||||
certificatesResolvers:
|
||||
letsencrypt:
|
||||
acme:
|
||||
email: your-email@example.com
|
||||
storage: acme.json
|
||||
httpChallenge:
|
||||
entryPoint: web
|
||||
|
||||
providers:
|
||||
docker:
|
||||
exposedByDefault: false
|
||||
```
|
||||
|
||||
Update `docker-compose.yml` to add Traefik labels:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
web:
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.web.rule=Host(`your-domain.com`)"
|
||||
- "traefik.http.routers.web.entrypoints=websecure"
|
||||
- "traefik.http.routers.web.tls.certresolver=letsencrypt"
|
||||
```
|
||||
|
||||
### Nginx
|
||||
|
||||
Create `/etc/nginx/sites-available/project-e`:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
server_name your-domain.com;
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name your-domain.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;
|
||||
|
||||
location / {
|
||||
proxy_pass http://localhost:3000;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection 'upgrade';
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_cache_bypass $http_upgrade;
|
||||
|
||||
# SSE support for realtime endpoint
|
||||
proxy_buffering off;
|
||||
proxy_read_timeout 300s;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Enable the site and reload:
|
||||
Restore a backup into a new or empty database:
|
||||
|
||||
```bash
|
||||
sudo ln -s /etc/nginx/sites-available/project-e /etc/nginx/sites-enabled/
|
||||
sudo nginx -t
|
||||
sudo systemctl reload nginx
|
||||
docker compose exec -T db psql -U project_e -d project_e < backups/project-e-YYYYMMDD_HHMMSS.sql
|
||||
```
|
||||
|
||||
## SSL/TLS
|
||||
|
||||
### Let's Encrypt with Certbot
|
||||
|
||||
```bash
|
||||
# Install certbot
|
||||
sudo apt install certbot python3-certbot-nginx
|
||||
|
||||
# Get certificate
|
||||
sudo certbot --nginx -d your-domain.com -d pb.your-domain.com
|
||||
|
||||
# Auto-renewal is configured automatically
|
||||
```
|
||||
|
||||
### Caddy
|
||||
|
||||
Caddy obtains and renews certificates automatically. No configuration needed beyond the domain name in the `Caddyfile`.
|
||||
|
||||
### Internal Communication
|
||||
|
||||
The web and worker containers connect to PocketBase over the internal Docker network (`http://db:8090`). This traffic does not need SSL.
|
||||
|
||||
Only expose ports 3000 and 8090 to the reverse proxy, not directly to the internet.
|
||||
|
||||
## Backups
|
||||
|
||||
### PocketBase Database
|
||||
|
||||
The database is a single SQLite file at `/pb_data/data.db`.
|
||||
|
||||
**Manual backup:**
|
||||
|
||||
```bash
|
||||
# Stop the container to ensure consistency
|
||||
docker compose stop db
|
||||
|
||||
# Copy the database file
|
||||
docker cp project-e-db:/pb_data/data.db ./backups/data-$(date +%Y%m%d).db
|
||||
|
||||
# Restart the container
|
||||
docker compose start db
|
||||
```
|
||||
|
||||
**Automated backup script:**
|
||||
|
||||
Create `scripts/backup.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
BACKUP_DIR="/path/to/backups"
|
||||
DATE=$(date +%Y%m%d_%H%M%S)
|
||||
|
||||
mkdir -p $BACKUP_DIR
|
||||
|
||||
# Use PocketBase's backup API (no downtime)
|
||||
curl -X POST http://localhost:8090/api/backup \
|
||||
-H "Authorization: Admin your-admin-token" \
|
||||
-o "$BACKUP_DIR/backup-$DATE.zip"
|
||||
|
||||
# Keep only last 30 backups
|
||||
find $BACKUP_DIR -name "backup-*.zip" -mtime +30 -delete
|
||||
|
||||
echo "Backup completed: backup-$DATE.zip"
|
||||
```
|
||||
|
||||
Schedule with cron:
|
||||
|
||||
```bash
|
||||
# Run daily at 2 AM
|
||||
0 2 * * * /path/to/scripts/backup.sh
|
||||
```
|
||||
|
||||
### Upload Files
|
||||
|
||||
Uploaded files are stored in the `project-e-web-uploads` volume.
|
||||
|
||||
**Backup uploads:**
|
||||
Back up the `project-e-web-uploads` volume if your deployment stores uploads there:
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
@@ -408,232 +130,29 @@ docker run --rm \
|
||||
tar czf /backup/uploads-$(date +%Y%m%d).tar.gz -C /source .
|
||||
```
|
||||
|
||||
### Restore
|
||||
Test restores on a separate environment before relying on a backup.
|
||||
|
||||
**Restore database:**
|
||||
## Monitoring and logs
|
||||
|
||||
```bash
|
||||
# Stop containers
|
||||
docker compose stop db
|
||||
|
||||
# Remove old data
|
||||
docker volume rm project-e-pb-data
|
||||
|
||||
# Copy backup into new volume
|
||||
docker volume create project-e-pb-data
|
||||
docker run --rm \
|
||||
-v project-e-pb-data:/pb_data \
|
||||
-v $(pwd)/backups:/backup \
|
||||
alpine \
|
||||
sh -c "cp /backup/data-YYYYMMDD.db /pb_data/data.db"
|
||||
|
||||
# Restart
|
||||
docker compose start db
|
||||
```
|
||||
|
||||
**Restore uploads:**
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
-v project-e-web-uploads:/target \
|
||||
-v $(pwd)/backups:/backup \
|
||||
alpine \
|
||||
sh -c "cd /target && tar xzf /backup/uploads-YYYYMMDD.tar.gz"
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Container Health
|
||||
|
||||
All containers include health checks. Check status:
|
||||
Check service status and logs:
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
### Application Health
|
||||
|
||||
The web container exposes a health endpoint:
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000/api/health
|
||||
```
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"timestamp": "2024-01-15T10:30:00.000Z",
|
||||
"version": "0.1.0"
|
||||
}
|
||||
```
|
||||
|
||||
### Logs
|
||||
|
||||
View logs from all containers:
|
||||
|
||||
```bash
|
||||
# All containers
|
||||
docker compose logs -f
|
||||
|
||||
# Specific container
|
||||
docker compose logs -f web
|
||||
docker compose logs -f db
|
||||
docker compose logs -f worker
|
||||
```
|
||||
|
||||
### Resource Usage
|
||||
|
||||
Monitor container resource usage:
|
||||
|
||||
```bash
|
||||
docker stats
|
||||
```
|
||||
|
||||
### Uptime Monitoring
|
||||
|
||||
Use an external service to monitor your deployment:
|
||||
|
||||
- **UptimeRobot** (free tier): HTTP monitoring with email/SMS alerts
|
||||
- **Healthchecks.io**: Cron job monitoring
|
||||
- **Better Stack**: Status pages and incident management
|
||||
|
||||
Set up a check for `https://your-domain.com/api/health` with a 60-second interval.
|
||||
|
||||
## Scaling
|
||||
|
||||
### Vertical Scaling
|
||||
|
||||
The application runs as a single instance of each container. To handle more load:
|
||||
|
||||
1. **Increase server resources**: Add more CPU and RAM to your host
|
||||
2. **Increase Node.js memory**: Set `NODE_OPTIONS=--max-old-space-size=4096` in the web container
|
||||
3. **Increase PocketBase limits**: PocketBase handles thousands of concurrent connections on modest hardware
|
||||
|
||||
### Horizontal Scaling
|
||||
|
||||
Horizontal scaling requires changes to the architecture:
|
||||
|
||||
**Current limitations:**
|
||||
- MCP sessions are stored in memory (not shared between instances)
|
||||
- SSE connections are tied to a specific container
|
||||
- File uploads go to a local volume
|
||||
|
||||
**To scale horizontally:**
|
||||
1. Use a shared session store (Redis)
|
||||
2. Use a load balancer with sticky sessions for SSE
|
||||
3. Use object storage (S3) for file uploads
|
||||
4. Run multiple web containers behind a load balancer
|
||||
|
||||
For most personal and small-team use cases, a single instance handles the load. PocketBase with SQLite performs well up to hundreds of concurrent users.
|
||||
|
||||
### Worker Scaling
|
||||
|
||||
The worker uses polling with exponential backoff. For high-throughput job processing:
|
||||
|
||||
1. Run multiple worker containers (they coordinate through the database)
|
||||
2. Reduce the base poll interval (currently 5 seconds)
|
||||
3. Use a dedicated job queue (Bull, BullMQ) instead of database polling
|
||||
The web service exposes `GET /api/health` on port 3000. Configure uptime monitoring for `https://your-domain.example/api/health`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Container Won't Start
|
||||
| Problem | Fix |
|
||||
|---------|-----|
|
||||
| Database connection fails | Confirm that `db` is healthy and that `POSTGRES_PASSWORD` matches the value in `DATABASE_URL`. |
|
||||
| The web service will not start | Set `NEXTAUTH_SECRET`, `INITIAL_ADMIN_EMAIL`, and `INITIAL_ADMIN_PASSWORD` in `.env`, then restart the service. |
|
||||
| Sign-in fails | Confirm the email and password match the initial-admin values. Those values create an account only before any user exists. |
|
||||
| A schema change is missing | Apply the generated Drizzle SQL. PostgreSQL initialization scripts do not rerun for an existing volume. |
|
||||
| Realtime updates stop | Disable proxy buffering and set a read timeout of at least 300 seconds. |
|
||||
| Disk space runs low | Inspect `project-e-pg-data` and `project-e-web-uploads`, back up data, and expand the host disk. |
|
||||
|
||||
**Check logs:**
|
||||
## Scaling
|
||||
|
||||
```bash
|
||||
docker compose logs web
|
||||
docker compose logs db
|
||||
docker compose logs worker
|
||||
```
|
||||
|
||||
**Common issues:**
|
||||
|
||||
| Problem | Solution |
|
||||
|---------|----------|
|
||||
| `Cannot connect to PocketBase` | Ensure the `db` container is healthy. Check `docker compose ps`. |
|
||||
| `Port 3000 already in use` | Change the port mapping in `docker-compose.yml` |
|
||||
| `POCKETBASE_ADMIN_TOKEN not set` | Set the token in `.env` and restart |
|
||||
| `Migration failed` | Check migration files in `pocketbase/pb_migrations/` |
|
||||
|
||||
### Web App Returns 500 Errors
|
||||
|
||||
1. Check the web container logs: `docker compose logs web`
|
||||
2. Verify PocketBase is running: `docker compose exec db wget -qO- http://localhost:8090/api/health`
|
||||
3. Verify the admin token is correct: `docker compose exec web env | grep POCKETBASE`
|
||||
|
||||
### Realtime Events Not Arriving
|
||||
|
||||
1. Check the SSE connection: `curl -N http://localhost:3000/api/realtime`
|
||||
2. Verify the reverse proxy is not buffering SSE responses
|
||||
3. For Nginx, ensure `proxy_buffering off` is set
|
||||
4. Check browser console for connection errors
|
||||
|
||||
### Worker Not Processing Jobs
|
||||
|
||||
1. Check worker logs: `docker compose logs worker`
|
||||
2. Verify the admin token is set correctly
|
||||
3. Check for pending jobs in PocketBase admin: `http://localhost:8090/_/#/collections/queue_jobs`
|
||||
4. Jobs retry automatically with exponential backoff (max 5 minutes between retries)
|
||||
|
||||
### Database Corruption
|
||||
|
||||
If the SQLite database becomes corrupted:
|
||||
|
||||
1. Stop all containers: `docker compose down`
|
||||
2. Restore from the most recent backup (see [Backups](#backups))
|
||||
3. If no backup exists, try SQLite's recovery:
|
||||
|
||||
```bash
|
||||
sqlite3 data.db ".recover" | sqlite3 new-data.db
|
||||
```
|
||||
|
||||
4. Replace the corrupted file and restart
|
||||
|
||||
### Out of Disk Space
|
||||
|
||||
PocketBase stores the database and uploads in the `project-e-pb-data` volume.
|
||||
|
||||
**Check disk usage:**
|
||||
|
||||
```bash
|
||||
docker system df
|
||||
docker volume inspect project-e-pb-data
|
||||
```
|
||||
|
||||
**Clean up:**
|
||||
|
||||
```bash
|
||||
# Remove unused images
|
||||
docker image prune -a
|
||||
|
||||
# Remove unused volumes (WARNING: deletes all data)
|
||||
docker volume prune
|
||||
```
|
||||
|
||||
**Expand volume:**
|
||||
|
||||
Docker volumes use the host filesystem. If the host disk is full, expand it or move the volume to a larger disk.
|
||||
|
||||
### SSL Certificate Errors
|
||||
|
||||
**Caddy:** Check the Caddy logs for ACME errors. Ensure port 80 is accessible from the internet for the HTTP challenge.
|
||||
|
||||
**Let's Encrypt:** Certificates renew automatically. Force renewal:
|
||||
|
||||
```bash
|
||||
sudo certbot renew --force-renewal
|
||||
```
|
||||
|
||||
**Common errors:**
|
||||
- `Connection refused`: Port 80 is blocked by firewall
|
||||
- `DNS problem`: Domain does not resolve to this server
|
||||
- `Rate limit exceeded`: Too many certificate requests. Wait and retry.
|
||||
|
||||
### Performance Issues
|
||||
|
||||
1. **Slow page loads**: Check if the server has enough RAM. Node.js needs at least 512MB.
|
||||
2. **Slow API responses**: Check PocketBase query performance in the admin panel
|
||||
3. **SSE disconnects**: Ensure the reverse proxy has appropriate timeout settings (300s+)
|
||||
4. **Worker falling behind**: Increase the poll frequency or add more worker instances
|
||||
The default deployment runs one web service, one worker, and one PostgreSQL instance. Add CPU and memory before changing the topology. Multiple web services require shared session-aware infrastructure and a reverse proxy that supports long-lived SSE connections. Multiple workers coordinate through the PostgreSQL-backed job records.
|
||||
|
||||
+86
-669
@@ -1,729 +1,146 @@
|
||||
# Development Guide
|
||||
|
||||
This guide covers the development workflow for Project E. Read this before contributing code.
|
||||
Use this guide to run Project E locally, change the database schema, and prepare a pull request.
|
||||
|
||||
## Table of Contents
|
||||
## Prerequisites
|
||||
|
||||
- [Environment Setup](#environment-setup)
|
||||
- [Project Structure](#project-structure)
|
||||
- [Code Organization](#code-organization)
|
||||
- [Adding a New Feature](#adding-a-new-feature)
|
||||
- [Database Schema Changes](#database-schema-changes)
|
||||
- [Testing Strategy](#testing-strategy)
|
||||
- [Code Style and Conventions](#code-style-and-conventions)
|
||||
- [Git Workflow](#git-workflow)
|
||||
- [PR Review Process](#pr-review-process)
|
||||
- [Common Tasks](#common-tasks)
|
||||
- Node.js 22.13.0 or later
|
||||
- npm 10.0.0 or later
|
||||
- Docker and Docker Compose, for PostgreSQL 16
|
||||
- Git
|
||||
|
||||
## Environment Setup
|
||||
## Set up your local environment
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **Node.js** 22.13.0 or later (use `nvm` to manage versions)
|
||||
- **npm** 10.0.0 or later
|
||||
- **Git**
|
||||
- **A code editor** (VS Code recommended)
|
||||
- **PocketBase** binary (download from [pocketbase.io](https://pocketbase.io))
|
||||
|
||||
### Initial Setup
|
||||
|
||||
1. **Clone the repository**
|
||||
1. Clone the repository and install dependencies.
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd ProjectE
|
||||
```
|
||||
|
||||
2. **Install dependencies**
|
||||
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
3. **Start PocketBase**
|
||||
|
||||
In a separate terminal:
|
||||
2. Copy the environment template.
|
||||
|
||||
```bash
|
||||
pocketbase serve \
|
||||
--dir=./pb_data \
|
||||
--publicDir=./pb_public \
|
||||
--migrationDir=./pocketbase/pb_migrations
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Or use Docker:
|
||||
3. Set the database and authentication values in `.env`.
|
||||
|
||||
```bash
|
||||
DATABASE_URL=postgresql://project_e:your_postgres_password@localhost:5432/project_e
|
||||
POSTGRES_PASSWORD=your_postgres_password
|
||||
NEXTAUTH_SECRET=your_long_random_secret
|
||||
INITIAL_ADMIN_EMAIL=admin@example.com
|
||||
INITIAL_ADMIN_PASSWORD=your_initial_admin_password
|
||||
```
|
||||
|
||||
`DATABASE_URL` connects local processes to PostgreSQL. `POSTGRES_PASSWORD` must match the password in that URL. Generate `NEXTAUTH_SECRET` with `openssl rand -base64 32`.
|
||||
|
||||
4. Start PostgreSQL 16.
|
||||
|
||||
```bash
|
||||
docker compose up db -d
|
||||
```
|
||||
|
||||
4. **Set environment variables**
|
||||
Docker initializes the `project_e` database and applies `drizzle/0000_first_mauler.sql` when it creates an empty database volume.
|
||||
|
||||
Create `apps/web/.env.local`:
|
||||
|
||||
```bash
|
||||
POCKETBASE_URL=http://localhost:8090
|
||||
POCKETBASE_ADMIN_TOKEN=your_admin_token
|
||||
```
|
||||
|
||||
Get the admin token from PocketBase after creating your first admin account.
|
||||
|
||||
5. **Start the development server**
|
||||
5. Start the app.
|
||||
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
|
||||
This starts the Next.js app at `http://localhost:3000` with Turbopack.
|
||||
6. Open `http://localhost:3000` and sign in with `INITIAL_ADMIN_EMAIL` and `INITIAL_ADMIN_PASSWORD`.
|
||||
|
||||
6. **Verify everything works**
|
||||
The credentials create the first admin account only when the `users` table has no accounts.
|
||||
|
||||
- Open `http://localhost:3000` in your browser
|
||||
- Open `http://localhost:8090/_/` for the PocketBase admin UI
|
||||
- Run `npm run typecheck` to verify TypeScript compiles
|
||||
|
||||
### VS Code Setup
|
||||
|
||||
Recommended extensions:
|
||||
- ESLint
|
||||
- Tailwind CSS IntelliSense
|
||||
- TypeScript and JavaScript Language Features (built-in)
|
||||
- Prettier - Code formatter
|
||||
|
||||
Create `.vscode/settings.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"editor.defaultFormatter": "esbenp.prettier-vscode",
|
||||
"editor.formatOnSave": true,
|
||||
"typescript.preferences.importModuleSpecifier": "relative",
|
||||
"tailwindCSS.experimental.classRegex": [
|
||||
["cva\\(([^)]*)\\)", "[\"'`]([^\"'`]*).*?[\"'`]"]
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
## Project structure
|
||||
|
||||
```
|
||||
project-e/
|
||||
├── apps/
|
||||
│ └── web/ # Next.js application (monorepo app)
|
||||
│ ├── app/ # App Router (pages + API routes)
|
||||
│ │ ├── (auth)/ # Auth route group (login, signup)
|
||||
│ │ ├── (dashboard)/ # Dashboard route group
|
||||
│ │ └── api/ # REST API endpoints
|
||||
│ ├── components/ # React components
|
||||
│ │ ├── ui/ # shadcn/ui primitives
|
||||
│ │ └── ... # Feature components
|
||||
│ ├── hooks/ # Custom React hooks
|
||||
│ ├── lib/ # Core utilities
|
||||
│ │ ├── mcp/ # MCP server and tools
|
||||
│ │ ├── services/ # Business logic
|
||||
│ │ ├── stores/ # Zustand stores
|
||||
│ │ ├── events/ # Event bus
|
||||
│ │ ├── auth.ts # Auth middleware
|
||||
│ │ ├── pocketbase.ts # PocketBase client
|
||||
│ │ └── errors.ts # Error handling
|
||||
│ └── types/ # TypeScript type definitions
|
||||
├── apps/web/ # Next.js application
|
||||
│ ├── app/ # App Router pages and API routes
|
||||
│ ├── components/ # React components
|
||||
│ ├── hooks/ # Custom React hooks
|
||||
│ └── lib/ # Services, database adapter, and NextAuth config
|
||||
├── packages/
|
||||
│ └── shared/ # Shared package (@project-e/shared)
|
||||
│ └── src/
|
||||
│ ├── schemas/ # Zod validation schemas
|
||||
│ ├── types/ # Shared TypeScript types
|
||||
│ └── constants/ # Shared constants
|
||||
├── pocketbase/
|
||||
│ ├── pb_migrations/ # Database migrations
|
||||
│ └── schema.ts # TypeScript types for collections
|
||||
├── worker/ # Background job worker
|
||||
│ └── index.ts # Worker entry point
|
||||
├── e2e/ # Playwright E2E tests
|
||||
├── tests/ # Unit and component tests
|
||||
└── docker-compose.yml # Docker Compose configuration
|
||||
│ ├── db/ # Drizzle schema and PostgreSQL client
|
||||
│ └── shared/ # Shared schemas, types, and constants
|
||||
├── drizzle/ # Generated PostgreSQL migrations
|
||||
├── worker/ # Background job worker
|
||||
├── e2e/ # Playwright tests
|
||||
├── tests/ # Unit and component tests
|
||||
├── drizzle.config.ts # Drizzle Kit configuration
|
||||
└── docker-compose.yml # Web, PostgreSQL, and worker services
|
||||
```
|
||||
|
||||
## Code Organization
|
||||
## Data access and authentication
|
||||
|
||||
### Layers
|
||||
The app uses Drizzle ORM with PostgreSQL. `packages/db/src/schema.ts` defines the schema, and `packages/db/src/index.ts` creates the database client from `DATABASE_URL`.
|
||||
|
||||
The application follows a three-layer architecture:
|
||||
Application records live in the `records` table as JSONB data grouped by collection. The `users` table stores email addresses and bcrypt password hashes. API routes use the database adapter in `apps/web/lib/database.ts` for collection operations.
|
||||
|
||||
1. **Presentation**: React components in `apps/web/components/` and pages in `apps/web/app/`
|
||||
2. **Business Logic**: Services in `apps/web/lib/services/` and shared schemas in `packages/shared/`
|
||||
3. **Data Access**: PocketBase client in `apps/web/lib/pocketbase.ts` and API routes in `apps/web/app/api/`
|
||||
NextAuth uses the credentials provider. It creates JWT sessions after a user signs in with an email address and password. Keep `NEXTAUTH_SECRET` stable for an environment; changing it invalidates existing sessions.
|
||||
|
||||
### Naming Conventions
|
||||
## Add a feature
|
||||
|
||||
| Type | Convention | Example |
|
||||
|------|-----------|---------|
|
||||
| Components | PascalCase | `TaskCard.tsx` |
|
||||
| Hooks | camelCase with `use` prefix | `use-task-filter.ts` |
|
||||
| Utilities | camelCase | `format-date.ts` |
|
||||
| Types | PascalCase | `Task.ts` |
|
||||
| Schemas | camelCase with `Schema` suffix | `taskSchema` |
|
||||
| API routes | kebab-case directory | `api/habit-logs/route.ts` |
|
||||
| Stores | camelCase with `use` prefix | `use-dashboard-store.ts` |
|
||||
|
||||
### Import Paths
|
||||
|
||||
Use the `@/` alias for imports within `apps/web`:
|
||||
|
||||
```typescript
|
||||
import { createPocketBaseClient } from '@/lib/pocketbase';
|
||||
import { TaskCard } from '@/components/task-card';
|
||||
```
|
||||
|
||||
Use the package name for shared imports:
|
||||
|
||||
```typescript
|
||||
import { createTaskSchema } from '@project-e/shared';
|
||||
```
|
||||
|
||||
## Adding a New Feature
|
||||
|
||||
Follow these steps to add a feature end-to-end. This example adds a "bookmarks" feature to notes.
|
||||
|
||||
### Step 1: Define the Schema
|
||||
|
||||
Add the field to the PocketBase collection schema. Create a migration file:
|
||||
|
||||
```bash
|
||||
# pocketbase/pb_migrations/20240115120000_add_bookmarks.js
|
||||
export default {
|
||||
up(db) {
|
||||
const collection = db.findCollectionByNameOrId("notes");
|
||||
collection.fields.add(new Field({
|
||||
name: "bookmarked",
|
||||
type: "bool",
|
||||
options: { default: false }
|
||||
}));
|
||||
return db.saveCollection(collection);
|
||||
},
|
||||
down(db) {
|
||||
const collection = db.findCollectionByNameOrId("notes");
|
||||
collection.fields.removeByName("bookmarked");
|
||||
return db.saveCollection(collection);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Update TypeScript Types
|
||||
|
||||
Update the type definition in `pocketbase/schema.ts`:
|
||||
|
||||
```typescript
|
||||
export interface Note extends BaseRecord {
|
||||
// ... existing fields
|
||||
bookmarked: boolean;
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Add Validation Schema
|
||||
|
||||
Update the Zod schema in `packages/shared/src/schemas/note.ts`:
|
||||
|
||||
```typescript
|
||||
export const noteSchema = z.object({
|
||||
// ... existing fields
|
||||
bookmarked: z.boolean().default(false),
|
||||
});
|
||||
```
|
||||
|
||||
### Step 4: Update the API
|
||||
|
||||
If the API route needs changes, update it in `apps/web/app/api/notes/route.ts`. Most CRUD operations work automatically through PocketBase, so you may not need API changes.
|
||||
|
||||
### Step 5: Build the UI
|
||||
|
||||
Create or update components:
|
||||
|
||||
```tsx
|
||||
// apps/web/components/note-bookmark-button.tsx
|
||||
"use client";
|
||||
|
||||
import { Bookmark } from "lucide-react";
|
||||
import { Button } from "@/components/ui/button";
|
||||
|
||||
interface NoteBookmarkButtonProps {
|
||||
noteId: string;
|
||||
bookmarked: boolean;
|
||||
onToggle: (noteId: string) => void;
|
||||
}
|
||||
|
||||
export function NoteBookmarkButton({ noteId, bookmarked, onToggle }: NoteBookmarkButtonProps) {
|
||||
return (
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
onClick={() => onToggle(noteId)}
|
||||
aria-label={bookmarked ? "Remove bookmark" : "Add bookmark"}
|
||||
>
|
||||
<Bookmark className={bookmarked ? "fill-current" : ""} />
|
||||
</Button>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
### Step 6: Add State Management
|
||||
|
||||
If needed, update the Zustand store:
|
||||
|
||||
```typescript
|
||||
// apps/web/lib/stores/use-notes-store.ts
|
||||
interface NotesState {
|
||||
// ... existing state
|
||||
toggleBookmark: (noteId: string) => Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
### Step 7: Write Tests
|
||||
|
||||
Add tests for the new functionality:
|
||||
|
||||
```typescript
|
||||
// tests/note-bookmark.test.ts
|
||||
describe("NoteBookmarkButton", () => {
|
||||
it("toggles bookmark state on click", () => {
|
||||
// ...
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Step 8: Update MCP Tools (if applicable)
|
||||
|
||||
If the feature should be accessible to AI agents, add or update MCP tools in `apps/web/lib/mcp/tools/`.
|
||||
|
||||
### Step 9: Verify
|
||||
|
||||
1. Run `npm run typecheck`: TypeScript compiles without errors
|
||||
2. Run `npm run lint`: No lint errors
|
||||
3. Run `npm run test`: All tests pass
|
||||
4. Run `npm run test:e2e`: E2E tests pass (if applicable)
|
||||
5. Test manually in the browser
|
||||
|
||||
## Database Schema Changes
|
||||
|
||||
### Creating Migrations
|
||||
|
||||
PocketBase migrations are JavaScript files in `pocketbase/pb_migrations/`.
|
||||
|
||||
**Naming convention:** `YYYYMMDDHHMMSS_description.js`
|
||||
|
||||
**Example (add a new collection):**
|
||||
|
||||
```javascript
|
||||
export default {
|
||||
async up(db) {
|
||||
const collection = new Collection({
|
||||
name: "bookmarks",
|
||||
type: "base",
|
||||
fields: [
|
||||
{ name: "title", type: "text", required: true },
|
||||
{ name: "url", type: "url", required: true },
|
||||
{ name: "note_id", type: "relation", options: { collectionId: "notes" } },
|
||||
],
|
||||
});
|
||||
return db.saveCollection(collection);
|
||||
},
|
||||
|
||||
async down(db) {
|
||||
return db.deleteCollection("bookmarks");
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
### Running Migrations
|
||||
|
||||
Migrations run automatically when PocketBase starts. To run them manually:
|
||||
|
||||
```bash
|
||||
pocketbase migrate --dir=./pocketbase/pb_migrations --dir=./pb_data
|
||||
```
|
||||
|
||||
### Updating TypeScript Types
|
||||
|
||||
After changing the schema, update the TypeScript types in `pocketbase/schema.ts` to match. This keeps the type system in sync with the database.
|
||||
|
||||
### Rules for Schema Changes
|
||||
|
||||
1. **Always provide both `up` and `down`**: Migrations must be reversible
|
||||
2. **Never modify existing migrations**: Create new ones instead
|
||||
3. **Test migrations locally** before committing
|
||||
4. **Update TypeScript types** in the same PR as the migration
|
||||
5. **Update Zod schemas** in `packages/shared/` if the change affects validation
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
|
||||
Unit tests cover pure functions and business logic. They run with Jest.
|
||||
|
||||
```bash
|
||||
npm run test
|
||||
```
|
||||
|
||||
**Location:** `tests/` directory or alongside source files.
|
||||
|
||||
**What to test:**
|
||||
- Zod schema validation
|
||||
- Utility functions (date formatting, string manipulation)
|
||||
- Service layer logic
|
||||
- Store actions
|
||||
|
||||
**Example:**
|
||||
|
||||
```typescript
|
||||
// tests/format-duration.test.ts
|
||||
import { formatDuration } from "@/lib/utils";
|
||||
|
||||
describe("formatDuration", () => {
|
||||
it("formats minutes to hours and minutes", () => {
|
||||
expect(formatDuration(90)).toBe("1h 30m");
|
||||
});
|
||||
|
||||
it("handles zero minutes", () => {
|
||||
expect(formatDuration(0)).toBe("0m");
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Component Tests
|
||||
|
||||
Component tests verify React components render correctly and handle user interactions.
|
||||
|
||||
**Location:** `tests/` directory or alongside component files.
|
||||
|
||||
**What to test:**
|
||||
- Components render with required props
|
||||
- User interactions trigger correct callbacks
|
||||
- Conditional rendering works as expected
|
||||
|
||||
### E2E Tests
|
||||
|
||||
E2E tests verify complete user flows using Playwright. They run against a real browser.
|
||||
|
||||
```bash
|
||||
# Run all E2E tests
|
||||
npm run test:e2e
|
||||
|
||||
# Run with UI mode (interactive debugging)
|
||||
npm run test:e2e:ui
|
||||
|
||||
# View test report
|
||||
npm run test:e2e:report
|
||||
```
|
||||
|
||||
**Location:** `e2e/` directory.
|
||||
|
||||
**Browser configurations:**
|
||||
- Chromium (Desktop)
|
||||
- Firefox (Desktop)
|
||||
- WebKit (Desktop Safari)
|
||||
- Mobile Chrome (Pixel 5)
|
||||
- Mobile Safari (iPhone 12)
|
||||
|
||||
**What to test:**
|
||||
- Complete user flows (login → create task → complete task)
|
||||
- Navigation between pages
|
||||
- Form submissions
|
||||
- Realtime updates
|
||||
- Error states
|
||||
|
||||
**Example:**
|
||||
|
||||
```typescript
|
||||
// e2e/tasks.spec.ts
|
||||
import { test, expect } from "@playwright/test";
|
||||
|
||||
test.describe("Tasks", () => {
|
||||
test("create and complete a task", async ({ page }) => {
|
||||
await page.goto("/dashboard/tasks");
|
||||
await page.click('[data-testid="create-task-button"]');
|
||||
await page.fill('[data-testid="task-title"]', "New task");
|
||||
await page.click('[data-testid="create-button"]');
|
||||
|
||||
await expect(page.locator('[data-testid="task-item"]')).toContainText("New task");
|
||||
|
||||
await page.click('[data-testid="task-checkbox"]');
|
||||
await expect(page.locator('[data-testid="task-status"]')).toHaveText("done");
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Test Naming
|
||||
|
||||
- Unit tests: `describe("functionName", () => { ... })`
|
||||
- Component tests: `describe("ComponentName", () => { ... })`
|
||||
- E2E tests: `test.describe("Feature", () => { ... })`
|
||||
|
||||
### Running Tests in CI
|
||||
|
||||
CI runs all tests automatically on every PR:
|
||||
|
||||
```bash
|
||||
npm run typecheck # TypeScript check
|
||||
npm run lint # Linting
|
||||
npm run test # Unit + component tests
|
||||
npm run test:e2e # E2E tests
|
||||
```
|
||||
|
||||
## Code Style and Conventions
|
||||
|
||||
### TypeScript
|
||||
|
||||
- Use strict mode (enabled in `tsconfig.json`)
|
||||
- Prefer interfaces for object shapes, types for unions and utilities
|
||||
- Use `unknown` instead of `any` for external data
|
||||
- Add JSDoc comments for exported functions
|
||||
|
||||
### React
|
||||
|
||||
- Use functional components with hooks
|
||||
- Mark client components with `"use client"` directive
|
||||
- Keep components small and focused
|
||||
- Extract reusable logic into custom hooks
|
||||
- Use `React.memo` only when profiling shows a need
|
||||
|
||||
### Styling
|
||||
|
||||
- Use Tailwind CSS utility classes
|
||||
- Use `cn()` from `lib/utils.ts` to merge class names
|
||||
- Use `cva` for component variants
|
||||
- Avoid inline styles unless dynamic values are required
|
||||
|
||||
### Error Handling
|
||||
|
||||
- Use `ApiError` and `AuthError` classes from `lib/auth.ts`
|
||||
- Return consistent error responses: `{ error: { code, message, details? } }`
|
||||
- Catch errors at the API route boundary
|
||||
- Log errors with context (user ID, request path)
|
||||
|
||||
### Async/Await
|
||||
|
||||
- Use async/await instead of `.then()` chains
|
||||
- Handle errors with try/catch
|
||||
- Use `AbortSignal.timeout()` for fetch requests with timeouts
|
||||
|
||||
## Git Workflow
|
||||
|
||||
### Branch Naming
|
||||
|
||||
```
|
||||
feature/description : New features
|
||||
fix/description : Bug fixes
|
||||
refactor/description : Code refactoring
|
||||
docs/description : Documentation changes
|
||||
test/description : Test additions
|
||||
chore/description : Maintenance tasks
|
||||
```
|
||||
|
||||
### Commit Messages
|
||||
|
||||
Use conventional commits:
|
||||
|
||||
```
|
||||
feat: add bookmark support to notes
|
||||
fix: resolve realtime SSE reconnection loop
|
||||
refactor: extract task filtering into custom hook
|
||||
docs: update API documentation for tasks endpoint
|
||||
test: add E2E tests for habit logging flow
|
||||
chore: upgrade Next.js to 15.3.0
|
||||
```
|
||||
|
||||
### Commit Guidelines
|
||||
|
||||
- One logical change per commit
|
||||
- Keep commits atomic and reversible
|
||||
- Write the subject line in imperative mood ("add feature" not "added feature")
|
||||
- Keep subject lines under 72 characters
|
||||
- Add a body for complex changes explaining the "why"
|
||||
|
||||
### Before Pushing
|
||||
|
||||
1. Run `npm run typecheck`: Must pass
|
||||
2. Run `npm run lint`: Must pass
|
||||
3. Run `npm run test`: Must pass
|
||||
4. Run `npm run test:e2e`: Must pass (for feature/fix branches)
|
||||
5. Review your diff: `git diff --stat`
|
||||
|
||||
## PR Review Process
|
||||
|
||||
### Creating a PR
|
||||
|
||||
1. Push your branch to the remote
|
||||
2. Open a PR against `main`
|
||||
3. Fill in the PR template:
|
||||
- What does this PR do?
|
||||
- Why is this change needed?
|
||||
- How was it tested?
|
||||
- Screenshots (for UI changes)
|
||||
|
||||
### Review Checklist
|
||||
|
||||
Reviewers check:
|
||||
|
||||
- [ ] Code compiles without TypeScript errors
|
||||
- [ ] Lint passes
|
||||
- [ ] Tests pass (unit + E2E)
|
||||
- [ ] Code follows project conventions
|
||||
- [ ] No unnecessary dependencies added
|
||||
- [ ] Error handling is complete
|
||||
- [ ] UI is accessible (keyboard navigation, ARIA labels)
|
||||
- [ ] Documentation updated (if API changed)
|
||||
|
||||
### Merging
|
||||
|
||||
- PRs require at least one approval
|
||||
- All CI checks must pass
|
||||
- Squash merge preferred for clean history
|
||||
- Delete the branch after merging
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Adding a New API Endpoint
|
||||
|
||||
1. Create a directory under `apps/web/app/api/`:
|
||||
1. Define or update the data shape in `packages/db/src/schema.ts`.
|
||||
2. Generate a Drizzle migration.
|
||||
|
||||
```bash
|
||||
mkdir apps/web/app/api/bookmarks
|
||||
npm run db:generate
|
||||
```
|
||||
|
||||
2. Create `route.ts`:
|
||||
3. Review and commit the generated SQL in `drizzle/`.
|
||||
4. Apply the migration to your local PostgreSQL database before testing. The initial Docker setup applies `drizzle/0000_first_mauler.sql`; apply later migrations through your deployment migration process.
|
||||
5. Update shared Zod schemas in `packages/shared/` when validation changes.
|
||||
6. Update the relevant API route, service, state, and UI.
|
||||
7. Add tests for the new behavior.
|
||||
|
||||
```typescript
|
||||
import { NextRequest, NextResponse } from "next/server";
|
||||
import { withAuth } from "@/lib/auth";
|
||||
import { createPocketBaseClient } from "@/lib/pocketbase";
|
||||
## Database schema changes
|
||||
|
||||
export const GET = withAuth(async (request: NextRequest, user) => {
|
||||
const pb = createPocketBaseClient();
|
||||
const result = await pb.collection("bookmarks").getList(1, 50);
|
||||
return NextResponse.json(result);
|
||||
});
|
||||
- Do not edit a migration after another environment has applied it.
|
||||
- Keep the Drizzle schema and generated SQL in the same pull request.
|
||||
- Test a migration against a database with representative data.
|
||||
- Add indexes for fields used in common filters or sorts.
|
||||
- Back up production data before applying a migration.
|
||||
|
||||
export const POST = withAuth(async (request: NextRequest, user) => {
|
||||
const body = await request.json();
|
||||
const pb = createPocketBaseClient();
|
||||
const bookmark = await pb.collection("bookmarks").create(body);
|
||||
return NextResponse.json(bookmark, { status: 201 });
|
||||
});
|
||||
```
|
||||
`drizzle.config.ts` reads `DATABASE_URL` and writes generated migrations to `drizzle/`.
|
||||
|
||||
3. For dynamic routes, create `[id]/route.ts`:
|
||||
## Testing
|
||||
|
||||
```typescript
|
||||
export const GET = withAuth(async (request: NextRequest, user, context: { params: { id: string } }) => {
|
||||
const { id } = await context.params;
|
||||
const pb = createPocketBaseClient();
|
||||
const bookmark = await pb.collection("bookmarks").getOne(id);
|
||||
return NextResponse.json(bookmark);
|
||||
});
|
||||
```
|
||||
Run the checks that match your change:
|
||||
|
||||
### Adding a New Zustand Store
|
||||
|
||||
```typescript
|
||||
// apps/web/lib/stores/use-bookmarks-store.ts
|
||||
import { create } from "zustand";
|
||||
|
||||
interface Bookmark {
|
||||
id: string;
|
||||
title: string;
|
||||
url: string;
|
||||
}
|
||||
|
||||
interface BookmarksState {
|
||||
bookmarks: Bookmark[];
|
||||
loading: boolean;
|
||||
fetchBookmarks: () => Promise<void>;
|
||||
addBookmark: (bookmark: Bookmark) => void;
|
||||
}
|
||||
|
||||
export const useBookmarksStore = create<BookmarksState>((set) => ({
|
||||
bookmarks: [],
|
||||
loading: false,
|
||||
|
||||
fetchBookmarks: async () => {
|
||||
set({ loading: true });
|
||||
const response = await fetch("/api/bookmarks");
|
||||
const data = await response.json();
|
||||
set({ bookmarks: data.items, loading: false });
|
||||
},
|
||||
|
||||
addBookmark: (bookmark) =>
|
||||
set((state) => ({
|
||||
bookmarks: [...state.bookmarks, bookmark],
|
||||
})),
|
||||
}));
|
||||
```bash
|
||||
npm run typecheck
|
||||
npm run lint
|
||||
npm run test
|
||||
npm run test:e2e
|
||||
```
|
||||
|
||||
### Adding a New MCP Tool
|
||||
Unit tests cover validation, utilities, services, and store actions. Component tests cover rendering and user interactions. Playwright tests cover browser flows such as signing in, creating a task, and completing it.
|
||||
|
||||
1. Add the tool to the appropriate file in `apps/web/lib/mcp/tools/`:
|
||||
## Code conventions
|
||||
|
||||
```typescript
|
||||
// apps/web/lib/mcp/tools/bookmarks.ts
|
||||
server.tool("create_bookmark", "Create a new bookmark", {
|
||||
title: z.string(),
|
||||
url: z.string().url(),
|
||||
note_id: z.string().optional(),
|
||||
}, async (args) => {
|
||||
try {
|
||||
const bookmark = await pb.collection("bookmarks").create({
|
||||
title: args.title,
|
||||
url: args.url,
|
||||
note_id: args.note_id || "",
|
||||
});
|
||||
return textContent(JSON.stringify({ success: true, bookmark }));
|
||||
} catch (error) {
|
||||
return textContent(JSON.stringify({ success: false, error: String(error) }));
|
||||
}
|
||||
});
|
||||
```
|
||||
- Use TypeScript strict mode.
|
||||
- Prefer `unknown` over `any` for external input.
|
||||
- Keep React components focused and extract shared logic into hooks.
|
||||
- Use Tailwind utility classes, `cn()` for class merging, and `cva` for variants.
|
||||
- Validate API input with shared Zod schemas.
|
||||
- Catch errors at API boundaries and log request context.
|
||||
|
||||
2. Register the tool in `apps/web/lib/mcp/server.ts`:
|
||||
## Git workflow
|
||||
|
||||
```typescript
|
||||
import { registerBookmarkTools } from "./tools/bookmarks";
|
||||
// ...
|
||||
registerBookmarkTools(server);
|
||||
```
|
||||
Use conventional commits and keep each commit focused. Before pushing, run the relevant checks and review `git diff --stat`.
|
||||
|
||||
### Adding a New Background Job Type
|
||||
Use these branch prefixes:
|
||||
|
||||
1. Add a case to the worker's `processJob` function in `worker/index.ts`:
|
||||
```
|
||||
feature/description
|
||||
fix/description
|
||||
refactor/description
|
||||
chore/description
|
||||
```
|
||||
|
||||
```typescript
|
||||
case "send_notification":
|
||||
await handleSendNotification(job);
|
||||
break;
|
||||
```
|
||||
|
||||
2. Implement the handler:
|
||||
|
||||
```typescript
|
||||
async function handleSendNotification(job: QueueJob): Promise<void> {
|
||||
const payload = job.payload as { user_id: string; message: string };
|
||||
const pb = createAdminClient();
|
||||
await pb.collection("notifications").create({
|
||||
user_id: payload.user_id,
|
||||
message: payload.message,
|
||||
type: "info",
|
||||
read: false,
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
3. Schedule the job from your API route or service:
|
||||
|
||||
```typescript
|
||||
await pb.collection("queue_jobs").create({
|
||||
type: "send_notification",
|
||||
queue: "default",
|
||||
payload: { user_id: "user123", message: "Task completed" },
|
||||
status: "pending",
|
||||
retry_count: 0,
|
||||
max_retries: 3,
|
||||
scheduled_at: new Date().toISOString(),
|
||||
});
|
||||
```
|
||||
Pull requests should explain the change, its reason, and the tests you ran. Include screenshots for UI changes.
|
||||
|
||||
Reference in New Issue
Block a user