refactor: migrate to monorepo structure with Docker, PocketBase, and e2e tests

- Reorganize into apps/, packages/, docs/, e2e/, pocketbase/ directories
- Add Dockerfiles for web, worker, and PocketBase services
- Add docker-compose.yml for local orchestration
- Add turbo.json for monorepo task management
- Add Playwright e2e test infrastructure
- Add PocketBase backend with migrations
- Remove Vite/Next.js/ESLint/PostCSS config files
- Update package.json with workspace dependencies
- Add .env.example and .dockerignore
This commit is contained in:
2026-07-16 06:19:58 -04:00
parent ec14645a4b
commit 8f55626e03
286 changed files with 31992 additions and 9245 deletions
+1027
View File
File diff suppressed because it is too large Load Diff
+505
View File
@@ -0,0 +1,505 @@
# Architecture Documentation
Project E is a monorepo application built on Next.js with PocketBase as the data layer. This document explains the system design, data flow, and key architectural decisions.
## Table of Contents
- [System Overview](#system-overview)
- [Three-Layer Architecture](#three-layer-architecture)
- [Data Flow](#data-flow)
- [Realtime Architecture](#realtime-architecture)
- [Worker Architecture](#worker-architecture)
- [MCP Server Architecture](#mcp-server-architecture)
- [Security](#security)
- [Performance](#performance)
- [Accessibility](#accessibility)
## System Overview
```
┌──────────────────────────────────────────────────────────────────────┐
│ Client (Browser) │
│ Next.js (React 19) · Zustand · SSE Client · PocketBase SDK │
└──────────────────────────────┬───────────────────────────────────────┘
┌──────────┴──────────┐
│ │
REST API SSE /api/realtime
/api/* (PocketBase events)
│ │
┌───────────────────▼─────────────────────▼────────────────────────────┐
│ Next.js Server (apps/web) │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ ┌───────────┐ │
│ │ API Routes │ │ MCP Server │ │ Auth │ │ Middleware │ │
│ │ (CRUD ops) │ │ (61 tools) │ │ (cookies) │ │ (routing) │ │
│ └──────┬──────┘ └──────┬───────┘ └──────┬──────┘ └───────────┘ │
│ │ │ │ │
│ ┌──────▼────────────────▼──────────────────▼──────────────────────┐ │
│ │ Service Layer │ │
│ │ task-service · habit-service · note-service · report-service │ │
│ │ webhook-service · agent-mention-service · project-service │ │
│ └──────────────────────────┬──────────────────────────────────────┘ │
│ │ │
│ ┌──────────────────────────▼──────────────────────────────────────┐ │
│ │ PocketBase Client │ │
│ │ createPocketBaseClient() · createAdminClient() │ │
│ └──────────────────────────┬──────────────────────────────────────┘ │
└─────────────────────────────┼────────────────────────────────────────┘
┌─────────────────────────────▼────────────────────────────────────────┐
│ PocketBase (SQLite) │
│ Auth · Collections · Realtime · File Storage · Admin UI │
│ 30+ collections: tasks, habits, projects, notes, reports, ... │
└──────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────┐
│ Background Worker │
│ Polls queue_jobs collection · Exponential backoff · 5 job types │
│ webhook_delivery · agent_mention · report_generation │
│ recurring_task · cleanup │
└──────────────────────────────────────────────────────────────────────┘
```
## Three-Layer Architecture
The application separates concerns into three layers.
### Presentation Layer
**Location:** `apps/web/app/` (pages) and `apps/web/components/` (UI)
The presentation layer handles rendering, user interaction, and client-side state.
**Key components:**
- **Pages**: Next.js App Router pages organized by route groups: `(auth)` for login/signup, `(dashboard)` for the main application
- **UI Components**: shadcn/ui primitives in `components/ui/`, feature components in `components/`
- **Client State**: Zustand stores in `lib/stores/` manage UI state (sidebar, filters, theme, timer)
- **Hooks**: Custom hooks in `hooks/` encapsulate reusable logic
The presentation layer communicates with the business logic layer through:
1. REST API calls (fetch to `/api/*`)
2. Zustand store actions (which call the API)
3. SSE events from the realtime endpoint
### Business Logic Layer
**Location:** `apps/web/lib/services/` and `packages/shared/`
The business logic layer handles validation, data transformation, and orchestration.
**Services:**
| Service | Responsibility |
|---------|---------------|
| `task-service` | Task CRUD, recurring task spawning, dependency resolution |
| `habit-service` | Habit tracking, streak calculation, completion scoring |
| `note-service` | Note CRUD, wikilink parsing, word count |
| `report-service` | Report generation, template rendering |
| `project-service` | Project progress calculation, milestone tracking |
| `webhook-service` | Webhook event dispatch, delivery tracking |
| `agent-mention-service` | Agent @mention parsing, task dispatch |
**Shared package (`@project-e/shared`):**
Contains Zod schemas and TypeScript types shared between the web app, worker, and potentially other consumers. Every entity has:
- A Zod schema for runtime validation
- Inferred TypeScript types for compile-time safety
- Create and update schema variants (omit server-generated fields)
### Data Access Layer
**Location:** `apps/web/lib/pocketbase.ts` and `apps/web/app/api/`
The data access layer handles communication with PocketBase.
**PocketBase client (`lib/pocketbase.ts`):**
Provides two factory functions:
- `createPocketBaseClient(token?)`: Creates a client with optional user auth
- `createAdminClient()`: Creates a client with admin privileges (uses `POCKETBASE_ADMIN_TOKEN`)
Also provides generic helpers: `getRecord`, `listRecords`, `createRecord`, `updateRecord`, `deleteRecord`.
**API routes (`app/api/`):**
Each entity has a directory under `app/api/` with a `route.ts` file implementing GET (list) and POST (create). Dynamic routes use `[id]/route.ts` for GET (single), PATCH (update), and DELETE.
All API routes use the `withAuth` middleware to enforce authentication.
## Data Flow
### Creating a Task
```
User fills form → React component
POST /api/tasks (JSON body)
withAuth middleware validates session
Zod schema validates body (createTaskSchema)
PocketBase client creates record
PocketBase triggers realtime event
├──▶ SSE /api/realtime → Browser updates UI
└──▶ Worker picks up job (if webhook subscribed)
```
### Realtime Update Flow
```
User A updates a task
PATCH /api/tasks/abc123
PocketBase updates SQLite record
PocketBase emits realtime event
├──▶ User A's SSE connection receives event → UI updates
├──▶ User B's SSE connection receives event → UI updates
└──▶ Webhook delivery queued (if subscribed)
```
### Agent Tool Call Flow
```
AI agent sends MCP request
GET/POST /api/mcp (Authorization: Bearer <api_key>)
Authenticate: look up agent by API key
MCP server routes to tool handler
Tool handler calls PocketBase (admin client)
Return JSON result to agent
Agent activity logged to agent_activity collection
```
## Realtime Architecture
Project E uses Server-Sent Events (SSE) to push realtime updates from PocketBase to the browser.
### Why SSE Instead of WebSockets
1. **Simpler infrastructure**: SSE works over standard HTTP, no special proxy configuration needed
2. **Unidirectional**: The server pushes events; the client sends commands through REST. This matches the data flow.
3. **Auto-reconnect**: The browser's `EventSource` API handles reconnection automatically
4. **PocketBase native**: PocketBase has built-in realtime subscriptions. SSE is a natural proxy.
### SSE Proxy Architecture
PocketBase's realtime uses WebSockets internally. The Next.js app acts as a proxy:
```
Browser ──SSE──▶ /api/realtime ──WebSocket──▶ PocketBase
```
The proxy:
1. Opens a PocketBase WebSocket connection for each subscribed collection
2. Translates PocketBase events into SSE `data:` frames
3. Sends a `:ping` comment every 30 seconds to keep the connection alive
4. Cleans up all subscriptions when the client disconnects
### Subscription Management
Clients specify which collections to subscribe to via query parameters:
```
GET /api/realtime?collections=tasks,habits,projects
```
Default subscriptions (if no parameter): `tasks`, `habits`, `projects`, `notes`, `reports`, `milestones`, `notifications`.
### Event Format
```
data: {"type":"connected","collections":["tasks","habits"]}
data: {"type":"create","collection":"tasks","record":{"id":"abc","title":"New task"}}
data: {"type":"update","collection":"tasks","record":{"id":"abc","status":"done"}}
data: {"type":"delete","collection":"tasks","record":{"id":"abc"}}
:ping
```
### Client-Side Handling
The Zustand stores subscribe to SSE events and update local state:
```typescript
const eventSource = new EventSource("/api/realtime?collections=tasks");
eventSource.onmessage = (event) => {
const data = JSON.parse(event.data);
if (data.type === "update" && data.collection === "tasks") {
useTaskStore.getState().updateTask(data.record);
}
};
```
## Worker Architecture
The background worker processes asynchronous jobs stored in the `queue_jobs` PocketBase collection.
### Job Lifecycle
```
API creates job → status: "pending"
Worker polls for pending jobs (every 5s base)
Worker marks job as "in_progress"
├──▶ Success → status: "completed"
└──▶ Failure → retry_count < max_retries?
├──▶ Yes → status: "pending", scheduled_at: now + backoff
└──▶ No → status: "failed", error: message
```
### Polling Strategy
The worker uses exponential backoff to reduce database load when idle:
| Condition | Poll Interval |
|-----------|--------------|
| Jobs found | Reset to 5 seconds |
| No jobs | Multiply by 1.5 (max 60 seconds) |
| Error | Keep current interval |
### Job Types
| Type | Handler | Description |
|------|---------|-------------|
| `webhook_delivery` | `handleWebhookDelivery` | POST payload to webhook URL with HMAC signature |
| `agent_mention` | `handleAgentMention` | Dispatch task to agent webhook |
| `report_generation` | `handleReportGeneration` | Collect data and populate report content |
| `recurring_task` | `handleRecurringTask` | Compute next due date from RRULE and spawn task |
| `cleanup` | `handleCleanup` | Purge old webhook deliveries and error logs |
### Retry Strategy
Failed jobs retry with exponential backoff:
```
Retry 1: 5 seconds
Retry 2: 10 seconds
Retry 3: 20 seconds
...
Max backoff: 5 minutes
Max retries: 3 (configurable per job)
```
### Cleanup
The worker schedules a daily cleanup job that:
- Deletes webhook deliveries older than 90 days
- Deletes error logs older than 30 days
## MCP Server Architecture
The MCP server runs inside the Next.js API at `/api/mcp`. It uses the `@modelcontextprotocol/sdk` package.
### Transport
The server uses **Streamable HTTP** transport:
1. `GET /api/mcp`: Establish a session, returns `mcp-session-id` header
2. `POST /api/mcp`: Send JSON-RPC requests with `mcp-session-id` header
3. `DELETE /api/mcp`: End the session
Sessions are stored in an in-memory `Map<string, Transport>`.
### Authentication
Every request validates the `Authorization: Bearer <api_key>` header against the `agents` collection. Only active agents can connect.
### Tool Organization
Tools are organized by entity in `apps/web/lib/mcp/tools/`:
```
lib/mcp/
├── server.ts : Creates McpServer, registers all tools
└── tools/
├── tasks.ts : 8 tools
├── habits.ts : 7 tools
├── projects.ts : 6 tools
├── notes.ts : 6 tools
├── reports.ts : 5 tools
├── milestones.ts : 5 tools
├── domains.ts : 5 tools
├── tags.ts : 5 tools
├── agents.ts : 5 tools
├── webhooks.ts : 5 tools
└── analytics.ts : 4 tools
```
Each file exports a `register*Tools(server)` function that adds tools to the server.
### Tool Execution
Tools use the admin PocketBase client (`createAdminClient()`) to bypass row-level security. This is safe because:
1. The MCP endpoint requires a valid agent API key
2. Agent permission tiers control which tools are available
3. All tool calls are logged to `agent_activity`
## Security
### Authentication
**User authentication** uses PocketBase's built-in auth system:
- Email/password login
- JWT tokens stored in `httpOnly` cookies
- 7-day token expiry with refresh endpoint
- Tokens validated on every API request via `withAuth` middleware
**Agent authentication** uses API keys:
- Each agent has a unique API key (UUID)
- Keys validated against the `agents` collection
- Disabled agents cannot authenticate
- Used for both MCP server and agent webhook callbacks
### Authorization
**API routes** use the `withAuth` middleware. All routes require a valid user session. PocketBase row-level security provides additional protection at the database level.
**MCP tools** use agent permission tiers. The `read_only` tier can only call `get_*` and `list_*` tools. The server enforces this at the tool registration level.
### Data Protection
- **Passwords**: Hashed by PocketBase using bcrypt
- **API keys**: Stored as plain text (UUIDs). Treat them like passwords.
- **Webhook secrets**: Used for HMAC-SHA256 payload signing
- **Admin token**: Stored in environment variable, not in the database
- **Cookies**: `httpOnly`, `secure` (production), `sameSite: lax`
### Input Validation
All API endpoints validate request bodies with Zod schemas from `@project-e/shared`. Invalid input returns a `400 VALIDATION_ERROR` with details about which fields failed.
### CORS
Configure CORS in `next.config.ts` for cross-origin requests. By default, Next.js allows same-origin requests only.
### Rate Limiting
The application does not enforce rate limiting. Configure it at the reverse proxy or infrastructure layer. See [Deployment Guide](DEPLOYMENT.md#rate-limiting) for recommendations.
## Performance
### Database
PocketBase uses SQLite, which handles read-heavy workloads well.
**Optimizations:**
- PocketBase enables WAL mode by default for concurrent reads
- Indexes on frequently queried fields (status, domain, project_id)
- Pagination limits prevent large result sets
**Limits:**
- SQLite handles thousands of concurrent reads
- Write throughput is limited by single-writer model
- For most personal/team use cases, this is not a bottleneck
### API Responses
**Caching:**
- List endpoints set `Cache-Control: private, max-age=60, stale-while-revalidate=300`
- This allows the browser to use stale data while revalidating in the background
**Compression:**
- Next.js compresses responses automatically (gzip/brotli)
- The reverse proxy should also enable compression
### Realtime
**Connection management:**
- Each SSE connection holds a PocketBase WebSocket subscription
- The proxy cleans up subscriptions on client disconnect
- Keepalive pings prevent idle connection timeouts
**Scaling limits:**
- Each SSE connection uses a server-side WebSocket to PocketBase
- For hundreds of concurrent users, consider connection pooling or a dedicated realtime service
### Frontend
**Next.js optimizations:**
- App Router with React Server Components for initial page loads
- Client components only where interactivity is needed
- Automatic code splitting per route
- Image optimization via `next/image`
**Bundle size:**
- shadcn/ui components are tree-shaken (only imported components are included)
- Zustand stores are lightweight (no boilerplate)
- Tiptap editor loads lazily
## Accessibility
Project E follows WCAG 2.1 AA guidelines.
### Keyboard Navigation
- All interactive elements are reachable via keyboard
- Focus order follows visual layout
- Focus indicators are visible (Tailwind `focus-visible:ring-2`)
- Modal dialogs trap focus and return it on close
### ARIA Labels
- Icon-only buttons include `aria-label` attributes
- Form fields have associated `<label>` elements
- Dynamic content uses `aria-live` regions
- Dialogs use `role="dialog"` and `aria-modal="true"`
### Color and Contrast
- Text meets 4.5:1 contrast ratio against backgrounds
- Status indicators use both color and text/icons
- Domain colors are customizable but default to accessible palette
### Screen Readers
- Semantic HTML elements (`<nav>`, `<main>`, `<section>`, `<article>`)
- Headings follow proper hierarchy (h1 → h2 → h3)
- Tables use `<th>` with `scope` attributes
- Toast notifications announce via `aria-live="polite"`
### Motion
- Animations respect `prefers-reduced-motion`
- No auto-playing content
- Transitions are subtle and brief
+639
View File
@@ -0,0 +1,639 @@
# Deployment Guide
Deploy Project E with Docker Compose and Nginx Proxy Manager. Three containers run the application: web (Next.js), db (PocketBase), and worker (background jobs).
## Table of Contents
- [Prerequisites](#prerequisites)
- [Quick Deploy](#quick-deploy)
- [Environment Configuration](#environment-configuration)
- [Nginx Proxy Manager Setup](#nginx-proxy-manager-setup)
- [PocketBase Setup](#pocketbase-setup)
- [Backups](#backups)
- [Monitoring](#monitoring)
- [Scaling](#scaling)
- [Troubleshooting](#troubleshooting)
## Prerequisites
- **Docker** 24.0 or later
- **Docker Compose** 2.20 or later
- **Nginx Proxy Manager** installed and running
- **At least 1GB RAM** and 10GB disk space
## Quick Deploy
1. **Clone the repository on your server**
```bash
git clone <repository-url>
cd ProjectE
```
2. **Create the environment file**
```bash
cp .env.example .env
```
Edit `.env` and set the required variables:
```bash
POCKETBASE_ADMIN_TOKEN=your-secure-random-token
PUBLIC_URL=http://project-e.local
COOKIE_SECURE=false
ALLOWED_HOSTS=project-e.local,localhost
```
Generate a secure token:
```bash
openssl rand -hex 32
```
3. **Build and start containers**
```bash
docker compose up -d
```
4. **Verify all containers are running**
```bash
docker compose ps
```
You should see three containers with status `Up (healthy)`:
- `project-e-web` (internal only, no exposed ports)
- `project-e-db` (internal only, no exposed ports)
- `project-e-worker` running in the background
5. **Configure Nginx Proxy Manager** (see below)
6. **Create your admin account**
Once NPM is configured, access Project E through your domain. Create your first user account.
## Environment Configuration
### Required Variables
| Variable | Description |
|----------|-------------|
| `POCKETBASE_ADMIN_TOKEN` | Admin token from PocketBase. Required for the worker and server-side API operations. |
### Optional Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `POCKETBASE_URL` | `http://db:8090` | PocketBase URL (internal Docker network) |
| `NODE_ENV` | `production` | Node environment |
| `PUBLIC_URL` | `http://localhost:3000` | Public URL where users access Project E (used for generating absolute URLs) |
| `COOKIE_SECURE` | `false` | Set to `true` if using HTTPS through NPM, `false` for HTTP-only LAN access |
| `ALLOWED_HOSTS` | `localhost` | Comma-separated list of domains that can access the app |
### Setting Variables
Create a `.env` file in the project root:
```bash
POCKETBASE_ADMIN_TOKEN=abc123def456...
POCKETBASE_URL=http://db:8090
PUBLIC_URL=http://project-e.local
COOKIE_SECURE=false
ALLOWED_HOSTS=project-e.local,localhost
```
Docker Compose reads this file automatically.
## Nginx Proxy Manager Setup
### Add Proxy Host
1. Open Nginx Proxy Manager admin interface
2. Go to **Hosts** → **Proxy Hosts** → **Add Proxy Host**
3. Configure the following:
**Details Tab:**
- **Domain Names:** `project-e.local` (or your chosen domain)
- **Scheme:** `http`
- **Forward Hostname/IP:** `project-e-web` (the Docker container name)
- **Forward Port:** `3000`
**SSL Tab:**
- If using HTTPS: Select your SSL certificate
- If HTTP-only on LAN: Leave SSL disabled
**Advanced Tab:**
Add these custom Nginx configuration lines:
```nginx
# WebSocket support for realtime features
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# Forward real client information
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
# SSE support for realtime endpoint
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 300s;
# Increase timeout for long-running requests
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
```
4. Click **Save**
### Testing the Connection
1. From a device on your LAN, open a browser
2. Navigate to `http://project-e.local` (or your configured domain)
3. You should see the Project E login page
### Optional: PocketBase Admin Access
If you need direct access to the PocketBase admin UI:
1. In NPM, add another Proxy Host:
- **Domain Names:** `pb.project-e.local`
- **Scheme:** `http`
- **Forward Hostname/IP:** `project-e-db`
- **Forward Port:** `8090`
2. Or temporarily expose the port in `docker-compose.yml`:
```yaml
db:
ports:
- "8090:8090"
```
## PocketBase Setup
### Initial Configuration
PocketBase runs as a standalone container. After starting it for the first time:
1. Access the admin UI at `http://your-server:8090/_/`
2. Create your admin account
3. Configure authentication settings under **Settings > Auth**
4. Enable the auth methods you need (email/password is enabled by default)
### Migrations
Database migrations are in `pocketbase/pb_migrations/`. They run automatically when the PocketBase container starts.
To add a new migration:
1. Create a file in `pocketbase/pb_migrations/` with the naming convention `YYYYMMDDHHMMSS_description.js`
2. Restart the PocketBase container: `docker compose restart db`
### Data Directory
PocketBase stores all data (SQLite database, uploads, logs) in the `/pb_data` volume. This volume persists across container restarts.
## Reverse Proxy
Put a reverse proxy in front of the application to handle SSL, compression, and routing.
### Caddy (Recommended)
Caddy handles SSL automatically.
Create a `Caddyfile`:
```
your-domain.com {
reverse_proxy localhost:3000
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains"
X-Frame-Options "DENY"
X-Content-Type-Options "nosniff"
}
}
pb.your-domain.com {
reverse_proxy localhost:8090
}
```
Install and run Caddy:
```bash
# Install Caddy
sudo apt install caddy # Debian/Ubuntu
# or
brew install caddy # macOS
# Start Caddy
caddy start
```
### Traefik
Create a `traefik.yml`:
```yaml
entryPoints:
web:
address: ":80"
http:
redirections:
entryPoint:
to: websecure
scheme: https
websecure:
address: ":443"
certificatesResolvers:
letsencrypt:
acme:
email: your-email@example.com
storage: acme.json
httpChallenge:
entryPoint: web
providers:
docker:
exposedByDefault: false
```
Update `docker-compose.yml` to add Traefik labels:
```yaml
services:
web:
labels:
- "traefik.enable=true"
- "traefik.http.routers.web.rule=Host(`your-domain.com`)"
- "traefik.http.routers.web.entrypoints=websecure"
- "traefik.http.routers.web.tls.certresolver=letsencrypt"
```
### Nginx
Create `/etc/nginx/sites-available/project-e`:
```nginx
server {
listen 80;
server_name your-domain.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name your-domain.com;
ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;
location / {
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
# SSE support for realtime endpoint
proxy_buffering off;
proxy_read_timeout 300s;
}
}
```
Enable the site and reload:
```bash
sudo ln -s /etc/nginx/sites-available/project-e /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
```
## SSL/TLS
### Let's Encrypt with Certbot
```bash
# Install certbot
sudo apt install certbot python3-certbot-nginx
# Get certificate
sudo certbot --nginx -d your-domain.com -d pb.your-domain.com
# Auto-renewal is configured automatically
```
### Caddy
Caddy obtains and renews certificates automatically. No configuration needed beyond the domain name in the `Caddyfile`.
### Internal Communication
The web and worker containers connect to PocketBase over the internal Docker network (`http://db:8090`). This traffic does not need SSL.
Only expose ports 3000 and 8090 to the reverse proxy, not directly to the internet.
## Backups
### PocketBase Database
The database is a single SQLite file at `/pb_data/data.db`.
**Manual backup:**
```bash
# Stop the container to ensure consistency
docker compose stop db
# Copy the database file
docker cp project-e-db:/pb_data/data.db ./backups/data-$(date +%Y%m%d).db
# Restart the container
docker compose start db
```
**Automated backup script:**
Create `scripts/backup.sh`:
```bash
#!/bin/bash
BACKUP_DIR="/path/to/backups"
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p $BACKUP_DIR
# Use PocketBase's backup API (no downtime)
curl -X POST http://localhost:8090/api/backup \
-H "Authorization: Admin your-admin-token" \
-o "$BACKUP_DIR/backup-$DATE.zip"
# Keep only last 30 backups
find $BACKUP_DIR -name "backup-*.zip" -mtime +30 -delete
echo "Backup completed: backup-$DATE.zip"
```
Schedule with cron:
```bash
# Run daily at 2 AM
0 2 * * * /path/to/scripts/backup.sh
```
### Upload Files
Uploaded files are stored in the `project-e-web-uploads` volume.
**Backup uploads:**
```bash
docker run --rm \
-v project-e-web-uploads:/source:ro \
-v $(pwd)/backups:/backup \
alpine \
tar czf /backup/uploads-$(date +%Y%m%d).tar.gz -C /source .
```
### Restore
**Restore database:**
```bash
# Stop containers
docker compose stop db
# Remove old data
docker volume rm project-e-pb-data
# Copy backup into new volume
docker volume create project-e-pb-data
docker run --rm \
-v project-e-pb-data:/pb_data \
-v $(pwd)/backups:/backup \
alpine \
sh -c "cp /backup/data-YYYYMMDD.db /pb_data/data.db"
# Restart
docker compose start db
```
**Restore uploads:**
```bash
docker run --rm \
-v project-e-web-uploads:/target \
-v $(pwd)/backups:/backup \
alpine \
sh -c "cd /target && tar xzf /backup/uploads-YYYYMMDD.tar.gz"
```
## Monitoring
### Container Health
All containers include health checks. Check status:
```bash
docker compose ps
```
### Application Health
The web container exposes a health endpoint:
```bash
curl http://localhost:3000/api/health
```
**Response:**
```json
{
"status": "ok",
"timestamp": "2024-01-15T10:30:00.000Z",
"version": "0.1.0"
}
```
### Logs
View logs from all containers:
```bash
# All containers
docker compose logs -f
# Specific container
docker compose logs -f web
docker compose logs -f db
docker compose logs -f worker
```
### Resource Usage
Monitor container resource usage:
```bash
docker stats
```
### Uptime Monitoring
Use an external service to monitor your deployment:
- **UptimeRobot** (free tier): HTTP monitoring with email/SMS alerts
- **Healthchecks.io**: Cron job monitoring
- **Better Stack**: Status pages and incident management
Set up a check for `https://your-domain.com/api/health` with a 60-second interval.
## Scaling
### Vertical Scaling
The application runs as a single instance of each container. To handle more load:
1. **Increase server resources**: Add more CPU and RAM to your host
2. **Increase Node.js memory**: Set `NODE_OPTIONS=--max-old-space-size=4096` in the web container
3. **Increase PocketBase limits**: PocketBase handles thousands of concurrent connections on modest hardware
### Horizontal Scaling
Horizontal scaling requires changes to the architecture:
**Current limitations:**
- MCP sessions are stored in memory (not shared between instances)
- SSE connections are tied to a specific container
- File uploads go to a local volume
**To scale horizontally:**
1. Use a shared session store (Redis)
2. Use a load balancer with sticky sessions for SSE
3. Use object storage (S3) for file uploads
4. Run multiple web containers behind a load balancer
For most personal and small-team use cases, a single instance handles the load. PocketBase with SQLite performs well up to hundreds of concurrent users.
### Worker Scaling
The worker uses polling with exponential backoff. For high-throughput job processing:
1. Run multiple worker containers (they coordinate through the database)
2. Reduce the base poll interval (currently 5 seconds)
3. Use a dedicated job queue (Bull, BullMQ) instead of database polling
## Troubleshooting
### Container Won't Start
**Check logs:**
```bash
docker compose logs web
docker compose logs db
docker compose logs worker
```
**Common issues:**
| Problem | Solution |
|---------|----------|
| `Cannot connect to PocketBase` | Ensure the `db` container is healthy. Check `docker compose ps`. |
| `Port 3000 already in use` | Change the port mapping in `docker-compose.yml` |
| `POCKETBASE_ADMIN_TOKEN not set` | Set the token in `.env` and restart |
| `Migration failed` | Check migration files in `pocketbase/pb_migrations/` |
### Web App Returns 500 Errors
1. Check the web container logs: `docker compose logs web`
2. Verify PocketBase is running: `docker compose exec db wget -qO- http://localhost:8090/api/health`
3. Verify the admin token is correct: `docker compose exec web env | grep POCKETBASE`
### Realtime Events Not Arriving
1. Check the SSE connection: `curl -N http://localhost:3000/api/realtime`
2. Verify the reverse proxy is not buffering SSE responses
3. For Nginx, ensure `proxy_buffering off` is set
4. Check browser console for connection errors
### Worker Not Processing Jobs
1. Check worker logs: `docker compose logs worker`
2. Verify the admin token is set correctly
3. Check for pending jobs in PocketBase admin: `http://localhost:8090/_/#/collections/queue_jobs`
4. Jobs retry automatically with exponential backoff (max 5 minutes between retries)
### Database Corruption
If the SQLite database becomes corrupted:
1. Stop all containers: `docker compose down`
2. Restore from the most recent backup (see [Backups](#backups))
3. If no backup exists, try SQLite's recovery:
```bash
sqlite3 data.db ".recover" | sqlite3 new-data.db
```
4. Replace the corrupted file and restart
### Out of Disk Space
PocketBase stores the database and uploads in the `project-e-pb-data` volume.
**Check disk usage:**
```bash
docker system df
docker volume inspect project-e-pb-data
```
**Clean up:**
```bash
# Remove unused images
docker image prune -a
# Remove unused volumes (WARNING: deletes all data)
docker volume prune
```
**Expand volume:**
Docker volumes use the host filesystem. If the host disk is full, expand it or move the volume to a larger disk.
### SSL Certificate Errors
**Caddy:** Check the Caddy logs for ACME errors. Ensure port 80 is accessible from the internet for the HTTP challenge.
**Let's Encrypt:** Certificates renew automatically. Force renewal:
```bash
sudo certbot renew --force-renewal
```
**Common errors:**
- `Connection refused`: Port 80 is blocked by firewall
- `DNS problem`: Domain does not resolve to this server
- `Rate limit exceeded`: Too many certificate requests. Wait and retry.
### Performance Issues
1. **Slow page loads**: Check if the server has enough RAM. Node.js needs at least 512MB.
2. **Slow API responses**: Check PocketBase query performance in the admin panel
3. **SSE disconnects**: Ensure the reverse proxy has appropriate timeout settings (300s+)
4. **Worker falling behind**: Increase the poll frequency or add more worker instances
+729
View File
@@ -0,0 +1,729 @@
# Development Guide
This guide covers the development workflow for Project E. Read this before contributing code.
## Table of Contents
- [Environment Setup](#environment-setup)
- [Project Structure](#project-structure)
- [Code Organization](#code-organization)
- [Adding a New Feature](#adding-a-new-feature)
- [Database Schema Changes](#database-schema-changes)
- [Testing Strategy](#testing-strategy)
- [Code Style and Conventions](#code-style-and-conventions)
- [Git Workflow](#git-workflow)
- [PR Review Process](#pr-review-process)
- [Common Tasks](#common-tasks)
## Environment Setup
### Prerequisites
- **Node.js** 22.13.0 or later (use `nvm` to manage versions)
- **npm** 10.0.0 or later
- **Git**
- **A code editor** (VS Code recommended)
- **PocketBase** binary (download from [pocketbase.io](https://pocketbase.io))
### Initial Setup
1. **Clone the repository**
```bash
git clone <repository-url>
cd ProjectE
```
2. **Install dependencies**
```bash
npm install
```
3. **Start PocketBase**
In a separate terminal:
```bash
pocketbase serve \
--dir=./pb_data \
--publicDir=./pb_public \
--migrationDir=./pocketbase/pb_migrations
```
Or use Docker:
```bash
docker compose up db -d
```
4. **Set environment variables**
Create `apps/web/.env.local`:
```bash
POCKETBASE_URL=http://localhost:8090
POCKETBASE_ADMIN_TOKEN=your_admin_token
```
Get the admin token from PocketBase after creating your first admin account.
5. **Start the development server**
```bash
npm run dev
```
This starts the Next.js app at `http://localhost:3000` with Turbopack.
6. **Verify everything works**
- Open `http://localhost:3000` in your browser
- Open `http://localhost:8090/_/` for the PocketBase admin UI
- Run `npm run typecheck` to verify TypeScript compiles
### VS Code Setup
Recommended extensions:
- ESLint
- Tailwind CSS IntelliSense
- TypeScript and JavaScript Language Features (built-in)
- Prettier - Code formatter
Create `.vscode/settings.json`:
```json
{
"editor.defaultFormatter": "esbenp.prettier-vscode",
"editor.formatOnSave": true,
"typescript.preferences.importModuleSpecifier": "relative",
"tailwindCSS.experimental.classRegex": [
["cva\\(([^)]*)\\)", "[\"'`]([^\"'`]*).*?[\"'`]"]
]
}
```
## Project Structure
```
project-e/
├── apps/
│ └── web/ # Next.js application (monorepo app)
│ ├── app/ # App Router (pages + API routes)
│ │ ├── (auth)/ # Auth route group (login, signup)
│ │ ├── (dashboard)/ # Dashboard route group
│ │ └── api/ # REST API endpoints
│ ├── components/ # React components
│ │ ├── ui/ # shadcn/ui primitives
│ │ └── ... # Feature components
│ ├── hooks/ # Custom React hooks
│ ├── lib/ # Core utilities
│ │ ├── mcp/ # MCP server and tools
│ │ ├── services/ # Business logic
│ │ ├── stores/ # Zustand stores
│ │ ├── events/ # Event bus
│ │ ├── auth.ts # Auth middleware
│ │ ├── pocketbase.ts # PocketBase client
│ │ └── errors.ts # Error handling
│ └── types/ # TypeScript type definitions
├── packages/
│ └── shared/ # Shared package (@project-e/shared)
│ └── src/
│ ├── schemas/ # Zod validation schemas
│ ├── types/ # Shared TypeScript types
│ └── constants/ # Shared constants
├── pocketbase/
│ ├── pb_migrations/ # Database migrations
│ └── schema.ts # TypeScript types for collections
├── worker/ # Background job worker
│ └── index.ts # Worker entry point
├── e2e/ # Playwright E2E tests
├── tests/ # Unit and component tests
└── docker-compose.yml # Docker Compose configuration
```
## Code Organization
### Layers
The application follows a three-layer architecture:
1. **Presentation**: React components in `apps/web/components/` and pages in `apps/web/app/`
2. **Business Logic**: Services in `apps/web/lib/services/` and shared schemas in `packages/shared/`
3. **Data Access**: PocketBase client in `apps/web/lib/pocketbase.ts` and API routes in `apps/web/app/api/`
### Naming Conventions
| Type | Convention | Example |
|------|-----------|---------|
| Components | PascalCase | `TaskCard.tsx` |
| Hooks | camelCase with `use` prefix | `use-task-filter.ts` |
| Utilities | camelCase | `format-date.ts` |
| Types | PascalCase | `Task.ts` |
| Schemas | camelCase with `Schema` suffix | `taskSchema` |
| API routes | kebab-case directory | `api/habit-logs/route.ts` |
| Stores | camelCase with `use` prefix | `use-dashboard-store.ts` |
### Import Paths
Use the `@/` alias for imports within `apps/web`:
```typescript
import { createPocketBaseClient } from '@/lib/pocketbase';
import { TaskCard } from '@/components/task-card';
```
Use the package name for shared imports:
```typescript
import { createTaskSchema } from '@project-e/shared';
```
## Adding a New Feature
Follow these steps to add a feature end-to-end. This example adds a "bookmarks" feature to notes.
### Step 1: Define the Schema
Add the field to the PocketBase collection schema. Create a migration file:
```bash
# pocketbase/pb_migrations/20240115120000_add_bookmarks.js
export default {
up(db) {
const collection = db.findCollectionByNameOrId("notes");
collection.fields.add(new Field({
name: "bookmarked",
type: "bool",
options: { default: false }
}));
return db.saveCollection(collection);
},
down(db) {
const collection = db.findCollectionByNameOrId("notes");
collection.fields.removeByName("bookmarked");
return db.saveCollection(collection);
}
}
```
### Step 2: Update TypeScript Types
Update the type definition in `pocketbase/schema.ts`:
```typescript
export interface Note extends BaseRecord {
// ... existing fields
bookmarked: boolean;
}
```
### Step 3: Add Validation Schema
Update the Zod schema in `packages/shared/src/schemas/note.ts`:
```typescript
export const noteSchema = z.object({
// ... existing fields
bookmarked: z.boolean().default(false),
});
```
### Step 4: Update the API
If the API route needs changes, update it in `apps/web/app/api/notes/route.ts`. Most CRUD operations work automatically through PocketBase, so you may not need API changes.
### Step 5: Build the UI
Create or update components:
```tsx
// apps/web/components/note-bookmark-button.tsx
"use client";
import { Bookmark } from "lucide-react";
import { Button } from "@/components/ui/button";
interface NoteBookmarkButtonProps {
noteId: string;
bookmarked: boolean;
onToggle: (noteId: string) => void;
}
export function NoteBookmarkButton({ noteId, bookmarked, onToggle }: NoteBookmarkButtonProps) {
return (
<Button
variant="ghost"
size="icon"
onClick={() => onToggle(noteId)}
aria-label={bookmarked ? "Remove bookmark" : "Add bookmark"}
>
<Bookmark className={bookmarked ? "fill-current" : ""} />
</Button>
);
}
```
### Step 6: Add State Management
If needed, update the Zustand store:
```typescript
// apps/web/lib/stores/use-notes-store.ts
interface NotesState {
// ... existing state
toggleBookmark: (noteId: string) => Promise<void>;
}
```
### Step 7: Write Tests
Add tests for the new functionality:
```typescript
// tests/note-bookmark.test.ts
describe("NoteBookmarkButton", () => {
it("toggles bookmark state on click", () => {
// ...
});
});
```
### Step 8: Update MCP Tools (if applicable)
If the feature should be accessible to AI agents, add or update MCP tools in `apps/web/lib/mcp/tools/`.
### Step 9: Verify
1. Run `npm run typecheck`: TypeScript compiles without errors
2. Run `npm run lint`: No lint errors
3. Run `npm run test`: All tests pass
4. Run `npm run test:e2e`: E2E tests pass (if applicable)
5. Test manually in the browser
## Database Schema Changes
### Creating Migrations
PocketBase migrations are JavaScript files in `pocketbase/pb_migrations/`.
**Naming convention:** `YYYYMMDDHHMMSS_description.js`
**Example (add a new collection):**
```javascript
export default {
async up(db) {
const collection = new Collection({
name: "bookmarks",
type: "base",
fields: [
{ name: "title", type: "text", required: true },
{ name: "url", type: "url", required: true },
{ name: "note_id", type: "relation", options: { collectionId: "notes" } },
],
});
return db.saveCollection(collection);
},
async down(db) {
return db.deleteCollection("bookmarks");
},
};
```
### Running Migrations
Migrations run automatically when PocketBase starts. To run them manually:
```bash
pocketbase migrate --dir=./pocketbase/pb_migrations --dir=./pb_data
```
### Updating TypeScript Types
After changing the schema, update the TypeScript types in `pocketbase/schema.ts` to match. This keeps the type system in sync with the database.
### Rules for Schema Changes
1. **Always provide both `up` and `down`**: Migrations must be reversible
2. **Never modify existing migrations**: Create new ones instead
3. **Test migrations locally** before committing
4. **Update TypeScript types** in the same PR as the migration
5. **Update Zod schemas** in `packages/shared/` if the change affects validation
## Testing Strategy
### Unit Tests
Unit tests cover pure functions and business logic. They run with Jest.
```bash
npm run test
```
**Location:** `tests/` directory or alongside source files.
**What to test:**
- Zod schema validation
- Utility functions (date formatting, string manipulation)
- Service layer logic
- Store actions
**Example:**
```typescript
// tests/format-duration.test.ts
import { formatDuration } from "@/lib/utils";
describe("formatDuration", () => {
it("formats minutes to hours and minutes", () => {
expect(formatDuration(90)).toBe("1h 30m");
});
it("handles zero minutes", () => {
expect(formatDuration(0)).toBe("0m");
});
});
```
### Component Tests
Component tests verify React components render correctly and handle user interactions.
**Location:** `tests/` directory or alongside component files.
**What to test:**
- Components render with required props
- User interactions trigger correct callbacks
- Conditional rendering works as expected
### E2E Tests
E2E tests verify complete user flows using Playwright. They run against a real browser.
```bash
# Run all E2E tests
npm run test:e2e
# Run with UI mode (interactive debugging)
npm run test:e2e:ui
# View test report
npm run test:e2e:report
```
**Location:** `e2e/` directory.
**Browser configurations:**
- Chromium (Desktop)
- Firefox (Desktop)
- WebKit (Desktop Safari)
- Mobile Chrome (Pixel 5)
- Mobile Safari (iPhone 12)
**What to test:**
- Complete user flows (login → create task → complete task)
- Navigation between pages
- Form submissions
- Realtime updates
- Error states
**Example:**
```typescript
// e2e/tasks.spec.ts
import { test, expect } from "@playwright/test";
test.describe("Tasks", () => {
test("create and complete a task", async ({ page }) => {
await page.goto("/dashboard/tasks");
await page.click('[data-testid="create-task-button"]');
await page.fill('[data-testid="task-title"]', "New task");
await page.click('[data-testid="create-button"]');
await expect(page.locator('[data-testid="task-item"]')).toContainText("New task");
await page.click('[data-testid="task-checkbox"]');
await expect(page.locator('[data-testid="task-status"]')).toHaveText("done");
});
});
```
### Test Naming
- Unit tests: `describe("functionName", () => { ... })`
- Component tests: `describe("ComponentName", () => { ... })`
- E2E tests: `test.describe("Feature", () => { ... })`
### Running Tests in CI
CI runs all tests automatically on every PR:
```bash
npm run typecheck # TypeScript check
npm run lint # Linting
npm run test # Unit + component tests
npm run test:e2e # E2E tests
```
## Code Style and Conventions
### TypeScript
- Use strict mode (enabled in `tsconfig.json`)
- Prefer interfaces for object shapes, types for unions and utilities
- Use `unknown` instead of `any` for external data
- Add JSDoc comments for exported functions
### React
- Use functional components with hooks
- Mark client components with `"use client"` directive
- Keep components small and focused
- Extract reusable logic into custom hooks
- Use `React.memo` only when profiling shows a need
### Styling
- Use Tailwind CSS utility classes
- Use `cn()` from `lib/utils.ts` to merge class names
- Use `cva` for component variants
- Avoid inline styles unless dynamic values are required
### Error Handling
- Use `ApiError` and `AuthError` classes from `lib/auth.ts`
- Return consistent error responses: `{ error: { code, message, details? } }`
- Catch errors at the API route boundary
- Log errors with context (user ID, request path)
### Async/Await
- Use async/await instead of `.then()` chains
- Handle errors with try/catch
- Use `AbortSignal.timeout()` for fetch requests with timeouts
## Git Workflow
### Branch Naming
```
feature/description : New features
fix/description : Bug fixes
refactor/description : Code refactoring
docs/description : Documentation changes
test/description : Test additions
chore/description : Maintenance tasks
```
### Commit Messages
Use conventional commits:
```
feat: add bookmark support to notes
fix: resolve realtime SSE reconnection loop
refactor: extract task filtering into custom hook
docs: update API documentation for tasks endpoint
test: add E2E tests for habit logging flow
chore: upgrade Next.js to 15.3.0
```
### Commit Guidelines
- One logical change per commit
- Keep commits atomic and reversible
- Write the subject line in imperative mood ("add feature" not "added feature")
- Keep subject lines under 72 characters
- Add a body for complex changes explaining the "why"
### Before Pushing
1. Run `npm run typecheck`: Must pass
2. Run `npm run lint`: Must pass
3. Run `npm run test`: Must pass
4. Run `npm run test:e2e`: Must pass (for feature/fix branches)
5. Review your diff: `git diff --stat`
## PR Review Process
### Creating a PR
1. Push your branch to the remote
2. Open a PR against `main`
3. Fill in the PR template:
- What does this PR do?
- Why is this change needed?
- How was it tested?
- Screenshots (for UI changes)
### Review Checklist
Reviewers check:
- [ ] Code compiles without TypeScript errors
- [ ] Lint passes
- [ ] Tests pass (unit + E2E)
- [ ] Code follows project conventions
- [ ] No unnecessary dependencies added
- [ ] Error handling is complete
- [ ] UI is accessible (keyboard navigation, ARIA labels)
- [ ] Documentation updated (if API changed)
### Merging
- PRs require at least one approval
- All CI checks must pass
- Squash merge preferred for clean history
- Delete the branch after merging
## Common Tasks
### Adding a New API Endpoint
1. Create a directory under `apps/web/app/api/`:
```bash
mkdir apps/web/app/api/bookmarks
```
2. Create `route.ts`:
```typescript
import { NextRequest, NextResponse } from "next/server";
import { withAuth } from "@/lib/auth";
import { createPocketBaseClient } from "@/lib/pocketbase";
export const GET = withAuth(async (request: NextRequest, user) => {
const pb = createPocketBaseClient();
const result = await pb.collection("bookmarks").getList(1, 50);
return NextResponse.json(result);
});
export const POST = withAuth(async (request: NextRequest, user) => {
const body = await request.json();
const pb = createPocketBaseClient();
const bookmark = await pb.collection("bookmarks").create(body);
return NextResponse.json(bookmark, { status: 201 });
});
```
3. For dynamic routes, create `[id]/route.ts`:
```typescript
export const GET = withAuth(async (request: NextRequest, user, context: { params: { id: string } }) => {
const { id } = await context.params;
const pb = createPocketBaseClient();
const bookmark = await pb.collection("bookmarks").getOne(id);
return NextResponse.json(bookmark);
});
```
### Adding a New Zustand Store
```typescript
// apps/web/lib/stores/use-bookmarks-store.ts
import { create } from "zustand";
interface Bookmark {
id: string;
title: string;
url: string;
}
interface BookmarksState {
bookmarks: Bookmark[];
loading: boolean;
fetchBookmarks: () => Promise<void>;
addBookmark: (bookmark: Bookmark) => void;
}
export const useBookmarksStore = create<BookmarksState>((set) => ({
bookmarks: [],
loading: false,
fetchBookmarks: async () => {
set({ loading: true });
const response = await fetch("/api/bookmarks");
const data = await response.json();
set({ bookmarks: data.items, loading: false });
},
addBookmark: (bookmark) =>
set((state) => ({
bookmarks: [...state.bookmarks, bookmark],
})),
}));
```
### Adding a New MCP Tool
1. Add the tool to the appropriate file in `apps/web/lib/mcp/tools/`:
```typescript
// apps/web/lib/mcp/tools/bookmarks.ts
server.tool("create_bookmark", "Create a new bookmark", {
title: z.string(),
url: z.string().url(),
note_id: z.string().optional(),
}, async (args) => {
try {
const bookmark = await pb.collection("bookmarks").create({
title: args.title,
url: args.url,
note_id: args.note_id || "",
});
return textContent(JSON.stringify({ success: true, bookmark }));
} catch (error) {
return textContent(JSON.stringify({ success: false, error: String(error) }));
}
});
```
2. Register the tool in `apps/web/lib/mcp/server.ts`:
```typescript
import { registerBookmarkTools } from "./tools/bookmarks";
// ...
registerBookmarkTools(server);
```
### Adding a New Background Job Type
1. Add a case to the worker's `processJob` function in `worker/index.ts`:
```typescript
case "send_notification":
await handleSendNotification(job);
break;
```
2. Implement the handler:
```typescript
async function handleSendNotification(job: QueueJob): Promise<void> {
const payload = job.payload as { user_id: string; message: string };
const pb = createAdminClient();
await pb.collection("notifications").create({
user_id: payload.user_id,
message: payload.message,
type: "info",
read: false,
});
}
```
3. Schedule the job from your API route or service:
```typescript
await pb.collection("queue_jobs").create({
type: "send_notification",
queue: "default",
payload: { user_id: "user123", message: "Task completed" },
status: "pending",
retry_count: 0,
max_retries: 3,
scheduled_at: new Date().toISOString(),
});
```
+827
View File
@@ -0,0 +1,827 @@
# MCP Server Documentation
Project E includes a native Model Context Protocol (MCP) server that lets AI agents read and write data through a standardized interface. The server exposes 61 tools organized across 11 categories.
## Table of Contents
- [Overview](#overview)
- [Connection](#connection)
- [Authentication](#authentication)
- [Transport](#transport)
- [Tools](#tools)
- [Tasks (8 tools)](#tasks-8-tools)
- [Habits (7 tools)](#habits-7-tools)
- [Projects (6 tools)](#projects-6-tools)
- [Notes (6 tools)](#notes-6-tools)
- [Reports (5 tools)](#reports-5-tools)
- [Milestones (5 tools)](#milestones-5-tools)
- [Domains (5 tools)](#domains-5-tools)
- [Tags (5 tools)](#tags-5-tools)
- [Agents (5 tools)](#agents-5-tools)
- [Webhooks (5 tools)](#webhooks-5-tools)
- [Analytics (4 tools)](#analytics-4-tools)
- [Error Handling](#error-handling)
- [Examples](#examples)
## Overview
The MCP server runs inside the Next.js application at `/api/mcp`. It uses the Streamable HTTP transport from the `@modelcontextprotocol/sdk` package.
AI agents connect to the server using their API key, then call tools to interact with Project E data. Every tool call is authenticated and attributed to the calling agent.
## Connection
### Endpoint
```
GET /api/mcp : Establish a session
POST /api/mcp : Send tool calls
DELETE /api/mcp : End the session
```
### Connection Flow
1. Agent sends `GET /api/mcp` with `Authorization: Bearer <api_key>` header
2. Server validates the API key and returns a session ID in the `mcp-session-id` response header
3. Agent includes `mcp-session-id` in all subsequent `POST` requests
4. Agent sends `DELETE /api/mcp` with the session ID to close the connection
### Session Management
Sessions are stored in memory on the server. If the server restarts, active sessions are lost and agents must reconnect.
Sessions do not expire automatically. Send a `DELETE` request to clean up.
## Authentication
All MCP requests require an API key in the `Authorization` header:
```
Authorization: Bearer your_agent_api_key
```
API keys are generated when you create an agent through the web UI or the `create_agent` tool. Each agent has a unique key tied to its permission tier.
### Permission Tiers
| Tier | Access |
|------|--------|
| `full_access` | All tools |
| `read_only` | `get_*` and `list_*` tools only |
| `content_creator` | Read tools + `create_note`, `create_report` |
| `task_manager` | Read tools + all task and habit tools |
| `custom` | Defined by `custom_permissions` field |
The server validates the API key against the `agents` collection. Disabled agents cannot connect.
## Transport
The server uses **Streamable HTTP** transport. This is a request-response protocol over HTTP with server-sent events for streaming responses.
### Capabilities
The server advertises the `tools` capability. Agents can list available tools and call them by name.
### Message Format
Tool calls follow the MCP JSON-RPC format:
```json
{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {
"name": "create_task",
"arguments": {
"title": "New task",
"domain": "work"
}
}
}
```
Responses:
```json
{
"jsonrpc": "2.0",
"id": 1,
"result": {
"content": [
{
"type": "text",
"text": "{\"success\":true,\"task\":{...}}"
}
]
}
}
```
## Tools
All tools return a JSON object with a `success` boolean. On success, the response includes the created or fetched data. On failure, the response includes an `error` string.
### Tasks (8 tools)
| Tool | Description |
|------|-------------|
| `create_task` | Create a new task |
| `get_task` | Get a task by ID |
| `list_tasks` | List tasks with optional filters |
| `update_task` | Update an existing task |
| `delete_task` | Delete a task |
| `bulk_create_tasks` | Create multiple tasks at once |
| `bulk_update_tasks` | Update multiple tasks at once |
| `bulk_delete_tasks` | Delete multiple tasks at once |
#### `create_task`
Create a new task.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `title` | string | yes | Task title |
| `description` | string | no | Task description |
| `status` | enum | no | `todo`, `in_progress`, `done`, `cancelled` (default: `todo`) |
| `priority` | enum | no | `low`, `medium`, `high`, `urgent` (default: `medium`) |
| `due_date` | string | no | ISO 8601 date string |
| `project_id` | string | no | Related project ID |
| `milestone_id` | string | no | Related milestone ID |
| `tags` | string[] | no | Array of tag IDs |
| `domain` | string | yes | Domain ID |
| `assignee` | string | no | Assignee user ID |
| `estimate` | number | no | Time estimate in minutes |
**Example:**
```json
{
"name": "create_task",
"arguments": {
"title": "Write API documentation",
"description": "Document all REST endpoints",
"priority": "high",
"domain": "work_id",
"tags": ["tag_docs", "tag_api"],
"due_date": "2024-02-01T00:00:00.000Z"
}
}
```
#### `list_tasks`
List tasks with optional filters.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `status` | enum | no | Filter by status |
| `priority` | enum | no | Filter by priority |
| `project_id` | string | no | Filter by project |
| `milestone_id` | string | no | Filter by milestone |
| `domain` | string | no | Filter by domain |
| `limit` | number | no | Items per page (default: 20) |
| `offset` | number | no | Offset for pagination |
**Response:**
```json
{
"success": true,
"tasks": [...],
"total": 42,
"page": 1,
"limit": 20
}
```
#### `bulk_create_tasks`
Create multiple tasks in one call.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `tasks` | object[] | yes | Array of task objects |
Each task object accepts the same fields as `create_task` (minus `domain` being required per task).
**Response:**
```json
{
"success": true,
"created": [...],
"count": 5
}
```
### Habits (7 tools)
| Tool | Description |
|------|-------------|
| `create_habit` | Create a new habit |
| `get_habit` | Get a habit by ID |
| `list_habits` | List habits with optional filters |
| `update_habit` | Update an existing habit |
| `delete_habit` | Delete a habit |
| `log_habit_completion` | Log a habit completion |
| `get_habit_streaks` | Get streak information for all active habits |
#### `create_habit`
Create a new habit.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `name` | string | yes | Habit name |
| `description` | string | no | Habit description |
| `domain` | string | yes | Domain ID |
| `frequency` | enum | no | `daily`, `weekly`, `custom` (default: `daily`) |
| `difficulty` | enum | no | `easy`, `medium`, `hard` (default: `medium`) |
| `goal_per_period` | number | no | Target completions per period (default: 1) |
| `tags` | string[] | no | Array of tag IDs |
#### `log_habit_completion`
Log a habit completion and update streaks.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `habit_id` | string | yes | Habit ID |
| `completed` | boolean | no | Whether completed (default: true) |
| `notes` | string | no | Completion notes |
| `value` | number | no | Quantity value |
| `mood` | number | no | Mood rating (1-5) |
| `logged_at` | string | no | ISO 8601 timestamp (default: now) |
**Response:**
```json
{
"success": true,
"log": {...},
"habit_id": "habit_id",
"current_streak": 7
}
```
#### `get_habit_streaks`
Get streak information for all active habits. No arguments required.
**Response:**
```json
{
"success": true,
"streaks": [
{
"habit_id": "id",
"name": "Meditation",
"current_streak": 14,
"best_streak": 21,
"total_completions": 45,
"score": 85
}
]
}
```
### Projects (6 tools)
| Tool | Description |
|------|-------------|
| `create_project` | Create a new project |
| `get_project` | Get a project by ID |
| `list_projects` | List projects with optional filters |
| `update_project` | Update an existing project |
| `delete_project` | Delete a project |
| `get_project_progress` | Get project progress based on task completion |
#### `create_project`
Create a new project.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `name` | string | yes | Project name |
| `description` | string | no | Project description |
| `status` | enum | no | `active`, `paused`, `archived` (default: `active`) |
| `domain` | string | yes | Domain ID |
| `color` | string | no | Hex color code |
| `icon` | string | no | Icon name |
| `tags` | string[] | no | Array of tag IDs |
| `owner` | string | no | Owner user ID |
| `start_date` | string | no | ISO 8601 date |
| `target_date` | string | no | ISO 8601 date |
#### `get_project_progress`
Calculate project progress based on task completion.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `project_id` | string | yes | Project ID |
**Response:**
```json
{
"success": true,
"project_id": "project_id",
"total_tasks": 20,
"completed_tasks": 15,
"progress": 75
}
```
### Notes (6 tools)
| Tool | Description |
|------|-------------|
| `create_note` | Create a new note |
| `get_note` | Get a note by ID |
| `list_notes` | List notes with optional filters |
| `update_note` | Update an existing note |
| `delete_note` | Delete a note |
| `get_note_graph` | Get the note graph showing connections between notes |
#### `create_note`
Create a new note. Word count is calculated automatically from content.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `title` | string | yes | Note title |
| `content` | string | no | Note content (markdown) |
| `domain` | string | yes | Domain ID |
| `tags` | string[] | no | Array of tag IDs |
| `project_id` | string | no | Related project ID |
| `is_pinned` | boolean | no | Pin the note (default: false) |
#### `get_note_graph`
Get the full note graph with all notes and their links. No arguments required.
**Response:**
```json
{
"success": true,
"graph": {
"nodes": [
{ "id": "note_id", "title": "Note Title", "domain": "domain_id" }
],
"edges": [
{ "source": "note_1", "target": "note_2", "label": "related to" }
]
}
}
```
### Reports (5 tools)
| Tool | Description |
|------|-------------|
| `create_report` | Create a new report |
| `get_report` | Get a report by ID |
| `list_reports` | List reports with optional filters |
| `update_report` | Update an existing report |
| `delete_report` | Delete a report |
#### `create_report`
Create a new report.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `title` | string | yes | Report title |
| `type` | enum | yes | `weekly`, `monthly`, `project`, `habit`, `custom` |
| `domain` | string | yes | Domain ID |
| `date_range_start` | string | yes | ISO 8601 date |
| `date_range_end` | string | yes | ISO 8601 date |
| `summary` | string | no | Report summary |
| `tags` | string[] | no | Array of tag IDs |
| `is_draft` | boolean | no | Save as draft (default: true) |
### Milestones (5 tools)
| Tool | Description |
|------|-------------|
| `create_milestone` | Create a new milestone |
| `get_milestone` | Get a milestone by ID |
| `list_milestones` | List milestones with optional filters |
| `update_milestone` | Update an existing milestone |
| `delete_milestone` | Delete a milestone |
#### `create_milestone`
Create a new milestone.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `name` | string | yes | Milestone name |
| `description` | string | no | Milestone description |
| `project_id` | string | yes | Parent project ID |
| `domain` | string | yes | Domain ID |
| `status` | enum | no | `planned`, `in_progress`, `complete` (default: `planned`) |
| `target_date` | string | no | ISO 8601 date |
| `sort_order` | number | no | Display order (default: 0) |
| `tags` | string[] | no | Array of tag IDs |
#### `update_milestone`
When status is set to `complete`, the server automatically sets `completed_at` to the current timestamp.
### Domains (5 tools)
| Tool | Description |
|------|-------------|
| `create_domain` | Create a new domain |
| `get_domain` | Get a domain by ID |
| `list_domains` | List all domains |
| `update_domain` | Update an existing domain |
| `delete_domain` | Delete a domain |
#### `create_domain`
Create a new domain.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `name` | string | yes | Domain name |
| `color` | string | no | Hex color code |
| `icon` | string | no | Icon name |
| `sort_order` | number | no | Display order (default: 0) |
### Tags (5 tools)
| Tool | Description |
|------|-------------|
| `create_tag` | Create a new tag |
| `get_tag` | Get a tag by ID |
| `list_tags` | List all tags |
| `update_tag` | Update an existing tag |
| `delete_tag` | Delete a tag |
#### `create_tag`
Create a new tag.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `name` | string | yes | Tag name |
| `color` | string | no | Hex color code |
### Agents (5 tools)
| Tool | Description |
|------|-------------|
| `create_agent` | Create a new agent |
| `get_agent` | Get an agent by ID |
| `list_agents` | List agents with optional filters |
| `update_agent` | Update an existing agent |
| `delete_agent` | Delete an agent |
#### `create_agent`
Create a new agent. The server generates an API key automatically.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `name` | string | yes | Agent name |
| `description` | string | no | Agent description |
| `domain` | string | yes | Domain ID |
| `status` | enum | no | `active`, `disabled` (default: `active`) |
| `permission_tier` | enum | no | `full_access`, `read_only`, `content_creator`, `task_manager`, `custom` (default: `read_only`) |
| `tags` | string[] | no | Array of tag IDs |
**Response:**
```json
{
"success": true,
"agent": {
"id": "agent_id",
"name": "Code Assistant",
"api_key": "generated-uuid-key",
"permission_tier": "read_only",
"status": "active"
}
}
```
Save the `api_key` from the response. It is not shown again.
### Webhooks (5 tools)
| Tool | Description |
|------|-------------|
| `create_webhook` | Create a new webhook |
| `get_webhook` | Get a webhook by ID |
| `list_webhooks` | List webhooks with optional filters |
| `update_webhook` | Update an existing webhook |
| `delete_webhook` | Delete a webhook |
#### `create_webhook`
Create a new webhook subscription.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `name` | string | yes | Webhook name |
| `url` | string | yes | Delivery URL |
| `events` | string[] | yes | Event types to subscribe to |
| `domain` | string | yes | Domain ID |
| `secret` | string | no | HMAC secret for payload signing |
| `active` | boolean | no | Enable webhook (default: true) |
| `retry_count` | number | no | Max retry attempts (default: 3) |
### Analytics (4 tools)
| Tool | Description |
|------|-------------|
| `get_analytics` | Get analytics data for a given period |
| `get_time_summary` | Get aggregated time tracking summary |
| `search` | Search across tasks, habits, projects, notes, and reports |
| `get_agent_activity` | Get recent agent activity |
#### `get_analytics`
Get analytics data for a given period.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `period_days` | number | no | Number of days to analyze (default: 30) |
**Response:**
```json
{
"success": true,
"analytics": {
"taskCompletionRate": 75,
"habitConsistency": 82,
"totalTimeMinutes": 1240,
"activeStreaks": 5,
"bestStreak": 21,
"period": 30
}
}
```
#### `get_time_summary`
Get aggregated time tracking data broken down by domain, project, and tag.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `start_date` | string | no | Start date (default: 30 days ago) |
| `end_date` | string | no | End date (default: now) |
**Response:**
```json
{
"success": true,
"time_summary": {
"totalMinutes": 1240,
"byDomain": { "work": 800, "personal": 440 },
"byProject": { "project_id_1": 600 },
"byTag": { "frontend": 300 },
"startDate": "2024-01-01T00:00:00.000Z",
"endDate": "2024-01-31T00:00:00.000Z"
}
}
```
#### `search`
Full-text search across multiple entity types.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `query` | string | yes | Search query |
| `types` | string[] | no | Entity types to search (default: all) |
| `limit` | number | no | Max results per type (default: 10) |
Supported types: `tasks`, `habits`, `projects`, `notes`, `reports`.
**Response:**
```json
{
"success": true,
"results": [
{ "type": "tasks", "items": [...] },
{ "type": "notes", "items": [...] }
]
}
```
#### `get_agent_activity`
Get recent agent activity logs.
**Arguments:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `limit` | number | no | Items per page (default: 20) |
| `offset` | number | no | Offset for pagination |
## Error Handling
All tools catch errors and return them in a consistent format:
**Success:**
```json
{
"content": [
{
"type": "text",
"text": "{\"success\":true,\"task\":{...}}"
}
]
}
```
**Failure:**
```json
{
"content": [
{
"type": "text",
"text": "{\"success\":false,\"error\":\"ClientResponseError: The requested resource wasn't found.\"}"
}
]
}
```
Tools never throw exceptions. Errors are returned as text content with `success: false`.
### Common Errors
| Error | Cause |
|-------|-------|
| `Unauthorized` | Missing or invalid API key |
| `Session not found` | Session expired or never created |
| `The requested resource wasn't found` | Invalid record ID |
| `Failed to parse request body` | Invalid JSON in tool arguments |
## Examples
### Python Client
```python
import requests
API_URL = "http://localhost:3000/api/mcp"
API_KEY = "your-agent-api-key"
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
}
# Step 1: Establish session
response = requests.get(API_URL, headers=headers)
session_id = response.headers.get("mcp-session-id")
headers["mcp-session-id"] = session_id
# Step 2: List available tools
list_tools = {
"jsonrpc": "2.0",
"id": 1,
"method": "tools/list",
}
response = requests.post(API_URL, headers=headers, json=list_tools)
tools = response.json()
# Step 3: Create a task
create_task = {
"jsonrpc": "2.0",
"id": 2,
"method": "tools/call",
"params": {
"name": "create_task",
"arguments": {
"title": "Review pull request",
"domain": "work_domain_id",
"priority": "high",
},
},
}
response = requests.post(API_URL, headers=headers, json=create_task)
result = response.json()
# Step 4: Close session
requests.delete(API_URL, headers=headers)
```
### JavaScript Client (using MCP SDK)
```typescript
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js";
const API_URL = "http://localhost:3000/api/mcp";
const API_KEY = "your-agent-api-key";
const transport = new StreamableHTTPClientTransport(
new URL(API_URL),
{ requestInit: { headers: { Authorization: `Bearer ${API_KEY}` } } }
);
const client = new Client({ name: "my-agent", version: "1.0.0" });
await client.connect(transport);
// List tools
const { tools } = await client.listTools();
console.log(`Available tools: ${tools.length}`);
// Create a task
const result = await client.callTool({
name: "create_task",
arguments: {
title: "Review pull request",
domain: "work_domain_id",
priority: "high",
},
});
console.log(result);
// Close connection
await client.close();
```
### cURL
```bash
# Establish session
curl -X GET http://localhost:3000/api/mcp \
-H "Authorization: Bearer your-api-key" \
-D headers.txt
# Extract session ID from headers
SESSION_ID=$(grep -i 'mcp-session-id' headers.txt | awk '{print $2}' | tr -d '\r')
# Create a task
curl -X POST http://localhost:3000/api/mcp \
-H "Authorization: Bearer your-api-key" \
-H "mcp-session-id: $SESSION_ID" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {
"name": "create_task",
"arguments": {
"title": "New task",
"domain": "work_domain_id"
}
}
}'
# Close session
curl -X DELETE http://localhost:3000/api/mcp \
-H "Authorization: Bearer your-api-key" \
-H "mcp-session-id: $SESSION_ID"
```