BuzzbeeSCD/ProjectE
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

feat: migrate from PocketBase to PostgreSQL with Drizzle ORM

Browse Source 
- Add @project-e/db package with Drizzle schema and migrations
- Replace PocketBase client with PostgreSQL-based database client
- Migrate auth from custom to NextAuth.js
- Add Docker Compose with PostgreSQL container
- Update worker to use new database client
- Remove PocketBase-specific files and migrations
- Add drizzle config and initial migration
This commit is contained in:
mbatchelder
2026-07-24 07:08:29 -04:00
parent 6c438eab32
commit 73335484f8
42 changed files with 2895 additions and 2796 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/DEPLOYMENT.md
+66 -547
View File
@@ -1,404 +1,126 @@
# Deployment Guide
Deploy Project E with Docker Compose and Nginx Proxy Manager. Three containers run the application: web (Next.js), db (PocketBase), and worker (background jobs).
## Table of Contents
- [Prerequisites](#prerequisites)
- [Quick Deploy](#quick-deploy)
- [Environment Configuration](#environment-configuration)
- [Nginx Proxy Manager Setup](#nginx-proxy-manager-setup)
- [PocketBase Setup](#pocketbase-setup)
- [Backups](#backups)
- [Monitoring](#monitoring)
- [Scaling](#scaling)
- [Troubleshooting](#troubleshooting)
Deploy Project E with Docker Compose and a reverse proxy such as Nginx Proxy Manager. Compose runs three services: the Next.js web app, PostgreSQL 16, and the background worker.
## Prerequisites
- **Docker** 24.0 or later
- **Docker Compose** 2.20 or later
- **Nginx Proxy Manager** installed and running
- **At least 1GB RAM** and 10GB disk space
- Docker 24.0 or later
- Docker Compose 2.20 or later
- A reverse proxy for TLS and public routing
- At least 1 GB RAM and 10 GB disk space
## Quick Deploy
## Deploy Project E
1. **Clone the repository on your server**
1. Clone the repository on the server.
```bash
git clone <repository-url>
cd ProjectE
```
2. **Create the environment file**
2. Create the environment file.
```bash
cp .env.example .env
```
Edit `.env` and set the required variables:
3. Set the required values in `.env`.
```bash
POCKETBASE_ADMIN_TOKEN=your-secure-random-token
PUBLIC_URL=http://project-e.local
COOKIE_SECURE=false
ALLOWED_HOSTS=project-e.local,localhost
POSTGRES_PASSWORD=your_postgres_password
DATABASE_URL=postgresql://project_e:your_postgres_password@localhost:5432/project_e
NEXTAUTH_SECRET=your_long_random_secret
INITIAL_ADMIN_EMAIL=admin@example.com
INITIAL_ADMIN_PASSWORD=your_initial_admin_password
PUBLIC_URL=https://project-e.example.com
COOKIE_SECURE=true
ALLOWED_HOSTS=project-e.example.com
```
Generate a secure token:
Generate secrets with `openssl rand -base64 32`. Use the same `POSTGRES_PASSWORD` in `DATABASE_URL`. The web and worker containers use an internal database URL that Compose builds from `POSTGRES_PASSWORD`.
```bash
openssl rand -hex 32
```
3. **Build and start containers**
4. Build and start the services.
```bash
docker compose up -d
```
4. **Verify all containers are running**
5. Confirm that the services are healthy.
```bash
docker compose ps
```
You should see three containers with status `Up (healthy)`:
- `project-e-web` (internal only, no exposed ports)
- `project-e-db` (internal only, no exposed ports)
- `project-e-worker` running in the background
6. Route your domain to `project-e-web:3000` through your reverse proxy.
5. **Configure Nginx Proxy Manager** (see below)
7. Open the app and sign in with `INITIAL_ADMIN_EMAIL` and `INITIAL_ADMIN_PASSWORD`. Project E creates that account only when the database contains no users.
6. **Create your admin account**
## Environment variables
Once NPM is configured, access Project E through your domain. Create your first user account.
| Variable | Required | Description |
|----------|----------|-------------|
| `DATABASE_URL` | Yes | PostgreSQL connection string for local tools and processes outside Compose. |
| `POSTGRES_PASSWORD` | Yes | Password for the `project_e` PostgreSQL user. |
| `NEXTAUTH_SECRET` | Yes | Secret used to sign NextAuth JWT sessions. |
| `INITIAL_ADMIN_EMAIL` | Yes | Email address for the first account. |
| `INITIAL_ADMIN_PASSWORD` | Yes | Password for the first account. |
| `PUBLIC_URL` | No | Public application URL. Defaults to `http://localhost:3000`. |
| `COOKIE_SECURE` | No | Set to `true` behind HTTPS. Defaults to `false`. |
| `ALLOWED_HOSTS` | No | Comma-separated allowed hostnames. |
## Environment Configuration
Keep `.env` out of version control. Rotate `NEXTAUTH_SECRET` only when you intend to end active sessions.
### Required Variables
## PostgreSQL and Drizzle
| Variable | Description |
|----------|-------------|
| `POCKETBASE_ADMIN_TOKEN` | Admin token from PocketBase. Required for the worker and server-side API operations. |
The `db` service runs `postgres:16-alpine`, stores its data in the `project-e-pg-data` Docker volume, and exposes port 5432. The web and worker services connect to `db:5432` over the Compose network.
### Optional Variables
Docker mounts `drizzle/0000_first_mauler.sql` into PostgreSQL's initialization directory. PostgreSQL runs that file only while it initializes an empty data volume. For a later Drizzle migration, apply the reviewed SQL as part of your release process. Do not expect a container restart to apply a new migration to an existing volume.
| Variable | Default | Description |
|----------|---------|-------------|
| `POCKETBASE_URL` | `http://db:8090` | PocketBase URL (internal Docker network) |
| `NODE_ENV` | `production` | Node environment |
| `PUBLIC_URL` | `http://localhost:3000` | Public URL where users access Project E (used for generating absolute URLs) |
| `COOKIE_SECURE` | `false` | Set to `true` if using HTTPS through NPM, `false` for HTTP-only LAN access |
| `ALLOWED_HOSTS` | `localhost` | Comma-separated list of domains that can access the app |
### Setting Variables
Create a `.env` file in the project root:
To apply a migration from the host, run:
```bash
POCKETBASE_ADMIN_TOKEN=abc123def456...
POCKETBASE_URL=http://db:8090
PUBLIC_URL=http://project-e.local
COOKIE_SECURE=false
ALLOWED_HOSTS=project-e.local,localhost
docker compose exec -T db psql -U project_e -d project_e < drizzle/<migration>.sql
```
Docker Compose reads this file automatically.
Back up the database before applying schema changes.
## Nginx Proxy Manager Setup
## Nginx Proxy Manager
### Add Proxy Host
1. Open Nginx Proxy Manager admin interface
2. Go to **Hosts** → **Proxy Hosts** → **Add Proxy Host**
3. Configure the following:
**Details Tab:**
- **Domain Names:** `project-e.local` (or your chosen domain)
- **Scheme:** `http`
- **Forward Hostname/IP:** `project-e-web` (the Docker container name)
- **Forward Port:** `3000`
**SSL Tab:**
- If using HTTPS: Select your SSL certificate
- If HTTP-only on LAN: Leave SSL disabled
**Advanced Tab:**
Add these custom Nginx configuration lines:
1. In Nginx Proxy Manager, open **Hosts** > **Proxy Hosts** > **Add Proxy Host**.
2. Set the domain name, choose `http`, set the forward hostname to `project-e-web`, and set the forward port to `3000`.
3. Select a certificate and force SSL for HTTPS deployments.
4. Add the following advanced configuration to support Server-Sent Events:
```nginx
# WebSocket support for realtime features
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# Forward real client information
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
# SSE support for realtime endpoint
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 300s;
# Increase timeout for long-running requests
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
```
4. Click **Save**
5. Save the host and open the configured domain. You should reach the Project E sign-in page.
### Testing the Connection
## Back up and restore PostgreSQL
1. From a device on your LAN, open a browser
2. Navigate to `http://project-e.local` (or your configured domain)
3. You should see the Project E login page
### Optional: PocketBase Admin Access
If you need direct access to the PocketBase admin UI:
1. In NPM, add another Proxy Host:
- **Domain Names:** `pb.project-e.local`
- **Scheme:** `http`
- **Forward Hostname/IP:** `project-e-db`
- **Forward Port:** `8090`
2. Or temporarily expose the port in `docker-compose.yml`:
```yaml
db:
ports:
- "8090:8090"
```
## PocketBase Setup
### Initial Configuration
PocketBase runs as a standalone container. After starting it for the first time:
1. Access the admin UI at `http://your-server:8090/_/`
2. Create your admin account
3. Configure authentication settings under **Settings > Auth**
4. Enable the auth methods you need (email/password is enabled by default)
### Migrations
Database migrations are in `pocketbase/pb_migrations/`. They run automatically when the PocketBase container starts.
To add a new migration:
1. Create a file in `pocketbase/pb_migrations/` with the naming convention `YYYYMMDDHHMMSS_description.js`
2. Restart the PocketBase container: `docker compose restart db`
### Data Directory
PocketBase stores all data (SQLite database, uploads, logs) in the `/pb_data` volume. This volume persists across container restarts.
## Reverse Proxy
Put a reverse proxy in front of the application to handle SSL, compression, and routing.
### Caddy (Recommended)
Caddy handles SSL automatically.
Create a `Caddyfile`:
```
your-domain.com {
reverse_proxy localhost:3000
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains"
X-Frame-Options "DENY"
X-Content-Type-Options "nosniff"
}
}
pb.your-domain.com {
reverse_proxy localhost:8090
}
```
Install and run Caddy:
Create a logical backup without stopping the database:
```bash
# Install Caddy
sudo apt install caddy # Debian/Ubuntu
# or
brew install caddy # macOS
# Start Caddy
caddy start
mkdir -p backups
```
### Traefik
Create a `traefik.yml`:
```yaml
entryPoints:
web:
address: ":80"
http:
redirections:
entryPoint:
to: websecure
scheme: https
websecure:
address: ":443"
certificatesResolvers:
letsencrypt:
acme:
email: your-email@example.com
storage: acme.json
httpChallenge:
entryPoint: web
providers:
docker:
exposedByDefault: false
```
Update `docker-compose.yml` to add Traefik labels:
```yaml
services:
web:
labels:
- "traefik.enable=true"
- "traefik.http.routers.web.rule=Host(`your-domain.com`)"
- "traefik.http.routers.web.entrypoints=websecure"
- "traefik.http.routers.web.tls.certresolver=letsencrypt"
```
### Nginx
Create `/etc/nginx/sites-available/project-e`:
```nginx
server {
listen 80;
server_name your-domain.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name your-domain.com;
ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;
location / {
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
# SSE support for realtime endpoint
proxy_buffering off;
proxy_read_timeout 300s;
}
}
```
Enable the site and reload:
Restore a backup into a new or empty database:
```bash
sudo ln -s /etc/nginx/sites-available/project-e /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
docker compose exec -T db psql -U project_e -d project_e < backups/project-e-YYYYMMDD_HHMMSS.sql
```
## SSL/TLS
### Let's Encrypt with Certbot
```bash
# Install certbot
sudo apt install certbot python3-certbot-nginx
# Get certificate
sudo certbot --nginx -d your-domain.com -d pb.your-domain.com
# Auto-renewal is configured automatically
```
### Caddy
Caddy obtains and renews certificates automatically. No configuration needed beyond the domain name in the `Caddyfile`.
### Internal Communication
The web and worker containers connect to PocketBase over the internal Docker network (`http://db:8090`). This traffic does not need SSL.
Only expose ports 3000 and 8090 to the reverse proxy, not directly to the internet.
## Backups
### PocketBase Database
The database is a single SQLite file at `/pb_data/data.db`.
**Manual backup:**
```bash
# Stop the container to ensure consistency
docker compose stop db
# Copy the database file
docker cp project-e-db:/pb_data/data.db ./backups/data-$(date +%Y%m%d).db
# Restart the container
docker compose start db
```
**Automated backup script:**
Create `scripts/backup.sh`:
```bash
#!/bin/bash
BACKUP_DIR="/path/to/backups"
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p $BACKUP_DIR
# Use PocketBase's backup API (no downtime)
curl -X POST http://localhost:8090/api/backup \
-H "Authorization: Admin your-admin-token" \
-o "$BACKUP_DIR/backup-$DATE.zip"
# Keep only last 30 backups
find $BACKUP_DIR -name "backup-*.zip" -mtime +30 -delete
echo "Backup completed: backup-$DATE.zip"
```
Schedule with cron:
```bash
# Run daily at 2 AM
0 2 * * * /path/to/scripts/backup.sh
```
### Upload Files
Uploaded files are stored in the `project-e-web-uploads` volume.
**Backup uploads:**
Back up the `project-e-web-uploads` volume if your deployment stores uploads there:
```bash
docker run --rm \
@@ -408,232 +130,29 @@ docker run --rm \
tar czf /backup/uploads-$(date +%Y%m%d).tar.gz -C /source .
```
### Restore
Test restores on a separate environment before relying on a backup.
**Restore database:**
## Monitoring and logs
```bash
# Stop containers
docker compose stop db
# Remove old data
docker volume rm project-e-pb-data
# Copy backup into new volume
docker volume create project-e-pb-data
docker run --rm \
-v project-e-pb-data:/pb_data \
-v $(pwd)/backups:/backup \
alpine \
sh -c "cp /backup/data-YYYYMMDD.db /pb_data/data.db"
# Restart
docker compose start db
```
**Restore uploads:**
```bash
docker run --rm \
-v project-e-web-uploads:/target \
-v $(pwd)/backups:/backup \
alpine \
sh -c "cd /target && tar xzf /backup/uploads-YYYYMMDD.tar.gz"
```
## Monitoring
### Container Health
All containers include health checks. Check status:
Check service status and logs:
```bash
docker compose ps
```
### Application Health
The web container exposes a health endpoint:
```bash
curl http://localhost:3000/api/health
```
**Response:**
```json
{
"status": "ok",
"timestamp": "2024-01-15T10:30:00.000Z",
"version": "0.1.0"
}
```
### Logs
View logs from all containers:
```bash
# All containers
docker compose logs -f
# Specific container
docker compose logs -f web
docker compose logs -f db
docker compose logs -f worker
```
### Resource Usage
Monitor container resource usage:
```bash
docker stats
```
### Uptime Monitoring
Use an external service to monitor your deployment:
- **UptimeRobot** (free tier): HTTP monitoring with email/SMS alerts
- **Healthchecks.io**: Cron job monitoring
- **Better Stack**: Status pages and incident management
Set up a check for `https://your-domain.com/api/health` with a 60-second interval.
## Scaling
### Vertical Scaling
The application runs as a single instance of each container. To handle more load:
1. **Increase server resources**: Add more CPU and RAM to your host
2. **Increase Node.js memory**: Set `NODE_OPTIONS=--max-old-space-size=4096` in the web container
3. **Increase PocketBase limits**: PocketBase handles thousands of concurrent connections on modest hardware
### Horizontal Scaling
Horizontal scaling requires changes to the architecture:
**Current limitations:**
- MCP sessions are stored in memory (not shared between instances)
- SSE connections are tied to a specific container
- File uploads go to a local volume
**To scale horizontally:**
1. Use a shared session store (Redis)
2. Use a load balancer with sticky sessions for SSE
3. Use object storage (S3) for file uploads
4. Run multiple web containers behind a load balancer
For most personal and small-team use cases, a single instance handles the load. PocketBase with SQLite performs well up to hundreds of concurrent users.
### Worker Scaling
The worker uses polling with exponential backoff. For high-throughput job processing:
1. Run multiple worker containers (they coordinate through the database)
2. Reduce the base poll interval (currently 5 seconds)
3. Use a dedicated job queue (Bull, BullMQ) instead of database polling
The web service exposes `GET /api/health` on port 3000. Configure uptime monitoring for `https://your-domain.example/api/health`.
## Troubleshooting
### Container Won't Start
| Problem | Fix |
|---------|-----|
| Database connection fails | Confirm that `db` is healthy and that `POSTGRES_PASSWORD` matches the value in `DATABASE_URL`. |
| The web service will not start | Set `NEXTAUTH_SECRET`, `INITIAL_ADMIN_EMAIL`, and `INITIAL_ADMIN_PASSWORD` in `.env`, then restart the service. |
| Sign-in fails | Confirm the email and password match the initial-admin values. Those values create an account only before any user exists. |
| A schema change is missing | Apply the generated Drizzle SQL. PostgreSQL initialization scripts do not rerun for an existing volume. |
| Realtime updates stop | Disable proxy buffering and set a read timeout of at least 300 seconds. |
| Disk space runs low | Inspect `project-e-pg-data` and `project-e-web-uploads`, back up data, and expand the host disk. |
**Check logs:**
## Scaling
```bash
docker compose logs web
docker compose logs db
docker compose logs worker
```
**Common issues:**
| Problem | Solution |
|---------|----------|
| `Cannot connect to PocketBase` | Ensure the `db` container is healthy. Check `docker compose ps`. |
| `Port 3000 already in use` | Change the port mapping in `docker-compose.yml` |
| `POCKETBASE_ADMIN_TOKEN not set` | Set the token in `.env` and restart |
| `Migration failed` | Check migration files in `pocketbase/pb_migrations/` |
### Web App Returns 500 Errors
1. Check the web container logs: `docker compose logs web`
2. Verify PocketBase is running: `docker compose exec db wget -qO- http://localhost:8090/api/health`
3. Verify the admin token is correct: `docker compose exec web env | grep POCKETBASE`
### Realtime Events Not Arriving
1. Check the SSE connection: `curl -N http://localhost:3000/api/realtime`
2. Verify the reverse proxy is not buffering SSE responses
3. For Nginx, ensure `proxy_buffering off` is set
4. Check browser console for connection errors
### Worker Not Processing Jobs
1. Check worker logs: `docker compose logs worker`
2. Verify the admin token is set correctly
3. Check for pending jobs in PocketBase admin: `http://localhost:8090/_/#/collections/queue_jobs`
4. Jobs retry automatically with exponential backoff (max 5 minutes between retries)
### Database Corruption
If the SQLite database becomes corrupted:
1. Stop all containers: `docker compose down`
2. Restore from the most recent backup (see [Backups](#backups))
3. If no backup exists, try SQLite's recovery:
```bash
sqlite3 data.db ".recover" | sqlite3 new-data.db
```
4. Replace the corrupted file and restart
### Out of Disk Space
PocketBase stores the database and uploads in the `project-e-pb-data` volume.
**Check disk usage:**
```bash
docker system df
docker volume inspect project-e-pb-data
```
**Clean up:**
```bash
# Remove unused images
docker image prune -a
# Remove unused volumes (WARNING: deletes all data)
docker volume prune
```
**Expand volume:**
Docker volumes use the host filesystem. If the host disk is full, expand it or move the volume to a larger disk.
### SSL Certificate Errors
**Caddy:** Check the Caddy logs for ACME errors. Ensure port 80 is accessible from the internet for the HTTP challenge.
**Let's Encrypt:** Certificates renew automatically. Force renewal:
```bash
sudo certbot renew --force-renewal
```
**Common errors:**
- `Connection refused`: Port 80 is blocked by firewall
- `DNS problem`: Domain does not resolve to this server
- `Rate limit exceeded`: Too many certificate requests. Wait and retry.
### Performance Issues
1. **Slow page loads**: Check if the server has enough RAM. Node.js needs at least 512MB.
2. **Slow API responses**: Check PocketBase query performance in the admin panel
3. **SSE disconnects**: Ensure the reverse proxy has appropriate timeout settings (300s+)
4. **Worker falling behind**: Increase the poll frequency or add more worker instances
The default deployment runs one web service, one worker, and one PostgreSQL instance. Add CPU and memory before changing the topology. Multiple web services require shared session-aware infrastructure and a reverse proxy that supports long-lived SSE connections. Multiple workers coordinate through the PostgreSQL-backed job records.