feat(skills): curated GitHub catalog redesign (#3016)
* feat(skills): remove ClawHub catalog integration Drop the ClawHub registry as a skills catalog source across web server, shared UI, VS Code, docs, and locales. The catalog now serves git-based sources only: the curated Anthropic repo and user-defined repositories. Also removes the now-unused adm-zip dependency. * feat(skills): redesign catalog around curated GitHub repositories Replace the single-source dropdown with a card grid of curated GitHub repositories (Anthropic, OpenAI, Cursor pstack/skills, Matt Pocock) plus user-defined sources. Source cards show skill counts, GitHub stars, and last-updated time; a global search covers all loaded sources. Server: curated sources gain GitHub repo metadata (stars, pushed_at) fetched best-effort with a 3-hour in-memory and on-disk cache; scans run through a concurrency-limited, deduplicated cache with 3-hour TTL persisted across restarts. Refresh still bypasses the cache. Shared UI: source cards, global search with clear button, per-skill GitHub links, install/installed states. VS Code curated list updated to match. All new copy translated across 12 locales. * fix(skills): address catalog review findings - GitHub metadata fetch timeout drops to 1.5s (under the catalog client's 3s deadline) and failed lookups cache briefly (5 min) so repeated catalog loads do not re-hit a failing API. - Disk cache files are written with owner-only permissions (0o600); rename preserves the mode. - loadSource deduplicates concurrent in-flight requests per source and the shared isLoadingSource flag now clears only when the last active source load finishes.
This commit is contained in:
Bohdan Triapitsyn
committed by
GitHub
GitHub
parent
90d8868bfc
commit
1ed3f1f575
@@ -1,21 +1,17 @@
|
||||
# Skills Catalog Module Documentation
|
||||
|
||||
## Purpose
|
||||
This module provides skill discovery, scanning, and installation capabilities for OpenCode. It supports multiple skill sources including git repositories and the ClawHub registry, with caching and conflict resolution for skill installation.
|
||||
This module provides skill discovery, scanning, and installation capabilities for OpenCode. It supports skill sources backed by git repositories, with caching and conflict resolution for skill installation.
|
||||
|
||||
## Entrypoints and structure
|
||||
- `packages/web/server/lib/skills-catalog/`: Skills catalog module directory containing all skill-related functionality.
|
||||
- `cache.js`: In-memory cache for scan results with TTL support.
|
||||
- `curated-sources.js`: Predefined skill sources (Anthropic, ClawHub).
|
||||
- `curated-sources.js`: Predefined skill sources (Anthropic, OpenAI, Cursor, Matt Pocock).
|
||||
- `github-meta.js`: Best-effort GitHub repository metadata (stars, last push) with in-memory TTL cache.
|
||||
- `git.js`: Git operations helpers for cloning and auth error detection.
|
||||
- `install.js`: Skills installation from git repositories.
|
||||
- `scan.js`: Skills scanning from git repositories.
|
||||
- `source.js`: Source string parsing for git repositories.
|
||||
- `clawdhub/`: ClawHub registry integration.
|
||||
- `index.js`: Public API exports for ClawHub.
|
||||
- `scan.js`: Scanning ClawHub registry with pagination.
|
||||
- `install.js`: Installation from ClawHub (ZIP download).
|
||||
- `api.js`: ClawHub API client with rate limiting.
|
||||
|
||||
## Public API
|
||||
|
||||
@@ -24,13 +20,19 @@ The following functions are exported and used by the web server:
|
||||
### Cache (`cache.js`)
|
||||
- `getCacheKey({ normalizedRepo, subpath, identityId })`: Generate cache key for scan results.
|
||||
- `getCachedScan(key)`: Retrieve cached scan result if not expired.
|
||||
- `setCachedScan(key, value, ttlMs)`: Store scan result with TTL (default 30 minutes).
|
||||
- `setCachedScan(key, value, ttlMs)`: Store scan result with TTL (default 3 hours).
|
||||
- `scanWithCache(key, loader, { refresh })`: Run a scan loader with cache lookup, in-flight deduplication, and a global concurrency limit (2 concurrent scans); only `ok: true` results are cached.
|
||||
- `clearCache()`: Clear all cached scan results.
|
||||
- Scan results persist to `skills-catalog-cache.json` in the OpenChamber data dir (debounced, atomic rename) and survive server restarts within the TTL.
|
||||
|
||||
### Curated Sources (`curated-sources.js`)
|
||||
- `getCuratedSkillsSources()`: Return list of curated skill sources (Anthropic, ClawHub).
|
||||
- `getCuratedSkillsSources()`: Return list of curated skill sources (Anthropic, OpenAI, Cursor, Matt Pocock).
|
||||
- `CURATED_SKILLS_SOURCES`: Constant array of predefined sources.
|
||||
|
||||
### GitHub Repository Metadata (`github-meta.js`)
|
||||
- `fetchGitHubRepoMetas(normalizedRepos)`: Fetch `{ stars, repoUpdatedAt }` for GitHub `owner/repo` strings. Best-effort: failures resolve to `null`; in-flight requests deduplicate; results cached in memory and on disk (`skills-github-meta.json`) for three hours.
|
||||
- `clearGitHubMetaCache()`: Test-only cache reset.
|
||||
|
||||
### Source Parsing (`source.js`)
|
||||
- `parseSkillRepoSource(source, { subpath })`: Parse git repository source string into structured object with SSH/HTTPS clone URLs, normalized repo, and effective subpath. Supports SSH URLs, HTTPS URLs, and shorthand `owner/repo[/subpath]` format.
|
||||
|
||||
@@ -40,20 +42,6 @@ The following functions are exported and used by the web server:
|
||||
### Git Repository Installation (`install.js`)
|
||||
- `installSkillsFromRepository({ source, subpath, defaultSubpath, identity, scope, targetSource, workingDirectory, userSkillDir, selections, conflictPolicy, conflictDecisions })`: Install skills from git repository. Supports user/project scopes, opencode/agents targets, conflict resolution (prompt/skipAll/overwriteAll), and sparse checkout for efficiency.
|
||||
|
||||
### ClawHub Integration (`clawdhub/index.js`)
|
||||
- `isClawdHubSource(source)`: Check if source string refers to ClawHub.
|
||||
- `scanClawdHub()`: Scan entire ClawHub registry for all skills (paginated, max 20 pages).
|
||||
- `scanClawdHubPage({ cursor })`: Scan a single page of ClawHub results with cursor-based pagination.
|
||||
- `installSkillsFromClawdHub({ scope, targetSource, workingDirectory, userSkillDir, selections, conflictPolicy, conflictDecisions })`: Install skills from ClawHub by downloading ZIP files.
|
||||
- `fetchClawdHubSkills({ cursor })`: Fetch paginated skills list from ClawHub API.
|
||||
- `fetchClawdHubSkillVersion(slug, version)`: Fetch specific skill version details.
|
||||
- `fetchClawdHubSkillInfo(slug)`: Fetch skill metadata without version details.
|
||||
- `downloadClawdHubSkill(slug, version)`: Download skill package as ZIP buffer.
|
||||
|
||||
### ClawHub Constants (`clawdhub/index.js`)
|
||||
- `CLAWDHUB_SOURCE_ID`: Source identifier for curated sources.
|
||||
- `CLAWDHUB_SOURCE_STRING`: Source string format.
|
||||
|
||||
## Internal Helpers
|
||||
|
||||
The following functions are internal helpers used by exported functions:
|
||||
@@ -63,10 +51,10 @@ The following functions are internal helpers used by exported functions:
|
||||
- `looksLikeAuthError(message)`: Detect if error message indicates authentication failure (permission denied, publickey, etc.).
|
||||
- `assertGitAvailable()`: Check if git is available in PATH.
|
||||
|
||||
### Skill Name Validation (used in `install.js`, `scan.js`, `clawdhub/install.js`)
|
||||
### Skill Name Validation (used in `install.js`, `scan.js`)
|
||||
- `validateSkillName(skillName)`: Validate skill name against pattern `/^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$/` (1-64 chars, lowercase alphanumeric with hyphens).
|
||||
|
||||
### File System Helpers (`install.js`, `scan.js`, `clawdhub/install.js`)
|
||||
### File System Helpers (`install.js`, `scan.js`)
|
||||
- `safeRm(dir)`: Safely remove directory recursively (ignores errors).
|
||||
- `ensureDir(dirPath)`: Ensure directory exists with recursive creation.
|
||||
- `copyDirectoryNoSymlinks(srcDir, dstDir)`: Copy directory contents without symlinks, with path traversal protection.
|
||||
@@ -82,10 +70,6 @@ The following functions are internal helpers used by exported functions:
|
||||
- `toFsPath(repoDir, repoRelPosixPath)`: Convert POSIX path to filesystem path.
|
||||
- `getTargetSkillDir({ scope, targetSource, workingDirectory, userSkillDir, skillName })`: Determine target installation directory based on scope (user/project), targetSource (opencode/agents), and skill name.
|
||||
|
||||
### ClawHub API Helpers (`clawdhub/api.js`)
|
||||
- `rateLimitedFetch(url, options)`: Fetch with rate limiting (120 req/min limit, 100ms delay between requests, exponential backoff on 429/500 errors).
|
||||
- `mapClawdHubItem(item)`: Transform ClawHub API response to SkillsCatalogItem format.
|
||||
|
||||
## Response Contracts
|
||||
|
||||
### Scan Skills Repository Response
|
||||
@@ -101,12 +85,6 @@ The following functions are internal helpers used by exported functions:
|
||||
- `skipped`: Array of skipped skills with `{ skillName, reason }`.
|
||||
- `error`: Error object with `{ kind, message, conflicts? }` on failure. Kinds: `authRequired`, `networkError`, `conflicts`, `invalidSource`, `unknown`.
|
||||
|
||||
### ClawHub Scan Response
|
||||
- `ok`: Boolean indicating success.
|
||||
- `items`: Array of skill items with ClawHub-specific metadata in `clawdhub` property.
|
||||
- `nextCursor`: Pagination cursor for next page (only for `scanClawdHubPage`).
|
||||
- `error`: Error object with `{ kind, message }` on failure.
|
||||
|
||||
### Parse Source Response
|
||||
- `ok`: Boolean indicating success.
|
||||
- `host`: Git host (e.g., `github.com`, `gitlab.com`).
|
||||
@@ -129,7 +107,7 @@ The following functions are internal helpers used by exported functions:
|
||||
|
||||
### Skill Name Validation
|
||||
- All skill names must match `/^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$/` (1-64 chars).
|
||||
- Skill names are derived from directory basenames for git repos and slugs for ClawHub.
|
||||
- Skill names are derived from directory basenames for git repos.
|
||||
- Invalid names result in non-installable skills with appropriate warnings.
|
||||
|
||||
### Git Cloning Strategy
|
||||
@@ -144,17 +122,12 @@ The following functions are internal helpers used by exported functions:
|
||||
- Per-skill decisions override global policy via `conflictDecisions` map.
|
||||
- Conflict response includes `{ skillName, scope, source }` for each conflict.
|
||||
|
||||
### ClawHub Integration
|
||||
- ClawHub API base URL: `https://clawdhub.com/api/v1`.
|
||||
- Pagination uses cursor-based approach with `MAX_PAGES=20` safety limit.
|
||||
- Rate limiting: 120 req/min with 100ms delay between requests.
|
||||
- Downloaded skills are extracted from ZIP files using `adm-zip`.
|
||||
- Always validate `SKILL.md` exists before installation.
|
||||
|
||||
### Cache Management
|
||||
- Cache keys include `normalizedRepo`, `subpath`, and `identityId` for isolation.
|
||||
- Default TTL is 30 minutes; can be overridden via `ttlMs` parameter.
|
||||
- Cache is in-memory (not persisted across restarts).
|
||||
- Default TTL is 3 hours for both scan results and GitHub repository metadata.
|
||||
- Scan and GitHub metadata caches persist to JSON files in the OpenChamber data dir, so app restarts and page refreshes reuse previous results instead of re-hitting GitHub.
|
||||
- Scans run through a global concurrency limiter (2 at a time) with per-key in-flight deduplication.
|
||||
- The refresh button passes `refresh: true` and bypasses the cache.
|
||||
|
||||
### Security Considerations
|
||||
- Path traversal protection in `copyDirectoryNoSymlinks`: resolves real paths and checks containment.
|
||||
|
||||
Reference in New Issue
Block a user