A catalog browser and static-site generator for dfetch package registries.
Search, discover, and compose dfetch.yaml manifests from thousands of reusable components —
regardless of language or build system.
dfetch (docs) is a source-only, VCS-agnostic dependency aggregator. It vendors dependencies directly into your project by cloning Git (or SVN) repositories at a specific ref — no submodules, no subtrees. The same workflow handles C libraries, Rust crates, TypeScript packages, Kotlin modules, and anything else that lives in a repository.
The missing piece is discovery. When you need a networking helper, a JSON parser, or a
logging framework, where do you look? How do you find the canonical Git URL, pick the right
tag, and wire it into your dfetch.yaml?
dfetch-hub answers that question. It:
- Crawls existing package registries (vcpkg, Conan Center Index, clib, and custom sources) and normalises their metadata into a unified catalog.
- Serves a searchable web UI so you can browse and filter components by name, description, license, source registry, or language.
- Publishes the catalog as a fully static site that can be hosted anywhere — no server required.
The result is a single place to find a component, check its available tags, read its README,
and copy the dfetch.yaml snippet that vendors it into your project.
| Term | Meaning |
|---|---|
| Component | A self-contained, reusable unit of source code living in a Git repository. Language-agnostic. |
| Source | A package registry or monorepo that dfetch-hub crawls (e.g. vcpkg, Conan, clib). |
| Catalog | The normalized JSON database of all discovered components. |
| dfetch.yaml | The project manifest consumed by dfetch to vendor dependencies into your codebase. |
A dfetch.yaml manifest looks like this:
manifest:
version: '0.0'
projects:
- name: network-utils
url: https://github.com/user/network-utils.git
tag: v1.0.0
- name: io-helpers
url: https://github.com/org/io-helpers.git
branch: mainKey manifest attributes: url, tag, branch, revision (full SHA), src (partial
checkout of a subfolder), dst (destination path), ignore (files to exclude).
See the dfetch manifest reference
for the full syntax.
dfetch does not care whether the component is C, Rust, Java, or TypeScript — it clones the source tree. dfetch-hub supplies the metadata (Git URL, available tags, README, license) so you can make that choice confidently.
| Registry | Manifest parsed | Notes |
|---|---|---|
| vcpkg | vcpkg.json + portfile.cmake |
Upstream repo extracted from vcpkg_from_github() |
| Conan Center Index | conandata.yml + conanfile.py |
Full attribute extraction via AST |
| clib | Packages.md wiki + per-package package.json |
Category tags preserved |
| Custom Git repos | README (fallback) |
Works for monorepos with subfolder components |
New sources are added by writing a small manifest parser and registering it in dfetch-hub.toml.
Requirements: Python 3.11 or later.
pip install dfetch-hubFor development (includes all linters, formatters, and test dependencies):
git clone https://github.com/dfetch-org/dfetch-hub.git
cd dfetch-hub
pip install -e ".[development]"
pre-commit installPull fresh metadata from all configured sources:
dfetch-hub updateUse --limit N during development to cap the number of packages fetched per source:
dfetch-hub update --limit 10Target a single source by name:
dfetch-hub update --source vcpkgLaunch a local web server and open the browser automatically:
dfetch-hub serveNavigate to http://127.0.0.1:8000, type a package name in the search bar, and explore.
Press / to jump to the search box. Press Ctrl+K for keyboard shortcuts.
Build a deployable artifact (suitable for GitHub Pages, Netlify, etc.):
dfetch-hub publishThe output lands in public/ by default. All paths are rewritten so the site works
without a local dev server.
dfetch-hub is configured with a dfetch-hub.toml file in the current working directory.
[settings]
concurrency = 8 # parallel fetch workers
catalog_path = "dfetch_hub/data"
output_dir = "public"
[[source]]
name = "vcpkg"
strategy = "subfolders"
url = "https://github.com/microsoft/vcpkg"
path = "ports"
manifest = "vcpkg.json"
label = "vcpkg"
[[source]]
name = "conan"
strategy = "subfolders"
url = "https://github.com/conan-io/conan-center-index"
path = "recipes"
manifest = "conandata.yml"
label = "conan"
[[source]]
name = "clib"
strategy = "catalog-file"
url = "https://github.com/clibs/clib.wiki.git"
manifest = "Packages.md"
label = "clib"
[auth]
github_token = "GITHUB_TOKEN" # environment variable name (not the token itself)| Strategy | Description |
|---|---|
subfolders |
Each subdirectory under path is treated as one component. |
catalog-file |
Clones a repository and parses a file for component URLs. |
Skip a subfolder if it contains a specific file or directory — useful for filtering versioned recipe trees like Conan's:
[[source]]
name = "conan"
strategy = "subfolders"
url = "https://github.com/conan-io/conan-center-index"
path = "recipes"
manifest = "conandata.yml"
ignore_if_present = "conanfile.py" # skip top-level recipe dir; descend into version dirsMonorepos often tag releases with component-scoped prefixes (audio/v1.2.3,
VideoEncoder-2.0.0) or mix release and pre-release tags freely. Tag filters let
you declare exactly which tags to keep for each source.
Define a named filter in a [filter.<name>] block, then reference it from a
[[source]] block via filter = "<name>":
# Keep only tags that look like semantic versions — drop "latest", "nightly", etc.
[filter.semver-only]
include = [{kind = "semver"}]
[[source]]
name = "my-lib"
strategy = "subfolders"
url = "https://github.com/example/my-lib"
filter = "semver-only"The {{component}} placeholder in rule values is replaced at runtime with the
subfolder name of each component, so one filter definition covers the whole monorepo:
[filter.monorepo]
# Keep only tags scoped to this component and parseable as semver
include = [
{kind = "prefix", value = "{{component}}/", case = "sensitive"},
{kind = "semver", value = ""},
]
# Drop pre-release tags
exclude = [{kind = "regex", value = "-(alpha|beta|rc)\\d*$"}]
[[source]]
name = "audio-video-sdk"
strategy = "subfolders"
url = "https://github.com/example/av-sdk"
path = "components"
manifest = "readme"
filter = "monorepo"| Field | Values | Description |
|---|---|---|
kind |
prefix |
Tag must start with value (after normalisation). |
regex |
Tag must match the value regular expression. |
|
semver |
Tag must be a valid semantic version (vX.Y.Z or X.Y.Z, |
|
| with optional pre-release / build-metadata). | ||
value |
string | Prefix or regex pattern. May contain {{component}}. |
case |
smart (default) |
Split CamelCase/PascalCase, strip separators, lowercase. |
LowPassFilter, low-pass-filter, and LOW_PASS_FILTER all compare equal. |
||
insensitive |
Lowercase only; separators kept. | |
sensitive |
Exact byte-level comparison. | |
normalize-lower |
Strip separators then lowercase (no CamelCase split). | |
normalize-upper |
Strip separators then uppercase (no CamelCase split). |
Filtered tags are always stored with their original, unmodified name — normalisation only affects comparison logic, never the output.
Git Repos (any language)
│
▼
Crawler & Manifest Parsers
(vcpkg · Conan · clib · README)
│
▼
Catalog JSON files
(catalog.json + per-project detail files)
│
▼
Static Web UI ──▶ dfetch-hub serve / dfetch-hub publish
│
▼
User selects a component and ref
│
▼
dfetch.yaml snippet ──▶ dfetch update ──▶ vendored sources
dfetch-hub <command> [options]
Commands:
update Fetch sources and rebuild the catalog.
serve Launch a local web server to browse the catalog.
publish Build a deployable static site from catalog data.
dfetch-hub update
--config FILE Path to dfetch-hub.toml [default: dfetch-hub.toml]
--data-dir DIR Catalog output directory [default: from config]
--limit N Max packages per source (useful for testing)
--source NAME Process only this source
dfetch-hub serve
--port PORT HTTP port [default: 8000]
dfetch-hub publish
--config FILE Path to dfetch-hub.toml [default: dfetch-hub.toml]
--output DIR Output directory [default: public/]
# Activate the virtual environment
source .venv/bin/activate
# Run the full test suite (206 tests, fully mocked — no network required)
pytest
# Run all pre-commit hooks
pre-commit run --all-filesThe test suite mocks all network and filesystem calls. Tests must not hit the network or
write outside tmp_path.
MIT — see LICENSE.