Aegis documentation

Introduction

Aegis lets a website owner see, restrict, monetize, and authorize automated access to their sites. It sits at your edge or in your application as a lightweight connector, makes each access decision locally in well under a millisecond, and is managed from one console — with no per-request cloud lookup and, by default, no visitor data leaving your infrastructure (the one opt-in exception is shared threat intel).

Aegis is portable (it runs on any stack — Cloudflare, Node, Lambda, Deno, Bun, Vercel/Netlify Edge, Fly), transparent (open and auditable, no black-box ML), and owner-controlled (your policy, your revenue, your data). It is a control plane for AI access, not a reverse proxy and not a CDN.

Who this is for

Anyone running a website who wants to decide what AI crawlers and agents may do with it — block training bots, keep search bots, offer a paid licence, verify who's really crawling, price the actions agents take, or simply observe the traffic first.

Key concepts

Quick start (10 minutes)

1. Sign in. Open the console and sign in with your email and password (or Sign in with SSO). The first account on a new console creates its organization; everyone else joins by invitation.
2. Add a website. Click Add website, give it a name and the exact hostname(s) it serves (e.g. example.com and www.example.com).
3. Save the site key shown once — you'll paste it into the connector.
4. Download a connector from the Connect a site tab (start with the one matching your stack).
5. Configure it with AEGIS_CONTROL_URL, AEGIS_SITE_ID, AEGIS_SITE_KEY, and the AEGIS_BOOTSTRAP snapshot from the console.
6. Deploy the connector in front of / inside your app. It begins reporting traffic within about a minute.
7. Observe. New sites start in monitor mode — nothing is blocked. Watch the Overview tab and review who's reaching you.
8. Enforce. When you're comfortable, switch the site to enforce and save. Refresh the bootstrap snapshot for new instances.

For the full walkthrough with checks at every step, see the end-to-end implementation guide.

End-to-end implementation guide

This walks you from nothing to a site that is protected and enforcing, in the order that avoids surprises. Plan on about an hour of work plus a few days of observing traffic before you enforce. Each step says how to check it worked.

Step 0 — Plan (10 minutes)

Step 1 — Create the site in the console

1. Sign in (your operator created your account or sent an invitation).
2. + Add website: a name and every exact hostname the site serves, one per line (example.com and www.example.com are different hostnames).
3. Open Connect a site. Copy the site key now; it is shown once. Treat it like a password.
4. Copy the site id and the saved policy snapshot (AEGIS_BOOTSTRAP) from the same tab.

Check: the site appears in the list in Monitor mode.

Step 2 — Install the connector

Download your connector from Connect a site (one dependency-free file) and follow its page: Worker, Node, Fastify, Lambda, Fetch. Every connector needs the same four values:

AEGIS_CONTROL_URL = https://your-console.example      # this console
AEGIS_SITE_ID     = 64f1c2…                           # 24 hex characters
AEGIS_SITE_KEY    = (secret)                          # from step 1, stored as a secret
AEGIS_BOOTSTRAP   = {"siteId":"…","version":1,…}      # the snapshot, as a string

Mount it before your expensive routes so blocked requests cost you nothing. Configure the client-IP resolver you chose in step 0.

Complete example — Cloudflare Worker

# wrangler.toml
name = "aegis-example"
main = "aegis-worker.mjs"
compatibility_date = "2026-09-12"
workers_dev = false
routes = [{ pattern = "example.com/*", zone_name = "example.com" }, { pattern = "www.example.com/*", zone_name = "example.com" }]

[vars]
AEGIS_CONTROL_URL = "https://your-console.example"
AEGIS_SITE_ID = "64f1c2…"

# then, from the same folder:
wrangler secret put AEGIS_SITE_KEY        # paste the site key
wrangler secret put AEGIS_BOOTSTRAP       # paste the snapshot JSON (a secret is fine for a long string)
wrangler deploy

Complete example — Node / Express

// app.mjs  (Node 22+)
import express from 'express';
import { aegisMiddleware, xForwardedForHop } from './aegis-node.mjs';

const app = express();
const guard = aegisMiddleware({
  controlUrl: process.env.AEGIS_CONTROL_URL,
  siteId: process.env.AEGIS_SITE_ID,
  siteKey: process.env.AEGIS_SITE_KEY,
  bootstrap: process.env.AEGIS_BOOTSTRAP,                // string or parsed object
  getClientIp: xForwardedForHop({ trustedProxies: 1 }),   // ONLY if exactly one proxy you run sits in front
});
app.use(guard);                                           // before your routes
app.get('/', (req, res) => res.send('hello'));
const server = app.listen(3000);
process.on('SIGTERM', () => { guard.close(); server.close(); });

Step 3 — Verify the installation

1. Heartbeat: within 1–2 minutes the Overview tab shows a connector heartbeat, and the reported policy version equals your saved version.
2. Traffic is counted: send a test crawler request and see it in the decision breakdown (still allowed, because you're in monitor mode):

   curl -s -o /dev/null -w '%{http_code}\n' -A "GPTBot/1.2" https://example.com/     # 200 in monitor mode
   curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0" https://example.com/    # 200

   Refresh Overview after a minute: training · GPTBot appears under "would restrict".
3. No host_mismatch in the breakdown. If you see it, a hostname is missing from the site.
4. Every entry point shows traffic (check each hostname).

Step 4 — Observe in monitor mode (a few days)

Nothing is blocked yet. Use the time to learn what's reaching you:

• Would restrict shows what enforcement would do. Open the breakdown and read the reasons and bot names.
• Look for things you must not block: your uptime monitor, payment/CI webhooks, internal API clients, partner integrations. Most appear as unknown_automation (a library user-agent). Add their paths to excluded paths, or give those clients a real user-agent.
• Check search engines: with verified-bot metadata or IP ranges available, they show as verified_bot.
• If everything is rate_limit from one address, your client-IP source is a proxy; fix getClientIp.

Step 5 — Tune the policy

Turn features on one at a time, in roughly this order, re-checking "would restrict" after each (Protection rules tab):

1. AI categories: training (on by default), search, assistants.
2. Unknown automation: rate-limit (default) → block or challenge once exclusions are right.
3. Optional: monetize/x402 or data licensing instead of blocking.
4. Optional defenses: heuristics, TLS fingerprints, honeypot + trap links, challenge, tarpit, threat intel.
5. Optional: agent actions for write endpoints.

Use the policy simulator to test a specific request against the saved rules before you rely on them.

Step 6 — Enforce

1. Set the mode to Enforce and save. Connectors apply it within about a minute (two with the policy mirror).
2. Refresh AEGIS_BOOTSTRAP in the connector's configuration with the new snapshot, so newly started instances enforce immediately.
3. Re-run the step 3 checks: the GPTBot request now returns 403 (or 402 when monetizing).

Step 7 — Maintain

• After every policy change, refresh the bootstrap snapshot in your connector config.
• Review the Overview breakdown weekly; new crawlers and tools appear over time.
• If the site key leaks: Connect a site → Rotate, then update AEGIS_SITE_KEY. (Until you do, the connector keeps its last policy but can't sync.)
• When your operator announces a new connector version, re-download it and redeploy, one site in monitor mode first.

Recipes by goal

Each recipe gives the policy (Protection rules tab, or the JSON the console saves), anything needed in the connector, and how to check it. Start every recipe in monitor mode.

1. Block AI training, keep AI search and Google

{ "mode": "enforce", "blockTraining": true, "blockAiSearch": false, "blockAiAssistant": false,
  "allowVerifiedBots": true, "unknownAction": "rate-limit" }

Also merge the generated robots.txt fragment (Connect a site) into your robots.txt so compliant crawlers see the preference. Check: curl -A "GPTBot/1.2" → 403; curl -A "OAI-SearchBot/1.0" → 200.

2. Offer a paid licence instead of blocking (402)

{ "mode": "enforce", "blockTraining": true, "monetize": true, "licenseUrl": "https://example.com/ai-licence" }

AI crawlers get 402 with Link: <https://example.com/ai-licence>; rel="license". To collect on-chain payments per request, add the x402 settings and a facilitator.

3. Sell data access to AI companies with licence keys

1. Protection rules → Data licensing: turn on, set a purchase URL, define tiers (e.g. research | Research | /articles/ | 600), save.
2. Keep your AI-category rules as they are (the licence is the paid way past them).
3. Licensees → Issue licence key: licensee name, tier, validity. Copy the key (shown once) and send it to the customer securely.
4. The customer sends it on every request:

   curl -H "Aegis-License: aegl1.eyJ2IjoxLC…" -A "ExampleAIBot/1.0" https://example.com/articles/42    # 200

5. Invoice monthly from Download usage CSV (month, licence, licensee, tier, requests). You bill; you keep 100%.
6. Revoke a key any time; connectors refuse it within about a minute.

An unlicensed AI crawler gets 402 with Link: <purchase URL>; rel="license" and Aegis-License-Purchase on paths a tier covers, and your normal 403 (with the purchase header) elsewhere. See Data licensing.

4. Catch scrapers that ignore robots.txt (trap links)

{ "honeypot": { "enabled": true, "trapLinks": true, "trapPrefix": "/assets/r/", "banSeconds": 3600, "action": "tarpit" } }

• Worker: nothing else; links are injected into your HTML automatically.
• Fetch runtimes: return (await guard(req)) ?? guard.decorate(req, await origin(req));
• Node/Express: add the link to your page template:

  app.get('/', async (req, res) => {
    const trap = await guard.trapLink(req);                 // '' when traps are off or IP unknown
    res.send(`<html><body>…your page…${trap}</body></html>`);
  });

Needs a trustworthy client IP. Check: view source of a page and find the hidden <a href="/assets/r/…" hidden …>; requesting that URL from the same machine bans your IP for banSeconds (use a test IP!).

5. Stop scripts that pretend to be browsers

{ "heuristics": { "enabled": true, "requireBrowserHeaders": true, "tlsConsistency": true, "action": "rate-limit" },
  "tlsBlocklist": ["t13d1812h1_85036bcba153_375ca2c5e164"], "unknownAction": "block" }

TLS checks need a fingerprint source: Cloudflare Worker with Bot Management (automatic), or behind CloudFront:

import { aegisLambda, cloudfrontJa4, lambdaSourceIp } from './aegis-lambda.mjs';
const guard = aegisLambda({ controlUrl, siteId, siteKey, bootstrap,
  getClientIp: lambdaSourceIp, getTlsFingerprint: cloudfrontJa4 });

Add CloudFront-Viewer-JA4-Fingerprint to the distribution's origin request policy, and make sure only CloudFront can reach the origin. Start the heuristics action at Observe only if you have users behind corporate TLS-inspecting proxies.

6. Protect checkout and APIs from AI agents

{ "agentActions": { "enabled": true, "methods": ["POST","PUT","PATCH","DELETE"], "paths": ["/checkout", "/api/"],
    "requireSigned": true, "allowedAgents": ["ExampleAgent"], "action": "block" } }

Add the agent operator's public key under Verified agents with the name ExampleAgent. Signed writes from that agent pass; unsigned or unknown agent writes get 403; human browsers are unaffected. Use "action": "monetize" to charge per action instead.

7. Join shared threat intel

{ "honeypot": { "enabled": true, "trapLinks": true },
  "threatIntel": { "contribute": true, "consume": true, "action": "rate-limit" } }

Contributing shares the IPs your trap links catch (read what is shared first); consuming applies a stricter limit to IPs caught on at least two other organizations' sites.

8. Run many connector instances (policy mirror)

Ask your operator to enable the mirror; the Connect tab then shows AEGIS_POLICY_MIRROR_URL. Add it to every connector (Worker env var, or the policyMirrorUrl option). Check: the heartbeat keeps updating; policy changes arrive within about two minutes.

9. One policy across several stacks

Use one site with all hostnames, and install the matching connector on each stack (e.g. a Worker for www.example.com and the Node connector on api.example.com), all with the same site id and key. Both enforce the same policy and report into one dashboard. Or use separate sites if the policies should differ.

10. A global rate limit across instances

import { aegisMiddleware, createSharedLimiter } from './aegis-node.mjs';
const backend = { async incr(key, windowSeconds) {             // any atomic counter; Upstash Redis REST shown
  const call = p => fetch(`${process.env.REDIS_URL}/${p}`, { headers: { authorization: `Bearer ${process.env.REDIS_TOKEN}` } }).then(r => r.json());
  const { result } = await call(`incr/${encodeURIComponent(key)}`);
  if (result === 1) await call(`expire/${encodeURIComponent(key)}/${windowSeconds}`);
  return result;
} };
const guard = aegisMiddleware({ controlUrl, siteId, siteKey, bootstrap, getClientIp: trustedResolver,
  sharedLimiter: createSharedLimiter(backend) });

Fails open (a backend outage allows traffic). Not available in the Worker connector.

11. Serve Markdown to licensed and verified AI agents

Give identified agents a clean Markdown copy of your pages instead of HTML: a licensee (Aegis-License), an agent with a verified Web Bot Auth signature, or a crawler verified by IP range or your CDN. Everyone else gets the normal page, and your AI-category blocks still apply first.

// Cloudflare Worker: set the variable AEGIS_MARKDOWN = "true"
// Fetch / Edge:
const guard = createFetchGuard({ controlUrl, siteId, siteKey, bootstrap, markdownOrigin: req => origin(req) });
// Node / Express (after the guard, before your routes):
app.use(guard);
app.use(guard.markdown(async (path, req) => renderPageHtml(path)));   // return null for no such page
// Fastify:
fastify.addHook('onRequest', guard);
fastify.addHook('onRequest', guard.markdown(async (path, req) => renderPageHtml(path)));

An agent asks with Accept: text/markdown or by adding .md to a path (/pricing.md, /index.md for the home page). Accept: application/vnd.aegis.page+json returns JSON. Check: curl -H 'Accept: text/markdown' https://example.com/ from an ordinary machine returns HTML (you are not identified); a licensee's request with its key returns Markdown. Then publish the llms.txt from Connect a site with “My connector serves Markdown” ticked.

Connector options reference

Options for aegisMiddleware (Node), aegisFastify, aegisLambda and createFetchGuard. The Cloudflare Worker takes the environment variables shown instead.

Helpers returned with the guard

Exported resolvers (every bundle)

How it works

Three components:

• The engine classifies each request and returns a decision with zero DNS, database, logging, or HTTP calls. It caps user-agents at 2,048 characters and paths at 4,096, and performs no arbitrary user-supplied regular expressions.
• Connectors run the engine at your edge/app. On each request they evaluate locally and either short-circuit (403/402/429/200-tarpit) or pass the request through untouched. In the background — at most about once a minute — they pull the latest policy and push privacy-safe aggregate counts.
• The console stores each site's policy durably (in PostgreSQL, with optimistic-concurrency version checks), serves it to authenticated connectors, and shows aggregate metrics. It also fetches published crawler IP-range feeds and distributes them, plus your trusted keys and your data-licensing public key, in the policy envelope.

Optional pieces: a policy mirror (a bucket/CDN connectors read the policy from instead of the console), data licences (keys the console signs and connectors verify locally), and shared threat intel (opt-in exchange of IPs caught by trap links).

Visitor ─▶ [ Aegis connector at your edge/app ] ─▶ your origin
                 │ local, sub-ms decision (no per-request cloud call)
                 │ allow · block(403) · monetize(402) · rate-limit(429) · tarpit(200)
                 ├── background: pull policy (or read it from the mirror) + push aggregate metrics ──▶ Console ─▶ Postgres
                 └── opt-in: report trap-caught IPs / pull the shared threat list

Request lifecycle (evaluation order)

For each request, the engine evaluates rules in a fixed order and returns on the first match:

1. Disabled mode → allow everything.
2. Excluded paths → bypass all checks. /health covers /health and /health/… but not /health-data; / exempts the whole site.
3. Honeypot (if enabled) → a client already banned is refused; a hit on a decoy path bans the source IP; a request under the trap-link prefix is refused, and bans the IP only if its token was minted for that IP. Checked before user-agent handling.
4. Agent actions (if enabled) → a write request from an identified/automated agent is authorized here (allowed) or refused (blocked / charged). A human's browser write is not an agent action and continues.
5. Data licence (if enabled) → a request with a valid Aegis-License key whose tier covers the path is allowed, limited per licence by its tier.
6. AI-category restrictions → a matching training / search / assistant crawler produces a 403 candidate (or a 402 licence when monetizing), even if the bot is verified or signed.
7. Link-preview crawlers → allowed so shared links unfurl (if allowSocialPreview).
8. Verified signed agent (Web Bot Auth) → may bypass the remaining checks, when allowSignedAgents is on.
9. Infrastructure-verified bots → may bypass the remaining checks, when allowVerifiedBots is on.
10. Challenge passed → a client with a valid proof-of-work pass is treated like a browser and skips the remaining gates.
11. Shared threat intel (if consuming) → an IP on the shared list gets a stricter rate limit, a challenge, or a block.
12. Heuristics (if enabled) → a browser impersonator (missing common headers, or a non-browser TLS handshake) is observed, rate-limited on a stricter bucket, or blocked.
13. Unknown automation → an empty or automation-signature user-agent, or a blocklisted TLS fingerprint, is allowed / rate-limited / blocked / challenged per unknownAction.
14. Rate limit → the token bucket limits unverified clients; excess yields a 429 with Retry-After.
15. Monitor mode → the candidate action is reported but the request is always allowed.

A block may instead be served as a tarpit maze when tarpit is enabled. See decision reasons for the full list of outcomes.

Data & privacy

Aegis is designed to keep visitor data on your infrastructure:

• Evaluation makes no external calls, so visitor requests never reach the console.
• The only telemetry is a private, best-effort aggregate: decision reasons and counts, plus the AI bot's public, self-declared name (e.g. GPTBot).
• By default, no visitor IPs, raw user-agents, paths, or query strings leave the connector. Metrics are grouped into 24 hourly UTC buckets by receipt time and de-duplicated per site.
• Trusted keys, crawler IP ranges and the licensing public key are public material; nothing secret is carried in the policy envelope.
• Per-licence request counts are reported for invoicing; a licence id identifies your business customer, not a visitor.

Overview dashboard

The Overview tab summarizes the last 24 hours for the selected site:

• Observed / Blocked / Would restrict / Rate limited — headline counts. "Would restrict" is what enforce mode would have done while you're in monitor mode.
• Traffic at a glance — an hourly bar chart of observed requests.
• Decision breakdown — each decision reason with counts, and the specific AI bot name where known (e.g. training · GPTBot).
• Connector heartbeat — the most recent report time and the policy version the connector last reported, versus your saved version.

Onboarding a website

1. Click + Add website.
2. Enter a display name (1–80 characters).
3. Enter hostnames, one per line — exact matches, including www if you use it. Use plain hostnames like example.com; no scheme, wildcard, port, or path. 1–20 hostnames per site.
4. Create the site. It starts in monitor mode with default rules.
5. Save the site key shown in the Connect tab — it is displayed once. It authorizes policy fetch and metric submission for this site.

Protection rules

The Protection rules tab is where you configure a site's policy. Changes take effect on connected connectors after their next sync (usually within 60–65 seconds). Sections:

• Rollout — the mode: Disabled, Monitor, or Enforce.
• Bot categories — block AI training, AI search, or user-triggered AI fetches; allow infrastructure-verified bots; choose what to do with unknown automation.
• AI licensing (monetize) — offer a 402 licence instead of blocking, and optionally accept on-chain payment with x402.
• Verified agents — the trusted Web Bot Auth keys and the toggle to honor signatures; also shows the auto-maintained IP ranges.
• Agent actions — authorize/price the actions agents take.
• Data licensing — tiers, the purchase URL, and the Licensees list (issue, revoke, usage CSV).
• Active defense — browser-impersonator heuristics, TLS fingerprint checks and blocklist, honeypot decoys and trap links, shared threat intel, and the proof-of-work challenge.
• Traffic limits — the rate limiter, plus excluded paths and the tarpit toggle.
• Website — the display name and hostnames.

See the configuration reference for every field. Click Save protection rules to persist; the app checks the version to prevent overwriting a change made in another session.

Connect a site

The Connect a site tab provides everything needed to install a connector:

• Site key — shown once on creation/rotation; copy it now.
• Connector downloads — Cloudflare Worker, Node, and the portable Fetch/Lambda/Fastify connectors. All are dependency-free single files.
• Configuration snippets — the environment variables and example code for your stack.
• Saved policy snapshot — the AEGIS_BOOTSTRAP JSON. Copy it into your connector so new instances start with your policy. It is pinned to version 1 so the live console policy always wins once a connector syncs.
• robots.txt & AI preferences — the generated declaration to merge into your robots.txt.
• Policy mirror URL — shown in the snippets when your operator has enabled the mirror (AEGIS_POLICY_MIRROR_URL).
• Rotate this site's key — generate a new key (invalidates the old one).

Policy simulator

On the Protection rules tab, the Policy simulator evaluates your saved policy against a request you describe — a user-agent, a path, an HTTP method, and optionally a JA4 TLS fingerprint — and shows the exact decision JSON. Use it to verify a rule before enforcing.

Accounts & invitations

Each person has an email/password account and belongs to one or more organizations. Roles, highest first: owner (full control), admin (manage members/config), editor (manage sites & policies), viewer (read). Every tenant-scoped action is authorized server-side by role.

Invite-only by default. The very first account to register becomes the owner of a new organization (bootstrap). After that, self-serve sign-up is disabled and new people join by invitation: an admin opens Team → Invitations, enters an email + role, and shares the single-use link (/?invite=…, valid 7 days). The invitee sets a password and joins at the assigned role; their email is treated as already verified. Operators can allow open self-serve registration by setting AEGIS_ALLOW_SIGNUP=true — invitations work either way.

Add existing account-holders to an org directly under Team → Add a member; invite people who don't yet have an account under Invitations. Seat limits come from the org's plan.

SSO, SAML, SCIM & two-factor

• OIDC SSO (Okta, Entra ID, Google Workspace, Auth0, …): an owner configures the issuer, client id/secret, and optional email-domain restriction under Team. Members sign in via “Sign in with SSO.” The ID token's RS256 signature, issuer, audience, nonce, and expiry are all verified; the client secret is encrypted at rest.
• SAML 2.0: for IdPs that use SAML — paste the entry point, issuer, and signing certificate; hand your IdP the SP metadata URL shown in the panel. Responses are signature- and replay-checked (InResponseTo + NotOnOrAfter).
• SCIM 2.0: auto-provision/deprovision members from your IdP with a generated bearer token + base URL.
• Two-factor (TOTP): any user can enable an authenticator app under Security, with one-time recovery codes. If someone loses both, an owner can reset it under Team → Reset 2FA (the owner needs 2FA on and re-enters their password and a code; they must own every organization the person belongs to). The person is signed out, emailed, and it's audited.

SSO/SAML/SCIM are plan-gated (the sso entitlement). Self-hosted consoles without Stripe default to the top plan, so these are available out of the box.

Email verification & password reset

When an email provider is configured (AEGIS_EMAIL_ENDPOINT), a verification link is sent on sign-up and a reset link on “Forgot your password?”. Tokens are random, stored only as a hash, single-use, and short-lived (verify 24h, reset 1h); completing a reset signs the account out of all other sessions. The “forgot password” endpoint always responds identically whether or not the address exists, to limit account enumeration.

Install & configure a connector

Every connector reads the same four environment values:

Every option, helper and resolver is listed in the connector options reference. Licensees you've issued keys to send them in the Aegis-License header; connectors verify them locally with no extra configuration.

These are server/edge variables, not browser variables. Each connector short-circuits with 403 (block), 402 (licence/payment), 429 (rate limit), or a 200 tarpit, and otherwise passes the request through untouched. Install a connector at every entry point to your origin; an unprotected alternate origin bypasses an edge-only layer.

Cloudflare Worker

Deploy the downloaded aegis-worker.mjs on a proxied Cloudflare Route (not a Worker Custom Domain), in front of an existing proxied origin. Example Wrangler config:

name = "aegis-example"
main = "aegis-worker.mjs"
compatibility_date = "2026-09-12"
routes = [{ pattern = "example.com/*", zone_name = "example.com" }]

[vars]
AEGIS_CONTROL_URL = "https://YOUR-CONSOLE.run.app"
AEGIS_SITE_ID = "YOUR_SITE_ID"
# AEGIS_BOOTSTRAP as a Text variable (the JSON snapshot as a string).
# Set AEGIS_SITE_KEY with: wrangler secret put AEGIS_SITE_KEY

• Client IP comes from CF-Connecting-IP; verified-bot status and the JA4 TLS fingerprint from request.cf.botManagement (requires the appropriate Bot Management plan — never inferred from a user-agent).
• The adapter forwards the original request once and does not buffer bodies. It modifies an allowed response only when trap links are on: the hidden link is streamed into HTML pages before </body>.
• Optional: AEGIS_POLICY_MIRROR_URL to read the policy from the edge mirror.
• Emergency bypass: set AEGIS_BYPASS=true in the Worker environment, or remove the Aegis route.
• If you already have a Worker that does routing/auth, integrate the engine into it — do not replace it.

Node.js / Express / Connect

Download aegis-node.mjs into your server project (Node 22+):

import { aegisMiddleware } from './aegis-node.mjs';

const guard = aegisMiddleware({
  controlUrl: process.env.AEGIS_CONTROL_URL,
  siteId: process.env.AEGIS_SITE_ID,
  siteKey: process.env.AEGIS_SITE_KEY,
  bootstrap: JSON.parse(process.env.AEGIS_BOOTSTRAP),
  // Behind a proxy, supply a verified resolver (see Client IP):
  // getClientIp: xForwardedForHop({ trustedProxies: 1 }),
  // Optional: getTlsFingerprint: ja4FromHeader('x-ja4'), policyMirrorUrl: process.env.AEGIS_POLICY_MIRROR_URL,
  //           sharedLimiter: createSharedLimiter(backend)
});
app.use(guard);                 // mount before expensive routes
// Trap links (if enabled): put `await guard.trapLink(req)` into your HTML before </body>
// On shutdown: guard.close();   // stops the refresh timer (does not disable decisions)

Plain Node HTTP: guard(req, res, () => yourHandler(req, res)). The middleware is async so it can verify signatures and consult a shared limiter; ordinary traffic still takes the fast path.

Fastify

import { aegisFastify } from './aegis-fastify.mjs';
const guard = aegisFastify({ controlUrl, siteId, siteKey, bootstrap });
fastify.addHook('onRequest', guard);
fastify.addHook('onClose', (i, done) => { guard.close(); done(); });
// Behind a trusted proxy: enable Fastify trustProxy and pass getClientIp: req => req.ip
// Trap links (if enabled): include `await guard.trapLink(req)` in the HTML you send

AWS Lambda

For API Gateway HTTP API (payload v2) or a Lambda Function URL. The guard is async — await it:

import { aegisLambda } from './aegis-lambda.mjs';
const guard = aegisLambda({ controlUrl, siteId, siteKey, bootstrap });
export const handler = async (event) => await guard(event) ?? myHandler(event);
// Source IP is event.requestContext.http.sourceIp; pass getClientIp: lambdaSourceIp
// only where that boundary is trusted. Behind CloudFront: getTlsFingerprint: cloudfrontJa4.
// Trap links (if enabled): include `await guard.trapLink(event)` in HTML bodies you return.

Web-Fetch / Edge runtimes

For Deno, Bun, Vercel/Netlify Edge, Fly, or any standard fetch handler. The guard is async — await it, and it returns a Response to short-circuit or null to pass through:

import { createFetchGuard } from './aegis-fetch.mjs';
const guard = createFetchGuard({ controlUrl, siteId, siteKey, bootstrap: JSON.parse(bootstrap) });
export default { fetch: async (req, env, ctx) => await guard(req, ctx) ?? origin(req) };

// With trap links, let the guard decorate your origin's HTML:
export default { fetch: async (req, env, ctx) => (await guard(req, ctx)) ?? guard.decorate(req, await origin(req)) };

guard.decorate only rewrites 200 text/html UTF-8 responses that aren't compressed; everything else passes through unchanged.

Bootstrap & cold starts

AEGIS_BOOTSTRAP is a cold-start snapshot of your policy (plus trusted keys and IP ranges), copied from the console's Connect tab. It is pinned to version 1, so the live console policy always wins once a connector syncs (about once a minute). Its purpose is to give a brand-new instance immediate protection instead of failing open until the first sync.

• The snapshot is accepted as a JSON string or an already-parsed object.
• A malformed snapshot never disables the connector — it cold-starts and syncs from the console, and logs a warning.
• Without a bootstrap, a new/unconfigured instance fails open (allows traffic) until the first successful sync. This favors availability; refresh your snapshot after policy changes so new instances start close to your intended rules.
• Last-known-good policy stays active through console outages.

Policy sync: conditional polls & edge mirror

Problem: every connector instance polls for its policy once a minute. With thousands of instances, those polls land on the console and its database even though the policy rarely changes.

• Conditional polls (always on). The console tags each policy response with an ETag. A connector that already runs that exact policy gets a bodiless 304 Not Modified, so an unchanged policy costs almost no bandwidth.
• Edge mirror (optional). Set AEGIS_POLICY_MIRROR_BUCKET on the console and AEGIS_POLICY_MIRROR_URL (option policyMirrorUrl) on connectors. The console publishes each site's policy to the bucket on every change, and connectors read it from there instead of from the console.
• The mirror is untrusted. Each object is AES-256-GCM encrypted under a key derived from that site's key, at a name derived the same way. The bucket host can't read, forge or alter a policy, and rotating the site key moves the object. Deleting an organization deletes its mirror objects.
• Fallbacks: a connector still asks the console directly about every 10 minutes. It also falls back to the console whenever the mirror copy is missing, fails to decrypt, has expired (the seal lasts 7 days) or is older than the policy it already runs. It never downgrades.
• Constraint: with the mirror, a change can take up to about two minutes to arrive (a 60-second edge cache plus the poll interval). Metrics reports still go to the console.

Client IP & proxy trust

Rate limiting and IP-range verification use the client IP, so trust it deliberately.

• By default the Node/Fastify/Fetch/Lambda connectors ignore forwarded headers (X-Forwarded-For, CF-Connecting-IP) and any visitor-claimed verified-bot header.
• Behind a CDN, proxy, or load balancer, supply getClientIp(req) using a resolver that reflects your actual trusted proxy boundary. Do not blindly use the first forwarded address.
• If you can't verify the boundary, pass getClientIp: () => '' to skip IP-keyed checks rather than grouping every visitor behind a shared proxy.
• getVerifiedBot(req) must likewise come from authenticated infrastructure metadata, never a visitor-supplied claim.
• The Cloudflare Worker uses CF-Connecting-IP and request.cf automatically.

Ready-made resolvers. adapters/clientip.mjs — re-exported from every downloadable connector bundle — provides safe getClientIp factories so you don't hand-roll one:

• clientIpFromHeader('cf-connecting-ip') — trust exactly one header your platform authenticates.
• xForwardedForHop({ trustedProxies }) — read the client from the right of X-Forwarded-For, counting only the hops your infrastructure appends, so a client-prepended forgery is ignored.
• cloudflareClientIp, directSocketIp (raw TCP peer), lambdaSourceIp.

Each validates the value and returns '' (skip, don't guess) when it can't be trusted.

Configuration reference — policy fields

A site's policy is a JSON object. Unknown fields are rejected. All fields have safe defaults, so a minimal policy such as { "mode": "monitor" } is valid.

rateLimit

Limits are per process/isolate unless you add a shared limiter (see Rate limiting). Counters reset on policy changes and instance restarts.

x402

agentActions

Recommended for write paths: keep requireSigned: true (the default) and scope paths to your write endpoints, naming only trusted agents in allowedAgents. The engine exports a recommendedAgentActions({ paths, allowedAgents }) preset that sets this up.

heuristics

Detects browser impersonators — a UA that claims a browser but omits headers every real browser sends, or that a connector flags via a TLS hint. Only applies to a claimed browser that is not a verified/signed agent or a known AI category, and only when the connector forwards request headers (all bundled connectors do). Never affects a real browser or a verified agent.

A connector may also pass tlsSuspicious: true on a request to flag even a header-complete browser from its own signal.

The top-level tlsBlocklist (array, default empty, up to 256 entries) lists JA4 fingerprints, full or as the a / a_b prefix, that are treated as unknown automation whatever the user-agent. See TLS fingerprints.

honeypot

Decoy paths no human would visit. They're added to the generated robots.txt as Disallow, so compliant crawlers avoid them while a scraper that mines robots or follows an invisible link walks in. A hit bans the source IP for the window (block, or tarpit maze) on every path. Bans are per instance, in memory; a trustworthy client IP is required.

challenge

Tunes the proof-of-work interstitial used when unknownAction: challenge. See the challenge guide.

The interstitial also refuses to solve in a browser showing common headless-automation markers (navigator.webdriver, a HeadlessChrome user-agent, PhantomJS/Nightmare/Selenium/Playwright globals), so no pass is issued. There is no setting for this; it is always on with the challenge.

Trusted keys (Verified agents)

Each site holds a list of trusted Web Bot Auth public keys (up to 32). Each entry:

Add and remove keys in the Verified agents panel. They are distributed to connectors in the policy envelope and the bootstrap snapshot.

AI crawler categories

Enable any combination of the three category toggles. Each produces a 403 (or 402 when monetizing) for its crawlers, even if the bot is otherwise verified.

{ "mode": "enforce", "blockTraining": true, "blockAiSearch": false }
// GPTBot → 403 (reason: training) · PerplexityBot → allowed

Verified identity — Web Bot Auth

A user-agent is spoofable. Web Bot Auth (RFC 9421 HTTP Message Signatures, ed25519) lets an agent sign its requests; Aegis verifies the signature against public keys you trust, turning identity into cryptographic proof.

Setup

1. Obtain the operator's published public key(s) — a key id and a base64 ed25519 public key.
2. In Protection rules → Verified agents, keep Honour verified agent signatures on.
3. Add each key (name optional), then Save.
4. Refresh your connector's bootstrap snapshot (or wait for the next sync) so it has the keys.

Behavior

• Connectors verify signatures locally — no per-request network call. Verification is capped per request to bound CPU.
• A verified agent (reason verified_signature) may bypass the unknown-automation and rate-limit gates, but an AI-category block or licence offer still applies first — identity is proof of who, not permission.
• Verification fails closed: an unknown key, wrong tag, unsupported algorithm, expired/early window, or a signature not bound to the request host is treated as unsigned and handled by the normal user-agent rules.

Verified crawlers by IP range

Most official crawlers publish the IP ranges they crawl from. Aegis confirms a claimed crawler by matching the client IP against those ranges, so a genuine Googlebot is recognized and an impostor sending its user-agent from a random address is not.

• The console fetches the published feeds (OpenAI, Google, Bing, and others) in the background and distributes merged ranges to connectors, which match locally — no Cloudflare plan required.
• A confirmed crawler feeds the verified_bot decision (subject to allowVerifiedBots); an AI-category block still applies first.
• Matching uses your trusted client IP — supply a getClientIp resolver behind a proxy (see Client IP).
• The Verified agents panel shows how many ranges are loaded. If none load, outbound access to the feed domains may be blocked — harmless; connectors just fall back to user-agent handling.

Data licensing: licence keys and tiers

Monetize mode tells an AI crawler where to license content. Data licensing lets you sell access and let paying licensees in. Define tiers in the policy, issue a signed licence key per licensee, and every connector verifies the key locally.

• Tiers: an id, a name, the paths covered (empty = whole site) and a per-licence rate limit.
• Keys: issued in the policy console (Licensees) for a tier and 1–1095 days. They are signed with the site's own ed25519 key, whose private half stays in the console (encrypted at rest with AEGIS_ENCRYPTION_KEY). A key is shown once; the licensee sends it as Aegis-License: <key>.
• Enforcement: a valid key for a tier covering the path passes the AI-category and automation rules, limited per licence by the tier (not per IP). Anything invalid, for another site, edited, expired or revoked is ignored, and the request gets the normal rules.
• Revocation reaches connectors on their next sync. Rotating the site key doesn't affect licences.
• Metering: counts per licence per month, recorded once per connector batch (retries are never double-billed). Download them as CSV for invoicing: GET /api/sites/:id/licenses.csv.
• Discovery: with a purchaseUrl, a restricted AI crawler without a valid key is answered 402 linking to it where a tier covers the path, otherwise 403; both carry Aegis-License-Purchase: <url>. A monetize licence offer keeps precedence.

{ "licensing": { "enabled": true, "purchaseUrl": "https://example.com/ai-data",
    "tiers": [{ "id": "research", "name": "Research", "paths": ["/articles/"], "requestsPerMinute": 600 }] } }
// Licensee: curl -H "Aegis-License: aegl1.…" https://example.com/articles/42

API: GET/POST /api/sites/:id/licenses (list with usage / issue: { licensee, tier, days }, returns the key once), DELETE /api/sites/:id/licenses/:licenceId (revoke), GET /api/sites/:id/licenses.csv, POST /api/sites/:id/licensing/rotate ({ version, graceDays: 0–90, reissue }; admin; returns the new key id and any re-issued keys once).

Monetize & x402 payments

Licence handshake (402)

Turn on Offer a licence and set a licence URL. A restricted AI crawler then receives 402 Payment Required with a Link: <url>; rel="license" header instead of a 403 — a good-faith commercial signal pointing at your terms.

Collecting with x402

1. Enable x402 and set network, asset (token), pay-to wallet, price (atomic units), and a facilitator URL. Optionally set the token's EIP-712 name/version under extra.
2. A restricted AI bot receives a JSON x402 payment manifest; the Link header is still sent for non-x402 clients.
3. When an agent retries with an X-PAYMENT header, the connector calls your facilitator's /settle endpoint and lets the request through only on success. It fails closed — no facilitator, a malformed payment, a timeout, or a rejection keeps the 402.

// licence signpost
{ "monetize": true, "licenseUrl": "https://you.com/ai-licence" }
// + on-chain payment
{ "x402": { "enabled": true, "network": "base", "asset": "0xUSDC…",
            "payTo": "0xYourWallet", "maxAmountRequired": "10000",
            "facilitatorUrl": "https://facilitator.example" } }
// GPTBot → 402 + Link (and x402 manifest); access only after /settle succeeds

Agent actions

Govern AI agents that act (writes: checkouts, form posts, tool calls), not just crawlers that read.

How it decides

A request is governed by the action policy only when it is both:

• an action — its method is in methods and, if paths is set, its path is under one of them; and
• an agent — a signed agent, a known AI-crawler user-agent, or an automation signature. A normal browser user-agent is not an agent, so a human's write is never governed and falls through.

Outcome

• Authorized (a verified identity when requireSigned; on the allowedAgents list if set) → allowed (reason agent_action).
• Unauthorized → block (403) or monetize (402 / x402) per action.

{ "agentActions": { "enabled": true, "paths": ["/checkout", "/api/"],
    "requireSigned": true, "allowedAgents": ["Acme-Agent"], "action": "block" } }
// unsigned curl POST /checkout → 403 · signed "Acme-Agent" → allowed · human browser write → untouched

Browser-impersonator heuristics

A user-agent that claims to be Chrome but omits the headers every real browser sends is very likely a script in a costume. With heuristics on, such a request is treated as suspicious and answered by your chosen action: observe only, a stricter rate limit (a separate, tighter bucket that never touches normal traffic), or a block.

• Only applies to a claimed browser (a Mozilla/… UA that isn't already automation) that is not a verified/signed agent or a known AI category.
• Needs the connector to forward request headers — all bundled connectors do. The check looks for missing Accept / Accept-Language.
• With tlsConsistency on, a claimed modern browser whose JA4 TLS fingerprint offers no TLS 1.3 or no HTTP/2 first is also flagged, even with complete headers. This needs a trusted fingerprint source; see TLS fingerprints. (A runtime with its own TLS signal can also pass tlsSuspicious: true.)

The fixed automation-signature list (curl, wget, python-requests, httpx, aiohttp, Go-http-client, okhttp, Java, Guzzle, Postman, scanners, …) also feeds the ordinary unknown-automation gate. Tune via the heuristics reference.

{ "heuristics": { "enabled": true, "action": "rate-limit",
    "requestsPerMinute": 30, "burst": 15 } }
// "Chrome" UA with no Accept-Language → stricter bucket · real Chrome with full headers → untouched

TLS fingerprints (JA4)

A user-agent is text anyone can type; the TLS handshake is harder to fake. HTTP libraries build a different TLS ClientHello from real browsers, and the JA4 fingerprint summarizes it, e.g. t13d1516h2_8daaf6152771_02713d6af862. Two opt-in uses:

• Blocklist (tlsBlocklist): fingerprints you've identified as automation clients, full or as the a / a_b prefix. A match is treated as unknown automation whatever the user-agent, so it gets your unknown-automation action. Verified/signed agents and AI-category rules are unaffected.
• Fingerprints seen: connectors with a trusted JA4 source report the fingerprints they see (top 50 per batch, with the share the engine attributed to automation). Active defense lists the top ones for the last 24 hours with Add to blocklist; save the rules to apply.
• Browser consistency (heuristics.tlsConsistency): a claimed modern Chrome/Edge/Firefox/Safari whose handshake offers no TLS 1.3, or doesn't offer HTTP/2 (or HTTP/3) first, is flagged by the heuristics.
• Source of the fingerprint: only the component that terminates TLS can compute it. The Cloudflare Worker reads request.cf.botManagement.ja4 automatically (Bot Management required). Other connectors take getTlsFingerprint: cloudfrontJa4 for CloudFront's CloudFront-Viewer-JA4-Fingerprint header, or ja4FromHeader('x-your-header') for your own load balancer. Both are only safe when the origin accepts traffic from that edge alone. With no trusted source, nothing changes.

{ "unknownAction": "block",
  "tlsBlocklist": ["t13d1812h1_85036bcba153_375ca2c5e164"],
  "heuristics": { "enabled": true, "action": "allow", "tlsConsistency": true } }
// Node: aegisMiddleware({ ..., getTlsFingerprint: ja4FromHeader('x-ja4') })

Honeypot decoy paths

Configure a few decoy paths no human would visit. They're added to your generated robots.txt as Disallow, so a compliant crawler avoids them — while a scraper that mines robots for "interesting" paths, or follows an invisible link you place on your pages, walks into the trap. Any client that fetches a decoy is banned for a window (block, or tarpit maze) on every subsequent path.

• Bans are keyed by client IP and held in memory per instance (like the local rate limiter) — a trustworthy client IP is required. To share them across instances, give the connector a shared ban store: sharedBans: createSharedBans({ get, set }) over Redis or similar, or bind a KV namespace as AEGIS_BAN_KV on the Worker.
• Place an invisible link to a decoy in your markup (e.g. a visually-hidden anchor) so only crawlers that ignore robots.txt follow it.

Configure via the honeypot reference.

{ "honeypot": { "enabled": true, "paths": ["/api/internal-export"],
    "banSeconds": 3600, "action": "block" } }
// robots.txt now Disallows /api/internal-export; any client that fetches it is banned

Dynamic trap links

Static decoy paths can be learned and skipped. With honeypot.trapLinks on, connectors add a fresh hidden link to every HTML page, under your trapPrefix, carrying a token signed with the site key and bound to the IP of the client the page was served to. People never see it (hidden, aria-hidden, no tab stop, rel="nofollow", and the prefix is Disallowed in robots.txt). A crawler that follows it presents a valid token for its own IP and is banned.

• No framing. If someone copies a trap URL, for example as an <img> on another site so that visitors' browsers fetch it, the token fails from every other IP. Those requests are refused and nobody is banned. A crawler that changes IP between fetching and following is also refused without a ban (an IP ban wouldn't hold it anyway).
• Proven identities are never banned. A signed agent or verified crawler that requests a trap path gets a 403, but no ban.
• Cloudflare Worker: automatic. Allowed HTML responses are streamed through and the link is inserted before </body>, including compressed origins.
• Web-Fetch runtimes: return (await guard(req)) ?? guard.decorate(req, await origin(req));. Only uncompressed UTF-8 HTML is rewritten; anything else passes through untouched.
• Node, Fastify, Lambda: these don't own your response body, so add the link in your template: const trap = await guard.trapLink(req);, then place it before </body>.
• Needs a trustworthy client IP (the same resolver as rate limiting); without one no link is injected. Works in monitor mode too, so you can see honeypot counts before enforcing.

{ "honeypot": { "enabled": true, "trapLinks": true, "trapPrefix": "/assets/r/",
    "banSeconds": 3600, "action": "tarpit" } }
// Node: res.send(html.replace('</body>', (await guard.trapLink(req)) + '</body>'))

Shared threat intel (opt-in)

An attack on one Aegis site can harden the others. Two independent, per-site switches:

• Contribute (threatIntel.contribute): when a trap link catches a client that followed a link minted for its own IP, the connector reports that IP to the console. Nothing else is reported.
• Consume (threatIntel.consume): the connector pulls the shared list (a conditional GET each minute) and answers a listed IP with a stricter rate limit (default), a challenge, or a block. Verified agents, verified crawlers, licensees and clients that solved a challenge are never affected.
• Safeguards: an IP is shared only once at least AEGIS_THREAT_MIN_REPORTERS (default 2) different organizations reported it. Private, reserved and crawler-range addresses are refused at intake. Reports expire after AEGIS_THREAT_TTL_HOURS (default 24, max 72), and connectors stop applying a list past its expiry. Consumers never learn who reported an IP.

{ "honeypot": { "enabled": true, "trapLinks": true },
  "threatIntel": { "contribute": true, "consume": true, "action": "rate-limit", "requestsPerMinute": 30, "burst": 15 } }

Proof-of-work challenge

Set Unknown automation to Challenge (and enable it in challenge) and an unverified GET navigation is served a tiny JavaScript proof-of-work interstitial instead of a hard block. The browser searches for a value whose SHA-256 has N leading zero bits (you tune N) — a moment for a real browser, far more for a headless client — then posts it back. On success the connector sets a short-lived signed pass cookie and the visitor is let through.

• Never shown to a verified/signed agent or a client that already passed; it's off the critical path for normal browsers (they solve it once).
• Wired in every connector (Node, Fetch/Edge, Cloudflare Worker, Lambda, Fastify); a pass issued by one validates on any other for the same site.
• All challenge crypto is WebCrypto and signed with your per-site key. A genuine non-browser API client set to Challenge is effectively blocked (it can't run the JS) — the engine only challenges GET/HEAD navigations.

{ "mode": "enforce", "unknownAction": "challenge",
  "challenge": { "enabled": true, "difficulty": 16, "ttlSeconds": 300 } }
// an unknown GET gets the PoW page; solving it sets a pass cookie valid 5 min

Tarpit mode

With tarpit on, a blocked request is answered with 200 OK and a page of plausible-looking junk whose links lead only to more junk pages — wasting a non-compliant scraper's compute instead of cleanly telling it to leave.

• It only ever replaces a block. Real visitors, monetize offers, and rate-limit responses are never tarpitted, and monitor mode never serves it.
• Pages are generated deterministically per path (a scraper can't detect the trap by refetching), carry X-Robots-Tag: noindex, add no artificial delay, and are bounded in size.

robots.txt & AIPREF

Alongside enforcement, the console generates a machine-readable declaration of the same policy for compliant crawlers to read first. Find it in Connect a site → robots.txt & AI preferences and merge it into your existing robots.txt (don't replace the file).

• An AIPREF Content-Usage directive (train-ai / search), derived from your category toggles.
• A Disallow block for each restricted AI crawler — including the robots-only tokens Google-Extended and Applebot-Extended.
• When a licence URL is set, an RSL License: directive pointing at your terms.

Connectors also send the Content-Usage header on the responses they generate. This layer is advisory for compliant crawlers — the connector still enforces.

Rate limiting

A token bucket limits unverified clients per client key (IP). Defaults: 240 requests/minute refill, 120-request burst. Set them under Traffic limits. Unknown automation set to rate-limit is limited even when the general limiter is off.

Global (shared) rate limiting

The built-in limiter is per process/isolate. For a limit enforced across every instance, pass an optional sharedLimiter to the connector (Node, Fastify, Lambda, Fetch; not the Worker). It fails open (a limiter outage never blocks traffic), only enforces in enforce mode, and needs a trusted client IP. Licensed requests are exempt: a data licence is limited per licence by its tier instead of per IP.

import { aegisMiddleware, createSharedLimiter } from './aegis-node.mjs';

// Backend over Upstash Redis REST (atomic INCR + first-write EXPIRE):
const backend = {
  async incr(key, windowSeconds) {
    const call = p => fetch(`${process.env.REDIS_URL}/${p}`,
      { headers: { authorization: `Bearer ${process.env.REDIS_TOKEN}` } }).then(r => r.json());
    const { result } = await call(`incr/${encodeURIComponent(key)}`);
    if (result === 1) await call(`expire/${encodeURIComponent(key)}/${windowSeconds}`);
    return result;
  }
};
const guard = aegisMiddleware({ controlUrl, siteId, siteKey, bootstrap,
  getClientIp: trustedResolver, sharedLimiter: createSharedLimiter(backend) });

createSharedLimiter is exported from every connector bundle and works with Redis, a Cloudflare Durable Object, or any atomic counter (incr(key, windowSeconds) → count).

HTTP responses & headers

All connector-generated responses carry Cache-Control: private, no-store.

Headers Aegis sets or reads

A decorated HTML page (trap links on) has no Content-Length and a weak ETag.

Decision reasons

Every decision includes a machine-readable reason:

The decision also carries action (which now includes challenge), wouldAction (the enforce-mode outcome while monitoring), mode, retryAfter, bot (public name), license, payment (x402), tarpit, and challenge (serve the interstitial).

x402 payment manifest

A monetized request with x402 enabled returns this JSON body with the 402:

{
  "x402Version": 1,
  "error": "Payment required to access this resource",
  "accepts": [{
    "scheme": "exact",
    "network": "base",
    "maxAmountRequired": "10000",
    "resource": "https://example.com/article",
    "description": "AI access licence",
    "mimeType": "",
    "payTo": "0xYourWallet",
    "maxTimeoutSeconds": 60,
    "asset": "0xTokenContract",
    "extra": { "name": "USDC", "version": "2" }
  }]
}

The agent settles by retrying with an X-PAYMENT header (a base64 payment payload), which the connector forwards to your facilitator's /settle endpoint as { x402Version, paymentPayload, paymentRequirements }.

Agent API (connector ↔ console)

Connectors authenticate with the site key as a Bearer token. You normally never call these directly.

Limited to 600 requests/minute per site (burst 120) per console instance.

Admin API (console)

Admin endpoints use the signed session cookie and strict same-origin checks. The console UI uses them for you.

Testing

Two layers.

Layer 1 — console & simulator (no connector)

• Health: curl <console>/health.
• Set a test site to Enforce, then use the Policy simulator with a user-agent, path, and method to see the decision JSON — this covers AI-category blocks, agent actions, monetize/x402, and the tarpit flag.
• Check the generated robots.txt and the IP-range count.

Layer 2 — a local connector (real responses)

Run a small local server that uses the Node connector, pointed at this console with a test site's id/key/bootstrap, then curl it with -H "Host: your-test-hostname" and various user-agents/methods to see real 403/402/429, the tarpit maze, the x402 manifest, and the Content-Usage header.

# examples
curl -i -H "Host: test.example" -A "Mozilla/5.0"  http://localhost:3000/   # allowed
curl -i -H "Host: test.example" -A "GPTBot/1.2"   http://localhost:3000/   # 403 / 402 / tarpit
curl -i -H "Host: test.example" -A "curl/8.0" -X POST http://localhost:3000/checkout  # agent_action

Testing the newer features

Observability

Aggregate metrics reach the console in the background. Open the Overview tab to see the hourly chart, the per-reason and per-bot breakdown, and the connector heartbeat with the reported vs. saved policy version. Metrics use 24 hourly UTC buckets by receipt time, are deduplicated per site, and are best-effort (they may be lost on shutdown or failure). They confirm a reporting connector, not coverage of every instance.

Crawler-feed freshness is surfaced at /api/status (public) and on the status page — per-feed last-success and an overall operational / stale / empty reading, so you can tell when IP-range verification is degrading. Prometheus /metrics (bearer-gated) exposes request counts/latency, auth failures, rate-limit counters, and SSO logins for your own dashboards/alerts.

Troubleshooting & FAQ

My traffic isn't being blocked

• The site is in monitor mode — switch to enforce and save.
• The connector cold-started without a bootstrap and hasn't synced yet (reason unconfigured, fail-open). Add a bootstrap snapshot.
• The request's Host isn't a registered hostname (reason host_mismatch). Add it to the site.
• No connector is installed on that origin. Registering a site does not protect it.

A legitimate search bot got blocked

• Enable Allow infrastructure-verified bots, and make sure verified-bot metadata or IP ranges are available. On accounts without trusted verification, blocking unknown automation can catch search bots.

My API client / tool got blocked

• It likely presents an automation user-agent (curl, axios, …). Give it a real user-agent, add its path to excluded paths, or (for writes) a signed identity / the agent allow-list.

The connector isn't syncing

• The site key was rotated — update the connector with the new key.
• Check AEGIS_CONTROL_URL is the console's HTTPS URL and reachable from your edge/app.

A licensee's key is rejected

• Licensing is off, or the key's tier no longer exists, or the path isn't covered by the tier's paths.
• The key was issued for a different site, has expired, or was revoked.
• The header name must be exactly Aegis-License, with the whole key on one line.
• The connector hasn't synced since the first key was issued (the site's public key arrives with the policy). Wait a minute or refresh the bootstrap.

Trap links don't appear in my pages

• Honeypot and Inject trap links must both be on, and the connector needs a trusted client IP.
• Worker: only allowed 200 text/html UTF-8 responses are decorated. Fetch: only uncompressed ones via guard.decorate. Node/Fastify/Lambda: you must place await guard.trapLink(req) in your template.

TLS fingerprint rules do nothing

• No fingerprint reaches the connector: Worker needs Bot Management; others need getTlsFingerprint reading a header your edge actually sets (e.g. CloudFront's, added to the origin request policy).
• tlsConsistency lives under heuristics, which must be enabled.

Real users are rate-limited as a group

• The connector sees your proxy's IP for everyone. Fix getClientIp (Client IP) or unset it.

Policy changes take a couple of minutes

• Expected with the policy mirror (60 s edge cache + 60 s poll). Idle Workers sync only when a request arrives.

IP verification / x402 isn't working

• Both need outbound HTTPS (the console to feed domains; the connector to your facilitator). Where blocked, they degrade gracefully.
• Behind a proxy, IP-keyed checks need a trusted getClientIp.

Limits & caveats

• User-agent rules are identity hints, not proof. Web Bot Auth and IP verification harden this; adoption is still early.
• Rate limiting is per process/isolate unless you add a shared limiter; it is not volumetric DDoS protection.
• Monetize states terms and (with x402) settles a payment through a facilitator you provide — Aegis is not itself a bank or marketplace.
• robots.txt is advisory for compliant crawlers, not an access barrier.
• No public-content tool can promise to stop all scraping or prevent reuse of content already collected.
• Zero overhead is not guaranteed — measure tail latency and false positives on your own deployment before broad enforcement.
• Local rate limits and honeypot bans are per running instance, in memory, unless you add the shared limiter and the shared ban store.
• Data licence keys are bearer credentials: whoever holds one can use it until it expires or is revoked. Revocation reaches connectors on their next sync. At most 500 revoked, unexpired licences per site.
• Shared threat intel works on IPs, which many people can share (carrier NAT, VPNs); it's a rate limit by default for that reason.
• Trap links catch crawlers that follow hidden links; crawlers that render CSS or skip hidden elements won't fall in.
• JA4 checks need a trusted fingerprint source; a TLS-inspecting corporate proxy presents its own handshake.

Glossary