Skip to main content

SEO & robots

robots.txt

/robots.txt is always served: permissive (Allow: /), with a Sitemap: line pointing at /sitemap-index.xml and a Content-Signal directive on the User-agent: * block. Content-Signal states how crawlers may use your content, independent of whether they may fetch it. Three flags, each defaulting to true (emitted as yes):

Flag Directive Meaning
search search Use the content to build a search index
aiTrain ai-train Use the content to train an AI model
aiInput ai-input Use the content as AI input (RAG, grounding, live answers)

Set any to false to emit no. For example, to allow search indexing but opt out of AI training and retrieval:

docsTheme({
  // ...
  robots: { contentSignal: { aiTrain: false, aiInput: false } },
});

produces:

User-agent: *
Content-Signal: search=yes, ai-train=no, ai-input=no
Allow: /

Sitemap: https://your-site/sitemap-index.xml

Content-Signal is a preference expressed to crawlers, not an access control; it does not block fetching.

SEO and LLM Endpoints

When docs is configured in the integration, four endpoints are auto-generated:

Endpoint Description
/llms.txt Structured index: project name, description, per-doc ## sections
/llms-full.txt All docs concatenated into a single markdown file
/[slug].md Individual markdown for each doc (and extra entries with body)
/sitemap-index.xml Standard sitemap via @astrojs/sitemap

Docs are sorted by the order field in frontmatter. Frontmatter and import statements are stripped from the output.

When docs.extraEntries is configured, those entries are appended after docs in llms.txt and llms-full.txt. Entries with a body field also get individual /[id].md routes.

Extra Entries

Each entry has id (URL path segment), title, description, order, and optional body (markdown). Pass a static array for simple cases, or a module path for dynamic data:

// src/extra-entries.ts
import type { ExtraEntry } from "astro-pigment";
import { getCollection } from "astro:content";

export default async function (): Promise<ExtraEntry[]> {
  const examples = await getCollection("examples");
  return examples.map((ex) => ({
    id: `examples/${ex.id}`,
    title: ex.data.title,
    description: ex.data.description,
    order: 101,
  }));
}

The module must default-export an ExtraEntry[] or a function returning Promise<ExtraEntry[]>.

Theme copied