Knowledge

Knowledge bases provide context to your agent’s AI models. They allow your agent to answer questions based on documents, websites, or other data sources.

Creating a knowledge base

Create a knowledge source in src/knowledge/:

import { Knowledge, DataSource } from "@botpress/runtime";

export default new Knowledge({
  name: "documentation",
  description: "Product documentation",
  sources: [
    // Add knowledge sources
  ],
});

Data sources

You can add knowledge from several different data sources:

Website source

There are multiple ways to index websites:

From a sitemap

Index a website using a sitemap XML file:

import { Knowledge, DataSource } from "@botpress/runtime";

const WebsiteSource = DataSource.Website.fromSitemap(
  "https://example.com/sitemap.xml",
  {
    filter: ({ url }) => !url.includes("/admin"),
    maxPages: 1000,
    maxDepth: 10,
  }
);

export default new Knowledge({
  name: "website-docs",
  sources: [WebsiteSource],
});

From a base URL

Crawl a website starting from a base URL (requires the Browser integration):

import { Knowledge, DataSource } from "@botpress/runtime";

const WebsiteSource = DataSource.Website.fromWebsite(
  "https://example.com",
  {
    filter: ({ url }) => !url.includes("/admin"),
    maxPages: 500,
    maxDepth: 5,
  }
);

export default new Knowledge({
  name: "website-docs",
  sources: [WebsiteSource],
});

From llms.txt

Index pages referenced in an llms.txt file:

import { Knowledge, DataSource } from "@botpress/runtime";

const WebsiteSource = DataSource.Website.fromLlmsTxt(
  "https://example.com/llms.txt"
);

export default new Knowledge({
  name: "website-docs",
  sources: [WebsiteSource],
});

From specific URLs

Index a specific list of URLs:

import { Knowledge, DataSource } from "@botpress/runtime";

const WebsiteSource = DataSource.Website.fromUrls([
  "https://example.com/page1",
  "https://example.com/page2",
  "https://example.com/page3",
]);

export default new Knowledge({
  name: "website-docs",
  sources: [WebsiteSource],
});

Website source options

Option	Type	Description
id	string	Optional unique identifier for the source
filter	(ctx) => boolean	Filter function receiving { url, lastmod?, changefreq?, priority? }
fetch	string | function	Fetch strategy: 'node:fetch' (default), 'integration:browser', or custom function
maxPages	number	Maximum pages to index (1-50000, default: 50000)
maxDepth	number	Maximum sitemap depth (1-20, default: 20)

File source

Index files from a local directory (development only):

const FileSource = DataSource.Directory.fromPath("./src/knowledge/docs", {
  filter: (filePath) => filePath.endsWith(".md") || filePath.endsWith(".txt"),
});

export default new Knowledge({
  name: "local-docs",
  sources: [FileSource],
});

Directory sources only work during development (adk dev). For production, use website sources or upload files to Botpress Cloud.

Directory source options

Option	Type	Description
id	string	Optional unique identifier for the source
filter	(filePath: string) => boolean	Filter function to include/exclude files

Using knowledge in conversations

Provide knowledge bases to your conversation handlers:

import { WebsiteKB } from "../knowledge/docs";

export default new Conversation({
  channel: "*",
  handler: async ({ execute }) => {
    await execute({
      instructions: "You are a helpful assistant.",
      knowledge: [WebsiteKB],
    });
  },
});

Searching knowledge

You can search a knowledge base directly using the search() method:

import { DocsKB } from "../knowledge/files";

const results = await DocsKB.search("How do I deploy my agent?");

for (const passage of results.passages) {
  console.log(passage.content);
  console.log(passage.metadata.url);
}

The search() method accepts optional options:

const results = await DocsKB.search("deployment", {
  contextDepth: 4,  // Surrounding passages for context (1–20, default: 4)
  limit: 10,        // Max passages to return (1–50, default: 20)
});

In most cases you don’t need to call search() directly. When you pass a knowledge base to execute() via the knowledge prop, the AI model automatically searches it as needed.

Refreshing knowledge

Refresh knowledge bases to update their content:

import { Workflow } from "@botpress/runtime";
import WebsiteKB from "../knowledge/docs"

export default new Workflow({
  name: "refresh-knowledge",
  schedule: "0 0 * * *", // Daily
  handler: async () => {
    await WebsiteKB.refresh();
  },
});

Force refresh

Force re-indexing of all content even if unchanged:

await WebsiteKB.refresh({ force: true });

Refresh a specific source

Refresh a single data source within a knowledge base:

await WebsiteKB.refreshSource("my-source-id", { force: true });

Reference

Knowledge props

Show Knowledge props

name · string · required

Unique name for the knowledge base.

description · string

Optional description of what the knowledge base contains.

sources · DataSource[] · required

Array of data sources to index. Can include Website or Directory sources.

Knowledge methods

Show Knowledge methods

search · (query: string, options?) => Promise<{ passages }>

Search the knowledge base for relevant passages. Options: contextDepth (1–20, default 4), limit (1–50, default 20). Returns passages with content and metadata (url, title, file, description, etc.).

refresh · (options?) => Promise<void>

Refresh the knowledge base by re-indexing all sources. Pass { force: true } to re-index even if unchanged.

refreshSource · (dsId: string, options?) => Promise<void>

Refresh a specific data source by its ID. Pass { force: true } to force re-indexing.