How to Index Web Pages Into a Knowledge Collection From a Webhook

Build a Spojit workflow where a webhook hands you a page URL, the Knowledge node embeds that page into a persistent collection, and Slack confirms the page was indexed.

What This Integration Does

Teams constantly find pages worth keeping: a new help article, a competitor's pricing page, a policy update, a release note. This workflow lets any system or person hand Spojit a URL through a single HTTP request, and Spojit captures that page into a searchable Knowledge collection without anyone uploading a file. Once a page lives in a persistent collection, every other workflow in your workspace can query it with the Knowledge node in Query mode, so your retrieval-augmented answers stay current.

The workflow is triggered by a Webhook trigger: an external HTTP POST carries a JSON body containing the page URL (and optionally a name and a tag). Spojit parses the body, the Knowledge node fetches and embeds the page as a Web Page URL document into a named persistent collection, and a Connector node posts a confirmation to Slack. The webhook returns 202 with an executionId immediately, then the run continues asynchronously. Re-sending the same URL with the same file name overwrites the existing document, so re-runs refresh the page rather than creating duplicates.

Prerequisites

A persistent Knowledge collection to write into. Create one from the Knowledge section of the sidebar with New Collection, and note its name and embedding model. See Creating a Knowledge Collection.
A signing connection for the webhook trigger (scheme Spojit, Custom, or another supported scheme) to verify incoming requests. See Setting Up a Webhook Connection.
A Slack connection with permission to post messages, and the target channel ID or name. See the Slack connector article.
The sender that will call the webhook (a script, form, or another tool) able to send a JSON POST with the page URL.

Step 1: Add a Webhook trigger that carries the URL

Add a Trigger node and set its type to Webhook. Pick your signing connection so Spojit can verify each request. Copy the generated webhook URL and configure your sender to POST a JSON body like this:

{
  "url": "https://example.com/blog/whats-new",
  "name": "whats-new.html",
  "tag": "release-notes"
}

The parsed body is available downstream as {{ input }}, so the page address is {{ input.url }}, the file name is {{ input.name }}, and your optional tag is {{ input.tag }}. The trigger responds to the caller with 202 and an executionId right away; the rest of the workflow runs asynchronously.

Step 2: Validate the incoming URL

Add a Connector node in Direct mode on the validation connector and choose the url tool. Map its input to {{ input.url }}. This rejects malformed or empty values before you spend an embedding step on them. Follow it with a Condition node that continues only when the value is a valid URL; route the invalid branch to a Send Email node or a Slack message so a bad request never fails silently. Validating up front keeps your collection free of junk documents.

Step 3: Embed the page into your persistent collection

Add a Knowledge node and set its mode to Embed. Configure the fields:

Collection: select your persistent collection by name (not Transient, because you want the page to live on for other workflows to query).
Document Type: choose Web Page URL. With this type, Spojit fetches the page at the supplied address and embeds its readable content.
Document Input: set this to the page address, {{ input.url }}.
File Name: set this to {{ input.name }} (required for a persistent collection). Reusing the same name overwrites the existing document, which is how a re-run refreshes a changed page.
Embedding Model: leave at the collection default. Always embed with the same model the collection was created with.
Output Variable: name it, for example, embedResult. It returns the chunk count and document metadata you will reference next.

On completion the page is chunked, embedded, and stored under that file name in the collection.

Step 4: Build a confirmation message

Add a Transform node to assemble a short, readable confirmation string from the embed result and the original request. For example, compose a message such as:

Indexed "{{ input.name }}" from {{ input.url }}
into the Knowledge collection ({{ embedResult.chunkCount }} chunks).

Keeping the message assembly in its own step makes the Slack call in the next step a clean, single mapping and makes the text easy to adjust later. Reference the chunk count from your Knowledge output variable so the confirmation reflects what was actually stored.

Step 5: Confirm indexing in Slack

Add a Connector node in Direct mode on the slack connector and choose the send-message tool. Map its inputs:

Channel: your target channel ID or name (for example #knowledge-feed).
Text: the confirmation string built in Step 4, or inline it as Indexed {{ input.name }} into the knowledge collection.

When the run reaches this step, your team sees a Slack note that the page is now searchable. If you would rather notify by email instead of Slack, swap this node for a Send Email node, which sends from Spojit's built-in mail service with no connection needed.

Step 6: Make the page queryable (verification)

To prove the page is retrievable, add a second Knowledge node set to Query mode pointing at the same persistent collection. Set its Prompt to a question the page should answer and a Result Count of 3, then inspect the output variable in the execution log. Because the collection is persistent, you can also run this query from any other workflow in your workspace. For more on this pattern, see Querying Your Knowledge Base.

Tips

Pass a stable name per source page (for example a slug-based file name). Reusing the same name overwrites the prior version, so re-sending a URL keeps one fresh copy instead of piling up duplicates.
Use one persistent collection per topic area so queries stay focused. The collection is workspace-scoped, so any workflow can read it once a page is indexed.
If you only need a page for a single run (embed, query, discard), pick Transient as the collection instead; here you want persistent so the page survives.
Ask Miraxa, the intelligent layer across your automation, to scaffold the canvas with a prompt like "Build a webhook workflow that embeds {{ input.url }} as a Web Page URL document into my Knowledge collection and posts a Slack confirmation," then fine-tune each node in the properties panel.

Common Pitfalls

Embedding with a different model than the collection was created with breaks retrieval. Leave the Embedding Model at the collection default for both embed and query.
Pages behind a login, paywall, or heavy client-side rendering may yield little or no readable text. Test with the exact URLs you expect, and validate the value first with the validation connector's url tool.
The webhook returns 202 before embedding finishes, so the caller cannot read the indexing result from the HTTP response. Use the Slack confirmation (or the execution log) to confirm success, not the webhook reply.
Webhook senders sometimes retry, delivering the same URL twice. Enable opt-in dedup via the event-id header on the trigger, or rely on the overwrite-by-file-name behavior so retries refresh rather than duplicate.

Testing

Before pointing real traffic at the workflow, send one POST to the webhook URL with a single known public page in the body and watch the execution log. Confirm the Webhook trigger parsed {{ input.url }}, the Knowledge node reported a non-zero chunk count, and the Slack message arrived. Open the collection in the Knowledge section and check the document table shows your file name with status READY. Then run the Step 6 query to confirm the page is retrievable. Once a single page round-trips end to end, enable the sender to deliver real URLs.