How to Keep a Persistent Product Catalog Collection in Sync from FTP Drops

On a schedule, download catalog spec sheets from an FTP server and embed them into a long-lived Knowledge collection that any of your Spojit workflows can query for product details.

What This Integration Does

Many suppliers and ERP systems publish product spec sheets, price lists, and catalog documents as files dropped onto an FTP server. This tutorial turns that drop folder into a continuously refreshed, searchable product catalog. Each run picks up the latest documents, embeds them into a single persistent Knowledge collection, and posts a short summary to Slack so your team knows the catalog is current. Once the collection is populated, any other workflow you build can ask natural-language questions like "What is the carton quantity for SKU AB-1200?" without re-reading the source files.

The workflow is driven by a Schedule trigger, so it runs unattended on a cron cadence you choose. On each run it lists the FTP drop folder, downloads the matching files, and feeds each one into a Knowledge node in Embed mode targeting a named, workspace-scoped collection. Because the collection is persistent (not Transient), the embedded documents survive after the run finishes and accumulate across runs. Re-running is safe: embedding a document under a file name that already exists overwrites that document in place, so updated spec sheets replace their older versions rather than creating duplicates.

Prerequisites

An FTP connection added under Connections -> Add connection, with read access to the folder where catalog files land.
A Slack connection for the summary message, plus the channel ID (or name) you want to post to.
A persistent Knowledge collection created ahead of time in the Knowledge section of the sidebar (New Collection). Give it a clear name such as product-catalog and note the embedding model you pick; the embedding model is fixed at creation and must stay the same for every embed and every query against this collection.
Knowing the exact remote folder path of the drop (for example /catalog/specs) and the file types you expect (PDF spec sheets, Excel price lists, CSV exports, and so on).

Step 1: Start the workflow on a Schedule trigger

Add a Trigger node and set its type to Schedule. Enter a 5-field Unix cron expression and an IANA timezone. For example, to run every weekday morning at 6am Sydney time, use 0 6 * * 1-5 with timezone Australia/Sydney. A single Schedule trigger can hold more than one schedule if you want both a morning and an evening refresh. The trigger output is { scheduledAt }, which you can reference later as {{ scheduledAt }} in messages.

Step 2: List the FTP drop folder

Add a Connector node in Direct mode, choose your FTP connection, and pick the list-directory tool. Set path to your drop folder, for example /catalog/specs. The tool returns the folder listing under data.entries, where each entry includes its name, type, size, and modification date. Set the node's output variable to something like listing so downstream steps can read {{ listing.data.entries }}.

path: /catalog/specs

Step 3: Loop over each catalog file

Add a Loop node in ForEach mode and point it at {{ listing.data.entries }}. Inside the loop body you will reference the current entry as {{ item }}. To skip subfolders, add a Condition node as the first step inside the loop that only continues when {{ item.type }} equals file. You can add a second condition to limit by extension (for example, only continue when {{ item.name }} ends with .pdf or .xlsx) so you do not try to embed unrelated files.

Step 4: Download each file as base64

Still inside the loop body, add a Connector node in Direct mode using your FTP connection and the download-file tool. Set path to the full remote path of the current file and set encoding to base64 so binary spec sheets and spreadsheets transfer intact. Name the output variable file.

path: /catalog/specs/{{ item.name }}
encoding: base64

The tool returns data.content (the base64 file bytes), along with data.encoding and data.size. You will feed {{ file.data.content }} straight into the Knowledge node next.

Step 5: Embed the document into your persistent collection

Add a Knowledge node in Embed mode inside the loop body. Configure it as follows:

Collection: pick your persistent collection, product-catalog (do not pick Transient: Transient collections are wiped at the end of each run, and you want this catalog to persist).
File Name: set it to {{ item.name }}. Because embedding under an existing file name overwrites that document, reusing the source file name keeps each spec sheet to exactly one entry that refreshes whenever the supplier publishes a new version.
Document Type: match the file. Use PDF for spec sheets, Excel for price lists, or CSV/TSV for exports. If your drop mixes types, drive this from your extension condition in Step 3.
Document Input: set it to the base64 reference {{ file.data.content }}.
Embedding Model: leave it on the model the collection was created with. The same model must be used for every embed and query of this collection.
Output Variable: name it embedResult so you can read the chunk count it returns.

Step 6: Post a refresh summary to Slack

After the loop completes, add a Connector node in Direct mode using your Slack connection and the send-message tool. Set channel to your team channel and compose text from the run details so the channel sees that the catalog is current.

channel: C0123CATALOG
text: Product catalog refreshed at {{ scheduledAt }}. Processed {{ listing.data.entries.length }} files from /catalog/specs.

If you prefer email over Slack, you can instead add a Send Email node, which sends from Spojit's built-in mail service with no extra connection. Either way, the catalog collection is now ready for any workflow to query.

Step 7: Query the catalog from any workflow

In a separate workflow (or later in this one), add a Knowledge node in Query mode, choose the same persistent product-catalog collection, and supply a natural-language Prompt. Set a Result Count (default 5) and, if you want machine-readable output, attach a Response Schema to force structured JSON. Because the collection is workspace-scoped, the catalog you built here is available to every workflow in your workspace.

Prompt: What is the carton quantity and case weight for SKU {{ order.sku }}?

Tips

Always download with encoding: base64 for PDFs, Excel, and other binary documents; UTF-8 will corrupt them. Plain CSV or text files can use UTF-8 if you prefer.
Reusing the source file name as the Knowledge File Name is what makes re-runs idempotent: updated spec sheets overwrite their prior version instead of piling up duplicates.
You can give one Schedule trigger several schedules (for example a morning and an evening refresh) rather than building two workflows.
Open the collection in the Knowledge section to watch document status move from PROCESSING to READY and to confirm chunk counts after a run.

Common Pitfalls

Picking Transient by mistake. A Transient collection is auto-cleaned when the run ends, so nothing persists. For a long-lived catalog you must select the named persistent collection.
Mismatched embedding models. The embedding model is fixed when the collection is created. If you embed with one model and query with another, results degrade. Keep every embed and query on the same model.
Embedding folders or junk files. Without the Step 3 condition on {{ item.type }} and file extension, the Loop will try to download and embed directories or unrelated files. Filter before downloading.
Cron timezone surprises. The Schedule trigger uses the IANA timezone you set, not your browser's. A run scheduled for 0 6 * * 1-5 fires at 6am in that timezone, which can shift across daylight-saving changes.

Testing

Before turning the schedule loose, validate on a small scope. Point the list-directory path at a test folder holding two or three sample spec sheets, run the workflow once with the Run button, and watch the execution log for each loop iteration. Open the product-catalog collection in the Knowledge section and confirm the expected documents show status READY with sensible chunk counts. Then add a quick Knowledge Query node and ask a question whose answer you already know from one of the sample files. Once the answer is correct, widen the FTP path to the real drop folder and let the Schedule trigger take over. If you get stuck, open Miraxa, the intelligent layer across your automation, and ask it to explain a step or investigate a failed run in context.