How to Enrich MongoDB Documents from a REST API on a Schedule

Build a Spojit workflow that runs on a schedule, finds MongoDB documents that are missing enrichment, calls a REST API for each one through a reusable subworkflow, and writes the returned data back to each document.

What This Integration Does

Most collections accumulate records that are only partially complete: a customer document has an email but no company size, an order has a postcode but no region, a lead has a domain but no firmographic data. This workflow closes that gap automatically. On a schedule it scans a MongoDB collection for documents that still lack an enrichment field, calls an external REST API for each one to fetch the missing data, and updates the document in place. You get a self-healing collection that fills itself in without anyone running a manual export-enrich-reimport cycle.

A Schedule trigger starts each run on a cron expression. A Connector node queries MongoDB for a batch of un-enriched documents, a Loop node walks that batch, and for each record a Subworkflow node calls a small reusable child workflow that does the HTTP lookup and the MongoDB write. The run leaves the matched documents enriched and flagged so the next run skips them. Because the query filters on the enrichment flag, re-runs are naturally idempotent: a document that was already enriched will not be picked up again, and a run that ends mid-batch simply continues from where it left off on the next schedule tick.

Prerequisites

A MongoDB connection added under Connections -> Add connection, scoped to a database whose credentials allow both read (find-documents) and write (update-documents).
A collection (for example customers) whose documents carry a stable identifier you can match on, such as _id or a unique business key like email.
The HTTP connector, which is built in and needs no authentication for public endpoints. If your enrichment API needs a key, store it on an HTTP connection or pass it in a request header.
The JSON connector (built in) for safely reading fields out of the API response.
One enrichment field name you will use as the "done" marker (for example enrichedAt), so the scan can tell finished documents from pending ones.

Step 1: Build the reusable enrichment child workflow

Create the child first so the parent can reference it. Make a new workflow named enrich-one-record with a Manual trigger. The data you pass into a Subworkflow node arrives as the child's trigger input, available as {{ input }}. Plan to pass one record's identifier and lookup key, for example {{ input.id }} and {{ input.email }}.

Add a Connector node in Direct mode using the HTTP connector and the http-get tool. Set url to your enrichment endpoint, templating the lookup key into the query string, and add any auth in headers:

url:     https://api.example-enrich.com/v1/lookup?email={{ input.email }}
headers: { "Authorization": "Bearer YOUR_API_KEY" }

If the API expects a body instead of query parameters, use http-post and set its body field to a JSON object. Name the node output lookup so later steps can read {{ lookup.body }}.

Step 2: Read the fields you want out of the API response

Add a Connector node in Direct mode using the JSON connector and the get tool to pull specific values out of {{ lookup.body }} by path, which keeps the workflow resilient if the response carries extra fields. For example, read the company name and employee count:

get  ->  data:  {{ lookup.body }}   path: company.name
get  ->  data:  {{ lookup.body }}   path: company.employeeCount

Name these outputs (for example companyName and employees). If the upstream shape is unpredictable, you can instead use the JSON query tool for a more expressive path, or pick to keep only the keys you care about. Keeping this extraction explicit means a schema change at the vendor surfaces here rather than silently writing wrong data into MongoDB.

Step 3: Write the enrichment back to the document

Add a Connector node in Direct mode using the MongoDB connector and the update-documents tool. Set collection to your collection name, set filter to match the single record by its identifier, and set update to a $set that writes the new fields plus the "done" marker. Leave upsert off so the node only ever updates an existing document:

collection: customers
filter:  { "_id": "{{ input.id }}" }
update:  { "$set": {
           "companyName": "{{ companyName }}",
           "employees":   {{ employees }},
           "enrichedAt":  "{{ input.runAt }}"
         } }

Writing enrichedAt in the same update is what makes the whole pipeline idempotent. The parent's scan in Step 5 excludes any document that already has this field, so a document is enriched exactly once. This child workflow is now complete; its final output (the update result with matchedCount and modifiedCount) returns to the parent. Save it.

Step 4: Create the parent workflow with a Schedule trigger

Create a second workflow named enrich-customers-nightly. Click the trigger node and set Trigger Type to Schedule. Enter a 5-field Unix cron expression and an IANA timezone. For a run every weekday at 2 AM Sydney time:

cron:     0 2 * * 1-5
timezone: Australia/Sydney

A single Schedule trigger can hold more than one schedule, so you can add a second entry if you want a midday top-up run. The trigger output is {{ scheduledAt }}, which you can reuse as the timestamp written into enrichedAt by passing it through to the child.

Step 5: Query MongoDB for a batch of un-enriched documents

Add a Connector node in Direct mode using the MongoDB connector and the find-documents tool. Set collection to your collection, and set filter to match documents that still lack the marker. Use limit to cap the batch so a single run stays predictable:

collection: customers
filter:  { "enrichedAt": { "$exists": false }, "email": { "$exists": true } }
limit:   200

The node returns { documents, count }. Name the output pending so the next step can iterate {{ pending.documents }}. Capping at a fixed limit means each scheduled run processes a bounded slice; the remaining documents are handled on subsequent ticks, and the workflow drains the backlog over a few runs without ever calling the API for an already-enriched record.

Step 6: Loop the batch and call the subworkflow per record

Add a Loop node in ForEach mode over {{ pending.documents }}; the current record is available inside the loop body as {{ item }}. Inside the loop body, add a Subworkflow node. For Workflow, pick enrich-one-record from Step 1. For Input, pass exactly the fields the child expects:

{
  "id":    "{{ item._id }}",
  "email": "{{ item.email }}",
  "runAt": "{{ scheduledAt }}"
}

Each loop iteration pauses the parent while the child runs through its own trigger and nodes, then resumes with the child's output. Children appear as their own entries in execution history, so you can open any single record's enrichment run to see its HTTP call and MongoDB write in isolation. Save and enable the parent workflow.

Tips

Keep the find-documents limit modest at first (50 to 200). It bounds both your API spend and the run duration, and the next schedule tick simply picks up the rest.
Index the field you filter on (the enrichment marker, and your lookup key) in MongoDB so the scan stays fast as the collection grows.
Put the per-record HTTP call and write in the Subworkflow rather than inline. Because changes to a child take effect immediately for every parent, you can swap enrichment providers or add a retry without touching the scheduler.
If a single enrichment source is slow, you can fan several records out at once by wrapping the subworkflow call in a Parallel node instead of a serial Loop, but watch the vendor's rate limits.

Common Pitfalls

Forgetting the "done" marker. If the child does not write enrichedAt, the parent's filter will keep re-selecting the same documents every run and you will pay for the same API call repeatedly. The marker is what makes re-runs safe.
Leaving upsert on in update-documents. With upsert enabled, a filter that matches nothing creates a stray document. Keep it off so the node only ever updates records that already exist.
Type mismatches in _id filters. If your identifiers are MongoDB ObjectIds rather than plain strings, match on a string business key like email instead, or store a stable string id, so the templated filter compares like with like.
Timezone drift on the cron. The Schedule trigger interprets the cron in the IANA timezone you set, not UTC. Confirm the timezone, especially around daylight-saving changes, so the job fires when you expect.
Unhandled API errors. A 4xx or 5xx response from http-get can surface as an empty extraction. Read only the fields you need with the JSON connector and consider a Condition node in the child to skip the write when the lookup came back empty.

Testing

Before enabling the schedule, validate the child workflow on its own: open enrich-one-record, press Run on its Manual trigger, and supply one real record's id and email as the request body. Confirm the HTTP call returns the expected JSON, the JSON extraction reads the right values, and MongoDB reports modifiedCount: 1. Then point the parent at a tiny slice by temporarily setting the find-documents limit to 1 and adding a narrow filter (for example a single test record's email), run the parent manually, and check that exactly one document gains companyName, employees, and enrichedAt. Once a single record flows end to end, raise the limit, widen the filter, and enable the Schedule trigger. If a run misbehaves, ask Miraxa, the intelligent layer across your automation, "Why did my last run fail?" to investigate the execution.