How to Load XML Feed Data into MongoDB on a Schedule

On a fixed schedule, this Spojit workflow fetches a remote XML feed over HTTP, converts it to JSON, reshapes each record, and inserts the results into a MongoDB collection so your data stays current without anyone touching it.

What This Integration Does

Plenty of suppliers, marketplaces, and legacy systems still publish their data as an XML feed sitting behind a URL: product catalogs, price lists, inventory snapshots, syndicated content. Re-importing that feed by hand is tedious and error-prone. This workflow does the import for you on a timetable, pulling the latest feed, turning its nested XML into clean JSON records, and writing those records into a MongoDB collection you can query from dashboards, other workflows, or reporting tools.

The workflow is driven by a Schedule trigger, so it runs on a cron expression in a timezone you choose (for example every morning before business hours). On each run it fetches the feed fresh over HTTP, converts and normalizes the payload, then inserts the records into MongoDB. Because each run reads the live feed and writes to the same collection, re-runs simply append the newest snapshot. To keep the collection from accumulating duplicates across runs, this tutorial shows a clear-and-reload pattern using the MongoDB delete-documents tool before inserting, so the collection always reflects the latest feed.

Prerequisites

A MongoDB connection added in Spojit under Connections → Add connection, pointing at the database where you want the records stored. See the guide on adding a new connection if you have not set one up yet.
The public (or authenticated) URL of the XML feed you want to import.
If the feed requires authentication, the API key, token, or header it expects, so you can supply it in the HTTP request.
A target collection name in mind (for example catalog_feed). MongoDB creates the collection automatically on first insert.
A rough idea of the feed's element structure (the repeating record element and the fields inside it) so you can map them correctly.

Step 1: Add a Schedule trigger

Open the Workflow Designer and add a Trigger node, then set its type to Schedule. A Schedule trigger uses a 5-field Unix cron expression plus an IANA timezone. For a daily 6 AM import in Sydney, set the cron to 0 6 * * * and the timezone to Australia/Sydney. A single trigger can hold more than one schedule if you want multiple runs per day. The trigger output is { scheduledAt }, which you can reference downstream as {{ scheduledAt }} if you want to stamp each record with the run time. For more detail, see the guide on setting up a Schedule trigger.

Step 2: Fetch the XML feed with an HTTP Connector node

Add a Connector node in Direct mode, choose the http connector, and select the http-get tool. Set url to your feed address, for example https://feeds.example.com/catalog.xml. If the feed needs authentication, add the required header under headers as a key-value pair (for example Authorization with your token value). Optionally raise timeout for large feeds. Name the step result something memorable such as feed. The http-get output is:

{
  "data": "<catalog>...</catalog>",
  "status": 200,
  "statusText": "OK",
  "headers": { "content-type": "application/xml" }
}

The raw XML string is available downstream as {{ feed.data }}.

Step 3: Convert XML to JSON with the xml connector

Add another Connector node in Direct mode, choose the xml connector, and select the to-json tool. Map the xml input to the feed body from the previous step: {{ feed.data }}. The tool parses the XML into a JSON object that mirrors the feed's element tree. If your feed leans on XML attributes (such as <item id="123">), the attribute keys are prefixed by default with @_ (configurable via attributePrefix), and text content lands under #text (configurable via textNodeName). Name this result parsed. You now have structured JSON at {{ parsed }} shaped like the feed, typically with a repeating array nested inside, for example {{ parsed.catalog.product }}.

Step 4: Reach the array of records and reshape them

The records you want to insert are almost always a nested array inside the parsed object. Add a Connector node in Direct mode using the json connector. To pull out the repeating element by path, use the get tool with data set to {{ parsed }} and path set to the dot-notation location of your records, such as catalog.product. Name the result records; the array is then available as {{ records.value }}.

If the field names from the feed do not match the schema you want stored, add a Transform node to reshape each record into your target shape (for example renaming @_id to sku and casting price strings to numbers). Keep the output as an array of plain objects ready for insertion. For guidance on shaping data between steps, see using Transform nodes in structured mode.

Step 5: Clear the previous snapshot (optional reload pattern)

To avoid duplicating records every run, add a Connector node in Direct mode using the mongodb connector and the delete-documents tool. Set collection to your target collection (for example catalog_feed) and provide a filter that matches everything you want replaced. To clear the whole collection each run, use an empty filter:

{
  "collection": "catalog_feed",
  "filter": {}
}

If you would rather keep history, skip this step and instead include a unique key on each record so you can deduplicate later with update-documents in upsert style. The clear-and-reload approach above is the simplest way to keep the collection equal to the latest feed.

Step 6: Insert the records into MongoDB

Add a final Connector node in Direct mode using the mongodb connector and the insert-documents tool. Set collection to the same collection name and map documents to your reshaped array, for example {{ records.value }} (or the output of your Transform step). The documents input expects an array of objects:

{
  "collection": "catalog_feed",
  "documents": "{{ records.value }}"
}

The tool returns { insertedCount }, so after the run you can confirm how many records landed. MongoDB creates the collection on the first insert if it does not already exist.

Step 7: Save, enable, and let the schedule run it

Save the workflow and enable it so the Schedule trigger becomes active. From here on, Spojit fires the workflow at each scheduled time, fetches the feed, converts and reshapes it, optionally clears the previous snapshot, and inserts the latest records. You can review each run, including the insertedCount, in the execution history; see monitoring workflow executions for where to find it.

Tips

Single-element feeds can be a trap: when a feed has only one record, some XML parsers return an object instead of an array. After to-json, you can use the json query tool with a JSONPath like $.catalog.product[*], which always returns a collection, to normalize one-or-many into a consistent array.
For very large feeds, insert in batches by wrapping the insert step in a Loop node over chunks of {{ records.value }} rather than sending tens of thousands of documents in one call.
Stamp each record with {{ scheduledAt }} from the trigger or a value from the date connector so you can tell which run produced which snapshot.
If you are unsure how to wire the nodes together, ask Miraxa, the intelligent layer across your automation, something like "Add a Connector node that runs the xml connector's to-json tool on {{ feed.data }} and connect it after the HTTP step." Miraxa can scaffold the canvas and then you fine-tune fields in the properties panel.

Common Pitfalls

Wrong record path: if insert-documents reports zero or oddly shaped documents, your json get path probably does not match the feed. Inspect {{ parsed }} in a test run and confirm the exact dot-notation path to the repeating element.
Attribute prefixes: values that live in XML attributes appear under keys like @_id after conversion. Map those explicitly in your Transform step, or set a cleaner attributePrefix on the to-json tool.
Duplicate growth: without the clear step or a unique upsert key, every run appends another full copy of the feed. Decide on clear-and-reload or upsert before you enable the workflow.
Timezone surprises: a cron of 0 6 * * * means 6 AM in the IANA timezone you set, not server time. Double-check the timezone field so the import lands when you expect.
Feed downtime: if the source returns a non-200 status, {{ feed.status }} will reflect it. Add a Condition node to stop before inserting when the status is not 200, so a bad fetch never wipes and reloads with empty data.

Testing

Before enabling the schedule, validate end to end on a small scope. Temporarily point the HTTP step at a small sample of the feed (or a feed with only a handful of records) and use a throwaway collection name such as catalog_feed_test. Run the workflow once with the Run button, then check each step's output in the execution log: confirm {{ feed.status }} is 200, that {{ parsed }} has the structure you expect, that {{ records.value }} is an array of clean objects, and that the final step's insertedCount matches the number of records. Use the MongoDB find-documents tool (or your database client) to spot-check a few inserted documents. Once the small run looks right, switch back to the real feed and collection, then enable the Schedule trigger.