How to Batch a Large Dataset into Chunks and Process Each Chunk in Parallel

Pull thousands of rows from MySQL, split them into evenly sized chunks with the array chunk tool, then loop over the chunks and fan out Parallel branches that push each batch to an API without overwhelming it.

What This Integration Does

When you need to move a large dataset from a database into a downstream system, sending one record at a time is slow and sending everything at once usually trips rate limits or timeouts. This Spojit workflow strikes the balance: it reads a big result set from MySQL once, divides it into fixed-size chunks (for example 50 rows each), and then walks through those chunks one at a time. Inside each chunk it uses a Parallel node to fire several HTTP requests concurrently, so a batch of records is delivered in a fraction of the time a sequential push would take, while the overall throughput stays inside a ceiling you control.

The workflow runs on a Schedule trigger, so it executes unattended on a cron cadence (for example every weekday morning). On each run it queries MySQL for the rows that are due for processing, the array chunk tool returns an items array of chunk arrays plus a count, a Loop in ForEach mode iterates those chunks, and a Parallel branch set sends each chunk's records to the target API. Because the run starts from a query, re-runs are naturally idempotent as long as your query only selects rows that still need work (filter on a status column and mark rows as processed at the end of each batch), so a retry will not double-send the same records.

Prerequisites

A MySQL connection added under Connections, with read access to the source table and (recommended) write access so you can mark rows as processed.
The destination API reachable over HTTPS. The built-in http connector needs no connection; if the API requires an auth header, store the value as a workspace variable or pass it in the request headers.
A column you can filter on to identify rows that still need processing (for example sync_status = 'PENDING') and, ideally, a primary key to update afterward.
A sense of the API's rate limit so you can pick a safe chunk size and branch width.

Step 1: Add a Schedule trigger

Add a Trigger node and set Trigger Type to Schedule. Enter a 5-field Unix cron expression and an IANA timezone. For a weekday 9am run in Sydney, use:

0 9 * * 1-5
Australia/Sydney

The trigger output is { scheduledAt }. A single Schedule trigger can hold multiple cron entries if you want the same workflow to run at more than one time.

Step 2: Pull the dataset from MySQL

Add a Connector node in Direct mode, choose the MySQL connector, and select the execute-query tool. Put your selection in the query field and use the params array for any bound values so the query is parameterized rather than string-concatenated. Cap the result with a LIMIT so a single run stays bounded:

SELECT id, email, full_name, payload
FROM customers
WHERE sync_status = 'PENDING'
ORDER BY id
LIMIT 5000

Save the result to a variable such as rows. You now have the full set of records to batch, available as {{ rows.data }}.

Step 3: Split the rows into chunks

Add a Connector node in Direct mode, choose the array connector, and select the chunk tool. Map the array input to your row list (for example {{ rows.data }}) and set size to the number of records per batch, such as 50. The tool returns items (an array where each element is itself an array of up to 50 records) and count (how many chunks were produced). Save the output to a variable such as batches. You will iterate {{ batches.items }} in the next step.

Step 4: Loop over the chunks in order

Add a Loop node in ForEach mode and point it at {{ batches.items }}. Each iteration exposes the current chunk (an array of records) as the loop item, which you can reference inside the loop body, for example {{ batch }}. Processing chunks sequentially in the Loop while parallelizing inside each chunk is the key to the pattern: it keeps a steady, predictable load on the API rather than launching every request at once. The records inside the current chunk are what the next step distributes across branches.

Step 5: Fan out a chunk's records with a Parallel node

Inside the Loop body, add a Parallel node so the records in the current chunk are sent concurrently. Add one branch per concurrent slot you want (for example 5 branches for up to 5 in-flight requests). In each branch place a Connector node in Direct mode using the http connector and the http-post tool, mapping the branch's record into the request body. A single record payload looks like:

{
  "url": "https://api.example.com/v1/contacts",
  "headers": { "Authorization": "Bearer {{ apiToken }}", "Content-Type": "application/json" },
  "body": {
    "email": "{{ record.email }}",
    "name": "{{ record.full_name }}",
    "data": "{{ record.payload }}"
  }
}

If your chunk size is larger than your branch count, add a nested Loop inside each branch to walk that branch's slice of the chunk. The Parallel node completes when all branches finish, so the Loop only advances to the next chunk once the current batch is fully delivered. This gives you a hard upper bound on concurrent requests equal to your branch count.

Step 6: Mark the batch as processed

Still inside the Loop body (after the Parallel node), add a Connector node in Direct mode on the MySQL connector. Use the update-rows tool to flip the status of the records you just sent so the next scheduled run does not pick them up again. Filter on the IDs from the current chunk and set the status column, for example to SYNCED. Doing this per chunk (rather than once at the very end) means that if a later chunk fails, the chunks you already delivered stay marked and are not re-sent on the next run.

Step 7: Report the outcome

After the Loop, add a Send Email node to summarize the run. Set Recipients to your ops address, a templated Subject such as Batch sync: {{ batches.count }} chunks processed, and a short Body noting how many chunks ran. Send Email uses Spojit's built-in mail service, so no connection is needed; just confirm external recipients are on the org allowlist under Settings > General > Email recipients.

Tips

Tune two dials together: chunk size (records per batch) and Parallel branch count (concurrent requests). Lower branch count or smaller chunks if you see the API return 429 responses.
Keep the MySQL LIMIT in step 2 sane so a single run finishes in a reasonable window. If your backlog is huge, let the schedule chip away at it run by run rather than pulling everything at once.
Use the params array on execute-query for any dynamic filter values instead of building the SQL string by hand, which keeps the query parameterized.
If you need Miraxa to scaffold this, try a prompt like "Add a Loop in ForEach mode over {{ batches.items }} and put a Parallel node with 5 branches inside it." Miraxa is the intelligent layer across your automation and can add and connect these nodes on the canvas, then you fine-tune fields in the properties panel.

Common Pitfalls

Forgetting to mark rows as processed. If your query in step 2 does not filter on a status column that step 6 updates, every scheduled run re-sends the same records. Always pair a status filter with a status update.
Setting chunk size too high. A large chunk combined with many branches can still burst past the rate limit. The effective concurrency is the branch count, but a giant chunk per iteration also means a larger memory and time footprint per loop pass.
Timezone surprises in the cron. The Schedule trigger uses the IANA timezone you set, not the viewer's local time. Confirm the zone (for example Australia/Sydney versus UTC) so the job runs when you expect.
Assuming Parallel preserves order. Branches run concurrently and may finish in any order. If downstream logic depends on record order, sort after collecting results rather than relying on branch completion order.

Testing

Before scheduling, validate on a tiny scope. Temporarily change the MySQL query to LIMIT 5 and set the array chunk size to 2, then disable the Schedule and run the workflow manually with the Run button. Open the execution log and confirm the chunk count matches what you expect, that each Parallel branch made its http-post call, and that the target API received the records. Once the small run looks correct end to end (including the per-chunk status update), raise the limit and chunk size, re-enable the Schedule trigger, and let it run on its cron cadence.