CSV and JSON: Streaming Transform Pipeline Template
A Spojit template that walks a large CSV one batch of rows at a time, reshaping each batch into JSON for an API, so memory stays bounded no matter how big the file is.
What It Builds
A Manual trigger kicks off the run. The csv connector parses your source file, and a Loop node iterates over the rows in fixed-size groups. Inside each pass, a Transform node maps the row fields and the json connector serializes the batch, which an http request posts to your destination API. Because only one group is in flight at a time, the workflow handles files far larger than a single in-memory pass could.
The Prompt
Paste this into Miraxa and it builds the workflow, connecting the tools for you:
Build a workflow with a manual trigger that parses an uploaded CSV file, then loops over the rows in batches of 100, transforms each batch into a JSON array, and posts each batch to https://api.example.com/v1/import. Continue to the next batch only after the previous request succeeds.Connectors Used
- Manual trigger - you start the run and supply the CSV file.
- csv - parses the source file into rows.
- json - serializes each batch into a JSON array for the API.
- http - Miraxa connects to your destination API to post each batch.
Customize It
Change the batch size of 100 to match what your API accepts per request, swap the endpoint URL, and adjust the field mapping in the prompt to rename or drop columns. To run on a timetable instead, ask for a Schedule trigger; to pull the file from a server, mention an ftp source.
Tips
- Keep the batch size small enough to stay under your API payload limit but large enough to cut request count.
- Use a Direct-mode http request for the exact endpoint and headers, including any API key your destination needs.
- Add a Condition node after the request if you want to branch on non-success responses before moving on.