HTTP and JSON: Paginated API Collector Template

A scheduled Spojit workflow walks a paginated REST API page by page with the http connector, accumulating every page into one collection with the json connector until the cursor runs out.

What It Builds

A Schedule trigger fires the workflow on a cron you set, then a Loop node in While mode calls the target REST API with the http connector's http-get tool, one page per pass. On each pass the json connector reads the next cursor or page token from the response and merges that page's records into a running list. When the API returns no further cursor, the loop stops and you have the complete data set assembled in a single variable, ready to hand to a downstream node such as Send Email or a database connector.

The Prompt

Paste this into Miraxa, the intelligent layer across your automation, and it builds the workflow, connecting the tools for you:

Build a workflow on a Schedule trigger that runs every morning at 7am. Use a While Loop that calls a paginated REST API with the http connector's GET tool, passing a page cursor query parameter. After each call, use the json connector to read the next cursor from the response and append that page's items to a running list. Keep looping while a next cursor is present, and stop when it is empty. When the loop finishes, log how many total records were collected.

Connectors Used

  • Schedule trigger - runs the collector on a 5-field cron and IANA timezone (for example 0 7 * * *, Australia/Sydney).
  • http connector - http-get fetches each page from the external REST API; pass your token in the Authorization header.
  • json connector - get reads the next cursor, merge or set grows the accumulated list across passes.

Customize It

The obvious knobs to change in the prompt: the cron expression and timezone for how often it runs, the API base URL and the name of the cursor or page-token field your API uses (some use ?cursor=, others ?page= or a Link header), and the records path inside each response. If your API caps page size, set the per-page count in the query string so each http-get pulls the largest batch allowed, which keeps the loop short.

Tips

  • Use Direct mode on the Connector node for both http-get and the json tools: pagination is deterministic, so you spend no AI credits walking the pages.
  • Reference the previous pass with a variable such as {{ page.nextCursor }} so each iteration requests the page after the last one, and start the first pass with an empty cursor.
  • Add a sensible iteration ceiling on the Loop node so a misbehaving API that never returns an empty cursor cannot spin forever.

Common Pitfalls

  • Off-by-one cursors: confirm whether the cursor in the response points at the next page or the current one before you wire it back into http-get, or you will skip or repeat records.
  • Rate limits: tight loops can trip an API's request quota. Space passes out or shrink the cron frequency if you see 429 responses.
  • Schema drift: if the API moves the records array or renames the cursor field, the json get path stops resolving. Pin the exact paths and revisit them when the vendor changes its API.

Related

Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.