How to Fetch Only New or Updated Shopify Records Since the Last Run
Build incremental Shopify syncs that read only the orders, products or customers changed since the previous run, using updated_at search filters and a saved checkpoint, instead of re-reading the whole store every time.
What This Integration Does
A sync that reads every Shopify order on every run gets slower as the store grows and burns through API limits. An incremental sync reads only what changed. In Shopify's REST Admin API this was the updated_at_min parameter; in the GraphQL Admin API, which Spojit's Shopify connector uses, it is a search query such as updated_at:>'2026-10-14T03:00:00Z', which list-orders, list-products and list-customers all accept. This tutorial adds the missing piece: a checkpoint that remembers where the last successful run stopped, so each run starts exactly there, even after a failure or a pause.
The pattern is simple and safe: read the checkpoint, fetch records updated after it (minus a small overlap), process them idempotently, and move the checkpoint forward only when everything succeeded. If a run fails, the checkpoint does not move and the next run retries the same window.
Prerequisites
- A Shopify connection with the read scopes for the objects you sync.
- A place to keep the checkpoint. This tutorial uses a small text file in your Spojit File Store; a row in MySQL works the same way.
- A destination that can upsert (insert or update by ID), such as MySQL, MongoDB, NetSuite or another store, so re-processing a record is harmless.
Step 1: Schedule Trigger
Add a Trigger node with type Schedule, for example */15 * * * *.
Step 2: Read the Checkpoint
Add a File Store node with List files and folders and a search for shopify-orders-checkpoint.txt, then a File Store node with Get a file on the file it finds. The file holds one ISO 8601 timestamp. On the very first run there is no file; add a Condition that falls back to a fixed start date (for example 30 days ago) when the list is empty. Name the result checkpoint.
Step 3: Subtract an Overlap
Records updated in the same second as the checkpoint, or whose update arrived late, can slip past a strict "greater than" filter. Use the Date & Time connector's subtract tool to take 5 minutes off the checkpoint and name the result since (the timestamp is in {{ since.iso }}). Also record the run's start time now, with now, as run_started: that becomes the next checkpoint.
Step 4: Fetch Changed Records
Add a Shopify step with list-orders (or list-products, list-customers):
query:updated_at:>'{{ since.iso }}'first:250
Put it in a Loop in While mode that follows the page cursor into after until there are no more pages. Combine filters with spaces: updated_at:>'{{ since.iso }}' financial_status:paid fetches only paid orders that changed.
Step 5: Process Idempotently
Write each record to the destination by its Shopify ID, so the overlap window and any retried run update rows instead of duplicating them. For example, a MySQL step with execute-query using INSERT ... ON DUPLICATE KEY UPDATE, or a MongoDB update-documents with upsert on. Set On iteration error to Stop the run so a failure leaves the checkpoint untouched.
Step 6: Move the Checkpoint Forward
As the last step, add a File Store node with Save a file: name shopify-orders-checkpoint.txt, content {{ run_started.result }}, content type text/plain, and If the name is taken set to Overwrite. Using the run's start time, not the time it finished, means records updated while the run was working are picked up next time.
Tips
- One checkpoint per object: keep separate files for orders, products and customers so each sync can fail and recover on its own.
- Deleted records: deletions do not show up in
updated_atqueries. Subscribe to Shopify's delete webhooks (products/delete,customers/delete) if the destination must remove them. - Webhooks plus polling: webhooks give speed and polling gives completeness. Many teams use both: webhooks for real-time updates and this incremental poll every hour as a safety net.
- Bulk history: for the first full load of a large store, Shopify's bulk operations (through
raw-graphql) are faster than paging; switch to this incremental pattern afterwards.
Common Pitfalls
- Quoting dates: the date in the query needs quotes:
updated_at:>'2026-10-14T03:00:00Z'. Without them Shopify may ignore the filter and return the most recent records instead. - Moving the checkpoint on failure: saving the checkpoint before processing (or in a branch that runs after errors) skips records forever. Save it only as the final step of a successful run.
- Timezones: use UTC timestamps with a
Zsuffix everywhere. Mixing the shop's timezone with UTC shifts the window by hours. - Old orders: without
read_all_orders, Shopify only returns orders from the last 60 days, even when they were updated recently.
Testing
Run the workflow once to create the checkpoint. Edit one order in Shopify (add a tag), run again, and check that only that order was processed and the checkpoint moved forward. Then make the run fail on purpose (for example, point the destination at a missing table) and confirm the checkpoint stayed where it was.