Guides

Recurring imports and syncs

Set up jobs that write to Airdash on a schedule without creating duplicates.

Many agents write to the same table again and again: a daily scrape, a weekly export, a sync from another tool. The key is that running the job twice gives the same result as running it once.

Give every row a natural key

Pick a value that identifies the thing in the outside world: a url, an email, an order number, an id from the other system. Make that field unique when you create the table (or with update_field later, if the existing values have no duplicates).

Upsert instead of insert

Pass match_on with that field to insert_rows or import_csv:

Rows whose key already exists update that row (only the fields you send).
New keys are added as new rows.
on_match: "skip" leaves existing rows alone instead, for when the first value should win.

The result tells you how many rows were inserted, updated and skipped.

Keep history when values change

If the job measures something that changes (a price, a ranking, a follower count), don't just overwrite it. Keep the thing in one table with a unique key, and add a row per measurement to a second table with a link back and a datetime. See the recurring tracker in Designing a database.

Removing things that disappeared

Upserts never delete. If rows that are no longer in the source should go, either:

add a last_seen date field, set it on every run, and delete rows with an old last_seen (find their ids with run_sql), or
add an active checkbox and clear it instead of deleting, which keeps the history.

Good habits for scheduled jobs

Write in batches of 100 rows, or use import_csv with a url for large files.
Check the result counts. A run that suddenly inserts everything again usually means the key changed format (http vs https, trailing slashes, upper vs lower case): normalise keys before writing.
Stay well inside the free plan's rows written per month. Use get_account to see usage. Writing unchanged rows still counts, so skip rows that haven't changed when you can.