Building a Scalable MLS Synchronization Engine

Where a sync engine breaks

The first version of an MLS sync is usually a scheduled script: ask the feed for records changed since the last run, save them, remember the time. It works for one feed on a quiet day. It stops being correct in five situations, and none of them raises an error on its own: the process dies halfway through a page; the feed re-stamps every record at once; the feed is unreachable for a day; a second and third board are added to the same job; a field arrives that the code has never seen.

RESO describes the job in one sentence: replicators initially pull an entire data set, and then continually request the most recent data changes. Our MLS data pipeline reference architecture places that job as stage two of nine and lists the platform limits around it. This article is about the inside of that stage: the loop, the checkpoint, the queue behind it, and what the engine does when each step fails.

One sync cycle

The same cycle as a list:

1. Read the checkpoint for this feed and resource. It holds the cursor.
2. Fetch one page of records changed since the cursor, through the rate limiter. On a 429, a 5xx or a timeout: back off and retry the same request. The cursor does not move.
3. Write the page to the raw store with an idempotent upsert keyed by the record key.
4. Advance the cursor in the same transaction as the write. If either fails, both roll back and the next run fetches the page again.
5. Enqueue downstream work (normalize, media, index). If enqueueing fails, the rows stay marked unprocessed and a sweeper re-queues them. A record that keeps failing downstream is quarantined; it never holds the cursor back.
6. If the response carried a next link, go back to the fetch step. If not, record the time of the successful run and wait for the next one.

Everything else in this article is a consequence of two decisions in that loop: the cursor only moves after a durable write, and every write can be repeated without harm.

The checkpoint: what to store and when to move it

On MLS Grid and Trestle the cursor is a timestamp. RESO's Web API Core tests page through results filtered on ModificationTimestamp because it is the only required field at this point, and providers must support server-driven paging using @odata.nextLink. Three platform rules shape what goes in the checkpoint row:

• The value comes from the feed, not from your clock. Trestle recommends tracking the latest value you received so that clock differences do not cause missed updates. RESO says the same of timestamp replication generally: clock drift can cause events to be out of order and consumers to miss data.
• It covers what you received, not what you chose to keep. MLS Grid requires the greatest ModificationTimestamp you have received, regardless of whether you store the records. Writing every received record to a raw store makes the two the same thing.
• There is one cursor per feed, originating system and resource. MLS Grid requires a single OriginatingSystemName in the filter of every request, and its replication guide gives a separate request for Property, Member, Office and OpenHouse. A checkpoint keyed only by feed cannot express that.

An illustrative checkpoint table, in PostgreSQL syntax:

-- Illustrative. One row per feed, originating system and resource.
CREATE TABLE sync_checkpoint (
  feed_id              text        NOT NULL,  -- for example 'mlsgrid-idx'
  originating_system   text        NOT NULL,  -- for example 'actris'
  resource             text        NOT NULL,  -- Property, Member, Office, OpenHouse
  phase                text        NOT NULL,  -- 'initial' or 'incremental'
  cursor_ts            timestamptz,           -- greatest feed timestamp durably stored
  last_attempt_at      timestamptz,
  last_success_at      timestamptz,           -- last run that drained the feed
  last_reconciled_at   timestamptz,           -- last full key comparison
  consecutive_failures integer     NOT NULL DEFAULT 0,
  paused_until         timestamptz,           -- set by the circuit breaker
  last_error           text,
  lease_owner          text,                  -- run id holding the lease
  lease_expires_at     timestamptz,
  PRIMARY KEY (feed_id, originating_system, resource)
);

Why the cursor moves only after a durable write. The cursor is a claim: everything up to this timestamp is safely in our store. If it advances when the page is fetched and the process dies before the write, nothing will ever re-request those records, because the next query starts after them. If it advances after the write, a crash costs one repeated page. So the write and the cursor update belong in one transaction where the raw store and the checkpoint share a database. Where they do not, write first, advance second, and rely on the write being repeatable.

Page boundaries. MLS Grid warns against range queries on ModificationTimestamp; the filter is “greater than the cursor”. That raises a question the documentation does not answer: if a page ends on timestamp T and the next page begins with more records stamped T, a cursor saved as T would skip them after a crash. Our recommendation is to treat the last timestamp of a page that has a next link as incomplete, and save the greatest timestamp below it. The final page, with no next link, saves its true maximum. This is more conservative than MLS Grid's own wording, which is to use the greatest timestamp received: it can only re-request records, never skip them, and the cost is a few records fetched twice after a restart.

The lease. MLS Grid says not to send more than one replication request at a time, because requests must be in sequential order. A lease on the checkpoint row, with an expiry so a dead worker cannot hold it forever, keeps a slow run and the next scheduled run from interleaving. Two details make it hold: the run renews the lease as it works, and the transaction that advances the cursor checks that the run still owns it, so a worker that stalled past its expiry cannot write.

Initial full import

The pipeline guide covers the platform limits for the first load. What matters for the engine is the handoff from import to incremental sync, because a gap there loses changes silently.

On MLS Grid the import is the same loop as incremental sync with one extra filter. Results are ordered by ModificationTimestamp so that you do not miss changes that occur during your download, and after an error the instruction is to not start over but continue from the greatest timestamp received. The initial query filters on MlgCanView equal to true; once it has drained, the engine flips the checkpoint to incremental and drops that filter, so that changes to MlgCanView arrive and removals can be applied. Forgetting to drop it is a real bug: the site keeps syncing and never removes anything.

Trestle is different above a million records, where its replication endpoint cannot skip records, so a process that fails part way through has to restart, and its links expire after 5 minutes of inactivity. Two recommendations follow. First, do nothing slow between pages: write the raw payload and request the next link. Second, do not derive the starting cursor from the bulk download, whose order the documentation does not state. We would read the greatest ModificationTimestamp in the feed before the bulk load begins and start incremental sync from that value. Records changed during the load are then fetched a second time, which is harmless, instead of being missed.

During an import, per-record downstream jobs are wasteful. A sound default is to mark rows unprocessed and let batch workers normalize and bulk-index them, switching to per-record jobs when the phase changes.

Incremental sync: the loop

Pseudocode for one run, matching the diagram. It is illustrative: names and helpers are invented for the example.

// Illustrative. One run for one (feed, originating system, resource).
async function runSync(key) {
  // Returns null if another run holds the lease or the circuit breaker is open.
  const cp = await checkpoints.acquireLease(key)
  if (!cp) return

  try {
    let url = buildQuery(key, { modifiedAfter: cp.cursor_ts, phase: cp.phase })
    while (url) {
      await limiter.take(key.feed_id)              // requests and bytes, under the published limits
      const page = await fetchWithBackoff(url)     // retries 429, 5xx and timeouts, then throws

      if (page.value.length > 0) {
        const maxTs  = maxTimestamp(page.value)
        const safeTs = page.nextLink
          ? greatestBelow(page.value, maxTs) ?? cp.cursor_ts   // last timestamp may continue on the next page
          : maxTs

        const changed = await db.transaction(async (tx) => {
          const ids = []
          for (const rec of page.value) {
            // Skips a copy older than the stored one. Returns true only if the payload hash changed.
            if (await tx.upsertRaw(key, rec)) ids.push(rec.key)
          }
          // Forward only. Throws, and so rolls the page back, if this run no longer owns the lease.
          await tx.advanceCursor(key, safeTs, cp.lease_owner)
          return ids
        })

        await queue.enqueueChanged(key, changed)   // a sweeper covers a failure here
        cp.cursor_ts = safeTs
      }
      await checkpoints.renewLease(key, cp.lease_owner)
      url = page.nextLink
    }
    await checkpoints.markSuccess(key, cp.lease_owner)      // last_success_at, failures = 0, release lease
  } catch (err) {
    await checkpoints.markFailure(key, cp.lease_owner, err) // failures + 1, last_error, release lease
    throw err
  }
}

Three details are easy to miss. The run only records success when the feed is drained, so a run that dies on page three leaves a truthful last_success_at. A failed run records the failure and releases the lease instead of holding it until expiry. And the cursor changes on every page, which keeps the engine inside another MLS Grid rule: do not send the same request multiple times.

Idempotent writes

The loop delivers every record at least once, and sometimes twice. Correctness therefore depends on a second delivery changing nothing. Three rules get there:

• Upsert by a stable key. The raw store is keyed by feed, originating system, resource and record key. In PostgreSQL, ON CONFLICT DO UPDATE guarantees an atomic INSERT or UPDATE outcome, so two workers cannot create two rows for one listing.
• Never let an older copy overwrite a newer one. Make the update conditional on the incoming ModificationTimestamp being greater than or equal to the stored one. This matters whenever a replay or a reconciliation job runs beside the incremental sync. A removal is a write too: on MLS Grid a record that arrives with MlgCanView false must be removed from your local data store, so the upsert replaces the payload with a removal marker instead of keeping it.
• Detect “changed” by content, not by arrival. A record's timestamp can move without the listing changing. On MLS Grid, ModificationTimestamp is the time the record was modified by MLS Grid, not by the MLS. On Trestle a record can be updated because mappings or enumerations that affected it were updated, and Trestle warns that source systems may update all of their records at once: being prepared for mass updates is important. Store a hash of the payload (excluding the timestamps) and enqueue downstream work only when it differs.

With the third rule a mass update still downloads every record, but the index, the media queue and the alert emails see only what really changed. A re-stamped record must never look like a new listing or a price change to anything that notifies people.

Queues: separate fetching from processing

Fetching is sequential by rule and limited by quota. Processing (normalization, media, indexing) is neither, and in one process a slow image host or an indexing error would stall replication. Separate them with a queue, and treat the raw store, not the queue, as the record of what is outstanding:

• Each raw row carries a processed marker. The transaction that writes the row clears it.
• After the commit, the engine enqueues a job per changed record. This is the only step outside the transaction.
• A sweeper periodically enqueues rows that have been unprocessed for longer than expected. It repairs a failed enqueue, a lost queue and a worker that crashed mid-job with one mechanism.
• Job identifiers are derived from the record key and version, so a duplicate enqueue collapses instead of doubling the work. BullMQ, for one, documents deduplication by an identifier you supply.
• Use separate queues per kind of work and, for fairness, per feed, so one board's import does not starve another board's price changes.

Queue mechanics in Node.js (retries, concurrency, stalled jobs) are the subject of background jobs with Redis and BullMQ.

Rate limiting and backoff

The limiter belongs to the feed credential, not to the job, because every job on that credential spends the same budget. The published budgets differ in shape:

• MLS Grid limits requests and bytes: no more than 2 requests per second, 7,200 requests in any hour, 4 GB downloaded in any hour and 40,000 requests per 24-hour period. Our arithmetic on those figures: two per second for an hour is exactly 7,200, but 40,000 spread over the 86,400 seconds of a day is one request every 2.16 seconds. The daily cap is the one a steady engine has to plan around, and the limiter has to count bytes as well as requests.
• Trestle sets a per-hour baseline quota and a per-minute burst quota, calculated per product and feed type pair, and reports them in response headers: Minute-Quota-Limit, Hour-Quota-Limit and Hour-Quota-ResetTime. Read the headers instead of hard-coding the baseline.

Backoff is what happens after the limiter was wrong. HTTP defines 429 as too many requests in a given amount of time and says the response may include a Retry-After header; honor it when present. Otherwise Trestle recommends an exponential wait, for example 1 second, then 2, then 4, 8, 16, 32, up to some number of retries. Add random jitter so several workers do not retry in step; BullMQ documents exponential backoff with a jitter option if the retries run as queue jobs.

On MLS Grid a 429 can mean the token is suspended: it then receives an HTTP 429 error in response to any requests until access is reinstated automatically once usage falls back within the limits. Retrying each request on its own schedule is the wrong response. Open a circuit breaker for the whole credential: set paused_until, stop every job that uses it, alert, and probe with a single request when the pause ends.

Monitoring: three numbers per feed

• Last successful sync. Now minus last_success_at. This is the alarm for a dead worker, an expired token or an open breaker. MLS Grid suggests a replication request once every 15 minutes, and its IDX rules require a refresh at least once every twelve (12) hours; alert long before the second figure. Your own license sets the real limit.
• Lag. Now minus cursor_ts. Lag alone misleads: a feed with no changes overnight shows hours of lag while every run succeeds. Read it together with the first number. Rising lag with recent successful runs means the engine is working and behind (an import or a mass update); rising lag with no successful run means it is stopped.
• Error rate. Failed requests over total, per feed, with 429s counted separately from 5xx and timeouts. They have different causes and different owners.

Four secondary signals explain the first three: records received per run against that feed's usual range (a spike is a mass update; a long run of zeros on a busy feed is a scoping or credential problem), the count of unprocessed raw rows and its oldest age, the size of the quarantine, and requests and bytes spent against each published limit. Keep a history row per run as well as the checkpoint.

Failure recovery

Replay from a checkpoint

Because writes are idempotent, recovery from most data problems is one operation: set cursor_ts back and let the loop run. It is a forward query from the earlier timestamp, never a window, in keeping with MLS Grid's rule on range queries. Know its limit. A replay re-delivers records that still exist in the feed. It cannot re-deliver a removal the feed no longer carries: on MLS Grid, records marked MlgCanView false are removed from the data feed entirely after 7 days. A replay from further back than that must be followed by a full key comparison. Handling deleted, expired and sold listings covers that reconciliation job.

Poison records

A poison record is one that fails every time it is processed. The design above confines it. The raw write does not interpret the payload, so an unexpected value cannot fail it, and the cursor moves on. The failure surfaces in a downstream worker, where the rule is: retry a bounded number of times with backoff, then move the record to a quarantine table with the error and the raw payload, and alert. After the mapping is fixed, re-queue the quarantine.

The harder case is a page that cannot be read at all. We would retry it with a smaller page size to narrow the fault (MLS Grid defaults to 500 records per request and allows up to 5,000, or 1,000 when the request uses $expand), and contact the platform. Advancing the cursor by hand past the fault should be a logged, deliberate act, followed by a reconciliation.

A feed that goes down

Isolation comes first: one feed's failures must not consume another feed's workers or stop its schedule. In our published real estate SaaS case study, the original sync ran as a single process with no error recovery, and one failed MLS feed would stall updates for all agents; the sync engine was rewritten with delta detection, retry logic and per-feed error isolation. In the design here that isolation is the checkpoint row: its own lease, failure count and breaker.

When the feed returns, the breaker closes and the loop resumes from the stored cursor with no special code. The one decision is whether the incremental query is still enough. Compare the length of the outage with the platform's retention for removal signals, and run a reconciliation if it was exceeded or if the platform, like Trestle, does not report records that are no longer available in incremental updates at all.

A field or schema change

A new field, a renamed field or a new lookup value should never stop replication, and with a raw store it does not: the payload is saved as received and the change is a normalization problem. Detect it there, with a count of unmapped fields and lookup values per run and a daily diff of the metadata document. MLS Grid publishes lookup values as a resource and asks consumers not to pull it more than once a day, which sets the cadence. After the mapping is extended, re-run normalization from the raw store; the cursor is not involved. MLS data normalization across boards covers the mapping layer.

The larger version of this event is an MLS moving to a new system. When MARIS did, MLS Grid told consumers to treat the change as an initial import of a new MLS source under a new OriginatingSystemName. Because the checkpoint is keyed by originating system, that is a new row, not a code change. Replacing your own platform works the same way; see data migration for legacy real estate platforms.

Timestamps are not the only cursor

RESO has ratified an alternative. The EntityEvent resource carries a sequence number that is durable, immutable and monotonic, and can only increase in value, and RESO notes that events are produced when records are no longer part of a user's feed. Where a provider offers it, the checkpoint stores a sequence number instead of a timestamp, the page-boundary problem disappears and removals arrive in the same stream. The loop, the transaction and the idempotent write stay exactly as described. Build the engine so the cursor type is a property of the connector.

Five tests before launch

• Kill the worker between the fetch and the commit. The next run must fetch the same page and store each record once.
• Deliver the same page twice. Row counts and downstream job counts must not change.
• Replay a full day by moving the cursor back. No older copy may overwrite a newer one.
• Return 429 for every request on one feed. That feed must pause and alert; the others must keep their schedule.
• Send a record with an unknown field and an unknown lookup value. Replication must continue, the record must be quarantined or flagged, and the unmapped count must rise.

Platform behavior and limits were read from the linked RESO, MLS Grid, Trestle, PostgreSQL, BullMQ and RFC pages on October 12, 2026, and checked again the next day. They change, and they differ by MLS and by license; the current documentation and your agreement decide. Code and schema blocks are illustrative.

Building or repairing a sync engine? See our MLS and IDX integration service. Every project starts with a 30-minute technical discovery call with an engineer.