2026 Property Search Architecture: Incremental Feeds for Fresh Listing Retrieval
The freshness problem
Short answer: use an append-only change stream to drive idempotent, per-listing index updates, keep periodic reconciliation as a safety net, and choose chunk size with a measured retrieval-quality versus latency test rather than rebuilding the whole corpus on every feed run.
A real-estate search result can be factually correct and still be operationally wrong. A buyer sees a property after its status changed to sold, misses a reduced price because an older event arrived last, or retrieves a paragraph whose address belongs to a previous version.
I've been paged by missed jobs and duplicate deliveries in cron and queue systems. The lasting lesson was not to trust "exactly once" as a scheduling promise - it was to make every consequence safe to repeat and every omission detectable.
That changes the design boundary. The unit of recovery is one listing version, not one nightly indexing job, and the key metric is bounded staleness by change type. A status transition deserves a tighter bound than an edited description. Retrieval quality still matters, but freshness failures must be visible before a user becomes the monitor.
Durable change log and ordering invariant
How should retrieval architecture handle incremental listing indexing for property discovery? Start with a durable change log between the source of truth and the retrieval index.
Each event needs a stable listing ID, a monotonically comparable source version, an operation, and enough provenance to trace it back to the source record. The consumer reads the latest canonical listing, normalizes it, produces chunks, computes embeddings when searchable text changed, and commits the searchable document with the source version.
Deletes and off-market transitions become tombstones or explicit visibility changes; they must not depend on a later full rebuild to disappear.
The ordering rule is the invariant: an index writer may apply version 42 after version 41, but it must never let version 41 overwrite version 42. Partitioning a stream by listing ID makes normal processing easier, yet the sink still needs that comparison because retries, backfills, and parallel consumers can disturb arrival order.
A duplicate of version 42 should produce the same final state. No drama.
Fast path and repair path
Keep two paths, with different jobs. The fast path consumes changes continuously. The repair path scans source IDs and versions on a slower schedule, compares them with index metadata, and republishes only missing or divergent records.
A full reindex belongs in a controlled migration path for schema, analyzer, or embedding-model changes; it isn't the everyday freshness mechanism. Build the replacement index beside the active one, validate it, then switch an alias or routing pointer. The old generation remains available for rollback until the new generation has passed checks.
The event does not need to carry the entire listing. A small notification followed by a canonical read reduces the risk that a partial event becomes a permanent search document, although it adds a source read and creates a race that the source-version check must close. Carrying a full snapshot lowers read pressure and preserves the exact event-time state, but it makes the log larger and requires every producer to serialize the complete schema correctly.
A compact change event plus canonical read is my default when the source can serve those reads within the freshness budget; a transactional outbox is the stronger choice when committing database state and publishing its notification must share one atomic boundary.
Treat every delivery as replay
The preventative code path is small. The hard part is refusing stale writes at the storage boundary, where two workers cannot both "check, then write" against an outdated view. This example leaves the storage engine generic: PutIfNewer must be one atomic compare-and-set operation, and DeleteIfNewer must preserve a version marker so an old upsert cannot resurrect a withdrawn property.
package indexing
import (
"context"
"errors"
"fmt"
)
type Change struct {
ListingID string
Version int64
Operation string
}
type Listing struct {
ID string
Version int64
Status string
Description string
Address string
}
type Source interface {
GetListing(ctx context.Context, id string) (Listing, error)
}
type Index interface {
PutIfNewer(ctx context.Context, listing Listing, chunks []string) (bool, error)
DeleteIfNewer(ctx context.Context, id string, version int64) (bool, error)
}
type Chunker interface {
Split(listing Listing) []string
}
type Consumer struct {
Source Source
Index Index
Chunker Chunker
}
func (c Consumer) Apply(ctx context.Context, change Change) error {
if change.ListingID == "" || change.Version < 1 {
return errors.New("invalid change identity")
}
if change.Operation == "delete" {
_, err := c.Index.DeleteIfNewer(ctx, change.ListingID, change.Version)
return err
}
listing, err := c.Source.GetListing(ctx, change.ListingID)
if err != nil {
return fmt.Errorf("read canonical listing: %w", err)
}
if listing.Version < change.Version {
return fmt.Errorf("source has version %d, event requires %d", listing.Version, change.Version)
}
_, err = c.Index.PutIfNewer(ctx, listing, c.Chunker.Split(listing))
return err
}
Acknowledgment comes after the atomic write. A transient read or write error leaves the event available for retry; invalid events go to a quarantined stream with their identity and reason. Set a retry ceiling, but do not discard the record when that ceiling is reached.
Alert on the oldest unprocessed event, not merely queue depth: ten old status changes can matter more than ten thousand recent description edits.
I'd also record source_version, indexed_at, event_id, chunker_version, and embedding_model beside every indexed listing. Those fields turn a vague "search looks stale" report into a comparison. They let a reconciliation job distinguish transport loss from a transformation change, and they make rollback decisions possible without guessing which generation produced a chunk.
Measure retrieval quality against the freshness budget
Chunking creates the central quality-latency trade-off. One vector for an entire listing is cheap to update and easy to delete, but a long description can dilute the signal for a query such as "south-facing study near the train." Very small chunks isolate details, yet they increase embedding work, index entries, fan-out, and the chance that a retrieved fragment loses the listing context needed to answer safely.
Structured filters for status, geography, price, bedrooms, and availability should remain typed metadata; don't hide exact constraints inside prose embeddings.
Use a versioned offline evaluation set made from representative discovery queries and judged relevant listing IDs. Report retrieval measures such as recall at a fixed cutoff and mean reciprocal rank, then run the same candidates through an end-to-end latency test. The original retrieval-augmented generation paper describes combining retrieved external memory with generation, but it does not choose a property-search chunking policy for you. That decision needs local relevance judgments and your own traffic shape.
I would test at least three policies:
- a listing-level document;
- field-aware chunks that keep address and listing ID in every chunk;
- smaller overlapping prose chunks.
The numbers here are test labels, not universal recommendations: try maximum text windows of 256, 512, and 1,024 tokens, keep the embedding model and query set fixed, and compare both quality and p95 indexing-to-searchable latency. I'm not sure which window wins for your descriptions. Language, boilerplate, and the proportion of structured queries can reverse the result; a held-out query set and production shadow traffic would resolve that uncertainty.
Watch the update amplification too. A price-only change should update typed metadata without re-embedding unchanged prose. An address correction probably invalidates every chunk because location context is repeated. A description edit can use content hashes to reuse unchanged chunk embeddings, but chunk boundaries must be deterministic or one inserted sentence will make the rest appear new.
This is where a nominal quality improvement can consume the freshness budget: more chunks mean more work per mutation, so measure time from the source commit to the new version becoming queryable, separated by operation type.
Failure matrix and ugly sequences
Use a small failure matrix during review:
- Failure: Duplicate event - Required behavior: Same visible listing version and chunks - Signal: Duplicate delivery count
- Failure: Older event arrives late - Required behavior: Atomic write rejects the older version - Signal: Stale-write rejection count
- Failure: Consumer stops - Required behavior: Resume from durable offset - Signal: Oldest-event age
- Failure: Event is missing - Required behavior: Reconciliation republishes the listing - Signal: Source/index version mismatch
- Failure: Listing is withdrawn - Required behavior: Visibility change or tombstone wins over old upserts - Signal: Withdrawn listing probe
- Failure: New schema or model - Required behavior: Build and validate a separate generation - Signal: Generation comparison report
Test the ugly sequences. Apply versions 7, 7, 6, 8; kill the worker after the index write but before acknowledgment; withdraw a listing while a description backfill is running; pause a partition long enough for the freshness objective to fail. The expected state is always derived from the greatest accepted source version. If a test cannot state that expected state precisely, the recovery contract is still underspecified.
Where this architecture stops fitting
Incremental indexing is not free. It adds a log, replay semantics, version metadata, reconciliation, and an on-call surface. For a small catalog that changes once a week and can be rebuilt comfortably inside its freshness objective, stick with a scheduled full rebuild; it is easier to reason about and may be the more reliable system.
If the source cannot expose ordered versions or a consistent change feed, periodic snapshots plus deterministic diffs may be safer than pretending timestamps provide strict ordering.
The storage choice follows workload boundaries rather than a ranking. PostgreSQL offers full-text search and transaction semantics close to relational listing data, which can reduce moving parts for modest workloads. Elasticsearch exposes near-real-time search and optimistic concurrency controls suited to a separate search index. Qdrant supports vector search with JSON payload filtering, fitting an architecture that keeps semantic vectors and listing filters together. Each option still needs application-level rules for source versions, tombstones, and reconciliation; none removes the delivery problem.
The catch is that adding a dedicated engine also adds capacity planning, migration, backup, and incident procedures, so keep the database path when its measured recall and latency already meet the objective.
Rollout and operational decision
Before rollout, define service-level indicators for source-to-searchable age, stale-visible listing rate, reconciliation mismatch rate, retry age, and retrieval latency. Deploy a consumer that writes a shadow generation, compare results on the evaluation set and sampled queries, then move a small traffic slice. Roll back routing - not source events - if quality regresses. The append-only log remains the record from which a corrected generation can be rebuilt.
The operational decision is therefore plain: use incremental delivery when the catalog's change rate or freshness target makes batch replacement inadequate, but retain reconciliation because queues provide delivery mechanisms, not proof that two stores agree. Make versions authoritative, make replay boring, and let measured retrieval quality decide how much chunking latency the system can afford.
Sources
- https://arxiv.org/abs/2005.11401
- https://www.postgresql.org/docs/current/textsearch.html
- https://www.elastic.co/guide/en/elasticsearch/reference/current/near-real-time.html
- https://www.elastic.co/guide/en/elasticsearch/reference/current/optimistic-concurrency-control.html
- https://qdrant.tech/documentation/concepts/filtering/
- https://microservices.io/patterns/data/transactional-outbox.html
Comments
No comments yet. Start the discussion.