Frontend Beauty
ESC

Type to search the knowledge base.

Keeping a Vector Index Fresh

Vector index updates keep retrieval honest: upserts, tombstones, and rebuilds so yesterday's policy does not outrank today's.

advanced7 min read
  • vector-db
  • rag
  • freshness

The chatbot quoted a price you deprecated last sprint because nobody deleted the old vectors. Similarity does not know about “latest.” It knows about nearest. Vector index updates are how yesterday’s policy stops outranking today’s: replace by hash, tombstone what died, and rebuild when the pipeline changes.

An index that can only insert is a museum. Museums are bad at refunds.

Key takeaways

  • Stale rows are retrieval bugs, not “the model forgot.”
  • Replace a source as a set: upsert new hashes, delete ids that vanished.
  • Soft deletes work only if every query filters them. Prefer hard delete when the store allows it.
  • Rebuilds need an alias or a dual-write window. Do not query a half-built namespace.
  • Event-driven updates for docs that change with deploys. Nightly is for corpora you do not own the firehose of.

Vector index updates are a retrieval problem

The user asked “what’s the refund window?” You retrieved a chunk from March. The June page exists, but its embedding is slightly farther from the query than the old heading. The model, told to cite sources, cites March.

Vector databases store vectors plus metadata and run ANN search. They will not expire a row because you shipped a PR. Document ingestion produces the new chunks. This page is the index-side contract: apply those changes without split-brain.

type VectorRow = {
  id: string;
  sourceId: string;
  hash: string;
  embedding: number[];
  metadata: {
    url: string;
    updatedAt: string;
    deletedAt: string | null;
    pipeline: string;
  };
};

export async function applySourceUpdate(
  sourceId: string,
  fresh: VectorRow[],
  store: {
    queryBySource(sourceId: string): Promise<VectorRow[]>;
    upsert(rows: VectorRow[]): Promise<void>;
    delete(ids: string[]): Promise<void>;
  },
) {
  const current = await store.queryBySource(sourceId);
  const freshIds = new Set(fresh.map((r) => r.id));
  const gone = current.filter((r) => !freshIds.has(r.id)).map((r) => r.id);
  await store.upsert(fresh);
  if (gone.length) await store.delete(gone);
}

Order matters less than atomicity. If you delete first and crash, retrieval is empty (usually better than stale). If you upsert first and crash, you may have duplicates until the next run. Idempotent re-runs fix both. Two overlapping jobs on the same sourceId need a lock.

Hash-based replace, not append

People “re-index” by inserting new ids (page.mdx#0-attempt-2). The old #0 still ranks. Always:

  1. Compute the desired set of chunk ids for the source.
  2. Upsert that set.
  3. Delete the complement.

Chunk id scheme should be stable for unchanged text (sourceId + hash or sourceId + heading path) so you can skip embed calls. When the heading path changes but the hash is new, treat it as a new row. Do not rename in place if the store keys by id and you already changed the id.

pgvector makes this a SQL transaction. Delete every row for the source, then insert the full fresh set. A single-row VALUES insert is how you keep March and drop June.

-- $1 source_id
-- $2 ids text[]
-- $3 hashes text[]
-- $4 embeddings vector[]
-- $5 metadata jsonb[]
BEGIN;
DELETE FROM chunks WHERE source_id = $1;
INSERT INTO chunks (id, source_id, hash, embedding, metadata)
SELECT t.id, $1, t.hash, t.embedding, t.metadata
FROM unnest(
  $2::text[],
  $3::text[],
  $4::vector[],
  $5::jsonb[]
) AS t(id, hash, embedding, metadata);
COMMIT;

If you already skipped unchanged hashes in application code, you can DELETE ... AND id <> ALL($2) and ON CONFLICT (id) DO UPDATE instead. The insert still has to be set-oriented (unnest or a loop in the same transaction). One VALUES tuple is not a replace.

Hosted ANN stores vary. Pinecone-style upserts overwrite by id. Deletes are explicit. If you cannot delete, you are waiting for a rebuild. Overwrite-by-id plus delete is the operational primitive.

Tombstones when delete is delayed

Some stores propagate deletes slowly, or you need an audit trail. Write deletedAt and require the filter on every query.

const CURRENT_PIPELINE = 'docs-mdx-v3';

export function liveFilter(tenantId: string) {
  return {
    tenantId,
    deletedAt: null,
    pipeline: CURRENT_PIPELINE,
  };
}

Metadata filters are the query-time half. A forgotten filter is a data leak or a zombie chunk. Tests should insert a tombstoned row whose text is a perfect match for a golden query and assert it never appears in the packed prompt.

export async function assertTombstoneHidden(
  search: (q: number[]) => Promise<Array<{ id: string }>>,
  tombstonedId: string,
  queryThatWouldMatch: number[],
) {
  const hits = await search(queryThatWouldMatch);
  if (hits.some((h) => h.id === tombstonedId)) {
    throw new Error(`tombstone ${tombstonedId} leaked into retrieval`);
  }
}

Do not “delete” by setting text = '' and leaving the vector. Empty text still has a vector. It will rank as something.

Rebuild without serving garbage

Pipeline v4 changes chunk boundaries. You cannot mix v3 and v4 in one unfiltered index. Two patterns:

Pattern How When
Blue/green alias Fill docs-v4, flip the query alias, drop docs-v3 You can pay for two copies
In-place + version filter Write v4 rows, query pipeline = v4, then delete v3 One cluster, careful filters
Dual-write Ingest to both during the soak, read from old until evals pass Long cutovers 
type NamespacedStore = {
  queryBySource(ns: string, sourceId: string): Promise<VectorRow[]>;
  upsert(ns: string, rows: VectorRow[]): Promise<void>;
  delete(ns: string, ids: string[]): Promise<void>;
  query(
    ns: string,
    vector: number[],
    k: number,
    filter: ReturnType<typeof liveFilter>,
  ): Promise<VectorRow[]>;
};

const WRITE = ['docs-v3', 'docs-v4'] as const;
const READ = process.env.READ_INDEX ?? 'docs-v3';

export async function applySourceUpdateBoth(
  store: NamespacedStore,
  sourceId: string,
  fresh: VectorRow[],
) {
  await Promise.all(
    WRITE.map((ns) =>
      applySourceUpdate(sourceId, fresh, {
        queryBySource: (id) => store.queryBySource(ns, id),
        upsert: (rows) => store.upsert(ns, rows),
        delete: (ids) => store.delete(ns, ids),
      }),
    ),
  );
}

export async function searchLive(
  store: NamespacedStore,
  query: number[],
  k: number,
  tenantId: string,
) {
  return store.query(READ, query, k, liveFilter(tenantId));
}

applySourceUpdateBoth is the same set-replace as applySourceUpdate, in every write namespace: upsert the fresh set and delete ids that vanished. A helper that only upserts is how v3 keeps March while v4 has June. searchLive takes tenantId from the authenticated request. Do not invent a global tenantFromRequest.

Never point production reads at an index that is 40% loaded. A half rebuild looks like “the bot forgot half the docs.”

Event-driven vs nightly

  • Event-driven: the MDX repo merged, the CMS webhook fired, the ticket changed. Enqueue that sourceId. Freshness measured in minutes. This is the default for a docs site.
  • Nightly: you scrape a partner catalog, or you cannot get events. Accept a staleness SLO (“24h”) and show “indexed as of” in the UI if the product is price-sensitive.
  • Hybrid: events for known sources, nightly reconcile to catch missed deletes. Reconcile is the liveSourceIds pass from ingestion.

If legal takes a page down, nightly is too slow. That path should be a privileged delete-by-source that does not wait for embeddings.

Footguns

  • Unique constraint on sourceId only. Multi-chunk sources will fail or overwrite each other.
  • Clock-based “freshness” (ORDER BY updatedAt after ANN). ANN already picked neighbors. A post-filter on time can drop all hits. Prefer delete.
  • Re-embedding unchanged bytes on every deploy. Cost and rate limits, no quality gain.
  • Namespace-per-user as a substitute for filters at large scale. It can work. It can also explode ops. Measure.
  • Serving during vacuum / rebuild without an alias. Empty results look like an outage.

Interview answer (out loud)

“Freshness is delete-and-replace, not hope. I update a source as a set of hashed chunk ids, upsert the set, and delete the complement, ideally in one transaction. Tombstones need a mandatory query filter. Rebuilds go to a new namespace or version tag, then I flip an alias after evals. Docs I own update on events. I reconcile nightly so missed deletes cannot linger.”

Further reading (primary sources)

Related guides