What are embeddings? Measuring semantic search on Persian enterprise documents

What does an embedding represent?

An employee may ask how a business trip is reimbursed while the relevant document is titled travel-expense settlement procedure. Exact word matching may miss that relationship. An embedding is a numerical representation that enables comparison between a query and document passages within a particular model's representation space. Similarity identifies candidates; it does not certify their truth.

Dense Passage Retrieval is a research example of passage retrieval with separate query and passage encoders. Its experimental results are not a quality guarantee for Persian or a company's scanned policies. The travel-policy exercise below is an original illustrative evaluation scenario.

Separate retrieval from answer generation

The system first needs the passages covering expense limits, required evidence and submission deadlines. Only then should an answer be assembled. A more articulate generator cannot compensate for a missing policy exception. Inspect the retrieved passages independently of the final answer when diagnosing failure.

Retain document ID, version, page and passage location. A similarity score without those connections is difficult to review. A document about the general subject is insufficient when the question depends on one particular condition. Record which passage actually supports the proposed answer.

Keep query and document representations compatible

Microsoft's integration guidance calls for a compatible embedding model at indexing and query time. Equal vector dimensions alone do not establish semantic compatibility. Record the encoder version and preprocessing rules with the index rather than treating a model change as a harmless parameter adjustment.

For a proposed migration, build a separate index and replay reference questions against both versions. Compare policy IDs, conversational formulations and department abbreviations separately. Transfer traffic only after checking quality and a rollback route. Mixing incompatible representations obscures the source of failures.

Build a useful Persian reference set

Use real employee questions rather than only paraphrased document titles. Include formal and colloquial language, half-spaces, Persian digits and mixed-language abbreviations. Add unanswerable questions about expenses that the available policies do not cover. A nearest passage should not become an invented policy.

Identify near-duplicate document versions before splitting evaluation data. Almost identical clauses on both sides of the split can produce misleadingly strong results. Have content owners label the necessary evidence and record disagreements; an ambiguous reference set cannot support a clear model comparison.

Measure evidence coverage, not attractive answers

Define the reference passages for each question. Recall@k measures the fraction of those passages covered by the first k results; finding one of three is not complete coverage. Do not count duplicate passages twice. Track coverage, rank and valid document version separately.

MTEB evaluates embeddings across different tasks rather than assuming a single universal winner. For this exercise, a public benchmark is a shortlist, not procurement evidence. The organization's Persian retrieval results, indexing cost and search latency should decide which candidate is appropriate.

For more on this topic, read Semantic search with PostgreSQL and pgvector: from model to index in the Liyan knowledge base.

Know when semantic similarity is insufficient

An exact policy or contract identifier requires identifier-aware matching. An old and a new policy may be nearly identical while specifying different reimbursement limits. Treat version and validity as explicit metadata rather than expecting vector distance to resolve the difference.

Evaluate lexical and hybrid alternatives on identifier-heavy questions. Permit a no-evidence outcome. The objective is not to return neighbors for every input; it is to supply evidence for this question, within this user's permitted information.

Vectorization is not anonymization

For the proposed design, treat passages, vectors, user IDs and search logs as a connected data lifecycle. Numerical representation does not remove the need for access, retention or deletion decisions. This is an architectural recommendation, not a claim about a particular product's guarantees.

Before using an external encoder, check processing location and retention arrangements. Test revocation across indexed passages and answer caches, not only the source download link. Two test roles asking the same question should receive results consistent with the organization's policy.

Deliver a decision-ready evaluation package

Provide questions, reference passages, model version and chunking configuration. Separate successes, missing evidence and high-impact errors. A wrong amount or deadline should not carry the same acceptance weight as a minor stylistic defect.

Name the content approver and the changes that trigger reevaluation: a revised policy, a new OCR process or a different encoder. This turns embedding selection into a repeatable knowledge-management decision, supported by evidence rather than a small collection of impressive answers.