FIELD ORDER SHOULD NOT MATTER: PERMUTATIONINVARIANT EMBEDDING MODEL FINE-TUNING FOR STRUCTURED METADATA RETRIEVAL
创建于 更新于
摘要
We study retrieval over catalogs of structured metadata, where each record is a small schema whose fields answer different kinds of query. Embedding a record with a text encoder first serializes its fields into a string, which forces a choice of field order. We show this choice silently controls retrieval quality once the encoder is fine-tuned. A standard fine-tune loses 7.4 nDCG@10 points when the index is rebuilt under a different field order, because it reads absolute position instead of the field labels. We propose permutationinvariant fine-tuning (PI-FT), which serializes each training record under a freshly sampled field order with random field dropout, so meaning binds to the labels rather than to position. The change is about two lines in the data loader; it costs negligible in-distribution accuracy and reduces the order-change penalty to 0.2 points. We study this in the discovery of development statistics, a catalog of nearly 10,000 indicators that should be searchable in many languages by a model small enough to self-host. DEVDATABENCH is a fully LLM-generated benchmark of grounded, facet-targeted queries across 15 languages, covering every indicator for both training and evaluation. A finetuned 118M-parameter encoder, runnable on a CPU, outperforms every zero-shot baseline including text-embedding-3-large (0.707 vs. 0.556 nDCG@10), with the largest gains on the low-resource languages those users speak. We release the benchmark, pipeline, and models, and the training framework applies to any labeled-field catalog. [page::0][page::1]
