Getting off the DIME: Dimension Pruning via Dimension Importance Estimation for Dense Information Retrieval

Dense Information Retrieval (IR) systems rely on neural networks to embed documents and queries within a latent low-dimensional space. Among the Dense IR approaches, bi-encoders are particularly popular, as they achieve state-of-the-art performance and allow for efficient encoding of documents and queries. Nevertheless, using this class of systems, by construction, all the documents and queries are represented using the same set of dimensions. In this article, we introduce the Manifold Clustering (MC) hypothesis which states that, for each query, there exists a query-dependent manifold of the original embedding space where the query and documents relevant to it cluster more effectively. We empirically validate the MC hypothesis showing that it is possible to find a query-dependent linear subspace of the original embedding space where high retrieval effectiveness is achieved. To find such subspaces, we propose the Dimension IMportance Estimators (DIMEs), a class of models that associate an importance score with each dimension of an embedding and can be used to project the dense representations only on the most important dimensions. We first demonstrate the effectiveness of the DIMEs by proposing an oracle DIME which employs annotated documents and induces performance improvements as big as +184% in terms of AP. To demonstrate the practical applicability of the DIMEs beyond the oracle, we also propose a set of DIMEs based on pseudo-relevance and active feedback that induce improvement as big as +49.6% in terms of AP and +55.9% in terms of nDCG@10. The effectiveness of such DIMEs not only empirically supports the MC hypothesis, but illustrates an actual strategy to outperform the state-of-the-art that does not require any form of retraining, fine-tuning or re-indexing and can be efficiently implemented at retrieval time.