Adaptive Search Filters

Breaking the Lens of the Telescope: A Game-Changing Approach to Document Retrieval I just read a fascinating paper from researchers at L3S Research Center, University of Glasgow, and TU Delft that introduces a novel paradigm called "Online Relevance Estimation" (ORE) for document retrieval systems. Traditional search systems use a "telescoping" approach - using fast but less accurate methods to filter documents before applying expensive but precise ranking models (like LLMs). The problem? This approach can miss relevant documents if they're filtered out early. >> How ORE Works: Instead of re-ranking a fixed set of top-k documents in one step, ORE continuously updates relevance estimates throughout the ranking process: - It iteratively re-scores smaller subsets of the most promising documents - It dynamically adjusts relevance scores for the remaining pool based on what it learns - It uses a bandit-based algorithm that balances exploration vs. exploitation - It captures documents that might have low initial retrieval scores but high relevance The framework models document selection as a top-arms selection problem in stochastic linear bandits, where: - Arms = candidate documents - Features = document properties (lexical similarity, semantic similarity, etc.) - Rewards = actual relevance scores from expensive rankers >> Impressive Results: The researchers validated ORE on TREC benchmarks under two scenarios: - In hybrid retrieval: Up to 14.12% gains on DL19 - In adaptive retrieval: Up to 30.55% gains on DL21 - Against standard telescoping: Up to 58.53% improvements on DL22 - Efficiency: 2x speedups over state-of-the-art, with ORE taking 10x less time than expensive ranker calls This approach bridges the gap between retrieval and ranking stages, offering a scalable solution that enhances both recall and precision while maintaining computational efficiency. As search systems increasingly rely on expensive LLM-based rankers, innovations like ORE that can maximize their effectiveness while minimizing computational costs will be crucial for building the next generation of information retrieval systems.

Weaviate just released a new filtered search method based on the popular ACORN paper that makes certain queries up to 10x faster 🤯 𝗧𝗵𝗲 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲: Picture searching for a "gold ring" with a price filter under $10. Seems simple, right? But there's a technical challenge here: • The query ("gold ring") and filter (price < $10) have 𝘭𝘰𝘸 𝘤𝘰𝘳𝘳𝘦𝘭𝘢𝘵𝘪𝘰𝘯 • Gold rings are typically expensive, so most similar items won't pass the filter • This creates inefficiencies in traditional search approaches 𝗣𝗿𝗲-𝗔𝗖𝗢𝗥𝗡 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻𝘀 𝗛𝗮𝗱 𝗧𝗿𝗮𝗱𝗲-𝗼𝗳𝗳𝘀: 1️⃣ Pre-filtering: Apply filters first, then search • Great for small result sets • Becomes slow with larger filtered datasets, because you need to brute force 2️⃣ Post-filtering: Search first, then filter • Challenging to determine how many results to fetch • Can waste resources or miss relevant results 𝗘𝗻𝘁𝗲𝗿 𝗔𝗖𝗢𝗥𝗡! 🌳 Original paper: https://lnkd.in/eJhPkeTK Acorn uses a two-hop based expansion of the neighborhood, evaluating nodes that are two hops away rather than one. This speeds up graph traversal and helps maintain connectivity even when filtered nodes are sparse. Weaviate took the concepts from the paper’s implementation, but made a few key changes: • Only uses two-hop expansion when needed • Keep regular HNSW graph, no reindexing • Additional entry points to better handles cases where filters and queries have low correlation Practical Benefits: • No reindexing required for existing users • Up to 10x performance improvement in challenging scenarios • Maintains good performance across different use cases 𝗧𝗵𝗲 𝗥𝗲𝘀𝘂𝗹𝘁𝘀? 📊 • With low correlation queries: ACORN significantly outperforms traditional methods • With high selectivity filters: Automatically switches back to optimize performance • Especially shines when filters are restrictive and have low correlation with queries Learn more in this blog post: https://lnkd.in/ejUFU3uv Or try it out with Weaviate 1.27: https://lnkd.in/e2EHCEty

If you're building an AI app with PostgreSQL/ pgvector, you'll probably need to add filters to your semantic search to get better results. Here's an overview of 5 most useful filter options for building search and RAG apps: 𝟏/ 𝐌𝐞𝐭𝐚𝐝𝐚𝐭𝐚 𝐟𝐢𝐥𝐭𝐞𝐫 Use case: A technical documentation search system for a software company with multiple products. This query searches for documents related to the CRM Software product, specifically API reference documents. 𝟐/ 𝐂𝐨𝐦𝐩𝐨𝐬𝐢𝐭𝐞 𝐟𝐢𝐥𝐭𝐞𝐫 Use case: An e-commerce product recommendation system that considers both user preferences and product attributes. This query combines multiple filters to find relevant products within specific categories, price range, stock status, and rating. 𝟑/ 𝐓𝐢𝐦𝐞-𝐛𝐚𝐬𝐞𝐝 𝐟𝐢𝐥𝐭𝐞𝐫 Use case: A news article recommendation system that prioritizes recent content. This query retrieves semantically similar news articles published within the last 7 days. 𝟒/ 𝐏𝐞𝐫𝐦𝐢𝐬𝐬𝐢𝐨𝐧𝐬-𝐛𝐚𝐬𝐞𝐝 𝐟𝐢𝐥𝐭𝐞𝐫 Use case: A company-wide knowledge base where access to certain documents is restricted based on user roles. (This is super common for internal RAG apps!) This query ensures that only documents the user has permission to access are included in the search results. 𝟓/ 𝐆𝐞𝐨-𝐬𝐩𝐚𝐭𝐢𝐚𝐥 𝐟𝐢𝐥𝐭𝐞𝐫 Use case: A location-based service recommendation system for tourists. This query combines semantic search with geospatial filtering to find tourist attractions within a 5km radius of the user's location, ordered by semantic relevance and distance. 𝐏𝐫𝐨-𝐭𝐢𝐩: Use pgvectorscale and the StreamingDiskANN index to get higher search accuracy for filtered search. You can learn more about how streaming filtering works in the blog post linked in the comments: pgvectorscale is open-source and free to use under the PostgreSQL license. Head over to GitHub to get started. I hope this thread was helpful, let me know what other topics you'd like me to cover regarding building RAG and search systems with Postgres and pgvector. #semanticsearch #vectordatabase #postgresql #rag #pgvector #devtool #pinecone #search