Recommendation Performance Metrics

Metrics Myopia: a common Information Retrieval affliction. 🧐📊 Symptoms include 95% precision but 0% user retention. Prescription: understand the metrics that actually matter. 💊 Order-Unaware Metrics: Precision in Simplicity 🎲 These metrics give you a straightforward view of your system's effectiveness, without worrying about results order. 1️⃣ Precision • What It Tells You: The accuracy of your retrieval—how many of the retrieved items are actually relevant. • When to Use: When users expect to get correct results right off the bat. 2️⃣ Recall • What It Tells You: The thoroughness of your retrieval—how many of all relevant items you managed to find. • When to Use: When missing information could be costly. 3️⃣ F1-Score • What It Tells You: The sweet spot between precision and recall, rolled into one metric. • When to Use: When you need to balance accuracy and completeness. Order-Aware Metrics: Ranking with Purpose 🏆 These metrics come into play when the order of results matters as much as the results themselves. 1️⃣ Average Precision (AP) • What It Tells You: How well you maintain precision across different recall levels, considering ranking. • When to Use: When assessing ranking quality for individual queries is crucial for your system's performance. 2️⃣ Mean Average Precision (MAP) • What It Tells You: Your system's average performance across multiple queries. • When to Use: For system evaluations, especially when comparing different models across diverse query types. 3️⃣ Normalized Discounted Cumulative Gain (NDCG) • What It Tells You: How well you're prioritizing the most relevant results and how quickly the first relevant result appears. • When to Use: In user-focused applications where top result quality can make or break the user experience. 4️⃣ Mean Reciprocal Rank (MRR) • What It Tells You: How quickly you're retrieving the first relevant item. • When to Use: When speed to the first correct answer is key, like in Q&A systems or chatbots. Choosing the Right Metric 🎯 The key is to align your metric choice with your system's goal. What matters most? • Precision? Go for Precision or MRR. • Completeness? Opt for Recall or F1-Score. • Ranking order? NDCG or MAP are your best bets. No single metric tells the whole story. Combine metrics strategically to gain a 360 review of your system's performance: • Pair Precision with Recall to understand both accuracy and coverage. • Use NDCG alongside MRR to evaluate both overall ranking quality and quick retrieval of top results. • Combine MAP with F1-Score to assess performance across multiple queries while balancing precision and recall. Finally, regularly reassess your metric choices as your system evolves and user needs change!

If you want to know where the money is in Machine Learning, look no further than Recommender Systems! Recommender systems are usually a set of Machine Learning models that rank items and recommend them to users. We tend to care primarily about the top-ranked items, the rest being less critical. If we want to assess the quality of a specific recommendation, typical ML metrics may be less relevant. Let’s take the search results of a Google search query, for example. All the results are somewhat relevant, but we need to make sure that the most relevant items are at the top of the list. To capture the level of relevance, it is common to hire human labelers to rate the search results. It is a very expensive process and can be quite subjective since it involves humans. For example, we know that Google performed 757,583 search quality tests in 2021 using human raters: https://lnkd.in/gYqmmT2S. Normalized Discounted Cumulative Gain (NDCG) is a common metric to exploit relevance measured on a continuous spectrum. Let’s break that metric down. Using the relevance labels we can compute diverse metrics to measure the quality of the recommendation. The cumulative gain (CG) metric answers the question: How much relevance is contained in the recommended list? To get a quantitative answer to that question, we simply add the relevance scores provided by the labeler: CG = relevance 1 + relevance 2 + ... The problem with cumulative gain is that it doesn’t take into account the position of the search results. Any order would give the same value however we want the most relevant items at the top. Discounted cumulative gain (DCG) discounts relevance scores based on their position in the list. The discount is usually done with a log function, but other monotonic functions could be used: DCG = relevance 1 / log(position 1) + relevance 2 / log(position 2) + ... DCG is quite dependent on the specific values used to describe relevance. Even with strict guidelines, some labelers may use high numbers and others low numbers. To put those different DCG values on the same level, we normalize them by the highest value DCG can take. The highest value corresponds to the ideal ordering of the recommended items. We call the DCG for ideal ordering the Ideal Discounted Cumulative Gain (IDCG). The Normalized Discounted Cumulative Gain (NDCG) is the normalized DCG NDCG = DCG / IDCG If the relevance scores are all positive, then NDCG is contained in the range [0, 1], where 1 is the ideal ordering of the recommendation. #MachineLearning #DataScience #ArtificialIntelligence

🚀 Part 2 of the '𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐘𝐨𝐮𝐫 𝐎𝐰𝐧 𝐑𝐞𝐜𝐨𝐦𝐦𝐞𝐧𝐝𝐞𝐫 𝐒𝐲𝐬𝐭𝐞𝐦𝐬!' Series is now live 🚀 Co-authored with Arun Subramanian, we dive into Evaluating Recommender Systems, covering: 🔹 Metrics like Precision, Recall, and Hit Rate—and how to use them. 🔹 Balancing accuracy, diversity, and novelty to meet user needs. 🔹 Real-world evaluation methods, from offline testing to A/B experiments. 💡 Evaluating isn’t just about accuracy—it’s about creating systems that are truly impactful for users. Read more: https://lnkd.in/eqh9-q35 Link to Part 1, which focused on different types of recommender systems: https://lnkd.in/e_4wmydi 📬 Want to follow along? Subscribe to the newsletter for updates and practical insights:  https://lnkd.in/eHdP_9Kr

𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞 𝐈𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 𝐐𝐮𝐞𝐬𝐭𝐢𝐨𝐧: Your model disproportionately recommends popular/expensive items, reducing diversity. How would you quantify and mitigate popularity and price bias in a recommender system? 𝐏𝐨𝐩𝐮𝐥𝐚𝐫𝐢𝐭𝐲 𝐁𝐢𝐚𝐬 Popular items are shown more frequently than less popular ones. To quantify this, we can calculate metrics like - ✅ Skewness of Item Frequency Distribution: Check the distribution of item exposure (how many times items are recommended to users). A high skew indicates over-representation of popular items. ✅ Popularity Bias Index: This could be a ratio of the proportion of recommendations from the top-N most popular items versus the rest. 𝐏𝐫𝐢𝐜𝐞 𝐁𝐢𝐚𝐬 Expensive items are disproportionately recommended. To quantify price bias, we can - ✅ Price Distribution of Recommended Items: Calculate the average price of recommended items and compare it with the average price of all items in the catalog. ✅ Price-to-CTR Correlation: Measure how price correlates with the click-through rate (CTR). A strong correlation suggests price bias. To address these we can use - 𝐖𝐞𝐢𝐠𝐡𝐭𝐞𝐝 𝐋𝐨𝐬𝐬 𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧: Modify the loss function during training to penalize over-prediction of popular items. This can be done by giving inverse popularity weights to the items during model training. 𝐃𝐢𝐯𝐞𝐫𝐬𝐢𝐭𝐲 𝐂𝐨𝐧𝐬𝐭𝐫𝐚𝐢𝐧𝐭𝐬: After the model has made its recommendations, we can enforce a diversity constraint by using a diversity penalty. The system can recombine the top-N recommendations by penalizing duplicate categories, genres, or other similar features. 𝐏𝐫𝐢𝐜𝐞 𝐍𝐨𝐫𝐦𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧: Normalize the predicted score for expensive items so that they don’t dominate the ranking. This can be achieved by applying a log transformation on the price before feeding it into the model, or using a price decaying factor that lowers the predicted score as the price increases. 𝐏𝐫𝐢𝐜𝐞 𝐑𝐞𝐛𝐚𝐥𝐚𝐧𝐜𝐢𝐧𝐠: Adjust the recommendation list by introducing a price constraint after inference, where we ensure that the final list has a mix of low, medium, and high-priced items. This can be done through a cost-based penalty that discourages recommending items above a certain price threshold. 𝗟𝗶𝗸𝗲 for more such content. 𝐒𝐮𝐛𝐬𝐜𝐫𝐢𝐛𝐞 to the FREE substack - https://lnkd.in/g5YDsjex and YT Channel - https://lnkd.in/gttKwJtd to land your next Data Science Role. 𝗙𝗼𝗹𝗹𝗼𝘄 Karun Thankachan for all things Data Science.