Recommendation Systems Walkthrough - Popularity Recommendations

This post discusses a different approach to recommending movies based on the movie’s popularity.

Unlocking Recommendations

The Foundational Power of Popularity-Based Systems

In this post, we will use the average rate available in the movies database. This approach is for building a more generalized recommendation widget based on the movie’s popularity.

This document outlines the concept and implementation of popularity-based recommendation systems, a foundational approach in movie recommendation engines. These systems prioritize items based on their broad appeal, making them essential for generalized recommendations and addressing the cold start problem.

Core Concept: Why Popularity Matters

Popularity-based recommendations suggest items based on their broad appeal to the general audience, rather than personalized user preferences. The underlying principle is that items with higher popularity are more likely to be enjoyed by a larger user base.

This method is crucial for creating generalized recommendation widgets (e.g., “Top 10 Movies This Week”) and for addressing the cold start problem (recommending to new users or new items with limited interaction data).

Mechanism & Challenges: Beyond Simple Averages

A simple approach involves sorting items by a pre-calculated aggregate metric, such as an average rating. However, a simple arithmetic average can be misleading. For instance, a movie with one 5-star rating is not as statistically reliable as a movie with 10,000 ratings averaging 4.0 stars. This highlights the need for more sophisticated weighting.

The approach here is pretty simple we sort the movies based on the pre-calculated average rate which is collected from different users.

IMDb’s Weighted Rating Formula (Bayesian Estimate)

IMDb uses a robust statistical approach for its “Top 250” lists, employing a Bayesian estimate for its weighted rating to stabilize ratings for items with fewer votes and prevent outliers from disproportionately influencing perceived popularity.

For the sake of demonstration, IMDB has a list of top-rated movies which is rated by different users. How is IMDb actually calculating these rates?

The formula for calculating the top rated 250 titles gives a true Bayesian estimate:

weighted rating (WR) = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C where:

• R = average for the movie (mean) = (Rating)
• v = number of votes for the movie = (votes)
• m = minimum votes required to be listed in the Top 250 (currently 25000)
• C = the mean vote across the whole report (currently 7.0)

Intuition: This formula blends a movie’s individual average rating (R) with the global average (C). When v (votes) is small, WR is pulled closer to C. As v increases and surpasses m (minimum votes), WR approaches R.

Python Implementation Example

def compute_weighted_average_rate(r, v, m, c):
    """
    Calculates the IMDb-style weighted rating for an item.

    Args:
        r (float): The average rating for the item.
        v (int): The number of votes for the item.
        m (int): The minimum votes required for consideration (regularization constant).
        c (float): The mean vote across the entire dataset (prior mean).

    Returns:
        float: The calculated weighted rating.
    """
    if v == 0: # Handle cases with no votes to avoid division by zero or NaN
        return c # Default to global average if no votes
    wr = (v / (v + m) * r) + (m / (v + m) * c)
    return wr

This criterion forms a solid basis for a generalized recommendation widget, useful when individual user profiles are unavailable or insufficient. Scores derived from this formula are typically recomputed periodically to reflect changing preferences and new data.

Data Representation & SQL Queries

Based on the previous criteria we can use it as a suggestion widget to suggest movies to all users given the IMDB popularity info or users’ average rate that will be recomputed from time to time.

Of course, it is not the best to give a recommendation, but I think it can be used along with other recommender algorithms.

A movies_app_movie table might store movie names, rates, and a pre-calculated popularity score. Assume for a moment we have the following table in our movies database which is fetched using this sql query:

SELECT name, rate, COALESCE(popularity, 0) AS popularity 
  FROM movies_app_movie 
 ORDER BY popularity DESC;

COALESCE(popularity, 0) ensures a default value of 0 if popularity is NULL. ORDER BY popularity DESC sorts from highest to lowest popularity. A movies_app_rating table stores individual movie ratings. Here’s how to aggregate ratings per movie:

Based on the previous table, we deduce that movies are sorted in descending order according to their popularity. We will go through how we can compute such info for every film in our database.

In order to compute the popularity property it is a bit tricky since we may need some of the following information which of course will be stored in a separate table to JOIN on later.

Let’s take a look at a separate table for the ratings of movies and it is required to do some aggregation to calculate the rating count of every movie.

  SELECT movie_id, 
         COUNT(movie_id), 
         AVG(rate) AS rate_avg 
    FROM movies_app_rating 
GROUP BY movie_id 
ORDER BY rate_avg DESC
   LIMIT 20;

COUNT(movie_id) calculates v (vote count). AVG(rate) calculates R (average rating). This aggregation highlights the issue of movies with high average > ratings but very few votes (e.g., 5.0 with 1 vote), reinforcing the need for weighted formulas.

Factors for Computing Popularity Scores

A robust popularity score often uses a weighted sum of dynamic and static factors, providing a comprehensive view of an item’s current and long-term appeal. The specific blend and weights are often proprietary and fine-tuned for the platform.

There are many ways to determine the popularity of a movie. There is no standard way of computing such a score, we can take the following factors into consideration for example:

• Number of votes for the day.
• Number of views for the day.
• Number of users who marked it as a “favorite” for the day.
• Number of users who added it to their “watchlist” for the day.
• Number of comments.
• Number of rates (Negative Vs. Positive).
• Number of total votes.

The Strategic Role of Popularity in Recommender Systems Architecture

1. Baseline and Fallback: They act as a robust baseline model and an excellent fallback mechanism for the cold-start problem for new users.
2. Addressing Cold-Start for Items (Partially): The IMDb weighted rating formula partially mitigates the cold-start problem for new items with few ratings by pulling them towards the global average.
3. Limitations and Trade-offs: Popularity-based systems inherently lack personalization, which can lead to a “filter bubble” and conflict with “beyond accuracy” metrics like diversity and novelty.
4. Component in Hybrid Systems: Popularity recommendations can be used alongside other algorithms in hybrid recommendation systems, serving as a candidate generator or a feature.

Conclusion

The “Popularity Recommendations” method, though simple, embodies the data science principle of extracting meaningful signals from noisy data, especially with sparsity. The Bayesian estimation in the IMDb formula transforms raw averages into statistically sound popularity scores. While modern systems focus on personalization through techniques like collaborative filtering and deep learning, popularity-based approaches remain vital for providing strong baselines, effective cold-start solutions for new users, and crucial components in sophisticated hybrid architectures. Its enduring appeal stems from its ease of implementation and its robust, generalized insight into collective preference.