In this tutorial, we will dive into recommendation systems.

You might not know what recommendation systems are but you see them everywhere on the internet.

Everytime you shop on Amazon and you see related products…

Or when Netflix recommends you something interesting to watch…

The purpose of a recommendation system is to predict a rating that a user will give to an item that they have not yet rated.

This rating is produced by analyzing either item characteristics or other user/item ratings (or both) to provide personalized recommendations to users.

There are 2 main approaches to recommendation systems:

• Content Filtering. Recommendations depend on item characteristics.
• Collaborative Filtering. Recommendations depend on user-item ratings.

In this tutorial we will work with the MovieLens Dataset. This dataset contains user generated movie ratings from the website MovieLens (https://movielens.org/).

It contains multiple files, but the ones we will use in this tutorial will be movies.dat and ratings.dat.

First we will download the dataset:

wget http://files.grouplens.org/datasets/movielens/ml-1m.zip
unzip ml-1m.zip
cd ml-1m/

Content Filtering

Here are the first rows of the movies.dat file. The file follows the format:

movie_id::movie_title::movie genre(s)

head movies.dat

1::Toy Story (1995)::Animation|Children's|Comedy
2::Jumanji (1995)::Adventure|Children's|Fantasy
3::Grumpier Old Men (1995)::Comedy|Romance
4::Waiting to Exhale (1995)::Comedy|Drama
5::Father of the Bride Part II (1995)::Comedy
6::Heat (1995)::Action|Crime|Thriller
7::Sabrina (1995)::Comedy|Romance
8::Tom and Huck (1995)::Adventure|Children's
9::Sudden Death (1995)::Action
10::GoldenEye (1995)::Action|Adventure|Thriller

With genres being separated by a pipe |.

We load now the movies file:

import pandas as pd
import numpy as np
movies_df = pd.read_table('movies.dat', header=None, sep='::', names=['movie_id', 'movie_title', 'movie_genre'])

movies_df.head()

Out[]:

In order to be able to work with the movie_genre column, we need to transform it to what is called “dummy variables”.

This is a way to convert a categorical variable (e.g. Animation, Comedy, Romance…), into multiple columns (one column named Action, one named Comedy, etc).

For each movie, these dummy columns will have a value of 0 except for those genres the movie has.

# we convert the movie genres to a set of dummy variables 
movies_df = pd.concat([movies_df, movies_df.movie_genre.str.get_dummies(sep='|')], axis=1)
movies_df.head()

Out[]:

So for example, the movie with an id of 1 Toy Story, belongs to the genres Animation, Children’s and Comedy, and thus the columns Animation, Children’s and Comedy have a value of 1.

movie_categories = movies_df.columns[3:]
movies_df.loc[0]

Out[]:

movie_id                                 1
movie_title               Toy Story (1995)
movie_genre    Animation|Children's|Comedy
Action                                   0
Adventure                                0
Animation                                1
Children's                               1
Comedy                                   1
Crime                                    0
Documentary                              0
Drama                                    0
Fantasy                                  0
Film-Noir                                0
Horror                                   0
Musical                                  0
Mystery                                  0
Romance                                  0
Sci-Fi                                   0
Thriller                                 0
War                                      0
Western                                  0
Name: 0, dtype: object

Content filtering is a simple way to build a recommendation system. Here, items (in this example movies) are mapped to a set of features (genres).

To recommend a user an item, first that user has to provide his/her preferences regarding those features.

So in this example, the user has to tell the system how much does he or she like each movie genre.

Right now we have all the movies mapped into genres. We just need to create a user and map that user into those genres.

Let’s create a user with strong preference for action, adventure and fiction movies.

from collections import OrderedDict


user_preferences = OrderedDict(zip(movie_categories, []))

user_preferences['Action'] = 5
user_preferences['Adventure'] = 5
user_preferences['Animation'] = 1
user_preferences["Children's"] = 1
user_preferences["Comedy"] = 3
user_preferences['Crime'] = 2
user_preferences['Documentary'] = 1
user_preferences['Drama'] = 1
user_preferences['Fantasy'] = 5
user_preferences['Film-Noir'] = 1
user_preferences['Horror'] = 2
user_preferences['Musical'] = 1
user_preferences['Mystery'] = 3
user_preferences['Romance'] = 1
user_preferences['Sci-Fi'] = 5
user_preferences['War'] = 3
user_preferences['Thriller'] = 2
user_preferences['Western'] =1

Once we have users with their movie genre preferences and the movies mapped into genres, to compute the score of a movie for a specific user, we just need to calculate the dot product of that movie genre vector with that user preferences vector.

# in production you would use np.dot instead of writing your own dot product function.
def dot_product(vector_1, vector_2):
    return sum([ i*j for i,j in zip(vector_1, vector_2)])

def get_movie_score(movie_features, user_preferences):
    return dot_product(movie_features, user_preferences)

Let’s compute the score of the movie ‘Toy Story’ (a children’s animation movie) for the sample user.

toy_story_features = movies_df.loc[0][movie_categories]
toy_story_features

Action         0
Adventure      0
Animation      1
Children's     1
Comedy         1
Crime          0
Documentary    0
Drama          0
Fantasy        0
Film-Noir      0
Horror         0
Musical        0
Mystery        0
Romance        0
Sci-Fi         0
Thriller       0
War            0
Western        0
Name: 0, dtype: object

toy_story_user_predicted_score = dot_product(toy_story_features, user_preferences.values())
toy_story_user_predicted_score

Out[]:

5

So for the user, Toy Story, has a score of 5. Which does not mean much by itself, but helps us comparing how good of a recommendation Toy Story is compared to other movies.

Let’s calculate the score for Die Hard (a thrilling action movie):

movies_df[movies_df.movie_title.str.contains('Die Hard')]

die_hard_id = 1036
die_hard_features = movies_df[movies_df.movie_id==die_hard_id][movie_categories]
die_hard_features.T 

Out[]:

note, 1023 is the dataframe row index for Die Hard, not the movie index in the movielens dataset

die_hard_user_predicted_score = dot_product(die_hard_features.values[0], user_preferences.values())
die_hard_user_predicted_score

Out[]:

8

So we see that Die Hard gets an score of 8 vs a 5 for Toy Story. So Die Hard would be recommended before Toy Story. Which makes sense, given this user’s preferences are skewed towards action packed movies.

Once we know how to calculate the score for one movie, providing movie recommendations for the user is as easy as calculating the score for all the movies and returning those with the highest scores.

def get_movie_recommendations(user_preferences, n_recommendations):
    # we add a column to the movies_df dataset with the calculated score for each movie for the given user
    movies_df['score'] = movies_df[movie_categories].apply(get_movie_score, 
                                                           args=([user_preferences.values()]), axis=1)
    return movies_df.sort_values(by=['score'], ascending=False)['movie_title'][:n_recommendations]
    
get_movie_recommendations(user_preferences, 10)    

Out[]:

2253                                       Soldier (1998)
257             Star Wars: Episode IV - A New Hope (1977)
2036                                          Tron (1982)
1197                              Army of Darkness (1993)
2559     Star Wars: Episode I - The Phantom Menace (1999)
1985                      Honey, I Shrunk the Kids (1989)
1192    Star Wars: Episode VI - Return of the Jedi (1983)
1111                                    Abyss, The (1989)
1848                                    Armageddon (1998)
2847                                  Total Recall (1990)
Name: movie_title, dtype: object

So the system recommends heavy action and scifi movies. Neat!

Content Filtering makes recommending to a new user very easy. Users just have to express their preferences once. However, Content Filtering shows some caveats:

• Need to map each item into the feature space. That means that any time a new item gets added, someone has to manually categorize that item.

• Recommendations are limited in scope. This means items can’t be categorized in new features.

So content filtering is maybe a too simple option nowadays, which leads us to…:

Collaborative Filtering

Collaborative filtering is another way of predicting user-item scores. This time though, we will use the existing user-item scores to predict the missing ones.

The assumption is that users get value from recommendations based on other users with similar tastes.

For this example we will use the ratings.dat file. This file follows the format:

user_id::movie_id::rating::timestamp

head ratings.dat

1::1193::5::978300760
1::661::3::978302109
1::914::3::978301968
1::3408::4::978300275
1::2355::5::978824291
1::1197::3::978302268
1::1287::5::978302039
1::2804::5::978300719
1::594::4::978302268
1::919::4::978301368

The MovieLens dataset provides us with a file that includes over 1 million movie ratings.

ratings_df = pd.read_table('ratings.dat', header=None, sep='::', names=['user_id', 'movie_id', 'rating', 'timestamp'])

# we dont care about the time the rating was given
del ratings_df['timestamp']

# replace movie_id with movie_title for legibility
ratings_df = pd.merge(ratings_df, movies_df, on='movie_id')[['user_id', 'movie_title', 'movie_id','rating']]

ratings_df.head()

Out[]:

The dataset is a matrix of users and movie ratings, so we convert the ratings_df to a matrix with a user per row and a movie per column.

ratings_mtx_df = ratings_df.pivot_table(values='rating', index='user_id', columns='movie_title')
ratings_mtx_df.fillna(0, inplace=True)

movie_index = ratings_mtx_df.columns

ratings_mtx_df.head()

Out[]:

We have a matrix of 6040 users and 3706 movies.

To compute similarities between movies, one way is to find the correlation between movies and then use that correlation to find similar movies to those the users have liked.

An easy way of doing this is in python is by using the numpy.corrcoef function, that calculates the Pearson Product Moment Correlation Coefficient (PMCC) between each item pair. the PMCC has a value between -1 and 1 that measures the correlation (positive or negative) between two variables.

A correlation matrix is a matrix of m x m shape, where element Mij represents the correlation between item i and item j.

corr_matrix = np.corrcoef(ratings_mtx_df.T)
corr_matrix.shape

Out[]:

(3706, 3706)

Note: We use the transposed ratings matrix to calculate the correlation matrix so it gives back the correlation between movies (rows). If we used the ratings matrix without transposing it, np.corrcoef would return the correlation between users.

Now, if we want to find similar movies to a specific movie, it’s just a matter of returning those movies that have a high correlation coefficent with that one.

favoured_movie_title = 'Toy Story (1995)'

favoured_movie_index = list(movie_index).index(favoured_movie_title)

P = corr_matrix[favoured_movie_index]


# only return those movies with a high correlation with Toy Story
list(movie_index[(P>0.4) & (P<1.0)])

Out[]:

['Aladdin (1992)',
 "Bug's Life, A (1998)",
 'Groundhog Day (1993)',
 'Lion King, The (1994)',
 'Toy Story 2 (1999)']

Now to provide recommendations to a user, we take the list of movies that user has rated. Then we sum the correlations of those movies with all the other ones and return a list of those movies sorted by their total correlation with the user.

def get_movie_similarity(movie_title):
    '''Returns correlation vector for a movie'''
    movie_idx = list(movie_index).index(movie_title)
    return corr_matrix[movie_idx]

def get_movie_recommendations(user_movies):
    '''given a set of movies, it returns all the movies sorted by their correlation with the user'''
    movie_similarities = np.zeros(corr_matrix.shape[0])
    for movie_id in user_movies:
        movie_similarities = movie_similarities + get_movie_similarity(movie_id)
    similarities_df = pd.DataFrame({
        'movie_title': movie_index,
        'sum_similarity': movie_similarities
        })
    similarities_df = similarities_df[~(similarities_df.movie_title.isin(user_movies))]
    similarities_df = similarities_df.sort_values(by=['sum_similarity'], ascending=False)
    return similarities_df

For example, let’s select a user with a preference for kid’s movies, and some action movies.

sample_user = 21
ratings_df[ratings_df.user_id==sample_user].sort_values(by=['rating'], ascending=False)

Out[]:

Now we provide movie recommendations to the sample user by using his list of rated movies as an input.

sample_user_movies = ratings_df[ratings_df.user_id==sample_user].movie_title.tolist()
recommendations = get_movie_recommendations(sample_user_movies)

# We get the top 20 recommended movies
recommendations.movie_title.head(20)

Out[]:

1939                     Lion King, The (1994)
324                Beauty and the Beast (1991)
1948                Little Mermaid, The (1989)
3055    Snow White and the Seven Dwarfs (1937)
647                     Charlotte's Web (1973)
679                          Cinderella (1950)
1002                              Dumbo (1941)
301                              Batman (1989)
3250            Sword in the Stone, The (1963)
303                      Batman Returns (1992)
2252                              Mulan (1998)
2924                Secret of NIMH, The (1982)
2808                         Robin Hood (1973)
3026                    Sleeping Beauty (1959)
1781                   Jungle Book, The (1967)
260         Back to the Future Part III (1990)
259          Back to the Future Part II (1989)
2558                          Peter Pan (1953)
2347             NeverEnding Story, The (1984)
97                  Alice in Wonderland (1951)
Name: movie_title, dtype: object

So we see that the system recommends mostly kid’s movies and some action movies. Neat!

Collaborative filtering is a widely used recommendation system nowadays. It is capable of recommending new items without having to manually define them. Also, it is able to find recommendations based on hidden features that an expert wouldn’t be able to find (for example, combination of genres or actors).

However, it has one mayor drawback. Collaborative filtering cannot recommend items for a new user until he/she has reviewed some items. This problem is called the Cold Start Issue.

One way recommender systems overcome this issue is by using a hybrid Content + Colaborative Filtering. That is, using colaborative filtering as well as content filtering when necessary.

Further reading

Here are a few interesting readings on Recommendation systems.

• A Programmer’s guide to Data Mining Ch:2
• Mining of Massive Datasets Ch:9. Recommendation Systems
• Collaborative Filtering: A tutorial