Wednesday, March 25, 2009

What is a good recommendation algorithm?

Posted by Danny Tarlow
Greg Linden has a post about what criteria should be used to evaluate how well a recommendation system works. He makes a case against the Netflix-style root mean squared error (RMSE) measure, and raises the point that the right model might be closer to web search, where the goal is to return the top N movies that you might want to see. He stays at a fairly high level and doesn't make too many strong statements, but it's a reasonably interesting read. One aspect that I think is important but he completely skips is the bias that people have towards rating things that they like. This makes it difficult even to evaluate the RMSE faithfully, because the ratings that you have access to for each user are typically not a truly random sample. Ben Marlin has done a lot of interesting work in this area that I think deserves mention. For example, Collaborative Filtering and the Missing at Random Assumption:
Abstract: Rating prediction is an important application, and a popular research topic in collaborative filtering. However, both the validity of learning algorithms, and the validity of standard testing procedures rest on the assumption that missing ratings are missing at random (MAR). In this paper we present the results of a user study in which we collect a random sample of ratings from current users of an online radio service. An analysis of the rating data collected in the study shows that the sample of random ratings has markedly different properties than ratings of user-selected songs. When asked to report on their own rating behaviour, a large number of users indicate they believe their opinion of a song \textit{does} affect whether they choose to rate that song, a violation of the MAR condition. Finally, we present experimental results showing that incorporating an explicit model of the missing data mechanism can lead to significant improvements in prediction performance on the random sample of ratings.
Related Posts:
Python recommendation system code

No comments: