Evaluating Recommender Systems: Choosing the best one for your business

Together with the endless expansion of E-commerce and online media in the last years, there are more and more Software-as-a-Service (SaaS) Recommender Systems (RSs) becoming available today. Unlike 5 years ago, when using RSs was a privilege of large companies building their own RS in-house, spending a ginormous budget on a team of data scientists, today’s popularity of SaaS solutions make it affordable to use recommendation even for small- and medium-sized companies. A question that CTOs of such companies are facing when looking for the right SaaS RS is: Which solution is the best? Assuming that you still don’t have a RS, or you are not satisfied with you current RS, which solution should you choose?

In this article, I will cover two approaches:

• Offline evaluation in academic world (plus the Netflix Prize), searching for low prediction errors (RMSE/MAE) and high Recall/Catalog coverage. TLDR; just know these measures exists and you probably don’t wanna use them. But I still give a brief summary of them in case you are interested.
• Online evaluation in business world, searching for high Customer Lifetime Values (CLV), going through A/B-testing, CTR, CR, ROI, and QA. You should read this section if you are seriously considering recommendations boosting your business.

The Offline World = How Academics Do It?

RSs have been investigated for decades in academic research. There are lot of research papers introducing different algorithms, and to make the algorithms comparable, they use academic measures. We call these measures the offline measures. You don’t put anything into production, you just play with the algorithms in your sandbox and fine-tune them according to these measures. I personally researched these measures a lot, but from my today’s point of view, they are rather prehistoric. But even in the middle ages of 2006 in the famous Netflix Prize, a purely academic measure called the RMSE (root mean squared error) has been used.

Just to briefly explain how it works, it supposes your users explicitly rate your products with say number of stars (1=strong dislike, 5=strong like), and you have a bunch of such ratings (records saying that user A rated item X with Y stars) from the past. A technique called the split validation is used: you take only a subset of these ratings, say 80% (called the train set), build the RS on them, and then ask the RS to predict the ratings on the 20% you’ve hidden (the test set). And so it may happen that a test user rated some item with 4 stars, but your model predicts 3.5, hence it has an error of 0.5 on that rating, and that’s exactly where RMSE comes from. Then you just compute the average of the errors from the whole test set using a formula and get a final result of 0.71623. BINGO! That’s how good (or, more precisely, bad) your RS is. Or you may also use different formula and get the MAE (mean absolute error), which does not penalize huge errors (true 4 stars, predicted 1 star) that much, so you might only get 0.6134.

One tiny drawback here is that such a data almost doesn’t exist in the real world, or at least there is too few of it.

Users are too lazy and they won’t rate anything. They just open a web page and if they like what they see, they might buy it/consume it; if it sucks, they leave as fast as they came. And so you only have so-called implicit ratings in you web-server log or a database of purchases, and you can’t measure the number-of-stars error on them, simply because there are no stars. You only have +1 = user viewed a detail or purchased a product, and, typically, nothing else. Sometimes these are called the unary ratings, which you know from Facebook’s “Like” button: the rating is either positive, or unknown (the user just mightn’t know the content exists).

You can still use the split-validation on such data, even for your own offline comparison of SaaS recommenders. Say you take, by example, your purchases database, submit history of 80% users to the RS, and then, for each test user, submit only a few purchases and ask the RS to predict the rest. You may have hidden 4 purchased items and ask the RS for 10 items. You may get 0%, 25%, 50%, 75%, or 100% accuracy for that user, depending on how many of the hidden 4 appeared in the recommended 10. And this accuracy is called the Recall. You may average it over your whole test set and TADAAA! You result is 31.4159%, that’s how good your RS is.

Now honestly, even though the Recall is much more sane than RMSE, it still sucks. Say a test user watched 20 episodes of the same TV series, and you measure recall on her. So you hide episodes #18–20 and ask the RS to predict them from #1–17. It is quite easy task as the episodes are strongly connected, so you get recall 100%. Now, did your user discover something new? Do you want to recommend her such a content at all? And what brings the highest business value to you anyway? Say in online store, do you wish to recommend alternatives, or accessories? You should feel you’re getting on a very thin ice with recall.

And one more secret I will tell you: In some cases (not always, depends on your business!), it’s a fair strategy to recommend only the globally most popular items (a.k.a. bestsellers) to achieve reasonable recall. So here comes the Catalog coverage. Do you wish you users to keep discovering new and new content to stay loyal? Then you might want to recommend as many different items as possible. In the simplest case, to compute the Catalog coverage, just take your test users, ask for recommendation for each one of them, and put all the recommended items together. You obtain a large set of different items. Divide the size of this set by the total number of items in your entire catalog, and you get… 42.125%! That’s the portion of items your RS is able to ever recommend.

Now consider a bestseller model. It might have quite good recall, but almost zero coverage (5 constants items?). And take a random recommender. It has almost zero recall and 100% coverage. You might feel you’d like some compromise.

The above image comes from my (now very outdated) original research. You can see about 1000 different RS models drawn in the Recall-Coverage plane. Geeky, ain’t it? :) You might feel dizzy when choosing the best one, but I hope you feel that choosing some from the upper-right (“Pareto-optimal front”) could be a good choice.

To make your offline estimate even more robust, you can use cross-validation (Xval) instead of split-validation. Just divide your users into 10 folds and go in loop: always take 9 folds to build the model, and use the remaining 1 fold to do the validation. Average the results over these 10 runs.

Now you might say: What about my business? Measuring recall and coverage might be fine, but how are they related to my KPIs?

And you are right. To put SaaS RS on X-axis and $$$ on the Y-axis, we have to leave the offline world and go into the production!