White Paper
 

Testing Predictive Models in Production

Learn how to create A/B and multi-armed bandit tests to assess the performance of predictive models already in production.

Many predictive models, from customer lifetime value models to recommendation engines, are trained on historical data that only captures the state of a business at some point in time. Once deployed, a model might not replicate its performance on new, unseen data — meaning you might not have picked the best model for the job. DataScience.com Lead Data Scientists Jean-René Gauthier, Ruslana Dalinina, and Pramit Choudhary explain how to overcome this barrier to success with testing methods that can be used on models already in production. 

model-testing-white-paper-cover
checkmark
Historical data may not capture interdependencies between different model predictions.
checkmark
Stopping an A/B test too soon is a very common mistake that can produce unreliable results.
checkmark
A cost-benefit analysis is an important part of deciding whether to implement a new predictive model.
checkmark
Compared to A/B testing, multi-armed bandits are a more adaptive method of testing model performance.