Nearly every time I lead a machine learning course, it becomes clear that there is a fundamental acceptance that cross validation should be done…and almost no understanding as to why it should be done—or how it is actually done in a real-world workflow. Finally I’ve decided to move my answers from the white board to the blog post. Hope this helps!
Cross validation is like standing in a line in San Francisco: Everyone’s doing it, so everybody does it, and if you ask someone about it, chances are they don’t know why they’re doing it.
Fortunately for those who might be doing so without fully knowing why, there is very good reason to cross validate (which is generally not true about joining any random Bay Area queue).
So, other than our now inbred inclination as data scientists to do the thing that everyone else is doing…why should we cross validate?
The Why & the When
To discuss why we perform cross-validation, it’s easiest to review how and when we incorporate cross validation into a data science workflow.
To set the stage, let’s pretend we’re not doing a GridSearchCV. Just good old fashioned cross validation.
And, let’s forget about time. Time of observation usually matters quite a bit…and that complicates things…quite a bit. For right now let’s pretend time of observation doesn’t matter.
First Partition of Data
Divide your data into a holdout set (maybe you prefer to call it a testing set?) and the rest of your data.
What’s a good practice here? If you’re data isn’t too big, import your entire_dataset, randomly sample [your chosen holdout percent]% of observation ids (or row indices). Create two new data frames, holdout_data and rest_of_data. Clear your variable entire_dataset. Export holdout_data to the file or storage type of your convenience, and clear your variable holdout_data.
If your data is too big, adapt a similar workflow by using indices instead of data frames and only loading data when necessary.
To clear a variable in python, you’re welcome to del your variable or set it to Null, as you’d like. Just make sure you can’t access the data by accident.
EDA & Feature Engineering
Let’s assume we then follow good practices around exploratory data analysis and creating a reusable feature engineering pipeline.
Notice that we’re only using the rest_of_data for EDA and designing our feature engineering pipeline. Let’s think about this for a second. Only doing this on the rest_of_data allows us to test the entire modeling process when we test on the holdout data. Clever, right?
Now, we’ve got our data ready to go and we have an automated way to process new data, via our pipeline.
Specify a Model
We need to specify a model, e.g. choose to use ‘random forest’. For this non-grid-search scenario, we’re going to also say this is where tuning occurs.
Cross Validation
Now we cross-validate.
You know the idea: take the rest_of_data, partition it into k folds (partition). For step j[j]_train_set (all the data but the jth partition) and a [j]_test_set (the jth partition). Train on the training set, test on the test set, record the test performance, and throw away the trained model. Do this for all k partitions.
Or, more likely, have sklearn do it for you.
Go ahead, we’ll wait.
Now we have k data points of “sample performance” of our specified model.
We inspect these results by looking at the mean (or median) score and the variance. This is the key to why we cross validate.
Why We Perform Cross Validation:
We want to get an idea of how well the specified model can generalize to unseen data and how reliable that performance is. The mean (or median) will gives us something like expected performance. The variance gives us an idea of how likely it is to actually perform near that ‘expected’ performance.
I’m not making this up–you can look at the sklearn docs and they, too, look at the variance. Some people even plot out the performances.
If you’re coming from a traditional analytics or statistics background, you might think “don’t we have theories for that? don’t we have p-values for that? don’t we have modeling assumptions for that?” The answer is, generally, no. We don’t have the same strong assumptions around the distribution of data or the performance of models in broader machine learning as we do in, say, linear models or GLMs. As a result, we need to engineer what we can’t theory [sic]. Cross validation is the engineered answer to “how well will this perform?”
Assuming the performance measures are acceptable, we’ve now found a specification of a model we like. Huzzah! (Otherwise, go back and specify a different model and repeat this process.)
Note: we did not get a trained model; we got a bunch of trained models, none of which we will actually use and all of which we will immediately discard. What we found was that the “specified model”, e.g. random forest, was a good choice for this problem.
Get the Performance Estimate
Armed with a specified model we like, we train this specified model on the entirety of rest_of_data. Sweet.
Now we load the holdout_data and run it through the feature engineering pipeline. We test our trained-on-rest-of-data specified model on the holdout_data. This gives us our performance estimate.
We hold a silent sort of intuition about performance at this step because rest_of_data is necessarily larger than any training set used during cross validation.
The intuition is that the specified model trained on this larger set would not have higher variation in performance and on average will perform at least as well as the average performance seen during cross validation. However, to check this we would need to repeat the steps from “First partition of Data” through this one multiple times.
NOTE: We do not assume it will perform at or better than average performance seen in cross validation. That is a very different assumption.
NOTE: we did not get a trained model; we got a trained model which we will immediately discard. What we found was a better estimate for in-production performance of the “specified model”, e.g. random forest.
Train Your Final Model
Assuming our performance estimate is sufficient, now we train the specified model on the entire_dataset. This is the trained model we use for production and/or decision making. This is our final model.
Monitoring & Maintenance
Until, as always, we iterate.
Hope this gave a little more insight to cross-validation in practice!
Data Science at Work 