Resolved: Can you restart/rerun code and test again?
Since we want to only test a model with data it truly hasn't seen, at what point is the model created? If I were to restart my Jupyter kernel and rerun the code, would this be a new model that could be tested again? What if I rerun the chunk where `model` is defined?
Thanks!
Hi,
Well, what we actually want to avoid is the making of a decision based on some result from the test set. The actual important component is our mind, as the experiment runner. Imagine you see that a model has 78% accuracy on a test dataset, and then you restart the kernel and try out a slightly different model that has 83% accuracy on the SAME test dataset. A reasonable conclusion one might make is that the second model is better than the first one. And that may be true, it probably is true. HOWEVER, you no longer can say that the model has 83% accuracy, because you have already used this test dataset a couple of times, so at this point you cannot say anything about the accuracy of the model. As soon as we start comparing test accuracies and making decisions based on those accuracies, we open up the possibility of overfitting on that dataset. That's why we say that we can only test a model on a dataset it has never seen before.
In reality, it may be more accurate to say that the test dataset should be one WE (not the model) have never seen before. If you test a model, but don't see the result (thus, you can't make any decisions based on the result), then you can safely retrain and retest the model again.
Hope this answers your question!
Best,
365 Team, Nikola