Since we want to only test a model with data it truly hasn't seen, at what point is the model created? If I were to restart my Jupyter kernel and rerun the code, would this be a new model that could be tested again? What if I rerun the chunk where `model` is defined? Thanks!

Hi, Well, what we actually want to avoid is the making of a decision based on some result from the test set. The actual important component is our mind, as the experiment runner. Imagine you see that a model has 78% accuracy on a test dataset, and then you restart the kernel and try out a slightly different model that has 83% accuracy on the SAME test dataset. A reasonable conclusion one might make is that the second model is better than the first one. And that may be true, it probably is true. HOWEVER, you no longer can say that the model has 83% accuracy, because you have already used this test dataset a couple of times, so at this point you cannot say anything about the accuracy of the model. As soon as we start comparing test accuracies and making decisions based on those accuracies, we open up the possibility of overfitting on that dataset. That's why we say that we can only test a model on a dataset it has never seen before. In reality, it may be more accurate to say that the test dataset should be one WE (not the model) have never seen before. If you test a model, but don't see the result (thus, you can't make any decisions based on the result), then you can safely retrain and retest the model again. Hope this answers your question! Best, 365 Team, Nikola

Follow this topic

Westley Winks

Last answered:

11 Jan 2022

Posted on:

11 Jan 2022

Resolved: Can you restart/rerun code and test again?

in Convolutional Neural Networks with TensorFlow in Python / Testing the trained CNN

1 answers ( 1 marked as helpful)

Nikola Pulev

Instructor

Posted on:

11 Jan 2022

Helpful answer

Resolved: Can you restart/rerun code and test again?

Submit an answer

related questions