Hi Hady,
That’s an excellent question and is especially relevant when you go on to learn about CNNs (Convolutional Neural Networks).
Consider one step of training your network, i.e. one input/output pair.
The basic forward propagation recipe is: input + weights = output, where “+” stands for all the linear algebra you perform under the hood of multiplying matrices etc.
After that you run backpropagation in order to adjust the weights: weights <= weights + correction.
Having this in mind, how would we approach the problem of finding an input that produces a certain output, given the trained network, i.e. the weights?
In the exact same way!
The recipe is the same: input + weights = output.
This time around, however, you keep the weights fixed, and you vary the input values, so the update step changes to: input <= input + correction.
Using this strategy you can find inputs that correspond to arbitrary outputs, not simply minimal / maximal outputs.
To summarise:
Training the network: fixed input + varying weights = output
Finding an input that produces a certain output: varying input + fixed weights = output
As I mentioned, this is particularly relevant for CNNs, and I would recommend the following article as an example description of why this is the case: https://arxiv.org/pdf/1412.0035.pdf
Hope this helped, and please feel free to ask more questions as they arise.
Stay safe and keep learning!
Iskren