Optimization in deep learning

Hi Hady, That's an excellent question and is especially relevant when you go on to learn about CNNs (Convolutional Neural Networks). Consider one step of training your network, i.e. one input/output pair.
The basic forward propagation recipe is: input + weights = output, where "+" stands for all the linear algebra you perform under the hood of multiplying matrices etc.
After that you run backpropagation in order to adjust the weights: weights <= weights + correction. Having this in mind, how would we approach the problem of finding an input that produces a certain output, given the trained network, i.e. the weights?
In the exact same way!
The recipe is the same: input + weights = output.
This time around, however, you keep the weights fixed, and you vary the input values, so the update step changes to: input <= input + correction. Using this strategy you can find inputs that correspond to arbitrary outputs, not simply minimal / maximal outputs. To summarise: Training the network: fixed input + varying weights = output
Finding an input that produces a certain output: varying input + fixed weights = output As I mentioned, this is particularly relevant for CNNs, and I would recommend the following article as an example description of why this is the case: https://arxiv.org/pdf/1412.0035.pdf Hope this helped, and please feel free to ask more questions as they arise. Stay safe and keep learning!
Iskren