Resolved: How does it work an activation function for a convolutional layer?
Good morning. I know what it means that a neuron, or a layer of neurons in feedforward NN, has a certain activation function. But a convolutional layer is not made up of neurons, it's just a set of different kernels of a particular size if I have well understood. So what does it mean to apply, for example, a ReLU to a convolutional layer? Maybe it means that every feature map will have the negative numbers replaced by 0? Or it means that every addend obtained by applying the kernel to a particular region of the image, if negative, is replaced by 0? Thank you
Hi,
Generally speaking, activation functions are independent of the underlying structure of the layer. In Neural Networks, the purpose of every layer is to transform numeric inputs into numeric outputs. So, at the end, every layer just "spits out" a bunch of numbers. The job of the activation functions is to further transform those numbers. They do not care how those numbers came to be, just what those numbers are.
In a convolutional layer, the output is a collection of feature maps. Thus, the activation function would act on all the numbers in those feature maps. Depending on the implementation and the function itself, it may behave differently for scalars, vectors, matrices or tensors (a feature map is a tensor).
Hope this helps!
Best,
Nikola, 365 Team
Quite clear now, thank you!