2
$\begingroup$

I am trying to learn convolutional neural networks from scratch.

I try to find a simple example that I can calculate by hand just to get the ideas.

There are many things that I do not understand so let's start simple.

I found some CNN examples that detect shapes like X, O, /, \ or faces like :) and :(, so just stuff that you can draw in a frame with 8x8 boxes.

In many examples filters are already given, but as I know filters are "trained" in the hidden layer via backpropagation.

So a CNN starts with filters with random values but I do not understand how the filters become what they are, I mean how a filter A becomes a detector for straight lines, or how a Filter B becomes a detector for curves. I just know that with some "magic" you end up with different filters and each filter is unique and detects a special shape/feature.

The other thing is who is telling me how many filter the CNN should have? Is it just intuition or just testing?

Whats about the filter size, in many example I saw that the dimensions of the filters 3x3, but why not 5x5 or other dimensions?

What would really help me is a small and simple step by step example.


Thanks for respond Firebug!

There are many things that I still do not get and that I have to study. For example I did not know that deep learning is largely based on heuristics/try and error.

I created a little example that I found and as you can see its very simplified enter CNN

In this examples the filters are given and the weights are manually changed so that we get an average of 0.8 for all votes. As far as I know I can decide what accuracy I accept, in this case we just decide that 80% is enough.

What really would help me is to see a step by step backpropagation by hand with all kernels and weights set to 1 so I can see how the kernel and the weights get the values that they get. I would also like to see the cost-function that is optimized.

It is very abstract for me at the moment but I hope you can help me out, then I can move on to the other details.

$\endgroup$
1
  • $\begingroup$ Do you understand how a fully-connected networks learns the parameter values? $\endgroup$
    – Dave
    Commented Oct 27, 2020 at 17:00

1 Answer 1

3
$\begingroup$

Deep learning is largely based on heuristics today. There are no hard answers for broad questions.

So a CNN starts with filters with random values but I do not understand how the filters become what they are, I mean how a filter A becomes a detector for straight lines, or how a Filter B becomes a detector for curves.

Optimization. When you present data and perform gradient descent, these types of Gabor-like filters tend to appear naturally in the first layers, implicating these forms are useful for natural images (the mammal visual system also use similar forms). There's no guarantee though.

The other thing is who is telling me how many filter the CNN should have? Is it just intuition or just testing?

Objectively? You.

You could, ideally, perform hyper-parameter tuning to decide that, or alternatively architectural search. The point is, never base such a decision on performance in validation/testing sets, that's the greatest sin in machine learning.

Whats about the filter size, in many example I saw that the dimensions of the filters 3x3, but why not 5x5 or other dimensions?

Again, heuristics. Keep in mind that two 3x3 filters have the same receptive field of a 5x5 filter. People use 3x3 most often, but many architectures employ many filter sizes as well.

$\endgroup$
2
  • $\begingroup$ Isn't hyper-parameter tuning based on the validation set performance? Completely agree that it is a huge error to change any hyperparameter based on the performance on the test set, though. $\endgroup$
    – Ciodar
    Commented Jun 16, 2023 at 15:45
  • 1
    $\begingroup$ @Ciodar yes, but when I wrote this answer I was actually referring to "test sets". It's just that a lot of people will call them "validation sets" as well. $\endgroup$
    – Firebug
    Commented Jun 16, 2023 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.