1
$\begingroup$

"Is a convolutional neural network biased?"

This came up in an interview I had a few years ago, and I’ve recently thought of it. I think it’s a misguided question. Imagine this related question.

“Is a linear model biased?”

I am not even sure what that means. Certainly we can say that the parameter estimates are unbiased under standard assumptions and the OLS estimator $\hat{\beta}_{ols}=(X^TX)^{-1}X^Ty$, but we get a biased parameter estimate if we use regularized regression like ridge or LASSO.

But it’s the same model either way, $\mathbb{E}[Y\vert X] = X\beta$, just different ways of estimating the parameters.

Back to CNNs, here’s what I think of a basic CNN with one $2\times 2$ filter over a $3\times 3$ image.

enter image description here

We have one parameter per color, and instead of drawing connections from (for instance) the top left pixel to the bottom hidden neuron, CNN sets that weight as $0$, so I have not drawn in a connection. Therefore, by forcing many parameters to be $0$ and others to be equal to each other, we regularize the fully-connected network that has the same number of neurons in the hidden layer (each pixel connected to each hidden neuron for a total of 36 parameters). That makes the CNN parameter vector biased (if the fully-connected later is unbiased). Throw in some maxpooling, and it certainly would be biased!

Is this about right?

$\endgroup$

2 Answers 2

1
$\begingroup$

If we consider a neural network to just be some mapping $f(x): X \mapsto Y,$ where $X$ and $Y$ are features and outcomes (basically, the term "AI" is just rebranding point estimation, which has been studied a lot in statistics), we can maybe make some progress on this question. In particular, we can note that a neural network, when trained on a dataset $(X_n,Y_n)$ is actually $f_n(x).$ The question of whether a neural network si "biased" is then whether $Ef_n(x)=f_0(x),$ where $f_0$ is the true mapping from $X$ to $Y$, chosen by nature.

In general, if a neural network can approximate any function, which I think it basically can, then definitely $f_n(x)\rightarrow^p f_0(x),$ for any $x$, or, in other words, as the sample size $n\rightarrow\infty$ then $f_n\rightarrow f_0$ in probability (ie, the probability $f_n$ and $f_0$ are different shrinks to zero), or $f_n$ is consistent. This is not the same as unbiased, and some biased estimators can be consistent. To show that a neural network is unbiased would requie that one evaluate the expectation $Ef_n(X)$ for a neural network by hand, which is, as far as I know, not really possible.

By regularizing in such a way that the regularization does not disappear in the limit, we are definitely causing bias that also will not disappear in the limit, and therefore also losing consistency. However, this can make sense to still do in the finite sample if the error from variance dominates the error from bias.

Looking at the "bias" of the individual weights $w_n$ in the network rather than of the function $f_n$ itself is problematic imo because the individual weights can be different and give the same function. There are not really "true" weights $w_0$ that we can assess in terms of their distance from $Ew_n$ to make statements about bias.

Now that we maybe have considered a fully flexible neural network, we can consider a CNN, which has tied parameters. This will definitely lead to bias. That said this is maybe a feature of the model, so maybe the function is consistent even under this constraint. Actually not sure about this.

$\endgroup$
1
  • 1
    $\begingroup$ +1 It seems that I was on the right track. $\endgroup$
    – Dave
    Commented Apr 2, 2023 at 17:36
0
$\begingroup$

You might want to look here and the references in the paper.

Chernozhukov, Victor, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins (2018). “Double/debiased machine learning for treatment and structural parameters”. In: The Econometrics Journal.

It would depend on what kind of activation function, but estimations done with CNNs would exhibit N^(1/4) convergence, with which CLT could work.

$\endgroup$
1
  • $\begingroup$ Welcome to Cross Validated! And thanks for the reference! What does the central limit theorem have to do with this, though? $\endgroup$
    – Dave
    Commented Nov 1, 2022 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.