1
$\begingroup$

First the problem: I am refering to lecture note, on page 480, 2nd paragraph, it mentions

If X is in fact useless for predicting Y given Z, then an adaptive bandwidth selection procedure (like cross-validation) should realize that giving any finite bandwidth to X just leads to over-fitting.

I am working with this UCI Dataset for room occupancy.

The dataset is simple, as an exercise, I am trying to find the causal Directed Acylic Graph for the following dimensions, Temperature, Humidity, CO2 and Light.

If I follow the quote above, if Temperature cannot predict CO2 given let's say Humidity, then it probably can be assumed that Temperature and CO2 are independent in causal DAG (ofcorse, I shall have to do it for Temperature with Light as well).

Now for prediction, I am considering a simple 3 layer feed forward neural network, where the inputs would be Temperature and Humidity and the output CO2. I shall train the model on the training set and validate it against the test set, which, if Temperature cannot predict CO2 given Humidity, the NN would overfit on the training set and give high Mean Square Error on test set.

However, I am not sure, if this scheme will work, as, may be Temperature does not have ability to predict CO2 but may be Humidity has (as the common confounder is occupancy in the dataset), so the model may predict CO2 with some good accuracy and will not tell me if Temperature and CO2 are cusally independent or not.

In this situation, how to proceed with the problem, given that I only have the observations. Any help n this will be appreciated

$\endgroup$
1
$\begingroup$

I was unaware of this algorithm for the test of independence. It is an interesting algorithm. I think the reason that this algorithm can determine the independence between $X$ and $Y$ given $Z$ is that if this independence relationship holds in data, adding $X$ beside $Z$ for predicting $Y$ gives no more information and adds unnecessary extra complexity to your model. This can cause overfitting since your model tries to take $X$ into account and because $X$ has nothing to say, the model just tries to model something like noise.

I also agree with you that this scheme may not work and in the lecture note that you referred, it is obviously mentioned: "slightly less rigorous". There are many other factors, such as the amount of the training data, that can affect overfitting which are neglected in this analysis. I suggest you use standard independence test such as chi-squared test. Also, this question seems also beneficial.

$\endgroup$
  • $\begingroup$ I refered the link to the other question, very interesting, I shall be going through the methods in more details. However, from your answer, if I train an NN with one input let's say Humidity and one output say CO2 and test it on test set and find some MSE. Again train an NN on 2 inputs, Humidity and Temperature and the 2nd NN's MSE on test set becomes higher than the 1st NN, then it would be reasonable to assume that adding Temperature added no more information but noise and Temperature and CO2 are independent to a large degree. $\endgroup$ – Ironluca Apr 6 '17 at 12:42
  • $\begingroup$ In my opinion, you should get such a conclusion with more caution. The higher MSE in the second NN can be due to other factors such as the low amount of the training data. In other words, it is possible that CO2 depends on Temperature besides Humidity, but your data is not enough to extract the true dependencies and therefore, it just overfits the data. $\endgroup$ – Hossein Apr 6 '17 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.