Timeline for How should evaluate a Testing set using a pattern learned with PCA?
Current License: CC BY-SA 4.0
18 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 30, 2021 at 3:00 | history | tweeted | twitter.com/StackStats/status/1410070713197813761 | ||
| Jun 29, 2021 at 21:30 | vote | accept | DavidZoy | ||
| Jun 29, 2021 at 21:02 | history | edited | DavidZoy | CC BY-SA 4.0 |
More details about the question
|
| S Jun 29, 2021 at 20:34 | history | suggested | patagonicus |
this is related to how to perform CV properly.
|
|
| Jun 29, 2021 at 20:27 | review | Suggested edits | |||
| S Jun 29, 2021 at 20:34 | |||||
| Jun 29, 2021 at 20:21 | comment | added | patagonicus | @gunes and Dave: Many thanks for the interesting discussion. I have already express my view, so won't comment further but there is another relevant discussion which has a similar nature to ours : stats.stackexchange.com/questions/239898/… | |
| Jun 29, 2021 at 19:39 | comment | added | gunes | @MehmetSuzen thanks for the article link. I couldn't exactly find parts supporting your idea (just skimmed). It's not only SO community, for example, the data leakage section in sklearn documentation exactly describes this kind of scenario: scikit-learn.org/stable/common_pitfalls.html . Simply, introducing information about the test set may introduce some trends in features that may not be readily available in training data. Thus, target or not, introduction of test set always carry a danger of leakage, despite one might get away with it on occasions. | |
| Jun 29, 2021 at 19:37 | comment | added | Dave | @MehmetSuzen That practice is a poor one. The test set mimics data that do not yet exist. Remember that Siri is supposed to be able to do speech recognition on people who have not yet been born. | |
| Jun 29, 2021 at 19:21 | comment | added | patagonicus | @gunes Thank you for the link. I could see the general practice in the community. But I am not convinced that this would create any "target leakage". PCA do not interact with the labels. With the strict definition of "target leakage" there is no leakage if PCA is build on entire-set, as long as there is no target leakage inherently. ( see dl.acm.org/doi/10.1145/2382577.2382579 ) . DavidZoy's question, if PCA is build on training, test set can be projected separately stats.stackexchange.com/questions/405660/… | |
| Jun 29, 2021 at 19:16 | answer | added | gunes | timeline score: 3 | |
| Jun 29, 2021 at 19:14 | comment | added | DavidZoy | That's right @gunes, so should not be wrong by me doing the PCA on the Testing Set during the Validation Step of the KDD, should it? The PCA on the Testing Set will be related to the basic practices, such as Data Cleaning and Data Scaling etc., because in order to validate the pattern learned from the Training Set through PCA, I have to bring back the Testing Set by performing the PCA with the same number of components (used for the Training Set), correct me if I'm wrong. | |
| Jun 29, 2021 at 19:02 | comment | added | gunes | @MehmetSuzen it causes data leakage, a somewhat related discussion: stats.stackexchange.com/questions/55718/… | |
| Jun 29, 2021 at 18:50 | comment | added | patagonicus | Not really. PCA is "unsupervised". There is no peaking here. Test set is still unseen data, instances on the test set are not used in building the model. | |
| Jun 29, 2021 at 16:27 | history | edited | DavidZoy |
More tag
|
|
| Jun 29, 2021 at 15:34 | comment | added | DavidZoy | That's totally wrong, the reason is that you should train your model only on the training data, without using any information regarding the testing data. If you apply PCA on the whole data (including the test data) before training the model (so, before the split), then you in fact use some information from the test data. Thus, you cannot really judge the behaviour of your model using the test data, because it is not an unseen data anymore. | |
| Jun 29, 2021 at 14:33 | comment | added | patagonicus | One would apply the PCA to all dataset before splitting to test/train. | |
| Jun 29, 2021 at 11:50 | review | First posts | |||
| Jun 29, 2021 at 14:33 | |||||
| Jun 29, 2021 at 11:46 | history | asked | DavidZoy | CC BY-SA 4.0 |