Timeline for Data snooping in selecting between ML models
Current License: CC BY-SA 3.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 19, 2016 at 11:37 | comment | added | PeterTschuschke | But now, in this differencial privacy new frame,¿how should i proceed? I try a lot of C values on training set, get my training errors, and test them one by one against the test set returning the test score according to the thresholdout algorithm? I think that this has no sense, because how do i choose between models with different C's? According to the test set score (which has noise) ? And how do i choose sigma and threshold? ¿sigma = 1.0/sqrt(len(sample))? threshold=k*sigma, k=2 or 3? | |
| Sep 19, 2016 at 11:22 | comment | added | PeterTschuschke | I'm having troubles trying to implement that. Let's say i have some classification problem, and i want a model, lets say a SVM with linear kernel.It only has a paremeter C. In the non-private way,roughly, i did cross-validation, select a value for C, try in the test set, and if test set accuracy is similar to cross validation estimation of the test error, that's the end. | |
| Sep 16, 2016 at 19:19 | comment | added | horaceT | @GeoMatt22 I've seen gaussian noise being used, too. I think the deal is about satisfying the $\delta, \epsilon$ bound. | |
| Sep 16, 2016 at 17:44 | comment | added | GeoMatt22 | I believe most of the differential-privacy theory uses Laplace noise for proofs. But I think the "ThresholdOut" demos do use Gaussian noise in practice (their Python demo for the Science paper, at least). | |
| Sep 16, 2016 at 17:09 | comment | added | horaceT | The type of noise and the dispersion of the noise variables determine the guarantee of differential privacy. See their Thresholdout algorithm, and Thm 9 following it for proof of the guarantee. | |
| Sep 16, 2016 at 16:56 | comment | added | PeterTschuschke | I must be missing something, why in slide 71 they pick normal noise? if (abs (sample_mean-holdout_mean)) < randon.normal.... @horaceT | |
| Sep 16, 2016 at 16:46 | history | edited | horaceT | CC BY-SA 3.0 |
add reference
|
| Sep 16, 2016 at 16:41 | comment | added | horaceT | @PeterTschuschke That's the general idea, but they pick Laplacian noise because of certain sample complexity guar.entee. See Theorem 14 in the paper cite above. | |
| Sep 15, 2016 at 18:23 | comment | added | PeterTschuschke | If I understood it, the idea is: add noise to the test set performance and use this as the real test set performance for that model. You've just made it re-usable. Depending on the similarity of train and test performances, you add noise in one way or another. I don't know if it works, but sounds good. Is this right? | |
| Sep 14, 2016 at 21:59 | comment | added | horaceT | @Cliff AB I hear you and I'll add an example to show how I think it should work. But your skepticism is justified, given what little else is out there right now. | |
| Sep 14, 2016 at 20:58 | comment | added | Cliff AB | To be clear on my side, I'm not saying there's nothing to it, but that I don't get it yet. There's a lot of things that work great that I don't get! I have it in my mind to thoroughly investigate this some day, but given that I haven't had the time, if someone else has I'm very interested. | |
| Sep 14, 2016 at 20:54 | comment | added | horaceT | @Cliff AB I think we crossed path before. Let me make it clear, I'm not saying differential privacy is the (only) holy grail in machine learning that completely solves the data snooping problem. But, this is a pretty neat trick on an otherwise intractable problem, isn't it. Before embarking on a 1000 model searches, this algorithm provides at least a sort of protective shield against overfitting. | |
| Sep 14, 2016 at 20:30 | comment | added | Cliff AB | Yes, and if you can explain why their method works, please answer the following question. I'm not saying there's nothing to this paper, but I am saying it doesn't make sense to me. I am very interested if someone can explain it a clear manner. | |
| Sep 14, 2016 at 19:10 | history | answered | horaceT | CC BY-SA 3.0 |