5
$\begingroup$

I'm currently working on a random forest classifier which is trained on sensitive data. I haven't been able to find specific answers to the following questions:

  • Can someone with access to the machine learning model extract individual sample data points from it? In particular I'm using the R package ranger.
  • In case the raw data is not accessible, can the training data points be recovered from the decision trees, and if, with how high accuracy?

To elaborate, the adversary would have access to the entire model, not just the API and predictions.

$\endgroup$
2
$\begingroup$

Potentially yes, the adversary that has access to the entire model can learn pretty much about your data.

Recall what random forests are: ensembles of decision trees trained on random subsets of data (usually both: subsets of rows and columns). Using the example from this tutorial, decision tree splits the data into branches, conditionally on the training data. The example below shows like the data is split based on the sex, age and the number of siblings & spouses abord, using the Titanic dataset. First split is based on sex and we learn from it that 36% of the dataset are females who survived the Titanic disaster. Next split shows that there is 61% of males older then 9.5 years who died during the disaster etc.

enter image description here

You can clearly see that the decision tree has memorized many of the characteristics of the dataset. The information is partial because we calculate only some conditionals (e.g. from the tree above we learned nothing about age of the males) and with continuous predictors we bin them into two categories at each split. Moreover, since while constructing random forest you took random samples from the data, the numbers won't be exact (the error depends on the fraction you subsample).

On another hand, since you usually use large number of such trees, taken together they give you more precise picture. As the trees were trained on random subsets, each tree can use different variables and different splits, together giving you more detailed view on the underlying data (single tree can make a split on age > 9.5, but multiple trees could make many different splits etc.). If you have many trees and many splits, then also the contentious variables will eventually get spitted into many different bins.

The answer on if data aggregated in such shattered was as it is done by random forest can be used to de-anonymize the underlying data, would depend on many things. First, it would depend on the data used for training the model: are the variables detailed enough to recognize the individual participants? If you recorded only some very broad demographic details, then possibly this may not be the case, but combinations of those features, or features with very skewed distributions (rare features) could be used for de-anonymization. Second, this would depend to great extent on the characteristics of the model: how big were the subsampling fractions?; how many trees did you use?; did you enforce the minimum number of cases per branch to be a relatively large value?; did you use any pruning? etc. Basically, the mode "tuned" the trees would be, the more noise you introduce and the more anonymized your data will be (but I guess, this would be very hard to measure and quantify!). If you use large fractions of the data for splits, many trees, allow for small branches and use no pruning, then the random forest may memorize quite detailed picture of your training data.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Well, learning characteristics from the training data is of course the purpose of a random forest model. I'm not too concerned with information leakage about the value distribution or rough measures of correlation, but rather with protecting the individual data points that make up the original data. $\endgroup$ – tremula111 Sep 10 '18 at 14:52
  • 1
    $\begingroup$ What do you mean by individual data? You possibly can learn about joint distributions of all the variables used for training. If you know some of the characteristics and they're detailed enough so that you can make educated guess about who is the individual described by the characteristics, that's what we call de-anonymization. $\endgroup$ – Tim Sep 10 '18 at 15:21
  • 1
    $\begingroup$ Say that the above decision tree had 100% accuracy and the protected information was if the individual from the training set died in Titanic disaster. Say that Jane took the trip by Titanic, can you tell me if she survived? What about her husband? $\endgroup$ – Tim Sep 10 '18 at 15:30
  • $\begingroup$ I see what you mean. Then that would require prior knowledge of the samples. I think, then, that the main thing is that a potential adversary could gain information on characteristics of the cohort. That would not be that bad. Thanks for the great explanation! $\endgroup$ – tremula111 Sep 10 '18 at 22:10
  • 1
    $\begingroup$ How else could you imaging de-anonymization based on (say) demographic characteristics? Surely random forest does not store full names and addresses (unless you trained it on them). $\endgroup$ – Tim Sep 10 '18 at 22:18
1
$\begingroup$

Decision tree is effectively a kind of lossy encoder (like hashmaps) meaning it is not possible to figure out exact inputs based on a model. I can imagine having a tree that is able to contain most or the whole knowledge about input data (extreme overfitting), but I guess it is not your case.

Also, training parameters come into play that adversaries may not know even if they have a model.

Having said that, the whole purpose of decision trees is to model training data and it all boils down to what is sensitive in your dataset. If this is distributions, trees very much pass this information on. If this is individual record attributes like ids, names, etc. this won't be retained. If this is relationships between features - one can use trees to reverse engineer that too.

Please keep in mind one has control over how precise a decision tree is and what data is being used to train it. This may be everything you need to keep it safe. Maybe some sensitive features are just not needed?

Please take a look at the concept of "decision tree autonecoders" this may shed some more light on the problem for you.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you! I do plan to explore other methods like anonymization and dimensionality reduction. To be clear, you seem to be talking about decision trees, but I'm using random forests - i.e. there can be an arbitrary number of trees. $\endgroup$ – tremula111 Sep 10 '18 at 14:47
  • $\begingroup$ I am aware of that. Random forests are built on top of decision trees. $\endgroup$ – Marcin Sep 10 '18 at 14:50
1
$\begingroup$

Do you mean from inside the R environment? It would be hard for someone to accomplish this, however they could go through the whole forest (if it's available at your_model_name$forest) and create some sort of summary / boundary for all the used variables and their respective values.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.