Incorporating the confidence in the training data into the ML model I am wondering if there has been any research into how to incorporate our  confidence in each training data into the machine learning model. Specifically I have a bunch of training data for each I know how reliable they are 
X1: 95%
X2: 70%
X3: 10%

This means that there is 95% probability that the label for X1 is "True" and 5% chance that it is "False". Similarly X2 is True with the probability of 70% and False with 30% probability. And finally the probability of X3 having label "True" is only 90%. Note that these are training data. 
I am using a random forest classification model and training on this data. Is there any trick for me to use the confidence to do a better training? 
I looked for research papers but unfortunately could not find anything relate to this problem. 
 A: You have a training set $\{ (X_i, p_i) \}_{i=1}^n$, where the corresponding label $y_i \sim \mathrm{Bernoulli}(p_i)$. I'll assume that the $y_i$s are independent of one another. Call $\mathcal P$ the distribution of $\{ (X_i, y_i) \}_{i=1}^n$.
Then one way forward is to consider drawing multiple datasets $\mathcal D_j = \{ (X_i, y_{ij}) \}_{i=1}^n$ from $\mathcal P$, training a forest on each $\mathcal D_j$, and predicting according to the most common prediction from the different classifiers. This approximates taking the expectation of the learned model under $\mathcal P$.
In random forests, it turns out that we can basically approximate this more directly.
Construct a weighted dataset $\{(X_i, 1, p_i)\}_{i=1}^n \cup \{ (X_i, 0, 1-p_i) \}_{i=1}^n$, i.e. you include each data point once with each label, weighted according to how likely you think it is that the data point has that label. Many random forest implementations support these weights, and use them to determine how likely you are to sample the data point into a bootstrap replication for a tree. For example, scikit-learn's RandomForestClassifier has a sample_weight argument to the fit method.
A: [edit: chose not to delete answer. What Dougal wrote is clearly a better answer]
Random forest will effectively apply an prior expectation matching the class distribution of the training set. You can modify this distribution with stratification/downsampling or apply class weights during training. You can also modify the voting aggregation rule. You can use the vote ratio as predicted pseudo probabilities. Use ROC plots to investigate sensitivity and selectivity.
Here's another answer including coding in R.
