# Random generation of scores similar to those of a classification model

Hello fellow number crunchers

I want to generate n random scores (together with a class label) as if they had been produced by a binary classification model. In detail, the following properties are required:

• every score is between 0 and 1
• every score is associated with a binary label with values "0" or "1" (latter is positive class)
• the overall precision of the scores should be e.g. 0.1 (<- parameter of the generator)
• the ratio of scores with label "1" should be higher than overall precision in the top-section and lower in the bottom section (<- the "model-quality" should also be a parameter of the generator)
• the scores should be in such a way, that a resulting roc curve is smooth (and not e.g. that a bunch of scores with label "1" are at the top and the rest of the scores with label "1" is at the bottom of the list).

Does anyone have an idea how to approach this? Maybe via generation of a roc-curve and then generating the points from that cure? Thanks in advance!

-
wrote "the overall precision of the scores should be e.g. 0.1" do you mean you want the score to be a random variable with mean 0.1 ? why don't you generate a classification problem (say with gaussian variables) and use something fast and simple (like LDA) to solve it ? can you tell us more on your motivations ? –  robin girard Nov 2 '10 at 12:45
@Robin: No, I am talking about "precision" as measure of the quality of a binary classification model (see e.g. en.wikipedia.org/wiki/Sensitivity_and_specificity here). Regarding your second question: I require the scores for a simulation where multiple binary classification models are combined (more in the area of data processing than Data Mining). It is hard to explain the task in details, but I am pretty sure I need these scores :D –  steffen Nov 3 '10 at 16:02

Some time has passed and I think I might have a solution at hand. I will describe my approach briefly to give you the general idea. The code should be enough to figure out the details. I like to attach code here, but it is a lot and stackexchange makes it not easy to do so. I am of course happy to answer any comments, also I appreciate any criticism.

The code can be found below.

The strategy:

1. Approximate a smooth ROC-Curve by using the Logistic function in the interval [0,6]
2. By adding a parameter k one can influence the shape of the curve to fit the desired model quality, measured by AUC (Area Under Curve). The resulting function is $f_k(x)=\frac{1}{(1+exp(-k*x))}$. If k-> 0, AUC approaches 0.5 (no optimization), if k -> Inf, AUC approaches 1 (optimal model). As a handy approach, k should be in the interval [0.0001,100]. By some basic calculus, one can create a function to map k to AUC and vice versa.
3. Now, given you have a roc-curve which matches the desired AUC, determine a score by sample from [0,1] uniformly. This represents the fpr (False-Positive-Rate) on the ROC-curve. For simplicity, the score is calculated then as 1-fpr.
4. The label is now determined by sampling from a Bernoulli Distribution with p calculated using the slope of the ROC-Curve at this fpr and the desired overall precision of the scores. In detail: weight(label="1"):= slope(fpr) mutiplicated by overallPrecision, weight(label="0"):= 1 multiplicated by (1-overallPrecision). Normalize the weights so that they sum up to 1 to determine p and 1-p.

Here is an example ROC-Curve for AUC = 0.6 and overall precision = 0.1 (also in the code below)

Notes:

• the resulting AUC is not exactly the same as the input AUC, in fact, there is a small error (around 0.02). This error originates from the way the label of a score is determined. An improvement could be to add a parameter to control the size of the error.
• the score is set as 1-fpr. This is arbitrary since the ROC-Curve does not care how the scores look like as long as they can be sorted.

code:

# This function creates a set of random scores together with a binary label
# n = sampleSize
# basePrecision = ratio of positives in the sample (also called overall Precision on stats.stackexchange)
# auc = Area Under Curve i.e. the quality of the simulated model. Must be in [0.5,1].
#
binaryModelScores <- function(n,basePrecision=0.1,auc=0.6){
# determine parameter of logistic function
k <- calculateK(auc)

res <- data.frame("score"=rep(-1,n),"label"=rep(-1,n))
randUniform = runif(n,0,1)
runIndex <- 1
for(fpRate in randUniform){
tpRate <- roc(fpRate,k)

# slope
slope <- derivRoc(fpRate,k)

labSampleWeights <- c((1-basePrecision)*1,basePrecision*slope)
labSampleWeights <- labSampleWeights/sum(labSampleWeights)

res[runIndex,1] <- 1-fpRate # score
res[runIndex,2] <- sample(c(0,1),1,prob=labSampleWeights) # label

runIndex<-runIndex+1
}
res
}

# min-max-normalization of x (fpr): [0,6] -> [0,1]
transformX <- function(x){
(x-0)/(6-0) * (1-0)+0
}

# inverse min-max-normalization of x (fpr): [0,1] -> [0,6]
invTransformX <- function(invx){
(invx-0)/(1-0) *(6-0) + 0
}

#  min-max-normalization of y (tpr): [0.5,logistic(6,k)] -> [0,1]
transformY <- function(y,k){
(y-0.5)/(logistic(6,k)-0.5)*(1-0)+0
}

# logistic function
logistic <- function(x,k){
1/(1+exp(-k*x))
}

# integral of logistic function
intLogistic <- function(x,k){
1/k*log(1+exp(k*x))
}

# derivative of logistic function
derivLogistic <- function(x,k){
numerator <- k*exp(-k*x)
denominator <- (1+exp(-k*x))^2
numerator/denominator
}

# roc-function, mapping fpr to tpr
roc <- function(x,k){
transformY(logistic(invTransformX(x),k),k)
}

# derivative of the roc-function
derivRoc <- function(x,k){
scalFactor <- 6 / (logistic(6,k)-0.5)
derivLogistic(invTransformX(x),k) * scalFactor
}

# calculate the AUC for a given k
calculateAUC <- function(k){
((intLogistic(6,k)-intLogistic(0,k))-(0.5*6))/((logistic(6,k)-0.5)*6)
}

# calculate k for a given auc
calculateK <- function(auc){
f <- function(k){
return(calculateAUC(k)-auc)
}
if(f(0.0001) > 0){
return(0.0001)
}else{
return(uniroot(f,c(0.0001,100))$root) } } # Example require(ROCR) x <- seq(0,1,by=0.01) k <- calculateK(0.6) plot(x,roc(x,k),type="l",xlab="fpr",ylab="tpr",main=paste("ROC-Curve for AUC=",0.6," <=> k=",k)) dat <- binaryModelScores(1000,basePrecision=0.1,auc=0.6) pred <- prediction(dat$score,as.factor(dat\$label))
performance(pred,measure="auc")@y.values[[1]]
perf <- performance(pred, measure = "tpr", x.measure = "fpr")
plot(perf,main="approximated ROC-Curve (random generated scores)")

-
I'd appreciate if anyone with more skills in latex could edit the calculation in 4. so that it is more readable. –  steffen Nov 3 '10 at 16:04
since this is exactly what I wanted, I will mark it as " the answer". However, I am still open for suggestions and ideas. –  steffen Nov 4 '10 at 14:54