Random generation of scores similar to those of a classification model Hello fellow number crunchers
I want to generate n random scores (together with a class label) as if they had been produced by a binary classification model. In detail, the following properties are required:


*

*every score is between 0 and 1

*every score is associated with a binary label with values "0" or "1" (latter is positive class)

*the overall precision of the scores should be e.g. 0.1 (<- parameter of the generator)

*the ratio of scores with label "1" should be higher than overall precision in the top-section and lower in the bottom section (<- the "model-quality" should also be a parameter of the generator)

*the scores should be in such a way, that a resulting roc curve is smooth (and not e.g. that a bunch of scores with label "1" are at the top and the rest of the scores with label "1" is at the bottom of the list).


Does anyone have an idea how to approach this? Maybe via generation of a roc-curve and then generating the points from that cure? Thanks in advance!
 A: Some time has passed and I think I might have a solution at hand. I will describe my approach briefly to give you the general idea. The code should be enough to figure out the details. I like to attach code here, but it is a lot and stackexchange makes it not easy to do so. I am of course happy to answer any comments, also I appreciate any criticism.
The code can be found below.
The strategy:


*

*Approximate a smooth ROC-Curve by using the Logistic function in the interval [0,6]

*By adding a parameter k one can influence the shape of the curve to fit the desired model quality, measured by AUC (Area Under Curve). The resulting function is $f_k(x)=\frac{1}{(1+exp(-k*x))}$. If k-> 0, AUC approaches 0.5 (no optimization), if k -> Inf, AUC approaches 1 (optimal model). As a handy approach, k should be in the interval [0.0001,100]. By some basic calculus, one can create a function to map k to AUC and vice versa. 

*Now, given you have a roc-curve which matches the desired AUC, determine a score by sample from [0,1] uniformly. This represents the fpr (False-Positive-Rate) on the ROC-curve. For simplicity, the score is calculated then as 1-fpr.

*The label is now determined by sampling from a Bernoulli Distribution with p calculated using the slope of the ROC-Curve at this fpr and the desired overall precision of the scores. In detail: weight(label="1"):= slope(fpr) mutiplicated by overallPrecision, weight(label="0"):= 1 multiplicated by (1-overallPrecision). Normalize the weights so that they sum up to 1 to determine p and 1-p.


Here is an example ROC-Curve for AUC = 0.6 and overall precision = 0.1 (also in the code below)


Notes: 


*

*the resulting AUC is not exactly the same as the input AUC, in fact, there is a small error (around 0.02). This error originates from the way the label of a score is determined. An improvement could be to add a parameter to control the size of the error.

*the score is set as 1-fpr. This is arbitrary since the ROC-Curve does not care how the scores look like as long as they can be sorted. 


code:
# This function creates a set of random scores together with a binary label
# n = sampleSize
# basePrecision = ratio of positives in the sample (also called overall Precision on stats.stackexchange)
# auc = Area Under Curve i.e. the quality of the simulated model. Must be in [0.5,1].
# 
binaryModelScores <- function(n,basePrecision=0.1,auc=0.6){
  # determine parameter of logistic function
  k <- calculateK(auc)

  res <- data.frame("score"=rep(-1,n),"label"=rep(-1,n))
  randUniform = runif(n,0,1)
  runIndex <- 1
  for(fpRate in randUniform){
    tpRate <- roc(fpRate,k)

    # slope
    slope <- derivRoc(fpRate,k)

    labSampleWeights <- c((1-basePrecision)*1,basePrecision*slope)
    labSampleWeights <- labSampleWeights/sum(labSampleWeights)

    res[runIndex,1] <- 1-fpRate # score
    res[runIndex,2] <- sample(c(0,1),1,prob=labSampleWeights) # label

    runIndex<-runIndex+1
  }
  res
} 

# min-max-normalization of x (fpr): [0,6] -> [0,1]
transformX <- function(x){
  (x-0)/(6-0) * (1-0)+0
}

# inverse min-max-normalization of x (fpr): [0,1] -> [0,6]
invTransformX <- function(invx){
  (invx-0)/(1-0) *(6-0) + 0
}

#  min-max-normalization of y (tpr): [0.5,logistic(6,k)] -> [0,1]
transformY <- function(y,k){
 (y-0.5)/(logistic(6,k)-0.5)*(1-0)+0
}

# logistic function
logistic <- function(x,k){
  1/(1+exp(-k*x))
}

# integral of logistic function
intLogistic <- function(x,k){
  1/k*log(1+exp(k*x))
}

# derivative of logistic function
derivLogistic <- function(x,k){
  numerator <- k*exp(-k*x)
  denominator <- (1+exp(-k*x))^2
  numerator/denominator
}

# roc-function, mapping fpr to tpr
roc <- function(x,k){
  transformY(logistic(invTransformX(x),k),k)
}

# derivative of the roc-function
derivRoc <- function(x,k){
    scalFactor <- 6 / (logistic(6,k)-0.5)
    derivLogistic(invTransformX(x),k) * scalFactor
}

# calculate the AUC for a given k 
calculateAUC <- function(k){
  ((intLogistic(6,k)-intLogistic(0,k))-(0.5*6))/((logistic(6,k)-0.5)*6)
}

# calculate k for a given auc
calculateK <- function(auc){
  f <- function(k){
      return(calculateAUC(k)-auc)
  }  
  if(f(0.0001) > 0){
     return(0.0001)
  }else{  
    return(uniroot(f,c(0.0001,100))$root)
  }
}

# Example
require(ROCR)

x <- seq(0,1,by=0.01)
k <- calculateK(0.6)
plot(x,roc(x,k),type="l",xlab="fpr",ylab="tpr",main=paste("ROC-Curve for AUC=",0.6," <=> k=",k))

dat <- binaryModelScores(1000,basePrecision=0.1,auc=0.6)

pred <- prediction(dat$score,as.factor(dat$label))
performance(pred,measure="auc")@y.values[[1]]
perf <- performance(pred, measure = "tpr", x.measure = "fpr") 
plot(perf,main="approximated ROC-Curve (random generated scores)")

