The second set is generated using an algorithm, and I am currently
trying to find the best set of parameters, so the ratings are similar
to my first set.
This sounds like parameter estimation rather than hypothesis testing, it is not neccesary to formulate null/alternative hypotheses. I would do something like the below (Approximate Bayesian Computation; ABC) as mentioned in the comments.
Say the data consists of 100 samples of count/score data of either 0, 1, 2, 3, 4, 5
. Then we can generate data like this from a binomial distribution with known probability of success p. Here p=0.5
is used. This is slightly different than what you describe but makes the example easier because the binomial distribution will have only one free parameter.
We can then summarize those 100 samples by taking the mean. This mean should reflect the underlying parameter p.
In reality our model (algorithm) of the data generating process is a guess. But in this case say that it is correct. We hypothesize that the data are samples from a binomial distribution, but we do not know what parameter p. Then we can find the "best set" of parameters by taking samples from binomial distributions with different p, calculating the mean of these simulated results, and comparing those to that observed for the data.
Then we reject simulations where abs(mean(simulated)-mean(real))
is beyond some tolerance level. Other distance measures can be used, but here the absolute difference should work. The smaller the tolerance (it can even be zero) the better your final estimate of the best parameters but the more simulations you will have to run.
We can also simulate M results using the same set of parameter values (here, just p) and see how many of these are less than the tolerance. We can require that e.g., 50% of results less than the tolerance, in the code below a parameter is accepted if any of the M=10 replications are below the tolerance.
R code:
RealData<-rbinom(100,5,.5) #generate "real" data
RealMean<-mean(RealData) #Calculate Summary Statistic
out=NULL #initialize output
tol=.01 #set the tolerance, simulations w/ summary stats closer than this are accepted
M=10 #Number of Replications with each parameter set
Nsim=10000 #Number of different parameter sets to test
for(i in 1:Nsim){
p=runif(1,0,1) #Sample parameter "p"
SimData<-replicate(M,rbinom(100,5,p)) #Simulate Data
Dev<-apply(SimData, 2, function(x) abs(RealMean-mean(x))) #Compute Distances
NumMatches<-length(which(Dev<=tol)) #Count the number of simulations closer than the tolerance
#If any simulations were closer, save the parameter values, etc. This "tolerance" can be adjusted as well
if(NumMatches>0){
out<-rbind(out,cbind(p,NumMatches))
}
}
hist(out[,1], xlab="p", main="")
From the histogram, we see that our estimate is near the parameter that generated the data *p=0.5*
. The estimate is not perfect because the mean of the "real data" differed somewhat from the expected value mean=2.5
, this could be improved by increasing sample size. Our estimate could also be made more precise by running more simulations and decreasing the tolerance.
If you really want to just compare means, only the following two lines need to be replaced:
p=runif(1,0,1) #Sample parameter "p"
SimData<-replicate(M,rbinom(100,5,p)) #Simulate Data
If you algorithm is more complex, just sample parameter 1, 2, 3, etc from some reasonable prior distributions. Then have your algorithm generate data using those parameters. Replace the function rbinom
with your algorithm. It may not be the fastest implementation but it will work. There are various schemes out there to gradually adjust tolerance levels down to some minimum, etc.