2
$\begingroup$

I have two sets of ratings between 1 to 5.

The second set is generated using an algorithm, and I am currently trying to find the best set of parameters, so the ratings are similar to my first set.

I have decided to use the Wilcoxon Signed Rank test for this, but am unsure on how to formulate the null and the alternative.

From my understanding, the null is assumed to be true which would make me believe it should be formulated as:

H0 = There is a difference between the ratings from set1 and set2.

The alternative would then become: Ha = There is no difference between the ratings from set1 and set3.

By setting alpha = 0.05, I would be able to rule H0 out if I get a p-value > alpha?

This is all new to me, so If someone could tell me if I doing something wrong, I would appreciate it.

$\endgroup$
5
  • $\begingroup$ You have reciprocated the null and the alternative hypotheses. Generally, the null hypothesis is a hypothesis of 'no difference/ no association/no effect' while the alternative hypothesis claims the opposite (there is difference/association/effect). $\endgroup$
    – Ayalew A.
    Jan 15, 2015 at 13:18
  • $\begingroup$ Yes, exactly. This feels wrong. But I don't know how to formulate what I want to show via null and the alternative. It seems, with the alternative you typically want to show a difference. In my case, I already have a difference and would like to show similarity with some statistical significance. $\endgroup$
    – Vger
    Jan 15, 2015 at 13:20
  • $\begingroup$ Some argue that null hypothesis is supposed to be predicted by the theory and using the opposite has been a major mistake.This paper is a good place to start: "Theory-Testing in Psychology and Physics: A Methodological Paradox". $\endgroup$
    – Livid
    Jan 15, 2015 at 13:52
  • $\begingroup$ You could look into equivalence tests. But a confidence interval for the pseudomedian might be enough for what you want. $\endgroup$ Jan 15, 2015 at 13:55
  • $\begingroup$ Also, it may be better to compare the similarity of the real data to those resulting from different sets of parameters to find the best one. For each parameter set, simulate e.g. 10k results and see how common results exactly like yours (or very similar) occurs.This is called ABC $\endgroup$
    – Livid
    Jan 15, 2015 at 13:57

1 Answer 1

1
$\begingroup$

The second set is generated using an algorithm, and I am currently trying to find the best set of parameters, so the ratings are similar to my first set.

This sounds like parameter estimation rather than hypothesis testing, it is not neccesary to formulate null/alternative hypotheses. I would do something like the below (Approximate Bayesian Computation; ABC) as mentioned in the comments.

Say the data consists of 100 samples of count/score data of either 0, 1, 2, 3, 4, 5. Then we can generate data like this from a binomial distribution with known probability of success p. Here p=0.5 is used. This is slightly different than what you describe but makes the example easier because the binomial distribution will have only one free parameter.

We can then summarize those 100 samples by taking the mean. This mean should reflect the underlying parameter p.

In reality our model (algorithm) of the data generating process is a guess. But in this case say that it is correct. We hypothesize that the data are samples from a binomial distribution, but we do not know what parameter p. Then we can find the "best set" of parameters by taking samples from binomial distributions with different p, calculating the mean of these simulated results, and comparing those to that observed for the data.

Then we reject simulations where abs(mean(simulated)-mean(real)) is beyond some tolerance level. Other distance measures can be used, but here the absolute difference should work. The smaller the tolerance (it can even be zero) the better your final estimate of the best parameters but the more simulations you will have to run.

We can also simulate M results using the same set of parameter values (here, just p) and see how many of these are less than the tolerance. We can require that e.g., 50% of results less than the tolerance, in the code below a parameter is accepted if any of the M=10 replications are below the tolerance.

R code:

RealData<-rbinom(100,5,.5) #generate "real" data
RealMean<-mean(RealData) #Calculate Summary Statistic

out=NULL #initialize output
tol=.01 #set the tolerance, simulations w/ summary stats closer than this are accepted
M=10 #Number of Replications with each parameter set
Nsim=10000 #Number of different parameter sets to test
for(i in 1:Nsim){
  p=runif(1,0,1) #Sample parameter "p"
  SimData<-replicate(M,rbinom(100,5,p))  #Simulate Data
  Dev<-apply(SimData, 2, function(x) abs(RealMean-mean(x))) #Compute Distances
  NumMatches<-length(which(Dev<=tol)) #Count the number of simulations closer than the tolerance

  #If any simulations were closer, save the parameter values, etc. This "tolerance" can be adjusted as well
  if(NumMatches>0){
    out<-rbind(out,cbind(p,NumMatches))
  }

}

hist(out[,1], xlab="p", main="")

enter image description here

From the histogram, we see that our estimate is near the parameter that generated the data *p=0.5*. The estimate is not perfect because the mean of the "real data" differed somewhat from the expected value mean=2.5, this could be improved by increasing sample size. Our estimate could also be made more precise by running more simulations and decreasing the tolerance.

If you really want to just compare means, only the following two lines need to be replaced:

  p=runif(1,0,1) #Sample parameter "p"
  SimData<-replicate(M,rbinom(100,5,p))  #Simulate Data

If you algorithm is more complex, just sample parameter 1, 2, 3, etc from some reasonable prior distributions. Then have your algorithm generate data using those parameters. Replace the function rbinom with your algorithm. It may not be the fastest implementation but it will work. There are various schemes out there to gradually adjust tolerance levels down to some minimum, etc.

$\endgroup$
1
  • $\begingroup$ Thanks Livid! The described approach is also the one I came to realize and thanks for the thorough description. $\endgroup$
    – Vger
    Jan 16, 2015 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.