0
$\begingroup$

Here are some data: (R language)

set.seed(1234)
dat <- data.frame( ins=sample(c(1,2,3,4,5,6,7,8), 100, replace=T), 
                  outs=sample(c(1,2,3,4,5,6,7,8), 100, replace=T) )
df  <- ifelse(with(dat, ins-outs)>2, 3,
              ifelse(with(dat, ins-outs)< -2, -3,
                     with(dat, ins-outs)))

The ins column is the levels of in-sample data (model development sample) and outs is the levels from the model (fitted).

I would like to test for under/over estimation. To do so I would like to only consider up to -/+3 notch difference.

# calculate the frequency table
tab   <- data.frame(t(table(df)))[c(1,2,3,5,6,7), c(2,3)]; 
tab$p <- rep(0, nrow(tab))
tab$p[1:3] <- tab$Freq[1:3]/sum(tab$Freq[1:3])
tab$p[4:6] <- tab$Freq[4:6]/sum(tab$Freq[4:6])

# expected frequency 
tab$exp_freq <- rev(c(floor(tab$p[1:3]*sum(rev(tab$Freq)[1:3])),
                      floor(tab$p[4:6]*sum(rev(tab$Freq)[4:6])) ))

\begin{align} &H_{0}: p_{-3}=u_{-3}, p_{-2}=u_{-2}, p_{-1}=u_{-1} \\[5pt] &H_{1}: \sum_{i=1}^{3} u_{i} - \sum_{i=1}^{3} p_{i} > 0 \end{align}

where $p_{i}$ represents left side and $u_{j}$ right side.

This is the chi-squared test from R:

with(tab, chisq.test(Freq, p=exp_freq, rescale.p=TRUE))

Well, I'm indifferent, would this be correct approach given setup hypothesis?

# insert equal p 
tab$eq_p <- 1/nrow(tab)

with(tab, chisq.test(Freq, p=eq_p))
Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ Aside from your R code it would help if you provide background on your question. Why do you think a chi square test is appropriate? What did you calculate by hand that differs from the result in R? You need more clarity or the question may be closed. $\endgroup$
    – Michael R. Chernick
    Commented Dec 25, 2016 at 15:44
  • $\begingroup$ @MichaelChernick, thank you for looking into this. I ammended the question slightly, but overall I feel the question is pretty straithforward, with data and calculation disclosed. $\endgroup$
    – Maximilian
    Commented Dec 25, 2016 at 15:56
  • $\begingroup$ I now completely redone the question, I believe this time is clear. Thanks. $\endgroup$
    – Maximilian
    Commented Dec 25, 2016 at 18:22
  • $\begingroup$ "Checking the calculation" is typically going to be off topic here. (Moreover, R isn't going to make a simple arithmetic error, so it isn't clear what the point is.) Can you say more about what is motivating this question? Are you wondering if / how to use a chi-squared test to determine if a model is overfitting or underfitting? $\endgroup$
    – gung - Reinstate Monica
    Commented Dec 25, 2016 at 18:26
  • $\begingroup$ Yes, I'm indeed wondering if / how to use a chi-squared test to determine if a model is overfitting or underfitting. $\endgroup$
    – Maximilian
    Commented Dec 25, 2016 at 18:31

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.