I want to find the model that best fits my observed data. There are 29 observed data points that are normally distributed. My proposed model has 2 categorical variables each with four possible values, total of 16 permutations. I propose using ANOVA and Scheffe contrasts with Dunn/Sidák or Bonferroni correction factors as the follow up test identify the best predictive model. What are the best statistical tests for this comparison?
1 Answer
$\begingroup$
$\endgroup$
5
Your best bet, if this study is meant for a peer review process, is to use a model that will preserve your limited degrees of freedom. There is a good chance that your model will suffer from sparsity or be unable to converge. On another note, I would also be extremely skeptical of an author telling me that their 29 data points were normally distributed.
I would recommend removing one of your variables from the model. That will at least allow you to estimate an effect on ~7 units per group and would address the ability of your model to converge. And you have to ask yourself, anyway. Do you want to make an claim of inference on a model that might have only 1 unit that is being used to make that inference?
If you wish to preserve both variables, I recommend condensing them into binary variables.
BTW, why would you be using a bonferroni? You are only testing a single model which is "Dependent variable = A + categorical variable1 + categorical variable2 + error". There is no need to adjust p values as you are not resampling from the same dataset.
Your goal, with 29 data points, should be to employ a model that can make inference with a limited number of data points. I could even see using two t tests on either independent variable and then using a correction in that regard. That might be the more sensible approach rather than having both effects in a single model and trying to estimate them at the same time.
To be honest, this is an instance where you should be letting your literature guide you as well on which statistical approach makes the most sense in the context of the corpus.
-
$\begingroup$ Thank you for your insights and I agree 29 data points isn't much to work with. I'm trying to empirically evaluate a relatively simple theory to estimate commercial aircraft leases. One of the difficulties of empirically testing lease pricing is the absence of data. Lessors consider the contracts proprietary and confidential. However, a few of the major lessors are publicly traded and financials are available for their aircraft portfolio. The actual formula is $\endgroup$– STSamJun 19, 2018 at 13:48
-
$\begingroup$ The formula is l = (r + v + d) * a. The 29 observations are values for l, d, and a, not enough for regression. Assume that I can reliably estimate r, I want to test various estimates of v for best fit. Lessors may use other values for d in their contracts, therefore, I also want to test alternate values for d. I'm not a stats guy and any suggestions on the best approach are welcome. $\endgroup$– STSamJun 19, 2018 at 14:32
-
$\begingroup$ One question I have, do you have multiple years of data? Or does your dataset include all lessors? $\endgroup$– JWH2006Jun 19, 2018 at 15:09
-
$\begingroup$ Good question. The data set covers 2013 - 2017. There are 6 companies in each year except for 2014, where I exclude one company because it quadrupled in size and the data looks like an outlier. From 2008 - 2012 there are another 22 observations with at least 4 companies reporting each year. The plan is to back test the model, if possible, against these prior years. $\endgroup$– STSamJun 19, 2018 at 20:51
-
$\begingroup$ I don't believe you have enough companies to verify a model. You just do not have the power to perform that analysis. If you are concerned about your approach, you can boot strap, though this is likely to be unhelpful. I cannot stress this enough that you should acknowledge and accept the limitations of your sample size and how that influences the extent of your inference. $\endgroup$– JWH2006Jun 21, 2018 at 15:32