2
$\begingroup$

How could we know if the coefficients and R-Squared in the following table are reliable and are correctly computed or not? Is there any way to validate the correctness of the results, considering that we do not have access to the data set?

enter image description here

The descriptive analysis of the original variables is as follows, but in regression they transformed variables!

enter image description here

$\endgroup$
  • 1
    $\begingroup$ This has a flavour of: I get 42 as the result of a calculation with data I won't show you. Did I do it right? Better answers: Your tag indicates use of SPSS. Does SPSS reproduce regression results from elsewhere reliably with known data? You can carry out your own checks. Have you searched for literature on bugs, limitations or reliability of SPSS? As of 2014, it's a fair guess that regression routines in all major statistical software have been banged on many, many times, but that's no absolute guarantee of correctness. $\endgroup$ – Nick Cox Jan 15 '14 at 15:13
  • $\begingroup$ Thanks a lot for response; I have descriptive analysis of variables and I edited my question. I do not have access to data set; I found these results in a paper and I want to be sure of the validity of results. $\endgroup$ – user31315 Jan 15 '14 at 16:12
  • 3
    $\begingroup$ There are several things fishy about this summary: [Last] and [Score] range from $0$ through $100$, so how did they take the logarithms of the zeros? Moreover, because it is clear that [Last] has some negative skewness (notice how close its mean is to its maximum value), using its logarithm would usually not improve anything--it would only give the extreme low values more leverage. Furthermore, it is strange that "Model 4" would include interactions of [Win R] and [Win F] with other variables without actually including [Win R] and [Win F] themselves. $\endgroup$ – whuber Jan 15 '14 at 20:55
  • $\begingroup$ There's no way to be certain of the results without the data (and even then, validity would be only for the data you were given, whose provenance may well be doubtful). You can however, often do some reasonableness checks. (You can rely on SPSS for basic regression calculations, since it's easy to check it works on your own data.) $\endgroup$ – Glen_b Jan 15 '14 at 23:18
10
$\begingroup$

In this specific case, with the extra details about the data, some questions may still be raised about the model.

  • The response variable, Score, is cited as varying from 0 to 100, but was analysed using ln(Score). But the logarithm of 0 is indeterminate, so precisely what did the authors do with value(s) equal to 0?

  • Similarly, what was done about the zero(s) in Last when using ln(Last)?

  • Further, it is a fair guess that the range 0 to 100 for Score is a limitation of principle, but the model ignores that. That may or may not be a problem: a particularly strong clue is that the mean and median are both above 90, so values in the 90s may be common, and in any case values at the bounds evidently occur. This boundedness is likely to imply some nonlinearity and it is not obvious that logarithms are the way to cope with that.

  • The use of natural logarithmic transforms probably reflects the idea that a multiplicative model makes more sense than an additive one; it is unlikely to be driven by the skewness of the response, which from the results here is likely to be negative hence not positive. Putting these two together, it is possible that the ln transformation induces moderate outliers!

Although without the data you cannot check the calculations, I'd suggest that the quality of the analysis might fairly be guessed by the extent to which the authors discuss

  1. The characteristics of the data. I would particularly expect to see a frank discussion of what might be awkward or problematic for any analysis and how it was handled.

  2. Theoretical explanations and/or what kinds of models make scientific or substantive sense. For many projects, this should come first, naturally.

  3. Their particular choices in analysis. Data very rarely can or should be modelled in just one way, so why particular models were chosen for report is vital.

More personally, I would hope to see, as diagnostic of careful work, graphs as well as tables to give a clear picture of data and/or results. Habits and traditions vary, and in some fields graphs are avoided on various grounds, say supposed lack of space, lack of an obvious graph to show off a complicated model, academic snobbery, a principle that graphs should not repeat tables, or prejudice that graphical inspection is a form of data snooping. But it is rare that authors don't have scope for, minimally, alluding to graphical exploration of their data. The lack of graphs or of evidence of graphical work often hints at superficial, naive or mechanistic work.

In short, be particularly sceptical if models just are presented without a background story or critical discussion.

A very personal, and very minute, detail is that authors who allow "Ln" to appear in print rather than "ln" lack mathematical taste, even when it starts a sentence, but it is possible that this should be blamed on reviewers, editors, or publishers' staff.

$\endgroup$
  • $\begingroup$ Thanks a lot for your insightful views and good analysis. $\endgroup$ – user31315 Jan 15 '14 at 17:36
  • $\begingroup$ +1 Good advice. The more I see stuff like these summaries when reviewing papers, the more it becomes apparent that deep scepticism tends to be the right attitude to take: it is frequently rewarded by discovering basic mistakes in the analysis. :-). $\endgroup$ – whuber Jan 15 '14 at 20:59
  • 2
    $\begingroup$ @whuber Thanks. Pertinent quotations from Breiman, Leo. 1973. Statistics: With a view towards applications. Boston: Houghton Mifflin (seemingly a little known text, but pleasingly incisive and individual): "The main thing to learn about statistics is what is sensible and honest and possible." (p.1) "Doubt and suspicion, as well as technical knowledge, are indispensable tools in statistics." (p.18) $\endgroup$ – Nick Cox Jan 15 '14 at 21:35
5
$\begingroup$

You cannot see that from the table or the descriptives.

You can look at the paper and see what they did (or at least what they tell you they did), and see if that makes sense.

The ideal way is to replicate the study, but you would need access to the data. This will depend on the willingness of the original authors to share their data with you. There are good reasons for not being able to do that, e.g. privacy laws may forbid it in some countries. There are some less good reasons but still understandable for not sharing, e.g. the data is such a state that it needs some work to make it useful to other persons, and the original authors just don't have time right now. There are some bad reasons for not sharing the data, e.g. the analysis is so sloppy (or worse) it won't stand a chance of being reproduced.

$\endgroup$
3
$\begingroup$

In addition to the other issues, this table is NOT the way the output would appear in SPSS Statistics, so you have no way of knowing whether there are typographical errors without the original output, so if your life depends on it, you need to be cautious.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.