I have a huge data set that looks roughly like this:

x = [x1, x2, x3, ..., x800]
x is how much percent the real project cost is different from the planned project cost

y1 = [y1.1, y1.2, y1.3, ..., y1.800]
y1 is the estimated project cost in dollar (or another currency)

y2 = [y2.1, y2.2, y2.3, ..., y2.800]
y2 is the real projet duration

y3 = [y3.1, y3.2, y3.3, ..., y3.800]
y3 is the project manager (in coded form like "mk" and "op")


y50 = [y50.1, y50.2, y50.3, ..., y50.800]
y50 is the estimated project duration

Now the aim is to predict the x (cost difference in percent) in dependency of other y variables. It is a priori not clear which variables influence it. And some of the y are not interdependent like for example the difference in percent x, the real costs and the estimated costs since x is calculated from the real and estimated costs.

I am now wondering how is the best practice way to analyse the data or how to build a model (linear or maybe logistic). The binary output for the logistic model could be difference bigger than 10 %.

Since I have a lot of data sets (over 800) almost everything could get significant on a e. g. 5 % level. And It's not easy to handle/control the power if you do not have a simple t-test.

What would you recommend? Is there a good book for "normal people" about that kind of statistics?

  • 1
    $\begingroup$ Unless you're looking for a run-of-the-mill Statistics textbook, it will depend a lot on the software that you plan on using. You might really like this article--it's geared towards a general audience (though still substantial). In addition, you can practice with the code/data the author provided on his website (and described throughout). $\endgroup$
    – Steve S
    Aug 6 '14 at 12:35
  • 1
    $\begingroup$ Also, you probably don't want to use the logistic regression here since the restriction that results be between zero and 1 is most likely an unreasonable assumption on your part. As an aside, your data will be more readable if you follow the convention of using x to signify your predictors and y to be the response. $\endgroup$
    – Steve S
    Aug 6 '14 at 12:39
  • 1
    $\begingroup$ But again, that article will cover some of the big issues that you'll have to consider (such as incorporating cross-validation into your analysis as well as regularization), giving you a better idea of where to go from there. $\endgroup$
    – Steve S
    Aug 6 '14 at 12:41
  • $\begingroup$ Thanks for the comments. And yes - I should switch x and y ;) $\endgroup$ Aug 6 '14 at 16:00

I highly recommend these two books:

  • Linear Models with R, by Julian J. Faraway
  • Extending the Linear Model with R, by Julian J. Faraway

Both of these books discuss how to do applied statistics in R (a free statistical software that can be downloaded). Linear Models with R discusses modelling continuous variables where Extending the Linear Models with R will discuss modeling other types of variables (count, binary).

  • $\begingroup$ I will have a look at this books. I also know R by the way. Thanks! $\endgroup$ Aug 6 '14 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.