0
$\begingroup$

I have data with features that looks like thisenter image description here

I am using ML regression in Sklearn to predict a final cost (in a separate df). Before fitting a linear regression I went to test the assumptions of linear regression and have a problem with autocorrelation. Running an autocorrelation test gave me this enter image description here

As you can see the Durbin Watson is too low. Someone suggested to me since some of my variables may be affected by time, to use ARIMA models to fix it. So to test if any of my variables were non-stationary I used an ADFULLER test and got these results.enter image description here

The P-value of 0 for all of the variables indicates that the variables are already stationary meaning time shouldn't be the issue. Does anyone have a suggestion to get rid of this autocorrelation in my data?

Improve this question
$\endgroup$
4
  • $\begingroup$ Quite confusing what you mean by autocorrelation. The autocorrelation is the concept in the time series. $\endgroup$
    – hbadger19042
    Jul 7, 2020 at 12:54
  • $\begingroup$ The Durbin Watson statistic is a test for autocorrelation, mine shows positive, my data isn't completely a time series, even so, autocorrelation also is a concept in non- time-series datasets I just don't know how to fix it. $\endgroup$
    – Joseph Hodson
    Jul 7, 2020 at 12:57
  • $\begingroup$ If it's not a time series, I don't know what's the meaning of order in your observations. $\endgroup$
    – hbadger19042
    Jul 7, 2020 at 12:59
  • $\begingroup$ So to clarify, the % complete takes ~10 rows to get to 100% then a new 'job' begins. I filtered out a column that is JOB ID. so those first 5 rows you see are for one job, then it will move on to another, and another, for 8000 rows making a whole bunch of mini time series. Not one giant one. $\endgroup$
    – Joseph Hodson
    Jul 7, 2020 at 13:02

1 Answer 1

Reset to default
1
$\begingroup$

The problem with traditional autocorrelation corrections (and the tests for autocorrelation you used) is that they assume equal distance between the observations for each job. Since this assumption is not met, the autocorrelation tests are wrong. Also autocorrelation tests on less than 30 observations have shaky validity. The bigger question is whether the project cost estimates are correlated with percent of project complete. If so, then figure out how to incorporate that into your models as predictors. If you have 8000 projects and don't want build separate models, maybe you can estimate normal under/overprediction of eventual project costs as a function of the percent stage of the project and the current estimate at that stage of the project.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks John! I was quite surprised checking my inbox and seeing this blast from the past haha. But even now, your answer makes sense and I appreciate it! $\endgroup$
    – Joseph Hodson
    Dec 17, 2022 at 20:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.