1
$\begingroup$

I have a dataset that has a number of instances that look like the following:

id        x                y
a         2015-05-04       1
          18:32:00
a         2015-05-04       1
          18:32:00
a         2015-05-04       1.5
          18:32:00
a         2015-05-05       4
          16:30:00
a         2015-05-08       5
          10:25:00
a         2015-05-10       6
          00:32:00
a         2015-05-11       7
          6:45:00

"id" is an identification variable, of which there are thousands in the data set, and I'm running various forms of linear regressions to predict the dependent variable ("y") over time ("x") within each identification value. Note that for the actual regression, "x" is converted to an integer corresponding to the number of days since the beginning of the data under that identification value.

The issue I'm having relates to the uneven nature of the time element of the regression (the "x"). In this example data set, each observation in an OLS regression would be given equal weight, regardless of when the observation occurred in time. However, I feel that this would cause issues with the predictive power of my model if it gives equal weight to 3 observations that happened at the same time, and then as a result less weight to other observations in time (as more recent observations should have more weight in the regression). This is especially relevant because its possible that the first 3 observations were given that time because of an error in uploading data to our database, and hence were given the same time as soon as they could be uploaded to the database (however, there's no way for me to check whether this actually occurred).

Is there a way to assign, within a regression, each observation within a day a weight of 1/count(observations in that day)? So that if one day has 3 observations, each observation within that day is given 1/3 weight in the regression, and one day corresponds to a weight of 1/count(unique days in data set)?

Or if this isn't the best way of dealing with this issue, what would you suggest I look into to help with this problem to smooth out the time element of the data?

Or, even from a theoretical perspective, is this a problem?

I would appreciate any thoughts and feedback and am happy to answer any questions you may have.

$\endgroup$
1
$\begingroup$

Yes, you can easily weight observations. If you are using R, in function lm you have an argument weights to that effect (alternatively, argument wt of function lsfit).

There are alternative, more elaborate ways of dealing with this issue. You might want to use an state-space model to implement a varying parameters regresion: this would yield estimations more precise where you have a higher density of observations in time.

$\endgroup$
  • $\begingroup$ Thank you for the response. I believe weighting would be the best for this, and I know it's possible to weight regressions, but my question would be how would you weight the regression on this specific issue? I primarily use Python, and WLS gives you the option to weight based on some value, and I'm wondering if I should create a column that corresponds to the number of observations that have occurred in that day (so, for example, creating a column "z" that for the first 3 observations above would have a value of 3, and all others a value of 1). Then weight on that. Does that make sense? $\endgroup$ – CSlater Jun 20 at 17:45
  • $\begingroup$ What you say in your comment is opposite to what you proposed in your question (where 3 observations in a day would be downweighted so as to have the same influence as a single one). Both things might make sense in different settings, you just have to decide which is right for your problem. $\endgroup$ – F. Tusell Jun 21 at 8:22
  • $\begingroup$ You're right, I wasn't clear in that answer. I meant that the weight would be the inverse of that column, so that if "z" has a value of 3, the weight would be 1/3. And if "z" has a value of 1, it would just be 1/1, or 1. So then "z" values of 1 would be given more weight than "z" values of 3. If that doesn't make sense, I would appreciate feedback as I'm always interested in learning new things in stats! $\endgroup$ – CSlater Jun 21 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.