3
$\begingroup$

Imagine you have two time-series of variables, x and y, which both are proxies for the same measure, growth per minutes, for a plant. Further assume that y is the better measure for growth, but it is usually only available since the year 2012. So before 2012, variable x has to be used. There are exceptions (plants), in which y can be used earlier.

Assume we analyze about 100 different plants (lots of data). The issue is that a common measure for growth should be used in subsequent analyses (e.g. regressions), but the original scale of the variables (before calculation of growth) is different. For example, x has a mean per minute of $8\times10^{(-6)}$, whereas y has a mean of $1\times10^{(-8)}$ only.

Assume there is standard deviation of roughly 1.2 for x and 1.7 for y.

My questions is how I can use both of these variables in the same analysis, accounting for the "jump" in scale in 2012. Imagine I want to do a simple regression of e.g. growth on amount of sunlight and water per minute. Or more simply, I would like to state a mean for the whole time-series of growth, not just separately based on two different measures, if possible.

I know of standardization, normalization, etc., but I am unsure if these techniques are the right ones to use for my case. The variables x and y are generally very similar to each other (correlation of about 80%), just the scale varies.

Does anyone have experience with similar issues? Any hints are helpful.

$\endgroup$
  • $\begingroup$ What's the problem with just normalizing so that they're both bounded from [0, 1]? It just seems like the easy solution. Maybe you do an instrumental variable regression where you use one measure as an IV for the other? $\endgroup$ – Parseltongue Oct 26 '18 at 1:35
  • $\begingroup$ @Parseltongue well that is an option, but if you normalize the resulting variables do not have the same mean/sd and you are losing the original scale completely. In my case, I feel like the scale of y would be preferred over the scale of x. But if that's the common approach, I could try it. $\endgroup$ – Talik3233 Oct 26 '18 at 2:32
  • $\begingroup$ then why not standardize so both scales have mean 0 and sd 1? I'd wait for an expert to answer, but that's what I'd do. $\endgroup$ – Parseltongue Oct 26 '18 at 2:57
  • $\begingroup$ @Parseltongue indeed that would be an option, but the disadvantage is that you will have negative values (as negative deviations from the mean divided by sd) as well, but there is actually no negative growth, which hinders the interpretation of the variable at first glance. But from a technical perspective that would be definitely possible. I am waiting for more replies, I think this case can often occur in many contexts. $\endgroup$ – Talik3233 Oct 26 '18 at 5:58
  • $\begingroup$ It can help to understand the regression itself a a part of the modeling process, rather than the entirety of the modeling process. In this view, you can A) prepare the data for regression, which might include scaling or taking logs, followed by B) regression, and then C) de-transform for final analysis of the modeling results, such as RMSE or R-squared calculations. $\endgroup$ – James Phillips Oct 26 '18 at 11:09
1
$\begingroup$

Since growth is your outcome, then you have two outcomes at some time points, but only one in others. According to you, one is of higher quality than the other. If you have a quantitative way of communicating model quality, then you could apply those as case weights. It does not seem like you do so here's what I recommend.

For cases where you have both measurements, transform those cases into two cases. Each case should retain a case ID so you know where the data come from. Also, have an identifier for each type of outcome.

Now you have data with dependence between rows. So you can take a mixed effects regression approach, clustering the data by original case. For your model, create two dummies, one for each original outcome. You can begin by entering both dummies into the model (remove the standard intercept) from your software so you can use both dummies. Also, depending on your software, you can enter both dummies as random slopes, also calculating a correlation between them. It is probably simpler to enter a single random intercept, and permit heteroskedastic variances by each type of outcome. The second approach is possible using the nlme package in R.

Finally, you have two options for entering your remaining predictors into the model. If you believe each predictor influences each outcome uniquely, you can enter them as interactions-only with the two dummies for each type of outcome. Otherwise, enter them into the model as standard predictors.

I have an example of data stacking and predictor entry here - MLE with unbalanced system of regressions. In that example, the corresponding nlme model would be:

library(nlme)
# baseline model
# outcome1 and outcome2 are outcome type identifiers
# Random intercept by plant, permit heteroskedasticity by outcome type
# Obtain average value for each outcome type using dummies
lme(
  y ~ 0 + outcome1 + outcome2,
  random = ~ 1 | plant_id,
  dat, weights = varIdent(form = ~ 1 | outcome1))
# Standard model
lme(
  y ~ 0 + outcome1 + outcome2 + sunlight + water_per_min,
  random = ~ 1 | plant_id,
  dat, weights = varIdent(form = ~ 1 | outcome1))

The more flexible specification where we permit each predictor to have a different coefficient depending on the outcome type is below. But since both outcomes are the same, there is probably no need to do this unless the measures really are different.

lme(
  y ~ 0 + outcome1 + outcome2 + outcome1:sunlight + outcome1:water_per_min +
    outcome2:sunlight + outcome2:water_per_min,
  random = ~ 1 | plant_id,
  dat, weights = varIdent(form = ~ 1 | outcome_type))

I think this is one approach. Obviously, it ignores the differences in quality between the different outcome types. But it allows you to use both outcome types in a single model.

You can also use the different correlation structures your package (nlme in my case) offers to account for the time-series nature of the data.

I would also run a quality measure only model, hoping that the results do not differ much.


If the growth from earlier periods affects future growth, then you need to account for growth using some special correlation structure (corClasses in nlme). Otherwise, if you just want to model growth simply overall, time simply enter the model as a predictor. You can specify it as linear, quadratic, exponential, ... Given discussion in comments, a model that would average over both outcome types would then be:

model.disc <- lme(
  y ~ 1 + f(time), random = ~ 1 | plant_id,
  dat, weights = varIdent(form = ~ 1 | outcome1))

where f() is some function on time, power, log, spline, ... The fitted values (fitted(model.disc)) from such a model would return the predicted outcome at each time point for each plant averaging over the two measures.

$\endgroup$
  • $\begingroup$ @Heteriskedastic Jim thanks, that is definitely one nice way to perform the regression, that is why I gave you an upvote. What I am missing a bit is the the second part of my question, where I asked about the simple mean over the whole time-series. Also it would be interesting to plot growth over the whole time-series. So the question to me is if the two variables must be treated totally separately, or if there is a way to combine them, at least for the purpose of illustration and to give an overall statement. $\endgroup$ – Talik3233 Oct 29 '18 at 19:13
  • $\begingroup$ @Talik3233 I can modify my answer. But how do you hope to represent time in the model? $\endgroup$ – Heteroskedastic Jim Oct 30 '18 at 15:50
  • $\begingroup$ Ok sure. Time is just equal to minutes (indexed by t). We measure growth from t-1 to t. At a sidenote, we have growth almost every minute, but sometimes obs can be missing due to no measurement being taken (affects rather earlier periods) $\endgroup$ – Talik3233 Oct 31 '18 at 5:20
  • $\begingroup$ @Talik3233 I modified my response. $\endgroup$ – Heteroskedastic Jim Nov 1 '18 at 17:18
  • $\begingroup$ thanks, I need to think more about this and will reply to you soon $\endgroup$ – Talik3233 Nov 5 '18 at 6:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.