1
$\begingroup$

Say I have a data set that I am trying to perform a linear least squares regression on. Suppose that the end goal is to predict y from x. The training data set I am working with has the form

y: (0.500,0.500,0.500,.300,.300,.300,.100,.100,.100)
x: (15.6, 15.2, 15.9, 11.2, 10.9, 11.0, 5.6, 5.3, 6.0)

The important thing to note here is that what I am trying to predict (y) has multiple, different (x) values in the training data set. Suppose that the data comes from an experiment. Say (y) is some density and (x) is some reading on a machine. Say the data comes from measuring some standard known densities, collecting a bunch of readings, and the end goal is to fit a model so that when unknown density is read, and the machine gives a value (x), we can use the model to predict the actual density.

How should regression be done here? My understanding is that there are dependencies in the data here, so a standard regression model to predict y given x cannot simply be fit and applied. Do I need to invert the model? Take averages first? What is the correct thing to do? I've been told to invert the model, but I am not sure I understand what the justification is. Assume that the intended solution is a linear regression model.

Improve this question
$\endgroup$
3
  • $\begingroup$ Maybe you could explain your example a bit more clear. I think you can just use regression. With enough training data the effect of the "density" should be taken care of. Unless you have weird distributions. Am I missing something? $\endgroup$
    – user209249
    Commented May 24, 2018 at 22:13
  • $\begingroup$ We need to calibrate a machine that will read an unknown density and give a reading. A model is going to be fit so that we can interpet the machine output and predict the actual density. To do this, standardized densities are read multiple times and all of the machine outputs are recorded. So for one particular density (y), we have several readings. We are assuming that the same machine is used for each reading, so the variance in machine outputs is from the error if the machine $\endgroup$
    – Marcel
    Commented May 24, 2018 at 22:23
  • $\begingroup$ Why don't you make a normalized histogram and calculate the standard error for each bin? Wouldn't that be more straightforward? $\endgroup$
    – user209249
    Commented May 24, 2018 at 22:26

1 Answer 1

Reset to default
0
$\begingroup$

You are likely looking for the vectorized version which is called multiple (linear) regression:

$y_i=\beta_0+\beta_1 x_{1,i}+\beta_2 x_{2,i}+\dots+\beta_nx_{n,i}+\varepsilon_i=\beta_0+\vec{x}_i~\vec{\beta}+\varepsilon_i$. Where the $\varepsilon_i$-s are assumed iid and $\varepsilon_i\sim N(0,\sigma^2)$

It's pretty much the same as univariate linear regression, except you're fitting a plane and not a line.

The $\vec\beta$ is estimated using the augmented notation: $\vec{b}=\hat{\vec\beta}=(X^{T}X)^{-1}X^{T}Y$ where $X$ is a matrix with all your $\vec x_i$-values, and augmented with 1s. Read more here under "Least squares estimations"

Improve this answer
$\endgroup$
5
  • $\begingroup$ I'm not sure I understand this: multiple regression (I think) typically assumes that each predictor variable is independent from the others. This is not so in my case, as each sequence of measurements for a given density comes from the same machine and same standardized density. They are necessarily dependent. Why is this method justified? $\endgroup$
    – Marcel
    Commented May 24, 2018 at 21:52
  • $\begingroup$ @Marcel In this model, the predictors are not stochastic, only the $\varepsilon$s are random. You are not making assumptions on the predictor variables, only the response via the errors. That is $Y_i=a+bx_i+\varepsilon_i$. $x_i$ is not random! $\endgroup$
    – Andreas Storvik Strauman
    Commented May 24, 2018 at 23:02
  • $\begingroup$ Ah, I see. This should work then, assuming the model ends up fitting well. Thanks. I'm going to leave the question up since I want some more opinions, but I'll mark your answer if no one bites. $\endgroup$
    – Marcel
    Commented May 24, 2018 at 23:58
  • $\begingroup$ @Marcel sure, that’s only fair! Anything you need me to clear up? You could do all kinds of regression if the model doesn’t fit well; polynomial, logistic etc. Maybe you’d even want to try some machine learning algorithms. $\endgroup$
    – Andreas Storvik Strauman
    Commented May 25, 2018 at 0:04
  • $\begingroup$ You indeed can treat this as a garden-variety multiple regression situation. There's no need to assume that all predictors are independent. And since you seek prediction rather than explanation -- or identification of stronger or weaker relationships -- it matters little how collinear the predictors are. Each x variable will supply a little more useful info to help with predictive accuracy. $\endgroup$
    – rolando2
    Commented May 25, 2018 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.