I have $(x,\ y)$ pairs with a strongly suspected linear correlation. So I want to fit the "best" linear function in order to make predictions for unknown $x$'s. These pairs don't represent a function, i.e. there are many different values of $y$ for some $x$. I want to know if there is an issue with that considering this as a machine learning problem. Should I use some unique "representative" (average, maximum, minimum, most frequent, etc.) and convert the input data in a function or it is OK to work with the data as it is?


If you perform linear regression on your raw data using Ordinary Least Squares to estimate the model parameters you will get a model that estimates the conditional mean of $Y$ given the observed value of $X$. I.e. $\hat{f}(x)=E[Y|X=x].$ Since your model outputs the conditional expectation of the response it wouldn't be necessary to convert your $Y|X=x$ into a descriptive statistic prior to your analysis.

  • $\begingroup$ OK. I see that. But now suppose that I have to evaluate how good are the estimated parameters of my linear model with, let's say, MAE or MSE. I want to do that with cross-validation. So I take my evaluation set and do all the predictions with my model and I want to compare it with the "real" one. Which value should I take as the "real" one given that the input data is not a function? Thanks for your answer! $\endgroup$ – Martin Chaia Nov 8 '13 at 13:30
  • $\begingroup$ You're right that your validation method wouldn't make much sense. Since you said, "there are many different values of y for some x," you know that there is no "real" value of $Y$ at a given level of $X$, i.e. the relationship isn't deterministic. Using linear regression with OLS estimators you can predict $E[Y|X=x]$, so you could compare these predictions to the average Y value at a given level of X to determine how accurate your model is. The regression output should provide you with diagnostics to make checks similar to this. $\endgroup$ – tjnel Nov 8 '13 at 21:42
  • $\begingroup$ You may also find prediction intervals useful to determine the range within which the model assumes most observations would fall at a given level of X. $\endgroup$ – tjnel Nov 9 '13 at 0:49
  • $\begingroup$ Thanks tjnel! I really appreciate your answer. Can you please suggest me a technique to "find prediction intervals"? $\endgroup$ – Martin Chaia Nov 11 '13 at 19:26
  • $\begingroup$ @MartinChaia if you are using statistical computing software to perform your analysis you can check if it supports prediction intervals for regression. $\endgroup$ – tjnel Nov 11 '13 at 23:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.