0
$\begingroup$

Do Probability Distributions Inherently "Capture More Information" Compared to Hyperplanes?

A regression model (whether linear or polynomial) is said to be represented as hyperplane through the data space:

enter image description here

On the other hand, a probability distribution model fits an entire distribution through the data space:

enter image description here

My Question: Compared to the hyperplanes generated by regression models, do probability distributions inherently have the ability to capture more information and make more informative inferences about the same data? Of course, this would depend on how well both types of model fit the same data - but in general, can such a conclusion be made comparing the expressiveness of both models? Or is this simply not true?

Thanks!

$\endgroup$
5
  • 1
    $\begingroup$ "Hyperplanes" and "distributions" are not comparable. However, most statistical regression procedures estimate the full distribution of the conditional responses. $\endgroup$
    – whuber
    Nov 24, 2021 at 14:54
  • $\begingroup$ @ Whuber: thank you for your reply! I had a question about your comment "statistical regression procedures estimate the full distribution of the conditional responses" : in this context, since a regression model is technically a conditional distribution P(y|x1, x2, .. xn) : should a regression model be considered as a probability distribution or a hyperplane? In what context do we interpret a statistical model as a hyperplane? Thank you! $\endgroup$
    – stats_noob
    Nov 24, 2021 at 15:14
  • 1
    $\begingroup$ When the model supposes some predefined function of the location parameter of the conditional distribution is a linear combination of $x_1, \ldots, x_n,$ it both determines a hyperplane (the locus of possible locations) and a joint distribution of the responses. This is the standard setting of ordinary least squares regression (OLS; the predefined function is the identity) and of the generalized linear model (GLM; the predefined function is called the "link function"). $\endgroup$
    – whuber
    Nov 24, 2021 at 15:25
  • $\begingroup$ @ Whuber: thank you for your reply! just to clarify - when creating a regression model y = b_o + b_1*x1 + b_2 *x_2 : in the frequentist setting, there is no multivariate probability distribution associated with this model? $\endgroup$
    – stats_noob
    Nov 24, 2021 at 22:11
  • 1
    $\begingroup$ There certainly is! The model, fully written out, is $$y_i = \beta_0 + \beta_1x_{1i} + \beta_2 x_{2i} + \varepsilon_i$$ and the standard minimal assumptions are that the vector $E=(\varepsilon_1, \ldots, \varepsilon_n)$ has a multivariate distribution with zero mean and covariance matrix $\sigma^2\mathbb{I}_n$ ($\sigma$ is a "nuisance parameter" in the model). Note that these are assumptions about the distribution of $Y=(y_1,\ldots, y_n)$ conditional on the design matrix (of values of the explanatory variables). $\endgroup$
    – whuber
    Nov 24, 2021 at 22:20

1 Answer 1

1
$\begingroup$

linear or any n-dimensional hyperplane cuts the data so it minimizes the MSE (or any other metric used to measure distance). However if you have a probability distribution (including multi-dimensional distributions), it provides the location of the cluster and the likelihood of finding a sample in a specific region.

If the data is clustered, then a distribution model will provide more information, while a regression model will just cut the plane and won't provide you with the distribution.

I believe the expressiveness depends on the situation. If you have the time, resources, and enough data points, you can run both models and sample from the model and use statistical tools to see which one better matches the data (as a way to measure the model's generalization of the data)

$\endgroup$
1
  • $\begingroup$ In general, probability distributions are more useful because they recognize there are close calls and that arbitrary classification is misleading. $\endgroup$ Nov 24, 2021 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.