Regression with summary statistics

The following setting should already be familiar:

Let $$X$$ be some space, $$\mathbb R^d$$ for simplicity, and let $$Y\subseteq \mathbb R$$. An unknown distribution $$\mu$$ is defined over $$X\times Y$$ and some loss function $$\mathcal l: Y\times Y\to\mathbb R$$ are also given.

We want to find a function $$f$$ from a family of function $$\mathfrak F$$ which minimizes the generalized error on $$\mu$$:

$$f^*=argmin_{f\in\mathfrak F} \mathbb E_{(x,y)\sim \mu}\mathcal l(f(x),y)$$

Should we have a sample $$S=\{(x_i,y_i ):1\le i \le n )\}$$, one approach could be to choose the one that minimizes the empirical error, i.e.:

$$\hat f=argmin_{f\in\mathfrak F} \frac{1}{n} \sum_{i=1}^n \mathcal l(f(x_i),y_i)$$

My question is as follows: Should we choose this approach, how one can learn $$f$$ if the sample is given in the form of summary statistics:

$$S=\{(Z_i,m_i,\bar y_i):1\le i \le n\}$$, where $$Z_i$$ denotes some region in $$X$$, $$m_i$$ denotes the # of points sampled in that region & $$\bar y_i$$ denote the average of these points (not of the entire region). Think of it as putting a grid on $$X$$, where each d-dimensional square is some $$Z_i$$.

Is there anything in the literature discussing this kind of setting?

Thanks.

First of all, packing $$d$$-dimensional data into bins is a complicated problem by itself, because of the course of dimensionality (see figure below, for visual explanation). The higher the number of dimensions $$d$$, the lower would be the individual $$m_i$$ counts per bin, and the harder would be to find the reasonable, non-empty bins to pack the data. Moreover, the way you choose the bins to pack and aggregate the data may influence your results.

(image source: Must-Know: What is the curse of dimensionality? by Prasad Pore, although I don't know if this is an author of the original image)

As about your general question, let me define new notation, where $$\bar y_i$$ is the mean value of the target variable $$y$$, $$\bar X_i$$ is the vector of means of the features $$X$$, and $$m_i \ge 0$$ are the counts for the $$i$$-th bin. In such case, you could just use weighted linear regression model

$$\bar y_i = \bar X_i \beta + \varepsilon_i$$

where you would be minimizing the weighted square loss

$$\hat\beta = \operatorname{arg\,min}_\beta \;\sum_i m_i\,( \bar y_i - \bar X_i \beta )^2$$

so the more observed samples $$m_i$$, the higher is their impact on the overall loss. This can be generalized to any other kind of regression model and any other kind of loss function, where the only change in the model would be replacing usual loss, with the weighted equivalent.

Almost (?) always it would be the better idea to work with raw, rather then with aggregated data, but I guess that you are asking the what if kind of question. With aggregated data you are loosing information, because it is both less precise, and you loose the multivariate relations between the individual samples. At worst, it could even lead to misleading results (e.g. Simpson's paradox). Of course, in some cases you simply don't have any other data then the aggregated one and need to deal with what you have, but by any means, this is not a preferable scenario. For example, if your problem is that your dataset is too big and you need to make it smaller for computational reasons, then in many cases it would be much wiser to randomly downsample it.

• Hi Tim, thanks for the answer. You guessed correctly, I'm interested in the scenario where I can only receive aggregated data, and what would be the best approach to deal with it. – aavital1 Sep 5 at 10:08