This problem is from Elements of Statistical Learning, exercise 2.6
The problem states:
Consider a regression problem with inputs $x_i$ and outputs $y_i$ and a parameterized model $f_\theta(x)$ to be fit by least squares. Show that if there are observations with tied or identical values of $x$, then the fit can be obtained from a reduced weighted least squares problem.
I'm having some trouble getting this one off the ground. The book actually doesn't say much about reduced weighted least squares before encountering this problem, so I'm not entirely clear on how to proceed.
Very roughly, my initial thought was to do something like the following: For each tied $k-$tuple of input values, $(x_{i,1},... x_{i,k})$, take the average of their corresponding output values and call the new output $y_i\prime$. Then perform least squares using the repeated $x$ value once and using the corresponding $y_i\prime$ as the output value, and weight each of the modified points by the number of repeated data points.
I have a few concerns with this approach. First, I'm not even sure if it's correct. But if it is correct, I'm not sure how to proceed with such few restrictions on $f_\theta(x)$.
Another concern I have. In class my professor gave us a hint on how to solve this one, but her statement of the problem seemed to assume we were trying to deal with heteroscedacity, which doesn't seem to come up in this problem at all.
She started out by saying we have the model $Y_i = X_i\beta+\epsilon_i$, where $\epsilon_i \sim N(0, \sigma_i)$. She said we then set $Y_i^* = Y_i/\sigma_i$ to obtain a corresponding $\epsilon_i^* \sim N(0,1)$. We then are supposed to minimize $\sum(Y_i^*-X_i^*\beta^*)^2 =\sum\frac{1}{\sigma_i^2}(Y_i-X_i\beta)^2$ to obtain our result.
I'm a little unsure about this approach because it assumes heteroscedacity as well as linearity in our $\beta$; neither of which is mentioned in the problem. Moreover, it doesn't seem to make any use of the fact that we have tied $x$ values. I'm wondering if she may have been thinking about a different problem when she went over this in class, but I also don't know if her solution will get me what I need if I just go further with it, or if it's connected with my initial thought on how to solve this.
Any thoughts or suggestions would be greatly appreciated.
Thanks in advance.
P.S. I'm fairly new to stack exchange, so I apologize if I've made any formatting errors or anything of the sort.
