Questions tagged [projection]

For on-topic questions involving the mathematical concept projection, a linear transformation $P$ such that $P=P^2$. Please include also a more statistical methods tag. For purely mathematical questions about projections it is better to ask on math SE https://math.stackexchange.com/

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

Cross-covariance in context of Andrews plot

As shown in this Cross-Validated post Close curves on an Andrews plot I don't understand how, in the accepted answer, the cross-covariance can be defined as, $$\int_{-\pi}^{\pi}f_xf_ydt$$ Considering ...
user avatar
1 vote
0 answers
15 views

Range of error in linear regression [closed]

In linear regression whose matrix form is $$Y = Xb + E$$ where $X=[X_1,X_2]$. If the error of using only $X_1$ (i.e $ Y = X_1b+E$ ) is $x_1$, and the error of using $X_2$ (i.e $ Y=X_2b+E$) is $e_2$, ...
user avatar
  • 111
0 votes
0 answers
21 views

What is the relationship between the conditional expected value, OLS and projection in linear regression?

I am learning about the linear regression. I was taught it from two perspectives. The first one was about an equation connecting the conditional expected value and the predictor. I saw the nice graphs ...
user avatar
  • 1
0 votes
0 answers
22 views

What function yields a hat matrix with the smallest Frobenius norm to a given hat matrix?

Phrasing Attempt 1 If I have one function $f_1: \mathcal{X} \rightarrow \mathbb{R}^{D_1}$ that yields a particular hat matrix $P_1 \in \mathbb{R}^{N \times N}$, how do I find the function $f_2: \...
user avatar
0 votes
0 answers
19 views

What is the orthogonal projection with expectation?

I am reading an advanced econometrics textbook. When it talks about least squares, it says that the orthogonal projection of A onto Z is $P_Z(A)=Z^\prime E[ZZ^\prime]^{-1}E[ZA_k]$ and when A is a ...
user avatar
0 votes
0 answers
7 views

Why the inverse for unknown coefficients vector? [duplicate]

From my understanding, this formula is used for least-squares when we're interested in minimizing the distance between a point and some space we are projecting on. Somebody can correct me if this is ...
user avatar
  • 1
0 votes
0 answers
62 views

Why Expectation for estimates of projection in least squares

I understand that in the least square setting, the projection of $\mathbf{y}$ onto the column space of $\mathbf{X}$ is the vector $\mathbf{Xb}$, where the vector $$\mathbf{b=(X'X)^{-1}Xy}$$ Thus the ...
user avatar
0 votes
0 answers
31 views

Projection operator: why squared norm of the sum of them is equal (or smaller) than the sum of the squared norms?

I am working through the proof of Lemma 2 in this paper (page 25, need it for my own research) and I am stuck at the very first step. Here, I will formulate a bit simplified version of this step. ...
user avatar
  • 703
0 votes
0 answers
33 views

Minimization of Midpoint Kullback-Leibler Divergence

Consider the following problems: $$ \arg\min \limits_{\gamma \in \mathcal{\Gamma}} \; \mbox{KL}\Big(\gamma \; \Big\| \;\frac{\gamma + \nu}{2} \Big) ,$$ $$ \arg\min_{\gamma \in \mathcal{\Gamma}} \; \...
user avatar
3 votes
1 answer
48 views

Prediction error for ARMA process

Let $X(t)= \phi X_{t-12}+Z_t+\theta Z_{t-1}$ where $Z_t\sim WN(0,1)$. I need to find prediction error for projecting $X_t$ onto $H_{t-3}(X)$ (Hilbert space). So, I know that $X_t \perp P_{H_{t-3}}X_t$ ...
user avatar
  • 249
1 vote
2 answers
152 views

How to show that smoothing spline fit preserves the local regression part of the fit

We need to show that a smoothing spline of $y_i$ to $x_i$ retains the local regression part of the fit. For linear regression, this problem seems trivial because it is relatively easy to move from $...
user avatar
  • 27
1 vote
0 answers
19 views

Finding Covariance from linear algebra projection chapter

I am trying to solid background of linear regression by using linear algebra. In linear algebra, there are some chapters that related to linear regression. (orthogonality, Projection) I learned some ...
user avatar
1 vote
0 answers
18 views

In PCA, why does maximising projections maximise variance?

My understanding could be wrong as the lecture series from my University isn't clear and there are no links to share. But having rewatched a few times, I think the point being made is: . We want the ...
user avatar
0 votes
1 answer
63 views

Linear Models Hat matrix

For OLS in matrix form, we are taught that Hat matrix is $X(X^TX)^-X^T$, and is idempotent etc, i.e. when it multiplies with itself, it will self cancel and thus lead back to the same Hat matrix. I ...
user avatar
4 votes
1 answer
320 views

In machine learning, especially deep learning, is it a principle to only do linear projections to a smaller dimension size?

Linear projection (or fully connected layer) is perhaps one of the most common operations in deep learning models. When doing linear projection, we can project a vector $x$ of dimension $n$ to a ...
user avatar
  • 153
1 vote
1 answer
211 views

Why are covariance matrices projected by both right and left multiply?

I've been doing a lot of Kalman filtering work recently. I've derived all the equations starting from a basic linear inverse problem, so strictly speaking I know where everything comes from. I also ...
user avatar
1 vote
1 answer
37 views

Can I use a single letter or symbol to represent "actual" in a table?

I have a chart that displays numbers for 2017, 2018, 2019 and 2020, but the 2020 numbers are based on projections - I want to display the "actual" value on the data label next to the ...
user avatar
  • 113
1 vote
0 answers
43 views

Finding a Projection Plane in Dimensionality Reduction (e.g., Multidimensional Scaling)

I have a set of data points in high-dimensional space that I wish to map onto a lower dimension (3D or 2D). Question : How do I obtain the Projection (Hyper)Plane (e.g., its normal vector or its set ...
user avatar
4 votes
1 answer
595 views

Intuition behind projection matrix

I'm new to machine learning and came across projection matrix . In a random thread it was interpreted as The matrix $X(X^\text{T} X)^{-1} X^\text{T}$ is a projection matrix, as it does precisely that:...
user avatar
2 votes
0 answers
395 views

How is conditional expectation related to projection?

I vaguely remember seeing somewhere that the conditional expectation $E(Y|X)$ can be interpreted as projection of random variable $Y$ onto random variable $X$. My question is: Is the aforementioned ...
user avatar
  • 1,682
2 votes
1 answer
101 views

Projection vs fixed effects

Suppose I have $n$ observation indexed by $i$ and that each observation is part of a group $g$. I want to compare two regressions. First regression: $$ Y_i=\beta X_i + \alpha F_{g(i)}+\varepsilon_i $$ ...
user avatar
1 vote
0 answers
58 views

Orthogonalization vs fixed effects in instrumental variable estimation

Suppose that I have a set of observations indexed by $i$. Each observation belongs to a group $g$. Let's define by $\hat{Z}_i=Z_i - E\left[Z_i\vert g(i)\right]$ is the residual after projecting $Z$ on ...
user avatar
2 votes
1 answer
391 views

Projecting a covariance matrix to a lower dimensional space

I have a point $\mathbf{x}$ in 3-dimensional space, which is measured with a degree of uncertainty. The point falls within a unit cube, and the uncertainty is assumed to follow a multivariate normal ...
user avatar
0 votes
1 answer
35 views

Project time series from previous time series examples and characteristics

Say I want to open a shop but first I want to project the likely sales in the first 5 years to see if it is a viable option. I have data pertaining to 100s of other start ups, including their success ...
user avatar
  • 133
1 vote
1 answer
206 views

Question about Regression error and the residual maker matrix

Starting with the 'residual maker' defined by M in: $e= y-\hat{y} = Y-X(X'X)^{-1}X'Y = [I-X(X'X)^{-1}X']Y =MY$ where e is the regression residual. one common equality i see relating the regression ...
user avatar
  • 385
3 votes
0 answers
67 views

Is $\hat \beta$ of general least squares an orthogonal projection?

$\hat \beta_{GLS} = (X'V^{-1}X)^{-1}X'V^{-1}Y \\ Y=X\hat \beta_{GLS} + \epsilon=X(X'V^{-1}X)^{-1}X'V^{-1}Y + \epsilon = \\X(X'V^{-1}X)^{-1}X'V^{-1}Y + (I-X(X'V^{-1}X)^{-1}X'V^{-1})Y$ It is clear that ...
user avatar
1 vote
0 answers
46 views

Intercept of linear projection

I'm reading a paper and wondering whether I get things right: Say I have a variable Y and two further variables $X_1$ and $X_2$. I want to determine the part of $Y$ which is not linearly explained by $...
user avatar
  • 11
2 votes
1 answer
798 views

Projecting new samples onto PCA space is failing

After performing PCA I would like to project any new samples to the principal component space (I would like to see how samples cluster together). I did the PCA analysis in R: ...
user avatar
1 vote
1 answer
315 views

Simple Linear Regression: Hat-Value $h_i$

I'm trying to finish proving that in simple-regression analysis, $h_i = \frac{1}{n} + \frac{(X_i - \bar{X})^2}{\sum_{j=1}^{n}(X_j - \bar{X})^2}$, where $h_i := h_{ii} = \sum_{j=1}^nh_{ij}^2$, the ...
user avatar
  • 157
2 votes
0 answers
51 views

Boundary point errors in PCA projection using sklearn

I am preparing a small example of a projection using python, numpy and sklearn to perform <...
user avatar
  • 433
1 vote
2 answers
213 views

Difficulty in understanding the proof of the Wold decomposition theorem

The proof of the Wold Decomposition [1] of $x_t$ involves the definition of the process $$w_t = x_t - P_{\mathcal{M}_{t-1}^x} x_t,$$ where $x_t$ is a stationary zero-mean process, $\mathcal{M}_n^x = ...
user avatar
  • 287
3 votes
0 answers
130 views

What is the geometric meaning of correlation matrix

I recently read this article explaining the geometric meaning of covariance matrix. http://www.visiondummy.com/2014/04/geometric-interpretation-covariance-matrix/ My question is : is there an ...
user avatar
  • 1,279
1 vote
1 answer
88 views

Best linear prediction as a projection in a Hilbert space $L^2$

Consider two random variables $Y$ and $X$. In the context of the best linear prediction, if we would like to predict $Y$ given $X$ known, we derive the solution solving the following minimize problem ...
user avatar
  • 877
1 vote
0 answers
116 views

The joint distribution of Y=AX and Z=BX given a projection matrix A and residual maker matrix B, and a random vector X with known pdf?

This question follows on from a previous question I asked which was answered. It turns out my question lacked some important details, which was revealed by the answer posted on that thread. This is ...
user avatar
  • 53
1 vote
0 answers
154 views

Least Squares Fit Covariance Matrix

How do you perform a least squares fit for $\Sigma$ in the equation $v = u^T\Sigma u$? The term $v$ is a vector of observed variances of a projected Gaussian distribution. The matrix $u$ is made of ...
user avatar
3 votes
1 answer
431 views

Higher moments of linear regression residuals?

I previously asked this on Math StackExchange, with no success, but this post will add to that with some simulations. Background In the following linear regression with i.i.d $\epsilon_i$ $(i = 1, \...
user avatar
  • 429
4 votes
1 answer
822 views

Expected value as an orthogonal projection

I'm reading a paper in which the expected value of a random variable, $\mathbb{E}[X]$, is characterized as an orthogonal projection. This is on page 10. I've seen the geometric interpretation of ...
user avatar
  • 327
0 votes
0 answers
36 views

Linear project when X includes a constant

In Hamilton's text I ran across the following statement: if $X$ includes a constant then the linear projection of $aY+b$ is $aP(Y\mid X)+b$ where $P(Y\mid X)$ is the linear projection of $y$ on $x$. I’...
user avatar
  • 243
0 votes
0 answers
199 views

Linear regression with feature representation confusion - relationship of design matrix column space to the feature space?

I am trying to visualise the geometry of linear regression with feature representation. I have a regression problem with $n$ data pairs $\mathcal{D}:=\{(\mathbf{x},y)_{i}\}_{i=1}^{n}$, independent ...
user avatar
  • 203
3 votes
0 answers
343 views

What's wrong with my solution to canonical correlation analysis (CCA) using the SVD? [duplicate]

I am working through the derivations for solving CCA in A Tutorial on Canonical Correlation Methods. Right now, I am trying to solve CCA using SVD (bottom of page 95:7). For completeness, I include ...
user avatar
  • 1,412
2 votes
1 answer
2k views

t-SNE, PCA and another technique - which one? [closed]

I am comparing dimension reduction techniques and I am utilizing them for data visualizations onto a plane – projections in 2D space. The input into a projection/dimension reduction techniques is a ...
user avatar
  • 21
5 votes
1 answer
524 views

Conditional expectation and variable decomposition

Suppose that $X$ and $Y$ have an uknown joint distribution $f_{XY}$. How can I formally demostrate that it always exists a unique decomposition of the form : $$ Y = E[Y|X] +\epsilon $$ without ...
user avatar
  • 407
0 votes
1 answer
19 views

Model Selection - 6 month forecast given the past 25 months

As the title states, the problem at hand is asking me to predict the next 6 months values when given the past 25 months. In my opinion, the training data will be quite thin so traditional time series ...
user avatar
  • 121
0 votes
0 answers
37 views

Can anyone help me with manipulating the projection matrices?

I am trying to prove the below equality to prove that the squared correlation of coefficient is equal to r-square(i.e. $r^2 = R^2$): $$Y'M_\iota \hat{Y} = \hat{Y}'M_\iota \hat{Y}$$ where $\hat{Y} = X\...
user avatar
0 votes
1 answer
1k views

Projection of a vector on unit sphere

I am currently reading the paper G.Salton et al, A Vector Space Model for Automatic Indexing, November 1975, Volume 18 It is about indexing documents. Representing them in vectors to find ...
user avatar
  • 229
0 votes
1 answer
190 views

Vertically translated depreciation curve: Update the exponential regression coefficient

I have an exponential regression equation that I use to predict the condition of roads. The equation can be found on page 53 of the original master's thesis: Development of a Flexible Framework for ...
user avatar
1 vote
1 answer
127 views

Road condition: Project future condition by appending part of a deterioration curve?

I inspect a road network's condition every three years: ...
user avatar
0 votes
1 answer
346 views

What is wrong with my computation of projections on the first principal component?

Following the links in parenthesis (link1, link2) I wrote a bit of code in MATLAB to simulate a PCA. After running the code and plotting the results I obtain this ...
user avatar
  • 101
2 votes
0 answers
346 views

Markov chain which is also a projection

Let $P$ be the transition matrix of a Markov chain, and assume that $P^2=P$. One immediate conclusion is that $P=P^\infty$. Furthermore, assume that there is a state $i$ such as each state $j$ (...
user avatar
  • 308
5 votes
1 answer
970 views

How to show that demeaning the data in design matrix does not change the hat matrix

When I have a design matrix $$X = \begin{bmatrix} 1 & x_{11} & \ldots & x_{1k}\\ 1 & x_{21} & \ldots & x_{2k}\\ \vdots & \vdots & \ddots & \vdots\\ 1 & ...
user avatar
  • 399