This is an interesting question. Surprisingly, it is possible to do something under certain assumptions, but there is a potential loss of information about the residual variance. It depends upon $X$ how much is lost.
Let's consider the following singular value decomposition $\newcommand{\t}{^\mathrm{t}}X = UDV\t$ of $X$ with $U$ an $n \times p$ matrix with orthonormal columns, $D$ a diagonal matrix with positive singular values $d_1 \geq d_2 \geq ... \geq d_p > 0$ in the diagonal and $V$ a $p \times p$ orthogonal matrix. Then the columns of $U$ form an orthonormal basis for the column space of $X$ and
$$Z = U\t Y = D^{-1} V\t V D U\t Y = D^{-1} V\t X\t Y$$
is the vector of coefficients for the projection of $Y$ onto this column space when expanded in the $U$-column basis. From the formula we see that $Z$ is computable from knowledge of $X$ and $X\t Y$ only.
Since the ridge regression predictor for a given $\lambda$ can be computed as
$$\hat{Y} = X(X\t X + \lambda I)^{-1} X\t Y = U D(D^2 + \lambda I)^{-1} D U\t Y = U D(D^2 + \lambda I)^{-1} D Z$$
we see that the coefficients for the ridge regression predictor in the $U$-column basis are
$$\hat{Z} = D (D^2 + \lambda I)^{-1} D Z.$$
Now we make the distributional assumption that $Y$ has $n$-dimensional mean $\xi$ and covariance matrix $\sigma^2 I_n$. Then $Z$ has $p$-dimensional mean $U\t \xi$ and covariance matrix $\sigma^2 I_p$. If we imagine an independent $Y^{\text{New}}$ with the same distribution as $Y$ (everything conditionally on $X$ from hereon) the corresponding $Z^{\text{New}} = U\t Y^{\text{New}}$ has the same distribution as $Z$ and is independent and
\begin{eqnarray*}
E ||Y^{\text{New}} - \hat{Y}||^2 &= & E || Y^{\text{New}} - U Z^{\text{New}} + U Z^{\text{New}} - U \hat{Z} ||^2 \\
& = & E || Y^{\text{New}} - U Z^{\text{New}}||^2 + E||U Z^{\text{New}} - U \hat{Z} ||^2 \\
& = & \text{Err}_0 + E||Z^{\text{New}} - \hat{Z} ||^2.
\end{eqnarray*}
Here the third equality follows by orthogonality of $Y^{\text{New}} - U Z^{\text{New}}$ and $U Z^{\text{New}} - U \hat{Z}$ and the fourth by the fact that $U$ has orthonormal columns.
The quantity $\text{Err}_0$ is an error that we cannot get any information about, but it does not depend upon $\lambda$ either. To minimize the prediction error on the left hand side we have to minimize the second term on the right hand side.
By a standard computation
\begin{eqnarray*}
E||Z^{\text{New}} - \hat{Z} ||^2 &= & E||Z - \hat{Z}||^2 + 2 \sum_{i=1}^p \text{cov}(Z_i, \hat{Z}_i) \\ & = &
E||Z - \hat{Z}||^2 + 2 \sigma^2 \underbrace{\sum_{i=1}^p \frac{d_i^2}{d_i^2 + \lambda}}_{\text{df}(\lambda)}.
\end{eqnarray*}
Here $\text{df}(\lambda)$ is known as the effective degrees of freedom for ridge regression with parameter $\lambda$. An unbiased estimator of $E||Z - \hat{Z}||^2$ is
$$\text{err}(\lambda) = ||Z - \hat{Z}||^2 = \sum_{i=1}^p \left(1 - \frac{d_i^2}{d_i^2 + \lambda}\right)^2 Z_i^2.$$
We combine this with the (unbiased) estimator
$$\text{err}(\lambda) + 2 \sigma^2 \text{df}(\lambda)$$
of $E||Z^{\text{New}} - \hat{Z} ||^2$ given that we know $\sigma^2$, which we then need to minimize. Obviously, this can only be done if we know $\sigma^2$ or have a reasonable guess at or estimator of $\sigma^2$.
Estimating $\sigma^2$ can be more problematic. It is possible to show that
$$E||Z - \hat{Z}||^2 = \sigma^2\left(p - \underbrace{\sum_{i=1}^p \frac{d_i^2}{d_i^2 + \lambda}\left(2 - \frac{d_i^2}{d_i^2 + \lambda}\right)}_{\text{d}(\lambda)}\right) + \text{bias}(\lambda)^2.$$
Thus if it is possible to choose $\lambda$ so small that the squared bias can be ignored we can try to estimate $\sigma^2$ as
$$\hat{\sigma}^2 = \frac{1}{p-\text{d}(\lambda)} ||Z - \hat{Z}||^2.$$
If this will work depends a lot on $X$.
For some details see Section 3.4.1 and Chapter 7 in ESL or perhaps even better Chapter 2 in GAM.