Provided that censoring isn't informative, there are data beyond year $X$ for $X$-year survival, and there are enough events, what you describe isn't necessarily a problem. Here's how to gauge how much of a problem it could be.
The Kaplan-Meier estimator and the Greenwood formula for its variance are shown on this page. I've copied the variance formula here:
$$\widehat{Var}\{\hat{S}(t)\} = \hat{S^2}(t) \sum_{j|t_j \le t} \dfrac{d_j}{n_j (n_j - d_j)}$$
where $\hat S(t)$ is the estimated survival through time $t$ and $j$ is the index of event times, with $n_j$ and $d_j$ the number of individuals at risk and having events at time $t_j$, respectively. Use this variance (or its square root, the standard error) as an estimate of the reliability of the Kaplan-Meier estimator to see the effects of reducing the numbers of individuals at risk and having events at late times.
Although the summation in the formula necessarily increases with time and as the number of deaths comes closer to the number at risk, the estimated survival also decreases with time. So you need to consider the specific situation to see how the variance changes at late times. A Kaplan--Meier curve should include confidence intervals to illustrate that directly.
Estimates at late times within the range of the data can be pretty good even with small number of cases. For example, Klein and Moeschberger's Kaplan-Meier example (Table 4.1) is based on only 21 individuals and 9 events, with median follow-up about 16 weeks and only 6 still at risk at 23 weeks. Nevertheless, the estimate of 23-week survival, 0.448, has a standard error of only 0.135. That's less than twice the standard error of 0.076 for the earliest event time in that example, when there were 3 events out of 21 at risk.