- Are MLEs obtained from fitting $P(x|x > a)$ to the truncated tend to be the same as MLEs obtained from fitting $P(x)$ to the non-truncated data?
This is not the case
We have
$$P(X|X > a,\theta) = \frac{P(X,\theta)}{P(X>a,\theta)}$$
and maximizing ${P(X,\theta)}/{P(X>a,\theta)}$ is different from maximizing ${P(X,\theta)}$.
Example 1
E.g. consider a truncated exponential distribution with rate parameter $\theta$ and truncating at $x_0$ (keeping only the values between $0$ and $x_0$). Then, the MLE is dependent on the point of truncating
$$\hat{\theta}_{MLE} = \begin{cases}
0 & \quad \text{for $\bar{x} \geq \frac{x_0}{2}$}\\
\frac{1}{\bar{x}} + \frac{ W \left(-x_0/\bar{x} e^{-x_0/\bar{x}}\right)}{x_0} & \quad \text{for $\bar{x} < \frac{x_0}{2}$}
\end{cases}$$
(If we allow the rate to be negative, ie an exponential distribution that is mirrored, then you get additional solutions for $\bar{x}>\frac{x_0}{2}$)
See for instance Deemer Jr, Walter L., and David F. Votaw Jr. "Estimation of parameters of truncated or censored exponential distributions." The Annals of Mathematical Statistics 26.3 (1955): 498-504. Related question: Maximum likelihood estimators for a truncated distribution
Also interesting: in an example below we compute a maximum entropy distribution, and in that example the relationship between the parameter and the distribution mean, has the same relationship as the relationship between parameter and the sample mean for the maximum likelihood estimator.
Example 2
An even simpler situation is truncating the left side of the exponential distribution, then the maximum likelihood estimate is
$$\hat{\theta}_{MLE} = \frac{1}{\bar{x}-x_0}$$
You can argue about this because the exponential distribution has the memoryless property. The distribution of $X-x_0$ conditional on $X>x_0$ is again exponential distributed with the same rate parameter. So the truncated/conditional distribution of $X|X>x_0$ is equivalent to a shifted exponential distribution. The shift relates to $x_0$ and will turn up in the expression for the maximum likelihood estimate as a shift of the data (that is: because of the shift one uses $\bar{x} - x_0$ instead of $\bar{x}$).
- Now, assuming that the training dataset is finite and the support of $x$ is not, there will be plenty of possible $x$ values that we don't see in a training data. Shouldn't we consider those unseen points to be "truncated" as well for the training part/parameter estimation?
There are two processes that reduce in some way the observations
Sampling process: when you randomly sample a population, then you won't get to see all the potential values that are in the entire population.
E.g. if you roll a six sided dice a few times, then you may observe only a subselection of all the possible numbers. (but it is not like the distribution of potential samples did not include those numbers)
Truncating: when you regard a population as following a certain mathematical function that is truncated.
This truncating can be some explicit physical truncation process. E.g. the distribution of people heights entering the rollercoaster in a theme park, while there os a height limit. Then it is like taking the distribution of all the people in the park and taking only a part of that distribution. The considered population is literally a subset of another population, where the condition for the subset is some boundary value. The truncating is very explicit here.
The truncating can also be more implicit. E.g. a maximum entropy distribution with some boundary limits.
For example regard the following process, take many values $x_i = 0.3$ and randomly have pairs interact by exchanging a random bit of their values $\pm d$ (the one plus the other minus such that the mean value remains 0.3), but only if the values do not end up below 0 or above 1 (such that the domain is constrained), then this will approach maximum entropy distribution and follow a truncated exponential distribution. See the image below with example computation at the end of this post.
It is not like we started with an exponential distribution that got truncated by some process. It is just that we can mathematically describe the distribution with an exponential function between values 0 and 1.
Truncating relates to the population distribution, or sample distribution, not to a realized sample itself. It is the distribution itself that is truncated and makes 'missing values'.
If you randomly sample a population then some numbers may not be observed, and you also have 'missing values', but that is not because the population distribution was truncated, you could have potentially observed other values.
Code to get implicitly a truncated exponential distribution
n = 1000
m = 4*10^6
set.seed(1)
### we start with 1000 values all the same value 0.3
mean = 0.3
x = rep(mean,n)
### the relating exponential term parameter
lambda = pracma::lambertWp(-1*exp(-1/mean)/mean)+1/mean
### randomly add and subtract some value
### from two random values
### repeat this many times untill you
### reach approximately a maximum entropy
for (i in 1:m) {
k = sample(1:n,2)
d = 0.01
new1 = x[k[1]]+d
new2 = x[k[2]]-d
if ((abs(new1-0.5) <= 0.5) & (abs(new2-0.5) <= 0.5)) {
x[k[1]] = new1
x[k[2]] = new2
}
}
### plot histogram
hist(x, breaks = seq(0,1,0.05), freq = 0, xlim = c(0,1))
### compare with exponential distribution
### truncated at 1
xs = seq(0,1,0.01)
lines(xs,dexp(xs,lambda)/pexp(1,lambda))
