Questions tagged [truncation]
Truncation is a process that results in the omission of data that are beyond a threshold.
167 questions
0
votes
0
answers
37
views
Which log-likelihood is to be maximized for left-truncated count data?
What is to be done, if the count data is missing the counts on the zeros (i.e. left truncated data)? Say one wants to estimate a Poisson regression and the goal is to derive the log-likelihood to be ...
- 677
2
votes
1
answer
79
views
Rejection sampling to obtain a random sample from a truncated version of a multivariate probability density
Suppose I have a multivariate probability density $f(\mathbf{y}|\boldsymbol{\theta})$ with support $\mathbb{R}^d$ that is analytically tractable, and I know how to randomly sample from $f(\mathbf{y}|\...
- 2,073
1
vote
1
answer
109
views
Difference between truncated and unseen data
I have 2 related questions.
Assume that we want to build a model to study of some random discrete variable $x$ that follows some known distribution with PMF $P(x)$, yet with unknown parameters that we ...
- 57
4
votes
0
answers
92
views
Truncated response variable in boosted regression trees
I was thinking about the differences in approaches between parametric and non-parametric statistics in regression.
I am working with a non-negative integer response $N\in\mathbb{N}_{0}$.
Let's imagine ...
- 2,891
3
votes
1
answer
55
views
How do right censoring and right truncation interact?
I am working with data related to time of event for some physical product. the data is current status data, also sometimes called interval censored type II data, which mean i only have one inspection ...
- 167
1
vote
2
answers
226
views
Can I estimate the mean of a dataset if I have its standard deviation and a portion of the full data that is higher than some threshold?
I have a partial set of measurement data that is limited due to my tool's sensitivity. I know that the data is approximately normally distributed and I have a standard deviation from another data set ...
- 21
0
votes
1
answer
24
views
Left Truncated Data survival data
I am working with a data that is I believe left truncated. I am looking at how Veterans with MPNs have a different timing of arterial thrombosis (AT) compared with Civilians with MPNs.
In the data ...
- 25
0
votes
1
answer
159
views
On the difference of truncated Gaussian and a new definition
Given a r.v. $X \sim N(0, 1)$, what is the density of $Z = X I(\lvert x \rvert < \lambda)$. I am confused with the truncated Gaussian $Y = X$ if $\lvert X \rvert < \lambda$ otherwise $Y = 0$.
My ...
- 43
0
votes
1
answer
57
views
Estimating a censored distribution curve
I have a sample of only 142 numbers from a distribution of 3852 numbers ranging from 0 to 53, but it is censored below 35 (The values exist, but I don't have access.), so I have only the values in ...
3
votes
0
answers
129
views
Distribution closed under convolution and truncation followed by convolution
Let $D(\theta)$ denote an absolutely continuous distribution on $\mathbb{R}$. (The finite dimensional vector $\theta$ collects the parameters of the distribution.) Assume that the p.d.f. of $D(\theta)$...
- 525
1
vote
1
answer
559
views
Left truncation or left censoring?
I am conducting a study in which I analyse the time to opt out after the free trial is ending. I have individuals who sign up for a free membership-trial period. At some point the period is ending and ...
user359974
1
vote
0
answers
59
views
Likelihood contribution of censored data - why conditional?
Let $x_i$ be a sample of survival time from right-truncated data with right-truncation time $Y_R$. I would like to find the likelihood contribution from the sample.
I thought I have to find the ...
- 299
1
vote
1
answer
96
views
Parameter estimation on exponential distribution from a bounded subset of that distribution
I have a random variable that is exponentially distributed with some $\lambda$. I'm sampling observations from this variable, but I'm limited to observing only those that are less than some maximum ...
- 111
2
votes
0
answers
103
views
Distribution of a constrained Gaussian distribution in frequency domain
We know that a Gaussian Distribution is not limited and it spans from $-\infty$ to $+\infty$ . However, practically if we sample the Gaussian with a finite sampling frequency, the maximum frequency is ...
- 315
1
vote
0
answers
55
views
Is my data right censored or right truncated?
I have a dataset of the percent of pavement area that is cracked and time as the explanatory variable for a city's streets. Engineering principles and logic tell us that as that as time progresses, ...
0
votes
0
answers
18
views
How to model an uncertain (or erroneous) truncation effect on a random variable?
Let $ V_a $ be a random variable which is truncated at a value $v_c$. Therefore, the updated density function of truncated $V_a$ is given by,
$$ f(v_a| V_a\leq v_c) = \frac{g(v_a)}{F(v_c)}$$
where $g(...
- 173
2
votes
1
answer
180
views
Survival analysis when event cannot happen after timepoint
People are contacted for a survey for a fixed number of days. For simplicity, let's say this happens over 5 days. People can complete their survey on day 1, day 2, ..., up to the end of day 5. After ...
- 2,692
2
votes
1
answer
829
views
Prediction metrics for left-truncated and right-censored data
I am trying to assess (out-of-sample) predictive performance of survival analysis models with left-truncated and right-censored data. Assume the training and test data, respectively, consist of ...
- 61
2
votes
0
answers
413
views
Formula for standard deviation of a normal distribution from truncated data [closed]
I have some samples taken from a normal distribution of unknown $\mu$ and $\sigma$, and I know someone took away the top and bottom $p$ percent of the original samples ($p$ is known).
Is there a ...
1
vote
1
answer
2k
views
How to deal with left truncation in cox model in R
I have a longitudinal data with 8 follow-ups. My aim is to see the effect of a disease on ability decline at old age. In my study, I have sibling pairs, one with disease (1) and one without disease (0)...
- 139
0
votes
0
answers
41
views
Expected value of a random variable with truncation
Let $f:[0,\infty)\to \mathbb R_+$ denote the PDF of a random variable $X$ and $c>0$ a constant. I want to evaluate the following integral:
$$I(c)=\int_0^\infty{\min(x,c)f(x)dx}.$$
This can be ...
- 113
3
votes
1
answer
1k
views
Distribution of a sample of uniformly distributed points in the 2D
Let there be a rectangle in the plane and a set of points distributed in the rectangle by a uniform distribution. I select a random point on the top and right border and draw the red line. The blue ...
- 157
4
votes
2
answers
318
views
Simulation of a truncated normal distribution over two intervals
Given $X$ a random variable with a normal distribution, what is the best procedure to simulate $X|X\in[a;b]\cup[c;d]$, i.e. we want to simulate the truncated normal distribution only on the intervals $...
- 141
2
votes
1
answer
1k
views
Better understanding of censoring and truncation (in particular right truncation) in survival analysis
I am taking a course in survival analysis where we follow the book "Survival Analysis: Techniques for Censored and Truncated Data" by John P. Klein and Melvin L. Moeschberger.
Although I ...
- 147
2
votes
2
answers
167
views
Is it possible to calculate or find out what the original distribution was of a dataset? [duplicate]
Is it possible to calculate or find out what the original distribution was of a dataset?
For example: I have (part of) a dataset with 800 weights and I know that the original dataset contained 1000 ...
Ken
2
votes
1
answer
378
views
What is the correlation of a truncated bivariate Normal distribution?
$X$ and $Y$ are independent standard normal variables. If we generate $n$ samples $(x,y),$ what is the correlation for samples $(x, y)$ with $x+y\gt0$?
- 21
2
votes
1
answer
160
views
Is trial with accrual period left truncated?
I am reading Dirk Moore's Applied Survival Analysis 11.2.
"...However, patients actually enter over an accrual period of length a, and then, after accrual to the trial has ended, they are ...
- 1,465
1
vote
1
answer
2k
views
Censoring, truncated and missing data in Survival Analysis
Can someone please explain the difference between censored, truncated and missing data in survival analysis? Suppose I have following information.
...
- 145
1
vote
1
answer
35
views
Baseline adjustment when both pre- and post values are subject to left-censoring
Assume $X$ is a binary treatment variable, $Y$ is a continuous variable measured pre- and post-treatment $(Y_{pre}$, $Y_{post})$, and $Z$ represents the remaining covariates. Both $Y_{pre}$ and $Y_{...
- 11
4
votes
1
answer
3k
views
Dealing with left-, right- and interval-censoring and left- and right-truncation
I have a survival data set with left-, right- and interval-censoring and left- and right-truncation. Now I want to fit a Cox proportional hazard and an AFT model to these data. What is the best way to ...
- 87
1
vote
1
answer
117
views
How to use logistic regression with original values from a dependent variable that is a ratio?
I have a dependent variable that is a ratio, i.e. it takes the values between 0 and 1. Some 30% of values are 1s. The dependent variable measures the distribution of funds, i.e. it is calculated just ...
- 351
3
votes
1
answer
1k
views
Why is it problematic to use OLS for a bounded dependent variable (e.g. a ratio variable)?
I have a question that emerged from my previous post. If you look at my previous post, the dependent variable there is a ratio variable and thus is bounded, i.e. it can only take the values between 0 ...
- 351
6
votes
1
answer
274
views
Flawed multiple linear regression in academia? Heteroscedasticity's effect on p-value?
I believe I have found a paper in academia that has used a flawed multiple linear regression. I have downloaded the data set and replicated their regression results. I have done some diagnostics and ...
- 351
4
votes
0
answers
119
views
truncation of bivariate normal under quadratic condition
Consider a complex normal variable $Z \sim \mathcal{CN}(\mu,2\sigma^2)$ with real component $X \sim \mathcal{N}(\mu,\sigma^2)$ and imaginary component $Y \sim \mathcal{N}(0,\sigma^2)$. We can write ...
- 121
4
votes
0
answers
97
views
Bivariate normal and truncated expectation [duplicate]
What is the expectation
$$\mathbb E[X_1 \lvert X_1 > X_2]$$
assuming that
$$(X_1,X_2) \sim \mathcal MVN(0,\Sigma),$$
with $\mathcal{MVN}$ being the multivarite normal.
I would expect this to have ...
- 5,565
3
votes
1
answer
489
views
Central Limit Theorem for Truncated observations
Consider a random variable $X$ with values in $\left[0,\infty\right)$ such that $E\left[X\right]=\infty$. Given $M > 0$ I want to estimate the expected value of $X$ truncated at $M$. That is I want ...
- 563
0
votes
1
answer
166
views
Truncating variable in survival analysis
this is my first time developing a survival analysis model so bare with me if the nomenclature is not on point.
Basically, I'm running a Cox PH model for the length a contract is active, where 1 is ...
- 265
0
votes
1
answer
43
views
Partially left truncated data in bivariate time series, one series is missing data what to do?
I have two time series of futures Settlement Prices where I am to model the systematic volatility of the settlement prices for these two time series.
My plan is to:
Calculate the logged returns.
Fit ...
- 15
0
votes
0
answers
23
views
Truncation based on two variables
I have some data, which has been constructed from multiple API pulls. Observations have an expiry date, $x_i$, which means that observations which expire before the API pull is done do not appear in ...
- 21
2
votes
0
answers
57
views
Is it Sufficient to Truncate a Left Censored Distribution?
A colleague explained their approach to dealing with left censored data in an analysis, and while I don't think it is the best approach, I am not sure if it is insufficient or not.
My colleague has ...
- 21
5
votes
1
answer
3k
views
Understanding left-truncated data in survival analysis
I am having a hard time grasping the concept of left truncation.
According to what I understand, left truncation occurs when we observe individuals only if their event of interest takes place after ...
- 79
2
votes
1
answer
44
views
How do we obtain the probability density of a truncated regression with an upper and lower bound
I know my density for $y$ is supposed to be something of this form $$g(y|x_{i},t)=\frac{f(y|x'\beta, \sigma^{2})}{F(t|x' \beta' \sigma^{2}}$$ where the numerator is the density of the normal ...
- 31
1
vote
1
answer
237
views
How to handle truncated or missing ranking data in a classification problem?
I'm preparing data for a classification problem that involves matches in a single-player sport. In each match, each competitor is either ranked and thus has a numeric rank; or unranked (rare but can ...
- 205
4
votes
2
answers
913
views
Moments of truncated Student's $t$-distribution
I performed random sampling on a Student's $t$-distribution. I used SciPy to calibrate my parameters and then truncated my allowable values to the maximum and minimum observation in the data for ...
- 171
0
votes
0
answers
182
views
Truncation versus censoring with Kaplan-Meier
I am trying to run Kaplan-Meier on a rather odd dataset and am having difficulty determining whether I should be truncating or censoring my data. I have looked at the other feeds, including this very ...
- 1
4
votes
1
answer
703
views
Fitting a gamma distribution to truncated data
I am faced with the following truncation problem:
$$
X_i \sim \Gamma(\alpha, \beta) \\
\delta_i = \chi(X_i \le \tau_i)
$$
I can observe only $(X_i, \tau_i)$ where $\delta_i = 1$ and I have no a-...
- 809
2
votes
1
answer
373
views
What is a sensible way to truncate data to a region that fits a model?
I want to use an exponential decay model in python to relate the flow rate in a device to the mass left inside it, in particular $flow=a−b×e^{−c×mass}$ where a, b and c are the parameters of the model....
- 33
3
votes
2
answers
477
views
Distributions with simple truncated expectations
For a project I'm looking for continuous distributions which have a somewhat simple closed form for upper-truncation expectation ($E[x|x>c]$).
Here are two examples I've found so far:
Exponential ...
0
votes
0
answers
133
views
Truncation of distance matrix
Given a sample $\textbf{X}_1$,...,$\textbf{X}_n\in\mathbb{R}^p$ from an arbitrary distribution with distribution function $F$ we can calculate the pairwise Mahalanobis distances between the sample ...
- 1
2
votes
1
answer
198
views
How do to deal with data missing NOT at random?
It seems that because values are missing from a specific range of my target variable, my model performs poorly when predicting samples that are actually in that range. My target variable is ...
- 155