Questions tagged [overlapping-data]
The overlapping-data tag has no usage guidance.
81
questions
0
votes
1
answer
17
views
Comparing distribution of a biomarker among patients with overlapping clinical features
I have a question how to compare the distribution of one tested biomarker among patients with different overlapping clinical syndromes.
In my case, I am investigating patients with multiple sclerosis ...
- 1
0
votes
1
answer
52
views
rolling mean series - strong autocorrelation
I need to works with time-series of rolling means of a certain variable that comes from the data. I have this rolling means for N different individuals. Call them Y. I would like to test some ...
- 158
0
votes
1
answer
21
views
How to organize an overlapping ordinal variables?
I am analyzing some data with attributes about how long it takes to commute, and it is an overlapping ordinal variable like the following:
Time
Number of People
1-5 minutes
10
6-10 minutes
24
less ...
0
votes
0
answers
35
views
Measuring similarity between two datasets
This is quite straightforward:
I have a dataset with 8 features containing cell measurements from a patient. I want to compare two patients to a reference one. To do this I perform UMAP to reduce the ...
- 13
1
vote
1
answer
55
views
Best way to rank list of values having overlapping uncertainties
I have a list of individuals and their success rates on a task. The count of tasks for each varies widely, so the uncertainty in the estimates of their proficiency varies. Furthermore, typical ...
- 111
0
votes
0
answers
5
views
Selectivity freedom with overlapping conditions
I'm facing a mathematical challenge related to selectivity freedom under overlapping conditions. To make it more relatable, let's consider an analogy.
Imagine a farmer who has just completed his apple ...
- 1
0
votes
0
answers
23
views
Overlapping Dummy Variables in Logistic Regression
I am developing a logistic regression model that predicts pregnancy. One of the variables to be included has multiple overlapping levels.
Someone may be coded as being as having a gestational age of ...
- 155
0
votes
1
answer
144
views
Overlapping vs non-overapping windows in random walk
Consider a simple random walk. I am trying to compute the variance of differences over a window of certain size dw which could, for example, model returns of a ...
- 217
2
votes
1
answer
141
views
When two distributions overlap, how to separate one distribution from the mixture distribution if I the other distribution is known?
I have a data which can be classified into two groups. As you see, Figure(a) shows that they are easily classified into group A and B. However, sometimes they are overlapped and it is impossible set a ...
1
vote
1
answer
125
views
If a class overlay is too high for one to another, is it possible to do classification?
This is a figure of PCA Map of several class data that i am trying to classify:
By looking at it, I wanted to be sure if this is not really applicable to be classified? (since it has a high overlap ...
- 159
1
vote
2
answers
96
views
Logistic regression for observations on a monthly grid - impact of overlapping information of a given customer
My question is based on the usual setting in credit scoring. Assume we have historical monthly observations of customers, their risk factors $R_{i,t}$ and the flag whether they are currently in ...
- 3,396
0
votes
0
answers
60
views
How many "effective" observations from sliding window sampling?
I have data on the trajectory of a particle and am interested in the displacement of the particle in a given time period. In particular, I have information on how much a particle has moved in each ...
- 1
0
votes
1
answer
113
views
Timeseries analysis with partially overlapping date ranges
I am working with a dataset of car/vehicle measurements at a Target location. In this dataset, there are multiple providers that give measurements about the number of vehicles at that Target (based on ...
1
vote
0
answers
202
views
Comparing proportions from overlapping samples with a complex survey design
I am trying to test whether there has been a statistically significant increase in the prevalence of a condition between two years of a complex survey using R. The samples are partly overlapping ...
0
votes
1
answer
156
views
What is correct way to calculate the expected overlap of protein sets in two related biological system
I have a question regarding the statistics of protein set differences in two biological samples.
I have performed quantitative proteomics analyses on both WT and KO cells (derived from the WT cells by ...
- 11
0
votes
0
answers
75
views
Can overlap impact the EM algorithm?
I am using EM algorithm for clustering. I see that if there is an overlap, the accuracy is reduced. However, it is improvesd if no overlap exists.
My question is, can overlap affect the performance of ...
- 640
2
votes
0
answers
108
views
How to quantify temporal overlap of species abundance?
I want to compare the extent of overlap of species abundances in the following two secanrios. In the first scenario, species barely overlap so the index or overlap value should be zero. in the second ...
- 603
3
votes
1
answer
242
views
Probability of outcomes from overlapping samples of a random variable
I have a feeling there is a straight forward answer to my question, but I'm not sure what the appropriate term to search for is. I'm trying to get a sense of what the probability is of finding some ...
- 31
1
vote
0
answers
185
views
What are pros and cons of AR and other time series models with overlapping datapoints
Anyone have any links or resources on pros/cons of building a timeseries model with overlapping data points? Generally, a lot of text build models on daily returns, but let's say the daily variable ...
- 3,083
1
vote
0
answers
967
views
Overlapping data in time series model?
I have some time series data that I am trying to model - basically historical elasticity reads with daily sales data. I really am just interested in finding the 'mean' value - not necessarily predict ...
- 3,083
0
votes
1
answer
84
views
Population overlap of random distributions of binary choices
I’m trying to figure out how to calculate the overlap (concurrence) of certain building features, for example: high-efficiency windows, thick insulation, efficient lights, etc. I know the probability ...
- 15
1
vote
2
answers
713
views
Average count of overlap between random samples
If I randomly sample n numbers from 1..m, x times, what would be the average count of ...
- 13
1
vote
1
answer
120
views
Properties of OLS estimator for overlapping observations' regression
I am running regressions of long-horizon gross financial returns $$R_{t+k} = \prod_{i=1}^{k} R_{t+i}$$ where $R_t=1+r_t$, on current dividend-price ratios $DP_t=\frac{D}{P}_t$:
$$R_{t+k}=\alpha +\beta ...
- 329
1
vote
1
answer
2k
views
How to deal with visually overlapping clusters
I have done hierarchal clustering (average) and silhouette method shows 6 as the best k. However visualising I see that most of the clusters are overlapping. Any way to deal with the overlapping ...
- 13
2
votes
2
answers
238
views
How to get the estimated means and standard deviations of two overlapping circular normal distributions?
I have the following plot:
Overlapping circular normal distributions
I would like to estimate the means and standard deviations of the apparent overlapping normal distributions. This is slightly ...
0
votes
0
answers
37
views
How to quantify the overlap of 2 distribution
I have three events in time that follows a Lognormal distribution and that, in theory, occur in order T0 then T1 then ...
- 151
2
votes
0
answers
51
views
Averaging methods to deal with overlapping time series
Consider the scenario where one is trying to fit a distribution to historical yearly excess returns of an equity index (say S&P 500 excess return over T bills). Monthly data are available but we ...
2
votes
1
answer
44
views
Why do two confidence intervals only overlap when their difference $(\theta_1 − \theta_2) \pm z(SE_1 + SE_2)$ includes 0?
I am able to check if the interval contains 0 which I can find out by just plugging numbers into the above formula but I am curious as to why this interval containing zero means that the two CI's of $\...
- 23
2
votes
1
answer
936
views
Quantifying overlap between two categorical distributions with some non-identical categories
I'm looking for a way to measure the overlap (or general similarity) between two categorical distributions in which some of the categories are shared between each and some are not. For example, if the ...
- 395
5
votes
0
answers
475
views
Robust standard errors under overlapping observations: confusing simulation results from alternative methods
I have a time series of $h$-step-ahead forecasts $\hat{y}_{t+h|t}$ for $t=1,\dots,T-h$ with $h>1$. I also have the corresponding realized values of their targets $y_{t+h}$ and the corresponding ...
- 64.8k
0
votes
0
answers
69
views
Stationarity in quarterly overlapping data
If I have a time series of quarterly default rates (e.g. of mortgages), by definition they will be 75% correlated, thus leading in non stationarity.
What should I do in this case? Can I ignore lags up ...
- 483
2
votes
1
answer
157
views
Bayesian framework - Prior and Likelihood independence
I've a question on how, if ever needed, one can deal with the possible overlapping dependence of prior information and current data.
Basically, I was given the output parameters of a regression that ...
- 337
3
votes
0
answers
215
views
Correction for sample overlap in a meta-analysis of prevalence (metafor package)
I want to conduct a meta-analysis of around 30 studies (from a systematic review).
Some background of the studies: The quantity of interest is the prevalence of RSV infection. Different studies ...
- 71
2
votes
0
answers
143
views
Standard Error Bias from Monte Carlo Simulations
I am currently working through this paper on overlapping observations by Britten-Jones, Neuberger and Nolte 1, which presents an estimation method for correcting standard errors when performing ...
2
votes
0
answers
319
views
How to adjust R^2 and the Coefficient of Determination when using Overlapping Observations?
I am currently working through the paper Improved Inference and Estimation in Regression With Overlapping Observations [1] which presents an elegant way to do inference on $\beta$ in a linear ...
2
votes
1
answer
213
views
Estimating moments of annual return distribution
I am interested in estimating moments of annual return distributions of indices with limited historical data. I.e. equity/fixed income indices which have only 10-20 years of daily data available. The ...
2
votes
0
answers
404
views
Effective sample size of a time series of overlapping observations
Question: How do I calculate the effective sample size of a time series under overlapping observations?
By overlapping observations of $x_t$ where the overlap is of length $k$ I mean a case where $...
- 64.8k
4
votes
1
answer
268
views
Bias of overlap sampling
Say I have a series $x_1, ...., x_N$ independently drawn from $Normal(0,1)$
Say I create sub-samples
$y_1 = x_1+..x_n$,
$y_2 = x_2+..x_{n+1}$,
$y_m = x_{N-n}+..x_N$,
I was expecting $std(\{y\}) =...
- 143
1
vote
0
answers
45
views
Is it allowed to reduce a dataset of moving averages to run an AR(1) model properly?
I run a simple AR(1) and AR(2) model with the following code:
ar.ols(df$y, order.max = 1)
ar.ols(df$y, order.max =2)
My dataset is as follows: I do have yearly ...
- 39
0
votes
1
answer
146
views
How to change the observation for the first lag in an AR(1) model?
I run a simple AR(1) model in my analysis using ols:
ar.ols(df$y, order.max = 1))
However, I work with generations as my unit of analysis. Therefore, the first ...
- 39
6
votes
1
answer
640
views
$R^2$ and adjusted $R^2$ in presence of overlapping observations
Given a linear model
$$
y=X\beta+\varepsilon,
$$
the population value of $R^2$ is
$$
R^2=1-\frac{\text{Var}(\varepsilon)}{\text{Var}(y)}.
$$
The vanilla estimator of $R^2$ is
$$
\hat R^2=1-\frac{\...
- 64.8k
1
vote
0
answers
18
views
Multiple Comparisons Problem for non-independant subsets
I'm investigating whether a program was effective or not using BACI analysis and noticed that overall it had no effect (The 2-way ANOVA and non-parametric approach gave a p-value of about 0.5). I want ...
- 11
2
votes
1
answer
450
views
Training ARIMA based on overlapping hourly weather forecasts
I am working with hourly water level data and I plan to forecast each day the next two weeks (on an hourly base meaning 14*24 = 336 forecasts each day).
My regressor is an hourly weather forecast.
...
- 3,493
0
votes
1
answer
39
views
Cross checking AI Selections vs Random Chance
Fifty Three Images
AI Selects 20 images based on various criteria.
I Select 20 images based on personal criteria.
In a two group selection of all 53 items what is the likely overlap due to random ...
0
votes
1
answer
279
views
Survival Analysis on recurrent behavior time series predictor
We are trying to build a credit model to predict the default time (or finally closed the loans as censored).
The predictor is a high-dimension time series of current observed previous payment behavior....
- 106
4
votes
0
answers
199
views
Spatial regression with overlapping areas
Is it statistically correct to calculate a regression with overlapping areas? I have market areas as the spatial unit with different sociodemographic and (macro)economic variables and I´m examining ...
- 141
1
vote
0
answers
31
views
(Another) What Statistical Test Should I Use?
If I ask 600 different people to rate 3 different images from 0-10 (i.e 3 samples of 200 people per image) and I calculate the mean scores per image as well as the overall mean for all 3 images, can I ...
1
vote
0
answers
154
views
Probability of Finding Two Matching Subsequences in a Sequence
I'm currently studying DNA sequencing and am trying to find a formula which gives the probability that a subsequence of length $k$ appears twice in a sequence of $L$ bases (characters); this is pretty ...
- 11
0
votes
0
answers
62
views
How to analyze combined data from studies with overlap in subjects?
I have data from multiple placebo-controlled cross-over trials (each with $n \simeq 20$) with unique phenotypes measured during the placebo phase. I'd like to combine the data from these measurements ...
4
votes
2
answers
145
views
The minimum number of classes in the case of overlapping classes
I've a set of elements that should be grouped in the minimum number of groups, each element belongs to overlapping classes, for eg:
a belongs to the classes x , y , z;
b belongs to the classes x , y;
...
- 93