All Questions
213,891
questions
3
votes
1
answer
8k
views
Overlapping standard errors and statistical significance
I have a paired data set which I have placed into $x$ and $y$ columns where $x$ are the control values and $y$ are the values following drug treatment. $N=10$ for both $x$ and $y$ columns as they are ...
1
vote
1
answer
150
views
Best fitted distribution to my dataset?
I'd like to find best distribution to my data set below, and I have used (fitdistrplus) package but I do not know which distribution is fitted my data because my ...
2
votes
1
answer
356
views
How to apply distance-based clustering or dimensionality reduction for too many samples
I have a dataset with 200K samples (cases) and 30 variables. Every distance-based method for clustering or dimension reduction technique that I use, such as DBSCAN, Hierarchical Clustering, LLE, ...
2
votes
1
answer
40
views
Argmax of transformed RV
Does performing a (nonlinear) transformation to a random variable, finding the value in that space that maximizes the PDF, and transforming that value back to the original space yield the same maximum ...
1
vote
1
answer
2k
views
Statistics test to compare counts
I have two processes that produce the same type of event. For example, after 48 hours I may have 10,321 events from one process and 11,548 events from the other.
What statistical test do I use to ...
3
votes
1
answer
2k
views
Generalized linear models and noncanonical link function
When using the canonical link function for GLMs, it is true that $g(\mu) = \theta = \mathbf{X}\boldsymbol\beta$. Is this not the case when using a link that isn't canonical? For example, say we're ...
1
vote
1
answer
3k
views
Figuring out Signal to Noise Ratio of Spectral data?
I have some Raman spectral data of a chemical:
Each spectrum consist of multiple tuples of (wavenumber, intensity)
I have multiple instances of ...
0
votes
1
answer
111
views
What's a good test for assessing which model fits a particular dataset the best?
I understand you can graph the variable distributions but is there an actual best fit test?
Edit:Sorry about the wording. Basically what I did is run two linear regression models. One containing ...
2
votes
1
answer
1k
views
Bootstrapping Kruskal-Wallis test of significance
I am working with a set of 6 variables collected at 4 time periods (I consider each time period as a class in the following). For each time period, I have 82 397 observations.
If I plot the mean and ...
3
votes
0
answers
177
views
Distribution of the test statistic?
Let $\{ \mathbf{x}_1, \cdots, \mathbf{x}_d \}$, where $\mathbf{x}_i \in \mathbb{R}^n$, be random vectors distributed as $\mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma)$. Let $\mathbf{S}_0$ and $\...
28
votes
2
answers
62k
views
Adding weights to logistic regression for imbalanced data
I want to model a logistic regression with imbalanced data (9:1).
I wanted to try the weights option in the glm function in R, but I'm not 100% sure what it does.
...
2
votes
1
answer
577
views
How are the results of multivariable quantile regression interpreted?
Is multivariable quantile regression interpreted the same way as a multivariable linear regression would be interpreted? For example, would I say something like "the coefficient represents the ...
2
votes
2
answers
3k
views
featurizing images of different sizes
I'm training a non linear svm to do classification on images. I'm featurizing the image by creating 3 features for each pixel, its rgb value. My question is: How should i normalize images of different ...
1
vote
3
answers
12k
views
Ordinal data in regression?
My data is almost only ordinal data. My dependent variable is a 4 point scale:
1 = completely agree
2 = agree to some extent
3 = disagree to some extent
4 = completely disagree
I want to use ...
1
vote
0
answers
375
views
Determining if data is normally distributed [duplicate]
I'm working on a case study where the number of phone calls (min = 0 calls and max = 10 calls) is measured every 30 seconds in a metro station. The experiment was done three times during on day where ...
1
vote
0
answers
158
views
Conditional Distribution in Structural Equation Modeling
Let $\{\mathcal{N}(y_i;\mu(\theta),\Sigma(\theta):\theta \in \Theta\}$ be an SEM, where $y_i$ is one observation and $\theta$ a parameter vector. Let's say there are a lot of missing values in our ...
1
vote
1
answer
1k
views
Using word embeddings / word2vec for classification of entiy
I am trying to used word2vec or word vectors for classification based on entity.
For example ,
I have to classify the following words in a sentence as :
" Google gives information about Nigeria "
...
4
votes
0
answers
1k
views
R package for active learning and sampling [closed]
I tried to find R package for active learning/sampling and stumbled over "activelearning" package. However, this package is not available on CRAN.
Does someone know how good is this package?
Are ...
2
votes
1
answer
869
views
Mean and variance for unequal samples
I have a sampling of variable sized plots. Each plot contains the number of trees present on the plot. Given:
$n=$ the number of plots
$s_i=$ the size of the $i^{th}$ plot
$y_i=$ the number of trees ...
3
votes
0
answers
198
views
Prediction intervals for forecasts using spectral analysis
I have circadian data which typically have a period of around 24 hours so using spectral analysis seems appropriate. I've used spectrum resampling which is quite robust to changes in period which ...
10
votes
6
answers
2k
views
Machine Learning Book (Python)
I'm in search of a good book about Machine Learning.
Requirements:
Good theoretical depth (while being a ML beginner, I hold a degree in mathematics), but accompained with good examples and plenty ...
1
vote
0
answers
471
views
How to analyze skewed and unskewed data on SPSS? [closed]
Study design:
I have shown participants a series of photos of themselves vs. other people's bodies (original and manipulated), and asked them to find the original. Their responses were recorded and ...
1
vote
0
answers
55
views
Methods to find the correlation between one variable and a set of variables
I have discrete measures (let's say $I(x,y,t)$, i.e., coordinates in space and time) on a geographical map that are sampled randomly. I also have a constant flow of optical images in several ...
10
votes
2
answers
2k
views
Random Forest: Class specific feature importance
I'm using the bigrf R-package to analyse a dataset with ca. 50.000 observations x 120 variables, classified into two groups.
After growing a forest of 1000 trees, ...
7
votes
2
answers
1k
views
Hypothesis test based on entropy
I am reading the wikipedia page on hypothesis testing, but a I can't find any reference to tests based on entropy. Which are good hypothesis tests based on entropy or quantities derived from it?
0
votes
0
answers
334
views
Still trying to learn lme4 syntax
If I have a sample of rats in which I want to measure protein levels in several different brain regions, the rats are of two genotypes and both sexes, and I want to take into account that the brain ...
4
votes
1
answer
2k
views
Best way to deal with forecasting with noisy data?
I have a bunch of sales data. It is from distributors of 2000 different items, who service big companies and large distributors to a number of small independent stores. They sell some items which do ...
1
vote
0
answers
323
views
comparing multiple proportions over time
I have a dataset of 22 fish species in a fish market, sampled monthly, from 2007 to 2011. I want to see if there is a statistically significant change in their relative proportions over time. I first ...
2
votes
1
answer
233
views
When testing for linearity to the logit, do I include all variables in the same step? [closed]
I'm testing for linearity to the logit of the continuous predictor variables in a logistic regression by entering X and Xln(X) terms into the regression. I have multiple continuous predictors. When I ...
0
votes
1
answer
484
views
Interpretation of Multiple Change point results and graph for offline analysis in R
Using the changepoint R package to detect and estimate multiple change points with non-normality assumption for mean with the following codes
...
1
vote
0
answers
52
views
EM algorithm for Blind separation and deconvolution of noisy signals using mixture models
I have a question from the paper: Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models (pdf). In section 2, equations 7, 8, 9 read:
\begin{align}
\pi_{zx}(\...
2
votes
0
answers
339
views
Calculate value needed to reach target by the end of a year [closed]
I have a set of values (scores to a particular KPI), these are reported monthly. I need to calculate what I'd have to score every month from now until the end of the year to reach that target.
I'm ...
2
votes
0
answers
393
views
What is the proper name for a Quantile Bar Chart?
What is the proper name for a Bar Chart plotting quantiles (like percentiles) in abscissa (X-axis) and a corresponding continuous variable in ordinate (Y-axis)? For example, if the chart is based on ...
16
votes
2
answers
35k
views
Output of Logistic Regression Prediction
I have created a Logistic Regression using the following code:
...
4
votes
1
answer
85
views
Intuition behind biclustering
I am self-studying Kevin Murphy's book Machine learning - A probabilistic perspective and stumbled upon the following paragraph on biclustering. I understand the independence assumption - but am not ...
0
votes
0
answers
693
views
How NA in a model will impact on predict?
I wrote a script that create a logistic model, for Email opening probability, for each user name.
...
2
votes
1
answer
89
views
Characterizing the distribution of manufacturing data
I have some data from manufacturing concerning the variability of certain geometric dimensions of a given machine. These are real numbers (diameters, angles, lengths, etc.), so the variables are ...
3
votes
0
answers
280
views
Hyperparameter optimization in 6-dimensional continuous space
I am using Random Forest and Stochastic Gradient Boosting to predict a categorical target variable exhibiting severe between-class imbalance. I am using oversampling to make sure that the models do ...
0
votes
1
answer
131
views
What type of regression model do I need to use?
I am looking at the relationship between the two concepts of psychological strategy (PS) usage and athlete engagement (AE), looking to see if PS predicts AE as a whole and if certain subscales of PS ...
1
vote
0
answers
78
views
Can we estimate VAR model using monthly data for 7 years?
I am interested in estimating the long run relation between electricity shortfall and industrial output with other variables. It forms a 6*6 matrix. Can I get reliable estimates? I am very much ...
7
votes
1
answer
3k
views
GARCH vs SV for Forecasting
I believe I am aware of how GARCH family and stochastic volatility models differ in their construction and assumptions on the volatility states, (i.e. GARCH family assumes deterministic volatility ...
3
votes
1
answer
358
views
MLE and Bayesian methods
I saw in some lecture the fact that as the number of data points N goes to infinity, the prediction of the Bayesian method goes to the prediction of the MLE. Can someone explain what exactly this ...
7
votes
1
answer
11k
views
Limitation of LDA (latent dirichlet allocation)
I'd like to get a list of limitations of LDA. I know that LDA does not work for short document set like a set of tweets very well.
Are there such known limitations of LDA? Some reference including a ...
1
vote
1
answer
58
views
A smoothed series still exibits strong seasonality
I have a monthly time series. It is basically a price level series (inflation data), and I converted it into monthly percentage changes (i.e. like the CPI measure). This time series exhibits extremely ...
2
votes
2
answers
375
views
Undefined central moments; How to show that calculating a sample mean does not make sense?
Let $f$ be some probability density function with undefined central moments. For example, suppose $f$ is a Cauchy distribution.
Say I draw two samples of size $N=100$ from that distribution. The mean ...
3
votes
1
answer
330
views
How can I recognize when I must apply "log transformation"?
I have some time series - http://ww2.coastal.edu/kingw/statistics/R-tutorials/simplenonlinear.html
In this article author try to use log transformation for pressure data.
How can I recognize that ...
2
votes
0
answers
88
views
Extreme Value Theory Data Scaling
I have a data set available of almost thirty years of data, with for each month the number of occurrences of a certain event and the total number in the set available.
What I would like to compute is ...
2
votes
1
answer
87
views
Can I use Logistic regression to remove 0s from a multiple linear regression?
I have a large data set (~17k data points) on which I would like to do a multiple regression. However, the explained variable has several instances of 0 (~6k).
In finding an appropriate model for ...
6
votes
1
answer
339
views
What's the maximum expectation of a conditional variance, $E[\operatorname{Var}(X+Z_1 \mid X+Z_2)]$?
Let $X,Z_1,Z_2$ be 3 mutually independent RV's, with $Z_1, Z_2$ assuming $N(0,1)$ distribution. $X$ is constrained to have unit 2nd moment, i.e. $E[X^2] =1$, but may take arbitrary distribution. The ...
5
votes
4
answers
9k
views
Can the coefficient of determination $R^2$ be more than one? What is its upper bound?
It is well known that if you add additional independent variables in a linear regression, the $R^2$ of the new model is at least as large as the previous model, so you obtain a lower bound for the $R^...