All Questions

Filter by
Sorted by
Tagged with
3 votes
1 answer
8k views

Overlapping standard errors and statistical significance

I have a paired data set which I have placed into $x$ and $y$ columns where $x$ are the control values and $y$ are the values following drug treatment. $N=10$ for both $x$ and $y$ columns as they are ...
Provo 's user avatar
  • 33
1 vote
1 answer
150 views

Best fitted distribution to my dataset?

I'd like to find best distribution to my data set below, and I have used (fitdistrplus) package but I do not know which distribution is fitted my data because my ...
Salam Abbas's user avatar
2 votes
1 answer
356 views

How to apply distance-based clustering or dimensionality reduction for too many samples

I have a dataset with 200K samples (cases) and 30 variables. Every distance-based method for clustering or dimension reduction technique that I use, such as DBSCAN, Hierarchical Clustering, LLE, ...
Matin Kh's user avatar
  • 133
2 votes
1 answer
40 views

Argmax of transformed RV

Does performing a (nonlinear) transformation to a random variable, finding the value in that space that maximizes the PDF, and transforming that value back to the original space yield the same maximum ...
Vivek Subramanian's user avatar
1 vote
1 answer
2k views

Statistics test to compare counts

I have two processes that produce the same type of event. For example, after 48 hours I may have 10,321 events from one process and 11,548 events from the other. What statistical test do I use to ...
benhamner's user avatar
  • 3,013
3 votes
1 answer
2k views

Generalized linear models and noncanonical link function

When using the canonical link function for GLMs, it is true that $g(\mu) = \theta = \mathbf{X}\boldsymbol\beta$. Is this not the case when using a link that isn't canonical? For example, say we're ...
norm850's user avatar
  • 33
1 vote
1 answer
3k views

Figuring out Signal to Noise Ratio of Spectral data?

I have some Raman spectral data of a chemical: Each spectrum consist of multiple tuples of (wavenumber, intensity) I have multiple instances of ...
Tinker's user avatar
  • 131
0 votes
1 answer
111 views

What's a good test for assessing which model fits a particular dataset the best?

I understand you can graph the variable distributions but is there an actual best fit test? Edit:Sorry about the wording. Basically what I did is run two linear regression models. One containing ...
Sara A's user avatar
  • 31
2 votes
1 answer
1k views

Bootstrapping Kruskal-Wallis test of significance

I am working with a set of 6 variables collected at 4 time periods (I consider each time period as a class in the following). For each time period, I have 82 397 observations. If I plot the mean and ...
Améline V.'s user avatar
3 votes
0 answers
177 views

Distribution of the test statistic?

Let $\{ \mathbf{x}_1, \cdots, \mathbf{x}_d \}$, where $\mathbf{x}_i \in \mathbb{R}^n$, be random vectors distributed as $\mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma)$. Let $\mathbf{S}_0$ and $\...
Maxtron's user avatar
  • 514
28 votes
2 answers
62k views

Adding weights to logistic regression for imbalanced data

I want to model a logistic regression with imbalanced data (9:1). I wanted to try the weights option in the glm function in R, but I'm not 100% sure what it does. ...
ching's user avatar
  • 757
2 votes
1 answer
577 views

How are the results of multivariable quantile regression interpreted?

Is multivariable quantile regression interpreted the same way as a multivariable linear regression would be interpreted? For example, would I say something like "the coefficient represents the ...
Sara A's user avatar
  • 31
2 votes
2 answers
3k views

featurizing images of different sizes

I'm training a non linear svm to do classification on images. I'm featurizing the image by creating 3 features for each pixel, its rgb value. My question is: How should i normalize images of different ...
mt88's user avatar
  • 551
1 vote
3 answers
12k views

Ordinal data in regression?

My data is almost only ordinal data. My dependent variable is a 4 point scale: 1 = completely agree 2 = agree to some extent 3 = disagree to some extent 4 = completely disagree I want to use ...
user84117's user avatar
1 vote
0 answers
375 views

Determining if data is normally distributed [duplicate]

I'm working on a case study where the number of phone calls (min = 0 calls and max = 10 calls) is measured every 30 seconds in a metro station. The experiment was done three times during on day where ...
Dimitri's user avatar
  • 111
1 vote
0 answers
158 views

Conditional Distribution in Structural Equation Modeling

Let $\{\mathcal{N}(y_i;\mu(\theta),\Sigma(\theta):\theta \in \Theta\}$ be an SEM, where $y_i$ is one observation and $\theta$ a parameter vector. Let's say there are a lot of missing values in our ...
Julian Karch's user avatar
  • 1,890
1 vote
1 answer
1k views

Using word embeddings / word2vec for classification of entiy

I am trying to used word2vec or word vectors for classification based on entity. For example , I have to classify the following words in a sentence as : " Google gives information about Nigeria " ...
Sarath R Nair's user avatar
4 votes
0 answers
1k views

R package for active learning and sampling [closed]

I tried to find R package for active learning/sampling and stumbled over "activelearning" package. However, this package is not available on CRAN. Does someone know how good is this package? Are ...
Michael's user avatar
  • 219
2 votes
1 answer
869 views

Mean and variance for unequal samples

I have a sampling of variable sized plots. Each plot contains the number of trees present on the plot. Given: $n=$ the number of plots $s_i=$ the size of the $i^{th}$ plot $y_i=$ the number of trees ...
deAtog's user avatar
  • 23
3 votes
0 answers
198 views

Prediction intervals for forecasts using spectral analysis

I have circadian data which typically have a period of around 24 hours so using spectral analysis seems appropriate. I've used spectrum resampling which is quite robust to changes in period which ...
outb4break's user avatar
10 votes
6 answers
2k views

Machine Learning Book (Python)

I'm in search of a good book about Machine Learning. Requirements: Good theoretical depth (while being a ML beginner, I hold a degree in mathematics), but accompained with good examples and plenty ...
1 vote
0 answers
471 views

How to analyze skewed and unskewed data on SPSS? [closed]

Study design: I have shown participants a series of photos of themselves vs. other people's bodies (original and manipulated), and asked them to find the original. Their responses were recorded and ...
Student's user avatar
  • 11
1 vote
0 answers
55 views

Methods to find the correlation between one variable and a set of variables

I have discrete measures (let's say $I(x,y,t)$, i.e., coordinates in space and time) on a geographical map that are sampled randomly. I also have a constant flow of optical images in several ...
matt's user avatar
  • 111
10 votes
2 answers
2k views

Random Forest: Class specific feature importance

I'm using the bigrf R-package to analyse a dataset with ca. 50.000 observations x 120 variables, classified into two groups. After growing a forest of 1000 trees, ...
Misconstruction's user avatar
7 votes
2 answers
1k views

Hypothesis test based on entropy

I am reading the wikipedia page on hypothesis testing, but a I can't find any reference to tests based on entropy. Which are good hypothesis tests based on entropy or quantities derived from it?
emanuele's user avatar
  • 2,088
0 votes
0 answers
334 views

Still trying to learn lme4 syntax

If I have a sample of rats in which I want to measure protein levels in several different brain regions, the rats are of two genotypes and both sexes, and I want to take into account that the brain ...
Bryan's user avatar
  • 1,190
4 votes
1 answer
2k views

Best way to deal with forecasting with noisy data?

I have a bunch of sales data. It is from distributors of 2000 different items, who service big companies and large distributors to a number of small independent stores. They sell some items which do ...
Sameer Patel's user avatar
1 vote
0 answers
323 views

comparing multiple proportions over time

I have a dataset of 22 fish species in a fish market, sampled monthly, from 2007 to 2011. I want to see if there is a statistically significant change in their relative proportions over time. I first ...
James Abbott's user avatar
2 votes
1 answer
233 views

When testing for linearity to the logit, do I include all variables in the same step? [closed]

I'm testing for linearity to the logit of the continuous predictor variables in a logistic regression by entering X and Xln(X) terms into the regression. I have multiple continuous predictors. When I ...
Kat's user avatar
  • 29
0 votes
1 answer
484 views

Interpretation of Multiple Change point results and graph for offline analysis in R

Using the changepoint R package to detect and estimate multiple change points with non-normality assumption for mean with the following codes ...
Adidul's user avatar
  • 75
1 vote
0 answers
52 views

EM algorithm for Blind separation and deconvolution of noisy signals using mixture models

I have a question from the paper: Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models (pdf). In section 2, equations 7, 8, 9 read: \begin{align} \pi_{zx}(\...
Fariborz Mafakheri's user avatar
2 votes
0 answers
339 views

Calculate value needed to reach target by the end of a year [closed]

I have a set of values (scores to a particular KPI), these are reported monthly. I need to calculate what I'd have to score every month from now until the end of the year to reach that target. I'm ...
Chris Andrews's user avatar
2 votes
0 answers
393 views

What is the proper name for a Quantile Bar Chart?

What is the proper name for a Bar Chart plotting quantiles (like percentiles) in abscissa (X-axis) and a corresponding continuous variable in ordinate (Y-axis)? For example, if the chart is based on ...
Ismael Ghalimi's user avatar
16 votes
2 answers
35k views

Output of Logistic Regression Prediction

I have created a Logistic Regression using the following code: ...
SeBee's user avatar
  • 161
4 votes
1 answer
85 views

Intuition behind biclustering

I am self-studying Kevin Murphy's book Machine learning - A probabilistic perspective and stumbled upon the following paragraph on biclustering. I understand the independence assumption - but am not ...
Wouter's user avatar
  • 2,192
0 votes
0 answers
693 views

How NA in a model will impact on predict?

I wrote a script that create a logistic model, for Email opening probability, for each user name. ...
Dima Ha's user avatar
  • 101
2 votes
1 answer
89 views

Characterizing the distribution of manufacturing data

I have some data from manufacturing concerning the variability of certain geometric dimensions of a given machine. These are real numbers (diameters, angles, lengths, etc.), so the variables are ...
DeltaIV's user avatar
  • 17.9k
3 votes
0 answers
280 views

Hyperparameter optimization in 6-dimensional continuous space

I am using Random Forest and Stochastic Gradient Boosting to predict a categorical target variable exhibiting severe between-class imbalance. I am using oversampling to make sure that the models do ...
Antoine's user avatar
  • 6,159
0 votes
1 answer
131 views

What type of regression model do I need to use?

I am looking at the relationship between the two concepts of psychological strategy (PS) usage and athlete engagement (AE), looking to see if PS predicts AE as a whole and if certain subscales of PS ...
James Tromans's user avatar
1 vote
0 answers
78 views

Can we estimate VAR model using monthly data for 7 years?

I am interested in estimating the long run relation between electricity shortfall and industrial output with other variables. It forms a 6*6 matrix. Can I get reliable estimates? I am very much ...
Bushra Faizi's user avatar
7 votes
1 answer
3k views

GARCH vs SV for Forecasting

I believe I am aware of how GARCH family and stochastic volatility models differ in their construction and assumptions on the volatility states, (i.e. GARCH family assumes deterministic volatility ...
Zachary Blumenfeld's user avatar
3 votes
1 answer
358 views

MLE and Bayesian methods

I saw in some lecture the fact that as the number of data points N goes to infinity, the prediction of the Bayesian method goes to the prediction of the MLE. Can someone explain what exactly this ...
cruvadom's user avatar
  • 195
7 votes
1 answer
11k views

Limitation of LDA (latent dirichlet allocation)

I'd like to get a list of limitations of LDA. I know that LDA does not work for short document set like a set of tweets very well. Are there such known limitations of LDA? Some reference including a ...
Light Yagmi's user avatar
1 vote
1 answer
58 views

A smoothed series still exibits strong seasonality

I have a monthly time series. It is basically a price level series (inflation data), and I converted it into monthly percentage changes (i.e. like the CPI measure). This time series exhibits extremely ...
phil_20686's user avatar
2 votes
2 answers
375 views

Undefined central moments; How to show that calculating a sample mean does not make sense?

Let $f$ be some probability density function with undefined central moments. For example, suppose $f$ is a Cauchy distribution. Say I draw two samples of size $N=100$ from that distribution. The mean ...
wor's user avatar
  • 21
3 votes
1 answer
330 views

How can I recognize when I must apply "log transformation"?

I have some time series - http://ww2.coastal.edu/kingw/statistics/R-tutorials/simplenonlinear.html In this article author try to use log transformation for pressure data. How can I recognize that ...
Max Usanin's user avatar
2 votes
0 answers
88 views

Extreme Value Theory Data Scaling

I have a data set available of almost thirty years of data, with for each month the number of occurrences of a certain event and the total number in the set available. What I would like to compute is ...
Pelotas's user avatar
  • 35
2 votes
1 answer
87 views

Can I use Logistic regression to remove 0s from a multiple linear regression?

I have a large data set (~17k data points) on which I would like to do a multiple regression. However, the explained variable has several instances of 0 (~6k). In finding an appropriate model for ...
Tom Malkin's user avatar
6 votes
1 answer
339 views

What's the maximum expectation of a conditional variance, $E[\operatorname{Var}(X+Z_1 \mid X+Z_2)]$?

Let $X,Z_1,Z_2$ be 3 mutually independent RV's, with $Z_1, Z_2$ assuming $N(0,1)$ distribution. $X$ is constrained to have unit 2nd moment, i.e. $E[X^2] =1$, but may take arbitrary distribution. The ...
syeh_106's user avatar
  • 826
5 votes
4 answers
9k views

Can the coefficient of determination $R^2$ be more than one? What is its upper bound?

It is well known that if you add additional independent variables in a linear regression, the $R^2$ of the new model is at least as large as the previous model, so you obtain a lower bound for the $R^...
Irland's user avatar
  • 91

15 30 50 per page