# All Questions

195,530 questions
Filter by
Sorted by
Tagged with
174 views

### What is the correct likelihood function for an sequential, adaptive data generation process?

Consider the following sequential, adaptive data generating process for $Y_1$, $Y_2$, $Y_3$. (By sequential I mean that we generate $Y_1$, $Y_2$, $Y_3$ in sequence and by adaptive I mean that $Y_3$ is ...
3k views

### How can one speed up this correlation calculation in R without multicore?

I have a colleague who calculates correlations in which one set of scores for a subject (e.g. 100 scores) is correlated with another set of scores for that same subject. The resulting correlation ...
• 17.3k
438 views

### How to limit my input data for Jaccard item-item similarity calculation?

I'm trying to compute item-item similarity using Jaccard (specifically Tanimoto) on a large list of data in the format (userid, itemid) An item is considered as ...
• 161
17k views

### Statistical models cheat sheet

I was wondering if there is a statistical model "cheat sheet(s)" that lists any or more information: when to use the model when not to use the model required and optional inputs expected outputs has ...
562 views

### Non-trivial bound for $E[\exp(Z^2)]$ when $Z \sim {\rm Bin}(n, n^{-\beta})$ with $\beta \in (0,1)$

How to find a non-trivial upper bound on $E[\exp(Z^2)]$ when $Z \sim {\rm Bin}(n, n^{-\beta})$ with $\beta \in (0,1)$? A trivial bound is obtained for substituting $Z$ with $n$. A background on this ...
• 457
5k views

### What do ROC curves tell you that traditional inference wouldn't?

When would you tend to use ROC curves over some other tests to determine the predictive ability of some measurement on an outcome? When dealing with discrete outcomes (alive/dead, present/absent), ...
• 263
1k views

### How to interpret a control chart containing a majority of zero values?

I am using a control chart to try to work on some infection data, and will raise an alert if the infection is considered "out of control". Problems arrive when I come to a set of data where most of ...
• 2,999
184 views

### Heterogeneity with two studies

With data from two centres I want to account for potential heterogeneity or confounders between two centers. So the analysis will initially be stratified by clinical center and a chi square test ...
9k views

### Robust outlier detection in financial timeseries

I'm looking for some robust techniques to remove outliers and errors (whatever the cause) from financial time-series data (i.e. tickdata). Tick-by-tick financial time-series data is very messy. It ...
181 views

### Analysis of sells (what to buy)

I have a list of sold items by size. Shoes in this case ...
35k views

### Period detection of a generic time series

This post is the continuation of another post related to a generic method for outlier detection in time series. Basically, at this point I'm interested in a robust way to discover the periodicity/...
• 1,921
14k views

### Two-sample T-test with weighted data

I want to perform a two-sample T-test to test for a difference between two independent samples which each sample abides by the assumptions of the T-test (each distribution can be assumed to be ...
643 views

### Ranking distributional data by similarity

The question in short: What methods can be used to quantify distributional relationships between data when the distribution is unknown? Now the longer story: I have a list of distributions and would ...
• 413
420 views

### How can I apply a Pareto tail to a truncated distribution?

Many income surveys (especially older ones) truncate key variables, such as household income, at some arbitrary point, to protect confidentiality. This point changes over time. This reduces inequality ...
• 71
39k views

### Practical thoughts on explanatory vs. predictive modeling

Back in April, I attended a talk at the UMD Math Department Statistics group seminar series called "To Explain or To Predict?". The talk was given by Prof. Galit Shmueli who teaches at UMD's Smith ...
3k views

### Introduction to index decomposition analysis

Could you recommend an introductory reference to index decomposition analysis, including different methods (e.g. methods linked to the Laspeyre index and methods linked to the Divisa index) ...
• 677
64k views

### How can I test if given samples are taken from a Poisson distribution?

I know of normality tests, but how do I test for "Poisson-ness"? I have sample of ~1000 non-negative integers, which I suspect are taken from a Poisson distribution, and I would like to test that.
• 1,191
4k views

### Alternative graphics to "handle bar" plots

In my area of research, a popular way of displaying data is to use a combination of a bar chart with "handle-bars". For example, The "handle-bars" alternate between standard errors and standard ...
• 11.9k
890 views

### CI for a difference based on independent CIs

I'm looking to check my logic here. Say you measure a quantity in group A, and find the mean is 2 and your 95% confidence interval ranges from 1 to 3. Then you measure the same quantity in group B ...
• 12.8k
7k views

### Why haven't robust (and resistant) statistics replaced classical techniques?

When solving business problems using data, it's common that at least one key assumption that under-pins classical statistics is invalid. Most of the time, no one bothers to check those assumptions so ...
• 10k
464 views

### What code would you put before/after your R session? [closed]

R allows us to put code to run in the beginning/end of a session. What codes would you suggest putting there? I know of three interesting examples (although I don't have "how to do them" under my ...
47k views

### Is there an intuitive explanation why multicollinearity is a problem in linear regression?

The wiki discusses the problems that arise when multicollinearity is an issue in linear regression. The basic problem is multicollinearity results in unstable parameter estimates which makes it very ...
80k views

### Simple algorithm for online outlier detection of a generic time series

I am working with a large amount of time series. These time series are basically network measurements coming every 10 minutes, and some of them are periodic (i.e. the bandwidth), while some other aren'...
• 1,921
19k views

### Multiple Chi-Squared Tests

I have cross classified data in a 2 x 2 x 6 table. Let's call the dimensions response, A and ...
• 2,267
2k views

### Regression-multiple observations per subject

I have data for about 1 year, 100 observations, multiple observations per subject, transactions occur on weekly basis but have 6-12 subjects per week, there is no order to this. There is a policy ...
• 59
578 views

### Reasons besides prediction to build models?

Joshua Epstein wrote a paper titled "Why Model?" available at http://www.santafe.edu/media/workingpapers/08-09-040.pdf in which gives 16 reasons: Explain (very distinct from predict) Guide data ...
425 views

### Modeling success rate with gaussian distribution

In many papers I see data representing a rate of success (i.e a number between 0 and 1) modeled as a gaussian. This is clearly a sin (the range of variation of the gaussian is all of R), but how bad ...
4k views

### Can I predict percentage change in earnings from percentage change in produced and percentage changed in price? [closed]

I have computed percentage change from time1 to time2 for several variables. Can I predict percentage change in earnings from percentage change in produced and percentage changed in price? When I ...
• 59
50k views

### How to represent an unbounded variable as number between 0 and 1

I want to represent a variable as a number between 0 and 1. The variable is a non-negative integer with no inherent bound. I map 0 to 0 but what can I map to 1 or numbers between 0 and 1? I could use ...
3k views

### How to handle count data (categorical data), when it has been converted to a rate?

I am working on disease infection data, and I am puzzled on whether to handle the data as "categorical" or "continuous". "Infection Count" the number of infection cases found in a specific period ...
• 2,999
388 views

### Tonga: low rank approximation of the natural gradient, question regarding Le Roux et al. 2007

This question concerns an implementation of the topmoumoute natural gradient (tonga) algorithm as described in page 5 in the paper Le Roux et al 2007 http://research.microsoft.com/pubs/64644/tonga....
• 121
1k views

### Density function for a multivariate Bernoulli-like distribution

I'm looking for a distribution to model a vector of $k$ binary random variables, $X_1, \ldots, X_k$. Suppose I have observed that $\sum_i X_i = n$. In this case I do not want to treat them as ...
2k views

### Equation to calculate a smooth line given an irregular time series?

I'm trying to visualize a set of data that represents human body mass over time, taken from (usually) daily weighings. Because body mass tends to fluctuate +/- 3 pounds based on hydration I would ...
• 43
2k views

### How to define the significance threshold for mutual information in terms of probability of that value occurring in surrogate set?

I am trying to assess the significance of the obtained MI matrix. The initial input was a array of 3000 genes by 45 timepoints. MI was computed resulting in a array of 3600 by 3600. I am thus ...
1k views

### Help me understand nMDS algorithm

I have been reading Zuur, Ieno and Smith (2007) Analyzing ecological data, and on page 262, they try to explain how nMDS (non-metric multidimensional scaling) algorithm works. As my background is in ...
• 3,338
1 vote
473 views

### Approximating density function for a non-normal distribution

My question is actually quite short, but I'll have to start by describing the context since I am not sure how to directly ask it. Consider the following "game": We have a segment of length n ("large ...
• 1,191
88k views

### Why are mean 0 and standard deviation 1 distributions always used?

My stats has been self taught, but a lot of material I read point to a dataset having mean 0 and standard deviation of 1. If that is the case then: Why is mean 0 and SD 1 a nice property to have? ...
3k views

### Orthogonal parametrization

In general inference, why orthogonal parameters are useful, and why is it worth trying to find a new parametrization that makes the parameters orthogonal ? I have seen some textbook examples, not so ...
• 539
1k views

### Why does AIC formula in R appear to use one extra parameter than expected?

I'll use an example so that you can reproduce the results ...
• 6,214
2k views

### R command for stcox in Stata

What is the equivalent command in R for the stcox command in Stata?
• 2,569
10k views

### References for survival analysis

I am looking for a good book/tutorial to learn about survival analysis. I am also interested in references on doing survival analysis in R.
12k views

### What is the major difference between correlation and mutual information?

my question particularly applies to network reconstruction
31k views

### Is Kolmogorov-Smirnov test valid with discrete distributions?

I'm comparing a sample and checking whether it distributes as some, discrete, distribution. However, I'm not enterily sure that Kolmogorov-Smirnov applies. Wikipedia seems to imply it does not. If it ...
• 730
1k views

### Is there a radical difference in how bayesian and frequentist approaches treat nuisance parameters?

The wiki article on credible intervals has the following statement: credible intervals and confidence intervals treat nuisance parameters in radically different ways. What is the radical ...
1k views

### What is the interpretation/meaning of confidence intervals in misspecified models?

Consider the following model $Y_i = f(X_i) + e_i$ from which we observe n iid data points $\left( X_i, Y_i \right)_{i=1}^n$. Suppose that $X_i \in \mathbb{R}^d$ is a $d$ dimensional feature vector. ...
• 457
9k views

### How to interpret KL divergence quantitatively?

I am comparing two distributions with KL divergence which returns me a non-standardized number that, according to what I read about this measure, is the amount of information that is required to ...
• 413
1k views

### Where can I find good statistics quizzes?

Introductory, advanced, and even obscure, please. Mostly to test myself. I like to make sure I know what the heck I'm talking about :) Thanks
7k views

### Is a variable significant in a linear regression model?

I've got a linear regression model with the sample and variable observations and I want to know: Whether a specific variable is significant enough to remain included in the model. Whether another ...
• 730
979 views

### Reliability in Elicitation Exercise

I am trying to calculate the reliability in an elicitation exercise by analysing some test-retest questions given to the experts. The experts elicited a series of probability distributions which were ...
• 525