All Questions

Filter by
Sorted by
Tagged with
2 votes
2 answers
174 views

What is the correct likelihood function for an sequential, adaptive data generation process?

Consider the following sequential, adaptive data generating process for $Y_1$, $Y_2$, $Y_3$. (By sequential I mean that we generate $Y_1$, $Y_2$, $Y_3$ in sequence and by adaptive I mean that $Y_3$ is ...
user avatar
6 votes
3 answers
3k views

How can one speed up this correlation calculation in R without multicore?

I have a colleague who calculates correlations in which one set of scores for a subject (e.g. 100 scores) is correlated with another set of scores for that same subject. The resulting correlation ...
user avatar
6 votes
2 answers
438 views

How to limit my input data for Jaccard item-item similarity calculation?

I'm trying to compute item-item similarity using Jaccard (specifically Tanimoto) on a large list of data in the format (userid, itemid) An item is considered as ...
user avatar
46 votes
5 answers
17k views

Statistical models cheat sheet

I was wondering if there is a statistical model "cheat sheet(s)" that lists any or more information: when to use the model when not to use the model required and optional inputs expected outputs has ...
7 votes
3 answers
562 views

Non-trivial bound for $E[\exp(Z^2)]$ when $Z \sim {\rm Bin}(n, n^{-\beta})$ with $\beta \in (0,1)$

How to find a non-trivial upper bound on $E[\exp(Z^2)]$ when $Z \sim {\rm Bin}(n, n^{-\beta})$ with $\beta \in (0,1)$? A trivial bound is obtained for substituting $Z$ with $n$. A background on this ...
user avatar
  • 457
12 votes
5 answers
5k views

What do ROC curves tell you that traditional inference wouldn't?

When would you tend to use ROC curves over some other tests to determine the predictive ability of some measurement on an outcome? When dealing with discrete outcomes (alive/dead, present/absent), ...
user avatar
  • 263
10 votes
5 answers
1k views

How to interpret a control chart containing a majority of zero values?

I am using a control chart to try to work on some infection data, and will raise an alert if the infection is considered "out of control". Problems arrive when I come to a set of data where most of ...
user avatar
  • 2,999
2 votes
1 answer
184 views

Heterogeneity with two studies

With data from two centres I want to account for potential heterogeneity or confounders between two centers. So the analysis will initially be stratified by clinical center and a chi square test ...
user avatar
18 votes
3 answers
9k views

Robust outlier detection in financial timeseries

I'm looking for some robust techniques to remove outliers and errors (whatever the cause) from financial time-series data (i.e. tickdata). Tick-by-tick financial time-series data is very messy. It ...
user avatar
2 votes
2 answers
181 views

Analysis of sells (what to buy)

I have a list of sold items by size. Shoes in this case ...
user avatar
61 votes
7 answers
35k views

Period detection of a generic time series

This post is the continuation of another post related to a generic method for outlier detection in time series. Basically, at this point I'm interested in a robust way to discover the periodicity/...
user avatar
  • 1,921
13 votes
1 answer
14k views

Two-sample T-test with weighted data

I want to perform a two-sample T-test to test for a difference between two independent samples which each sample abides by the assumptions of the T-test (each distribution can be assumed to be ...
user avatar
3 votes
4 answers
643 views

Ranking distributional data by similarity

The question in short: What methods can be used to quantify distributional relationships between data when the distribution is unknown? Now the longer story: I have a list of distributions and would ...
user avatar
7 votes
1 answer
420 views

How can I apply a Pareto tail to a truncated distribution?

Many income surveys (especially older ones) truncate key variables, such as household income, at some arbitrary point, to protect confidentiality. This point changes over time. This reduces inequality ...
user avatar
  • 71
74 votes
16 answers
39k views

Practical thoughts on explanatory vs. predictive modeling

Back in April, I attended a talk at the UMD Math Department Statistics group seminar series called "To Explain or To Predict?". The talk was given by Prof. Galit Shmueli who teaches at UMD's Smith ...
5 votes
2 answers
3k views

Introduction to index decomposition analysis

Could you recommend an introductory reference to index decomposition analysis, including different methods (e.g. methods linked to the Laspeyre index and methods linked to the Divisa index) ...
user avatar
48 votes
8 answers
64k views

How can I test if given samples are taken from a Poisson distribution?

I know of normality tests, but how do I test for "Poisson-ness"? I have sample of ~1000 non-negative integers, which I suspect are taken from a Poisson distribution, and I would like to test that.
user avatar
  • 1,191
15 votes
8 answers
4k views

Alternative graphics to "handle bar" plots

In my area of research, a popular way of displaying data is to use a combination of a bar chart with "handle-bars". For example, The "handle-bars" alternate between standard errors and standard ...
user avatar
  • 11.9k
5 votes
1 answer
890 views

CI for a difference based on independent CIs

I'm looking to check my logic here. Say you measure a quantity in group A, and find the mean is 2 and your 95% confidence interval ranges from 1 to 3. Then you measure the same quantity in group B ...
user avatar
86 votes
14 answers
7k views

Why haven't robust (and resistant) statistics replaced classical techniques?

When solving business problems using data, it's common that at least one key assumption that under-pins classical statistics is invalid. Most of the time, no one bothers to check those assumptions so ...
user avatar
  • 10k
2 votes
2 answers
464 views

What code would you put before/after your R session? [closed]

R allows us to put code to run in the beginning/end of a session. What codes would you suggest putting there? I know of three interesting examples (although I don't have "how to do them" under my ...
101 votes
9 answers
47k views

Is there an intuitive explanation why multicollinearity is a problem in linear regression?

The wiki discusses the problems that arise when multicollinearity is an issue in linear regression. The basic problem is multicollinearity results in unstable parameter estimates which makes it very ...
user avatar
104 votes
13 answers
80k views

Simple algorithm for online outlier detection of a generic time series

I am working with a large amount of time series. These time series are basically network measurements coming every 10 minutes, and some of them are periodic (i.e. the bandwidth), while some other aren'...
user avatar
  • 1,921
12 votes
4 answers
19k views

Multiple Chi-Squared Tests

I have cross classified data in a 2 x 2 x 6 table. Let's call the dimensions response, A and ...
user avatar
  • 2,267
4 votes
1 answer
2k views

Regression-multiple observations per subject

I have data for about 1 year, 100 observations, multiple observations per subject, transactions occur on weekly basis but have 6-12 subjects per week, there is no order to this. There is a policy ...
user avatar
  • 59
11 votes
10 answers
578 views

Reasons besides prediction to build models?

Joshua Epstein wrote a paper titled "Why Model?" available at http://www.santafe.edu/media/workingpapers/08-09-040.pdf in which gives 16 reasons: Explain (very distinct from predict) Guide data ...
2 votes
4 answers
425 views

Modeling success rate with gaussian distribution

In many papers I see data representing a rate of success (i.e a number between 0 and 1) modeled as a gaussian. This is clearly a sin (the range of variation of the gaussian is all of R), but how bad ...
user avatar
-1 votes
3 answers
4k views

Can I predict percentage change in earnings from percentage change in produced and percentage changed in price? [closed]

I have computed percentage change from time1 to time2 for several variables. Can I predict percentage change in earnings from percentage change in produced and percentage changed in price? When I ...
user avatar
  • 59
34 votes
10 answers
50k views

How to represent an unbounded variable as number between 0 and 1

I want to represent a variable as a number between 0 and 1. The variable is a non-negative integer with no inherent bound. I map 0 to 0 but what can I map to 1 or numbers between 0 and 1? I could use ...
user avatar
9 votes
6 answers
3k views

How to handle count data (categorical data), when it has been converted to a rate?

I am working on disease infection data, and I am puzzled on whether to handle the data as "categorical" or "continuous". "Infection Count" the number of infection cases found in a specific period ...
user avatar
  • 2,999
2 votes
1 answer
388 views

Tonga: low rank approximation of the natural gradient, question regarding Le Roux et al. 2007

This question concerns an implementation of the topmoumoute natural gradient (tonga) algorithm as described in page 5 in the paper Le Roux et al 2007 http://research.microsoft.com/pubs/64644/tonga....
user avatar
3 votes
2 answers
1k views

Density function for a multivariate Bernoulli-like distribution

I'm looking for a distribution to model a vector of $k$ binary random variables, $X_1, \ldots, X_k$. Suppose I have observed that $\sum_i X_i = n$. In this case I do not want to treat them as ...
user avatar
4 votes
1 answer
2k views

Equation to calculate a smooth line given an irregular time series?

I'm trying to visualize a set of data that represents human body mass over time, taken from (usually) daily weighings. Because body mass tends to fluctuate +/- 3 pounds based on hydration I would ...
user avatar
  • 43
5 votes
1 answer
2k views

How to define the significance threshold for mutual information in terms of probability of that value occurring in surrogate set?

I am trying to assess the significance of the obtained MI matrix. The initial input was a array of 3000 genes by 45 timepoints. MI was computed resulting in a array of 3600 by 3600. I am thus ...
user avatar
5 votes
1 answer
1k views

Help me understand nMDS algorithm

I have been reading Zuur, Ieno and Smith (2007) Analyzing ecological data, and on page 262, they try to explain how nMDS (non-metric multidimensional scaling) algorithm works. As my background is in ...
user avatar
1 vote
1 answer
473 views

Approximating density function for a non-normal distribution

My question is actually quite short, but I'll have to start by describing the context since I am not sure how to directly ask it. Consider the following "game": We have a segment of length n ("large ...
user avatar
  • 1,191
19 votes
3 answers
88k views

Why are mean 0 and standard deviation 1 distributions always used?

My stats has been self taught, but a lot of material I read point to a dataset having mean 0 and standard deviation of 1. If that is the case then: Why is mean 0 and SD 1 a nice property to have? ...
6 votes
2 answers
3k views

Orthogonal parametrization

In general inference, why orthogonal parameters are useful, and why is it worth trying to find a new parametrization that makes the parameters orthogonal ? I have seen some textbook examples, not so ...
user avatar
  • 539
7 votes
1 answer
1k views

Why does AIC formula in R appear to use one extra parameter than expected?

I'll use an example so that you can reproduce the results ...
user avatar
4 votes
2 answers
2k views

R command for stcox in Stata

What is the equivalent command in R for the stcox command in Stata?
user avatar
  • 2,569
44 votes
15 answers
10k views

References for survival analysis

I am looking for a good book/tutorial to learn about survival analysis. I am also interested in references on doing survival analysis in R.
15 votes
2 answers
12k views

What is the major difference between correlation and mutual information?

my question particularly applies to network reconstruction
user avatar
40 votes
3 answers
31k views

Is Kolmogorov-Smirnov test valid with discrete distributions?

I'm comparing a sample and checking whether it distributes as some, discrete, distribution. However, I'm not enterily sure that Kolmogorov-Smirnov applies. Wikipedia seems to imply it does not. If it ...
user avatar
  • 730
4 votes
1 answer
1k views

Is there a radical difference in how bayesian and frequentist approaches treat nuisance parameters?

The wiki article on credible intervals has the following statement: credible intervals and confidence intervals treat nuisance parameters in radically different ways. What is the radical ...
user avatar
4 votes
3 answers
1k views

What is the interpretation/meaning of confidence intervals in misspecified models?

Consider the following model $Y_i = f(X_i) + e_i$ from which we observe n iid data points $\left( X_i, Y_i \right)_{i=1}^n$. Suppose that $X_i \in \mathbb{R}^d$ is a $d$ dimensional feature vector. ...
user avatar
  • 457
14 votes
4 answers
9k views

How to interpret KL divergence quantitatively?

I am comparing two distributions with KL divergence which returns me a non-standardized number that, according to what I read about this measure, is the amount of information that is required to ...
user avatar
12 votes
2 answers
1k views

Where can I find good statistics quizzes?

Introductory, advanced, and even obscure, please. Mostly to test myself. I like to make sure I know what the heck I'm talking about :) Thanks
10 votes
5 answers
7k views

Is a variable significant in a linear regression model?

I've got a linear regression model with the sample and variable observations and I want to know: Whether a specific variable is significant enough to remain included in the model. Whether another ...
user avatar
  • 730
8 votes
2 answers
979 views

Reliability in Elicitation Exercise

I am trying to calculate the reliability in an elicitation exercise by analysing some test-retest questions given to the experts. The experts elicited a series of probability distributions which were ...
user avatar
  • 525
2 votes
2 answers
2k views

What would the calculated value of the standard deviation of a uniform distribution be?

A colleague wants to compare models that use either a Gaussian distribution or a uniform distribution and for other reasons needs the standard devation of these two distributions to be equal. In R I ...
user avatar

15 30 50 per page