All Questions

28
votes
7answers
11k views

Good sources for learning Markov chain Monte Carlo (MCMC)

Any suggestions for a good source to learn MCMC methods?
7
votes
1answer
2k views

What is the probability of picking p primes from n random numbers?

The probability of that a number between 1 to x is prime is $\frac{1}{\ln{x}}$ as per prime number theorem and also the total number of prime numbers between $1$ to $x$ will be $\frac{x}{\ln{x}}$. But ...
3
votes
2answers
429 views

How to best estimate the time remaining for a variable-length questionnaire?

The greatest gain of the statistics classes in both school and university seems to be that I now have an inkling of which QA site to use for this question. :) I'm a programmer and I'm making a ...
5
votes
1answer
915 views

Experiment design for proportion

I would like to test a hypothesis that a database contains data that are correct in 95% or more of the samples. I found this Hypothesis test for a proportion experiment design that looked appropriate....
3
votes
2answers
270 views

The range of non-identically distributed binomial variables

Lets say we have three independent variables: $$\eqalign{ X_{1}\sim &B(n,\frac{1}{2}+\beta) \cr X_{2}\sim &B(n,\frac{1}{2}) \cr X_{3}\sim &B(n,\frac{1}{2}-\beta). }$$ I'm looking for ...
3
votes
1answer
168 views

Basics of extreme values / high-water marks?

With real-valued $X_1, X_2, \ldots$, define $Max_n := \max(X_1,\ldots,X_n)$ record value or high-water mark $NextMax_n :=$ the next greater high water, $Max_{n+m} > Max_n$ $Up_n := NextMax_n - ...
18
votes
4answers
9k views

Survival analysis: continuous vs discrete time

I am confused about how to decide whether to treat time as continuous or discrete in survival analysis. Specifically, I want to use survival analysis to identify child- and household-level variables ...
19
votes
7answers
19k views

What are some alternatives to a boxplot?

I am working on creating a website, which displays the census data for a user selected Polygons & would like to graphically show the distribution of various parameters (one graph per parameter). ...
8
votes
2answers
15k views

Forcing a set of numbers to a gaussian bell-curve

(This relates to my programming question on Stack Overflow: Bell Curve Gaussian Algorithm (Python and/or C#).) On Answers.com, I found this simple example: Find the arithmetic mean (average) => Sum ...
4
votes
1answer
434 views

Order statistics: probability random variable is k-th out of n when ordered

Given a random variable $X_1$ drawn from a distribution with cdf $F$, and random variables $X_2, \cdots,X_n$ drawn from another distribution with cdf $G$, what is the formula for the probability that $...
3
votes
1answer
2k views

Statistical significance (chi squared) for percentages over 100%?

So, working in web analytics, it's common to use a chi-square test for A/B tests to test statistical significance. I often use a calculator like this: http://www.usereffect.com/split-test-calculator , ...
10
votes
2answers
732 views

Good text for resampling?

Can the group recommend a good introduction text/resource to applied resampling techniques? Specifically, I am interested in alternatives to classical parametric tests (e.g. t tests, ANOVA, ANCOVA) ...
3
votes
1answer
1k views

What is the connection between Kernel Logistic Regression and Smoothing Splines?

Working on probabilistic outputs of kernel methods I found the formulation of the SVM as a Penalized Method using the Binomial Deviance (described for example in "The Elements of Statistical Learning ...
6
votes
1answer
2k views

Cluster data points by distance between clusters

I have a load of point data in 3D, and I wish to cluster them like so: Each cluster contains points all of which are at most distance $d$ from another point in the cluster. All points in two ...
7
votes
3answers
1k views

Annotating graphs in R [closed]

This is more of a "how to use R" question than an actual hardcore statistics question, but I think the concentration of R masters here makes this a good forum for it. I'm refreshing a time series ...
2
votes
1answer
705 views

First passage time distribution in a irreducible transient discrete-time Markov chain (DTMC)

In a Markov chain, a state $j$ is transient if $f_{jj}<1$ ($f_{jj}$ is probability of ever visiting state $j$ starting from state $j$ ). Suppose, I have an irreducible transient DTMC (means all ...
16
votes
4answers
640 views

War stories where wrong decisions were made based on statistical information?

I think it is fair to say statistics is an applied science so when averages and standard deviations are calculated it is because someone is looking to make some decisions based on those numbers. ...
5
votes
3answers
2k views

How do I calculate and compare incidence of infection between age-groups

We work on cervical cancer detection and we started accumulating data in 2006 by performing HPV testing on all our samples. We have accumulated a data set with almost 400,000 women. Data enters the ...
10
votes
2answers
9k views

Knot selection for cubic regression splines [duplicate]

I was wondering if anybody had experience in how to set the knot points when using cubic regression splines. Some background: I have a response and predictor variable, and I want to determine the ...
4
votes
1answer
4k views

Kernel density estimate takes values larger than 1 [duplicate]

I am using scipy.stats.gaussian_kde to estimate a pdf for some data. The problem is that the resulting pdf takes values larger than 1. As far as I understand, this ...
2
votes
1answer
609 views

What is my model statement for mixed models (nlme) in R?

I want to know if a covariate for each subject interacts with three types of trials, and the difficulty of those trials. My dependent measures are accuracy and response times (RT). For this question, ...
7
votes
0answers
314 views

Help in setting up and solving a transportation / traffic problem [closed]

Traffic light synchronization nowadays is not a tedious project, Image: For a personal research project I'm trying to build a statistical model to solve such problems in two and higher dimensions. ...
1
vote
1answer
194 views

Demo for bioinformatics

I am a beginner level programmer preparing for the interview in medical research company. Job sounds damn interesting and I would like to get there. To show my skills and interest, I want to write a ...
7
votes
2answers
6k views

Checking factor / covariate independence in ANCOVA

One of the ANCOVA assumptions, as I read it here, says that: Independent variables orthogonal to covariates. In traditional ANCOVA, the independents area assumed to be orthogonal to the factors. If ...
6
votes
1answer
1k views

Leave-one-out cross validation and boosted regression trees

Colleagues of mine recently presented a work where they calibrate boosted regression trees (BRT) models on small data sets ($n= 30$). They validated the models using leave-one-out cross validation (...
1
vote
2answers
237 views

Is it possible to predict the likelihood of an order of random events?

Sorry if this is a n00b question, I'm just trying to wrap my head around the problem. I am trying to refute conventional wisdom here, so any help is greatly appreciated. And now for the question: ...
6
votes
2answers
350 views

Is there a bias correction for effect size in a data mining context?

Given $K$ possible 'treatments' of some kind, and independent observations of some response under those treatments, say $X_{i,k}$ for $i=1,\ldots,n_k$ and $k=1,\ldots,K$, I am faced with the classical ...
5
votes
1answer
6k views

R-square from rpart model

How can I extract the R-square from a fit rpart model? rsq.rpart(fit) plots the two graphs, but I simply want to extract the R-square value for the full tree. ...
7
votes
2answers
2k views

Alternatives to using Coefficient of Variation to summarize a set of parameter distributions?

Background I have a model with 17 parameters, and I currently use the coefficient of variation ($\text{CV}=\sigma/\mu$) to summarize the prior and posterior distributions of each parameter. All of ...
6
votes
1answer
3k views

How to tell if the slope of a line is 0 or there is just no relationship?

I am attempting to examine the change in slope between a predictor and response over two years. In year 1, it is definitely positive. (Linear regression, the 95% CI of the slope doesn't overlap 0). ...
10
votes
2answers
2k views

Which kernel method gives the best probability outputs?

Recently I have used Platt's scaling of SVM-outputs to estimate probabilities of default-events. More direct alternatives seem to be "Kernel logistic Regression" (KLR) and the related "Import Vector ...
31
votes
2answers
32k views

Variance of a function of one random variable

Lets say we have random variable $X$ with known variance and mean. The question is: what is the variance of $f(X)$ for some given function f. The only general method that I'm aware of is the delta ...
2
votes
3answers
601 views

Odds of winning 2 trials in a row

There are 8 people playing poker. So, the odds of winning the entire round = 1/8 2 rounds are played, and Bill wins both rounds. What are the odds this was random? (Hypothesis test?) NullH = Bill ...
140
votes
6answers
134k views

Can principal component analysis be applied to datasets containing a mix of continuous and categorical variables?

I have a dataset that has both continuous and categorical data. I am analyzing by using PCA and am wondering if it is fine to include the categorical variables as a part of the analysis. My ...
4
votes
1answer
836 views

Multiple comparisons on nested subsets of data

Suppose I am doing some experimental procedure on two treatment groups. The procedure has several stages, each of which may fail. Failure at any stage halts the experiment. If all stages are passed ...
3
votes
1answer
1k views

Can I test the effect of Block in a split plot design in R?

I have been confused with the problem for a very long time, and hope that somebody here can help me out. I have a experiment installed in split-plot design, with 3 temporal groups as blocks, and 2 ...
2
votes
2answers
1k views

Measure of closeness

Given a list of numbers, is it possible to find out (or in other words, is there a statistical measure to tells the) the closeness of the numbers (do note that i am not talking about correlation - ...
2
votes
3answers
483 views

What logistic regression is best to use?

My project is about whether early language scores (on a reading task) can predict later scores on a social attribution task. I have to conduct a logistic regression for my project but I am stuck on ...
6
votes
1answer
2k views

When is a deviation statistically significant?

In a double-blind study, when are deviations from the control group considered statistically significant? And is this related to the number of samples? I realise that every experiment is different ...
65
votes
4answers
20k views

Look and you shall find (a correlation)

I have several hundred measurements. Now, I am considering utilizing some kind of software to correlate every measure with every measure. This means that there are thousands of correlations. Among ...
3
votes
1answer
159 views

When is it better to average observations at the same abscissa?

If you want to regress y on x, where multiple y's are observed at each x, is it ever better to instead take the mean at each x, and the use those means for the regression? Does it depend on the ...
61
votes
8answers
22k views

If A and B are correlated with C, why are A and B not necessarily correlated?

I know empirically that is the case. I have just developed models that run into this conundrum. I also suspect it is not necessarily a yes/no answer. I mean by that if both A and B are correlated ...
4
votes
2answers
636 views

Optimal stopping from an unknown distribution

The Secretary problem has an algorithm for fixed N and immediate accept/reject (that is, reject reject ... accept one, stop). There are several variants; in mine, secretaries or samples come from a ...
3
votes
1answer
258 views
2
votes
4answers
5k views

Is there any library like LOESS or ARIMA in Java/C# or Python?

I try to implement a time series data analysis project, but I have to do in Java, C# or Python, is there any good libary such like LOESS, ARIMA in R you can recommend? Many thanks
10
votes
5answers
4k views

Generate random multivariate values from empirical data

I'm working on a Monte Carlo function for valuing several assets with partially correlated returns. Currently, I just generate a covariance matrix and feed to the the ...
6
votes
1answer
868 views

Why “supremum”, not “maximum” in Kolmogorov-Smirnov test?

Why the test function in Kolmogorov-Smirnov test takes the supremum of the set of differences, not maximum? When maximum (greatest element) of a set does not exist then supremum may exist, but if ...
5
votes
4answers
9k views

Reducing false positive rate

I am running a model for a problem in insurance domain. The final results show some false positive x and some false negative y. I am using SAS Enterprise Miner for this. Can somebody suggest me how to ...
17
votes
2answers
5k views

James-Stein estimator: How did Efron and Morris calculate $\sigma^2$ in shrinkage factor for their baseball example?

I have a question on calculating James-Stein Shrinkage factor in the 1977 Scientific American paper by Bradley Efron and Carl Morris, "Stein's Paradox in Statistics". I gathered the data for the ...
1
vote
1answer
195 views

Multiplicative unobservable component in state space model

I'm new here and wondering if anyone could give me some hints on how to estimate the time varying coefficient and state variable together. Here is my model: observation equation: $Y(t)= A(t)X(t)+ w(t)...

15 30 50 per page