All Questions

Filter by
Sorted by
Tagged with
55
votes
19answers
9k views

Mathematical Statistics Videos

A question previously sought recommendations for textbooks on mathematical statistics Does anyone know of any good online video lectures on mathematical statistics? The closest that I've found are: ...
10
votes
3answers
5k views

Is it valid to aggregate a time series to make it look more meaningful?

Another question about time series from me. I have a dataset which gives daily records of violent incidents in a psychiatric hospital over three years. With the help from my previous question I have ...
20
votes
2answers
4k views

How does random forest generate the random forest

I am not an expert of random forest but I clearly understand that the key issue with random forest is the (random) tree generation. Can you explain me how the trees are generated? (i.e. What is the ...
9
votes
4answers
960 views

Good line color for "threshold" line in a time-series graph?

We're plotting time-series metrics in the context of network/server operations. The data has a 5-minute sample rate, and consists of things like CPU utilization, error rate, etc. We're adding a ...
24
votes
6answers
61k views

Always Report Robust (White) Standard Errors?

It has been suggested by Angrist and Pischke that Robust (i.e. robust to heteroskedasticity or unequal variances) Standard Errors are reported as a matter of course rather than testing for it. Two ...
370
votes
80answers
173k views

What is your favorite "data analysis" cartoon?

Data analysis cartoons can be useful for many reasons: they help communicate; they show that quantitative people have a sense of humor too; they can instigate good teaching moments; and they can help ...
103
votes
32answers
36k views

What book would you recommend for non-statistician scientists?

What book would you recommend for scientists who are not statisticians? Clear delivery is most appreciated. As well as the explanation of the appropriate techniques and methods for typical tasks: ...
30
votes
4answers
9k views

Why isn't RANSAC most widely used in statistics?

Coming from the field of computer vision, I've often used the RANSAC (Random Sample Consensus) method for fitting models to data with lots of outliers. However, I've never seen it used by ...
57
votes
6answers
19k views

Introduction to statistics for mathematicians

What is a good introduction to statistics for a mathematician who is already well-versed in probability? I have two distinct motivations for asking, which may well lead to different suggestions: I'd ...
49
votes
6answers
13k views

Motivation for Kolmogorov distance between distributions

There are many ways to measure how similar two probability distributions are. Among methods which are popular (in different circles) are: the Kolmogorov distance: the sup-distance between the ...
3
votes
2answers
442 views

How to approximate measurement uncertainty?

At the moment I use standard deviation of the mean to estimate uncertainty: where N is in hundreds and mean is a time series (monthly) mean. I present it then like this: for each element (month) in ...
41
votes
15answers
6k views

What best practices should I follow when preparing plots?

I usually make my own idiosyncratic choices when preparing plots. However, I wonder if there are any best practices for generating plots. Note: Rob's comment to an answer to this question is very ...
5
votes
2answers
223 views

How to tell if something happened in a data set which monitors a value over time

I have a data set where a series of measurements are being taken each week. In general the data set shows a +/- 1mm change each week with a mean measurement staying at about 0mm. In plotting the data ...
7
votes
0answers
479 views

Testing (and proving) the randomness of numbers [duplicate]

Possible Duplicate: Testing random variate generation algorithms What's a good way to test a series of numbers to see if they're random (or at least psuedo-random)? Is there a good statistical ...
8
votes
1answer
2k views

What are pivot tables, and how can they be helpful in analyzing data?

What are pivot tables, and how can they be helpful in analyzing data?
42
votes
15answers
6k views

The Monty Hall Problem - where does our intuition fail us?

From Wikipedia : Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who ...
3
votes
2answers
2k views

What are the key statistical concepts that relate to data mining?

What topics in statistics are most useful/relevant to data mining?
6
votes
4answers
886 views

Working through a clustering problem

Say I've got a program that monitors a news feed and as I'm monitoring it I'd like to discover when a bunch of stories come out with a particular keyword in the title. Ideally I want to know when ...
87
votes
26answers
16k views

What is the single most influential book every statistician should read?

If you could go back in time and tell yourself to read a specific book at the beginning of your career as a statistician, which book would it be?
36
votes
2answers
127k views

What is the difference between the Shapiro-Wilk test of normality and the Kolmogorov-Smirnov test of normality?

What is the difference between the Shapiro-Wilk test of normality and the Kolmogorov-Smirnov test of normality? When will results from these two methods differ?
11
votes
2answers
1k views

The trinity of tests in maximum likelihood: what to do when faced with contradicting conclusions?

The Wald, Likelihood Ratio and Lagrange Multiplier tests in the context of maximum likelihood estimation are asymptotically equivalent. However, for small samples, they tend to diverge quite a bit, ...
14
votes
5answers
962 views

Bias towards natural numbers in the case of least squares

Why do we seek to minimize x^2 instead of minimizing |x|^1.95 or |x|^2.05. Are there reasons ...
53
votes
10answers
28k views

What is a good algorithm for estimating the median of a huge read-once data set? [duplicate]

I'm looking for a good algorithm (meaning minimal computation, minimal storage requirements) to estimate the median of a data set that is too large to store, such that each value can only be read once ...
8
votes
4answers
18k views

kNN and unbalanced classes

Do you think that unbalanced classes is a big problem for k-nearest neighbor? If so, do you know any smart way to handle this?
8
votes
9answers
4k views

Resources for learning about the Statistical Analysis of Financial Data

I realize that the statistical analysis of financial data is a huge topic, but that is exactly why it is necessary for me to ask my question as I try to break into the world of financial analysis. As ...
19
votes
6answers
4k views

Good introduction into different kinds of entropy

I'm looking for a book or online resource that explains different kinds of entropy such as Sample Entropy and Shannon Entropy and their advantages and disadvantages. Can someone point me in the right ...
15
votes
1answer
4k views

How does gentle boosting differ from AdaBoost?

There is a variant of boosting called gentleboost. How does gentle boosting differ from the better-known AdaBoost?
5
votes
1answer
150 views

What approach could be used for modelling virus infections?

I'm a physics graduate who ended up doing infosec so most of the statistics I ever learned is useful for thermodynamics. I'm currently trying to think of a model for working out how many of a ...
50
votes
7answers
34k views

When conducting a t-test why would one prefer to assume (or test for) equal variances rather than always use a Welch approximation of the df?

It seems like when the assumption of homogeneity of variance is met that the results from a Welch adjusted t-test and a standard t-test are approximately the same. Why not simply always use the Welch ...
196
votes
8answers
425k views

In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?

Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
9
votes
8answers
2k views

Resources for learning Stata

I know of Cameron and Trivedi's Microeconometrics Using Stata. What are other good texts for learning Stata?
7
votes
3answers
3k views

Estimating beta-binomial distribution

Suppose that I culture cancer cells in n different dishes g₁, g₂, … , gn and observe the number of cells ni in each dish that look different than normal. The total number of cells in dish gi is ti. ...
15
votes
1answer
5k views

What is the difference/relationship between method of moments and GMM?

Can someone explain to me the difference between method of moments and GMM (general method of moments), their relationship, and when should one or the other be used?
69
votes
7answers
86k views

What is a "saturated" model?

What is meant when we say we have a saturated model?
8
votes
3answers
272 views

How to deal with the effect of the order of observations in a non hierarchical cluster analysis?

When a non-hierarchical cluster analysis is carried out, the order of observations in the data file determine the clustering results, especially if the data set is small (i.e, 5000 observations). To ...
27
votes
2answers
13k views

Spatial statistics models: CAR vs SAR

When would one prefer to use a Conditional Autoregressive model over a Simultaneous Autoregressive model when modelling autocorrelated geo-referenced aerial data?
12
votes
4answers
10k views

How large should a sample be for a given estimation technique and parameters?

Is there a rule-of thumb or even any way at all to tell how large a sample should be in order to estimate a model with a given number of parameters? So, for example, if I want to estimate a least-...
17
votes
5answers
7k views

Poisson regression with large data: is it wrong to change the unit of measurement?

Due to the factorial in a poisson distribution, it becomes unpractical to estimate poisson models (for example, using maximum likelihood) when the observations are large. So, for example, if I am ...
45
votes
5answers
406k views

What is the difference between a population and a sample?

What is the difference between a population and a sample? What common variables and statistics are used for each one, and how do those relate to each other?
9
votes
2answers
9k views

How do I calculate if the degree of overlap between two lists is significant?

If I have two lists A and B, both of which are subsets of a much larger list C, how can I determine if the degree of overlap of A and B is greater than I would expect by chance? Should I just ...
15
votes
10answers
3k views

What is the best out-of-the-box 2-class classifier for your application? [closed]

Rules: one classifier per answer vote up if you agree downvote/remove duplicates. put your application in the comment
17
votes
7answers
11k views

What is the easiest way to create publication-quality plots under Linux?

We may assume that we have CSV file and we want a very basic line plot with several lines on one plot and a simple legend.
23
votes
2answers
9k views

How does boosting work?

What is the easiest way to understand boosting? Why doesn't it boost very weak classifiers "to infinity" (perfection)?
4
votes
1answer
727 views

Variance components

I have a set of $N$ bodies, which is a random sample from a population whose mean and variance I want to estimate. A property of each body is being measured $m_i$ times ($m_i>1$) and different for ...
13
votes
2answers
884 views

Using time series analysis to analyze/predict violent behavior

This is a bit of a flippant question, but I have a serious interest in the answer. I work in a psychiatric hospital and I have three years' of data, collected every day across each ward regarding the ...
8
votes
4answers
791 views

Why is the average of the highest value from 100 draws from a normal distribution different from the 98th percentile of the normal distribution?

Why is the average of the highest value from 100 draws from a normal distribution different from the 98% percentile of the normal distribution? It seems that by definition that they should be the ...
9
votes
10answers
2k views

Recommended visualization libraries for standalone applications

Which visualization libraries (plots, graphs, ...) would you suggest to use in a standalone application (Linux, .Net, Windows, whatever). Reasonable performance would be nice as well.
6
votes
3answers
361 views

Intro to statistics for an MD?

I have a friend who is an MD and wants to refresh his Statistics. So is there any recommended resource online (or offline) ? He did stats ~20 years ago.
86
votes
7answers
245k views

What are principal component scores?

What are principal component scores (PC scores, PCA scores)?
69
votes
3answers
80k views

How is the minimum of a set of IID random variables distributed?

If $X_1, ..., X_n$ are independent identically-distributed random variables, what can be said about the distribution of $\min(X_1, ..., X_n)$ in general?

15 30 50 per page