At first, what is important is what your whiskers mean - https://en.wikipedia.org/wiki/Box_plot - it can be standard deviation (I would say no), or Tukey boxplot (I'd say this is what you have), or something else - check this out in the manual of your statistical toolbox. At second, these boxplots may be a bit "misleading" in some sense. These "outliers" may ...
The patients at risk in a survival analysis must start time at risk at entry into the risk set. For a time-dependent covariate, time zero is the time when the at risk unit transfers from the initial to the new status. Thus time at risk in this new state begins at zero and everyone transferring is alive by definition and survival = 1.0 at time zero. The time ...
As advised in the comments (+1), since y-values are very small, you can't observe the asymptotical behaviors of the curves with this scale (at least clearly). Just add plt.yscale('log') to your plotting lines. It seems you're using Python matplotlib. Below is an example:
import numpy as np
import matplotlib.pyplot as plt
plt.figure(figsize = (18,6))
n = ...
I have read of your linked meta post, and it is an interesting situation. If I understand the issue correctly, there has been a proposal to create a tag specific to this particular controversy (such that all posts pertaining to the controversy would be easily identifiable together), and that proposal has been rejected. Instead, you have identified a number ...
Described here. Digits do not appear with uniform frequency in front of numbers, but rather follow a specific pattern: digit 1 is the most likely to be the first digit, with 30% chance, followed by 2 (17.6% chance), and so on. The following picture (from Wikipedia) shows the frequency of each digit at the beginning of each number, in some ...
The failure to show the association between launch temperature, and the effect of launch temperature, on the space shuttle o-rings, leading to the catastrophic failure of the Columbia soon after launch. An overview of the problem is here.
The hexbin package: my quick and painless go-to for visualizing overplotted data sets.
x.axis <- c(rnorm(2000), rnorm(2000, 4, 2))
y.axis <- c(rnorm(2000), rnorm(2000, 2, 3))
point.map <- cbind(x.axis, y.axis)
# Square plot region
par(pty = "s")
# Standard R plot
# Convert coordinate data into a ...
I don't know if this counts as "intuition falls short", but rather a "naive analysis gives a counter intuitive, and misleading, answer".
One of my stats professors introduced a study regarding the the connection between smoking and FEV in young students.
FEV can be considered to be a measure of lung volume. When the professor first introduced the data, ...
My favourite example, as an illustration of how faulty statistics can have long-term consequences when they are used to direct government policy, is the act of large-scale railway vandalism known as the Beeching Axe. It resulted from a Transport Minister with strong ties to the road-building industry (Ernest Marples) hiring a petrochemicals-industry expert (...
I find the false positive paradox remarkable because it is so counter-intuitive. A good example:
Cancer screening of the general population does not increase life expectancy, even though clearly lives are saved because some cancers are caught early and can be treated better. The U.S. Preventive Services Task Force accordingly stopped recommending routine ...
Nice QA! here my two cents: It is mainly about how correlation can be very suspicious and some traditional ways to work it out:
To elaborate a little, the canon for correlation vs. ...
Another interesting case as to how wrong gambling can go is the Monte Carlo Casino example.
In a game of roulette at the Monte Carlo Casino on August 18, 1913 the ball fell in black 26 times in a row. This was an extremely uncommon occurrence: the probability of a sequence of either red or black occurring 26 times in a row is around 1 in 66.6 million, ...
I really liked the German tank problem. It shows, how data which is usually considered as irrelevant becomes valuable information in the hand of a statistician. Furthermore, I liked the law of small numbers and the base rate fallacy.
R vs Sally Clark is a famous case of a woman being convicted for murder because the court was unaware of statistics and probability base principles.
But if I have to say the thing that impressed me the most, when I begun studying statistics, that is regression to the mean, which also gave the name to statistical regression (even if that is a completely ...
Modelling the data: You have multivariate count data over time, so you are going to need to look up models that are appropriate for that kind of data. There are a number of different classes of models that have been applied to this kind of data. One commonly used class of models are multivariate extensions to standard count models, such as the negative-...
I'm sorry that this probably isn't very encouraging, but if you've only got 7 datapoints and you want to perform a statistical test on the relationships between 5 independent variables and 7 dependent variables, your results probably won't be significant. You might have better luck simply checking the correlation between the total mosquito count and the ...
One way to better visualize bivariate relationships is shown below. I will use iris data an example
my_data <- iris[, c(1,3,4)] # note 1,3,4 refer to variables in your dataset
chart.Correlation(my_data, histogram=TRUE, pch=19)
As depicted on the plot:
On the lower triangle, ...
Often when there is a lot of data scatterplots simply look too bunched up like this. I would recommend another visualization, perhaps overlaid histograms or overlaid density plots.
To make the overlaid histograms you could use
and the overlaid density plots could be made using the "sm" package.
First, the data you are using is an hourly time series covering one complete year (2013). So in your plot there is a lot of overplotting, which hides structure. One way of avoiding it is
Code for this plot:
ggplot(weather, aes(x=wind_speed, y=temp)) + geom_hex() +xlim(0, 50) +xlab("wind_speed (mph)")+ylab("temp (F)")
But this is still ...
I can imagine examples where showing 3 to 5 distinct values in a histogram is perfectly reasonable. The fact that the histogram may well just show a few distinct spikes is not an objection in principle. (You don't have to show only a few bins just because of advice linking binning to sample size.) Easy examples arise with tossing coins, dealing cards, etc. ...
import matplotlib.pyplot as plt //import
def linear_regression(x, y, m_current=0, b_current=0, epochs=1000, learning_rate=0.0001):
cost_list =  // list to store the cost in each iteration
N = float(len(y))
for i in range(epochs):
y_current = (m_current * x) + b_current
cost = sum([data**2 for data in (y-y_current)]) / N
A sometimes useful alternative to the confidence band is to show the estimated line together with bootstrapped lines. Here is a simple, simulated example:
Bootstrapping regression can be done in different ways, for instance:
Resampling (with replacement) from the rows of the design matrix. This is what I have done here. A robust option, is valid even ...
Graphical comment: Here is a session in R, which takes random samples of size 500 from
uniform, gamma, exponential, and normal distributions.
Descriptive statistics are shown for each sample, with very brief
comments about the sample statistics. Histograms of all four samples
at the end. (Although $n = 500$ is a relatively large sample size, not
even a ...
In your study of subjects deciding whether or not to believe
20 videos are factual, it seems easy to get a reasonable confidence
interval. Roughly speaking, suppose you have $100$ subjects
who viewed the $20$ videos, and that on average 25% of the videos
Then you may have data similar to the 100 in the list x below (simulated using R). ...