All Questions

1
vote
2answers
4k views

Is it possible to use the caret package in R with non-numerical data?

I was wondering if it is possible to use the caret package with non numerical data. I know, for example, if I want to use a simple linear regression lm I could have ...
7
votes
1answer
4k views

Fit a robust regression line using an MM-estimator in R

Context. I'd like to fit a regression line to study to relation between some response variable $y$ and some continuous covariate $x$. Because of the presence of bad leverage points, I have opted for ...
1
vote
1answer
491 views

Optimal data structure for 2-way repeated measures ANOVA

This must be a very basic question, but I'm trying to do a 2-way repeated measures ANOVA (in Statistica and/or SPSS), and was wondering: in what way to (re)structure my data so it is most flexible ...
5
votes
2answers
2k views

How to compare two groups when there are many outcome variables?

Suppose we have two patient groups, healthy and ill patients, and we measure the chemical concentrations of multiple chemicals. Now we want to determine which chemical concentrations are different for ...
4
votes
2answers
28k views

What is the distribution of the sum of independent normal variables?

Just need to check the answer for the following question: Question Suppose $X$ and $Y$ are two independent standard normal variables: $X$ ~ $N(0,1)$ $Y$ ~ $N(0,1)$ What is the distribution of $X + ...
3
votes
3answers
493 views

Is goodness of fit needed for regression models when interpreted causally?

I'm investigating associations between socioeconomic factors and dichotomous outcome. I use generalised linear models (GLM) with log link for Bernoulli family, i.e., modelling the prevalence ratio. At ...
1
vote
2answers
132 views

What quantity can be used to robustly estimate a specific (dimensionless) variation (variance) of a sequence?

What quantity can be used to robustly estimate a specific variation of a sequence? By "specific" I mean "dimensionless, per-unit" etc, that is, independent on units of measurement and independent on ...
1
vote
0answers
382 views

Level change and trend change in the interrupted time series regression, does collinearity matters?

In the interrupted time series regression, we are interested in the level change or trend change due to a policy change or an intervention event occurance. If the dates of those interventions are ...
6
votes
2answers
57k views

How to find P(X > F) in ANOVA F-test?

(Apologies for the ASCII tables, Stackexchange doesn't allow HTML tables and since I'm not supposed to link to an image, this is the only way I know of showing the data) I'm learning about ANOVA F-...
12
votes
5answers
6k views

lme4 or other open source R package code equivalent to asreml-R

I want to fit mixed model using lme4, nlme, baysian regression package or any available. Mixed model in Asreml- R coding conventions before going into specifics, we might want to have details on ...
5
votes
2answers
850 views

What causes loss of power in hypothesis testing?

I am just wondering what kind of things could cause power to be lost in general (the opposite of reducing the noise of the data to increase the power of the test). Could anyone give some summary or ...
2
votes
1answer
607 views

Functions for regression diagnostics on mer objects in R

I'm particularly interested in plotting residuals against fitted values, and residuals against predictors. Often times I need to make boxplots of the residuals conditional on predictors. I'd be ...
3
votes
0answers
141 views

Estimating parameters in a model with a periodic design parameter

We recently studied a model with a likelihood function of the form $$\Pr(d_i|\omega,t_i)=(1-d_i)+(2d_i-1)\cos^2(\omega t_i),$$ where $d_i\in\{0,1\}$, $\omega$ is an estimation parameter, and $t_i$ ...
50
votes
2answers
30k views

Derivation of closed form lasso solution

For the lasso problem $\min_\beta (Y-X\beta)^T(Y-X\beta)$ such that $\|\beta\|_1 \leq t$. I often see the soft-thresholding result $$ \beta_j^{\text{lasso}}= \mathrm{sgn}(\beta^{\text{LS}}_j)(|\beta_j^...
2
votes
0answers
399 views

What is the probability of a team winning many golf tournaments when conducted with handicaps (i.e., has sandbagging occurred)?

For non golfers, each golfer has a handicap based on his scoring history. The handicap predicts, on average, how many shots above par that golfer will take to play 18 holes. In theory, when that ...
44
votes
7answers
42k views

What is the difference between estimation and prediction?

For example, I have historical loss data and I am calculating extreme quantiles (Value-at-Risk or Probable Maximum Loss). The results obtained is for estimating the loss or predicting them? Where can ...
10
votes
3answers
2k views

How to cluster longitudinal variables?

I have a bunch of variables which contain longitudinal data from day 0 to day 7. I am looking for an appropriate clustering approach which can cluster these longitudinal variables (not cases) into ...
0
votes
2answers
640 views

How to demonstrate failure of CLT in R?

I was assigned to demonstrate the Central Limit Theorem (CLT) in R in my statistics class. I already made some progress with simulation using simple.sim in R. I want to prepare 3 examples of the CLT ...
1
vote
0answers
329 views

Maximization of overall hit rate

I'm looking for some pointers with an optimization problem, specifically how to have my data formatted If I want to use R/Excel/Octave for optimization (those 3 are the only applications I have access ...
8
votes
1answer
1k views

Consistency of the learning process

I have two questions related to the concept of "learning consistency" for those who are familiar with statistical learning theory a la Vapnik. Question 1. The learning process is called consistent (...
3
votes
0answers
210 views

Does cross validation work with asymmetric loss functions?

Does cross validation work with an asymmetric loss function?
9
votes
1answer
4k views

Dashed lines in ACF plot in R

I'm going through the book 'Introductory Time Series with R' by Cowpertwait and Metcalfe. On page 36 Its says the lines are at: $-1/n \pm 2/\sqrt{n}$. I've read here R forum that the lines are at $\...
4
votes
0answers
2k views

Why would concatenating feature vectors lead to better estimates?

I wish to estimate the state of a system from two separate and disparate observations. A simple approach that I have seen in some literature is to combine the feature vectors (observations) by simply ...
6
votes
1answer
1k views

What are the sampling distributions of higher moments of the normal distribution?

Let $X_i$ be independent, normally distributed random variables, for $1\leq i\leq N$. What is the distribution of $Y_m=\frac1 N \sum_{i=1}^N X_i^m$? Every high school student knows part of the ...
1
vote
1answer
708 views

p value correlation for random vectors in R

I want to calculate correlation coefficient between random vectors and use this for hypothesis testing. The code ...
0
votes
1answer
189 views

What is the correlation of [1,2,3] and [1,5,7] to 8 decimal digits?

This stackoverflow post describes computing a Pearson correlation of [1,2,3] and [1,5,7] in several different ways in Python. The most straightforward implementation from the definition in Wikipedia ...
9
votes
1answer
242 views

How to compute $\mathbb P( 3 X_{(1)} \geq X_{(2)}+X_{(3)})$ for order statistics of a uniform distribution?

I am trying to solve a problem for my thesis and I don't see how to do it. I have 4 observations randomly taken from a uniform $(0,1)$ distribution. I want to compute the probability that $3 X_{(1)}\...
2
votes
4answers
24k views

Can the percentages in a pie chart add up to more than $100\%$ and the data or analyses will still make sense (or be logical)?

Can the percentages in a pie chart add up to more than $100\%$ and the data or analyses will still make sense (or be logical)? Let's say the percentage in a pie chart adds up to $100\%$ . Does this ...
6
votes
4answers
344 views

Visualize sets and their connections

I collected a data-sets which tested for many users (>100.000) which out of ten features in a software-product they use. They can use multiple features but for each feature there is only "use" or "don'...
8
votes
1answer
362 views

Standard error of sample standard deviation of proportions

I recently started reading Gelman and Hill's, "Data Analysis Using Regression and Multilevel/Hierarchical Models" and the question is based on that: The sample contains 6 observations on proportions: ...
2
votes
3answers
619 views

Sidak correction

For variables A (product research) and B (product purchase), I am proposing to do three levels of analysis as follows: Test 1: Overall: The goal here is to see if there is a relationship between A ...
3
votes
3answers
15k views

Writing a Monte Carlo simulation in R

I am trying to write a Monte Carlo simulation in R and I am really stuck! I want to know the probability distribution of a random person in the UK becoming ill from eating a cooked 100g piece of ...
2
votes
2answers
100 views

How to get continuous surface from collection of data points?

I have a collection of datapoints in three dimensions x,y,z. I'd like to be able to visualize the data as a smooth surface. Stated another way: Given those points, I'd like to be able to determine z ...
8
votes
1answer
1k views

How should one control for group and individual differences in pre-treatment scores in a randomised controlled trial?

Andrew Gelman, in the book he wrote with Jennifer Hill, states in Chapter 9, (section 9.3), on page 177: It is only appropriate to control for pre-treatment predictors, or, more generally, ...
4
votes
1answer
614 views

Feature selection with k-fold cross-validated least angle regression

I am using the least angle regression (LARS) to extract the most important predictors ($x_1, x_2,...,x_p$) for my response variable ($y$). I have seven predictors ($x_1,x_2,...,x_7$) for each ...
2
votes
0answers
345 views

Is it valid to model discrete numerical test scores as coming from a continuous random variable?

I'm working with a sample of test scores which range from 0 to 100. These scores are generated from a set of 100 binary responses (0 or 1), so the higher the resulting sum, the better the performance. ...
6
votes
2answers
588 views

Determining how well given real-life data fits to a given probability distribution

Let's say I have some data, i.e. a collection of real numbers. Are there any good general methods to determine whether this data conforms to a given probability distribution, e.g. the normal ...
6
votes
2answers
1k views

Parallel solving Ax=b?

Cross posted on StackOverflow. I have some extremely large sparse matrices created using spMatrix function from the matrix package. Using the solve() function works for my Ax=b issue, but it takes a ...
17
votes
2answers
11k views

Why does ridge regression classifier work quite well for text classification?

During an experiment for text classification, I found ridge classifier generating results that constantly top the tests among those classifiers that are more commonly mentioned and applied for text ...
3
votes
2answers
1k views

Weighted mean of two 2D Gaussian random variables

I have two 2D Gaussian random variables. I'd like to find the weighted average of the two (based on their covariance matrix around the mean, meaning that the mean of the final Gaussian should be ...
3
votes
2answers
181 views

Optimal basis for regression problem

Consider the training set $\{(x_i; y_i)\}_{i=1}^N, x_i \in \mathbb{R}^n, y_i \in \mathbb{R}$. The goal is to find regression function, like, $f(x) = \sum_{i=1}^K a_i g_i(x) + a_0$. The least-squares ...
17
votes
1answer
44k views

How to show that an estimator is consistent?

Is it enough to show that MSE = 0 as $n\rightarrow\infty$? I also read in my notes something about plim. How do I find plim and use it to show that the estimator is consistent?
2
votes
2answers
159 views

How to calculate correlation between consumers according their multi-aspect and overall rating?

In e-commerce site, consumers would provide a overall rating with multi-aspect ratings. For example, if a consumer purchase a camera, he would provide a rating info as follows: ...
-2
votes
2answers
160 views

How to find the expectation of $X_1X_2$ given $X_1$ and $X_2$ are i.i.d. normal? [closed]

$X_1, X_2 \sim \text{i.i.d.} N(\mu, \sigma^2/n)$. Any ideas?
5
votes
1answer
844 views

Computing confidence intervals for count data

I have a representative sample token on area of 60 mq (squared meter) of a territory of 54077 mq. This sample contains the number of little plants that there are for each mq. The sample is defined in ...
13
votes
1answer
10k views

Significance test on the difference of Spearman's correlation coefficient

(Thanks a lot for the quick responses! I did a poor job of asking the question, so let me retry.) I do not know how to find out whether or not the difference between two Spearman's correlations is ...
1
vote
1answer
156 views

How would you analyse this problem?

Following the confusion (on my part) here, and as suggested, I am presenting my question in a neutral way below: Problem: I am looking at the relationship between "product research" (variable A) and ...
5
votes
0answers
820 views

Effect size measure for doubly-repeated measures design?

Update: I haven't gotten any answers, so I am simplifying my question. Say I measure test scores of $100$ people in one of two conditions. Each test has $10$ items. On each item, each person can ...
2
votes
0answers
819 views

Distribution of the size of overlap between two random samples with duplicates?

Similar to this question, I want to know if the overlap between two samples is significant. However, my items are not unique; I have c distinct colors of items, there are $m_i$ items of color $\text{...
2
votes
0answers
128 views

Random coefficient models with fractional polynomial transformation selection method, which criterion to use?

I am trying to conduct a meta analysis for dose response studies where I am using fractional polynomial transformation from predefined family of powers. Now data fitting using all possible ...

15 30 50 per page