All Questions

3
votes
1answer
293 views

Can we compute bivariate from marginal distributions?

Can we do that? If yes, then what are the conditions which should be met?
9
votes
3answers
4k views

What to make of explanatories in time series?

Having worked mostly with cross sectional data so far and very very recently browsing, scanning stumbling through a bunch of introductory time series literature I wonder what which role explanatory ...
4
votes
1answer
191 views

Incremental one-way ANOVA

Imagine you have a set of four elements (A-D) with some numeric values of a measured property (several observations for each element): ...
6
votes
1answer
290 views

Principal components of spatial variables

I have a number of rasters of environmental data (~10) which may be important predictors for modelling species presence and abundance at ~10 different locations. I would like to know which of the ...
12
votes
2answers
1k views

How to quickly sample X if exp(X) ~ Gamma?

I have a simple sampling problem, where my inner loop looks like: v = sample_gamma(k, a) where sample_gamma samples from the ...
24
votes
3answers
540 views

Equations in the news: Translating a multi-level model to a general audience

The New York Times has a long comment on the 'value-added' teacher evaluation system being used to give feedback to New York City educators. The lede is the equation used to calculate the scores - ...
24
votes
7answers
5k views

Algorithm to dynamically monitor quantiles

I want to estimate the quantile of some data. The data are so huge that they can not be accommodated in the memory. And data are not static, new data keep coming. Does anyone know any algorithm to ...
5
votes
1answer
15k views

Why are fitted.values not part the R object returned from arima?

Starting out with arima models in R, I do not understand why fitted.values (of an AR(2) process for example) are not part of the output like they are in regressions. Did I miss them when running ...
1
vote
0answers
243 views

The usage of Newton's method in nonlinear regression

Would you please give an intuitive illustration of Newton's Method, when we deal with nonlinear regression? Basically I understand that if we can use Taylor's theorem to expand the RSS function of ...
17
votes
1answer
9k views

How to create coloured tables with Sweave and xtable? [closed]

I am using Sweave and xtable to generate a report. I would like to add some coloring on a table. But I have not managed to find any way to generate colored tables with xtable. Is there any other ...
2
votes
2answers
418 views

Do two variables need to be independent in order to obtain a correlation? [closed]

I would like to find the correlation between two variables. It was suggested to me that two variables should be independent; otherwise it is not meaningful statistically to calculate a correlation. ...
109
votes
8answers
96k views

When is it ok to remove the intercept in a linear regression model?

I am running linear regression models and wondering what the conditions are for removing the intercept term. In comparing results from two different regressions where one has the intercept and the ...
3
votes
2answers
3k views

What correlation coefficient and graph is appropriate with this data?

I'm not in Statistics field. I conducted the case study and collected the data as shown below I have data as shown in the table below: I would like to find correlation coefficient from this two table ...
15
votes
4answers
23k views

What is tied data in the context of a rank correlation coefficient?

I am not in statistics field. I have seen the word "tied data" while reading about Rank Correlation Coefficients. What is tied data? What is an example of tied data?
4
votes
1answer
1k views

Gaussian kernel estimator as Nadaraya-Watson estimator?

I'm working on a problem from "The Elements of Statistical Learning" (prob. 6.8): Suppose that for continuous response $Y$ and predictor $X$, we model the joint density of $X, Y$ using a ...
56
votes
8answers
37k views

What are disadvantages of using the lasso for variable selection for regression?

From what I know, using lasso for variable selection handles the problem of correlated inputs. Also, since it is equivalent to Least Angle Regression, it is not slow computationally. However, many ...
13
votes
2answers
19k views

Understanding the k lag in R's augmented Dickey Fuller test

I played around with some unit root testing in R and I am not entirely sure what to make of the k lag parameter. I used the augmented Dickey Fuller test and the Philipps Perron test from the tseries ...
13
votes
1answer
3k views

Scrambling and correlation in low discrepancy sequences (Halton/Sobol)

I am currently working on a project where I generate random values using low discrepancy / quasi-random point sets, such as Halton and Sobol point sets. These are essentially $d$-dimensional vectors ...
2
votes
1answer
2k views

How can I compare Likert scale data of two clusters in SPSS?

In SPSS, I want to compare two clusters of management sciences department faculty members in two universities. Which test should I use? Can you explain how to do it in SPSS?
7
votes
2answers
4k views

How are zero values handled in lm()?

I am working on a linear regression with R and there are many 0 values in my predictor variables. How are these handled in R's lm() function? Should I remove this ...
5
votes
2answers
2k views

Efficient way to classify with SVM

I'm doing a binary classification using SVM classfier, libsvm, where roughly 95% belongs to one class. The parameters C and gamma are to be set before the actual training takes place. I followed the ...
8
votes
3answers
6k views

Mahalanobis distance between two bivariate distributions with different covariances

The question is pretty much contained in the title. What is the Mahalanobis distance for two distributions of different covariance matrices? What I have found till now assumes the same covariance for ...
25
votes
2answers
1k views

Detecting patterns of cheating on a multi-question exam

QUESTION: I have binary data on exam questions (correct/incorrect). Some individuals might have had prior access to a subset of questions and their correct answers. I don’t know who, how many, or ...
3
votes
1answer
1k views

How to calculate confidence interval when only a part of the samples are valid?

I will simplify our problem in this way. Say there are 100,000 cases in total to examine. Due to the time limitation, we randomly selected 2,000 of them. Then we found 1,000 of them are invalid, so we ...
7
votes
2answers
2k views

Generate random strings based on regular expressions in R

I'm trying to generate random textual data based on regular expressions. I'd like to be able to do this in R, as I know that R does have regex capabilities. Any leads? This question has come up ...
9
votes
2answers
4k views

Complex regression plot in R

I need to draw a complex graphics for visual data analysis. I have 2 variables and a big number of cases (>1000). For example (number is 100 if to make dispersion less "normal"): ...
10
votes
2answers
949 views

Best methods of feature selection for nonparametric regression

A newbie question here. I am currently performing a nonparametric regression using the np package in R. I have 7 features and using a brute force approach I identified the best 3. But, soon I will ...
5
votes
1answer
738 views

ROC surfaces in R

If my response variable has 2 levels, 0 and 1, I can use a ROC curve to assess the accuracy of my predictive model. But what if my response variable has 3 levels, -1, 0, and 1? Is there a way to ...
3
votes
1answer
146 views

Max-margin clustering with size constraint

Given a dataset $D$ and a distance measure, I want to split the dataset into two disjoint subsets $X, Y$ of a specified size (say 80% and 20% of the original size), so that the minimum distance of all ...
3
votes
2answers
950 views

How to transform data to have a new mean and maximum value?

I have data relative to image intensity, which range from 0 to 255. For some models I need to run, I must transform this intensity data to a different scale, which must have a certain mean (for ...
14
votes
6answers
26k views

Fast ways in R to get the first row of a data frame grouped by an identifier [closed]

Sometimes I need to get only the first row of a data set grouped by an identifier, as when retrieving age and gender when there are multiple observations per individual. What's a fast (or the fastest) ...
7
votes
1answer
3k views

How to compute prediction error from Relevance Vector Machine and Gaussian Process Regression?

Does anyone know how to construct a confidence interval for predicting a new test value given a trained Relevance Vector Machine (rvm) and/or Gaussian Process ...
12
votes
1answer
6k views

Interpreting range bars in R's plot.stl?

I have trouble figuring out what the range bars in plot.stl exactly mean. I found Gavin's post on this question and read the documentation as well, I understand ...
8
votes
2answers
338 views

Discussing binomial regression and modeling strategies

Today I have got a question about binomial/ logistic regression, its based on an analysis that a group in my department have done and were seeking comments upon. I made up the example below to protect ...
4
votes
2answers
628 views

What statistical technique would be appropriate for optimising the weights?

Background: I have the following data (an example): ...
2
votes
2answers
241 views

Building a classification rule for geographical mapping of cell phone number

I have a database of some 10000 cell-numbers with mapping of geographical areas they are associated to. I believe the first 5-6 characters in the numbers can point to the geo. area the cell number ...
4
votes
1answer
718 views

What is the power of the Ljung-Box Test for auto-correlation?

How large sample size should be for Ljung-Box statistic to achieve a power $\ge 0.5$ when number of lags tested is 1? How does the power fall ( assuming an AR(1) process), with increasing number of ...
31
votes
3answers
10k views

Visualizing a million, PCA edition

Is it possible to visualize the output of Principal Component Analysis in ways that give more insight than just summary tables? Is it possible to do it when the number of observations is large, say ~...
35
votes
4answers
3k views

Justification of one-tailed hypothesis testing

I understand two-tailed hypothesis testing. You have $H_0 : \theta = \theta_0$ (vs. $H_1 = \neg H_0 : \theta \ne \theta_0$). The $p$-value is the probability that $\theta$ generates data at least as ...
6
votes
3answers
22k views

Calculating proportions by age in R

I working on this data set with marital status and age. I want to plot the percentage of never married man versus each age. Could you please help me to figure out the way how to do it in R? So far I ...
14
votes
3answers
5k views

Best way to put two histograms on same scale?

Let's say I have two distributions I want to compare in detail, i.e. in a way that makes shape, scale and shift easily visible. One good way to do this is to plot a histogram for each distribution, ...
1
vote
2answers
2k views

Percent error or percent difference?

Using either percent error or percent difference, I want to compare one of my measured values from a set to the mean of the set. By reading the Wikipedia article on percent difference, it's still not ...
18
votes
4answers
10k views

In regression analysis what's the difference between data-generation process and model?

In regression analysis what's the difference between 'data-generation process' and 'model'?
4
votes
4answers
176 views

Feature selection for classification, controlling for sub-population

I have a bunch of points that belong to one of population P1, P2, ... Pn AND to class A or B. Within each population I'll be doing classification between A and B, and I want to select features that ...
6
votes
1answer
790 views

Derivation of distance in TwoStep clustering

I am working with the twostep cluster process in SPSS Modeler (Clementine) and trying to get a sense for the distance function used. It is a log-likelihood function (as stated in docs) but I am unsure ...
7
votes
3answers
2k views

How to generate user-friendly summaries of cluster analysis?

I used BIRCH and HAC to do clustering on my data. I want to now what type of information I can include in reports that my users can generate to get more insights on the clusters. I would have to dumb ...
5
votes
2answers
2k views

Computing confidence intervals for prevalence for several types of infection

I have a dataset about a population in hospital, and what type of infections patients have. Let say the number of patients is 100, 10 of them have pneumonia ...
8
votes
3answers
6k views

How to do ANOVA on data which is still not normal after transformations?

I'm looking at the effect defeat and entrapment inducing conditions have on subjective ratings of defeat and entrapment at three different time points (among other things). However the subjective ...
7
votes
1answer
210 views

Example of discontinous effect of x on y dataset (for paper)

For a paper, I need an example of a dataset $(x_i,y_i)$ where the residuals are $iid$ (the $x$ do not represent time) with a discontinuity on the effect of $x$ on $y$. I have already found a dataset ...
107
votes
5answers
17k views

What skills are required to perform large scale statistical analyses?

Many statistical jobs ask for experience with large scale data. What are the sorts of statistical and computational skills that would be need for working with large data sets. For example, how about ...

15 30 50 per page