0
votes
0answers
9 views

Where to find a guide to encoding categorical features?

I am facing an ML task with various categorical variables. Some examples include the following: Binary variables (0,1). Multilevel factors that can be ordered (low, medium, high). Multilevel factors ...
1
vote
1answer
15 views

What does it mean in real-world terms that the Chi-Squared distribution is a special case of the Gamma distribution?

Among other applications, a gamma distribution answers the question: "If the average time (or other quantity) between events is β, what is the probability that x time will elapse before α events occur?...
0
votes
0answers
8 views

How common is it to train each network independently then train end-to-end in a neural network?

Say that I had 3 autoencoders stacked on each other. How common is it to ...
1
vote
1answer
39 views

When using a limited data set, how can I use excel to forecast future values?

To preface, I am a statistics novice, but I am faced with a problem that I cannot seem to be able to satisfactorily resolve. The problem is as follows: I am working to forecast future sales for one ...
4
votes
0answers
24 views

Approximate Metropolis algorithm - does it make sense?

Some time ago Xi'an asked What is the equivalent for cdfs of MCMC for pdfs? The naive answer would be to use "approximate" Metropolis algorithm in form Given $X^{(t)} = x^{(t)}$ 1. generate $Y ...
2
votes
2answers
43 views

Understanding the formula for the Confidence Interval of the difference between two sample proportions

I'm having trouble making sense out of the formula $$(\hat{p}_1-\hat{p}_2) \pm 1.96 \sqrt{\frac{\hat{p}_1 (1-\hat{p}_1)}{n_1} + \frac{\hat{p}_2 (1-\hat{p}_2)}{n_2}}$$ Is the idea that the sampling ...
0
votes
0answers
21 views

Too many models to choose from

I'm trying to find a good evaluation function to approximate the score of a position of the game Reversi. That is the score of mutually perfect play. A position of Reversi has 64 fields, with each one ...
0
votes
1answer
26 views

How to reduce the number of labels in regression

I'm working in a regression problem, related to bio-signals, where my labels are integer numbers between 0 and 10. I've tried a couple of regression algorithms already, mainly linear regression. Edit:...
1
vote
1answer
12 views

How can I test normality of data incrementally?

I plan to track the mean and variance of a data set as it grows over time, but I can't keep the actual data, just the current mean and variance. Later, I want to estimate the probability of a given ...
2
votes
1answer
24 views

Required sample size and degrees of freedom for a VAR

I have read in many textbooks that as the ratio between the number of coefficients to be estimated in a VAR and the number of time periods increases above a certain threshold, estimation becomes ...
0
votes
1answer
15 views

How does data augmentation reduce overfitting?

I'm trying to understant the benefit apported by the step of data augmentation in a classification algorithm. I have a vector of hexadecimal strings and a column vector containing the label ...
0
votes
1answer
17 views

What happens inbetween switching test and training sets in k-fold cross validation

The general idea of k-fold cross validation is to partition your test data from your training data, then within your training data you make another partition where all but one of those partitions are ...
0
votes
0answers
29 views

Calculating the probability of a successful marriage lasting N years

In reading the US HHS report on marriage trends, I was interested in calculating the probability that a marriage for a specific person with different qualities (male/female, education, race, age) ...
0
votes
0answers
22 views

how to preprocess/feature stacle multimodal input data?

I am wondering how to normalize data for the use of SVMs etc. that has a clear non Gaussian, i.e. non unimodal distribution. I wrongfully scaled the data by subtracting the mean and dividing the std ...
2
votes
1answer
57 views

Deriving the normalizing constant for the multivariate Gaussian

I am trying to derive the normalizing constant for the multivariate Gaussian. The book I'm following suggests diagonalizing the covariance matrix and then using a change of variables. So, we consider ...
0
votes
0answers
19 views

Cheap, available in used market AP Statistics textbook [on hold]

I am teaching an Advanced Placement (AP) Statistics class and the high school currently has no textbooks for it. What is a cheap, available-in-the-used-market Statistics textbook that people actually ...
0
votes
1answer
10 views

difference in meaning between Soft Clustering and Multi-classification techniques

Dear Weka subscribers i would like to know the difference in meaning, approach and concept between : Cluster similar objects into more than one cluster ( soft clustering ) where objects can be ...
1
vote
0answers
11 views

SPSS Chi Square goodness-of-fit expected frequencies distribution check

My question is about the Chi Square goodness-of-fit test. Not to be mistaken with the test for association or the test for correlation (both called Chi Square). Let's say I have following variable '...
1
vote
2answers
6 views

Finding lower proportion limit of a sample distribution

Problem description If a population proportion is 0.28, and if the sample size is 140, 30% of the time the sample proportion will be less than what value if you are taking random samples? This ...
0
votes
0answers
20 views

Confidence interval of first derivative of a loess smooth

I have plotted the evolution of parameter vs time. I would however prefer to plot the change in these parameter vs time. Here I have seen how to predict a confidence interval for a loess smooth and ...
0
votes
0answers
8 views

Manova, mixed models or what?

I have difficulties in choosing the appropriate statistics to use for my data. In particular I don't know whether I have to use a MANOVA or a mixed model. Let's suppose, I have three dependent ...
0
votes
2answers
24 views

Comparing Result of Permutation and bootstrapping

I am trying to verify the correlation between two parameters using bootstrap/permutation methods(classical example!). What i understand is that both permutation and bootstrap method involves ...
0
votes
0answers
5 views

How do I calculate sample size and What is the is the meaning of power in a cross-sectional study?

Here is an example: Previous research reported that the prevalence of HIV testing among gays is 19.2% . Sample size required to detect this percentage with 95% confidence and .05 precision is 273 (...
0
votes
0answers
21 views

The Causal Impact Package in R could not trace the right number in the test group

I have this sample dataset which is as follow: head(data) ...
0
votes
0answers
21 views

cross validation in weka and model accuracy

I am developing a sentence classifier using Weka. The number of class is 48. I trained the classifier using AdaBoost.M1 where weak classifier is multi-layered perceptron. And each class label is just ...
23
votes
9answers
2k views

What does it mean when they say that “Hillary has a 75% chance of winning”?

As the election is a one time event, it is not an experiment that can be repeated. So exactly what does the statement "Hillary has a 75% chance of winning" technically mean? I am seeking a ...
0
votes
0answers
13 views

A Question on Interactions in STATA [on hold]

I am trying to replicate one of Wooldridge examples to learn a bit. It is about interacting the dummy variable (black) with a continuous variable (percblck). Here is the code that I have written: <...
1
vote
0answers
17 views

Quade test in R [on hold]

I would like to perform a Quade test with more than one covariate. My question is if it is posible to use more than one covarite in the Quade test. If the answer is yes, I would be pleased if you ...
1
vote
0answers
21 views

Interaction between two random variables [on hold]

I have a machine learning system called Sys. Sys uses a variables set (Var) applied to a ...
0
votes
0answers
9 views

What is the junction tree of the following graph?

I got so confused in the remaining steps (from triangulated graph to junction tree) after turning my Bayesian network to a triangulated graph. Would anyone please help?
1
vote
1answer
20 views

bayesian classification unknown domain

Suppose I am building a naive Bayes model to classify text messages as either spam or legit. I am training my model using a dataset containing both classes and for which I know the domain (the number ...
0
votes
1answer
29 views

What method for classification with 24 variables, 14 categories and 100,000 obs.?

I have a dataset with more than 100,000 observations (rows) and 24 variables in which 23 are continuous and one is categorical variable. The categorical variable has 13 categories (1, 2, 3, ..., 13) ...
-1
votes
0answers
15 views

Save regression coefficients as variables in Stata [on hold]

Consider the following regression in Stata: reg y x1 x2 ibn.x3#ibn.x4 where $x_{3}$ and $x_{4}$ are dummy variables, for example, gender (male/female) and race (...
0
votes
0answers
22 views

Does constrained EM algorithm work with bad initial inputs?

When trying to perform constrained optimization using EM algorithm, does EM work if the initial solution (x0) violates the constraints?
1
vote
2answers
105 views

Is MCMC really better than raw MC to Sample a region?

I have implemented the Markov Chain Monte Carlo (MCMC) with the Metropolis-Hastings sample selection criteria. Basically, as I understand it and as I have implemented it is: ...
1
vote
0answers
11 views

How to work out the relationship between error α and threshold η

Reading up on the Neyman Pearson Lemma, I have a question about how to obtain the probability for a type I error, $\alpha$, when we establish a threshold $\eta$ for the liklihood ratio $\Lambda(x)=\...
0
votes
1answer
24 views

SPSS Chi square test proportions (less than 5 expected)

Let's say I have a (fictive) sample: White: 25 Foreign: 1 General population: White: 30 Foreign: 9 Now I want to compare if my frequencies do not differ from the population. Normally I could use a ...
0
votes
0answers
11 views

How to test significant interaction effect

If I have a model, lets say y = a + b1male + b2large + b3medium + b4malelarge + b5malemedium where male is dummy coded 1 for male and 0 for female and large and female are dummies (also 0,1 coded) for ...
0
votes
0answers
12 views

Using sentiment data to predict stock prices

I have implemented a model to predict "last" price of stock of 5 days ahead. I have used RandomForestRegressor. In addition to this, I have a data set of sentiments. It is the data from social media ...
0
votes
0answers
6 views

In Two Way Mixed ANOVA (with one IV as a Repeated Measure) what post hoc correction should I incorporate if my data has non-homogenous variance?

I have three groups of subjects (BS) which have been treated with 6 conditions (WS Repeated Measures). I have non-homogenous variance of data even if I square root transform it. I suppose I have to ...
0
votes
0answers
16 views

Matrix Calculations and Multivariate Bayesian Methods

Suppose I have the matrix given by: $$X = \begin{bmatrix}1 & 0 & 0\\ 1 & 1 & 0 \\ 1 & 1 & 1 \end{bmatrix}$$ This matrix actually represents whether a user interacted with a ...
0
votes
0answers
30 views

Robust M-estimator of location and dispersion by hand in R

I try to estimate a location and dispersion model with R, as described Maronna et al (2006, pp. 56). However, my estimate dispersion does not converge to the desired value. Do I have an error in the ...
0
votes
0answers
7 views

Step by step guide for GRS test in R?

I'm new to R. I have basic knowledge in R. I'm testing a factor model. I have to use GRS test. But not sure how to do it. Please help me. I have found this recipe for GRS online. http://faculty....
1
vote
0answers
10 views

statistics metrics or machine learning methods to quantify the relationship between a point and a group

There are several data groups: A1 = { A11, A12, A13, …, A1n} A2 = {A21, A22, A23, …, A2n} .. … … Am = {Am1, Am2, Am3, …, Amn} Here ...
0
votes
0answers
15 views

multivariate normal distribution. And marginals distributions [duplicate]

if $X_1$ and $X_2$ are random variables that marginally have a normal distribution, is it true that the joint $(X_1,X_2)$ has a distribution according to a multivariate normal density distribution? ...
0
votes
0answers
13 views

winsorization - reducing the effect of outliers

I have measured the response times each participant took to respond to 24 items, however, only times of the correct responses of each participant were considered, thus leaving me with distinct number ...
1
vote
1answer
9 views

Making a forecast: Confidence/Prediction/Tolerance Interval?

I understand the the difference between these three types of intervals but I will summarize briefly: Confidence Interval: an interval that will contain the true mean value say, 95%, of the time. ...
0
votes
0answers
17 views

Neural Networks for simple Regression: Y=X^2

All I'm trying to do is make a simple neural network to learn $y=x^2$ (Just as a toy example with x ranging over the integers between -100 and 100) with Tensorflow, but I'm having a surprising amount ...
3
votes
0answers
65 views

Is there a mathematical expression that shows how LASSO shrinks coefficients (including some to zero)?

By using singular value decomposition (SVD), I noticed from the derivation that ridge regression shrinks the coefficients by factor $\frac{D^2}{D^2+\lambda}$, where $D$ is the diagonal matrix of the ...
0
votes
0answers
5 views

How do I deal with new words in held out documents when evaluating generative models?

I'm trying out some different generative models and I want to evaluate them by holding out some documents, training the models the other documents, and then calculating the perplexity on the held out ...

15 30 50 per page