0
votes
0answers
2 views

How to exact prediction from over sampled data(Undoing oversampling)?

We are oversampling the data to use in logistic regression. Aim is to predict CTR(click probability) which is rare event scenario. I have predicted the probabilities of click but CTR results are ...
0
votes
0answers
2 views

Deriving the relation $\sum (y_i - \bar{y})^2 - \sum (y_i - \hat{y}_i)^2 = \sum (\bar{y} - \hat{y}_i)^2$

$y_i$ denotes the observed values of $y$, $\hat{y}_i$ are the projected values from the explanatory variables, and $\hat{y}$ is the average. This relation is supposed to hold up when one of the ...
0
votes
0answers
4 views

Support vector regression (LIBSVM) returns out of range outputs when I use out-of-sample data to predict one step ahead (MATLAB)?

I'm using SVR model in MATLAB R2016a using this option: options_z = ['-q -s 3 -t 2 -c ', C_param, ' -p ', epsilon, ' -g ,Kernel_scale]; I'm optimizing SVR ...
0
votes
0answers
4 views

Power calculation - categorised continuous exposure in case-control study

I have a continuous exposure that I would like to catagorise it into quantiles according to the distribution of control (i.e. 25% controls in each exposure level group). From pilot study, I know that ...
0
votes
2answers
13 views

Is feature engineering relevant at all for Random Forests?

Random forests is an ensemble of trees that learns the hidden patterns in the data. I have mostly tried doing some feature-engineering before running the Random Forest model but is it required or the ...
0
votes
1answer
8 views

Behaviour of neighbouring points in LOESS smoothing when data is not uniformly distributed

The purpose of loess is to create an 'average' value of the response for any $x$, by using points in the region of $x$ to create a local regression line. From what I understand, picking the ...
0
votes
0answers
6 views

How to decide what is the relevant group in a precsion and recall computation?

One of the most famous measurements for an information retrieval system is to compute its precision and recall. For both cases, we need to compute the number of total relevant documents and compare it ...
0
votes
0answers
9 views

For a Fisher Information matrix $I(\theta)$ of multiple variables, is it true that $I(\theta) = nI_1(\theta)$?

For a Fisher Information matrix $I(\theta)$ of multiple variables, is it true that $I(\theta) = nI_1(\theta)$? That is, if $\theta = (\theta_1, \ldots, \theta_n)$, will it be the case that the fisher ...
0
votes
0answers
7 views

WEKA / Support vector machine - does attribute have positive or negative impact?

I'm doing some machine learning with WEKA using around 20 numeric attributes to predict a true/false class. Support vector machine gives me good enough results on my data set. However, I'd also like ...
-3
votes
0answers
19 views

I need help with my statistics final [on hold]

i need to know what type of test to use for the following questions 1) Does Mark’s yards relate to the temperature during the start of the game? 2) Does Mark’s year in college impact the amount ...
0
votes
0answers
6 views

What methods exist to apply inference to correlated noise in experimental results?

What resources are available for applying inference/computational statistics to correlated noise (in various scientific experiments)? My idea is this: let's say we use some apparatus (e.g. a ...
0
votes
0answers
4 views

bayesian approach to estimate weekday and weekend shoppers

I am trying to predict customer is weekend or weekday shopper. The dataset has 60% of the customers as weekend shoppers. So for new shopper I can assume they 60% likely to be weekend shoppers. Now a ...
0
votes
0answers
7 views

How can we find the Fisher Information Matrix of a linear regression model?

Suppose we have a sequence of random variables $y_1, \ldots, y_n$ such that $y_i = \beta_0 + \beta_1 x_i + \epsilon_i$ where $\epsilon_i$ is assumed to be i.i.d. $N(0,\sigma^2)$, with $\sigma^2$ ...
0
votes
0answers
5 views

Parameter selection and k-fold cross validation

I have one dataset, and need to do cross-validation, for example, a 10-fold cross-validation, on the entire dataset. I would like to use radial basis function (RBF) kernel with parameter selection ...
0
votes
0answers
2 views

Small CFA Model with 0 Degrees of Freedom, 1.0 CFI, 0 RMSEA

Can someone please help explain this to me, as I've never encountered it when running CFA before. I ran a CFA on a simple model with 4 indicators. The initial model returned all very good fit indices ...
0
votes
0answers
6 views

Why is the Fisher Information sometimes written as a conditional expectation?

I was looking at the Fisher Information entry on Wikipedia just now, and saw that the Fisher Information Matrix was written as: $$ \mathcal{I}(\theta) = - \operatorname{E} \left[\left. ...
0
votes
0answers
4 views

Likert scale questionnaire analysis

I'm doing a study regarding the relationship between the challenges experienced by firms and the strategies they implement. I have two variables in the challenges: ...
0
votes
0answers
12 views

Numbers in reporting one way ANOVA

In reporting data for a one way ANOVA what do all the numbers in the following stand for? $F (1, 24) = 44.4$, $p < .01$.
0
votes
0answers
5 views

How to get monthly Treasury Bill time-series data from weekly quotes

I am trying to get monthly T-Bill rate data for Sri-Lanka, for 3 and 6 month maturities. The Central Bank website has a downloadable excel spreadsheet giving weekly quotes (thus 4 rates quoted per ...
0
votes
0answers
23 views

Better Predictive Model?

I'm running a bunch of different models trying to find one that is best at predicting using a Validation set and Root Average Squared Error(RASE) calculated from residuals as my main criteria. My data ...
0
votes
0answers
7 views

Treatment of this particular dummy variable

I have these two dummy variables and a continuous variable in my dataset and I was wondering how I should include them in my model? dad10: A dummy variable = 1 if the individual lived with their dad ...
0
votes
0answers
14 views

Variance Reduction by Conditioning

Say that given Y=y, (edit->) a R.V. (edit end) N have a probability function of $f_N$ and Y have a distribution function $F_Y\in(0,1)$. We want to estimate the the probability p for when N is larger ...
2
votes
1answer
7 views

Autoregressive model for time series with structural breaks

I'm using a structural break model (threshold model or regime switching model) to examine the dynamics of a time series. The ADF test shows that the series has a unit root. Right now I'm regressing ...
0
votes
0answers
6 views

Multi-class SVM Calibration

Say we have multiple SVMs used in a one-vs-all approach, such that classes a, b, c correspond to 3 SVMs trained positively on the class and then negatively on all ...
0
votes
0answers
9 views

How to recover the value of the coefficient of an omitted dummy variable?

Suppose I have a regression given by $S = \beta_0 + \beta_1X + \beta_2Y + \beta_3Z$ with dummy variables $X,Y,Z$ and $X + Y + Z = 1$. I plug into Stata or whatever and obtain estimates ...
0
votes
0answers
5 views

RandomForest undersampling non events

I have data that is about 50 to 1 nonevent to event (in this case, purchases). I was taking an equal, or close to equal, sample of nonevents to events with a random forest model, and running the ...
1
vote
1answer
10 views

Second order inclusion probabilities in With-Replacement Sampling

I'm reading the book "Model Assisted Survey Sampling" from Särndal et al. In chapter 2, there's a section about Sampling with replacement. I'll put this into context: We have $m$ independent draws, ...
1
vote
1answer
11 views

Diagnostics for generalized additive model vs linear model

I am doing an analysis in R and I have the model: lm(birthrate~education+employment+lo(latitude, longitude),data=data2). It seems to be a generalized additive ...
0
votes
0answers
6 views

Is F-score (precision-recall) meaningful for a balanced dataset?

In the binary classification scenario, for highly unbalanced datasets, when the task is to find a few needles in a haystack -- say, in information retrieval -- we use F-score (precision-recall) for ...
0
votes
0answers
10 views

Extract covariate p-value from cph output

I used to use coxph and extract model output by summary(model4) to extract each the p-value of each covariate. How can I do that in cph (rms packagee)? So far I get the HR and overall p-value of ...
1
vote
0answers
16 views

If you know the central moments of the data $X$, find a function $f$ for which $f(X)$ has arbitrary central moments

Say you are given one-dimensional data $X$, with mean $\mu$ and central moments $a_n$ which you know. Can you construct a function $f(x)$ which transforms the data such that $f(X)$ has the central ...
1
vote
2answers
16 views

Is Naive Bayes suitable for large datasets with thousands of features?

I have a data set with 100 million rows and 15,000 categorical variables each with 0/1 values. My target variable is also a 0/1 binary variable. Is Naive Bayes suitable in terms of computational ...
0
votes
1answer
11 views

Logistic regression repeated measures with binary outcome and continuous predictor

I am really a beginner in stats (and also in r) and would need a input on whether my statistical methodology for following research experiment is right: Study: I want to test the effect of the ...
0
votes
0answers
9 views

Pooled OLS from panel data

I have unbalanced panel data for two years (4 years apart). I am planning on running Fixed Effects estimation. I would also like to do a cross-sectional analysis. Would it be appropriate to use a ...
0
votes
0answers
5 views

Reshaping Inputs that contain continuous and discrete values

The inputs I am using are 2xN, where the first 1xN row is a continuous number, and the second 1xN row takes on only discrete values (that encodes a . I expect there to be a relation between A[1,i] and ...
0
votes
0answers
8 views

SPSS: Interpretation of coefficients - OLS

I could need some help interpreting my findings. I've been conducting a linear OLS regression with the following output: I'm trying to discover what the influences are from an acquisition on the ...
0
votes
0answers
4 views

efficiently locf by groups in a single R data.table [migrated]

I have a large, wide data.table (20m rows) keyed by a person ID but with lots of columns (~150) that have lots of null values. Each column is a recorded state / ...
0
votes
0answers
3 views

Test of Equality of mean for time series data

I want to test the equality of mean returns of 4 fund mangers for seven time series data of annual returns, which statistical tools would be appropriate, i am planning to use single factor ANOVA
0
votes
0answers
16 views

Multiple regression with all predictors or with each predictor separate?

Assume you have following hypotheses: H1. X is positively related to Y H2. Z is negatively related to Y And you also have some control variables such as age, gender, ... Is it recommended to ...
0
votes
0answers
7 views

Can mediation analysis be done with Cox proportional hazards models?

I am trying to understand what types of open source software projects succeed and stay active for a long time and what types die. I want to use survival analysis because some of the projects in my ...
0
votes
0answers
5 views

Unsupervised deep learning image clustering

I'm interested in trying to apply my (very) limited understanding of neural networks to a problem of image clustering. I realise that NN's are mainly used for regression or classification problems ...
0
votes
1answer
12 views

Stationarity of subsample

Consider that I have a weakly stationary series for the period 2003M1-2014M12. I want to make a VAR model for the subsample 2007M1-2014M12. Should I reconsider the weak stationarity of my series, so ...
1
vote
0answers
18 views

good practice for cluster analysis

I want to find out what are the best practice in conducting a reliable cluster analysis: 1. outliers: Is it necessary or not to remove the outliers in the variables to be used for cluster analysis? ...
1
vote
0answers
12 views

The name of technique, when main models explicitly split to submodels

For my project I work with some kind of random forest. Until recently I've had a huge dateset and I just built one model(one random forest) based on this dateset. The quality of this model was not ...
0
votes
0answers
10 views

R: Interpreting Fowlkes–Mallows index output for comparing dendrograms for hierarchical clustering

I have two data sets which contains information about subsystems in a bacterial metabolic model DataSet1: Behavior data of the subsystems DataSet2: Structural data of the same subsystems Then perform ...
0
votes
0answers
3 views

How to design a fitness function for binary logic network?

Assume we have a directed graph of connected nodes, where each node represents logical operator. Input for this logic operator are values on all edges leading to the node and result is outputted to ...
0
votes
0answers
12 views

Survival analysis with multiple factors

I want to do survival analysis in a situation where I expect the survival time depends on two factors: Environment. Each person is in one of three environments, $E_1,E_2,E_3$. I expect that the ...
0
votes
0answers
5 views

Detecting product taxonomy based on sparse information?

I'm trying to categorize a product into Google's Taxonomy based on it's store categories, title, and description. Eg: ...
1
vote
3answers
25 views

Why can I combine gamma and factorial coefficients for a beta conjugate posterior

A commenter reminds me to be clearer. In a Bayesian context, the product of a binomial likelihood and a beta prior probability is $$\left( {\begin{array}{*{20}{c}}n\\x\end{array}} \right)p_{}^x{(1 ...
3
votes
0answers
13 views

Is there anything wrong with keeping random intercept and slope terms that are perfectly correlated with oneanother?

I am following the advice to "keep it maximal", and am analyzing the results from several psycholinguistic experiments. My main interest is in the fixed effects, with the random effect terms in there ...

15 30 50 per page