All Questions

5 views

Scaling/weighting probabilities

I have data which looks like the table below. I am provided with the likelihood of a game to occur (Prob) and the current time in min for that game (Curr_time_min). I would like to create a formula ...
6 views

Estimating a smooth probability distribution

I have a data frame storing labels and numerical values, e.g. cars and measured speed ...
6 views

Mixed-Effects ANOVA with Contrasts in R (Categorical variables with four levels)

I am struggling with a mixed-effects anova with categorical variables and am wondering if stat-gurus here would be willing to help me. I've read West, Aiken, and Krull (1996) to implement effect-...
8 views

How to integrate expert knowledge to outlier detection algorithms?

Suppose I have a dataset of 20 features, X1, X2..X20. ...
12 views

How to include new data into existing algorithm? (any brainstorming is welcome)

I have a complex ensembel algorithm X (divide data with k means that learn ensembel for each subgroup). Learning time of X is approx. 20 hours. I cannot afford to relearn algorithm for every new ...
7 views

Does Regularized Logistic Regression Produce Callibrated Results?

It has been asked and addressed here that logistic regression modelling is calibrated already and there is no need for calibration of it. To me it seems the argument provided there does not follow ...
4 views

Statistical evaluation of a diagnostic model versus a prognostic model

I aim to compare the performance of a diagnostic model (with a binary outcome) to a prognostic model (survival). The diagnostic model was created using a random forest algorithm after univariate ...
13 views

On what types of datasets do tree-based models not do well?

Are there examples where splitting on the best feature/threshold combination is not actually the best way to split the tree, and that better results could be got by choosing a different feature but ...
4 views

Calculate the relative value of a subsection of data

Is it possible to calculate the relative error of a subsection of my data? I am analyzing nouns and I managed to divide abstract and concrete nouns. The correct predictions of concrete nouns are ...
6 views

Clustered standard errors vs. serial correlation

I have a probably simple question but I can't find an adequate answer. I have self-collected data from three monitors that measure temperature, humidity etc. I useed all three monitors simultaneously ...
8 views

With a given reward, what is the difference between reinforcement learning methods and relabel the data?

If a clear reward is given, we can train the model by RL algorithm. Or we can revise the label according to the reward and then retrain the model. What is the difference, judged by the final result?
3 views

Keeping a text classifier up to date

I have built a text classifier using Naive Bayes and TF-IDF. It is a fairly weak model (~94.7 accuracy) and is more of a proof of concept before I move on to more complex methods of analysing text. ...
6 views

DK responses with Yes/No Binary variable

I have variable with repospondent location if Urban 01=Yes 02= No 03 = Do not Know 04 = Not Asertained how can I deal with this variable if i combibe 03 and 04 as one Category as DK/NA 01=Yes 02= ...
5 views

pseudo R2 as xgboost objective function

I want to use a custom objective function with xgboost: 1 - (log(y) - log(p)) / (log(y) - log(q)) y = true value, p = my probabilities, q = some other base ...
3 views

How does SpaCy make its dependency trees?

I discovered that SpaCy had the ability to make dependency trees. For instance given a question “To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?” We can create its dependency ...
7 views

GLRT Inconclusive for Gaussian Detection

When constructing a detection test between two Gaussian vectors, I have $H_0: \mathcal{N}(0,\sigma_n^2)$ and $H_1: \mathcal{N}(0,\sigma_n^2+\sigma_s^2)$, and $\sigma_n^2$ while $\sigma_s^2$ in known. ...
20 views

Is Bias-Variance Trade Off and MSPE the same?

I've question regarding to the correct notation. I was reading about the Bias-Variance-Trade-Off in the textbook "Elements of statistical learning". Is the expected forecast error listed there the ...
8 views

Right techniques to cluster/segment categorical data?

I am new to this forum and to data science. I might be naive in asking my question. I am working on customer transactions data. I have got data of ~143k customers; for each customer I have monetary ...
11 views

How to formulate the regression model in R for trials in agriculture over multiple years?

I'm working in a research facility for agriculture where we do a lot of trials that continue over several years. For the example below i tried to find out how to formulate the regression model so ...
7 views

PCA combined with 2nd stage multinomial logit regression

I am running a PCA with the following kind of data. Var1 - Var 25 take only discrete values between -4 and 4 in an imposed normal distribution (i.e. 4 and -4 can be the chosen value once, -3 and 3 can ...
15 views

Neural network regression: seemingly bounded output

I have been working on a neural network based predictor for a project. The aim is to learn a certain quantity, say the signal strength of a cellular network, for each coordinate set in the dataset. ...
38 views

Why descriptive statistics contradicts with regression coefficents?

I am estimating a binary logistic regression with L1 norm. According to the regression coefficients, the sign of x1's ...
5 views

Query regarding ML feature extraction and aggregated features

Hopefully I've articulated my queries as clearly as possible. I've provided sample data below (assume this is 100,000+ rows) Data characteristics Each row is a unique observation The first 3 columns ...
121 views

In CNN, do we have learn kernel values at every convolution layer?

I'm new to machine learning and one of the things I don't understand about CNN is whether we have to learn the kernel values at every convolutional layer, or just learn a single set of kernel values ...
7 views

How do sample weights work in classification models?

What does it mean to provide weights to each sample in a classification algorithm? How does a classification algorithm (eg. Logistic regression, SVM) use weights to give more emphasis to certain ...
9 views

Neural network how to deal with comparison

I'm currently working on a DQN network and this question comes to me. As far as I know, neural networks are good at dealing with values that have never seen (generalisation). E.g. If a classification ...
4 views

Identify Heywood cases in fa function from psych package within a function being looped

I'm working on a simulation to examine parameter recovery from hierarchical factor analyses, and I've run into an issue with the psych package throwing warnings for ultra-Heywood cases and ...
7 views

Need help understanding output of a periodogram

In my effort to understand the output of a periodogram I created a series (s) where 1,1,1,1,1,1,1,1,1,10 is repeated 100 times and then created a periodogram of this series using the following R code: ...