0
votes
0answers
5 views

Scaling/weighting probabilities

I have data which looks like the table below. I am provided with the likelihood of a game to occur (Prob) and the current time in min for that game (Curr_time_min). I would like to create a formula ...
0
votes
0answers
6 views

Estimating a smooth probability distribution

I have a data frame storing labels and numerical values, e.g. cars and measured speed ...
0
votes
0answers
6 views

Mixed-Effects ANOVA with Contrasts in R (Categorical variables with four levels)

I am struggling with a mixed-effects anova with categorical variables and am wondering if stat-gurus here would be willing to help me. I've read West, Aiken, and Krull (1996) to implement effect-...
0
votes
0answers
8 views

How to integrate expert knowledge to outlier detection algorithms?

Suppose I have a dataset of 20 features, X1, X2..X20. ...
0
votes
0answers
12 views

How to include new data into existing algorithm? (any brainstorming is welcome)

I have a complex ensembel algorithm X (divide data with k means that learn ensembel for each subgroup). Learning time of X is approx. 20 hours. I cannot afford to relearn algorithm for every new ...
0
votes
0answers
7 views

Does Regularized Logistic Regression Produce Callibrated Results?

It has been asked and addressed here that logistic regression modelling is calibrated already and there is no need for calibration of it. To me it seems the argument provided there does not follow ...
0
votes
0answers
4 views

Statistical evaluation of a diagnostic model versus a prognostic model

I aim to compare the performance of a diagnostic model (with a binary outcome) to a prognostic model (survival). The diagnostic model was created using a random forest algorithm after univariate ...
0
votes
0answers
13 views

On what types of datasets do tree-based models not do well?

Are there examples where splitting on the best feature/threshold combination is not actually the best way to split the tree, and that better results could be got by choosing a different feature but ...
0
votes
0answers
4 views

Calculate the relative value of a subsection of data

Is it possible to calculate the relative error of a subsection of my data? I am analyzing nouns and I managed to divide abstract and concrete nouns. The correct predictions of concrete nouns are ...
1
vote
0answers
6 views

Clustered standard errors vs. serial correlation

I have a probably simple question but I can't find an adequate answer. I have self-collected data from three monitors that measure temperature, humidity etc. I useed all three monitors simultaneously ...
0
votes
0answers
8 views

With a given reward, what is the difference between reinforcement learning methods and relabel the data?

If a clear reward is given, we can train the model by RL algorithm. Or we can revise the label according to the reward and then retrain the model. What is the difference, judged by the final result?
1
vote
0answers
3 views

Keeping a text classifier up to date

I have built a text classifier using Naive Bayes and TF-IDF. It is a fairly weak model (~94.7 accuracy) and is more of a proof of concept before I move on to more complex methods of analysing text. ...
-1
votes
0answers
6 views

DK responses with Yes/No Binary variable

I have variable with repospondent location if Urban 01=Yes 02= No 03 = Do not Know 04 = Not Asertained how can I deal with this variable if i combibe 03 and 04 as one Category as DK/NA 01=Yes 02= ...
0
votes
0answers
5 views

pseudo R2 as xgboost objective function

I want to use a custom objective function with xgboost: 1 - (log(y) - log(p)) / (log(y) - log(q)) y = true value, p = my probabilities, q = some other base ...
0
votes
0answers
3 views

How does SpaCy make its dependency trees?

I discovered that SpaCy had the ability to make dependency trees. For instance given a question “To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?” We can create its dependency ...
0
votes
0answers
7 views

GLRT Inconclusive for Gaussian Detection

When constructing a detection test between two Gaussian vectors, I have $H_0: \mathcal{N}(0,\sigma_n^2)$ and $H_1: \mathcal{N}(0,\sigma_n^2+\sigma_s^2)$, and $\sigma_n^2$ while $\sigma_s^2$ in known. ...
0
votes
0answers
20 views

Is Bias-Variance Trade Off and MSPE the same?

I've question regarding to the correct notation. I was reading about the Bias-Variance-Trade-Off in the textbook "Elements of statistical learning". Is the expected forecast error listed there the ...
0
votes
0answers
8 views

Right techniques to cluster/segment categorical data?

I am new to this forum and to data science. I might be naive in asking my question. I am working on customer transactions data. I have got data of ~143k customers; for each customer I have monetary ...
0
votes
0answers
11 views

How to formulate the regression model in R for trials in agriculture over multiple years?

I'm working in a research facility for agriculture where we do a lot of trials that continue over several years. For the example below i tried to find out how to formulate the regression model so ...
0
votes
0answers
7 views

PCA combined with 2nd stage multinomial logit regression

I am running a PCA with the following kind of data. Var1 - Var 25 take only discrete values between -4 and 4 in an imposed normal distribution (i.e. 4 and -4 can be the chosen value once, -3 and 3 can ...
1
vote
0answers
15 views

Neural network regression: seemingly bounded output

I have been working on a neural network based predictor for a project. The aim is to learn a certain quantity, say the signal strength of a cellular network, for each coordinate set in the dataset. ...
0
votes
1answer
38 views

Why descriptive statistics contradicts with regression coefficents?

I am estimating a binary logistic regression with L1 norm. According to the regression coefficients, the sign of x1's ...
0
votes
0answers
5 views

Query regarding ML feature extraction and aggregated features

Hopefully I've articulated my queries as clearly as possible. I've provided sample data below (assume this is 100,000+ rows) Data characteristics Each row is a unique observation The first 3 columns ...
2
votes
2answers
121 views

In CNN, do we have learn kernel values at every convolution layer?

I'm new to machine learning and one of the things I don't understand about CNN is whether we have to learn the kernel values at every convolutional layer, or just learn a single set of kernel values ...
0
votes
0answers
7 views

How do sample weights work in classification models?

What does it mean to provide weights to each sample in a classification algorithm? How does a classification algorithm (eg. Logistic regression, SVM) use weights to give more emphasis to certain ...
0
votes
0answers
9 views

Neural network how to deal with comparison

I'm currently working on a DQN network and this question comes to me. As far as I know, neural networks are good at dealing with values that have never seen (generalisation). E.g. If a classification ...
0
votes
0answers
4 views

Identify Heywood cases in fa function from psych package within a function being looped

I'm working on a simulation to examine parameter recovery from hierarchical factor analyses, and I've run into an issue with the psych package throwing warnings for ultra-Heywood cases and ...
0
votes
0answers
7 views

Need help understanding output of a periodogram

In my effort to understand the output of a periodogram I created a series (s) where 1,1,1,1,1,1,1,1,1,10 is repeated 100 times and then created a periodogram of this series using the following R code: ...
1
vote
0answers
13 views

Is it possible for a model to have higher sensitivity/specificity but lower accuracy and AUC?

In the evaluation of classification models, I've found one model to have a higher accuracy and c-statistic (AUC) as compared to a second model. However, the second model has higher sensitivity, ...
1
vote
1answer
16 views

calculating or approximating the normalizing constant bayesian posterior

I am wondering if it is possible to re-calculate the normalizing constant of the posterior distribution for example the following $$\pi(\theta|\boldsymbol{Y}) = \frac{L(\boldsymbol{Y}|\theta)\pi(\...
0
votes
0answers
18 views

Probability of getting the bus at a fixed time

I am trying to solve an exercise that says: Your bus will arrive at a random time between 12:00 and 13:00. If you arrive at 12:30 at the bus stop, what is the probability of taking the bus.
0
votes
0answers
8 views

Convert datetime in dataframe to seconds [on hold]

Below is the output of [mydataframe].info() ...
0
votes
0answers
7 views

Varying predictions on validation set with LSTM network

I put together a model using Python/Tensorflow to do binary classification based on time series data. The model is working fairly well, with about 95% accuracy on the training set and ~80-85% on the ...
0
votes
0answers
6 views

statsmodels.tsa.arima_model.ARIMA.fit throws all kinds of errors

I'm trying to forecast time series using ARIMA. For low arima orders (p,d,q) the model runs. But when p is larger than ~15 , ARIMA.fit throws all kinds of errors, e.g. ...
0
votes
0answers
5 views

Parameter estimation of time-dependent R.Vs(random processes),

I have taken a number of probability and statistics courses which cover estimation and basic random processes but something which is not clear is how you can do parameter estimation for time-dependent ...
2
votes
0answers
19 views

Probabilities : One-tailed test implication for two-tailed test?

I had an exam with a question (multiple choice) that had this question: If I perform a test with a significance level of alpha = 0.05 and reject H0 for a one-sided test, it means that: A) I reject ...
1
vote
0answers
6 views

Boosted regression trees - clarification of algorithm step

In my statistic learning textbook, there is an algorithm, "Boosting for Regression Trees". As step 1 in the algorithm, it is said to set $\hat{f}(x) = 0$ and $r_i = y_i$ for all $i$ in the training ...
0
votes
0answers
9 views

Stata Outreg2 Horizontal Alignement

I am using Stata's outreg2 to produce LaTeX tables. I am running several regression and appending those. The default alignment of these regressions by using outreg2 is vertical (one column represents ...
0
votes
0answers
9 views

How to compare the training performance of a deep learning model on different data sets?

So I have a deep learning model and three data sets (images). My theory is that one of these data sets should function better when it comes to training a deep learning model (meaning that the model ...
0
votes
0answers
6 views

Multivariate hypothesis testing for proportions

It's very common in website optimisation to run A/B tests for conversion rates (e.g. 'buy' button is green by default, would the conversion rate from 'visit' to 'buy' be significantly larger if we ...
1
vote
1answer
15 views

Framework for reducing the dimension of the features of multiple correlated Time Series with a notable amount of memory

I have a dataset that I am trying to analyse that consists of multiple (~500) time series each with around 25 observations. For each observation I have a large number of covariates, some of which will ...
0
votes
0answers
6 views

Econometric test to examine the Pre-and-Post of policy on macroeconomic variable

Question: I am testing the impact of implementation of a policy on the Economic Growth Rate, and thinking to hire some tests that can examine if the policy has a significant impact on the growth rate. ...
0
votes
0answers
18 views

Generalizing the relationship between cross-entropy and maximum loglikelihood [on hold]

Let's say we are trying to learn a distribution $p(\cdot)$ by searching through a parametric family $p_\theta(\cdot)$. We can have access to $p(\cdot)$ either analytically or only through samples, and ...
0
votes
0answers
8 views

Testing regression assumptions on panel data in R

I'm studying the relationship between student retention rate (dependent variable) and institutional expenditures (independent variables) across 4 categories - instruction, student services, academic ...
1
vote
1answer
11 views

PCA returns the same pair of principal axes for completely different 2D datasets [duplicate]

I noticed a (seemingly) weird behavior while using sklearn's PCA on 2D standardized datasets: I kept getting the same principal axes: $\pm\left(\begin{gathered}\sqrt{0.5}\\ \sqrt{0.5} \end{gathered} \...
0
votes
2answers
50 views

Are my stock returns non-normally distributed?

Hello stats community, I have a Q-Q plot for my stock returns with a sample of n=262. I drew the plot with qqnorm and qqline(qtype=8). Most of the returns, except for 3 outliers, tend to follow the ...
0
votes
0answers
8 views

How can I interpret Generalized Ordered Logit model output?

As the title may suggest, how can I interpret the coefficients from generalized ordered logit model output? I know a little bit about interpretation of logit model, not generalized ordered logit model ...
1
vote
1answer
20 views

Conditional Expected Value

I'm trying to figure out the following case; suppose that I have a supermarket where I give points to my customers for they to redeem (1 point for 1 dollar). Once in a while I send offers to my ...
0
votes
1answer
11 views

Any way of holding the other variable constant at mean, not at zero for interaction?

Consider the following example in R: lme4: glmer(score ~ VarA + VarB + VarA:VarB) I'd like to test the main effect of each variable by holding the other ...
1
vote
1answer
40 views

Can neural networks learn $g(x)$ from $\mathbb{E}[g(X_t)] = \int_{-\infty}^{\infty} g(x)p_t(x)dx$

Let $\mathbb{E}_x[g(X_t)]$ be the expected value of a random variable $X_t$ with known probability density $f_t(x)$ then for the continuous case $$\mathbb{E}[g(X_t)] = \int_{-\infty}^{\infty} g(x)...

15 30 50 per page