22
It is very difficult to achieve what you want programmatically because there are so many different forms of nonlinear associations. Even looking at correlation or regression coefficients will not really help. It is always good to refer back to Anscombe's quartet when thinking about problems like this:
Obviously the association between the two variables is ...
7
As mentioned in the comments to the question, you are interested in inference, not prediction - that is, you are interested in how the explanatory variables influence the outcome.
Any kind of stepwise procedure, and any other automated method of variable selection (eg the lasso or other regularised procedure) will not work as it cannot account for bias due ...
6
I agree that this is confusing.
Manifest variables and "observed" variables are both observed, in the sense that they are part of the input dataset, and not latent variables.
The distinction made in the semopy package is that manifest variables are part of the measurement model (ie the definition of the latent variables), whereas "observed&...
5
Linear/nonlinear should not be a binary decision. No magic threshold exists for informing the analyst things like "definitely linear". It's all a matter of degree. Instead, consider quantifying the degree of linearity. This can be measured relative to explained variation in Y be two competing models: one that forces linearity and one that doesn'...
5
Are there any mathematical drawbacks with this idea?
Yes, you will lose statistical power as well as running into multiple testing problems. Don't split the dataset.
How do I compare the resulting n models to my other model? I am unsure on how to interpret the n resulting metrics (i.e., a cross-validated r squared score) to those of my other model.
You ...
3
The biggest problem you have here is that "non-linear relation" is not well defined. If you allow for any non-linear relation, there's basically no way to tell if something is "completely random" or just follows a non-linear relation that looks exactly like something that might come out of a "completely random" set up.
However, ...
3
XGBoost uses a diagonal approximation to the Hessian. A diagonal $n\times n$ matrix has at most $n$ nonzero elements. The diagonal approximation scales nicely, because it only grows linearly in $n$, as opposed to the dense Hessian which grows quadratically.
The diagonal approximation is the best when the off-diagonal elements are close to zero. However, even ...
3
Comment: You're having trouble with your second simulation because the 'span' of several uniform random variables is is not uniform.
In R, the function range gives the endpoints, taking the difference gives
what you call the 'span'.
Here is a simulation in R of the span lengths from 100,000 samples of size $n=5$ from $\mathsf{Unif}(0, 3).$
set.seed(2020)
...
3
Nice question.
I have more to say tomorrow, but for now the main point here is that your predictor matrix, changepoints is just a weighted matrix of piecewise linear basis functions, where the weights are set to fit the 25 equally-spaced points in your data.
It is common enough to use lasso to penalise regression splines, although other methods are more ...
3
I do not know Python, but as you can readily illustrate in R, setting the value of the intercept to 1 is really just a convention (a useful one, though, of course, allowing us to interpret the intercept as the expected effect when $x=0$).
n <- 10
y <- rnorm(n) # some random data
x <- rnorm(n)
intercept <- rep(1,n) # a "hand-...
2
Your gradients and update rules are correct. It's just you're using a large learning rate for your data, because your gradients are large. Try $\alpha=10^{-5}$ and $100$ iterations. You'll see that it'll converge.
2
XGBoost does not have "a clearly defined DoF" unfortunately.
Degrees of freedom are not well-defined for regression trees. This is even more pronounces in the case of gradient boosters where we use potentially hunderds of trees. Adding in the fact that we also regularise our fitting procedures both within a tree and across trees makes the whole ...
2
Sycorax is right to point out that “accuracy” has a specific, technical meaning in machine learning (and it is a surprisingly bad performance metric). However, if you mean “accuracy” more colloquially, what you mean makes sense. You want to see how good each particular measurement is.
You have the true observation, and you have the predicted observation.
...
2
RandomForestClassifier is a fundamentally different model than RandomForestRegression. The key difference is that the regression model is predicting some continuous value (e.g. a predicted profit/loss, or an estimated height, or something similar). The RandomForestClassifier predicted probability is the estimated probability that the sample belongs to each ...
1
All scipy distributions have a scale and loc parameter too. So, as scipy arguments:
$$a=\gamma, b=\delta, \text{scale}=\lambda, \text{loc}=\eta$$
The constraints look identical to me between scipy and wikipedia, as well.
1
Enrico Fermi claimed that John von Neumann said:
With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.
On that basis, the 4-parameter unbounded Johnson distribution provides a way to transform an elephant into a standard normal distribution. The statistical question here is whether that's worth doing.* In this case, it'...
1
To fit so much data, you have to use subsamples, for instance tensorflow you sub-sample at each step (using only one batch) and algorithmically speaking you only load one batch at a time in memory it is why it works. Most of the time this is done using a generator instead of the dataset right away. Your problem is that you always load the whole dataset in ...
1
Suppose you have a composite z-score that is roughly normal with mean approximately 0 and standard deviation approximately 1. Then transforming with the standard normal CDF $\Phi$ (pnorm in R) will give you scores that are approximately uniformly distributed on $(0,1).$ Multiplying by 100 will give you 'index' scores between $0$ and $100.$
Here is an example ...
1
It's not surprising, Epanechnikov kernel is negative outside the bounds and kernels cannot return negative values (see example in Julia below). When you combine the negative and positive "density" values in kernel density estimation, you get rubbish results. Restricting the bounds in important in here, only Gaussian kernel is an exception as it is ...
1
Since the data are nested, the answer is technically no.
2, and a bit of 3. Straight forward depends on your experience, but here is one approach.
This sounds like exactly the right setup for a mixed effect model. Let's work bottom up.
For subject $j$, we assume that the logit of the success rate is
$$ \operatorname{logit}(p_j) = \beta_{0,j} $$
Here, we ...
1
Two datasets that may be essentially the same for practical purposes, but are
found to be different by Welch two-sample t test:
set.seed(904)
x1 = rnorm(10000, 1001, 15)
x2 = rnorm(10000, 1002, 15)
t.test(x1, x2)
Welch Two Sample t-test
data: x1 and x2
t = -4.6203, df = 19998, p-value = 3.856e-06
alternative hypothesis:
true ...
1
It's a simple transformation of the MAPE, so it's as "statistically correct" as using the MAPE in the first place.
Note that the MAPE may exceed 100%, so your accuracy may become negative. This can be disconcerting, and lead to strange behavior by forecast consumers, like truncating accuracy at 0%, which is not a good idea.
You may find this thread ...
1
While it is true that your original data can be reconstructed from the principal components, even if you didn't center the data when calculating them, part of what one is usually trying to do in principal components analysis is dimensionality reduction. That is you want to find a subset of the principal components that captures most of the variation in the ...
1
PCA is done on the centered data matrix in the first place (because $X^TX$ is the (scaled) covariance estimate if $X$ is centered). So, while fitting, $X$ is centered internally. The same mathematical operation takes place while transforming new data points.
1
As it turned out, I misunderstood the concept of factor loadings. I tought that it's the correlation coefficient between latent and observed variable, but it's standarized regression coefficient instead and there is nothing wrong in value -1.17.
1
I think I found a solution! Using a different scipy algorithm there is a solution to the optimisation. The optimised factors provide a good fit for the model.
Here is the code:
k = 0.00000000000001
sigma = 0.00000000000001
T = np.array([1, 3, 6, 7, 8, 9, 10, 11, 12, 13, 16,20,24,28,32,34,35] , dtype=int) #maturities
x = k, sigma
#objective function
def ...
1
multiply (get the dot-product of) Xtrain and the weights that the lasso model found to be optimal. This will give you a lasso model-weighted version of Xtrain that now weights the entire sample set of each individual feature column anywhere between 0-100%. Now run whichever multicollinearity tests as you normally would on the matrix such as its correlation ...
Only top voted, non community-wiki answers of a minimum length are eligible
