New answers tagged python
0
How about using v-measure? It is a symmetric measure
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.v_measure_score.html
You might also want to read more about homogeneity score and completeness score.
4
Simply invert the distribution function. It's efficient.
Here are the details.
A standardized Lomax distribution with shape parameter $\alpha\gt 0$ has distribution function (CDF)
$$F(z;\alpha) = 1 - \frac{1}{(1+z)^{\alpha}},\quad x \ge 0.$$
Its quantile function is thereby straightforward to find as
$$F^{-1}(q;\alpha) = \left(1 - q\right)^{-1/\alpha}-...
1
When the gradient vanishes for a particular layer it means that the gradient of the update for that layer is very close to 0, or actually is 0. When the gradient is 0, this implies that the weights do not update. When the weights do not update, the distribution plots will be the same. So as a heuristic, if you see that the distribution plots remain the same, ...
answered Feb 20 at 20:48
Sycorax says Reinstate Monica
53.5k16 gold badges130 silver badges236 bronze badges
0
I find it useful to search github when I want to find a particular function that isn't in the usual places I'd look:
https://github.com/search?q=cliff+delta
This result looks like your best bet
https://github.com/neilernst/cliffsDelta
This is code to calculate the Cliff's Delta effect size metric, which strangely is not in NumPy/SciPy.stats.
1
The prediction that you are using is the expected value, i.e. the conditional expectation E(y | x)
This is the same as in standard count models like Poisson and in other models like GLM and linear models, where predict also returns the conditional expectation of the response variable.
y_test_pred = res.predict(x_test, exog_infl=x_test)
Here is a notebook ...
0
I would suggest you to get more data and build a binary classifier.
First combine all of your data into 1 class (recyclable)
Then craw internet and find items not recyclable. We should found items that people usually throw in the trash. But not trying to get irrelevant images such as cats and dogs.
0
In a well designed lottery system, lottery numbers should be randomly generated and it should not depending on any variables.
So, if the range of the ball is from 1 to 32, I am expecting to see it is uniformly distributed, and have nothing todo with other variables such as date.
And if we really want to guess what number will be next, we have no choice but ...
0
There are a couple of issues in this example that lead to the strange-looking predictions.
Non-stationary (i.e. explosive) behavior
Short time series
The first problem is that the autoregression model assumes the data fluctuates around a stable mean value. This is not true of your data, which appears to increase something like exponentially.
The second ...
5
In general, your approach may work, and it might even give you something that works somewhat well. However, I would strongly advise against it, or only use something like this as a first step to just get a feel for the problem.
Think about it this way: If you just shift one of the images one pixel to the left, how much would the vector representing that ...
4
If you final goal is using SVM, the problem is number of data points instead of the number of dimensions. See following question.
Can support vector machine be used in large data?
In real world SVM will not work very well if you have ~10K data and above.
Your problem is a standard image classification problem using convolutional neural network CNN may be ...
2
Linear regression against time ASSUMES a model form and uncorrelated residuals to actually test the significance of estimated parameters. Your series might be adequately described with a local time trend (NOT GLOBAL) and a few level shifts and possible pulses and a possible memory component (arima) but only your data knows for sure. Post your actual data and ...
1
If you want to know the direction point to point, I believe that you can evaluate easily the slope: $\frac{Y_{t+1}-Y_{t}}{x_{t+1}-x_t}$. If this result is positive, it is going upwards. On the other hand, if it is negative it is going downward. If it is zero, it is going sideway.
If your question is about to know the average direction, you should estimate a ...
3
What do you exactly mean by upwards, downwards or sideways?
You could fit a linear regression model on the data. You will get a coefficient for the slope and a confidence interval. If the coefficient is not significant then you could say that there is no trend.
0
To ignore the effects of the PADs(in PyTorch) I thought we can try two things:
First. use the states of the encoder(rather than the final state, c and h) for each character and take a summation or mean(or any manipulation) of them to embed the sentence, which can be implemented very easily by multiplying a mask of the sentences.
For illustration please ...
0
We can see/learn from the implementation of the bidirectional dynamic RNN in TensorFlow that the backward LSTM was just the reversed input(or forward input), then we can reverse the sequence and do padding. Once we get the states we just reverse them back and do masking to mask out the gradients for the pads.
Once the mask values for the pads are zeros the ...
0
You may select features considering the importance of the features for out of sample prediction. I suggest two very interesting and general methods that you can use:
1) Permutation importance: The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled
2) Shap values: It is not an easy ...
2
R uses an orthogonal basis expansion while PolynomialFeautres does not. Try passing raw=TRUE in poly. What are the results?
0
If your only goal is to make a transformation of your data to normal data, you can simply use the inverse formula.
If you denote by $X$ a random variable with cdf $F$, then $F(X)\sim U$ where $U$ is a uniform distribution. If you denote by $G$ the cdf of a standard normal distribution, you can then obtain normal data via
$$ G^{-1}(F(X))= G^{-1}(U) \sim N(...
0
You could just perform 5 independent regressions. If you use scikit-learn, you can do it in one step as LinearRegression supports multiple target variables, as documented
here.
3
The only way to accomplish this is through constraints on coefficients. This will not be a standard linear regression though. It'll be similar to regularization except the shrinkage coefficient will be a vector, not scalar.
For instance, you define a problem as a typical least squares with a penalty on coefficients:
$$\min_{\beta_j} \sum_i(y_i-\beta_0-\sum_{...
1
It sounds like you are checking each predictor separately against the binary outcome. That's not a good idea with logistic regression, as is has an inherent omitted-variable bias. Omitting from a logistic regression any predictor associated with outcome will bias the coefficients for the included predictors. Unlike in linear regression, the omitted predictor ...
0
The outlier has an OK absolute value (on the right hand side there are normal values just as high), it is only anomalous because it deviates significantly from the time steps around it. When only passing in single samples to the anomaly detector this is impossible to learn, instead have to use some time window.
This can be as simple as using the previous N ...
0
Using latent regression and person covariates is common in IRT; especially in large-scale assessments such as PISA. In PISA (and all others) plausible values are drawn from individuals posterior distributions in order to calculate population statistics containing measurement error components (for starters, see Wu, 2005, The role of plausible values in large-...
1
This is implemented in the POT: Python Optimal Transport package, for samples (or, generally, discrete measures): use ot.wasserstein_1d.
If you want to do it for weighted samples (or general discrete distributions with finite support), you can provide the a and b arguments.
2
So I think if I understand you correctly, what you are trying to do is to sample points from $B$ according to the distribution of $A$. In which case I would take a different approach to what you have laid out here.
First of all you talked about label encoding $A$ and then doing a KDE. I don't think this makes sense unless your categorical variables can ...
0
So to start with there are some default options inside TfidfVectorizer() that you are using without realising. To make things line up with what you expect you should use
tfidf_vectorizer = TfidfVectorizer(norm=None, smooth_idf=False)
Using this option the score computed will be
$$s_{ij} = tf_{ij}(1+log(N/df_{i})$$
where $s_{ij}$ is the score for the word ...
0
Not sure exactly what kind of model you want to use to predict the time series, but one method of doing it might be to use a SARIMA model with exogenous dependent variables.
See this blog post for some more details https://machinelearningmastery.com/time-series-forecasting-methods-in-python-cheat-sheet/
1
The KS test will (assuming less) test for the $H_1$ that the CDF of x lies below that of y. On the other hand, the MW (assuming less) will test for the $H_1$ that the location shift is less than 0. The results reported are perfectly aligned with that as they indeed suggest that "$x$ is smaller than $y$". The KS fails to reject its null hypothesis in favour ...
1
Unfortunately the implementation of random forests between Python and R are not always directly comparable. If anything, because the random forest algorithm inherently performs bagging and random selection of explanatory variables (i.e. it samples both the rows and the columns of our training set when training), if this resampling is not done in the same ...
1
Regarding your first question:
Sigmoid gives a valid probability distribution only in the binary classification case. If $z$ is the pre-activation of the output layer and $N$ the number of output neurons, then
$$ \sum_{i=1}^N \sigma(z_i) = \sum_{i=1}^N \frac{1}{1+e^{-z_i}}$$
which has no machanism that forces it to sum to $1$, since $\sigma(z_i)$ doesn't ...
0
I have encountered a similar issue where the OLS is giving different Rsquared and Adjusted Rsquared values compared to Sklearn LinearRegression model.
Reason for it: OLS does not consider, be default, the intercept coefficient and there builds the model without it and Sklearn considers it in building the model.
Solution: Add a column of 1's to the dataset ...
1
proportions_ztest seems to work exactly as documented.
Unfortunately what the documentation says it does is just not what you're expecting it to do.
By default this function uses the sample proportion in calculating the standard error of $p-p_0$. There's an option (via a boolean function argument) to change that default, after which the output should ...
answered Feb 3 at 1:49
Glen_b -Reinstate Monica
232k25 gold badges470 silver badges823 bronze badges
2
There is a lot to address here, and while I appreciate you providing so much detail, I will only focus on some of the most important issues I see.
So, the first thing I notice from looking at your model structure is that your model is perhaps misspecified. For one, the beta distribution is defined only for positive $\alpha, \beta$. However, your ...
1
Plotting a seaborn distplot needs an adjustment, as it is primarily meant for continuous distributions. The distplot will put the data in 16 equally size bins, that don't align with the integer numbers. For discrete distributions, distplot would need explicit bins, e.g. range(30). However, with that many bins, the default calculated kde will not be as ...
4
In the absence of enough data to perform stacking, just simple model averaging might be adequate for predictions. It will (typically) offer some improvement in terms of prediction errors. Dormann et al. (2018) "Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference" offers a great and ...
1
You're asked to calculate $P(7\leq X \leq 13)$, but you only find $P(X=7)$, where $X\sim \text{Bin(n,p)}$ denotes the number of low-birth weight babies.
So, your script should be:
sum((binomial>=7) & (binomial <= 13))/runs
And, it's better to increase your number of runs to decrease your estimate's variance. You can also calculate it ...
1
There is no "good", or "bad" scores, they are not interpretable. Those scores are used only for sorting the recommendations, where the higher score means the more recommended the product is in given setting.
As you can read in the Collaborative Filtering for Implicit Feedback Datasets paper by Hu et al, the algorithm minimizes
$$
\min_{x,y} \sum_{u,i} c_{...
0
You might be interested in TensorFlow Probability. It has a Python API, and has been chosen to replace Theano as the PyMC3 backend at some point in the future. Tensorflow Probability can also be used for MCMC directly, and it has dedicated functionality for Bayesian structural time series modelling. There is a nice blog post which provides an introduction.
0
It is possible for a-learning to stick in sub optimal loop just like your case where the best action to take in state A take you to state B and the best action in the state B takes you back into state A.
I found two solutions for this so far:
Keep track of the path you take and avoid returning to the same state or give negative reward if it returns to the ...
0
Since you have only the total number of citations at time T (for example, as of today), it makes sense to adjust by the age of the patent to account for the fact that older patents had more time to receive citations. One complication is that the number of citations a patent receives is not a linear function of time. Some patents might receive a lot of ...
3
Fitting 10K GLMs will be slow. A 2x2 Chi-squared test for each comparison of two event counts will be faster and will work well enough provided your event counts don't get too extreme, e.g. your event counts range from [5, 495] for your sample size of 500. You can work with the event counts directly so you won't need to create the vectors of zeroes and ones. ...
Top 50 recent answers are included
