Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [python]

Python is a general purpose programming language designed for ease of use. It is a commonly used platform for machine learning. Use this tag for any *on-topic* question that (a) involves `Python` either as a critical part of the question or expected answer, & (b) is not *just* about how to use `Python`.

0
votes
1answer
9 views

Tools in Data Science

I am a new one in Data Science and Machine Learning. I have some experience in Java, Python, SQL, Jupyter and most polular libraries like scikit-learn, numpy, pandas, tensorflow, keras. What tools ...
0
votes
0answers
6 views

Regression via neural network on training data with uncertainties

I have a regression task where all my training data that I want to predict is of the form ($y$, $\sigma$), where $\sigma$ is the Gaussian noise corresponding to $y$, i.e. I want to be able to predict ...
2
votes
1answer
27 views

Statistically compare similarity between images

I have two images/heatmaps (2d matrix) of identical size. I need to statistically compare the similarity between the two. With 'similarity', I mean that high and low values of one image appear in ...
1
vote
1answer
26 views

Improve precision/recall for class imbalance?

Trying to get better precision/recall for both classes ... any tips? I have heterogeneous features [a few num vars, a few cat vars, and 2 text vars] Target is a binary classification w/ class ...
1
vote
0answers
39 views

Wrong fitted parameters in multivariate linear regression?

I am implementing multivariate linear regression using numpy, pandas and matplotlib. I then ...
1
vote
1answer
11 views

Negative Feature Importance Value in CatBoost LossFunctionChange

I am using CatBoost for ranking task. I am using QueryRMSE as my loss function. I notice for some features, the feature importance values are negative and I don't know how to interpret them. It says ...
0
votes
0answers
11 views

Python: Compute Bootstrap Confidence Intervals for a Kernel Density Estimator [on hold]

I have plotted the probability density function of some data, using a Kernel Density Estimator. Both the data, and the code I am using can be found here As you can see, the txt-file contains a lot ...
2
votes
1answer
333 views

How to write diff() mathematically?

I am using R/Python diff() operation. e.g., https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.diff.html I would like to know if there is a ...
0
votes
0answers
9 views

Using EM algorithm to estimate Markov Switching Vector Autoregression but failed to converge [migrated]

Can anybody help me figure out the problem in my code? I write it to estimate the MSVAR model using the EM algorithm proposed by Hamilton J.D. 1990. Basically There're three parameters to estimate. ...
0
votes
0answers
5 views

Importance of tests summary in multiple regression using statsmodels

Greetings to the community. I have a practical question regarding the interpretation of the summary statistics that you get from statsmodels when you fit a model. I have e.g.,: model = smf.ols(...
0
votes
0answers
12 views

Uber's Pyro less accurate than expected on toy example

Trying to understand the pyro example here: https://pyro.ai/examples/svi_part_i.html which starts with a Beta(10,10) prior, adds 10 Bernoulli likelihood datapoints with a 6,4 split. The analytic ...
0
votes
0answers
14 views

Encoding quantitative outputs for regression

From Elements of Statistical Learning: For a two-class G, one approach is to denote the binary coded target as Y , and then treat it as a quantitative output. The predictions Yˆ will typically ...
2
votes
2answers
34 views

How effective is SVM over big datasets?

I have a dataset of 800,000 observations and 11 features that I am using for a classification problem. I tried to optimize my model many times but in vain. The one thing I haven't tried is using SVM. ...
2
votes
2answers
33 views

Randomized search on big dataset

I have a dataset of 700,000 rows that Im applying random search on. My parameter grid looks like this: ...
0
votes
0answers
24 views

what is a vector perpendicular to a plane of vectors

I have 3 or 4 vectors connected that forms a plane. How can I find the vector that is perpendicular to this plane? it can be a unit vector as long as it preserves this direction. each vector is on ...
0
votes
0answers
31 views

McNemar or Cochran's Q for evaluating multiclass classifiers?

full disclosure: I did a semi-cross post of this question due to low traffic. Once I get an answer on any of the two questions, I will link the answer back to the respective other. tl;dr For ...
0
votes
2answers
57 views

Two sample quantile-quantile plot in Python [closed]

Suppose I have two samples from unknown distributions, and I want to produce a quantile-quantile plot of one against the other. What's the easiest way to do this in Python? There is scipy.stats....
1
vote
0answers
14 views

Is there a way I can plot my data as an animated graph with respect to time? [closed]

I am looking to create an animation that plots data onto a graph so it appears on screen gradually. My data comes in the form of a row vector, where the actual values in the vector will be plotted ...
0
votes
0answers
15 views

Matrix formation and calculation for Collaborative filtering in Neural Network

Intution I am trying to implement Collaborative Filtering (User Based and Item Based) in Python Keras with neural networks. For user based CF with neural network, my input is rating matrix with ...
0
votes
1answer
7 views

Array of samples from multivariate gaussian distribution Python

I am trying to build in Python the scatter plot in part 2 of Elements of Statistical Learning. First it is said to generate 10 means mk from a bivariate Gaussian distribution N((1,0)T,I) and ...
0
votes
0answers
12 views

Bokeh create many circles by tap with different size [closed]

I want to create many circles by tap or double-tap, select and change size of each circle created by a Slider widget Thanks,
1
vote
1answer
44 views

Regression on Auction Prices, Multiple Prices Randomness

I'm currently building a model to predict internet auction sale prices of products in a marketplace. There are a lot of instances where a product goes for multiple prices but it's basically the same ...
0
votes
0answers
16 views

Estimation of Stationary Vector Autoregressive Processes with R or Python [closed]

What well-documented R or Python package would you recommend that deals with penalized estimation of stationary vector autoregressive processes? I am particularly interested in the lasso but open-...
0
votes
0answers
5 views

Routine for Quantile LOESS in Python [closed]

is anyone aware of the existence of a routine for performing Quantile LOESS in Python?
0
votes
0answers
9 views

Autocorrelation of stochastic process with python

So I am trying to simulate a SDE and find the corresponding correlation function. The equation is $$x=\beta +\sqrt{2D} \xi(t)$$ where $\xi(t)$ is a white noise term. After solving it using Euler-...
2
votes
0answers
42 views
+50

Is there any strategy for validating the result of a general comparison between several confusion matrices?

Disclaimer: Recently we have developed a python library named PyCM specialized for analyzing multi-class confusion matrices. A compare system has been added in <...
1
vote
1answer
7 views

Is it appropriate to use a confusion matrix for frequency data?

I am testing the accuracy of a machine learning approach that counts cars in images. I have both a predicted dataset and a "real" dataset that was generated by a human. For example, this is what my ...
0
votes
0answers
12 views

correct setting of eval_set in multiclass classification xgboost python , error is “ Check failed: preds.size() == info.labels_.size()” [closed]

i have a multiclass classification problem with 3 classes [-1,0,1] . i'd like to use eval_set in xgboost. but it fails with error: ...
0
votes
0answers
3 views

Truncate a Hierarchical Clustering tree in order to get the cophenetic coefficient

I've gone ahead and clustered a dataset using a Euclidian Hierarchical Clustering algorithm: ...
1
vote
0answers
20 views

How does one estimate error of extrapolated value with only 2 data points?

I am trying to do a simple linear regression. I have only two data points with errors. How do I estimate the y-intercept with errors? Currently I have: ...
0
votes
1answer
29 views

Chi Square Analysis Throws Error - The internally computed table of expected frequencies has a zero element at (0,)

I am trying to see the association between two variables. I used Chi-Square analysis in Scipy package in Python. Here is the crosstab result of the two variables: ...
0
votes
0answers
48 views

Kalman Filter with MLE giving bad estimates

I am trying to learn and implement the Kalman filter. Yesterday I successfully implemented a non linear kalman filter of the form: $$ x_t = a(x_{t-1}) + u_t \\ y_t = Gy_{t-1} + v_t $$ $u_t$ and $v_t$...
0
votes
0answers
12 views
0
votes
1answer
14 views

How Do I get 0 and 1 For multiclass multilabel problem in Keras prediction?

Let's say I have 3 classes, and each sample can belong to any of those classes. [ [1 0 0] [0 1 0] [0 0 1] [1 1 0] [1 0 1] [0 1 1] [1 1 1] ] I set my output ...
0
votes
0answers
14 views

ACF and PACF Interpretation 12 month difference/seasonality

Looking for help interpreting the following ACF and PACF plots, for a time series SARIMAX model analysis (p,d,q) and (P,D,Q,12). I have searched online and have not found an example that explains it ...
0
votes
1answer
15 views

How to tune the hyperparameters for oneclass SVM while doing unsupervised learning?

For my task, I am doing unsupervised learning and I am trying to find the best possible value of the parameters gamma and nu in ...
0
votes
1answer
18 views

How can I use scaling and log transforming together?

I'm creating a regular linear regression model to establish a baseline before moving on to more advanced techniques. I scaled my data as below: ...
1
vote
1answer
46 views

Monthly Times Series Modeling Approach

I have a machine learning problem and have been working in Sklearn/Pandas with Python to come up with an accurate model. I find myself deep in a rabbit hole trying to learn the best approach and how ...
0
votes
3answers
14 views

How to determine p(a) in bayes' formula with this example of symmetric conditional probabilities?

Why is P(A) in this example 1? In class, we were shown an illustrative example of bayes that no one including the professor could understand. P(A|B) = P(A) * P(B|A) / P(B) Question is: Use Bayes ...
0
votes
0answers
19 views

Overlap of Two Probability Distributions in Python

So I saw someone use this code to find out how much overlap two normal distributions have. ...
0
votes
1answer
44 views

Confusion between using ratio, bayes theory, or others

I am confused as to what I should be using to derive an appropriate metric. The objective here is to find out if knowing if a person has X, Y, Z can help indicate if they are in the binary bucket A or ...
0
votes
0answers
8 views

Got very poor accuracy whenever I train any deep neural network [duplicate]

I am trying to train a network on Alabone dataset downloaded from "UCI machine learning repository" site. The dataset is look like: ...
0
votes
0answers
18 views

Prediction Challenge AI. Machine learning and data analysis.(Inconsistency data)

I am working on a personal rento in which based on some input data I have to predict some output data. The challenge is to predict the expenses in transactions, receipts and cards that users will have ...
0
votes
0answers
6 views

Autoencoder - reconstructed image not matching the input image

I have trained a convolutional autoencoder on cifar10 dataset. The reconstruction loss on the test data is quite less (around 0.0225). However, the reconstructed training images do not look like ...
0
votes
0answers
17 views

Doing hyperparameter optimization, the smart way

Once our machine learning model is build, it takes quite a while to fine tune the hyper parameters to get good results. Though techniques like cross validation do exist, I wonder if there are some ...
0
votes
0answers
21 views

How to proceed when testin for non-linear relationships between variables

I'm trying to predict the amount of money a customer spends on a product by using linear regression. I had the hypothesis that there was a non-linear connection between the Product_Category, the ...
0
votes
0answers
27 views

Calculating R^2 for a linear mixed model in python

I have been looking around online in regards to R^2 calculations in mixed models and a lot of info has come up in R (lme4, MuMIn) where the lme4 package creates the mixed model fit and MuMIn ...
1
vote
0answers
28 views

Wrong calculation of feature importance of decision tree in R [closed]

I trained decision tree both in python and R, but I think the way feature importance is calculated in R may be wrong. Following is the sample code which you can use to reproduce the problem. Let's say ...
-1
votes
0answers
44 views

Unbalanced classes multiclass

Certain ML algorithms have parameters which can be used to deal with the effects of unbalanced dependent variable classes. For example the random forest implementation in Sci-kit learn has the class ...
1
vote
1answer
35 views

Bayesian approach to interval prediction?

I am working on a problem involving understanding/predicting customer frequency. The data I am working with is structured as a series of interval days between orders: ...