New answers tagged

0

Adding to Josh that using some tree based algorithm is lacking some clear transparency on how the features are impacting your sales (like you have in a classic linear regression framework, which is called a white-box model, because the coefficients are very easy to intertrep). What you can get as relative importance from these tree based algorithms, which ...


1

When considering $$KL(P||Q) = \int_{-\infty}^{\infty} p(x) \log \left(\frac{p(x)}{q(x)}\right)~\text{d}x$$ it can be expressed as $$KL(P||Q) = \mathbb E_p\left[\log \left(\frac{p(X)}{q(X)}\right)\right]$$ meaning it is the expectation of a transform of the random variable $X$ with density $p$ (or distribution $P$). There is therefore no assumption of $X$ ...


0

I realise this is a little late to the party, but I just had a similar issue with a logistic regression model of my own and wanted to understand why it wasn't at least orienting the decision plane to include more data-points and/or basing the minimum misclassification boundary more favourably. I was using iterated reweighted least squares fitting (Rubin, ...


0

One way to do this is to calculate pairwise similarity (or distance) between each of the samples using some metric. This gives a high-dimensional matrix which preserves a lot of the important information. In order to then use this, use some dimensionality reduction technique (e.g. PCA) --- because there is a high degree of dependency across the variables the ...


1

@RNG's answer is brilliant, so this is just to give a more conceptual answer. The most confusing thing when comparing estimation and information theories (to me) is that they have opposite definitions of information, Fisher information and Shannon information (entropy). They're opposite in the sense that as one goes up, the other tends to go down. This is ...


0

So I've discussed this with a colleague. Consider the marginalisation $$p(\mathbf{X} \mid \boldsymbol{\theta}) = \int p(\mathbf{X} \mid \mathbf{Z}, \boldsymbol{\theta})p(\mathbf{Z} \mid \boldsymbol{\theta}) d\mathbf{Z}.$$ This can be rewritten as the expectation $$\mathbb{E}_{\mathbf{Z} \mid \boldsymbol{\theta}}\left[ p(\mathbf{X} \mid \mathbf{Z}, \...


1

Sensitivity is $\text{TP}/(\text{TP+FN})$, and specificity is $\text{TN}/(\text{TN+FP})$. If specificity is $1$, there is no false positives. The sensitivity figure yields $0.175\times\text{TP}=0.825\times \text{FN}$. Even if you know the total number of samples, $\text{TP+FN+TN}=n$, you'd still need to know the number of true negatives in terms of others. ...


1

I do not think that there is a clear answer without looking at the specific data. But if you want to build a time-series model using linear regression methods, I suggest to try build some features that capture some seasonality like year and month encoding (maybe using a dict saying: 'January':1, 'February':2, .... and for year: '2020':0, '2021':1, ....). On ...


0

In theory you can use also all algorithms having categorical data. You would need to encode them in a way the algorithm can read the information. This is often done using one-hot-encoding, but this can be very computational expensive if your data set is big. Have you tried xgboost and catboost? Especially catboost was developed handling categorical features ...


0

It is not so clear to me how you want to split your data in your second approach. Maybe I can suggest the following: Uee k-fold cross validation for hyperparameter tuning. You take your data set and split 80-20 (90-10 or 80-30 would also work). The 80% you use for tuning using cross-validation. So you split your 80% in k many train-validation splits and try ...


0

You may also want to check the normalized version of Damerau–Levenshtein distance. It is technically a string metric for measuring the edit distance between two sequences. However, in this case, it can be used as a ranking metric. A python package called pyxDamerauLevenshtein implements this distance metric.


6

Remember that the error terms in the regression measure the deviations of the response variable from its conditional mean (given knowledge of the explanatory variables). Indeed, under the stipulated model form for regression, this is essentially the definition of what the error terms are. So you have: $$\varepsilon_i \equiv Y_i - \mathbb{E}(Y_i|X_i).$$ ...


2

SGD provably converges for convex functions. However, people discovered that it's also pretty decent for training neural networks which have very non-convex high dimensional loss functions. Soon, people discovered different tweaks and additions to SGD which make it even better at training neural networks. One of these tweaks is called ADAM. These tweaked ...


0

I'm going to add onto Brale's answer which I think is mostly right. The part that's slightly incorrect is to talk about the bias from initialization of the value neural network. I think it makes more sense to talk about the ideal value neural network (that minimizes the MSE objective). Suppose we train the value network to convergence, obtaining a local ...


0

Remember the definition of $R^2$. $$ R^2 = 1 - \dfrac{\sum\big(y_i - \hat y_i\big)^2}{\sum\big(y_i - \bar y\big)^2} $$ The numerator of that fraction is proportional to the $MSE$, and the denominator is proportional to the variance of the $y$ observations. Your objection is to getting an $R^2$ that looks good yet also getting an $MSE$ that looks big. If that ...


1

No, I fear this is not possible. Kernels are inner products, which means they live in vector spaces which are Hilbert spaces. The $l_2$ norm is the only $l_p$ norm induced by an inner product. The other norms only give you Banach spaces. Check wikipedia for this. It may be possible to generalise the notion of Kernels or Kernel-methods to Banach spaces. But ...


1

Let me try to answer your question. First of all, I wanted to highlight that what you are observing here in terms of distribution of the probability of the model is a common effect especially for boosted models, SVM, or even Naive Bayes. You tend to see a sigmoid distortion of the probabilities more prominent towards the extremes rather than an even ...


2

As already mentioned, it depends on what data you train your GAN. But it also depends on what you expect as an outcome of the GAN. Most methods focus on complete new synthetic data, but that's not the only option. This approach of landing AI seems to be more promising than just generating new artificial data, they augment existing data using a GAN to enrich ...


0

YES When you have one random state, you select certain data for training, develop a model, and then test the model on a holdout data set. When you change the random state, you select different data for training, develop a different mode, and then test this other mode on a different holdout data set. I would expect different models that are tested on ...


2

Without knowing specifically how random state 1 and 2 differ, what's likely happening is that different objects (records, subjects, animals, cars, etc) are being selected for training the the RF classifier. Once training is done, the out-of-bag (OOB) objects that are not in each bootstrap (for training each tree), are "dropped" down the trained ...


0

A method known as non-negative matrix factorization (NMF) is ideal for recommending other songs and videos a user might like. You don't have to drill down into songs and use words in each song and run text mining or natural language processing (NLP) of any sort. Instead, you construct a user-by-song matrix, and in each row for a user, place the number of ...


0

After cluster analysis, I always run ANOVA (Kruskal-Wallis) and chi-squared contingency table analysis on all the cluster input variables to see how their means and count proportions differ across clusters (groups). This allows me to describe to a user/customer what the characteristics are for each cluster. After all, without knowing if age, gender, race, ...


1

Consider the linear model $$y_{i} = \mu + \sum \limits_{j=1}^7 \alpha_{j} I_{x_1=j}(x_{1i}) + \sum \limits_{k=1}^M \beta_k x_{2ki} + \epsilon_{i}$$ $i$ is the number of data points and j=1-7 (categories of $x_1$). Thus $\alpha_j$ is the coefficient for the $j$th category. Note that R will restrict $\alpha_1=0$ (be careful about the interpretation). So $\mu$ ...


5

Scikit-Learn's SVM is based on LIBSVM (from the documentation of SVC), for which you can find the implementation document here. Specific solver information can be found in section 4. Generally, gradient descent is a first-order method, which means it utilizes minimal information about the problem (only gradients) and thus converges slowly and might suffer ...


1

When using cross-entropy for classification tasks, that can also be viewed as a conditional probability distribution (categorical - the output values are the parameters giving the probability of each catageory) and when using MSE for regression that is also specifying a conditional probability distribution (the conditional mean of a Gaussian distribution - ...


2

This sounds appealing at first. After all, if one model predicts $0.6$ and another $0.9$, you’d trust the confident mode making the confident $0.9$ prediction and not the wishy-washy $0.6$ prediction, right? The trouble is that the right answer might be that your case has ambiguity, and about $60\%$ of such cases will go for one class and $40\%$ for the ...


1

I'll give my two cents, though it's far from a "perfect" solution (if one exists). First, as for the issue of varying dataset sizes, you can normalize the token frequencies by the document count, and choose a threshold over the normalized frequency rather than the absolute one. Second, as you noticed, using a threshold in such cases is problematic, ...


5

You can transform your original predictors in different ways, e.g., using splines or sine/cosine transformations, then feed them into regression-type models (OLS, GAMs, ...). Frank Harrell's Regression Modeling Strategies discusses this at length.


1

No it's not a silly question. There are not a lot of statistics for comparing statistics. For example, you have a lot of t-tests (logistic models) and want to perform a hypothesis test to determine which t-test is the most significant. That is, hypothesis tests for hypothesis test results. For logistic, compare each model using a variety of test results ...


0

I will answer my own question. This is formulated in terms of conditional cross entropy. I will me expand on the details as follows.


0

I've never quite seen NxN, but I think it may be referring to this scenario: If N examples are available, N training sets (subsamples) with N-1 examples are created, such that each of those N training sets (subsamples) omits exactly one (qty. 1) example that (collectively) every other training set (subsamples) did not omit. This strategy is a limiting case ...


1

No, you don't want to label every slice. It sounds like you're doing classification, as opposed to segmentation. For a segmentation problem, you need to classify every pixel, which can be laborious work. I'll assume you're doing classification. Get started with 2D images While you're learning, it might be to treat this as a 2D problem. Choose the most ...


3

First, let's identify what we mean here: $$\text{MSE}=\frac{1}{m}\sum (X_iW-Y_i)^2$$ And the gradient is a vector of partial derivatives: $$\nabla_W\text{MSE}=\left[\frac{\partial \text{MSE}}{\partial W_j} \right]$$ $$\frac{\partial \text{MSE}}{\partial W_j}= \frac{1}{m}\frac{\partial}{\partial W_j}\sum (X_iW-Y_i)^2=\\ \frac{1}{m}\sum 2(X_iW-Y_i)\frac{\...


0

There's an example in the user guide. You can pass a list of column names for transforming. This column_trans can then be used as a part of a pipeline. It can be useful to know about FeatureUnion too in this context. column_trans = ColumnTransformer( [('categories', OneHotEncoder(dtype='int'), ['city']), ('title_bow', CountVectorizer(), 'title')], ...


0

There are closed-form solutions to the MLEs in this case since the membership probabilities are given (sample mean and sample variance-covariance matrix) for observations within each class. However, your set-up is flawed. Let $\boldsymbol{x}_{ik} \in \mathbb{R}^{64}$ denote the $i$th vectorized 8x8 image for the $k$th label. The last term of the full log-...


1

Precision is the probability of being in a category given a prediction of that category, conditioning on the known prediction rather than on the unknown true state. (If you know the true state, why do any fancy modeling that might get the wrong answer?) If you never predict a particular category, precision makes less sense.


1

Although I haven't worked with ICP specifically I have done work with point clouds, hopefully you will get a better answer soon! I don't think so. Consider a very simple example where you're just fitting a line to a pair of points. The (many) points on the line are each fit to the closest of the two points, and the the line is transformed to minimize this; ...


0

Yes, why not. It's not different from using an ML pipeline with steps [PCA, SVM] cascaded. The kernel trick only adds some nonlinearity to these steps/algorithms.


0

I don't think so. Based on the feature encoding used, the root split might be better for $T_1$ but not enough for $T_2$, because the consequent levels of the tree may complement the root very well. For example, if $T_1$ uses a 1D mapping for an inherently multi dimensional feature, the first split might be very informative, especially compared to splitting a ...


6

If you think of $L$ as a column vector, then I think both your sources agree that $\frac{dJ}{dL}$ should be a row vector. But what if you really want $L$ as a row vector. Surely, the math shouldn't "care" about how you arrange your collection of numbers. One way to clarify this is by designating dimensions of your objects as "covariant" ...


5

Unfortunately, I didn't come across a resource that doesn't leave gaps. It's a disputed area. Even the chain rule may sometimes not make a lot sense, e.g. some terms might be 3D tensors that the matrix multiplication is not well-defined because of matrices differentiated by vectors or vice versa. Having said that, these rules are also very useful if you ...


0

Shapley value calculations assume all the features are uncorrelated. Correlated features will split the Shapley values among themselves in unexpected way, usually depending on their scales. The optimal trees probably selected one for each correlated groups to minimize unwanted variance from extra dimensions in the model. If you run PCA on your features, you ...


1

Here is a related thread on power and prediction. The short answer is you should use/borrow data from a source that is representative of your target. If you are not sure which source is representative then perform several sensitivity analyses (each time borrowing from a different source) to see how the results vary, and give more importance to the more ...


1

If your training and test sets differed substantially in terms of average observations, as your question suggests, then perhaps there is a problem with your train/test split. The problem potentially posed by a difference in average observations between the training and test sets is that the two sets might not represent the same underlying population. That ...


0

Well, I depends on the type of Machine Learning model you will use. If this model is linear, such as Logistic Regression or LDA, it is better to engineer this variable before training the model. If the model is a Decision Tree, Random Forest for example, the initial distribution doesn't affect much the outcome as the splitting technique is optimized by the ...


0

This is an old good question that has not received a complete answer. I believe this is not only about the "kernel trick." I believe this is more about a theoretical result called "the Representer Theorem" [1; Theorem 16.1] of kernel methods which I find rather deep. Short answer Is there an easy way to explain how that works? Namely how ...


4

The decision tree makes binary splits. Each node is an if statement, that leads you to the branch on the left, or on the right. When training the decision tree, the branching continues until hitting some stopping criterion (e.g. minimum number of samples per leaf). The final leaf is where you make the classification decision. You can either calculate the ...


5

After splitting with respect to some criterion (e.g. gini, information gain etc.) the samples you're left with in the leaf node decides the class. Typically, the majority class is chosen, or in some situations, a probability might be more preferable (e.g. a probability vector). It doesn't matter if the problem is multi-class or binary.


0

This is not true in general. Imagine that there’s a causal relationship between two events. In such a case, knowing that the cause has happened would be enough to predict the effect. A single feature would be enough. Now imagine the opposite case, where you’re trying to predict a completely random, independent of everything, event. No matter how many ...


1

The second equation is minimizing the TD-error of the critic -- if $\delta_t$ is positive, then that means our estimate of the value of $x_t, a_t$ was too pessimistic, so we want to update $\omega$ in a way which pushes $Q^\omega(x_t,a_t)$ upwards (and vice-versa). The critic's job is not to maximize any reward, it's just to become a better critic.


Top 50 recent answers are included