Tag Info

0

So you can eliminate the last few principal components, as they will not cause a lot of loss of data, and you can compress the data. Right? yes, you're right. And if there are $N$ variables $V_1, V_2, \cdots , V_N$, you then have $N$ Principal Component $PC_1, PC_2, \cdots , PC_N$, and every variable $V_i$ has an information (a contribution) in every PC ...

0

Probability and statistics are essential. Some keywords are hypothesis test, multivariate normal distribution, Bayesian inference (joint probability, conditional probability), mean, variance, covariance, Kullback-Leibler divergence, ... Basic linear algebra is essential for machine learning. Topics that you could learn are Eigen decomposition and singular ...

0

As far as brushing very very basic math skills, i'm using these books: Elements of Mathematics for Economics and Finance. Mavron, Vassilis C., Phillips, Timothy N This books covers essential math skills (addition substraction), to partial differentiation, integration, matrix and determinants, and a small chapter on optimization, and also differential ...

0

Kernels can be extended to nearly any kind of data, but the modelling has to be done very carefully. And this is the reason, why many people are scared of it. Its difficult to handle non-understood methods. Its just not enough to apply some gaussian kernels and to perform grid search on the scaling. This critisism is not just related to kernel methods, but ...

1

The answer to this question describes how feature importances are computed in sklearn. Maybe it will help you with your questions #1 and #3. Regarding question #1: It does not seem that this definition of importance is explicitly related to statistical significance. Regarding question #2: You could still report the feature importances reported by ...

2

In some sense I think this question is unanswerable. I say this because how well a particular unsupervised method performs will largely depend on why one is doing unsupervised learning in the first place, i.e., does the method perform well in the context of your end goal? Obviously this isn't completely true, people work on these problems and publish results ...

1

The biggest issue with LDA is that it has no concept of correlation between topics. Blei et al create the Correlated Topic Model to work and getting around that limitation in particular. But this is just one step. Your issue I think (if I understand correctly) stems from the fact that we are doing "clustering" on human text, but we have thrown away all the ...

0

If you want to/are able to go nonparametric, this is implemented in the mgcv package, which implements penalized splines. If you use the option select=TRUE, the optimizer that selects smoothness penalties also adds a penalty term to the "main effect" of each smooth term, in addition to the penalty used for smoothness selection. It doesn't however implement ...

0

Adaptive Lasso (H.Zou, JASA 2006, Vol. 101, No. 476) achieves consistency in parameter estimates by using individual lambda for each variable. Lambda values are tuned based on OLS solution (which unfortunately is not available in many practical cases where Lasso is used).

0

Is it possible to do much better than the 2.8% test error on MNIST using Random Forests? Probably, yes. But that doesn't mean you'll be using the same features that you get by default. Decision trees in general don't work well on high dimensional problems like this, as you are only splitting on one feature at a time. Random Forest extends the usefulness ...

1

Software The software you list (SPSS, SAS) are statistics packages. They are hardly suitable for actual data mining (kernel methods, neural nets, deep learning, ...). That said, you could easily replace both of those by R which works perfectly on any platform. In terms of data mining software, to my experience, you should be looking at things like Python, ...

0

Pragmatically speaking, if you have a choice over how you partition the data (i.e. there are now time constraints or such), I would say you would be better of shuffling your dataset so as to distribute the classes more or less evenly over all partitions. If this is not an option, I would simply drop any classes that you don't have any training data on: if ...

1

I think your question should be split in two parts: Which OS should you use? This is a very controversial topic and it depends on the tools you will be using. I myself prefer to work on a Unix based system (i.e. Linux/Mac). However, since most of the software you mentioned are designed for MS Windows, I would recommend going for that option. ...

0

I found this tutorial which was extremely helpful. It doesn't answer all the pieces but I think it is a great start to the discussion: http://vikparuchuri.com/blog/intro-to-ensemble-learning-in-r/

4

Yes, it had been tried (including by myself - I tried it with neural nets, with rather mixed success). The Relevance Vector Machine (RVM) does pretty much exactly that, and the regularisation parameters are tuned by maximising the marginal likelihood. The advantage of this is that it leads to a sparse model where uninformative attributes end up with large ...

0

It's best to use models that can deal with structured output instead of doing a hierarchical classification. The fact there is a structure in the possible outcomes is prior information which you can use to obtain a better overall model. One technique that can handle this kind of information is the structured SVM, an implementation of which is available in ...

1

I can suggest a strategy from a classification performance viewpoint. Since the classification problem relies not only on the amount of classes and how 'similar' they are, but also on the features which describe this classification problem, I can't say if it is necessarily 'optimal'. Assuming your features have similar values for A1 and A2 (and also B1 and ...

0

Genetic algorithms and neural networks are not really competing methods. For instance one could use a genetic algorithm to train a neural network (a field called neuroEvolution). Genetic algorithms are used to optimise 'thing' and neural networks are 'something' to be optimised. i.e. one could optimise a neural network using backpropagation, some form of ...

2

I don't know if you split your dataset randomly (each sample receives a random subset of observations) or not. I assume that your split is random. why does the training error start so high, then suddenly drop, then start to rise again as training set size increases? This is merely a noise caused by small size of your training and test sets, as well as the ...

1

The notion of instance hardness my address what you are looking for. Instance hardness posits that each instance in a data set has a hardness property indicating the likelihood that it will be misclassified by a supervised learning algorithm. In a sense, instance hardness looks at the hardness of each individual rather than the hardness of the data. However, ...

0

Correctly Predicted is the intersection between the set of suggested labels and the set expected one. Total Instances is the union of the sets above (no duplicate count). So given a single example where you predict classes A, G, E and the test case has E, A, H, P as the correct ones you end up with Accuracy = Intersection{(A,G,E), (E,A,H,P)} / ...

2

To get you started, the Elements of Statistical Learning have a nice discussion about regularization, and also sound discussions of different models To judge whether a particular regularization is a good idea, you need to take into account you data as well. E.g. for the LASSO, does it make sense for your data to assume that you have noise-only variates ...

0

I presume your are from Poland? I do not think it is possible to answer for your question. Even in some statistics books about measures like AUC it is said it will be domain specific. My classification models had recently their MCC in range of 0.2-0.25 with AUC bit over 0.7. MCC can be calculated only if you specify a cutoff point when you have a ...

1

$\frac{\partial L}{\partial w} = 0$ is equivalent to: $\frac{\partial(\frac{1}{2} w^{T}w - \sum_{i=1}^{N}\alpha_{i}(y_{i}(w^{T}\phi(x_{i}) - b) - 1))}{\partial w} = 0$ which comes down to differentiation of four parts: $\frac{\partial \frac{1}{2} w^{T}w}{\partial w} = \frac{1}{2}*2*w = w$ (quadratic function of $w$) $\frac{\partial ... 4 The two functions aren't the same. They are only related by duality.$s_a$is not a normalizer, it is an arbitrary scale factor. 0 There might be something in the Octave forge image package that you could use or adapt to your purposes. 0 if i need more training samples or features: See the next part. if my hypothesis suffers of underfit/overfit? See this: http://en.wikipedia.org/wiki/File:Overfitting_svg.svg (from: http://en.wikipedia.org/wiki/Overfitting) If you are before the exclamation sign: underfit If you are after the exclamation sign: overfit 0 there is only one paradigm that can avoid models from overfiting -on future data- that is of course VC bound. Many researchers say that VC bound is a pecimistic case but i don't understand .if there is only one woman in the space, is it possible to make comment about the beauty of that the only one female... 0 You confusion shows that you are a very exact person! ;-) Assumptions on the notation: -$\theta $: Parameters -$ \mathbf{x} , \mathbf{z} $: Variables Among Bayesian people, when someone talks about estimation, they refer to estimation of almost anything. See this: ftp://ftp.cs.utoronto.ca/pub/radford/bayes-tut.pdf In Neal's tutorial, page 4 he ... 1 It depends on the ensemble method you use. Usually the VC-dimension increases. But in the case of AdaBoost, you can find the answer here: http://www.cs.princeton.edu/courses/archive/spr08/cos511/scribe_notes/0305.pdf http://cseweb.ucsd.edu/~yfreund/papers/IntroToBoosting.pdf 0 When people train a model using a dataset, they split the data into several parts and do cross-validation: http://en.wikipedia.org/wiki/Cross-validation_(statistics) If you scientifically want to find out the exact test performance of the model, you see on which portion of the data it is trained on, and test on the remaining. 1 Download the source code, go to the source of the method that returns your predictions - and add your constant. Or add your constant to the bias term. No, LibSVM doesn't provide an API for doing it for you. 1$\mathcal{Y}$: observed$\mathcal{X}$: unobserved In other words,X is the set of latent variables. So, your picture makes sense. yes! See, this is just density (probability distribution) which depends on our observations. It doesn't matter what it is (because it can be anything!), what matters is that, this is a function which dependents on our inputs, ... 0 If your features are categorical, the first idea that comes to my mind is to create a binary feature for every possible value in the category. Thus, if you have a feature corresponding to "mobile phone brand" which can only be "Samsung, Apple, HTC or Nokia", I would represent it as four categories (1, 0, 0, 0), (0, 1, 0, 0), (0, 0, 1, 0) and (0, 0, 0, 1) ... 1 In these 2 situations, comparative performance flexible vs. inflexible model also depends on: is true relation y=f(x) close to linear or very non-linear; do you tune/constrain flexibility degree of the "flexible" model when fitting it. If relation is close to linear and you don't constrain flexibility, then linear model should give better test error in ... 3 What you are looking for sounds a tad like Evidence Trees: Evidence Trees. We have developed a new approach to supervised learning in which ensembles of tree classifiers are applied not to make classification decisions but to select which training data points provide evidence relevant to making a decision or prediction. This evidence can then be ... 2 Actually a three layer neural network can model arbitrary function with the linear and logistic functions, which was proved by Kolmogorov in 1957 (Kolmogorov, Andrei Nikolaevich. "On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition." Dokl. Akad. Nauk SSSR. Vol. 114. No. 5. ... 0 There's a few options. One is to search these hyper-parameters randomly then pick the one with the best cross-validation. I tend to run small experiments, record my results and then narrow the parameters. RandomizedSearchCV is a good starting point. After getting a decent estimate of good hyperparameters, with some modules, you can set the pass the ... 3 Artificial neural network: computational power (Wikipedia): The multi-layer perceptron (MLP) is a universal function approximator, as proven by the Cybenko theorem. However, the proof is not constructive regarding the number of neurons required or the settings of the weights. Work by Hava Siegelmann and Eduardo D. Sontag has provided a proof ... 1 Leon Bottou's webpage has some good references on how to use SGD wisely(here is a link that you may find useful). 2 One way to quantify the usefulness of each feature (= variable = dimension), from the book Burns, Robert P., and Richard Burns. Business research methods and statistics using SPSS. Sage, 2008. (mirror), usefulness being defined by the features' discriminative power to tell clusters apart. We usually examine the means for each cluster on each dimension ... 0 As far as I understand, this is the structure of your data, where you have a closed set (size=n) of questions: Q1 Q2 ... Qn S1 0 1 -1 S2 1 0 0 ... ... ... Sn -1 1 -1 In that case, you may want to use a collaborative filter. Here is a light-weight introductory article. The underlying assumption of the ... 1$\Vert w \Vert$represents the norm of the vector$w$. There are different norms that you can consider. When it is not said, it's often the euclidian norm, the norm 2: $$\Vert w \Vert = \Vert w \Vert_2 = \sqrt{w_1^2 + \cdots + w_n^2}$$ Here are some other norms: norm 1: $$\Vert w \Vert_1 = |w_1| + \cdots + |w_n|$$ norm$p$:$$\Vert w \Vert_p = \left( ... 0 Right, one way to think of$w$is as a vector representing a separating hyperplane. However, in order to understand its usefulness, you better consider it a regularization value:$||w||\$ is a measure for the complexity of the model represented by the SVM. The idea is to find a model that on one hand accurately models the training data, and on the other ...

1

genetic algorithms were used to lower the prime gap to 4680 in the recent Zhang twin primes proof breakthrough & associated polymath project. the bound has been lowered by other methods but it shows some potential for machine-learning approaches in this or related areas. they can be used to devise/optimize effective "combs" or basically sieves for ...

0

The caret package has a method for getting that. You can use train as the interface. For example: > mod1 <- train(Species ~ ., + data = iris, + method = "cforest", + tuneGrid = data.frame(.mtry = 2), + trControl = trainControl(method = "oob")) > mod1 150 samples 4 predictors 3 classes: ...

1

caret has a stepLDA method available in train: slda <- train(Species ~ ., data = iris, method = "stepLDA", trControl = trainControl(method = "cv")) This uses stepclass in the klaR package. There are also LDA feature selection tools in caret using rfe and sbf that would be helpful. Max

2

Feature normalization is to make different features in the same scale. The scaling speeds up gradient descent by avoiding many extra iterations that are required when one or more features take on much larger values than the rest(Without scaling, the cost function that is visualized will show a great asymmetry). I think it makes sense that use the mean and ...

1

Cross validation is used to select your model. The out-of-sample error can be estimated from your validation error. Usually this validation error is the mean value of your ten validation errors. Please note that the model here not only means the feature number, but also refers to your function model (whether it is y=ax1+bx2+c or y=ax1^2+bx2+cx1x2+d...etc.), ...

2

Your approach is entirely correct. Although data transformations are often undervalued as "preprocessing", one cannot emphasize enough that transformations in order to optimize model performance can and should be treated as part of the model building process. Reasoning: A model shall be applied on unseen data which is in general not available at the time ...

Top 50 recent answers are included