Hot answers tagged

16

You may find helpful this extensive curated list of ML libraries, frameworks and software tools. In particular, it contains resources that you're looking for - ML lists for Java and for Scala.


10

When I am forced to use java for basic statistics, apache commons math is the way to go. For plots, I use and recommend JFreeChart. The latter is widely spread, so stackoverflow even has a populated tag for it. Edit If one looks for a suite, then maybe Deducer is an option. The GUI is based on JGR meanwhile the statistical parts are called in R. It seems ...


8

Question 1: Let's say you have observed a data matrix $X \in \mathbb R^{n \times p}$. From this you can compute the eigendecomposition $X^T X = Q \Lambda Q^T$. The question now is: if we get new data coming from the same population, perhaps collected into a matrix $Z \in \mathbb R^{m \times p}$, will $ZQ$ be close to the ideal orthogonal rotation of $Z$? ...


7

The equation is $$\log(\mu_i) = \beta_0 + \beta_1 x_i$$ where $\mu_i$ is the conditional expectation of $y_i$, $E(y | x)$, $\beta_0$ is the coefficient marked Intercept and $\beta_1$ the coefficient marked x. The $\log$ bit is the link function you specified. Hence to get actual predictions on the scale of your response data $y$, you need to apply the ...


6

As there seems to be a misunderstanding about the statistical aspect of the procedures you used, here are some hints: Your R model is actually a multiple linear regression with geno_A and geno_B treated as numeric variables, and it includes an interaction term, which is why you get four parameter estimates. I hope you really want geno_A to be treated as ...


6

At http://prng.di.unimi.it/ you can find a shootout of several random number generators tested using TestU01, the modern test suite for pseudorandom number generators that replaced diehard and dieharder. The Java LCG generator is unusable. You should avoid it like the plague. A xorshift generator is better, but still displays several statistical artifacts. ...


5

The z-table contains the area to the left of the z-number under a normal distribution with $\mu = 0$ and $\sigma = 1$. Your p-value will generally represent one minus that area (for a one tailed test). Transforming this p-value into it's corresponding z-score should not be that hard under these circumstances. Your normal distribution with $\mu = 0$ and $\...


5

Class labels aren't the same as clusters. If you look at e.g. the iris data set, it's fairly obvious that the best solution will have just 2 clusters, not three. Plot the unlabeled data and interview some people on the number of groups they see in this data set. If you set k to three, you will often get results like this, where the wrong cluster is split. ...


5

Apache Spark and specifically its component MLlib looks like exactly what you are looking for. MLlib contains implementations for classification, regression, dimensionality reduction etc. You can program in Scala,Java and Python. Its basically a very fast distributed computing framework that can be run in an Hadoop cluster. For development purposes, you can ...


4

Check out Suan Shu: NumericalMethod.com. It is not free in general, but it is free for academic use.


4

I suspect this is because you are using a Gaussian process with a zero mean function, so that unless the covariance function is non-local, the output will go to zero away from the datapoints. If you are using a local covariance function, such as the squared exponential (RBF), it is a prior over functions that says that the function should be smooth, i.e. ...


4

The method you link to in your comment should work, if you choose to follow the neural-network survival analysis approach in the article I linked to in my comment. For each patient in the model that approach uses a list of probabilities of being alive at each time of interest: 1/0 for patients known to have died, and for "censored" cases a 1 until last ...


4

As far as I can see there are two issues in your question. First, the question of the classifier: learn a classifier to detect cars. This is typical image classification. I suppose you're dealing with supervised learning. What you might want to do is something like, given you training database: resize all your images, extract features from them - look for ...


4

You can compute pairwise KL divergence as a function of parameters in closed form for two Gaussian distributions $p$ and $q$. The uni-variate case: $KL(p||q) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_{1}^{2} + (\mu_1-\mu_2)^2}{2\sigma_{2}^{2}} - \frac{1}{2}$ and the multi-variate case: $KL(p||q) = \frac{1}{2}\left[\log\frac{|\Sigma_2|}{|\Sigma_1|} - ...


4

You have created a basic nearest neighbor model and a predict() mechanism to find the closest "fit" to your model. I'd call that a simple machine learning algorithm. You might break out the code doing the modeling and prediction into a separate small method - both to make it more clear to yourself and to readers and also to allow to "override" the methods ...


3

Weka is very lacking and slow in clustering. KNIME and Rapidminer are frontend-heavy; often the actual functionality is provided by Weka in the background. You should have a look at ELKI, it has a lot more in clustering and outlier detection than Weka, and usually is a lot faster. There are quite some algorithms that I'm not aware of other implementations ...


3

You're only describing a problem. There are many different algorithms that can be applied to solve it, see e.g. http://lane.compbio.cmu.edu/courses/slides_ucb.pdf. The UCB1 algorithm is so dead simple, reading Java code to understand how it works is probably a bad idea - unless you want to learn Java: Play machine $j$, that maximizes $$\bar{x}_j+\sqrt{\...


3

check out the EDMA (euclidean distance matrix analysis), it's used for biological shape comparison and uses a nonparametric bootstrap of the differences in the coordinates between shapes, here is a link to the author's site about the text on the subject http://getahead.psu.edu/purplebook_new.html and the actual software package http://www.getahead.psu.edu/...


3

One thing I might do is some sort of local smoothing? I assume the smallest jitter would be noise that you don't want to influence your analysis. Not sure if scaling both series or subtracting out their means might help too. I'd follow up computing their cross correlation perhaps?


3

If you assume that customers demands are independant and Poisson, the total demand will be a Poisson distribution as well with parameter equal to the sum of the individual Poisson parameters (see wikipedia for example). A C++ code example (I am not so familiar with Java): double cumulative(double x, User * users, int nusers) { double lambda = 0.; ...


3

An easy way to build an ensemble is by using a random forest. I'm fairly sure weka has a random forest algorithm, and if other tree-based models are performing well it's worth trying out. You could also build your own ensemble by training multiple (say 50 or 100) J48 decision trees and using them to "vote" on the classification of each object. For example,...


3

Similar to steffen's suggestion of RapidMiner, you might want to consider Weka. It may be geared more specifically to machine learning than you are hoping for though. It has lots of algorithms for tasks like clustering, classification, and regression. Weka has a GUI, but it can also be used as a software library as well. I've seen histograms in the GUI ...


3

After running more tests with different parameters, I found that the issue was that the initial learning rate was set too low. If I'm perfectly honest I'm pretty frustrated with myself that the solution was that simple, but that's all it was.


3

The code you are using will invert the entire matrix. This is probably O(p^3) already. You can approximate the result in O(p^2) but that will still be slow (but probably 100x faster). Essentially, take an arbitrary vector and do power iterations. With high probability, you'll get a good approximation of the first eigenvector. Then remove this factor from the ...


2

Did you apply the StringToWordVector in Weka? If so, then you did more than just punctuation and stop-words removal. StringToWordVector outputs only the doc-term matrix of the input text files, so once the above mentioned preprocessing is done Weka will create 1 term for each unique word. 35k terms sounds logical for 40k texts. The preprocessing in R seems ...


2

I think this is a small n large p problem. If you have more parameters than cases it is very difficult to see much. You could be severely overfitting to the data and creating singularities or near singularities because of some variables being highly correlated with each other. I think that would explain why so many eigenvalues are nearly equal to 1.


2

For a small number of PCs, the choice algorithm seems to be Nipals.


2

Have a look at Apache Mahout. It has various dimensionality reduction techniques implemented.


2

A good place to start for implementations of online learning is vowpal wabbit. It implements linear regression, logistic regression, several extensions of logistic regression for multi-class problems, neural networks, and matrix factorization. It also has several other nifty features, such as on-the-fly ngram computation and spell-checking. I'm not aware ...


2

The fastest I know of is Vowpal Wabbit by John Langford and his teams, first in Yahoo! and then at Microsoft. The implementation tweaks and tricks in that code are exceptional. Its implementation is in C++. So, one option is to call it as an external tool, since it accepts data as files from disk or from stdin. More interesting option is to use it as a web ...


Only top voted, non community-wiki answers of a minimum length are eligible