4
$\begingroup$

I have been messing around in tensorflow playground. One of the input data sets is a spiral. No matter what input parameters I choose, no matter how wide and deep the neural network I make, I cannot fit the spiral. How do data scientists fit data of this shape?

$\endgroup$
  • 1
    $\begingroup$ Thank for the neat link, I had not seen this! Playing around quickly, from their default setup, if you just add the two "sin" features, it does a lot better. (Makes sense, as the spiral oscillates classes along any transect.) $\endgroup$ – GeoMatt22 Sep 18 '16 at 16:25
  • $\begingroup$ That was the first thing I tried but I got like 40-60% accuracy. $\endgroup$ – Souradeep Nanda Sep 18 '16 at 16:29
  • $\begingroup$ It seems to depend on the initialization a bit. I commonly got to 5% or so. (I think I took down the batch size to 5 also). You used sin(x) & sin(y) in addition to x & y? (this) $\endgroup$ – GeoMatt22 Sep 18 '16 at 16:33
  • $\begingroup$ Following that same strategy (add sin's, lower batch size), then widening base & adding a 3rd interior layer, this seems to get consistently < 1% error. $\endgroup$ – GeoMatt22 Sep 18 '16 at 17:28
  • 1
    $\begingroup$ I should have just trained it a bit longer and used more neurons. It was actually trivially simple. $\endgroup$ – Souradeep Nanda Sep 18 '16 at 18:07
3
$\begingroup$

You could use SVM with an RBF kernel. Example:

import numpy as np
import matplotlib.pyplot as plt
import mlpy # sudo pip install mlpy
f = np.loadtxt("spiral.data")
x, y = f[:, :2], f[:, 2]
svm = mlpy.LibSvm(svm_type='c_svc', kernel_type='rbf', gamma=100)
svm.learn(x, y)
xmin, xmax = x[:,0].min()-0.1, x[:,0].max()+0.1
ymin, ymax = x[:,1].min()-0.1, x[:,1].max()+0.1
xx, yy = np.meshgrid(np.arange(xmin, xmax, 0.01), np.arange(ymin, ymax, 0.01))
xnew = np.c_[xx.ravel(), yy.ravel()]
ynew = svm.pred(xnew).reshape(xx.shape)
fig = plt.figure(1)
plt.set_cmap(plt.cm.Paired)
plt.pcolormesh(xx, yy, ynew)
plt.scatter(x[:,0], x[:,1], c=y)
plt.show()

enter image description here

You can also use least squares support vector machine.


spiral.data:

1 0 1
-1 0 -1
0.971354 0.209317 1
-0.971354 -0.209317 -1
0.906112 0.406602 1
-0.906112 -0.406602 -1
0.807485 0.584507 1
-0.807485 -0.584507 -1
0.679909 0.736572 1
-0.679909 -0.736572 -1
0.528858 0.857455 1
-0.528858 -0.857455 -1
0.360603 0.943128 1
-0.360603 -0.943128 -1
0.181957 0.991002 1
-0.181957 -0.991002 -1
-3.07692e-06 1 1
3.07692e-06 -1 -1
-0.178211 0.970568 1
0.178211 -0.970568 -1
-0.345891 0.90463 1
0.345891 -0.90463 -1
-0.496812 0.805483 1
0.496812 -0.805483 -1
-0.625522 0.67764 1
0.625522 -0.67764 -1
-0.727538 0.52663 1
0.727538 -0.52663 -1
-0.799514 0.35876 1
0.799514 -0.35876 -1
-0.839328 0.180858 1
0.839328 -0.180858 -1
-0.846154 -6.66667e-06 1
0.846154 6.66667e-06 -1
-0.820463 -0.176808 1
0.820463 0.176808 -1
-0.763975 -0.342827 1
0.763975 0.342827 -1
-0.679563 -0.491918 1
0.679563 0.491918 -1
-0.57112 -0.618723 1
0.57112 0.618723 -1
-0.443382 -0.71888 1
0.443382 0.71888 -1
-0.301723 -0.78915 1
0.301723 0.78915 -1
-0.151937 -0.82754 1
0.151937 0.82754 -1
9.23077e-06 -0.833333 1
-9.23077e-06 0.833333 -1
0.148202 -0.807103 1
-0.148202 0.807103 -1
0.287022 -0.750648 1
-0.287022 0.750648 -1
0.411343 -0.666902 1
-0.411343 0.666902 -1
0.516738 -0.559785 1
-0.516738 0.559785 -1
0.599623 -0.43403 1
-0.599623 0.43403 -1
0.65738 -0.294975 1
-0.65738 0.294975 -1
0.688438 -0.14834 1
-0.688438 0.14834 -1
0.692308 1.16667e-05 1
-0.692308 -1.16667e-05 -1
0.669572 0.144297 1
-0.669572 -0.144297 -1
0.621838 0.27905 1
-0.621838 -0.27905 -1
0.551642 0.399325 1
-0.551642 -0.399325 -1
0.462331 0.500875 1
-0.462331 -0.500875 -1
0.357906 0.580303 1
-0.357906 -0.580303 -1
0.242846 0.635172 1
-0.242846 -0.635172 -1
0.12192 0.664075 1
-0.12192 -0.664075 -1
-1.07692e-05 0.666667 1
1.07692e-05 -0.666667 -1
-0.118191 0.643638 1
0.118191 -0.643638 -1
-0.228149 0.596667 1
0.228149 -0.596667 -1
-0.325872 0.528323 1
0.325872 -0.528323 -1
-0.407954 0.441933 1
0.407954 -0.441933 -1
-0.471706 0.341433 1
0.471706 -0.341433 -1
-0.515245 0.231193 1
0.515245 -0.231193 -1
-0.537548 0.115822 1
0.537548 -0.115822 -1
-0.538462 -1.33333e-05 1
0.538462 1.33333e-05 -1
-0.518682 -0.111783 1
0.518682 0.111783 -1
-0.479702 -0.215272 1
0.479702 0.215272 -1
-0.423723 -0.306732 1
0.423723 0.306732 -1
-0.353545 -0.383025 1
0.353545 0.383025 -1
-0.272434 -0.441725 1
0.272434 0.441725 -1
-0.183971 -0.481192 1
0.183971 0.481192 -1
-0.0919062 -0.500612 1
0.0919062 0.500612 -1
1.23077e-05 -0.5 1
-1.23077e-05 0.5 -1
0.0881769 -0.480173 1
-0.0881769 0.480173 -1
0.169275 -0.442687 1
-0.169275 0.442687 -1
0.2404 -0.389745 1
-0.2404 0.389745 -1
0.299169 -0.324082 1
-0.299169 0.324082 -1
0.343788 -0.248838 1
-0.343788 0.248838 -1
0.373109 -0.167412 1
-0.373109 0.167412 -1
0.386658 -0.0833083 1
-0.386658 0.0833083 -1
0.384615 1.16667e-05 1
-0.384615 -1.16667e-05 -1
0.367792 0.0792667 1
-0.367792 -0.0792667 -1
0.337568 0.15149 1
-0.337568 -0.15149 -1
0.295805 0.214137 1
-0.295805 -0.214137 -1
0.24476 0.265173 1
-0.24476 -0.265173 -1
0.186962 0.303147 1
-0.186962 -0.303147 -1
0.125098 0.327212 1
-0.125098 -0.327212 -1
0.0618938 0.337147 1
-0.0618938 -0.337147 -1
-1.07692e-05 0.333333 1
1.07692e-05 -0.333333 -1
-0.0581615 0.31671 1
0.0581615 -0.31671 -1
-0.110398 0.288708 1
0.110398 -0.288708 -1
-0.154926 0.251167 1
0.154926 -0.251167 -1
-0.190382 0.206232 1
0.190382 -0.206232 -1
-0.215868 0.156247 1
0.215868 -0.156247 -1
-0.230974 0.103635 1
0.230974 -0.103635 -1
-0.235768 0.050795 1
0.235768 -0.050795 -1
-0.230769 -1e-05 1
0.230769 1e-05 -1
-0.216903 -0.0467483 1
0.216903 0.0467483 -1
-0.195432 -0.0877067 1
0.195432 0.0877067 -1
-0.167889 -0.121538 1
0.167889 0.121538 -1
-0.135977 -0.14732 1
0.135977 0.14732 -1
-0.101492 -0.164567 1
0.101492 0.164567 -1
-0.0662277 -0.17323 1
0.0662277 0.17323 -1
-0.0318831 -0.173682 1
0.0318831 0.173682 -1
6.15385e-06 -0.166667 1
-6.15385e-06 0.166667 -1
0.0281431 -0.153247 1
-0.0281431 0.153247 -1
0.05152 -0.13473 1
-0.05152 0.13473 -1
0.0694508 -0.112592 1
-0.0694508 0.112592 -1
0.0815923 -0.088385 1
-0.0815923 0.088385 -1
0.0879462 -0.063655 1
-0.0879462 0.063655 -1
0.0888369 -0.0398583 1
-0.0888369 0.0398583 -1
0.0848769 -0.018285 1
-0.0848769 0.018285 -1
0.0769231 3.33333e-06 1
-0.0769231 -3.33333e-06 -1
| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

I had similar experiments comparing to Franck's answer. Please check this post.

Do all machine learning algorithms separate data linearly?

Tree decision boundary

Boosting decision boundary

KNN decision boundary

In the post we use tree, boosting and K nearest neighbor on spiral data.

  • KNN is most intuitive one, it make the classification according to a given point's neighbors. So, spiral data would not "break the neighbor rule"

  • For tree and boosting model, you can understand it as a "really complicated model that can achieve complied decisions". That is why you can see it can roughly learn the pattern, with some errors.

Finally you may search for special clustering or kernel PCA in google to see how can we deal with "connected components".

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Funny thing about the tensorflow playground in terms of linear separation: The spiral is their only case that is not linearly separable! So if all of their input features are allowed, no hidden layers are required to classify their other data. (Of course there are interesting things to do using deeper networks when only a subset of the inputs are given.) $\endgroup$ – GeoMatt22 Sep 19 '16 at 18:43
  • $\begingroup$ @GeoMatt22 in fact i haven't pay attention about OP's question on tensorflow playground... I think NN is powerful to do a lot of things. it cannot work well with the spiral data is because the the limitation of the web based tools? $\endgroup$ – Haitao Du Sep 19 '16 at 18:48
  • $\begingroup$ The tensorflow playground app can do the spiral fine (in my comments to OP I link to one simple solution from r/MachineLearning). I think when doing their experiments that the OP was mostly just not patient enough in waiting for training to converge. $\endgroup$ – GeoMatt22 Sep 19 '16 at 18:54
1
$\begingroup$

For this dummy problem you can increase the number of features. One particular way that I found to work is using extreme learning machines. Basically, you create a random matrix $K$ with columns equal to number of old features, $d$, and rows equal to number of new features $d'$(I had to use $d'=300d$). Also, create a random bias vector $b$ with length equal to $d'$. And you need a non-linear activation function $f$. Relu in particular works well --- $Relu(X) = max(X,0)$. Then perform linear logistic regression on the new data $X'=f(XK+b)$ (sloppy numpy or matlab notation for adding $b$ to every row of $XK$).

Here is a small code using the linear logistic regression of scikit-learn in python.

import numpy as np
import matplotlib.pyplot as plt
import sklearn.linear_model


f = np.loadtxt("spiral.data")
x, y = f[:, :2], f[:, 2]
new_feature_ratio = 300;
def relu(Y): return np.maximum(Y, 0)
cls = sklearn.linear_model.LogisticRegression(
    penalty='l2', C=1000, max_iter=1000)
K = np.random.randn(x.shape[1], x.shape[1]*new_feature_ratio)
b = np.random.randn(x.shape[1]*new_feature_ratio)
cls.fit( relu(np.matmul(x,K) + b) ,y)
xmin, xmax = x[:,0].min()-0.1, x[:,0].max()+0.1
ymin, ymax = x[:,1].min()-0.1, x[:,1].max()+0.1
xx, yy = np.meshgrid(np.arange(xmin, xmax, 0.01), np.arange(ymin, ymax,    0.01))
xnew = np.c_[xx.ravel(), yy.ravel()]
ynew = cls.predict(relu(np.matmul(xnew,K) + b)).reshape(xx.shape)
fig = plt.figure(1)
plt.set_cmap(plt.cm.Paired)
plt.pcolormesh(xx, yy, ynew)
plt.scatter(x[y>0,0], x[y>0,1], color='r')
plt.scatter(x[y<0,0], x[y<0,1], color='g')
plt.show()

Classfication results

spiral.data is the same as Frank's answer. This strategy is basically a neural network where the first layer is chosen randomly rather than being trained.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.