20
$\begingroup$

I am training a neural network (details not important) where the target data is a vector of angles (between 0 and 2*pi). I am looking for advice on how to encode this data. Here is what I am currently trying (with limited success):

1) 1-of-C encoding: I bin the set up possible angles into 1000 or so discrete angles and then indicate a particular angle by putting a 1 at the relevant index. The problem with this, is that the network simply learns to output all 0's (since this is almost exactly correct).

2) Simple scaling: I scaled the networks output range ([0,1]) to [0,2*pi]. The problem here is that angles naturally have a circular topology (i.e. 0.0001 and 2*pi are actually right next to each other). With this type of encoding, that information is lost.

Any suggestions would be appreciated!

$\endgroup$
  • 1
    $\begingroup$ You should not be having a problem with the network outputting all zeros, if you use a softmax output layer -- which you generally should do, if you are using a catagorical (i.e. 1-of-C) output. $\endgroup$ – Lyndon White Jun 11 '16 at 6:24
  • 7
    $\begingroup$ A purely speculative encoding idea (I've not seen it done or tested it, but I haven't looked) is the encode your angle ($\theta$) as a pair: $\theta \mapsto (\sin(\theta),\cos(\theta))$. I think then it would be a continous map with all values like $0$, and $2\pi$ close to each other. I think I might build a demo of this and test it out. $\endgroup$ – Lyndon White Jun 11 '16 at 6:28
  • $\begingroup$ i've been thinking about this some more, and I think it might actually jsut be all in your loss function. I want to try a bunch of things. I made the demo, but didn't finish tests on it. Expect a detailed answer with experimental backing tomorrow sometime. (Poke me if I don't) $\endgroup$ – Lyndon White Jun 11 '16 at 14:18
  • $\begingroup$ I am not currently using a softmax layer, and that is probably the problem. I'll implement that today if I get a chance! Your (cos,sin) idea is very interesting and I especially like that it automatically puts that range into [-1,1] (good if you are working with a tanh activation function). I look forward to seeing your results1 $\endgroup$ – Ari Herman Jun 11 '16 at 15:29
  • $\begingroup$ A quick update: I tried implementing a softmax layer, and am still having no luck. The issue, I think, is that for this problem it is essential that the "angularity" of the data be represented somehow in the encoding. With a categorical encoding, the topology of the target data is lost. Hence an error of 0.5*pi and 0.05*pi look the same to the network (it sees both as incorrect categorizations). $\endgroup$ – Ari Herman Jun 11 '16 at 20:06
18
$\begingroup$

Introduction

I find this question really interesting, I'm assume someone has put out a paper on it, but it's my day off, so I don't want to go chasing references.

So we could consider it as an representation/encoding of the output, which I do in this answer. I remain thinking that there is a better way, where you can just use a slightly different loss function. (Perhaps sum of squared differences, using subtraction modulo 2 $\pi$).

But onwards with the actual answer.

Method

I propose that an angle $\theta$ be represented as a pair of values, its sine and its cosine.

So the encoding function is: $\qquad\qquad\quad\theta \mapsto (\sin(\theta), \cos(\theta))$
and the decoding function is: $\qquad(y_1,y_2) \mapsto \arctan\!2(y_1,y_2)$
For arctan2 being the inverse tangents, preserving direction in all quadrants)

You could, in theory, equivalently work directly with the angles if your tool use supported atan2 as a layer function (taking exactly 2 inputs and producing 1 output). TensorFlow does this now, and supports gradient descent on it, though not it intended for this use. I investigated using out = atan2(sigmoid(ylogit), sigmoid(xlogit)) with a loss function min((pred - out)^2, (pred - out - 2pi)^2). I found that it trained far worse than using outs = tanh(ylogit), outc = tanh(xlogit)) with a loss function 0.5((sin(pred) - outs)^2 + (cos(pred) - outc)^2. Which I think can be attributed to the gradient being discontinuous for atan2

My testing here runs it as a preprocessing function

To evaluate this I defined a task:

Given a black and white image representing a single line on a blank background Output what angle that line is at to the "positive x-axis"

I implemented a function randomly generate these images, with lines at random angles (NB: earlier versions of this post used random slopes, rather than random angles. Thanks to @Ari Herman for point it out. It is now fixed). I constructed several neural networks to evaluate there performance on the task. The full details of implementation are in this Jupyter notebook. The code is all in Julia, and I make use of the Mocha neural network library.

For comparison, I present it against the alternative methods of scaling to 0,1. and to putting into 500 bins and using soft-label softmax. I am not particularly happy with the last, and feel I need to tweak it. Which is why, unlike the others I only trial it for 1,000 iterations, vs the other two which were run for 1,000 and for 10,000

Experimental Setup

Images were $101\times101$ pixels, with the line commensing at the center and going to the edge. There was no noise etc in the image, just a "black" line, on a white background.

For each trail 1,000 training, and 1,000 test images were generated randomly.

The evaluation network had a single hidden layer of width 500. Sigmoid neurons were used in the hidden layer.

It was trained by Stochastic Gradient Decent, with a fixed learning rate of 0.01, and a fixed momentum of 0.9.

No regularization, or dropout was used. Nor was any kind of convolution etc. A simple network, which I hope suggests that these results will generalize

It is very easy to tweak these parameters in the test code, and I encourage people to do so. (and look for bugs in the test).

Results

My results are as follows:

|                        |  500 bins    |  scaled to 0-1 |  Sin/Cos     |  scaled to 0-1 |  Sin/Cos     |
|                        | 1,000 Iter   | 1,000 Iter     | 1,000 iter   | 10,000 Iter    | 10,000 iter  |
|------------------------|--------------|----------------|--------------|----------------|--------------|
| mean_error             | 0.4711263342 | 0.2225284486   | 2.099914718  | 0.1085846429   | 2.1036656318 |
| std(errors)            | 1.1881991421 | 0.4878383767   | 1.485967909  | 0.2807570442   | 1.4891605068 |
| minimum(errors)        | 1.83E-006    | 1.82E-005      | 9.66E-007    | 1.92E-006      | 5.82E-006    |
| median(errors)         | 0.0512168533 | 0.1291033982   | 1.8440767072 | 0.0562908143   | 1.8491085947 |
| maximum(errors)        | 6.0749693965 | 4.9283551248   | 6.2593307366 | 3.735884823    | 6.2704853962 |
| accurancy              | 0.00%        | 0.00%          | 0.00%        | 0.00%          | 0.00%        |
| accurancy_to_point001  | 2.10%        | 0.30%          | 3.70%        | 0.80%          | 12.80%       |
| accurancy_to_point01   | 21.90%       | 4.20%          | 37.10%       | 8.20%          | 74.60%       |
| accurancy_to_point1    | 59.60%       | 35.90%         | 98.90%       | 72.50%         | 99.90%       |

Where I refer to error, this is the absolute value of the difference between the angle output by the neural network, and the true angle. So the mean error (for example) is the average over the 1,000 test cases of this difference etc. I am not sure that I should not be rescaling it by making an error of say $\frac{7\pi}{4}$ be equal to an error of $\frac{\pi}{4}$).

I also present the accuracy at various levels of granularity. The accuracy being the portion of test cases it got corred. So accuracy_to_point01 means that it was counted as correct if the output was within 0.01 of the true angle. None of the representations got any perfect results, but that is not at all surprising given how floating point math works.

If you take a look at the history of this post you will see the results do have a bit of noise to them, slightly different each time I rerun it. But the general order and scale of values remains the same; thus allowing us to draw some conclusions.

Discussion

Binning with softmax performs by far the worst, as I said I am not sure I didn't screw up something in the implementation. It does perform marginally above the guess rate though. if it were just guessing we would be getting a mean error of $\pi$

The sin/cos encoding performs significantly better than the scaled 0-1 encoding. The improvement is to the extent that at 1,000 training iterations sin/cos is performing about 3 times better on most metrics than scaling is at 10,000 iterations.

I think, in part this is related to improving generalization, as both were getting fairly similar mean squared error on the training set, at least once 10,000 iterations were run.

There is certainly an upper limit on the best possible performance at this task, given that the Angle could be more or less any real number, but not all such angels produce different lines at the resolution of $101\times101$ pixels. So since, for example, the angles 45.0 and 45.0000001 both are tied to the same image at that resolution, no method will ever get both perfectly correct.

It also seems likely that on an absolute scale to move beyond this performance, a better neural network is needed. Rather than the very simple one outlined above in experimental setup.

Conclusion.

It seems that the sin/cos representation is by far the best, of the representations I investigated here. This does make sense, in that it does have a smooth value as you move around the circle. I also like that the inverse can be done with arctan2, which is elegant.

I believe the task presented is sufficient in its ability to present a reasonable challenge for the network. Though I guess really it is just learning to do curve fitting to $f(x)=\frac{y1}{y2} x$ so perhaps it is too easy. And perhaps worse it may be favouring the paired representation. I don't think it is, but it is getting late here, so I might have missed something I invite you again to look over my code. Suggest improvements, or alternative tasks.

$\endgroup$
  • $\begingroup$ This is certainly the most thorough response I have ever received on stack exchange. Since I am not familiar with Julia, it is difficult for me to examine your code...so, instead I am going to try to replicate your results using Python. I'll post me findings later today or tomorrow. $\endgroup$ – Ari Herman Jun 12 '16 at 17:37
  • $\begingroup$ While I was not surprised that binning performed poorly, I was surprised at the degree to which (0,1) scaling was outperformed by the (cos,sin) method. I noticed that you generated your examples by randomly picking the rise and run of the lines. This would, I think, generate lines whose angles are not uniformly distributed, but whose slopes are. Is it possible that this is why the (cos,sin) method performed so much better? What would happen if you made the targets tan(angle)...? $\endgroup$ – Ari Herman Jun 12 '16 at 17:47
  • $\begingroup$ you're right about random slope vs random angle. but I don't think targetting tan(angle) will go so well, given than tan isn't defined for all angles (eg $\pi/4$). I will rerun it with randomly generated angles and edit the posts. $\endgroup$ – Lyndon White Jun 13 '16 at 0:51
  • $\begingroup$ There should be a close to one to one map between julia and numpy, and between Mocha and Caffe, if you really want to reimplement it. Is there a particular part of the code you find hard to read? Julia should be an easy to understand language. So perhaps I have done something odd. $\endgroup$ – Lyndon White Jun 13 '16 at 0:54
  • $\begingroup$ I did end up reading your code, and everything seems correct. Still, I wanted to write my own version, since doing so is usually instructive. My implementation is slightly different from yours, so it will be interesting to compare results. I will be posting them within the next couple of hours. $\endgroup$ – Ari Herman Jun 13 '16 at 1:20
5
$\begingroup$

Here's another Python implementation comparing Lyndon White's proposed encoding to a binned approach. The code below produced the following output:

Training Size: 100
Training Epochs: 100
Encoding: cos_sin
Test Error: 0.017772154610047136
Encoding: binned
Test Error: 0.043398792553251526

Training Size: 100
Training Epochs: 500
Encoding: cos_sin
Test Error: 0.015376604917819397
Encoding: binned
Test Error: 0.032942592915322394

Training Size: 1000
Training Epochs: 100
Encoding: cos_sin
Test Error: 0.007544091937411164
Encoding: binned
Test Error: 0.012796594492198667

Training Size: 1000
Training Epochs: 500
Encoding: cos_sin
Test Error: 0.0038051515079569097
Encoding: binned
Test Error: 0.006180633805557207

As you can see, while the binned approach performs admirably in this toy task, the $(\sin(\theta), \cos(\theta))$ encoding performs better in all training configurations, sometimes by a considerable margin. I suspect as the specific task became more complex, the benefits of using Lyndon White's $(\sin(\theta), \cos(\theta))$ representation would become more pronounced.

import matplotlib.pyplot as plt
import numpy as np
import torch
import torch.nn as nn
import torch.utils.data

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")


class Net(nn.Module):
    def __init__(self, input_size, hidden_size, num_out):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.sigmoid = nn.Sigmoid()
        self.fc2 = nn.Linear(hidden_size, num_out)

    def forward(self, x):
        out = self.fc1(x)
        out = self.sigmoid(out)
        out = self.fc2(out)
        return out


def gen_train_image(angle, side, thickness):
    image = np.zeros((side, side))
    (x_0, y_0) = (side / 2, side / 2)
    (c, s) = (np.cos(angle), np.sin(angle))
    for y in range(side):
        for x in range(side):
            if (abs((x - x_0) * c + (y - y_0) * s) < thickness / 2) and (
                    -(x - x_0) * s + (y - y_0) * c > 0):
                image[x, y] = 1

    return image.flatten()


def gen_data(num_samples, side, num_bins, thickness):
    angles = 2 * np.pi * np.random.uniform(size=num_samples)
    X = [gen_train_image(angle, side, thickness) for angle in angles]
    X = np.stack(X)

    y = {"cos_sin": [], "binned": []}
    bin_size = 2 * np.pi / num_bins
    for angle in angles:
        idx = int(angle / bin_size)
        y["binned"].append(idx)
        y["cos_sin"].append(np.array([np.cos(angle), np.sin(angle)]))

    for enc in y:
        y[enc] = np.stack(y[enc])

    return (X, y, angles)


def get_model_stuff(train_y, input_size, hidden_size, output_sizes,
                    learning_rate, momentum):
    nets = {}
    optimizers = {}

    for enc in train_y:
        net = Net(input_size, hidden_size, output_sizes[enc])
        nets[enc] = net.to(device)
        optimizers[enc] = torch.optim.SGD(net.parameters(), lr=learning_rate,
                                          momentum=momentum)

    criterions = {"binned": nn.CrossEntropyLoss(), "cos_sin": nn.MSELoss()}
    return (nets, optimizers, criterions)


def get_train_loaders(train_X, train_y, batch_size):
    train_X_tensor = torch.Tensor(train_X)

    train_loaders = {}

    for enc in train_y:
        if enc == "binned":
            train_y_tensor = torch.tensor(train_y[enc], dtype=torch.long)
        else:
            train_y_tensor = torch.tensor(train_y[enc], dtype=torch.float)

        dataset = torch.utils.data.TensorDataset(train_X_tensor, train_y_tensor)
        train_loader = torch.utils.data.DataLoader(dataset=dataset,
                                                   batch_size=batch_size,
                                                   shuffle=True)
        train_loaders[enc] = train_loader

    return train_loaders


def show_image(image, side):
    img = plt.imshow(np.reshape(image, (side, side)), interpolation="nearest",
                     cmap="Greys")
    plt.show()


def main():
    side = 101
    input_size = side ** 2
    thickness = 5.0
    hidden_size = 500
    learning_rate = 0.01
    momentum = 0.9
    num_bins = 500
    bin_size = 2 * np.pi / num_bins
    half_bin_size = bin_size / 2
    batch_size = 50
    output_sizes = {"binned": num_bins, "cos_sin": 2}
    num_test = 1000

    (test_X, test_y, test_angles) = gen_data(num_test, side, num_bins,
                                             thickness)

    for num_train in [100, 1000]:

        (train_X, train_y, train_angles) = gen_data(num_train, side, num_bins,
                                                    thickness)
        train_loaders = get_train_loaders(train_X, train_y, batch_size)

        for epochs in [100, 500]:

            (nets, optimizers, criterions) = get_model_stuff(train_y, input_size,
                                                             hidden_size, output_sizes,
                                                             learning_rate, momentum)

            for enc in train_y:
                optimizer = optimizers[enc]
                net = nets[enc]
                criterion = criterions[enc]

                for epoch in range(epochs):
                    for (i, (images, ys)) in enumerate(train_loaders[enc]):
                        optimizer.zero_grad()

                        outputs = net(images.to(device))
                        loss = criterion(outputs, ys.to(device))
                        loss.backward()
                        optimizer.step()


            print("Training Size: {0}".format(num_train))
            print("Training Epochs: {0}".format(epochs))
            for enc in train_y:
                net = nets[enc]
                preds = net(torch.tensor(test_X, dtype=torch.float).to(device))
                if enc == "binned":
                    pred_bins = np.array(preds.argmax(dim=1).detach().cpu().numpy(),
                                         dtype=np.float)
                    pred_angles = bin_size * pred_bins + half_bin_size
                else:
                    pred_angles = torch.atan2(preds[:, 1], preds[:, 0]).detach().cpu().numpy()
                    pred_angles[pred_angles < 0] = pred_angles[pred_angles < 0] + 2 * np.pi

                print("Encoding: {0}".format(enc))
                print("Test Error: {0}".format(np.abs(pred_angles - test_angles).mean()))

            print()


if __name__ == "__main__":
    main()
$\endgroup$
3
$\begingroup$

Here is my Python version of your experiment. I kept many of the details of your implementation the same, in particular I use the same image size, network layer sizes, learning rate, momentum, and success metrics.

Each network tested has one hidden layer (size = 500) with logistic neurons. The output neurons are either linear or softmax as noted. I used 1,000 training images and 1,000 test images which were independently, randomly generated (so there may be repeats). Training consisted of 50 iterations through the training set.

I was able to get quite good accuracy using binning and "gaussian" encoding (a name I made up; similar to binning except that the target output vector has the form exp(-pi*([1,2,3,...,500] - idx)**2) where idx is the index corresponding to the correct angle). The code is below; here are my results:

Test error for (cos,sin) encoding:

1,000 training images, 1,000 test images, 50 iterations, linear output

  • Mean: 0.0911558142071

  • Median: 0.0429723541743

  • Minimum: 2.77769843793e-06

  • Maximum: 6.2608513539

  • Accuracy to 0.1: 85.2%

  • Accuracy to 0.01: 11.6%

  • Accuracy to 0.001: 1.0%

Test error for [-1,1] encoding:

1,000 training images, 1,000 test images, 50 iterations, linear output

  • Mean: 0.234181700523

  • Median: 0.17460197307

  • Minimum: 0.000473665840258

  • Maximum: 6.00637777237

  • Accuracy to 0.1: 29.9%

  • Accuracy to 0.01: 3.3%

  • Accuracy to 0.001: 0.1%

Test error for 1-of-500 encoding:

1,000 training images, 1,000 test images, 50 iterations, softmax output

  • Mean: 0.0298767021922

  • Median: 0.00388858079174

  • Minimum: 4.08712407829e-06

  • Maximum: 6.2784479965

  • Accuracy to 0.1: 99.6%

  • Accuracy to 0.01: 88.9%

  • Accuracy to 0.001: 13.5%

Test error for gaussian encoding:

1,000 training images, 1,000 test images, 50 iterations, softmax output

  • Mean: 0.0296905377463
  • Median: 0.00365867335107
  • Minimum: 4.08712407829e-06
  • Maximum: 6.2784479965
  • Accuracy to 0.1: 99.6%
  • Accuracy to 0.01: 90.8%
  • Accuracy to 0.001: 14.3%

I cannot figure out why our results seem to be in contradiction with one another, but it seems worth further investigation.

# -*- coding: utf-8 -*-
"""
Created on Mon Jun 13 16:59:53 2016

@author: Ari
"""

from numpy import savetxt, loadtxt, round, zeros, sin, cos, arctan2, clip, pi, tanh, exp, arange, dot, outer, array, shape, zeros_like, reshape, mean, median, max, min
from numpy.random import rand, shuffle
import matplotlib.pyplot as plt

###########
# Functions
###########

# Returns a B&W image of a line represented as a binary vector of length width*height
def gen_train_image(angle, width, height, thickness):
    image = zeros((height,width))
    x_0,y_0 = width/2, height/2
    c,s = cos(angle),sin(angle)
    for y in range(height):
        for x in range(width):
            if abs((x-x_0)*c + (y-y_0)*s) < thickness/2 and -(x-x_0)*s + (y-y_0)*c > 0:
                image[x,y] = 1
    return image.flatten()

# Display training image    
def display_image(image,height, width):    
    img = plt.imshow(reshape(image,(height,width)), interpolation = 'nearest', cmap = "Greys")
    plt.show()    

# Activation function
def sigmoid(X):
    return 1.0/(1+exp(-clip(X,-50,100)))

# Returns encoded angle using specified method ("binned","scaled","cossin","gaussian")
def encode_angle(angle, method):
    if method == "binned": # 1-of-500 encoding
        X = zeros(500)
        X[int(round(250*(angle/pi + 1)))%500] = 1
    elif method == "gaussian": # Leaky binned encoding
        X = array([i for i in range(500)])
        idx = 250*(angle/pi + 1)
        X = exp(-pi*(X-idx)**2)
    elif method == "scaled": # Scaled to [-1,1] encoding
        X = array([angle/pi])
    elif method == "cossin": # Oxinabox's (cos,sin) encoding
        X = array([cos(angle),sin(angle)])
    else:
        pass
    return X

# Returns decoded angle using specified method
def decode_angle(X, method):
    if method == "binned" or method == "gaussian": # 1-of-500 or gaussian encoding
        M = max(X)
        for i in range(len(X)):
            if abs(X[i]-M) < 1e-5:
                angle = pi*i/250 - pi
                break
#        angle = pi*dot(array([i for i in range(500)]),X)/500  # Averaging
    elif method == "scaled": # Scaled to [-1,1] encoding
        angle = pi*X[0]
    elif method == "cossin": # Oxinabox's (cos,sin) encoding
        angle = arctan2(X[1],X[0])
    else:
        pass
    return angle

# Train and test neural network with specified angle encoding method
def test_encoding_method(train_images,train_angles,test_images, test_angles, method, num_iters, alpha = 0.01, alpha_bias = 0.0001, momentum = 0.9, hid_layer_size = 500):
    num_train,in_layer_size = shape(train_images)
    num_test = len(test_angles)

    if method == "binned":
        out_layer_size = 500
    elif method == "gaussian":
        out_layer_size = 500
    elif method == "scaled":
        out_layer_size = 1
    elif method == "cossin":
        out_layer_size = 2
    else:
        pass

    # Initial weights and biases
    IN_HID = rand(in_layer_size,hid_layer_size) - 0.5 # IN --> HID weights
    HID_OUT = rand(hid_layer_size,out_layer_size) - 0.5 # HID --> OUT weights
    BIAS1 = rand(hid_layer_size) - 0.5 # Bias for hidden layer
    BIAS2 = rand(out_layer_size) - 0.5 # Bias for output layer

    # Initial weight and bias updates
    IN_HID_del = zeros_like(IN_HID)
    HID_OUT_del = zeros_like(HID_OUT)
    BIAS1_del = zeros_like(BIAS1)
    BIAS2_del = zeros_like(BIAS2)

    # Train
    for j in range(num_iters):
        for i in range(num_train):
            # Get training example
            IN = train_images[i]
            TARGET = encode_angle(train_angles[i],method) 

            # Feed forward and compute error derivatives
            HID = sigmoid(dot(IN,IN_HID)+BIAS1)

            if method == "binned" or method == "gaussian": # Use softmax
                OUT = exp(clip(dot(HID,HID_OUT)+BIAS2,-100,100))
                OUT = OUT/sum(OUT)
                dACT2 = OUT - TARGET
            elif method == "cossin" or method == "scaled": # Linear
                OUT = dot(HID,HID_OUT)+BIAS2 
                dACT2 = OUT-TARGET 
            else:
                print("Invalid encoding method")

            dHID_OUT = outer(HID,dACT2)
            dACT1 = dot(dACT2,HID_OUT.T)*HID*(1-HID)
            dIN_HID = outer(IN,dACT1)
            dBIAS1 = dACT1
            dBIAS2 = dACT2

            # Update the weight updates 
            IN_HID_del = momentum*IN_HID_del + (1-momentum)*dIN_HID
            HID_OUT_del = momentum*HID_OUT_del + (1-momentum)*dHID_OUT
            BIAS1_del = momentum*BIAS1_del + (1-momentum)*dBIAS1
            BIAS2_del = momentum*BIAS2_del + (1-momentum)*dBIAS2

            # Update the weights
            HID_OUT -= alpha*dHID_OUT
            IN_HID -= alpha*dIN_HID
            BIAS1 -= alpha_bias*dBIAS1
            BIAS2 -= alpha_bias*dBIAS2

    # Test
    test_errors = zeros(num_test)
    angles = zeros(num_test)
    target_angles = zeros(num_test)
    accuracy_to_point001 = 0
    accuracy_to_point01 = 0
    accuracy_to_point1 = 0

    for i in range(num_test):

        # Get training example
        IN = test_images[i]
        target_angle = test_angles[i]

        # Feed forward
        HID = sigmoid(dot(IN,IN_HID)+BIAS1)

        if method == "binned" or method == "gaussian":
            OUT = exp(clip(dot(HID,HID_OUT)+BIAS2,-100,100))
            OUT = OUT/sum(OUT)
        elif method == "cossin" or method == "scaled":
            OUT = dot(HID,HID_OUT)+BIAS2 

        # Decode output 
        angle = decode_angle(OUT,method)

        # Compute errors
        error = abs(angle-target_angle)
        test_errors[i] = error
        angles[i] = angle

        target_angles[i] = target_angle
        if error < 0.1:
            accuracy_to_point1 += 1
        if error < 0.01: 
            accuracy_to_point01 += 1
        if error < 0.001:
            accuracy_to_point001 += 1

    # Compute and return results
    accuracy_to_point1 = 100.0*accuracy_to_point1/num_test
    accuracy_to_point01 = 100.0*accuracy_to_point01/num_test
    accuracy_to_point001 = 100.0*accuracy_to_point001/num_test

    return mean(test_errors),median(test_errors),min(test_errors),max(test_errors),accuracy_to_point1,accuracy_to_point01,accuracy_to_point001

# Dispaly results
def display_results(results,method):
    MEAN,MEDIAN,MIN,MAX,ACC1,ACC01,ACC001 = results
    if method == "binned":
        print("Test error for 1-of-500 encoding:")
    elif method == "gaussian":
        print("Test error for gaussian encoding: ")
    elif method == "scaled":
        print("Test error for [-1,1] encoding:")
    elif method == "cossin":
        print("Test error for (cos,sin) encoding:")
    else:
        pass
    print("-----------")
    print("Mean: "+str(MEAN))
    print("Median: "+str(MEDIAN))
    print("Minimum: "+str(MIN))
    print("Maximum: "+str(MAX))
    print("Accuracy to 0.1: "+str(ACC1)+"%")
    print("Accuracy to 0.01: "+str(ACC01)+"%")
    print("Accuracy to 0.001: "+str(ACC001)+"%")
    print("\n\n")


##################
# Image parameters
##################
width = 100 # Image width
height = 100 # Image heigth
thickness = 5.0 # Line thickness

#################################
# Generate training and test data
#################################
num_train = 1000
num_test = 1000
test_images = []
test_angles = []
train_images = []
train_angles = []
for i in range(num_train):
    angle = pi*(2*rand() - 1)
    train_angles.append(angle)
    image = gen_train_image(angle,width,height,thickness)
    train_images.append(image)
for i in range(num_test):
    angle = pi*(2*rand() - 1)
    test_angles.append(angle)
    image = gen_train_image(angle,width,height,thickness)
    test_images.append(image)
train_angles,train_images,test_angles,test_images = array(train_angles),array(train_images),array(test_angles),array(test_images)



###########################
# Evaluate encoding schemes
###########################
num_iters = 50

# Train with cos,sin encoding
method = "cossin"
results1 = test_encoding_method(train_images, train_angles, test_images, test_angles, method, num_iters)
display_results(results1,method)

# Train with scaled encoding
method = "scaled"
results3 = test_encoding_method(train_images, train_angles, test_images, test_angles, method, num_iters)
display_results(results3,method)

# Train with binned encoding
method = "binned"
results2 = test_encoding_method(train_images, train_angles, test_images, test_angles, method, num_iters)
display_results(results2,method)

# Train with gaussian encoding
method = "gaussian"
results4 = test_encoding_method(train_images, train_angles, test_images, test_angles, method, num_iters)
display_results(results4,method)
$\endgroup$
  • $\begingroup$ Cool, On key different. You are only training on each image once. I am training on each image 1,000 times, or 10,000 times. Multiple Iterations though the training data are normal, particularly when training on relatively small amounts of data (and it only took me one unpublishable undergrad thesis to learn this, but that is another story). With that said, I should add a 1 iter column to my table. that would be informative $\endgroup$ – Lyndon White Jun 13 '16 at 13:37
  • $\begingroup$ I would think that training on similar (but not identical) images with similar targets would affect that network similarly. If that is true, it should work fine to simply increase the number of random images being trained on, rather iterating many times through a smaller training set. Are you saying this is not the case? $\endgroup$ – Ari Herman Jun 13 '16 at 19:45
  • $\begingroup$ It is similar, but for this example task it doesn't has the issue that eventually you will show all possible images so your test will overlap with your train, so testing generalistion won't work. More significantly though you do 100,000 training images, which is <1000*1000 training images*Iterations. $\endgroup$ – Lyndon White Jun 14 '16 at 0:00
  • $\begingroup$ You are correct, I will fix that issue. There is an even more significant problem with my code: I am using logistic neurons which are incapable of producing the negative values required by the (cos,sin) representation. D'oh! I will be revising my code and re-posting as soon as possible. $\endgroup$ – Ari Herman Jun 14 '16 at 0:40
  • $\begingroup$ You might (if you haven't already done so) be interested in doing a Graident Check, which is worthwhile when implemented neural networks from scratch, as it is very easy to make a minor mistake and have your network still mostly work. Re: Neuron: yeah, I have a linear output layer, onto of the sigmoid hidden layer $\endgroup$ – Lyndon White Jun 14 '16 at 0:46
1
$\begingroup$

Another way to encode the angle is as a set of two values:

y1 = max(0,theta)

y2 = max(0,-theta)

theta_out = y1 - y2

This would have the similar problem to arctan2 in that the gradient is undefined at theta = 0. I don't have the time to train a network and compare to the other encodings but in this paper the technique seemed reasonably successful.

New contributor
DerekG is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
  • 1
    $\begingroup$ This seems like an answer mixed in with another question in one post. This site operates a bit differently from the way forum would. Here answers should concentrate on answering the original question. And if you have another question or a comment - it should be posted as such. $\endgroup$ – Karolis Koncevičius Oct 18 at 22:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.