1
$\begingroup$

I've implemented a rather simple stacked autoencoder using lasagne and Theano

from functools import reduce
import itertools

import numpy as np
import theano
import lasagne

This is how I initialise the weights.

class SigmoidInit(lasagne.init.Initializer):
    def __init__(self, n_hid, n_vis):
        """
        :type n_hid: int
        :param n_hid: the number of hidden units
        :type n_vis: int
        :param n_vis: the number of visible (output) units
        """

        if not isinstance(n_hid, int) or n_hid < 1:
            raise ValueError("`n_hid` must be a positive integer")
        if not isinstance(n_vis, int) or n_hid < 1:
            raise ValueError("`n_vis` must be a positive integer")

        self.n_hid = n_hid
        self.n_vis = n_vis

    def sample(self, shape):
        weights_array = np.asarray(
            np.random.uniform(low=-4 * np.sqrt(6. / (self.n_hid + self.n_vis)),
                              high=4 * np.sqrt(6. / (self.n_hid + self.n_vis)),
                              size=shape),
            dtype=theano.config.floatX
        )
        return theano.shared(value=weights_array, name='W', borrow=True)

And the rest of the stuff

def yield_batches(arrays, batch_size, shuffle=True):
    """
    Yields batches of arrays.
    :type arrays: collections.Sequence[np.ndarray]
    :param arrays: a sequence of arrays to generate batches from, e.g.
                   X-values arrays and Y-values array
    :type batch_size: int
    :param batch_size: the number of entries per batch
    :type shuffle: bool
    :param shuffle: shuffle order (doesn't affect `x_train` and `y_train`
                    objects)
    :rtype: Generator[tuple[np.ndarray]]
    :return: a generator, yielding tuples of array batches
    """
    if len(set(len(array) for array in arrays)) != 1:
        raise ValueError("arrays have different length")

    n_entries = len(arrays[0])

    indices = np.arange(n_entries)
    if shuffle:
        np.random.shuffle(indices)

    return (tuple(array[indices[i:i+batch_size]] for array in arrays)
            for i in range(0, n_entries, batch_size))

def create_theano_variable(ndim, dtype, name=None):
    try:
        return {1: tensor.vector(name, dtype=dtype),
                2: tensor.matrix(name, dtype=dtype),
                3: tensor.tensor3(name, dtype=dtype),
                4: tensor.tensor4(name, dtype=dtype)}[ndim]
    except KeyError:
        raise ValueError("`ndim` must be an integer in [1, 4]")


def tensor_from_array(array, name=None):
    # TODO docs
    """
    :type array: np.ndarray
    :param array:
    :type name: str
    :param name:
    :rtype: T.TensorVariable
    """
    return create_theano_variable(ndim=array.ndim,
                                  dtype=str(array.dtype).split(".")[-1],
                                  name=name)


def inject_random_noise(x, p=0.5):
    mask = np.random.binomial(1, p, x.size).reshape(x.shape).astype(bool)
    x_hat = x.copy()
    x_hat[mask] = 0
    return x_hat


def build_autoencoder(n_inp, n_hid, nonlinearity):

    input_shape = (None, n_inp)

    l_inp = lasagne.layers.InputLayer(input_shape)
    l_hid = lasagne.layers.DenseLayer(l_inp, n_hid,
                                      W=SigmoidInit(n_hid, n_inp),
                                      nonlinearity=nonlinearity)

    # init output with tied weights
    l_out = lasagne.layers.DenseLayer(l_hid, n_inp, W=l_hid.W.T)

    return l_out


def train(network, x, y, epochs, batchsize, loss_fn, update_fn, learning_rate,
          **kwargs):
    # create target var
    # note: I use my own `tensor_from_array` instead of `theano.shared`,
    #       because for whatever reason Theano says I can't use a shared
    #       variable here and that I should pass it via the `givens`
    #       parameter, whatever that is.
    input_var = lasagne.layers.get_all_layers(network)[0].input_var
    target_var = tensor_from_array(x)

    # training functions
    prediction = lasagne.layers.get_output(network,
                                           deterministic=True)
    loss = loss_fn(prediction, target_var).mean()
    params = lasagne.layers.get_all_params(network, trainable=True)
    updates = update_fn(loss, params, learning_rate=learning_rate, **kwargs)
    train_fn = theano.function([input_var, target_var],
                               loss, updates=updates)

    def run_epoch(x_, y_):
        train_batches = yield_batches((x_, y_), batchsize)
        train_err = np.mean([train_fn(*batch) for batch in train_batches])
        return train_err

    return (run_epoch(x, y) for _ in itertools.repeat(None, epochs))


def pretrain(autoencoders, x, y, epochs, batchsize, loss_fn, update_fn,
             learning_rate, **kwargs):
    """
    :param networks: a sequence of autoencoders.
    """
    ... # simply iteratively train each subsequent autoencoder to recover the former's hidden representation.

My data are 100000k vectors of 16 real numbers in $[0, 1]$. I'm using adadelta update function with $\rho = 0.95$ and $\varepsilon = {10}^{-6}$. Learning rate is 1.0, input random noise level is set to 0.2, batchsize is 5000. The first denoising autoencoder with 30 hidden units, which gets the original data, shows some shocking error rates. I trained it for 10k generations, just to find the squared error decreased from ~1200 to ~1000 (yes, this high) and got stuck. Subsequent autoencoders (with 20, 14, 20 and 30 hidden units respectively) trained pretty well reaching error rates of ~ 0.002. After stacking the encoders together and starting the fine-tuning, the overall error rate decreased from 1100 to 750 in 20000 generations. Still, this is something insane. Any recommendations? I can give the data and the rest o the code, if needed. Not all variables are pair-wise correlated, but for each variable there is at least 1 pair with significant Spearman correlation.

$\endgroup$
  • $\begingroup$ Have you tried a smaller learning rate? 1.0 seems awfully high. $\endgroup$ – horaceT Aug 18 '16 at 16:51
  • $\begingroup$ @horaceT yes, I tried all the way from 0.05 to 0.5 (step 0.05) with Nesterov momentum updates. I ended up with 1 because that's what lasagne's authors recommend (due to this paper: Zeiler, M. D. (2012): ADADELTA: An Adaptive Learning Rate Method. arXiv Preprint arXiv:1212.5701) $\endgroup$ – Eli Korvigo Aug 18 '16 at 17:07
  • $\begingroup$ For some problems, ADADELTA can perform badly. I would try full batch gradient descent update with line search. $\endgroup$ – Curious Aug 19 '16 at 6:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.