Model: I am working on a "shallow" (3-layer) auto encoder neural network. The input layer receives a, say 25-dimensional, vector $x$ of numerical elements representing client purchases. Several thousand of these vectors form the training set. The network should learn purchase patterns by reconstructing the inputs at the output layer (it is trained by minimizing the MSE between input and output).

Data: The large majority of purchases are rather small (below \$1k) but some can get quite large (several \$million). Overall, the distribution of purchases is highly skewed to the lower side with a long tail on the right. What feature transformation is required before using the data optimally in the neural network?:

  1. log-transformation: I assume this is necessary to combat the skeweness? is taking $x=\log(1+x)$ the correct approach to avoid $\log(0)$ for very small numbers?
  2. scaling to $[0,1]$ range: I assume this is done for algorithmic optimization as it yields a steeper gradient in e.g. sigmoid or tanh activation functions? However, taking a look at these functions (where the steepest gradient is at $x=0$), wouldn't we rather scale to $[-1,1]$ instead?
  3. Would the above 2 transformations of $x$ be sufficient for simply passing the samples from left to right through the network? Should further normalization and standardization be applied on top of the above to bring the data distribution closer to $\mathcal{N}(\mu,\sigma^2)$ and $\mathcal{N}(0,1)$?
  4. Data Extension: if the data in $x$ instead consists not of the client purchases themselves but rather of distributional properties of these purchases (i.e. $x$ now being numericals representing mean, skew, kurtosis, standard deviation), would we still apply above transformations to it? I am not sure if taking the log of a number representing the skew would make sense
  • $\begingroup$ Taking logarithms of price makes sense as a purchase of 0$\$$, is not a purchase, and normally defined purchases are not negative numbers of dollars. Whether this is optimal or not depends on the population and in general one tests several transformations of data to see which is best based on criteria that are decided upon before testing. $\endgroup$ – Carl Dec 31 '18 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.