2
$\begingroup$

If a simple ANN was trained to predict the next step in a sequence, such as a univariate time series, can it be considered a generative model?

$\endgroup$

1 Answer 1

1
$\begingroup$

A natural way to specify a generative model for sequences is $$P(x_{1\ldots t}) = \prod_{i=1}^t P(x_i | \{x_j \text{ for } 0<j<i\})$$

So we need to specify a $P(x_i | \{x_j \text{ for } 0<j<i\})$. A reasonable choice could be something like $P(x_i | \{x_j \text{ for } 0<j<i\}) = \mathcal{N}(\mu=f(x_{i-n}, \ldots, x_{i-1}; \theta), \sigma^2=1)$ where $f$ is your NN and $\theta$ are it's parameters. There are still some details to resolve here, like what happens if $i < n$? (Maybe impute missing values with 0?). Is $\sigma^2=1$ a reasonable choice? If you used L1 loss, should we use a laplace distribution instead of normal?

Anyway, the point is that a NN itself isn't a generative model, it's just one part of the model. In simple cases like binary classification, it might be "obvious" what the model is, but often it's a bit trickier and there are some important details to take care of.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.