# From Bayesian Networks to Neural Networks: how multivariate regression can be transposed to a multi-output network

I'm dealing with a Bayesian Hierarchical Linear Model, here the network describing it.

$Y$ represents daily sales of a product in a supermarket(observed).

$X$ is a known matrix of regressors, including prices, promotions, day of the week, weather, holidays.

$S$ is the unknown latent inventory level of each product, which causes the most problems and which I consider a vector of binary variables, one for each product with $1$ indicating stockout and so the unavailability of the product. Even if in theory unknown I estimated it through a HMM for each product, so it is to be considered as known as X. I just decided to unshade it for proper formalism.

$\eta$ is a mixed effect parameter for any single product where the mixed effects considered are the product price, promotions and stockout.

$\beta$ is the vector of fixed regression coefficients, while $b_1$ and $b_2$ are the vectors of mixed effects coefficient. One group indicates brand and the other indicates flavour (this is an example, in reality I have many groups, but I here report just 2 for clarity).

$\Sigma_{\eta}$ , $\Sigma_{b_1}$ and $\Sigma_{b_2}$ are hyperparameters over the mixed effects.

Since i have count data let's say that I treat each product sales as Poisson distributed conditional on the Regressors (even if for some products the Linear approximation holds and for others a zero inflated model is better). In such a case I would have for a product $Y$ (this is just for who's interested in the bayesian model itself, skip to the question if you find it uninteresting or non trivial :)):

$\Sigma_{\eta} \sim IW(\alpha_0,\gamma_0)$

$\Sigma_{b_1} \sim IW(\alpha_1,\gamma_1)$

$\Sigma_{b_2} \sim IW(\alpha_2,\gamma_2)$, $\alpha_0,\gamma_0,\alpha_1,\gamma_1,\alpha_2,\gamma_2$ known.

$\eta \sim N(\mathbf{0},\Sigma_{\eta})$

$b_1 \sim N(\mathbf{0},\Sigma_{b_1})$

$b_2 \sim N(\mathbf{0},\Sigma_{b_2})$

$\beta \sim N(\mathbf{0},\Sigma_{\beta})$, $\Sigma_{\beta}$ known.

$\lambda _{tijk} = \beta*X_{ti} + \eta_i*X_{pps_{ti}} + b_{1_j} * Z_{tj} + b_{2_k} Z_{tk}$,

$Y_{tijk} \sim Poi(exp(\lambda_{tijk}))$

$i \in {1,\dots,N}$, $j \in {1,\dots,m_1}$, $k \in {1,\dots,m_2}$

$Z_i$ matrix of mixed effects for the 2 groups, $X_{pps_i}$ indicating price, promotion and stockout of product considered. $IW$ indicates inverse Wishart distributions, usually used for covariance matrices of normal multivariate priors. But it's not important here. An example of a possible $Z_i$ could be the matrix of all the prices, or we could even say $Z_i=X_i$. As regards the priors for the mixed-effects variance-covariance matrix, I would just try to preserve the correlation between the entries, so that $\sigma_{ij}$ would be positive if $i$ and $j$ are products of the same brand or either of the same flavour.

The intuition behind this model would be that the sales of a given product depend on its price, its availability or not, but also on the prices of all the other products and the stockouts of all the other products. Since I don't want to have the same model (read: same regression curve) for all the coefficients, I introduced mixed effects which exploit some groups I have in my data, through parameter sharing.

My questions are:

1. Is there a way to transpose this model to a neural network architecture? I know that there are many questions looking for the relationships between bayesian network, markov random fields, bayesian hierarchical models and neural networks, but I didn't find anything going from the bayesian hierarchical model to neural nets. I ask the question about neural networks since, having a high dimensionality of my problem (consider that I have 340 products), parameter estimation through MCMC takes weeks (I tried just for 20 products running parallel chains in runJags and it took days of time). But I don't want to go random and just give data to a neural network as a black box. I would like to exploit the dependence/independence structure of my network.

Here I just sketched a neural network. As you see, regressors($P_i$ and $S_i$ indicate respectively price and stockout of product $i$) at the top are inputed to the hidden layer as are those product specific (Here I considered prices and stockouts). (Blue and black edges have no particular meaning, it was just to make the figure more clear) . Furthermore $Y_1$ and $Y_2$ could be highly correlated while $Y_3$ could be a totally different product (think about 2 orange juices and red wine), but I don't use this information in neural networks. I wonder if the grouping information is used just in weight inizialization or if one could customize the network to the problem.

Edit, my idea:

My idea would be something like this: as before, $Y_1$ and $Y_2$ are correlated products, while $Y_3$ is a totally different one. Knowing this a priori I do 2 things:

1. I preallocate some neurons in the hidden layer to any group I have, in this case I have 2 groups {($Y_1,Y_2$),($Y_3$)}.
2. I initialize high weights between the inputs and the allocated nodes (the bold edges) and of course I build other hidden nodes to capture the remaining 'randomness' in the data.

• @Tomasso Guerrini here is possible the answer for you: stats.stackexchange.com/questions/4498/… – Anton Danilov Feb 11 '17 at 7:22
• thanks @AntonDanilov , unfortunately the accepted answer says 'While neural networks come with "graphs" they generally don't encode dependence information, and the nodes don't represent random variables' :/ – Tommaso Guerrini Feb 11 '17 at 7:51
• Have you tried Stan, or it's not feasible for your problem? Hamiltonian Monte Carlo can be orders of magnitude faster than Gibbs sampling, and scales well to hundreds (or even thousands) of variables. – lacerbi Mar 28 '17 at 0:09
• Have you tried posing the same question(s) to the Stan users mailing list? They are usually extremely helpful with technical aspects to make a model work. For example, it's possible that issues in your case can be solved with a better parameterization. (Hamiltonian Monte Carlo should mix much faster than Gibbs sampling.) – lacerbi Mar 28 '17 at 0:43
• Probably it's better to say I spammed that mailing list .. Thank you very much Luigi by the way.. I'm in that situation where I have no more time to dig into the problems as I should, since I have an incoming deadline.. It seems like STAN is a great tool, but the learning curve is a little steep to really realize its incredible performance (as of now I realized its speed up wrt JAGS) – Tommaso Guerrini Mar 28 '17 at 0:50

• I don't currently have the time to work out the details, but as a thought experiment imagine a data set where the independent variables were all statisitically significant, but were not all correlated. For example $y = \sum x_i + \sum z_i$ where $z_i = A \times \sigma^{-1}(x_i)$ with A very large in absolute value and $\sigma$ the (non-linear) activation function of the network. Because of the non-linear relationship of y and the z one could have that the correlation between y and the $z_i$ is approximately zero. Yet they should be a part of any good model. – aginensky Feb 15 '17 at 14:11