Tensorflow InvalidArgumentError: The determinant is not finite [closed]

I'm trying to fit a Mixture of Gaussians to a data set. First the data is clustered using K-Means Clustering. Each cluster is then fitted with a Gaussian.To avoid inversion of large covariance matrix, Wood-bury matrix inversion lemma is used. $$$$\mathcal{L} = \sum_{n=1}^{N} \log \sum_{i=1}^{K} \pi_i P(x_n | \mu_i, \Sigma_i) = \sum_{n=1}^{N} \log \sum_{i=1}^{K} e^{[log (\pi_i) + \log P(x_n | \mu_i, \Sigma_i)]} ,$$$$ where, $$$$log P(x | \mu, \Sigma) = -\frac{1}{2}\big[d \log(2\pi) + \log \det(\Sigma) + (x-\mu)^{T} \Sigma^{-1} (x-\mu)\big]$$$$ $$$$\Sigma^{-1} = D^{-1} - D^{-1} A L_{l \times l}^{-1} A^T D^{-1}$$$$ $$$$\log \det(AA^T+D) = \log \det(I + A^T D^{-1} A) + \log \det D = \log \det L_{l \times l} + \sum_{j=1}^{d} \log d_j$$$$ I do the following:

K_exp_sum = 0
logK_exp_sum = 0
for i in xrange(k):
dvar_slice= tf.slice(dvar, [i, 0], [1, num_features])
D         = tf.squeeze(tf.diag(dvar_slice), axis=[0, 2])
Din       = tf.squeeze(tf.diag(1/dvar_slice), axis=[0, 2])
Asl       = tf.squeeze(tf.slice(A, [i, 0, 0], [1, num_features, latent_dim]), axis=0)
ATDin     = tf.matmul(tf.transpose(Asl), Din)
L         = tf.eye(latent_dim)+ tf.matmul(ATDin, Asl)
Sigin     = Din - tf.matmul(tf.matmul(Din, Asl), tf.matmul(tf.matrix_inverse(L), ATDin))
logdetL   = tf.log(tf.clip_by_value(tf.matrix_determinant(L), 1e-10, np.inf))
logdetD   = tf.log(tf.clip_by_value(tf.reduce_prod(dvar_slice), 1e-10, np.inf))
const     = num_features*tf.log(2*3.1415926535)
logphi    = tf.transpose(tf.log(tf.slice( tf.matmul(phi, tf.ones([1, batch_size])), [i, 0], [1, batch_size])))
xSiginxt  = tf.diag_part(tf.matmul(tf.matmul(X-K_centers[i], Sigin), tf.transpose(X-K_centers[i])))
K_exp     = tf.exp(-0.5*(tf.reshape(xSiginxt, [batch_size, 1])+logdetL+logdetD+const)+logphi)
K_exp_sum = K_exp_sum+K_exp
# print("Step %i/%i completed"% (i, k))
logK_exp_sum  = logK_exp_sum+tf.reduce_sum(tf.log(tf.clip_by_value(K_exp_sum,1e-10,np.inf)))
logK_exp_sum = -logK_exp_sum


I get the following error:

InvalidArgumentError: The determinant is not finite.

Caused by op u'Model_likelihood/MatrixDeterminant', defined at:
File "/home/user/bin/anaconda2/envs/condaenv/bin/ipython", line 11, in <module>
sys.exit(start_ipython())
File "/home/user/bin/anaconda2/envs/condaenv/lib/python2.7/site-packages/IPython/__init__.py", line 119, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "/home/user/bin/anaconda2/envs/condaenv/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/home/user/bin/anaconda2/envs/condaenv/lib/python2.7/site-packages/IPython/terminal/ipapp.py", line 355, in start
self.shell.mainloop()
File "/home/user/bin/anaconda2/envs/condaenv/lib/python2.7/site-packages/IPython/terminal/interactiveshell.py", line 495, in mainloop
self.interact()
File "/home/user/bin/anaconda2/envs/condaenv/lib/python2.7/site-packages/IPython/terminal/interactiveshell.py", line 486, in interact
self.run_cell(code, store_history=True)
File "/home/user/bin/anaconda2/envs/condaenv/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2714, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/home/user/bin/anaconda2/envs/condaenv/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2818, in run_ast_nodes
if self.run_code(code, result):
File "/home/user/bin/anaconda2/envs/condaenv/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-1-da7fdb6d9d66>", line 65, in <module>
logdetL   = tf.log(tf.clip_by_value(tf.matrix_determinant(L), 1e-10, np.inf))
File "/home/user/bin/anaconda2/envs/condaenv/lib/python2.7/site-packages/tensorflow/python/ops/gen_linalg_ops.py", line 273, in matrix_determinant
result = _op_def_lib.apply_op("MatrixDeterminant", input=input, name=name)
File "/home/user/bin/anaconda2/envs/condaenv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/user/bin/anaconda2/envs/condaenv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/user/bin/anaconda2/envs/condaenv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): The determinant is not finite.