Searching for the way to initialize the matrix weights as orthogonal (i.e. W*W^T = I
and all the eigenvalues are equal either 1 or -1),(I was wrong) I found this post with a code from Lasagne:
https://stats.stackexchange.com/questions/228704
However, this code does not produce orthogonal, but unitary matrix with complex eigenvalues, which still satisfies equality W*W^T = I quite well:
import numpy as np
np.random.seed(1000)
def floatX(arr):
"""Converts data to a numpy array of dtype ``theano.config.floatX``.
Parameters
----------
arr : array_like
The data to be converted.
Returns
-------
numpy ndarray
The input array in the ``floatX`` dtype configured for Theano.
If `arr` is an ndarray of correct dtype, it is returned as is.
"""
return np.asarray(arr, dtype=np.float32)
def sample(shape,gain=1.0):
flat_shape = (shape[0], np.prod(shape[1:]))
a = np.random.normal(0, 1, flat_shape).astype(np.float32) #get_rng().normal(0.0, 1.0, flat_shape)
u, _, v = np.linalg.svd(a, full_matrices=False)
# pick the one with the correct shape
q = u if u.shape == flat_shape else v
q = q.reshape(shape)
res = floatX(gain * q)
#eigenvalues:
eigenvalues, _ = np.linalg.eig(u)
print(str(np.min(eigenvalues))+" "+ str(eigenvalues[1024])+ " "+ str(np.max(eigenvalues)) )
return res
W = sample([2048, 2048])
print(np.linalg.norm(W.dot(np.transpose(W))-np.eye(2048)))
The output is
(-1+0j) (0.376566+0.92639j) (1+0j)
for some of the eigenvalues and
1.00117376019e-05
for the proximity of the W*W^T
and I
.
1) Is this a bug in Lasagne, as numpy.linalg.svd
indeed produces unitary matrix as it is stated in its manual, and
2) How can I initialize a (non-square) orthogonal matrix for myself in Python, except the unit one?
Thank you!
EDIT: Thanks everyone! From all the explanations I've managed to create a simple Python solution:
import numpy as np
## Using a subset of vectors
np.random.seed(1000)
p = 4096
A = np.random.normal(0,1,(p,p))
q, _ = np.linalg.qr(A) # q -orthonormal
print(q.shape) #(4096,4096)
q = q[:,::2] #cut q
print(q.shape) #(4096,2048) m>n ==> q^T * q = I_n
print(np.linalg.norm(np.transpose(q).dot(q)-np.eye(2048))) # 4.26e-14
1.00117376019e-05
is because you use 32 bit float numbers. If you use 64 bit instead, you norm would be even smaller. In my machine I got2.0298795662912376e-13
$\endgroup$ – itdxer Nov 15 '17 at 20:15