Are weights 1-D or 2-D in softmax Regression?

Question

I've learned ML and have been learning DL from Andrew Ng's coursera courses, and every time he talks about a linear classifier, the weights are just a 1-D vector. Even during the assignments, when we roll an image into a 1-D vector (pixels * 3), the weights would still be a 1-D vector.

I now have started O'Reilly's "Learning TensorFlow" book, and came across the first example. The weights initialization in tensorflow was a bit different.

The book says the following (Page 14):

Since we are not going to use the spatial information at this point, we will unroll our image pixels as a single long vector denoted x (Figure 2-2). Then $xw^0 = ∑x_i w^0_i$ will be the evidence for the image containing the digit 0 (and in the same way we will have $w^d$ weight vectors for each one of the other digits, d = 1, . . . , 9).

and the corresponding TensorFlow code:

data = input_data.read_data_sets(DATA_DIR, one_hot=True)

x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 10]))

y_true = tf.placeholder(tf.float32, [None, 10])
y_pred = tf.matmul(x, W)

Why are the weights 2-D here. Are weights 2-D in softmax Linear Classifier? In the coursera course, when he taught Softmax Linear Classifier, he still says the weights are 1-D. Can anyone explain this?

Answer 1

In neural networks, the weights in a layer are always a matrix of shape [number of inputs, number of outputs], connecting every input to every output. When one of those is 1, the weights can be seen as 1D vector. In your case, there are 784 inputs and 10 outputs, so the weights are 2D.

Note aside: what you are doing is classification, not regression.

Answer 2

In the sample network below, the weight of the hidden layer will be a 2D matrix (say 4x5 (or) 5x4 depending upon the implementation in the framework) and the weights of the Output layer will be a 1D matrix (5x1).

The inputs will be 1D array of shape (1x4).

If the hidden layer weights are stored as (4x5), then they direct matrix multiplication is possible with the inputs i.e, x.w = (1x4) x (4x5) = (1x5)

If the hidden layer weights are stored as (5x4), then the weights matrix is transposed before matrix multiplication is possible with the inputs i.e, x.w_trans (1x4) x (4x5) = (1x5)

Image source: https://www.researchgate.net/profile/Ahmed_Thabit/publication/315561192/figure/fig4/AS:475189663277059@1490305452719/Figure-4-MLP-neural-network-structure-The-weights-of-the-neural-network-are-updated-in.jpg