Why is not a scalar feature enough to encode 3-component binary numbers in an autoencoder?

Question

I am trying to build an intuition on what really a feature is. I created a toy example as following. In my mind a scalar feature should be enough to represent my data. Couldn't the model in this case learn a mapping from binary numbers to decimal? For example, the scalar feature for [0,0,0] = 0, [0,0,1] = 1, [0,1,0] = 2 etc. I know feature space is continuous but I could draw the same assumption for continuous feature values. I.e, [0,0,0] belongs to the [-0.5, 0.5) feature space range, [0,0,1] to [0.5, 1.5) etc.

In my example, the autoencoder predicts [0,0,0] for every input? Is my intuition just wrong about the meaning of features? If so, how would you explain it?

import torch
import torch.utils.data
from torch import nn
import tqdm


# This autoencoder has only one feature as latent space
class AE(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(nn.Linear(3, 1, bias=False), nn.ReLU())
        self.decoder = nn.Sequential(nn.Linear(1, 3, bias=False), nn.ReLU())

    def forward(self, x):
        z = self.encoder(x)
        xhat = self.decoder(z)
        return xhat


class dummyData(torch.utils.data.Dataset):
    def __init__(self):
        self.samples = [[0, 0, 0],
                        [0, 0, 1],
                        [0, 1, 0],
                        [0, 1, 1],
                        [1, 0, 0],
                        [1, 0, 1],
                        [1, 1, 0],
                        [1, 1, 1]]

    def __getitem__(self, item):
        return torch.tensor(self.samples[item]).type(torch.float32)

    def __len__(self):
        return len(self.samples)

autoencoder = AE()
optimizer = torch.optim.Adam(autoencoder.parameters())
dummyDataset = dummyData()
dataloader = torch.utils.data.DataLoader(dummyDataset, batch_size=2, shuffle=False)
criterion = nn.MSELoss()

for epoch in tqdm.tqdm(range(10000)):
    for sample in dataloader:
        pred = autoencoder(sample)
        loss = criterion(pred, sample)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

print("TEST")
autoencoder.eval()
print(autoencoder.encoder(torch.tensor([0., 0., 0.])))
print(autoencoder(torch.tensor([0., 0., 0.])))
print(autoencoder.encoder(torch.tensor([0., 0., 1.])))
print(autoencoder(torch.tensor([0., 0., 1.])))
print(autoencoder.encoder(torch.tensor([0., 1., 1.])))
print(autoencoder(torch.tensor([0., 1., 1.])))
print(autoencoder.encoder(torch.tensor([1., 0., 0.])))
print(autoencoder(torch.tensor([1., 0., 0.])))
´´´

Answer 1

If we write out the problem, the neural network you're using is trying to estimate $A, B$ in

$$ x = (Ax)B $$ where $x$ has shape $3 \times 1$, $A$ has shape $1 \times 3$ and $B$ has shape $1 \times 3$. (I've omitted the activation here.)

It would be tempting to write $$ B^{-1}x=Ax $$ but this isn't possible because in the proposed network $A,B$ are vectors, not square matrices. (Indeed, if you use $3\times 3$ matrices instead, then the problem is easily solved, but one must wonder if this demonstration has accomplished anything at all.)

The encoding step $Ax$ can be expressed as a linear operation, because these 3-bit vectors $x$ have identical floats $y$, because $[4,2,1]^\top x=y$.

However, the decoding task is to go from float $y$ back to binary $x$: $z ^\top y = z$ for a fixed $z$ of shape $1 \times 3$. This isn't expressible as a vector-vector product.

In general, we're seeking solutions of the form

$$ x = [ay, by, cy] $$

but there is not a way to choose fixed $a,b,c$ from among real numbers so that $x$ is binary and $y$ is the integers 0 through 7. For $y=7$, you want $a = 1/7$, but that fails for $y=2$ because the result is $2/7$.