I can make it work in R
X = matrix(c(1, -1, -0.3,
1, 1, 0.6,
1, 20, 20), 3 , byrow = TRUE)
beta = c(-0.2,0.4,1.2)
y = X %*% beta
est = solve(t(X) %*% X) %*% t(X) %*% y
est ### output = -0.2, 0.4, 1.2
I have no matlab on my computer to test what is going wrong with your equation. I believe that the \ is used correctly. Possibly you could test the expression solve(A.'*A) * (A.'*Y); instead.