I can make it work in R

    X = matrix(c(1, -1, -0.3,
                 1,  1,  0.6,
                 1, 20, 20), 3 , byrow = TRUE)
    
    beta = c(-0.2,0.4,1.2)
    
    y = X %*% beta
    
    est = solve(t(X) %*% X) %*% t(X) %*% y
    est  ### output = -0.2, 0.4, 1.2

I have no matlab on my computer to test what is going wrong with your equation. I believe that the `\` is used correctly. Possibly you could test the expression `solve(A.'*A) * (A.'*Y);` instead.