From Elements of Statistical Learning:
For a two-class G, one approach is to denote the binary coded target as Y , and then treat it as a quantitative output. The predictions Yˆ will typically lie in [0,1], and we can assign to Gˆ the class label according to whether ˆ y > 0.5. This approach generalizes to K-level qualitative outputs as well.
I am python programming the least square regression in section 2.3.1 of the book.
p_or set of orange points,
p_bl set of blue points which i concatenate to
#X the set of all points X = p_or+p_bl m = len(X) #Add column of ones to account for the bias term X = np.array([list(np.ones(m)),[x for x in X],[x for x in X]]).T #y is coded as -1 for orange, 1 for blue y = np.array( list(-1*np.ones(len(p_or))) + list(np.ones(len(p_bl))) ) #calculating βˆ = (X_T*X)^(−1)*X_T*y beta = np.linalg.inv( X.T @ X )@( X.T @ y ) #Plot the boundary! line_x = np.linspace(-5, 5) line_y = -beta / beta - (beta / beta) * line_x plt.plot(line_x, line_y)
Question is the following - I encoded categorical outputs with -1 and 1
y = np.array( list(-1*np.ones(len(p_or))) + list(np.ones(len(p_bl))) ). This way the regression looks great. If I encode with 0 and 1 regression doesn't look good : regression boundary is biased towards one of the two sets of points ( graphically on the plot ). More rigorously, how should be a categorical variable encoded ? is there a demonstrable rule for this ?