Does bias input work like constant value in linear regression? and if bias input is not used then resulting boundary will always pass through origin?
Thanks in advance.
Yes, that is exactly right. The name "bias unit" is just a different name for a "constant" that would be included in regular linear (or logistic) regressions.
Just as you mention in your question, it allows the fitted line (decision boundary) not to pass through the origin.
Look at this question for greater detail: Importance of the bias node in neural networks