Q Learning Function Approximation When I have the following function
$$Q(s,a;w) = w_1f_1(s,a)+\cdots +w_nf_n(s,a)$$ in reinforcement learning. $f(s,a)$ is the feature vector.
How is $f(s,a)$ defined? When I have state $s_0$ and two actions, can I choose {left, right}? How would the feature vectors look?
 A: 
How is $f(s,a)$ defined?

$$f(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}^n$$

When I have state $s_0$ and two actions, can I choose {left, right}?

Yes. The feature vector function $f(s,a)$ is something you have to implement that converts the domains of your state and action into a vector, in order to use it with your chosen function approximation method (in your example, a linear approximation using an inner product with another vector of learnable function parameters).
You can choose "raw" representations that suit the problem description  naturally. It will then be your task to define how those map to a useful feature vector.

How would the feature vectors look?

They will be a vector of real numbers of fixed dimension $n$. This is necessary because of the type of function approximation you have chosen.
You are free to choose how the action part maps to values in the feature vector. Two simple options are:

*

*$\{left, right\} \rightarrow \{[1,0], [0,1]\}$ i.e. one-hot coding

*$\{left, right\} \rightarrow \{-1, 1\}$
When using a linear approximation, you may also want to add interaction terms with the state. You may also decide to perform some other transformation on the raw data using e.g. radial basis functions, tile coding, and make the resulting vector your feature vector.
Ideally, you should choose a mapping to a feature vector that you have reason to believe will have a close to linear relationship with expected future return, because that is what the suggested approximation will work with best. If such a mapping is not immediately obvious then you probably need to look at some form of coding (e.g. radial basis functions, tile coding) over some more natural mapping. You may also be better off looking at non-linear function approximation (e.g. DQN) when you suspect the relationship between state, action and action value could be complex.
