We are measuring data with a high sample rate (20 kHz) and calculated a big standard error due to our system setup. Currently we are only interested in slow signals (in the order of Hz's) is it valid to use averaging (once per 20.000 samples) and thus lower our standard error with an order 20.000?
Averaging is just a low pass filter so if you want to filter out the high frequency component of your signal, it will do that.
Obviously different averaging techniques and different parameters will filter out high frequency components in a different way. See for instance this illustration of the frequency response of a simple moving average.
Yes, you can reduce the noise by averaging or in general by reducing the sampling rate of the signal. This is a very common technique in signal processing commonly refereed to as oversampling. It is widely used in sigma-delta converters. Oversampling basically means that the sampling rate of the signal is much higher than the needed Nyquist limit (i.e. 2 times higher than bandwidth of the signal). By low-pass filtering and decimating the signal, you reduce the standard deviation of the noise by the decimation factor (assuming a uniform distribution for the noise).
Having said that, I doubt in your example you can downsample by a factor of 20000. From your description it seems that your "signal" include frequency up to few Hertz. Notice that to get best/correct results you should sample your signal higher than the Nyquist limit. For example if you want to preserve everything below 20 Hz, you should not decimate by a factor more than 500 (i.e. 20000 / 500 = 40 Hz = 2 x 20 Hz). If you are not much into signal processing, to do the downsampling correctly I suggest using a ready made package so that you know the lowpass-filter is implemented correctly. In MATLAB you can use the resample function in the signal-processing toolbox. In R you may use the decimate function from the signal package.