# Deal with noise data

The following picture represents a graph with price over time. I am a mathematical student, but also a trader. I want to create a function which could localize the good entry and exit points for sale and buy stocks. However, I am struggling with the noisy data. Simply using $p_t > p_{t+k}$ to determine the direction of the price would introduce unmanageable amount of noise, since the smallest change would register as an upward or downward movement. The green circle on the picture shows an unacceptable noise, while the pink circle is an acceptable noise and induce a new exit and entry point.

How could I deal with noisy points? I want to set a point of entry and exit when the initial price has increased by a certain percentage, but I need to tell my function not to consider a drop of a certain percentage (noise). If the drop reaches that percentage and there is a sufficient increase in the initial price, then I will an exit point.

Some people explain it is possible to set up a threshold. How could we explain that mathematically in that context?

## UPDATE

We want to predict the direction towards which the price will change. The term price is used to refer to the mid-price of a stock, which is defined as the mean between the best bid price and best ask price at time $t$: $$p_t = \frac{p_a^{(1)}(t)+p_b^{(1)}(t)}{2}$$

This is a virtual value for the price since no order can happen at that exact price, but predicting its upwards or downwards movement provides a good estimate of the price of the future orders. A set of discrete choices must be constructed from our data to use as targets for our classification model. Simply using $p_t > p_{t+k}$ to determine the direction of the mid-price would introduce unmanageably amount of noise, since the smallest change would be registered as an upward or downward movement.

In order to filter the noise, I think it is good to following smoothed approach. First, the mean of the previous $k$ mid-prices, denoted by $m_b$, and the mean of the next $k$ mid-prices, denoted by $m_a$, are defined as: $$m_a(t) = \frac{1}{k} \sum_{i=1}^{k} p_{t-i}$$ $$m_b(t) = \frac{1}{k} \sum_{i=0}^{k} p_{t+i}$$

A label $l_t$ that express the direction of price movement at time $t$ is extracted by comparing the previously defined quantities ($m_b$ and $m_a$):

$$l_t = \begin{cases} 1, & m_b(t) > m_a(t) (1+α)\\ -1, & m_b(t) < m_a(t) (1-α) \\ 0, & \text{otherwise} \end{cases}$$

where the threshold $α$ is set as the least amount of change in price that must occur for it to be considered upward or downward. If the price does not exceed this limit, the sample will be considered to belong to the stationary class. Therefore, the resulting label expresses the current trend we wish to predict. Note that this process is applied for every time step in our data.

Is it a good way to handle the noise? I could take into account a threshold $\alpha=0.05$ and an horizon $k=10$. A big advantage of using the mean of surrounding prices is that we can prevent drastic drops. In other words, if there was a drastic drop a time $t$, then we can set up an exit point at a time $t'$, where $t'<t$, in order to establish a certain security. To not be caught in a descent too fast.

• You should give this question a title that reflects the effort you put into the question itself! Apr 24, 2018 at 15:01
• @MatthewDrury What do you suggest? Apr 24, 2018 at 15:02

Let me save you a lot of money. There is a lot of information that you do not have, but I will give you a solution to a similar problem.

For starters, let us assume that the chart you have drawn was, in fact, a real chart. For starters, there is a critical piece of information that you and most traders are lacking. The data in the chart doesn't have to happen in the order on the chart. Large orders are taken "off the tape," and so trades happen that never appear as individual trades. They are aggregated together and placed in block orders later in the tape at a point where the order won't impact the trade price.

Your entry points may have happened an hour before they are recorded, likewise, your exits may have also happened earlier as well. The charts do not provide actual informative data. Second, the actual order is never placed in the public record. If there is a single market-maker, then the market-maker has the true trade order, but no one else does. If there are multiple market makers, then no one has the true order. Charts and data feeds are created from recording order and not trade order. This protects large orders from shocking the market up or down. Those steep dips you marked in green could actually be a block order being recorded late because the market-maker couldn't find a place to put them. They are also recorded as weighted average prices and not the actual prices that were paid.

That is the first part of your problem.

The second part of your problem is that you are probably using the wrong type of mathematical tools. Consider the apparent function $$p_{t+1}=Rp_t+\epsilon_{t+1}.$$ It isn't a function though until after the trade is complete. $\epsilon_{t+1}$ is a multivalued constant. In fact, if it follows a normal distribution, it has an infinite number of solutions, so ex-ante $x_{t+1}\in\mathbb{R}^{++}$. That isn't a useful piece of knowledge. It is a relation.

Now let us add some assumptions to make some math possible. We are ignoring the quantity purchased and sold since we could normalize it to unity anyway over a short period of time. For long periods bankruptcy or mergers could happen.

Now, it is known by theorem that no non-Bayesian solution exists for the equation $p_{t+1}=Rp_t+\epsilon_{t+1}.$ This is because $R>1$ or nobody would invest. A Bayesian solution does exist, but it won't really answer the question you are asking anyway because, as I said above, the data isn't in the true order in which it happened.

What you can do is note that risk falls with the price for any security, if you define risk in terms of reaching a goal. This is because when return is defined as $$r_t=\frac{p_{t+1}}{p_t},$$ then the smaller the bottom number is the higher the return is. Further, if you define a cost function where your cost is zero when your goal is met or exceeded and $c$ when the goal fails, then your cost function becomes the expected loss, which is now just the CDF of the values between -100% and your target return.

The risk falls because as the price falls, the cdf falls below 50% and in a binomial distribution risk is maximized at 50%. If the probability of loss is greater than 50% the risk falls because it becomes increasingly certain that you will fail. If the probability of loss is less than 50% then the probability of gain becomes increasingly certain.

In practice, since the goal is now binary, you can speak in terms of the binomial distribution. This is great since returns for going concerns should be approximately a Cauchy distribution, which has neither a mean or a variance. To check this, note that the expectation of the standard Cauchy is $$\frac{1}{\pi}\int_{-\infty}^\infty\frac{x}{1+x^2}\mathrm{d}x.$$ If you check the integral, you will see it diverges and so a solution does not exist. If you check the integral of the second moment about zero you get $$\frac{1}{\pi}\int_{-\infty}^\infty\frac{x^2}{1+x^2}\mathrm{d}x.$$

For this to work, however, you really need to ground prices in operating earnings. So you should base your returns as a function of adjusted operating income, as opposed to reported earnings. It is better to use adjusted operating cash flows, but that may require data that is a bit more difficult to come by. I say adjusted because accounting data is not always reflective of its true value. Most accounting records need very substantive adjustments. Consider a firm with a deferred tax liability of \$100 million. Imagine the tax would be first due in 20 years. Then that is a \$100 million interest-free loan from the Treasury. The difference in the debt and the present value is equity. That equity needs to be adjusted as the tax date gets closer. Likewise, goodwill is worthless and should be immediately written off.

If you keep doing the above, then you should save money by donating your portfolio to charity. People like me will take you for everything you have got, and if you make margin purchases, we may take more than 100% of what you have got. You cannot trade like that and make money because you are not a financial institution and cannot get access to the data you need to solve this problem.

NOTE FOR COMPLETENESS

In case you are wondering why it would follow a Cauchy distribution consider a trade on the NYSE. Prices happen in a double auction so the winner's curse does not obtain. Since the winner's curse is not operative the rational behavior is to bid your expectation. If there are many potential buyers and sellers then the static distribution of potential bids is the normal distribution. Statically, prices in equilibrium should converge to a normal distribution. This is not to say that the limit book is normal as not all orders are in the book. A person does not have to place an order to have a valuation.

By well known theorem, the ratio of two normal distributions, the sell price divided by the buy price, is a Cauchy distribution if you are centered around (0,0) which you would be since in equilibrium the joint prices would be there if you have translated the prices into the error space from the price space. Hence, no variance or, for that matter, a mean.

• You need to know the function I want to build is for deep neural network purposes. So I need to label data first before training my models. So in my case, at a time $t$, I know the future prices. In other words, if I am at a time $t$, I know the price $p_{t+1}, p_{t+2}, ..., p_{t+k}$. So I want to use that argument to find the entry and exit points. Apr 24, 2018 at 3:37
• Your deep neural network will suffer from the same real-world problem that trades do not happen in the order on the tape. You also need to validate that the neural network maps to a Bayesian solution or it won't matter. There is a non-existence proof out there. There is no point in building something that does not exist. Apr 24, 2018 at 3:40
• Alternatively, you could construct a neural network that uses a strictly concave and increasing utility function, but I believe it would be biased, possibly quite biased. Apr 24, 2018 at 3:42
• In fact, I am using a recurrent neural network with depth market data in input (32 features). Apr 24, 2018 at 3:43
• You do not know the prices at $p_{t+k}$ because the tape is inaccurate on purpose. The data record is reordered so that block orders do not interefere with liquidity management. Apr 24, 2018 at 3:44