2
$\begingroup$

I'm trying to understand the "Combinatorial Purged Cross-Validation" technique for time series data described in Marcos Lopez de Prado's "Advances in Financial Machine Learning" book (p. 163).

The setup is described as the researcher wanting to test "a number $\phi$ of backtest paths." I'm not really sure what that means, but here's what I have so far:

  • A time series is split into $N$ sequential groups
  • A number $k$ is chosen for cross validation
  • A combinatoric equation is used to calculate the "number of paths": $$ \phi(N, k) = \frac{k}{N}{N \choose N - k}. $$

For the case of $N=6$ groups and $k=2$, there are $\phi(6, 2)=5$ paths and Figure 12.1 from the book lays them out as a table. The number of train / test CV split" is 15 (6 choose 2), which are indexed as the columns in the table below. The rows are the 6 groups, and the numbers inside are the path ids from 1 to 5.

enter image description here

The book states, "Path 2 is the result of combining forecasts from (G1,S2), (G2,S6), (G3,S6), (G4,S7), (G5,S8) and (G6,S9)." The passage of time through the G-groups, I can see. What I'm not following is how the splits relate to the groups.

People obviously think highly of this book. Here's a video of someone explaining Combinatorial Purged Cross Validation, but it didn't answer my questions. Can anybody tell me what's going on here? Is this truly an advancement over Walk Forward Cross Validation?

$\endgroup$
1
$\begingroup$

Another popular question of mine. Well here's to self-help, starting with a sketch illustrating the 3 paths in the $N = 4, k =2$ situation:

enter image description here

The number of ways to arrange the 2 test sets to occur in 4 time periods is ${4 \choose 2} = 6$, and $\frac{k}{N} = .5$ is the fraction of the combinations that will start with a test set. Since a "path" is a continuous group of blocks from the first to the last sequential group, there are .5 * 6 = 3 paths, which aligns with $\phi(N, k)$ from the question.

Here's a sketch for a more complicated example with N = 5 and k = 2, which leads to 4 paths:

enter image description here

You might not like my question, but at least I have great handwriting.

$\endgroup$
1
$\begingroup$

I had the same question I asked the person who created that methodology on twitter. Here is a link to his response.

Suppose that you have $6$ folds. A CV where you leave $2$ folds out (instead of the standard 1) at each split allows you to compute $5$ estimates for each datapoint. Instead of one PnL line, you can now estimate $5$ PnL lines.

New contributor
Max 1000000 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.