1
$\begingroup$

I have multivariate time series data of the EURUSD financial vehicle. In this data each variable represents a different metric. There are ~200,000 rows and ~20 variables. There are no NULL values for any variable at any row. All data is numerical.

Alongside this data, at each time point I have the univariate data "Profit."

I want to curve-fit a function to transforms my multivariate data set into a new univariate data set which having the MAXIMAL correlation to my "Profit" variable.

In other words, I want to iterate through different mathematical transformations of my multivariate data set until I find the one that is optimally correlated with my "Profit" data.

What is the best way to do this? From what I understand, a genetic algorithm should work well.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ This is a relatively ill-posed problem. You can achieve a correlation of 1 by creating 200000 variables from your 20 variables using powers, interactions, and other nonlinear functions of the data. To make this problem reasonable, you need to impose some regularity conditions on your data. If you are talking about curve fitting, these may be restrictions on the curvature (2nd derivative) of your resulting curve. I personally tend to trust splines a little bit more than the neural networks and other fancy stuff: just run a regression with splines as basis functions... doable in any package. $\endgroup$
    – StasK
    Commented Jan 24, 2012 at 1:33
  • $\begingroup$ What I am trying to do is to create 1 variable out of my 20, not 200000 out of 20. However, if need be, here are some constraints: 1. p value must maximally signififcant. 2. the formula must be symmetrically optimal for "buy" orders and "sell orders". But this is optional. How do I run a regression through a custom formula in, say, R? (pearson's correlation coefficient) $\endgroup$
    – Mike Furlender
    Commented Jan 24, 2012 at 3:48
  • 1
    $\begingroup$ Stask's point is that you can create 1 variable via any arbitrarily large number of intermediate additional variables (interactions, transformations, etc) so that by the time you have n of them you are guaranteed to have the maximum correlation with profit. $\endgroup$
    – Peter Ellis
    Commented Jan 24, 2012 at 10:37

2 Answers 2

Reset to default
3
$\begingroup$

The traditional approach to this sort of problem is:

  • if you have a theoretical reason for a relationship between your explanatory variables and your response (profit), then base a model on that, and test it rigorously...
  • if you don't, then look at the 20 plots with each of your variables on the horizontal axis and the response (profit) on the vertical axis, and look for obvious relationships, or transformations (logarithm normally the first one) that make relationships reasonably straightforward - if not linear, at least easily approximated by splines or locally linear regressions (see StasK's comment)
  • then, create a set of plausible linear models with profit as your response and your transformed or splined (if that's a word) variables as explanatory variables. Compare the models against some criterion of goodness of fit eg AIC or BIC (plenty of debate on which to use). Be careful to adjust p-values downwards to allow for the fact that you have implicitly looked at 2^20 different models.

Unfortunately, any of those dot points above could be a major chapter or book. R can do anything necessary. I'd use plots rather than correlation co-efficients; and read some of the large literature on model selection and fitting.

Improve this answer
$\endgroup$
1
$\begingroup$

The best way to find most 'significant' parameters in such problems is Principal Component Analysis. Alternatively, finding the correlation matrix of your data will do the job too. Using principal component analysis, you can basically identify the parameters with most variance which are often of most interest. Once, you have identified the most 'significant' ones, you may actually reduce dimensions of your data and then simply look for underlying relations among them. In the new reduced order dimensions, you can find the correlations of new dimensions (also called as principal components) with the original parameters. Finally, the parameters having most correlation with principal components will have the maximum and most significant variance / correlation. Hope it helps.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Can you say more about how they would be used, & why they would work? $\endgroup$
    – gung - Reinstate Monica
    Commented Jan 3, 2016 at 0:43
  • $\begingroup$ For that you will need to do a little study on the technique itself. As a quick start if you have matlab, arrange all your data in a matrix form with columns representing variables and rows their samples / values. Import it into matlab. If you don't know how to import it, after arranging the data in excel, simply copy it all and leave the column headings unselected. Next, declare any variable in Matlab like A = 1. Double click it from the workspace and paste the excel data into it. Next use this command, R = corrcoef(A), it will create correlation matrix. In next comment I will explain it $\endgroup$
    – Mist
    Commented Jan 3, 2016 at 0:48
  • 1
    $\begingroup$ I don't think @gung is seeking instruction on the technique or on the code you might use in any particular software. His point is, I suspect, rather that this answer is highly undeveloped and essentially two unsubstantiated assertions. $\endgroup$
    – Nick Cox
    Commented Jan 3, 2016 at 0:52
  • 1
    $\begingroup$ Thanks for extending your answer. There is some slackness of usage over "variables" and "parameters". Every mention of parameters is, in conventional statistical terms, strictly a reference to variables. $\endgroup$
    – Nick Cox
    Commented Jan 3, 2016 at 1:04
  • 1
    $\begingroup$ Thanks; I am familiar with correlations. None of these thresholds is especially convincing. I've seen the same arguments about 0.3 and 0.7: all could be used as a personal rule of thumb to separate weak from strong; none is anything but arbitrary. $\endgroup$
    – Nick Cox
    Commented Jan 3, 2016 at 1:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.