Let $f$ be a mapping from dataset space $\mathcal{D}$ to $\mathbb{R}$, where $$ \mathcal{D}:=\{M_1,M_2,\ldots,M_k\} $$ with $M_i\in\mathbb{R}^{n\times n}$ a symmetric matrix. Typically, $n=36$.

We can, for a given matrix $M$, compute $f(M)$; however, even though the function $f$ is well-defined, it is computationally pretty complicated, so it costs usually a lot of time to determine the function values when we are dealing with large dataset spaces $\mathcal{D}$, i.e. large$=k=1$mill. Hence, I am exploring new fields to speed up this process.

I came across a machine learning cheat-sheet and I think that ML might be able to help us out here to solve our problem. So, I am aiming at predicting the function value of $f$ for a given matrix $M$.

My, probably naive, idea was to divide the dataset space into disjunct spaces \begin{align} \mathcal{D}&:=\{M_1,M_2,\ldots,M_k\}\\ &=\{M_1,M_2,\ldots,M_\ell\}\cup \{M_{\ell+1},\ldots,M_k\}\\ &=\mathcal{D}_{train-test}\cup \mathcal{D}_{predict}. \end{align}

Additionally, I was thinking of representing matrices $M$ of the train/test set as rows of data using its elements $m_{ij}$, i.e. to write the space $\mathcal{D}_{train-test}$ as a "proper dataset":

\begin{align} \text{matrix } M_1:\quad & m_{11}^1\quad & m_{12}^1 \quad& \cdots &m_{nn}^1 \quad & f(M_1)\\ \text{matrix } M_2:\quad & m_{11}^2\quad & m_{12}^2 \quad& \cdots &m_{nn}^2 \quad & f(M_2)\\ \vdots\quad & \vdots\quad & \vdots \quad& \cdots &\vdots \quad & \vdots\\ \text{matrix } M_\ell:\quad & m_{11}^\ell\quad & m_{12}^\ell \quad& \cdots &m_{nn}^\ell \quad & f(M_\ell) \end{align}

to predict by using some machine learning methods (SVR according to the scikit-learn algorithm cheat-sheet) the function values in the following table: \begin{align} \text{matrix } M_{\ell+1}:\quad & m_{11}^{\ell+1}\quad & m_{12}^{\ell+1} \quad& \cdots &m_{nn}^{\ell+1} \quad & ?\\ \vdots\quad & \vdots\quad & \vdots \quad& \cdots &\vdots \quad & \vdots\\ \text{matrix } M_k:\quad & m_{11}^k\quad & m_{12}^k \quad& \cdots &m_{nn}^k \quad & ? \end{align}

On the other hand, deep learning methods might be interesting as well, but I am by no means an expert. Therefore, I would really appreciate your suggestions. How can I approach this problem?

  • $\begingroup$ So you already know what this function looks like? Is it continuous and differentiable? If not, deep learning won't work. $\endgroup$ – horaceT Mar 8 '18 at 17:44
  • $\begingroup$ It doesn't have to be continuous and differentiable -- a network can be trained to approximate the step function. It just needs to be reasonably well behaved for this to work (no crazy discontinuities, sharp peaks, etc). $\endgroup$ – shimao Mar 8 '18 at 21:08

You're on the right track.

  • You definitely want to vectorize your matrices so that you're in the classic supervised learning, regression paradigm of predicting a scalar value $y_i$ using a vector $\mathbf{x}_i$. $$ \mathbf{x}_i = \begin{bmatrix} m_{11} \\ \ldots \\ m_{1n} \\ m_{21} \\ \ldots \end{bmatrix} \quad \quad y_i = f(\mathbf{x}_i)$$ Now your data is a bunch of pairs $(y_i, \mathbf{x}_i)$, and you can apply totally off the shelf techniques. You'll almost certainly be better off though having a background in theory (eg. go through Andrew NG's online course etc...). What regression technique can best capture structure in $f$ will depend on what $f$ is.

  • The simplest split is indeed to partition your data into a single $\mathcal{D}_{\mathrm{train}}$ and $\mathcal{D}_{\mathrm{test}}$. Many techniques involve an additional partition $\mathcal{D}_{\mathrm{validate}}$ for model selection. (Eg. train $k$ algorithms on $\mathcal{D}_{\mathrm{train}}$, find which of the set performs best in $\mathcal{D}_{\mathrm{validate}}$, then estimate final performance of that algorithm on $\mathcal{D}_{\mathrm{test}}$. Taking the next step, an extremely important and useful technique is cross-validation.

    • If you're fitting a function, you probably want to randomly assign observations to training and test set. (Many tools will do this automatically btw.) And a point of caution, you want your training set independent of your test set (eg. in the medical context, you wouldn't want to assign different observations from the same patient to both training and test set).

Anyway, the key thing is to vectorize your data then you can apply standard techniques you'll learn in any machine learning or regression course.

This question in a sense is overly broad (and lacks context). It basically reduces to, "I have a regression problem, how should I approach it?" That's too broad to answer (and indeed impossible to say much on at all without context). I think you are on the right track though.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.