Any insights onto how one can predict coefficients in lets say a regular or a polynomial equation using machine-learning/neural networks, i.e.,

$\alpha x_i + \beta x_i^2 + \gamma x_i^3 = y_i,$

where $i=1,2,3,...N.$

I have training data with inputs x and corresponding y. Coefficients to predict are a, b, c, and d. I am yet to spot valid literature related to the subject via google.

Thank you


This is a linear system, and if $N \geq 4$ and all your $x_i$ are unique and nonzero, then you can solve exactly for $a, b, c, d$ using gaussian elimination (or your preferred solver).

  • $\begingroup$ Don't you mean $N\le 4$? For $N \gt 4,$ this is a multiple regression problem and Gaussian elimination won't solve it. $\endgroup$ – whuber Nov 20 '19 at 23:02
  • $\begingroup$ I know that we have other ways to solve this but the question mentions, finding coefficients through the use of machine-learning/ neural networks techniques. $\endgroup$ – Skyward Nov 20 '19 at 23:11
  • $\begingroup$ @Skyward Your definition of machine learning excludes linear algebra? $\endgroup$ – Sycorax says Reinstate Monica Nov 20 '19 at 23:23

How you decide to solve a problem depends on what is known and what is unknown

First, we can observe that the function $f(x_i) = \sum_{j=0}^3 x_i^j\beta_j$ is linear in the coefficients to be estimated. So if we know for a fact that $f$ has this form, we need nothing more complicated than regression. Indeed, the design matrix of this regression problem is a Vandermonde matrix. The biggest challenge you'll face when approaching this as a regression problem is that Vandermonde matrices are typically ill-conditioned, so finite-precision arithmetic may lose significant digits. There are lots of tricks to improve on using the raw polynomials, such as rescaling the columns of the design matrix.

However, if your purpose is to model an unknown function and you have chosen for the sake of an experiment to model this $f$, we can use any of the many, many statistical methods for making inferences about smooth functions in general. Many modern statistical methods were developed to approximate these types of functions. These are useful when all you have are the $\left(f(x_i), x_i\right)$ tuples and little-to-no additional information. Neural networks are one example, but we also have Gaussian processes, spline regression, nearest neighbors, random kitchen sinks, among many others.

I draw this distinction because it's unclear to me which assumptions OP is making. OP's comment to @shimao suggests OP is not interested in linear regression as a solution, but instead prefer a "modern" approach. This is all well and good; it can be instructive to understand under which circumstances different alternative methods achieve a certain result. For example, we can evaluate how well some particular method recovers $f$ to some degree of precision.

However, in OP's answer, OP proposes a "neural network" which has a functional form that is exactly identical to the regression strategy, because we can rewrite OP's "neural network" as a linear function, and discover that it has the form of linear regression where the design matrix is a Vandermonde. So, if we can do that, then how is OP's "neural network" different from @shimao's regression solution? (Or, we can accept OP's premise that OP's answer is a neural network, then we can view @shimao's answer as a "neural network" which you can estimate using any linear regression method.)

The only quantities which OP's "neural network" is estimating are the regression coefficients $\beta$, so the only difference between @shimao's approach and OP's is whether the linear system is solved using matrix factorization or gradient descent. Choosing whether to use gradient descent or matrix factorization depends on circumstances which are not in evidence in the question (size of the data, computational resources available, desired precision, whether or not the data arrives in a "streaming" fashion, etc.).

The reason that neural networks are interesting is that they do not have to know the form of $f$ to make a good approximation.* For example, the Cybenko Universal Approximation Theorem finds that certain functions can be approximated by a shallow neural network with sigmoid activations. This is an interesting result, because even though $f$ is not itself sigmoidal, there are certain conditions under which we can approximate $f$ arbitrarily well with a certain number of hidden sigmoid units. This is why neural networks do not typically fix certain weights, nor do they use the known functional form of $f$ as its activation. The point of a neural network is that it has generality.

* Subject to some terms and conditions. Consult the appropriate UAT for more details.

The difference between machine learning and statistics is largely a matter of branding

I am extremely skeptical of anyone who belabors the distinction between statistics and machine learning. These people are trying to sell you something, and if their pitch hinges on disparaging statistics in favor of machine learning, the thing they're selling you is probably snake oil.

Machine learning is statistics; statistics is machine learning.


Please refer to http://ceur-ws.org/Vol-284/page107.pdf

The neural network architecture adopted is shown in the figure. From the input layer to the only hidden layer, I have fixed the weights to be equal to 1 for both connections. In first neuron of the hidden layer I a squaring my weighted input, i.e., $(1*x)^2$, and in the second neuron of the hidden layer I am taking cube of the weighted input, i.e., $(1*x)^3$. The output layer has its inputs as stated in the question (also given below),

$\alpha x_i + \beta x_i^2 + \gamma x_i^3 = y_i$

where $\alpha$, $\beta$, and $\gamma$ are the coefficients to be learnt.

Below is the code

import torch
from torch.optim import Adam

mat1 = torch.tensor([[1.,1.]],requires_grad=False)
mat2 = torch.randn(2,1,requires_grad=True)
mat3 = torch.randn(1,1,requires_grad=True)

optim = Adam([ {'params': mat2},  {'params' :mat3} ], lr=1e-4)

print("Mat1 size : {}".format(mat1.size()))
print("Mat2 size : {}".format(mat2.size()))
print("Mat3 size : {}".format(mat3.size()))

def forward(x) :
   middle = x*mat1

   print("Middle : {}".format(middle))

   middle[0,0] = middle[0,0]**2
   middle[0,1] = middle[0,1]**3

   print("Middle : {}".format(middle))

   final = torch.matmul(middle,mat2)+ mat3*x
   return final

def loss(pred,truth) :
   return (pred-truth)**2

target = torch.tensor([30.]) #y is set equal to 30.
x = torch.tensor([40.])      #x is set equal to 40.

num_epochs =100000

while(num_epochs>0) :
   prediction = forward(x)
   loss_val = loss(prediction,target)

   print("Prediction : {}".format(prediction))
   print("Alpha : {}".format(mat3[0,0]))
   print("Beta : {}".format(mat2[0,0]))
   print("Gamma : {}".format(mat2[1,0]))

num_epochs -= 1

Please run the code for at least epochs>3000 to get reasonable values for the coefficients.

Neural network architecture adopted

  • $\begingroup$ This answer seems to miss the point of using a neural network. By fixing some weights at 1, and using squares etc., you've defined a network that will always exactly match a regression aimed at the same task. One reason neural networks are useful is that a good network requires no feature engineering: the network/back-prop figures out on its own that approximating $y$ requires some functions that are kind of like squares and cubes of $x$, and how to approximate those functions using compositions of activation functions. This is why people care about universal approximation theorems etc. $\endgroup$ – Sycorax says Reinstate Monica Nov 23 '19 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.