How to discover relationships between features? I'm an undergraduate fairly new to statistics. I am trying to find relationships between the features of a dataset. The dataset in question consists of 5000 objects each with up to 23 features.
I was looking for methods which could relate many features. The idea would be to find a sort of "equation" that would relate some features.
My guess is that this problem would have something to do with regression analysis. In the past MIC was applied to this dataset, but without any results. From what I understand, it is limited to only analyzing the relationship between 2 features at a time. 
I was suggested to use Neural Networks. However, in my limited understanding, I'm not sure the data would be big enough to do so. Furthermore, arent NN like black-boxes, would it be possible to extract how it does a prediction?
Any pointers of possibly useful techniques/resources to learn from, are greatly appreciated!
Many thanks for the help.
 A: EDA  (Exploratory data analysis)

One of the basic techniques is using correlation and with that scatterplots, but like you mentioned that limits you to two features at a time.
As you have quite a number of features, I would suggest that you use something like a biplot instead. A biplot will plot all the features at the same time. I actually used it in an EDA for a data set last year. It had about 65000 observations and 65 features.
Depending on the nature of your data, you might also consider using histograms (mostly for categorical data).
I have also seen many people using heat maps. I like to use these to see where missing values are, but they can also be helpful to some degree in determining the relationship between features.
Model building


I was suggested to use Neural Networks. However, in my limited understanding, I'm not sure the data would be big enough to do so. Furthermore, aren't NN like black-boxes, would it be possible to extract how it does a prediction?

When it comes to model building you can possibly look at multiple regression. Of course the great thing about multiple regression is that the model is still interpretable. When it comes to neural networks, these are very difficult to interpret. You can use back propagation, but in the workplace I cannot see how you will explain to a client how that works.
As for the size of your data set, a lecturer told me that in their opinion you need at least 10000 observations before you can start thinking about fitting a neural network (I don't have any other source that confirms this).
Some other models that I would consider,

*

*LASSO and/or Ridge regression

*Random forest
