I'm dealing with a dataset that has about 300 input features, and about 200 response variables and consists of 25000 samples. These response variables are basically a profile of 200 different values of the same response and these are 200 sequential geometrical points along the 1D line. You can visualize this as a line graph of 200 points for each observation. These points are the different regions/zones across a plate at which the response variable is being measured during manufacturing. The input features are different process conditions set in different processes during the manufacturing of this plate such as thickness, temperature, pressure etc.
There are two objectives:

  1. To build a model/s that predicts the profile of the response variable. In other words, the model should predict the response at 200 different points across the plate.

  2. Determine the important features (predictors) that affect the response variable. Ideally, this response variable should be uniform across the entire profile. So by tweaking the features, I should be able to control the response variable.
    Now, predicting these 200 values is equivalent to building 200 models and I'm not sure how to draw inference from these 200 models. I tried doing a PCA on these 200 response variables to reduce them to a reasonable size but the variance is unstable and I'd require about 130 PC's to explain 90% of the variance. So PCA doesn't make sense. The points in a neighborhood seem to be closely correlated with each other. I'm not sure if you can call it spatial correlation because these points are along 1D line as opposed to 2D plane or 3D space. I'd greatly appreciate if someone can point me in the right direction in dealing with this problem.

The layout of the plate is about 7ft*5ft and some of the process parameters(input features) also are set and measured at different geometrical points across the plate like the target variable. After building the model and extracting significant predictors, I probably have to do some kind of optimization for finding the best combination of all input variables to achieve the uniformity of the target variable across all the measurements.

  • $\begingroup$ Assuming the 200 response variables are in the same metric, one solution would be to stack up a matrix of 500,000 records (25,000 x 200 rows) by 300 features and include a single factor with a categorical enumeration from 1 to 200 corresponding to the geometrical points. This would amount to an ANCOVA model or panel data model, depending on how specified. You don't mention any temporal component to the measures but if time is included, you can use it as a linear trend or as a set of categorical features with each time period having its own 0,1 dummy variable. $\endgroup$ – user234562 Apr 29 '20 at 16:58
  • $\begingroup$ Thanks! Could you point me to any resource/s? No, there is no temporal component. The response variable is measured at different geometrical points arranged in a sequential manner. As I said in the post, ideally the values across all these points should be uniform. That's the whole point of building the model so that you can control these values by adjusting input features. $\endgroup$ – Mohamad Sahil Apr 29 '20 at 17:59
  • $\begingroup$ If your interest is in finding process conditions that produce a uniform set of values across all 200 measurements, is there some reason why you can't just use some measure of non-uniformity (mean-square error, mean absolute error, median absolute deviation, etc) as a single outcome variable and avoid modeling all 200 outcomes? What do you specifically hope to gain from the much more detailed modeling? $\endgroup$ – EdM Apr 29 '20 at 20:07
  • $\begingroup$ It's a huge plate (~7ft * 5ft) and some of the process parameters(input features) also are set and measured at different geometrical points across the plate like the target variable. If you are averaging out all the values, you are losing a lot of information. My interest is both prediction as well as the inference. After building the model and extracting significant predictors, I probably have to do some kind of optimization (sensitivity analysis or something) for finding the best combination of all input variables to achieve the uniformity of the target variable across all the measurements $\endgroup$ – Mohamad Sahil Apr 29 '20 at 21:05
  • $\begingroup$ These are measurements on 25000 plates, with 200 corresponding points? How are the layout of the points? Some zigzag line covering the whole plate? If you have coordinates of each point, you can calculate distances and use spatial methods. Maybe. $\endgroup$ – kjetil b halvorsen Apr 29 '20 at 21:26

If I understood your data correctly, one method would be calculating feature importances for each model and maybe plotting them. Below is an example beeswarm plot and code in R for 198 different models that predict wind speed of storms using four features, with the features in decreasing order by average importances. Each point represents one feature importance of one model for the specific feature, and the vertical lines represent the mean feature importances for each predictor. If you have many predictors, you could limit them to the n most important as calculated by average feature importance.

enter image description here

library(dplyr) # data wrangling
library(purrr) # map function
library(tidyr) # nest function and pivot
library(caret) # varImp function
library(ggplot2) # plotting
library(ggbeeswarm) # plot type


# Nest data for each storm
nested <- storms %>% 
  group_by(name) %>%
  select(name, wind, month, pressure, lat, long) %>% 
  nest(data = c(wind, month, pressure, lat, long))

# Make models for each storm and store in tibble
models <- nested %>% 
  mutate(model = map(.x = data, ~lm(wind ~ month + pressure + lat + long,
                                    data = .x)))

# Combine the importances in a data frame
importances <- c()
for(i in 1:nrow(models)){
  x <- as.data.frame(t(varImp(models$model[[i]])))
  if(i == 1){
    importances <- x
  } else {
    importances <- suppressMessages(full_join(importances, x))

# Replace invalid values and min-max scale
importances_scaled <- importances %>% 
  mutate_all(function(x) ifelse(is.infinite(x) | is.na(x) | is.nan(x), 0, x)) %>% 
  mutate_all(function(x) {(x - min(x, na.rm = TRUE)) /
      (max(x, na.rm = TRUE) - min(x, na.rm = TRUE)) * 100

# Pivot for plotting and join means
to_plot <- importances_scaled %>% 
  pivot_longer(month:long, names_to = "Feature", values_to = "Importance") %>% 
  inner_join(importances_scaled %>% 
               summarise_all(mean) %>% 
               pivot_longer(month:long, names_to = "Feature", values_to = "Mean"))

# Plot from most important to least important
to_plot %>% 
  ggplot(aes(x = reorder(Feature, Mean),
             y = Importance,
             color = Feature)) +
  geom_quasirandom() +
  geom_point(data = to_plot %>%
               select(Feature, Mean) %>%
               distinct(), aes(Feature, Mean),
             shape = "|", size = 20, color = "black", alpha = 0.4) +
  coord_flip() +
  theme_minimal() +
  theme(legend.position = "none") +
  xlab("") +
  ylab("Feature importance") +
  ggtitle("Models for predicting wind speeds of tropical storms",
          subtitle = paste("Average feature importance from 0 to 100 for",
                           nrow(models), "different storms, means as vertical lines"))
  • 1
    $\begingroup$ Thanks. This idea did come to my mind. Thanks for sharing the code. I'm not sure if I'm going with this approach. If I do, this will be super helpful. $\endgroup$ – Mohamad Sahil Apr 29 '20 at 21:51

Not a full answer so long, but trying to answer the literal question in the title what to do with so many regressions. So, you have 200 similar regression models for some parallel data. Each response is "the same" variable measured at different points. So presumably, the estimated coefficients ought to be similar. So make some plots:

  • For each estimated model, for each coefficient make a plot of $\hat{\beta}$ versus position along the line. Can you see some structure? For important coefficients, the spread should be away from zero. For unimportant ones spread around zero. That could give an idea for some descriptive statistics to calculate on the 200 coefficients ...

  • If there are outliers in the plots, do they tend to occur at the same positions or not? Do the plots look approximately horizontal, or some structure? Structure the same/different for different plots? Effects near the edges? ...

Generally, with such rich/complex data it is maybe a good idea not to go straight into complex modeling, but really look at/visualize the data in many ways. There is a lot of internal replication, exploit that!. Andrew Gelman has written about that here.

  • 1
    $\begingroup$ Thanks! That totally makes sense. The issue is that there is multicollinearity as many of the input variables are correlated with each other. So I'm thinking of using Lasso as it will wash out some of the predictors by shrinking their coefficients to zero. As a result, some of these estimated betas will have zero values in some positions. Maybe I can think of using ridge instead. I'm also thinking of trying more complex models like Random Forests. $\endgroup$ – Mohamad Sahil May 1 '20 at 3:05
  • $\begingroup$ Thanks for suggesting the paper. I'll probably try that after I'm done training the models. I have a question though regarding using the right regularization method. Out of 300 input features, some of them are highly correlated with each other as mentioned in the previous comment. I was initially thinking of using Lasso but I read that Lasso arbitrarily chooses a variable among the highly correlated ones. If I choose Ridge, interpretation of 300 coefficients would be a challenge. I'm also trying Elastic net. $\endgroup$ – Mohamad Sahil May 2 '20 at 16:23
  • $\begingroup$ When performing cv on all the three methods, I found that the optimum lambda value for any alpha value is close to zero considering RMSE as my chosen metric. There is not much deviation in RMSE of the optimum model across all apha values. $\endgroup$ – Mohamad Sahil May 2 '20 at 16:27
  • $\begingroup$ Can I use PLS(Partial Least Squares) regression for this problem? $\endgroup$ – Mohamad Sahil May 4 '20 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.