Improving linear regression by transforming predictors

Question

I'm working on the insurance dataset with R, and I'm trying to do a lm with charges as the target.

I did the following:

• Removed the severe outliers (3*IQR)
• Removed the multivariate outliers with Moutliers using chemometrics
• Checked the boxcox lambda value and transformed the target with log
• Applied boxTidwell for children and age (I had to add 0.05 to children, so it doesn't contain a 0) and got 1/sqrt(variable) as transformation
• Removed severe outliers using Cook's distance

After these transformations, the model still being pretty poor. However, some friends assessed me telling that maybe another transformation for the target could improve the model. What else could I try?

Edit:

this is the code, with some of the proposals. Not quite working tho when checking the plot of the last lm

library(tidyverse)
library(chemometrics)
library(cowplot)
library(lmtest)
library(ggplot2)
library(corrplot)
library(patchwork)
library(dplyr)
library(fitdistrplus)
library(FactoMineR)

db<-read_csv("https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/insurance.csv")

plot_theme = theme_classic() + 
  theme(plot.title = element_text(hjust = 0.5, size = 14,face = 'bold'),
        axis.title.x = element_text(size = 14),
        axis.title.y = element_text(size = 14),
        axis.text.x  = element_text(size = 12),
        axis.text.y  = element_text(size = 12))


db$sex <- as.factor(db$sex) 
db$region <- as.factor(db$region)
db$smoker <- as.factor(db$smoker)
db$children <- as.factor(db$children)

is_severe_outlier <- function(x) {
    tmp <- 3 * IQR(x, na.rm = TRUE)
    a <- quantile(x, 0.25, na.rm = TRUE) - tmp
    b <- quantile(x, 0.75, na.rm = TRUE) + tmp
    !dplyr::between(x, a, b)
}

# We create a new dataset with new logical columns labelling values as outliers
df_outliers <- db %>%
    mutate(across(where(is.numeric),
        is_severe_outlier,
        .names = "sout_{col}")) %>%
    mutate(qty_outliers = rowSums(across(starts_with("sout_"))))


outliers_charge <- which( df_outliers$sout_charges == TRUE)
outliers_age <- which( df_outliers$sout_age == TRUE)
outliers_bmi <- which( df_outliers$sout_bmi == TRUE)

db_no_outliers <- db[-c(outliers_charge,outliers_age, outliers_bmi), ]

mult_outliers <- db_no_outliers %>%
    select_if(is.numeric) %>%
    Moutlier(quantile = 0.999, plot = FALSE)

mult_outliers <- db_no_outliers %>%
    add_column(
        moutlier_md = mult_outliers$md,
        moutlier = mult_outliers$md > mult_outliers$cutoff
    )

mv_out <- which(mult_outliers$moutlier == TRUE)

db_no_outliers <- db_no_outliers[c(-mv_out), ]


library(forecast)

BoxCox.lambda(db_no_outliers$charges) # close to 0 -> normal distro

db_log_target_no_outliers <- db_no_outliers %>%
  mutate(charges = log10(charges))

library(car)
num_for_tidwell <- db_log_target_no_outliers %>%
  select_if(is.numeric)

boxTidwell(charges ~ ., data = num_for_tidwell)

#0.5, -1

num_tidwell <- num_for_tidwell %>%
  mutate(age = sqrt(age),
         bmi = bmi^(-1))

db_factors <- db_no_outliers %>%
  select_if(is.factor)

db_tidwell <- db_factors %>%
  cbind(num_tidwell)

model_pre_cook <- lm(charges ~ ., data = db_tidwell)
summary(model_pre_cook)
plot(model_pre_cook)

dcook <- cooks.distance(model_pre_cook)
idx_sev_outlier_cookDistance <- is_severe_outlier(dcook)

db_tidwell_cooked <- db_tidwell[-idx_sev_outlier_cookDistance]

db_tidwell_cooked

model_cooked <- lm(charges ~ children + smoker + region + I(age^2) + I(age) + smoker*bmi, data = db_tidwell_cooked)
summary(model_cooked)
plot(model_cooked)

Answer 1

I would try

• log10 transform of charges. Variables that span orders of magnitude often benefit from this.
• categorical treatment of children because there aren't many levels
• Possible interaction term between sex and bmi and between sex and age
• Possibly age^2 in addition to age