# Improving linear regression by transforming predictors

I'm working on the insurance dataset with R, and I'm trying to do a lm with charges as the target.

I did the following:

• Removed the severe outliers (3*IQR)
• Removed the multivariate outliers with Moutliers using chemometrics
• Checked the boxcox lambda value and transformed the target with log
• Applied boxTidwell for children and age (I had to add 0.05 to children, so it doesn't contain a 0) and got 1/sqrt(variable) as transformation
• Removed severe outliers using Cook's distance

After these transformations, the model still being pretty poor. However, some friends assessed me telling that maybe another transformation for the target could improve the model. What else could I try?

Edit:

this is the code, with some of the proposals. Not quite working tho when checking the plot of the last lm

library(tidyverse)
library(chemometrics)
library(cowplot)
library(lmtest)
library(ggplot2)
library(corrplot)
library(patchwork)
library(dplyr)
library(fitdistrplus)
library(FactoMineR)

plot_theme = theme_classic() +
theme(plot.title = element_text(hjust = 0.5, size = 14,face = 'bold'),
axis.title.x = element_text(size = 14),
axis.title.y = element_text(size = 14),
axis.text.x  = element_text(size = 12),
axis.text.y  = element_text(size = 12))

db$$sex <- as.factor(db$$sex)
db$$region <- as.factor(db$$region)
db$$smoker <- as.factor(db$$smoker)
db$$children <- as.factor(db$$children)

is_severe_outlier <- function(x) {
tmp <- 3 * IQR(x, na.rm = TRUE)
a <- quantile(x, 0.25, na.rm = TRUE) - tmp
b <- quantile(x, 0.75, na.rm = TRUE) + tmp
!dplyr::between(x, a, b)
}

# We create a new dataset with new logical columns labelling values as outliers
df_outliers <- db %>%
mutate(across(where(is.numeric),
is_severe_outlier,
.names = "sout_{col}")) %>%
mutate(qty_outliers = rowSums(across(starts_with("sout_"))))

outliers_charge <- which( df_outliers$$sout_charges == TRUE) outliers_age <- which( df_outliers$$sout_age == TRUE)
outliers_bmi <- which( df_outliers$sout_bmi == TRUE) db_no_outliers <- db[-c(outliers_charge,outliers_age, outliers_bmi), ] mult_outliers <- db_no_outliers %>% select_if(is.numeric) %>% Moutlier(quantile = 0.999, plot = FALSE) mult_outliers <- db_no_outliers %>% add_column( moutlier_md = mult_outliers$$md, moutlier = mult_outliers$$md > mult_outliers$cutoff
)

mv_out <- which(mult_outliers$moutlier == TRUE) db_no_outliers <- db_no_outliers[c(-mv_out), ] library(forecast) BoxCox.lambda(db_no_outliers$charges) # close to 0 -> normal distro

db_log_target_no_outliers <- db_no_outliers %>%
mutate(charges = log10(charges))

library(car)
num_for_tidwell <- db_log_target_no_outliers %>%
select_if(is.numeric)

boxTidwell(charges ~ ., data = num_for_tidwell)

#0.5, -1

num_tidwell <- num_for_tidwell %>%
mutate(age = sqrt(age),
bmi = bmi^(-1))

db_factors <- db_no_outliers %>%
select_if(is.factor)

db_tidwell <- db_factors %>%
cbind(num_tidwell)

model_pre_cook <- lm(charges ~ ., data = db_tidwell)
summary(model_pre_cook)
plot(model_pre_cook)

dcook <- cooks.distance(model_pre_cook)
idx_sev_outlier_cookDistance <- is_severe_outlier(dcook)

db_tidwell_cooked <- db_tidwell[-idx_sev_outlier_cookDistance]

db_tidwell_cooked

model_cooked <- lm(charges ~ children + smoker + region + I(age^2) + I(age) + smoker*bmi, data = db_tidwell_cooked)
summary(model_cooked)
plot(model_cooked)

• Removing outliers makes regression worse. It attenuates effects and reduces power. If an observation in the dataset is wrong, it doesn't matter if it's an outlier or not, they should be removed. But outlying observations are not data errors necessarily. Commented Nov 17, 2022 at 18:31
• @AdamO but, if one observation is an outlier and has great leverage it can perjudicate the lm, right? Commented Nov 17, 2022 at 19:00
• No. It is rather the removal of outliers based on their incidental findings after collecting the data, that is fundamentally unscientific, that "prejudicates" the evidence as you say. By all means, removing outliers is an appropriate exploratory analysis. Commented Nov 17, 2022 at 19:15
• @AdamO still dont understand your point. Lets say that there is a perfectly aligned line with a single point that doesnt follow the line. Is not better to remove that single point due to the leverage? Commented Nov 17, 2022 at 21:01
• No, it's not. That single point is the most interesting (and arguably important) data point. If you don't know the reason for an observation being an outlier (e.g., an instrument malfunctioned), you should (I would even say must ) not remove it. A possible exception are values that are physically impossible (like a negative height). However, such values would bring into question the validity of the whole dataset. Commented Nov 18, 2022 at 7:23

• log10 transform of charges. Variables that span orders of magnitude often benefit from this.
• categorical treatment of children because there aren't many levels
• Possible interaction term between sex and bmi and between sex and age
• Possibly age^2 in addition to age