3
$\begingroup$

I am trying to do factor analysis on a few variables and one particular variable (given in the example below) is covering/ explaining all the variance due to some outliers. I am not sure what else I can do apart from removing this variable from factor analysis. But doing so will lose a lot of information. Hence, I am trying to fix this variable by fixing outliers and then normalizing it.

Initially, the histogram of this vector looks like this -

enter image description here

vec <- c(-10172.83, -8745.45, -8745.45, -7344.71, -4700.3, -3588, -1293.15, 
-1006.84, -995.62, -791.5, -634.49, -607.19, -591.17, -402.27, 
-397.21, -355.54, -301.01, -295.31, -272.22, -233.65, -168.68, 
-148.08, -132.47, -129.05, -115.94, -89.63, -89.63, -85.59, -83.79, 
-83.61, -83.61, -83.59, -81.05, -79.73, -76.99, -72.72, -71.02, 
-71.02, -70.2, -69.4, -67.56, -64.92, -62.8, -62.61, -60.09, 
-58.97, -58.97, -58.48, -55.39, -50.54, -50.54, -43.43, -43.39, 
-43.3, -42.85, -42.35, -40.46, -38.9, -37.9, -36.01, -35.41, 
-35.22, -34.92, -34.61, -34.23, -32.33, -30.96, -30.35, -30.16, 
-29.89, -29.8, -29.16, -28.81, -28.17, -27.56, -27.47, -26.49, 
-26.05, -24.4, -23.9, -22.87, -21.92, -20.07, -19.92, -19.62, 
-19.31, -18.8, -18.8, -18.38, -17.17, -17.17, -17.06, -16.74, 
-16.25, -16.2, -15.84, -15.75, -15.24, -14.78, -14.71, -14.48, 
-14.06, -13.88, -13.88, -13.4, -12.93, -12.89, -12.19, -11.98, 
-11.83, -11.75, -11.25, -11.23, -11.14, -10.87, -10.72, -10.68, 
-10.59, -10.59, -10.58, -10.56, -10.17, -10.15, -9.49, -9.45, 
-9.2, -9.13, -8.97, -8.73, -8.66, -8.49, -8.21, -8.1, -8.1, -8.02, 
-7.97, -7.95, -7.95, -7.93, -7.83, -7.61, -7.27, -7.23, -6.94, 
-6.85, -6.84, -6.7, -6.38, -6.32, -6.21, -6.2, -6.02, -6.01, 
-5.98, -5.91, -5.87, -5.87, -5.86, -5.86, -5.86, -5.85, -5.82, 
-5.74, -5.66, -5.61, -5.56, -5.56, -5.51, -5.42, -5.31, -5.26, 
-5.26, -5.18, -5.17, -4.98, -4.98, -4.74, -4.71, -4.71, -4.7, 
-4.7, -4.68, -4.57, -4.56, -4.51, -4.5, -4.5, -4.48, -4.48, -4.46, 
-4.36, -4.35, -4.2, -4.16, -4.08, -4.08, -4.07, -4.07, -4.05, 
-4.05, -4.01, -3.96, -3.85, -3.85, -3.79, -3.64, -3.51, -3.49, 
-3.48, -3.4, -3.39, -3.39, -3.38, -3.37, -3.34, -3.33, -3.33, 
-3.32, -3.26, -3.26, -3.25, -3.24, -3.24, -3.21, -3.21, -3.19, 
-3.19, -3.19, -3.18, -3.17, -3.16, -3.16, -3.14, -3.13, -3.09, 
-3.09, -3.03, -3.01, -2.96, -2.96, -2.91, -2.89, -2.89, -2.88, 
-2.84, -2.68, -2.62, -2.62, -2.6, -2.6, -2.6, -2.6, -2.57, -2.55, 
-2.54, -2.46, -2.45, -2.42, -2.42, -2.4, -2.38, -2.37, -2.35, 
-2.34, -2.28, -2.28, -2.28, -2.28, -2.26, -2.22, -2.22, -2.21, 
-2.2, -2.2, -2.2, -2.2, -2.19, -2.16, -2.15, -2.1, -2.08, -2.03, 
-1.98, -1.93, -1.92, -1.91, -1.91, -1.9, -1.89, -1.89, -1.89, 
-1.88, -1.88, -1.88, -1.87, -1.87, -1.86, -1.86, -1.85, -1.84, 
-1.83, -1.83, -1.82, -1.8, -1.79, -1.78, -1.76, -1.76, -1.74, 
-1.74, -1.73, -1.73, -1.72, -1.69, -1.63, -1.62, -1.62, -1.6, 
-1.6, -1.59, -1.59, -1.57, -1.56, -1.56, -1.55, -1.55, -1.54, 
-1.54, -1.53, -1.51, -1.48, -1.48, -1.47, -1.47, -1.46, -1.44, 
-1.44, -1.44, -1.43, -1.42, -1.4, -1.39, -1.39, -1.37, -1.34, 
-1.34, -1.34, -1.32, -1.31, -1.3, -1.3, -1.28, -1.26, -1.26, 
-1.24, -1.22, -1.22, -1.21, -1.19, -1.19, -1.18, -1.15, -1.13, 
-1.12, -1.12, -1.11, -1.11, -1.1, -1.1, -1.09, -1.09, -1.09, 
-1.09, -1.09, -1.08, -1.08, -1.08, -1.07, -1.07, -1.06, -1.05, 
-1.04, -1.03, -1.03, -1.03, -1.02, -1.02, -1.01, -1.01, -1, -1, 
-0.99, -0.99, -0.99, -0.99, -0.99, -0.99, -0.98, -0.98, -0.97, 
-0.97, -0.96, -0.96, -0.96, -0.95, -0.95, -0.94, -0.94, -0.94, 
-0.93, -0.93, -0.92, -0.92, -0.92, -0.91, -0.91, -0.91, -0.9, 
-0.9, -0.9, -0.9, -0.89, -0.88, -0.88, -0.87, -0.86, -0.85, -0.85, 
-0.85, -0.84, -0.84, -0.84, -0.83, -0.82, -0.81, -0.81, -0.81, 
-0.81, -0.81, -0.81, -0.8, -0.79, -0.79, -0.79, -0.79, -0.78, 
-0.78, -0.77, -0.76, -0.76, -0.76, -0.76, -0.75, -0.75, -0.74, 
-0.73, -0.72, -0.71, -0.71, -0.71, -0.71, -0.7, -0.7, -0.7, -0.7, 
-0.69, -0.69, -0.68, -0.68, -0.68, -0.67, -0.67, -0.67, -0.66, 
-0.66, -0.66, -0.66, -0.65, -0.65, -0.65, -0.65, -0.65, -0.64, 
-0.64, -0.64, -0.63, -0.63, -0.63, -0.63, -0.63, -0.63, -0.62, 
-0.62, -0.61, -0.61, -0.61, -0.61, -0.6, -0.6, -0.6, -0.6, -0.6, 
-0.6, -0.6, -0.58, -0.58, -0.58, -0.58, -0.58, -0.57, -0.57, 
-0.57, -0.57, -0.57, -0.57, -0.56, -0.56, -0.56, -0.56, -0.56, 
-0.55, -0.55, -0.55, -0.55, -0.55, -0.55, -0.55, -0.55, -0.55, 
-0.54, -0.54, -0.53, -0.53, -0.53, -0.53, -0.53, -0.53, -0.52, 
-0.52, -0.52, -0.52, -0.52, -0.51, -0.51, -0.51, -0.51, -0.51, 
-0.51, -0.51, -0.5, -0.5, -0.5, -0.5, -0.5, -0.49, -0.49, -0.49, 
-0.49, -0.49, -0.49, -0.48, -0.48, -0.48, -0.48, -0.48, -0.48, 
-0.48, -0.48, -0.48, -0.47, -0.47, -0.47, -0.47, -0.47, -0.47, 
-0.46, -0.46, -0.46, -0.46, -0.45, -0.45, -0.45, -0.45, -0.45, 
-0.45, -0.45, -0.44, -0.44, -0.44, -0.44, -0.44, -0.44, -0.44, 
-0.44, -0.44, -0.43, -0.43, -0.43, -0.43, -0.43, -0.43, -0.43, 
-0.43, -0.42, -0.42, -0.42, -0.42, -0.42, -0.42, -0.42, -0.42, 
-0.41, -0.41, -0.41, -0.41, -0.41, -0.41, -0.41, -0.41, -0.41, 
-0.4, -0.4, -0.4, -0.4, -0.4, -0.4, -0.4, -0.4, -0.39, -0.39, 
-0.39, -0.39, -0.39, -0.39, -0.39, -0.39, -0.38, -0.38, -0.38, 
-0.38, -0.38, -0.38, -0.38, -0.38, -0.37, -0.37, -0.37, -0.37, 
-0.37, -0.37, -0.37, -0.37, -0.36, -0.36, -0.36, -0.36, -0.36, 
-0.36, -0.36, -0.36, -0.35, -0.35, -0.35, -0.35, -0.35, -0.35, 
-0.35, -0.35, -0.34, -0.34, -0.34, -0.34, -0.34, -0.34, -0.34, 
-0.34, -0.33, -0.33, -0.33, -0.33, -0.33, -0.33, -0.33, -0.33, 
-0.33, -0.33, -0.32, -0.32, -0.32, -0.32, -0.32, -0.32, -0.32, 
-0.32, -0.32, -0.32, -0.31, -0.31, -0.31, -0.31, -0.31, -0.31, 
-0.31, -0.3, -0.3, -0.3, -0.3, -0.3, -0.3, -0.3, -0.3, -0.3, 
-0.3, -0.3, -0.3, -0.3, -0.3, -0.29, -0.29, -0.29, -0.29, -0.29, 
-0.29, -0.29, -0.29, -0.29, -0.29, -0.29, -0.29, -0.29, -0.29, 
-0.29, -0.29, -0.29, -0.28, -0.28, -0.28, -0.28, -0.28, -0.28, 
-0.28, -0.28, -0.28, -0.28, -0.28, -0.28, -0.27, -0.27, -0.27, 
-0.27, -0.27, -0.26, -0.26, -0.26, -0.26, -0.26, -0.26, -0.26, 
-0.26, -0.26, -0.26, -0.26, -0.26, -0.25, -0.25, -0.25, -0.25, 
-0.25, -0.25, -0.25, -0.25, -0.25, -0.25, -0.25, -0.25, -0.24, 
-0.24, -0.24, -0.24, -0.24, -0.24, -0.24, -0.24, -0.24, -0.24, 
-0.24, -0.24, -0.24, -0.24, -0.24, -0.24, -0.24, -0.24, -0.24, 
-0.23, -0.23, -0.23, -0.23, -0.23, -0.23, -0.23, -0.23, -0.23, 
-0.23, -0.22, -0.22, -0.22, -0.22, -0.22, -0.22, -0.22, -0.21, 
-0.21, -0.21, -0.21, -0.21, -0.21, -0.21, -0.21, -0.21, -0.21, 
-0.21, -0.2, -0.2, -0.2, -0.2, -0.2, -0.2, -0.2, -0.2, -0.2, 
-0.2, -0.2, -0.2, -0.2, -0.2, -0.2, -0.19, -0.19, -0.19, -0.19, 
-0.19, -0.19, -0.19, -0.19, -0.19, -0.19, -0.19, -0.19, -0.19, 
-0.19, -0.19, -0.19, -0.18, -0.18, -0.18, -0.18, -0.18, -0.18, 
-0.18, -0.18, -0.18, -0.18, -0.18, -0.18, -0.18, -0.18, -0.18, 
-0.18, -0.18, -0.18, -0.18, -0.18, -0.18, -0.17, -0.17, -0.17, 
-0.17, -0.17, -0.17, -0.17, -0.17, -0.17, -0.17, -0.17, -0.17, 
-0.17, -0.17, -0.17, -0.17, -0.16, -0.16, -0.16, -0.16, -0.16, 
-0.16, -0.16, -0.16, -0.16, -0.16, -0.16, -0.16, -0.15, -0.15, 
-0.15, -0.15, -0.15, -0.15, -0.15, -0.15, -0.15, -0.15, -0.15, 
-0.15, -0.15, -0.15, -0.15, -0.15, -0.15, -0.15, -0.15, -0.15, 
-0.15, -0.15, -0.15, -0.14, -0.14, -0.14, -0.14, -0.14, -0.14, 
-0.14, -0.14, -0.14, -0.14, -0.14, -0.14, -0.14, -0.14, -0.14, 
-0.14, -0.14, -0.14, -0.14, -0.14, -0.13, -0.13, -0.13, -0.13, 
-0.13, -0.13, -0.13, -0.13, -0.13, -0.13, -0.13, -0.13, -0.13, 
-0.13, -0.13, -0.12, -0.12, -0.12, -0.12, -0.12, -0.12, -0.12, 
-0.12, -0.12, -0.12, -0.12, -0.12, -0.12, -0.12, -0.12, -0.12, 
-0.12, -0.12, -0.12, -0.12, -0.11, -0.11, -0.11, -0.11, -0.11, 
-0.11, -0.11, -0.11, -0.11, -0.11, -0.11, -0.11, -0.11, -0.11, 
-0.11, -0.11, -0.11, -0.11, -0.11, -0.11, -0.11, -0.11, -0.11, 
-0.11, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, 
-0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, -0.1, 
-0.1, -0.1, -0.1, -0.09, -0.09, -0.09, -0.09, -0.09, -0.09, -0.09, 
-0.09, -0.09, -0.09, -0.09, -0.09, -0.09, -0.09, -0.09, -0.09, 
-0.09, -0.09, -0.09, -0.09, -0.09, -0.09, -0.09, -0.08, -0.08, 
-0.08, -0.08, -0.08, -0.08, -0.08, -0.08, -0.08, -0.08, -0.08, 
-0.08, -0.08, -0.08, -0.08, -0.08, -0.08, -0.08, -0.08, -0.08, 
-0.08, -0.08, -0.08, -0.08, -0.08, -0.08, -0.08, -0.08, -0.07, 
-0.07, -0.07, -0.07, -0.07, -0.07, -0.07, -0.07, -0.07, -0.07, 
-0.07, -0.07, -0.07, -0.07, -0.07, -0.07, -0.07, -0.07, -0.07, 
-0.07, -0.07, -0.07, -0.07, -0.07, -0.07, -0.06, -0.06, -0.06, 
-0.06, -0.06, -0.06, -0.06, -0.06, -0.06, -0.06, -0.06, -0.06, 
-0.06, -0.06, -0.06, -0.06, -0.06, -0.06, -0.06, -0.06, -0.06, 
-0.06, -0.06, -0.06, -0.06, -0.06, -0.06, -0.06, -0.06, -0.06, 
-0.06, -0.06, -0.06, -0.06, -0.06, -0.05, -0.05, -0.05, -0.05, 
-0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05, 
-0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05, 
-0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.05, 
-0.05, -0.05, -0.05, -0.05, -0.05, -0.05, -0.04, -0.04, -0.04, 
-0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, 
-0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, 
-0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, 
-0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, 
-0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.04, -0.03, 
-0.03, -0.03, -0.03, -0.03, -0.03, -0.03, -0.03, -0.03, -0.03, 
-0.03, -0.03, -0.03, -0.03, -0.03, -0.03, -0.03, -0.03, -0.03, 
-0.03, -0.03, -0.03, -0.03, -0.03, -0.03, -0.03, -0.03, -0.03, 
-0.03, -0.03, -0.03, -0.03, -0.03, -0.03, -0.03, -0.03, -0.03, 
-0.03, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, 
-0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, 
-0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, 
-0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, 
-0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.02, -0.01, 
-0.01, -0.01, -0.01, -0.01, -0.01, -0.01, -0.01, -0.01, -0.01, 
-0.01, -0.01, -0.01, -0.01, -0.01, -0.01, -0.01, -0.01, -0.01, 
-0.01, -0.01, -0.01, -0.01, -0.01, -0.01, -0.01, -0.01, -0.01, 
0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 
0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 
0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 
0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 
0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.02, 0.02, 0.02, 0.02, 0.02, 
0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 
0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 
0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 
0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 
0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 
0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 
0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 
0.02, 0.02, 0.02, 0.02, 0.02, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 
0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 
0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 
0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 
0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 
0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 
0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 
0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 
0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 
0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 
0.03, 0.03, 0.03, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 
0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 
0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 
0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 
0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 
0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 
0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 
0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 
0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 
0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 0.04, 
0.04, 0.04, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 
0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 
0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 
0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 
0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 
0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 
0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 
0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 
0.05, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 
0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 
0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 
0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 
0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 
0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 
0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 0.06, 
0.06, 0.06, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 
0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 
0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 
0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 
0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 
0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 
0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 
0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 
0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 
0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.07, 0.08, 
0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 
0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 
0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 
0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 
0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 
0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 
0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 
0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.09, 0.09, 0.09, 0.09, 
0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 
0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 
0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 
0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 
0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 
0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 
0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 
0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.1, 
0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 
0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 
0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 
0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 
0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 
0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 
0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 
0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 
0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 
0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 
0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 
0.11, 0.11, 0.11, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 
0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 
0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 
0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 
0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 
0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 
0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 
0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 
0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 
0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 
0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 
0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 
0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.15, 0.15, 0.15, 0.15, 0.15, 
0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 
0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 
0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 
0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 
0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 
0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 
0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 
0.16, 0.16, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 
0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 
0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.18, 0.18, 0.18, 0.18, 0.18, 
0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 
0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 
0.18, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 
0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 
0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 0.19, 
0.19, 0.19, 0.19, 0.19, 0.19, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 
0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 
0.2, 0.2, 0.2, 0.2, 0.2, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 
0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 
0.21, 0.22, 0.22, 0.22, 0.22, 0.22, 0.22, 0.22, 0.22, 0.22, 0.22, 
0.22, 0.22, 0.22, 0.22, 0.22, 0.22, 0.22, 0.22, 0.22, 0.22, 0.22, 
0.22, 0.22, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 
0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 
0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.24, 0.24, 0.24, 0.24, 0.24, 
0.24, 0.24, 0.24, 0.24, 0.24, 0.24, 0.24, 0.24, 0.24, 0.24, 0.24, 
0.24, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.26, 
0.26, 0.26, 0.26, 0.26, 0.26, 0.26, 0.26, 0.26, 0.26, 0.26, 0.26, 
0.26, 0.26, 0.27, 0.27, 0.27, 0.27, 0.27, 0.27, 0.27, 0.27, 0.28, 
0.28, 0.28, 0.28, 0.28, 0.28, 0.28, 0.29, 0.29, 0.29, 0.29, 0.29, 
0.29, 0.29, 0.29, 0.29, 0.29, 0.29, 0.29, 0.3, 0.3, 0.3, 0.3, 
0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.31, 0.31, 
0.31, 0.31, 0.31, 0.31, 0.31, 0.31, 0.31, 0.31, 0.31, 0.32, 0.32, 
0.32, 0.32, 0.32, 0.32, 0.32, 0.33, 0.33, 0.33, 0.33, 0.33, 0.33, 
0.33, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.35, 0.35, 
0.35, 0.35, 0.35, 0.35, 0.35, 0.35, 0.36, 0.36, 0.36, 0.36, 0.36, 
0.36, 0.36, 0.36, 0.37, 0.37, 0.37, 0.37, 0.37, 0.37, 0.38, 0.38, 
0.39, 0.39, 0.39, 0.39, 0.39, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.41, 
0.41, 0.41, 0.41, 0.41, 0.41, 0.41, 0.41, 0.42, 0.42, 0.42, 0.42, 
0.42, 0.42, 0.42, 0.42, 0.43, 0.43, 0.43, 0.43, 0.44, 0.44, 0.44, 
0.44, 0.45, 0.45, 0.45, 0.45, 0.46, 0.46, 0.46, 0.47, 0.47, 0.47, 
0.48, 0.48, 0.48, 0.48, 0.48, 0.49, 0.49, 0.49, 0.5, 0.51, 0.51, 
0.51, 0.52, 0.52, 0.53, 0.53, 0.54, 0.55, 0.57, 0.57, 0.57, 0.58, 
0.58, 0.58, 0.6, 0.6, 0.61, 0.62, 0.64, 0.65, 0.65, 0.67, 0.67, 
0.67, 0.74, 0.76, 0.77, 0.83, 0.83, 0.83, 0.84, 0.89, 0.89, 0.89, 
0.94, 0.95, 0.96, 0.96, 1, 1.07, 1.07, 1.12, 1.12, 1.26, 1.26, 
1.26, 1.27, 1.43, 1.55, 1.59, 1.59, 2.01, 2.05, 2.4, 2.45, 2.52, 
3.12, 4.38, 10.97, 10.97, 11.84, 14.57, 19.12, 52.07, 145.74, 
145.74)

To remove outliers, I used the update_outliers function (given below) which removes the negative and positive outliers using quantiles. However, the problem with this method is that it overdoes outlier fixing as can be seen in the image below (see the red circles).

update_outliers <- function(x, na.rm = TRUE, ...) {
  qnt <- quantile(x, probs=c(.05, .95), na.rm = na.rm, ...)
  H <- 1.5 * IQR(x, na.rm = na.rm)
  y <- x
  y[x < (qnt[1] )] <- qnt[1]
  y[x > (qnt[2] )] <- qnt[2]
  y
}

new_vec = update_outliers(vec)
hist(new_vec)

Question - How can I fix the outliers so that outlier fixing is not overdone as above and the data gets normalized? I want a bell-shaped histogram (as shown by the blue line in the image below) without any spikes on both ends.

Most importantly, during normalization, the negative values should not turn positive and vice versa.

My approach might be totally wrong as I am trying to fit data to a model instead of the fitting model to the data. I am open to any ideas to solve this problem in a better way.

enter image description here

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ The explicit purpose of the function is to replace values that are (according to the function) "too large" or "too small" with a fixed value. Since this is what the function does, It's hard to understand what the question could possibly mean -- don't use that function if it's not achieving your goal. $\endgroup$
    – Sycorax ♦
    Commented Aug 3, 2022 at 20:11
  • $\begingroup$ There are all kinds of ways to deal with this, depending on what vec represents and how you conceive of its role in your analysis. (it's a question of "feature engineering.") But do you really have to supply 4006 values to explain your problem?? Let's have a minimal reproducible example, please. $\endgroup$
    – whuber ♦
    Commented Aug 3, 2022 at 21:35
  • 1
    $\begingroup$ Instead of "removing the variable" you can remove the outlying observation, which loses less information, see my first comment. However robust approaches are usually better (unless there's evidence that the outlier is indeed wrong). $\endgroup$
    – Christian Hennig
    Commented Aug 3, 2022 at 21:46
  • 1
    $\begingroup$ If you want to force the vector of observations to have a distribution close to normal, you can use inverse normal scores transformation. There are a few different varieties such as Elfving, Blom, van der Waerden, Tukey, and rankit methods. This will not preserve the sign of the original values as you request, and may not be meaningful for your purpose. With the caveat that I wrote the function, in R you can use, library(rcompanion); vec2 = blom(vec); hist(vec2) $\endgroup$
    – Sal Mangiafico
    Commented Aug 4, 2022 at 13:20
  • 3
    $\begingroup$ I had no trouble using the logarithm. The idea is to create an auxiliary binary variable determined by the signs of your net margins along with the log of the size of the net margin. This yields a pair of variables that ought to work nicely in a factor analysis. But whether this is suitable for your intended analysis depends on many aspects of that analysis; and there are plenty of other approaches to the problem. $\endgroup$
    – whuber ♦
    Commented Aug 4, 2022 at 13:47

2 Answers 2

Reset to default
6
$\begingroup$

Techniques of Exploratory Data Analysis (EDA) can help with this feature engineering problem. I want to emphasize how just a couple of well-chosen plots tell us, forcibly, how we ought to proceed.

The histogram (unavoidably) hides important details, so let's begin with an N-letter summary of the batch of $2930$ data found in the array vec:

         M      H      E     D       C       B       A      Z       Y     X      *
Upper 0.02  0.100  0.190  0.30   0.430   0.650    1.10    2.4    11.0    24    150
Mid   0.02 -0.095 -0.455 -2.15  -8.285 -29.175  -64.45 -298.8  -944.5 -3738  -4925
Lower 0.02 -0.290 -1.100 -4.60 -17.000 -59.000 -130.00 -600.0 -1900.0 -7500 -10000

These letter statistics M, H, E, etc, designate upper and lower $1/2,$ $1/4,$ $1/8,$ etc. quantiles. (These letters started out as mnemonics: "M" = median, "H" = "hinge" (Tukey's term for the shoulders of a boxplot), "E" = "eighth," and thence ascending through the alphabet.) The final column "*" gives the extremes. The middle row labeled "Mid" computes the values midway between corresponding letter statistics; for example, the E-mid is $-0.455 = (0.190 + -1.100)/2.$

A glance at the large, systematic decreases in the Mid row shows us how negatively skewed the data are. (After all, in any symmetric batch of data the mid-values will all be approximately equal.)

The temptation is to transform the data, but how? If all values were positive, a diagnostic plot would provide an indication. This is a plot of the spreads (the distances between corresponding letter statistics) against the mid-values on logarithmic axes. Often the points on this plot trend almost linearly and, if so, the slope of the trend indicates how to transform the data to make their distribution more nearly symmetric.

Although taking logarithms is impossible due to the negative mid values, the consistent signs of those mid-values suggest looking at their sizes (absolute values) alone. (The initial value of $+0.02$ is essentially zero and oughtn't to be a problem. Fortunately, none of the data values are exactly zero.)

Here is the resulting plot.

Plot

It is extraordinary: usually the points do not line up so nicely, but here the linearity is almost perfect. When data are positive and the slope is $p,$ a Box-Cox transformation of power $1-p$ is indicated. Because $1-p = 1-0.95 = 0.05$ is essentially $0,$ and the Box-Cox transformation of power $0$ is the logarithm,

the data tell us we should be expressing them in terms of the logarithms of their absolute values.

Because taking the absolute value loses one bit of information -- the signs of the data -- we ought to retain the signs as a separate variable in the analysis. Although there's no guarantee this will be effective, it's worth a try.

Histograms of positive and negative parts

(Red is positive and blue is negative.)

The histograms show the positive and negative parts of the data have similar distributional shapes but different scales and locations. For some analyses it might help to standardize the logs of the positive and negative parts separately to make them more comparable; but for a factor analysis or PCA the differing spreads and locations of the two parts of the data might help reveal the patterns or behaviors the analysis is looking to discover.

Reference: John Tukey (1977), EDA. Addison-Wesley.

Appendix: R code for N-letter summaries

# Compute the letter statistics for `vec`.
q <- (1/2)^seq_len(floor(log2(length(vec)))-1)
stats <- c(min(vec), quantile(vec, c(rev(q), 1 - q[-1])), max(vec))

# Display them along with the mid-letter statistics.
k <- length(q) + 1
N.letter <- signif(rbind(stats[-seq_len(k-1)], rev(stats[seq_len(k)])), 2)
tletters <- c("M", "H", rev(LETTERS[1:5]), rev(LETTERS[-(1:8)]))[seq_along(q)]
colnames(N.letter) <- c(tletters, "*")
N.letter <- rbind(Upper = N.letter[1,], Mid = colMeans(N.letter), Lower = N.letter[2,])
(N.letter)
Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks, whuber, is there a package in R to see N-letter summary? $\endgroup$
    – Saurabh
    Commented Aug 5, 2022 at 13:21
  • 1
    $\begingroup$ Probably. But I found it easier and faster just to compute it directly rather than hunt down a package. I'll add the code to this answer. As you can see, it's just a couple of lines of (simple) code. I have included additional code to format and print the results nicely. $\endgroup$
    – whuber ♦
    Commented Aug 5, 2022 at 13:24
  • $\begingroup$ There is a small typo: The H-mid is: $-0.095=(0.100+-0.290)/2$. $\endgroup$
    – COOLSerdash
    Commented Aug 5, 2022 at 13:44
  • 1
    $\begingroup$ @COOL Thanks! My eye slipped and read the E column, so I'll just change "H" to "E" :-). $\endgroup$
    – whuber ♦
    Commented Aug 5, 2022 at 13:45
1
$\begingroup$

If you want to force the vector of observations to have a distribution close to normal, you can use inverse normal scores transformation. There are a few different varieties such as Elfving, Blom, van der Waerden, Tukey, and rankit methods.

I don't know that this transformation will be meaningful for your purpose.

And this will not preserve the sign of the original values as you request.

With the caveat that I wrote the function, in R you can use,

library(rcompanion)

vec2 = blom(vec)

hist(vec2)

If it's necessary to preserve the sign of the original data, it may be possible to combine this approach with the creation of a second dichotomous variable which captures the sign of the original data, as suggested in a comment by @whuber, in which a log transformation was used.

So, perhaps, with this approach:

vecAbs = abs(vec)

vecSign = sign(vec)

vecNormAbs = blom(vecAbs)

hist(vecNormAbs)

qqnorm(vecNormAbs)
qqline(vecNormAbs, col="red")

table(vecSign)
Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.