I have a trip duration dataset that looks like this: I want to use other parameters to predict the waiting time (wait_sec). The waiting time refers to the time the vehicle is stuck in traffic or so. The data is clean (without outliers/missing values) already.

pickup_longitude   pickup_latitude  dropoff_longitude   dropoff_latitude    trip_duration    dist_meters    wait_sec    hour_of_day month   day_of_week day_of_year week_of_year    trip_duration_log   dist_meters_log avg_speed
-78.5092           -0.2215           -78.4920           -0.2069              955             3587           403         6           10        3            301         43           6.8628               8.1853          3.7560

I tried using LinearRegression

X = df[['pickup_longitude', 'pickup_latitude','dropoff_longitude','dropoff_latitude','trip_duration','dist_meters','hour_of_day','day_of_week','trip_duration_log','dist_meters_log', 'avg_speed']]
y = df['wait_sec']

x_train, x_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 42)
LR = LinearRegression()
# fitting the training data
y_prediction =  LR.predict(x_test)

print('mean_abs is==',mean_absolute_error(y_test,y_prediction))

but the mean abs error I get is quite high (97). What other approach can I use? Or how can I optimise Linear Regression itself here? Original dataset can be found here: https://www.kaggle.com/mnavas/taxi-routes-for-mexico-city-and-quito?select=uio_clean.csv


Your outcome is a duration. It is unlikely (albeit possible) that a linear (OLS) regression model will be appropriate. By default, when I the outcome is a duration, I start with survival analyses (see threads categorized under our tag). In general, I would look at a proportional hazards model to start.

The other issue is you want to know how to get a more precise prediction than +/- 97. This is a common type of concern. Be aware that there may be no more information available in your data. It's possible that there are more informative variables, but you don't have them and no one has ever thought of them. It may be that there is a certain amount of irreducible randomness in the data, and that's it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.