# Accounting for sensitivity of model to small changes in assumptions

I am running a Monte Carlo simulation of correlated asset returns, based on this Matlab function.

The model's inputs are a vector of expected means and standard deviations for 9 assets, and their covariance matrix. It takes these inputs and simulates correlated returns for each asset over a fixed horizon for many trials, and then calculates the average likelihood that a portfolio of these assets will exceed a certain return threshold at the end of the simulation period.

The model works fine, but I am worried about estimation error as the probability output is very sensitive to small changes in the assumptions (ie the mean and standard deviation for the individual assets).

Ideally I would want to resample the data based on the observations, but I don't have the underlying raw data ; all I have are the initial assumptions for the assets. For simplicity, I can assume the assets all follow a normal distribution with an unknown true mean/st dev.

To account for the sensitivity of the inputs, I had the following approach in mind. Instead of giving a fixed probability as the output, I think it might be a better idea to give an interval to account for the estimation error (i.e instead of saying the probability is 60%, I'll instead say I am 95% sure the probability of exceeding the threshold is between 50% and 75%).

I will construct this interval using this approach:

• Using the original input mean and covariance matrix, I generate N sets of 500 random drawings for each of the assets
• For each of these sets, I use the simulated drawings to estimate the sample mean, st.dev and covariance, and then run my model using these new inputs to calculate the probability. This process is repeated N times.
• Now that I have N test statistics based on the simulated data, I can rank order them and create a distribution. The 5th percentile is the lower bound for my interval, and the 95th is the upper bound.

My question is: Is this a good solution, considering these bounds will still be sensitive to my starting assumptions? Given the limitations of my dataset, is there a better way to account for the estimation error?

• The issue is not just the variances of assets, but the correlation matrix estimation. Look up "portfolio optimization" in interweb. – Aksakal May 14 '18 at 19:44
• I may be missing something -- but isn't your interval just dependent on the number of random drawings? I.e. what if instead of 500 you had a million draws? Asymptotically your sample means etc. are just the original means, covariance, etc. – Weihuang Wong May 14 '18 at 19:47
• @WeihuangWong, his problem is that covariance matrix is unknown, and must be estimated. The outcome is very sensitive to estimation error. Increasing the number of draws from the estimated covariance matrix doesn't do anything to address the estimation error sensitivity – Aksakal May 14 '18 at 19:50
• From the question it seems like OP is provided with a covariance matrix, but is worried about estimation error in the provided information. But maybe we are talking about the same thing. – Weihuang Wong May 14 '18 at 20:00
• Right, I'm provided only with an estimated covariance matrix and estimated means/st devs for the assets. These are my starting inputs. The problem is that the model outcome is really sensitive to a small change in these inputs, and the inputs that I am using might not exactly be the "true" parameters for the assets due to estimation error. I don't have access to the observations that these estimates are based on, but I believe the underlying distribution for the assets is normal. – beeba May 14 '18 at 20:05

## 1 Answer

To the question of "is this a good solution," your bounds are just going to depend on the number of draws you do (i.e. 500 in your example) and possibly the number of Monte Carlo simulations. If instead of 500 draws you do 10 draws, the interval will be wider. In this sense, the bounds you get are somewhat arbitrary. More generally, since you have no information on the sampling distribution of the estimates you are given, there's no way to create this uncertainty out of thin air.

You might consider doing a sensitivity analysis, e.g. estimate the probability of exceeding the threshold given different shocks to the inputs (e.g. what if covariance is 50% of the estimates? 200%? What if asset returns are uncorrelated? etc.).

• Thank you for clarifying, that makes sense. I have run a sensitivity analysis to peturb the inputs one at a time and measure the output response. However, I also want to be able to model the risk of having multiple similar inputs wrong at the same time. Would it make sense to simulate a random "estimation error" vector and add it to the simulated drawings in each trial? This vector would be multivariate normal with mean 0 and correlation equivalent to my sample covariance matrix. The confidence interval could then be constructed based on the same process as before, but with noisy drawings. – beeba May 15 '18 at 18:26
• Isn't that just another way of drawing from a distribution with the same means you're given, but with double the variance? Let X be your original distribution, and Y be this "estimation error" vector with mean 0 and var(Y) = var(X). Then Var(X + Y) = Var(X) + Var(Y) = 2Var(X) since X and Y are uncorrelated. – Weihuang Wong May 15 '18 at 19:00
• Yes - it's equivalent to scaling the volatility up by sqrt(2). It widens the confidence interval, and the "uncertanity" scales with the correlation/variance of the inputs so it is not arbitrary compared to the previous approach, but I suspect there is probably a better way to model the parameter uncertainty. Do you have any suggestions for modifying the simulation so that it captures uncertainty in the initial point estimates? Thanks again for your help. – beeba May 15 '18 at 19:35
• Yeah, I think that's what I mean in my answer about doing a sensitivity analysis by scaling up and down the covariance. I don't know if there's a way of capturing the uncertainty in the initial estimates -- for all you know, they may be based on a large number of observations and hence very precise. They could also be based on a small number of observations; there's really no way to know without more information. – Weihuang Wong May 15 '18 at 20:52