more-arw search

Q&A Forum

Is Monte Carlo simulation appropriate for forecasting models with a lot of variables?

Monte Carlo simulationMy company has a complex mix of products and verticals. Some of our products have seasonality and a number of assumptions that go into our revenue model. Is Monte Carlo simulation appropriate for forecasting models with a lot of variables or would you stick to a smaller number of controlled assumptions?

Answers

Topic Expert
Ric Ratkowski
Title: Domain Expert Finance & Analytics
Company: SAP
(Domain Expert Finance & Analytics, SAP) |

For the type of forecasting you are describing I've typically used ARIMA, Stepwise Regression and Time Series techniques (especially when products have seasonality). There are tools out there developed by the two big statistical software companies that do a good job of what you describe: allowing you to throw a bunch of variables (over time) in and it will produce out a model and identify the dependent, independent and non-significant variables that drive sales. I've used those solutions with the above forecasting methods, but not Monte Carlo simulation.

I have used Monte Carlo simulation as it relates to business planning only as a diagnostic tool to help define a smaller number of control assumptions by using it to re-forecast history and identify the dependent and independent variables. I've also used Monte Carlo simulation more in a workbench environment to perform business performance optimization using existing historic data sets to help me understand what areas (drivers) provide the best leverage to accomplish a financial goal when my eyes couldn't see a correlation.

Just some cautions, a lot of folks are looking for a magic bullet to create forecasts and I've never seen that work effectively as that. I see it as a valuable tool that can create some "ahhhhs" when you study and break down the results, but it isn't a magic bullet.

Some of the keys to getting better results out of a statistical model are:
1) how you (and the software) handle outliers and nulls in the data (which skews the results) Not all tools are created equal.
2) having lots of comparable history (which is hard to do). To forecast the next three months typically requires 36 months of history.

To put it in perspective, companies like Coke-Cola have entire departments that will work weeks on creating a sales estimate using statistical methods for a new product introduction based on sales in a target market.

Hope this helps.

Cheers,

Ric

Topic Expert
David Wittenberg
Title: Director of Financial Strategy
Company: World Vision
(Director of Financial Strategy , World Vision) |

I heartily endorse Ric's guidance and have a few more cautions of this very powerful but usually misused tool.

In addition to the technical challenges Ric provides, there are several behavioral pitfalls that produce dangerous results. Here are three of the more common biases:

#1 - We are abysmal at estimating probabilities: This has nothing to do with intelligence -- it is the way our brains are wired. As Ric Ratkowski shared in his answer, Coke will have teams working weeks on the statistical inputs. An hour of sales folks tossing around numbers until a consensus is reached is woefully inadequate. There are some fun exercises in estimating probabilities that expose how bad we are at this.

#2 - We are systematically overly-optimistic: We systematically over-estimate the likelihood of good things happening (e.g. market share) and under-estimate the likelihood of bad things happening (e.g. cost overruns). This is very well documented in research.

#3 - We often goal-seek: When we get weird results (i.e. anything we don't agree with), we typically go back and tweak the inputs until they "look reasonable." It becomes an exercise in proving our initial position.

I warn of "dangerous results" because, even with very misguided inputs, Monte Carlo will still produce an array of very impressive graphs. You may notice that all three errors under-represent the risks of the situation. We would have been better off not using this tool at all than using it poorly because we embark on decisions underestimating their risks.

My consistent advice to companies is to avoid using the tool until they have invested heavily to train their experts in overcoming the biases. That costs a great deal more than the Excel Add-In, but the alternative is deluding ourselves into under-estimating risk. It can be hard to convince some folks of this -- the less we know about something the easier it looks.

All the Best,
Dave

Topic Expert
John Kogan
Title: CEO/CFO
Company: Proformative, Inc.
(CEO/CFO, Proformative, Inc.) |

...courtesy of Christos Tzelepis, former head of ING Risk Management in Greece, via another channel:

"The seasonality and trends are noise in a time series. You must "de-seasonalize" and "de-trend" your curve in order to have small residuals. Then you have to establish a Forecast method (ARIMA (seasonal) or Winters method or Decomposition etc.) to the clean time series. You must always look if the back-forecast model fits the curve well (Minimum Errors in MAPE, MSE, etc). Finally the forecasts you get are a mathematical (statistical) view. This view must be adjusted by the persons running the business, as they know what they expect in the future.

Assumptions must be validated with common sense. The most accurate forecast is the combination of mathematics and judgment. Monte Carlo simulation is highly complex, and is taking into account different distributions in order to have a final one as an output. I think you should avoid this. Its more transparent to have historical data, a good fit model forecasts, and judgment on this. SPSS can produce multiple time series forecasts.

I have used @Risk software to make financial models with many variables. I was changing the volume of the variables (+/-) by x1%,x2%.....xn% to see how the outcome behaves. I produce Tornado Diagrams to find which variable has the most important influence. I produce sensitivity diagrams to see how much. Monte Carlo simulations deal with distributions. Every variable is one different distribution. To mix all of them is a complicated procedure. I think Top Management may get lost with that. It's better to see the variables separately. For Sales,Revenues, Cost of Sales and all the figures of a General Ledger and Cash flows, its better to use Time Series (Monthly Historical Data) and forecast the next months. Then you have in your hands something that is "on the earth" and not "on the moon".

Christos Tzelepis
Title: Risk Management-Quality System Auditor-6..
Company: ----
(Risk Management-Quality System Auditor-6 Sigma Black Belt, ----) |

Thanks John for the post!

13304 views
Topics
Products and Companies

Get Free Membership

By signing up, you will receive emails from Proformative regarding Proformative programs, events, community news and activity. You can withdraw your consent at any time. Contact Us.

Business Exchange

Browse the Business Exchange to find information, resources and peer reviews to help you select the right solution for your business.

Learn more

Contribute to Community

If you’re interested in learning more about contributing to your Proformative community, we have many ways for you to get involved. Please email content@proformative.com to learn more about becoming a speaker or contributing to the blogs/Q&A Forum.