Chapter 9: Simulation

The Art of Modeling with Spreadsheets

S.G. Powell and K.R. Baker
John Wiley and Sons, Inc.

PowerPoint Slides Prepared By: Tava Olsen Washington University in St. Louis
7-1
9-1

Simulation
Tool for modeling uncertainty Describes probabilities of outcomes of a given decision, or set of decisions Outcomes can be averaged with probabilities to find expected, or average, outcome Excel add-in: Crystal Ball

On CD that accompanies text

9-2

Simulation Modeling Process

Develop a model that behaves like the real problem, with a special consideration of the random or probabilistic input variables Conduct a series of computer runs (called trials) to learn the behavior of the simulation model Compute the summary (output) statistics and make inferences about the real problem
9-3

Monte Carlo Simulation

A model that mimics the behavior of a (random) system
Probabilistic Input (random number)

r1, r2, , rn
Controllable Input Model

Analysis

Outcome

9-4

Entering CB Assumptions
A random input cell in Crystal Ball is called an assumption cell Select Cell Define Assumption

If an error occurs enter a dummy value in the cell first CB needs to recognize the cell as a number

Enter name for random variable Pick distribution Enter parameters

9-5

CB Forecasts
The output from a CB simulation is called a forecast Steps for defining:

Select a cell or a range of formula cells Select Cell Define Forecast Enter the name and units (optional) in text Click OK

When you run the simulation the forecast values will be calculated and output for each run
9-6

Running Crystal Ball

Set Run Preferences:

Select Run Run Preferences Can change number of trials One iteration: select Run Single Step Multiple iterations: select Run Run (may need to select Run Reset first)
9-7

To run:

Forecast Window
Number of trials Trials displayed

View Frequency
9-8

Preferences Display Range

Set to +Infinity to view all trials

Makes histogram easier to interpret

9-9

Conditions When Simulation is Not Necessary

If the relationship between uncertain input parameters and the output is linear

Then we can substitute mean value of uncertain input to calculate output E[aX+b] = aE[X] + b E[X2] (E[X]) 2
9 - 10

ONLY true for linear relationships

Steps in Simulation
Select uncertain parameters

Each requires own probability distribution Tornado charts can help select random input parameters Data analysis and judgment
What do we want to analyze? How many trials?
9 - 11

Select probability distributions

Select output(s)

Run

Analyze output

CB Sensitivity
Found under Sensitivity Toolkit Runs entire set of simulations while varying one or two parameters Similar to Data Sensitivity except each parameter choice requires a complete set of simulation trials

9 - 12

CB Sensitivity Input Window 1

***insert Figure 9.16a

9 - 13

CB Sensitivity Input Window 2

***insert Figure 9.16b

9 - 14

Generating Random Distributions

Can be done through Crystal Ball assumptions

Makes functions implicit and therefore model harder to read, debug, and understand. Useful if correlations are needed (see later) Will see list of all CB functions and associated parameters Faster than using assumption cells

Under paste function, select Crystal Ball

9 - 15

Useful Crystal Ball Distributions

CB.Binomial(p,n)

setting n=1 makes a 0-1 r.v.

CB.Normal(mean, std.dev.) CB.Uniform(min, max)

Use trunc(CB.Uniform(min, max+1)) to get integers uniformly distributed between min and max

CB.Triangular(min, likeliest, max)

9 - 16

Assumption Cells
Can enter parameters as cell addresses

e.g., =E6 or =C7 Cell colors will be copied but distribution will not

Do not use Edit Copy and Paste

Use Cell Copy Data and Cell Paste Data to copy and paste Only sampled when CB is run

9 - 17

Correlating Assumptions
Choose Cell Define Assumption Click on correlate

Must have another assumption defined

Select appropriate assumption from Select Assumption menu Enter correlation coefficient

Calc option calculates it from data

Press Enter
9 - 18

Selecting Uncertain Parameters

First carry out deterministic analysis with model before considering simulation

Establish base case set of inputs Calculate base case outputs Test model and learn about range of outcomes Decide whether simulation indicated Tornado chart indicates parameters with significant impact on the outcome

Perform sensitivity analysis

9 - 19

10/90 Percentile Tornado Chart

Enter data in two columns next to original parameters 1st column should be 10th percentile

There is a 90% probability parameter lies above this point There is a 90% probability parameter lies below this point
9 - 20

2nd column should be 90th percentile

Selecting Appropriate Probability Distributions

Empirical data

Numerical observations from experience Estimates made by experts May consist of mean, minimum, 10th or 90th percentiles Usually more readily available than empirical data
9 - 21

Judgmental data

Empirical Data
Usually not available, at least not immediately Information may be biased or otherwise inappropriate Requires judgment to determine whether best-fit distribution is appropriate Results of interest often depend on mean and variance but less on specific probability distribution Seldom sufficient due to above reasons
9 - 22

Most Highly Used Distributions

Discrete

Uncertain parameter takes one of a few outcomes Entered using CB Custom distribution Outcome equally likely to fall anywhere between minimum and maximum Useful when have reasonable guess about minimum and maximum but no reason to suspect any value in between more likely than others

Uniform

9 - 23

Most Highly Used Distributions (Continued)

Triangular distribution

More flexible than uniform Can have peak anywhere in its range Need minimum, maximum, and most likely values for outcomes Infinite tails negative values are possible Symmetric most likely outcome is midpoint Appropriate for representing uncertain quantities influenced by a large number of independent factors
9 - 24

Normal distribution

Crystal Ball Distributions

Option to enter alternative input parameters

Parms button in assumption window

Mean and standard deviation (standard) 10th and 90th percentile 5th and 95th percentile Mean and 90th percentile Mean and 95th percentile Custom (any percentiles)
9 - 25

Normal Distribution

Using Empirical Data

Create histogram Calculate mean and standard deviation Sample average generally accurate estimate of population mean Generally advisable to widen range from observed values

Actual results underestimate the extremes

CB provides Fit (in Gallery window) and CBTools Batch Fit

9 - 26

Ensuring Appropriate Precision in Outputs

Running a simulation is performing an experiment Purpose of experiment is to estimate some quantity There is some difference, or error, between the estimate and the true value A well-planned simulation study requires effort to measure this error Must ensure conclusions not compromised by error

9 - 27

Sources of Error
Model error

Model is abstraction of reality Caused by fact simulation is an experiment Usually a lesser problem than model error but should not be ignored Should be measured and controlled
9 - 28

Simulation error

Simulation Error
Small samples lead to large errors Even 1000 trials usually not sufficient for great accuracy Get closer to right answer as number of trials increases Mean Standard Error (MSE) measures precision

A 95% confidence interval for true mean extends to 1.96 MSEs on either side of the sample average
9 - 29

Precision Control in CB
Many times want to run simulation to some desired level of precision

e.g., want to be 90% confident within 2% of the true mean

Can do this with the Precision Control feature in CB CB will run as many trials as is needed to reach desired level of precision Must specify precision in:

Forecast window Run Preferences

9 - 30

Precision Control in Forecast Windows

Select a forecast cell and click on Define Forecast Click on More>> Click on Precision Make sure the check box for Select Precision Using is checked Chose the precision and statistic of interest

9 - 31

Precision Control Window

Absolute value of allowable difference Relative percentage difference allowed Statistic to measure precision over

9 - 32

Precision Control in Run Preferences

Open Run Run Preferences Check the box for Precision control Enter the percentage for the desired confidence interval Set the number of trials to a very large number CB will run then simulation until all confidence intervals (at given level) for defined forecasts are within given precisions
9 - 33

Precision Control Window

Confidence level for precision control

9 - 34

Probabilities of an Event
In many cases we want to calculate the probability of an event

e.g., probability that profit is less than $30,000

This can be done by defining a (0,1) r.v. as follows:

=IF(Profit<30,000,1,0) By defining this cell as a forecast, the average over n trials will be an estimate of the probability of the event The MSE shows accuracy of estimate

9 - 35

Precision and Simulation Error Wrap-up

Trade-off between precision of results and time taken to get them Easy to create models with excessive runtimes

Make sure only run to necessary level of precision

Remember that ultimate goal is a good decision not highly precise simulation results
9 - 36

Interpreting Simulation Outcomes

Raw result of a simulation is a file of n values for each forecast, where n is the number of trials CB displays and summarizes this data Display distribution of output

Histogram Cumulative distribution Reverse cumulative distribution

9 - 37

Forecast Window Preferences

Format

Change way numerical data displayed on chart

Chart Display Range

Choose range over which data displayed

9 - 38

Forecast Window Preferences Chart

Changes size of bins for histogram

Adds mean value to chart

9 - 39

Certainty Sliders

Can enter numbers directly Move to select range of outcomes

9 - 40

Forecast Window Statistics

Most useful

MSE View Statistics

9 - 41

Outputting Output Into a Spreadsheet

In many cases it is useful to store the CB output directly in the spreadsheet CB.GetForeStatFN(cell,stat)

Cell represents the forecast cell Stat represents the statistic calculated Statistics are in the same order as the statistics are output in the statistics table stat 1 = number of trials, stat 2 = mean, stat 13 = MSE

CB.GetForePercentFN(cell, percent) returns the percentile value of the given forecast cell
9 - 42

Percentiles View
***insert Figure 9.38

9 - 43

Interpreting Simulations
Take care in defining outcome variables

Capture essential aspects of model

Ensure estimates sufficiently precise to support meaningful interpretation Observe both expected outcomes and variability in outcomes

9 - 44

Optimization in Simulation
Approaches of Chapter 8 assume a deterministic model

Cannot be used when randomness in model Grid search Replicating the model CB Sensitivity OptQuest add-in
9 - 45

Simulation optimization approaches

Grid Search
Select a series of values to test for a decision variable Run the simulation at each value Can refine grid with smaller step size after initial runs

9 - 46

Replicating the Model

Replicate model on same sheet for each value of decision variable Each random outcome is simulated only once

Cuts down on simulation run time Removes one source of variability in differences between outcomes for different decision variables

Consider recording outcomes directly on sheet with CB.GetForeStatFN command

Can graph results to see trends

9 - 47

Trend Chart
Run Open Trend Chart Choose Forecasts

Select outputs to plot Forecast Axis Vertical usually preferred Value Axis Zero Based usually preferred Choose certainty bands to plot
9 - 48

Chart Preferences

CB Sensitivity
Select the forecast values and statistics of interest Select the decision variables to vary (up to two) Select the range of values for the decision variables Run then view values that are best choices for the forecasts of interest May refine range of values to test and rerun

9 - 49

OptQuest Add-in
Maximizes (or minimizes) an objective function by changing the values of decision variables possibly subject to one or more constraints

Use when objective depends on simulation outcomes Objective can be any summary measure of a forecast cell

9 - 50

Decision Variables
Highlight cell then Define Decision Variable Lower and upper limits defined

The tighter the limits the better OptQuest will work

Discrete

Specify number of evenly spaced alternatives

Fractional values are allowed

Continuous

Linear constraints on decision variables allowed

9 - 51

Requirements on Forecast Cells

Play the role of constraints for forecasts Not imposed before running simulation

Unlike constraints on decision variables Output is described as feasible or infeasible

Imposed afterwards

9 - 52

Steps to Use OptQuest

1a.
b.

Define decision variables, specify lower and upper bounds for the searching process Make sure at least one explicit assumption cell is defined through distribution gallery

9 - 53

Steps to Use OptQuest (Continued)

2. Load OptQuest
CBTools OptQuest Choose File New Best to close spreadsheets not related to optimization

3. In OptQuest
a. Tools Decision variables: specify LB and UB for the searching process
9 - 54

Steps to Use OptQuest (Continued)

b. Tools Forecasts: specify forecast statistics process

Select objective

Specify forecast statistic

9 - 55

Steps to Use OptQuest (Continued)

c. Tools Constraints: specify constraints on the decision variable (e.g., must sum to at most 1000) d. May specify requirements on forecast values e. Tools Options: specify time and other preferences

9 - 56

Steps to Use OptQuest (Continued)

4. Copy OptQuest solution to Excel In OptQuest: Edit Copy to Excel

5. In Excel, test the OptQuest result with more iterations Run Run Preferences

9 - 57

OptQuest Solutions
Heuristic approach

No guarantee solution is optimum

Extra care must be taken to ensure solution sensible Task complicated by the fact that the results of each set of trials is an estimate

Even if one combination of decision variables dominates another combination over one set of trials the second may actually be the better combination

9 - 58

OptQuest Solution Techniques

Scatter search

An initial population of starting values for the decision variables are randomly sampled and then simulated The initial population are combined in such a way that solutions already sampled are not resampled and the population of alternatives remains diverse A predictive model is derived based on past solutions and then used to filter out candidates that are likely to be poor (and therefore need not be simulated) 9 - 59

Tabu search

Neural networks

OptQuest Notes
Keep the simulation run length relatively short

Use MSE to determine acceptable level of error

Start with discrete decision variables and a coarse grid

Refine the grid in later runs

Consult the performance graph to see if it seems to have converged Use OptQuest Solution Analysis Experiment with different values of decision variables before running model
9 - 60

OptQuest Solution Analysis

First run OptQuest, then select Run Solution Analysis Enter percentage from best (e.g., 10%) Optquest will list all combinations tested that came within that percentage of best If there are many solutions then the response curve is probably quite flat and the solution quite robust to minor perturbations
9 - 61

Final OptQuest Note

OptQuest MUST have at least one explicit assumption cell defined through distribution gallery

This peculiar feature might be called a bug

This cell does not need to be linked to the model

A cell to the side that generates a Uniform(0,1) random variable will do the trick

If no such cell is defined then OptQuest will assume the model is deterministic and only run one replication for each value of the decision variables
9 - 62

Embedded Decision Process

Stage 1: Must make a set of long-term decisions in the face of uncertainty Subsequently uncertainty is resolved and conditions become known Stage 2: Need to make short-term decision given current conditions and constraints persisting from long-term decisions
9 - 63

Modeling Embedded Decision Process

Requires the use of both Solver and CB First build deterministic model Next reserve separate part for random inputs Invoke Solver for short-term model Use a Macro to use Solver on random inputs

Stored series of Excel commands

Coordinate with CB
9 - 64

Activities for Simulation Analysis Summary

Selecting uncertain parameters

Should have significant impact on outcome Uniform, triangular, normal, and custom most used Choose suitable simulation run lengths

Choosing probability distributions

Ensuring appropriate precision in outcomes

Interpreting outcome distributions

Results must be summarized and simplified

9 - 65

Summary
Simulation answers: What are the risks? CB convenient tool within Excel Appropriate sensitivity analysis should always be carried out on deterministic version of the model before simulating Optimizing within simulation not routine but number of approaches available

Grid search, OptQuest, etc.

9 - 66