Verification and validation

Verification and validation (V&V) are two checks that together show whether a calculation method in fire modelling can be trusted for a given task. Verification asks whether a model correctly represents the description of its developers, or in short, whether the model was "built" correctly. Validation asks whether the model is a suitable representation of the real world and reproduces the phenomena of interest, that is, whether the right model was "built". Verification is mainly a mathematical and programming task. Validation compares model predictions with measurements from fire experiments.

Fire models are used to show that a building or plant meets performance requirements. Guidance documents therefore expect the methods of analysis to be verified and validated, and to be used only within their field of application. The results of a validation study describe how accurate a model is for particular quantities, such as gas temperature or heat flux, over a particular range of conditions. Outside that range, the model may still work, but its accuracy has not been shown. A 2016 validation study expresses the accuracy of each model as a bias factor and a relative standard deviation, which can be applied to new predictions as uncertainty bounds.

Definitions

A widely used pair of definitions comes from a guide of the American Institute of Aeronautics and Astronautics, quoted in the SFPE Handbook of Fire Protection Engineering. Validation is the process of determining the degree to which a model is an accurate representation of the real world, from the perspective of its intended users. Verification is the process of determining that a model implementation accurately represents the conceptual description of its developer and the solution of the model.

For a computational fluid dynamics (CFD) fire model, the two checks cover different parts of the work. The submodels for turbulence, combustion and radiation must be validated to test how well they represent reality. Whether the model equations are then solved with adequate accuracy is a question of verification. A 2016 report for the United States Nuclear Regulatory Commission (NRC) puts it more briefly. Validation compares the models with measurements in full-scale fire experiments, while verification concerns the accuracy of the numerical solution of the governing equations.

Verification checks that the program solves the model equations correctly; validation checks the results against fire experiments. Based on Cox and Kumar (2002), p. 3-203, and Najafi et al. (2007), p. 1-2.

The two checks answer different questions, and neither replaces the other. A program can solve its equations accurately and still be wrong if the equations do not suit a fire. The turbulence models of many general CFD codes, for example, were developed for shear flows rather than for buoyant fire plumes. Equally, a model with sound physics can give wrong answers if its code contains an error, as the 2007 study discussed below found for several models. Every model also has a field of application. The user has to establish whether the scenario at hand fits within it, and has to know the simplifications that the model contains.

Why verification and validation matter

In the United States, fire models used to show compliance with NFPA 805 at nuclear power plants must be acceptable to the authority having jurisdiction. They must be applied only within their limitations and must be verified and validated. The analyst also has to justify that the chosen model suits the specific application.

In Australia, the handbook for the Fire Safety Verification Method of the Australian Building Codes Board (ABCB, 2020 edition) states that a validation and verification review of the methods of analysis should be undertaken. Many common equations and computer models have already been validated, for example those published in standards or in the International Fire Engineering Guidelines (2005). The review then checks that the method is used within its field of application, with appropriate inputs, and that it is fit for purpose. Use outside the field of application has to be justified in the design report. The International Fire Engineering Guidelines (2005) likewise state that methods of analysis should be well documented and well validated. They should also be suitable for the task and have clearly documented limitations and assumptions.

Independent checking belongs to the same picture. Fire safety engineering analyses can be so complex that a building authority cannot check them in detail, and CFD is a clear example. Authorities often have no CFD specialist on their staff, so the boundary conditions and assumptions of a simulation may not be checked at all. An independent second analysis is then advisable, for example when the results are unexpected or when the party that made them is unknown.

The ASTM E 1355 framework

The standard guide ASTM E 1355 describes how to evaluate the predictive capability of deterministic fire models. ASTM International distributes it as a general guide for this purpose. It identifies four steps:

1. define the model and the scenarios or phenomena to be evaluated;
2. assess the theoretical basis and the assumptions of the model;
3. assess the soundness of the mathematics and numerical methods of the model;
4. validate the model by quantifying the uncertainty and relative difference of its results.

A large V&V study for nuclear power plants, published by the NRC and the Electric Power Research Institute (EPRI) in 2007, followed this guide. It evaluated five tools: two libraries of empirical correlations, the zone models CFAST and MAGIC, and the CFD model Fire Dynamics Simulator (FDS). Each model report has chapters on the model definition, its theoretical basis, the soundness of its mathematics and numerics, a sensitivity analysis and the validation. The documentation of FDS follows the same framework. It consists of four volumes: the mathematical model, verification, experimental validation and configuration management.

A V&V study traditionally applies to the specific fire scenarios in its tests. The 2007 study did not cover scenarios outside the capabilities of the models, such as high-energy arcing faults or fire spread between control panels. It remains the responsibility of the user to decide whether a model can be applied to a given scenario.

Verification methods

ASTM E 1355 suggests three kinds of verification. Analytical tests run the program on a problem with a known mathematical solution. Code checking examines the structure of the computer code, by hand or with a checking program, to find irregularities and inconsistencies. Numerical tests examine the size of the residuals of the solved equations and how they decrease, as indicators of accuracy and convergence.

There is no exact solution for the full turbulent, time-dependent flow equations. Developers therefore test submodels that do have analytical solutions. The FDS verification suite contains several hundred test cases, which are all rerun with each new release to make sure that no new errors have entered the code. Examples include heat conduction into a very thick (semi-infinite) solid, the evaporation of water droplets, radiation from a hot object and the pressure rise in a sealed or slightly leaky compartment. For the simple empirical correlations, verification mainly means checking that the published formulae are correctly coded in the spreadsheets.

For CFD models, an important part of verification is a study of the mesh resolution, because the numerical error depends on the size of the grid cells. If the grid is refined to about 1 mm or less, FDS becomes a direct numerical simulation. Such runs are too costly for practical fire calculations, but they are useful for checking the numerical method. For the zone model CFAST, the error from the numerical solution is far smaller than the error that comes from the assumptions of the model.

Verification is not a formality. During the 2007 study, a number of modifications and corrections to the five models were identified and implemented. Their effects ranged from inconsequential to errors that could have led to incorrect results.

Validation against experiments

Validation compares model predictions with measurements from fire experiments. Confidence in a model can only be as good as the experiments and the model inputs derived from them. The 2007 study used the peak value of each predicted and measured quantity, measured from its ambient value, and expressed the comparison as a relative difference:

$$\epsilon =\frac{\mathrm{\Delta }M-\mathrm{\Delta }E}{\mathrm{\Delta }E}=\frac{(M_p-M_o)-(E_p-E_o)}{E_p-E_o}$$

A positive value means that the predicted rise is larger than the measured rise. For the oxygen concentration and the height of the hot gas layer, the minimum value is used instead of the peak. A relative difference alone says little, because the experiment is uncertain too. The study therefore compared each relative difference with a combined uncertainty. This combined value includes the uncertainty of the measuring instruments and the uncertainty of the model inputs, such as the measured heat release rate. If the predictions fall within this band, they are judged to be as accurate as the experiments and data.

The 2007 study summarised its results as colour grades. GREEN means that the physics of the model suits the calculation and that the relative differences lie within, or very near, the experimental uncertainty. YELLOW means that caution is needed, and YELLOW+ or YELLOW− indicates a consistent over-prediction or under-prediction. RED means that the capability should not be used. No model received a RED grade. The table shows some of the grades.

Quantity	CFAST	MAGIC	FDS
Hot gas layer temperature, room of origin	GREEN	GREEN	GREEN
Hot gas layer height	GREEN	GREEN	GREEN
Ceiling jet temperature	YELLOW+	GREEN	GREEN
Plume temperature	N/A	GREEN	YELLOW
Smoke concentration	YELLOW	YELLOW	YELLOW
Radiant heat flux	YELLOW	YELLOW	YELLOW

Validation is specific to each quantity. In a blind test of the CFD model JASMINE in the SFPE Handbook, gas temperatures and gas concentrations were reproduced reasonably well. The heat flux to the walls was the poorest prediction. The authors judged the method fit for gas-phase questions but not for heat transfer into solids.

In a blind test, the modellers receive the measurements only after submitting their results. Many published comparisons, by contrast, were made with prior knowledge of the experimental results. For its World Trade Center tests, NIST ran FDS before the experiments to obtain such "blind" predictions.

Bias factor and model uncertainty

A 2016 supplement to the NRC study expressed the accuracy of each model with two statistics. The bias factor $\delta$ shows how far the model over-predicts or under-predicts a quantity on average. The relative standard deviation ${\tilde{\sigma }}_M$ describes the scatter of the predictions. The experiments are assumed to be unbiased, with a relative standard deviation ${\tilde{\sigma }}_E$. For $n$ pairs of measured values $E_i$ and predicted values $M_i$, the statistics follow from the logarithm of their ratio:

Reading a validation plot: diagonal of perfect agreement, 95 % bounds of the experiments and of the model, here for FDS ceiling jet temperatures. Based on Stroup et al. (2016), pp. 6-1 and 6-57.

$$\overline{\ln (M/E)}=\frac{1}{n}\sum _{i=1}^n\ln \left(\frac{M_i}{E_i}\right)$$

$$\sqrt{{\tilde{\sigma }}_M^2+{\tilde{\sigma }}_E^2}\approx \sqrt{\frac{1}{n-1}\sum _{i=1}^n{[\ln \left(\frac{M_i}{E_i}\right)-\overline{\ln (M/E)}]}^2}$$

$$\delta =\exp (\overline{\ln (M/E)}+\frac{{\tilde{\sigma }}_M^2-{\tilde{\sigma }}_E^2}{2})$$

For a new prediction $M$, the true value is then treated as a normally distributed quantity with mean $\mu =M/\delta$ and standard deviation $\sigma ={\tilde{\sigma }}_M\mu$. In this way, a single prediction becomes a range of possible true values. The probability that it exceeds a critical value $x_c$ is:

$$P(x>x_c)=\frac{1}{2}\mathrm{erfc}\left(\frac{x_c-\mu }{\sigma \sqrt{2}}\right)$$

Symbol	Description	Unit
$\epsilon$	relative difference between prediction and measurement	–
$M_p,M_o$	peak and ambient value of the model prediction	unit of the quantity
$E_p,E_o$	peak and ambient value of the measurement	unit of the quantity
$M_i,E_i$	predicted and measured rise of pair $i$	unit of the quantity
$n$	number of pairs	–
$\delta$	bias factor of the model	–
${\tilde{\sigma }}_M,{\tilde{\sigma }}_E$	relative standard deviation of model and experiment	–
$\mu ,\sigma$	mean and standard deviation of the true value	unit of the quantity
$x_c$	critical value, measured as a rise above ambient	unit of the quantity
erfc	complementary error function	–

The method assumes that the relative differences are normally distributed. For some quantities the data are skewed, and the bounds then do not straddle the data evenly. The data are also weighted, so that a range with few measurements counts as much as a range with many.

The table lists some of the 2016 results for CFAST and FDS. A bias factor above 1 means over-prediction. Both the zone models and FDS over-predict the smoke concentration by about a factor of three, possibly because they ignore smoke losses to walls and ceiling. In general, FDS was the most accurate, followed by the zone models and then the empirical correlations. FDS needs hours or days for calculations that the other models do in less than a minute.

Quantity	CFAST $\delta$	CFAST ${\tilde{\sigma }}_M$	FDS $\delta$	FDS ${\tilde{\sigma }}_M$	Experiment ${\tilde{\sigma }}_E$
Hot gas layer temperature rise, natural ventilation	1.21	0.38	1.02	0.07	0.07
Hot gas layer depth	1.01	0.29	1.04	0.06	0.05
Ceiling jet temperature rise	1.06	0.42	0.99	0.12	0.07
Target temperature rise	1.25	0.49	0.99	0.17	0.07
Smoke concentration	3.42	0.68	2.63	0.60	0.19
Sprinkler activation time	1.01	0.20	0.93	0.15	0.06

Worked example

The 2016 report illustrates the method with electrical cables that fail when their surface reaches 330 °C. FDS predicts a maximum cable temperature of 300 °C in a room with an ambient temperature of 20 °C. The predicted rise is therefore $M$ = 280 °C. For target temperatures, FDS has a bias factor of 0.99 and a relative standard deviation of 0.17.

The adjusted prediction is $\mu$ = 280 / 0.99, or about 283 °C, and the standard deviation is $\sigma$ = 0.17 × 283, or about 48 °C. The critical rise is 330 − 20 = 310 °C. The argument of the error function is (310 − 283) / (48 × √2), or about 0.40, and the probability of failure is ½ erfc(0.40), or about 0.29. A prediction of 300 °C therefore does not mean that the cables survive with certainty. There is roughly a three in ten chance that the real temperature exceeds the failure value.

The same prediction from CFAST gives a different answer, because CFAST has a bias factor of 1.25 and a relative standard deviation of 0.49 for target temperatures. The adjusted mean rise is then about 224 °C and the standard deviation about 110 °C, giving a probability of about 0.22. The lower probability reflects the average over-prediction of CFAST, while the larger scatter shows that its result is less certain.

Possible true cable temperatures for a predicted 300 °C, using the FDS and CFAST validation statistics; shaded areas lie above the 330 °C failure temperature. Based on Stroup et al. (2016), pp. 6-3–6-4 and 6-57.

Validation domain and limitations

A GREEN grade or a small bias factor applies only within the range of conditions covered by the experiments. The 2016 supplement therefore lists that range as non-dimensional parameters. Adding experiments, from 26 to almost 500, widened the validated range of the fire Froude number from 0.4–2.4 to 0.2–9.1. The ratio of flame height to ceiling height widened from 0.2–1.0 to 0.0–1.6. In most cases, the bias factors and standard deviations changed very little. Room pressure rise was an exception: its agreement became worse, because the added experiments included tests with poorly characterised leakage.

Validated ranges of six non-dimensional parameters in the NRC study of 2007 and its 2016 supplement. Based on Stroup et al. (2016), p. 7-3.

Validation results also belong to a particular version of a model. The 2016 results may not apply to earlier versions, so users should consult the latest V&V reports for the version they use. Some limits are physical. The FDS equations are formulated for flows below about Mach 0.3, but the current validation of FDS extends only to about Mach 0.1.

Model uncertainty is only part of the total uncertainty of a result. The bias factor and standard deviation can serve as uncertainty bounds, but they must be combined with the uncertainty in the inputs chosen by the user. Depending on that input uncertainty, a sensitivity study may be needed. This combination is the subject of uncertainty analysis.

Validation is especially difficult for evacuation modelling. The International Fire Engineering Guidelines (2005) note that the analysis of occupant evacuation, particularly the pre-movement phase, is hampered by a lack of validated methods. Data from the literature, field studies or simulated evacuations, or well-substantiated engineering judgement, then have to be used. Evacuation software cannot reproduce real human behaviour. Its convincing output is largely the result of choices made by the modeller, such as the route that each person takes. The ABCB handbook points to ISO/TR 16730-5 for an example of the verification and validation of an evacuation model.

Common misconceptions

A common misconception is that a validated model is valid for any scenario. Validation shows accuracy only for the quantities and the range of conditions that were tested. A two-zone model may perform well in simple test rooms because those rooms suit its two-layer assumption. The same model can be less accurate in more complex scenarios or for quantities such as heat flux.

A second misconception is that impressive results prove a model right. However convincing a fire simulation looks, it cannot predict the course of a fire. It can only show the effects of a fire for the assumptions chosen by the modeller. For quantitative analyses, comparison with carefully controlled laboratory experiments remains the best measure of what models can and cannot do.

A third misconception is that verification and validation are the job of the software developer only. Commercial CFD codes include standard test cases, but the responsibility for showing validity for fire applications lies with the fire community. In each project, the user still has to show that the model is applied within its field of application, with suitable inputs and documented limitations.

See also

• Ethics in fire safety engineering
• CFAST
• Fire Dynamics Simulator
• Sensitivity analysis
• Zone model

References

• Australian Building Codes Board. (2020b). Handbook: Fire Safety Verification Method (Print version 1.1).
• Cox, G., & Kumar, S. (2002). Modeling enclosure fires using CFD. In P. J. DiNenno, D. Drysdale, C. L. Beyler, W. D. Walton, R. L. P. Custer, J. R. Hall Jr., & J. M. Watts Jr. (Eds.), SFPE handbook of fire protection engineering (3rd ed., pp. 3-194–3-218). National Fire Protection Association; Society of Fire Protection Engineers.
• McGrattan, K. B., Bouldin, C., & Forney, G. P. (2005). Computer simulation of the fires in the World Trade Center towers (NIST NCSTAR 1-5F). National Institute of Standards and Technology.
• McGrattan, K., Hostikka, S., Floyd, J., McDermott, R., Vanella, M., Mueller, E., & Paul, C. (2026). Fire Dynamics Simulator technical reference guide: Vol. 1. Mathematical model (6th ed., NIST Special Publication 1018-1). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.1018
• Najafi, B., Joglar, F., & Dreisbach, J. (2007). Verification and validation of selected fire models for nuclear power plant applications: Vol. 1. Main report (NUREG-1824; EPRI 1011999). U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research; Electric Power Research Institute.
• National Research Council of Canada, International Code Council, Department of Building and Housing, & Australian Building Codes Board. (2005). International fire engineering guidelines (2005 ed.). Australian Building Codes Board.
• Stroup, D., Lindeman, A., McGrattan, K., Peacock, R., Overholt, K., Joglar, F., LeStrange, S., & Montanez, S. (2016). Verification and validation of selected fire models for nuclear power plant applications: Supplement 1 (NUREG-1824, Supplement 1; EPRI 3002002182). U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research; Electric Power Research Institute.
• Van Rossum, E. (2027). Introduction to Fire Safety Engineering. Fire Safety Academy.