New FSE wiki article: Fire service performance measurement (written by Claude Opus 5.5) |
Add SVG figures fse-fire-service-performance-measurement-v1.svg, fse-fire-service-performance-measurement-v2.svg, fse-fire-service-performance-measurement-v3.svg, fse-fire-service-performance-measurement-v4.svg, fse-fire-service-performance-measurement-v5.svg (list-8 visual plan) |
||
| (One intermediate revision by one other user not shown) | |||
| Line 5: | Line 5: | ||
== Concepts and definitions == | == Concepts and definitions == | ||
Cline (2008) uses the definitions of the US Governmental Accounting Standards Board (GASB) to create a common language.<ref name="Cline-12-26">Cline (2008), p. 12-26.</ref> An input is the amount of financial and non-financial resources applied to producing a service, and an output is a measure of the quantity of service provided, which may include a quality component.<ref name="Cline-12-27">Cline (2008), Table 12.2.1, p. 12-27.</ref> An outcome is a measure of progress towards an objective, and effectiveness is the degree to which goals and objectives are achieved.<ref name="Cline-12-27" /> Efficiency is the relation between inputs and outputs or outcomes, measured as the resources used or cost per unit of output.<ref name="Cline-12-27" /> Explanatory factors are conditions that affect performance, either largely outside the organisation's control, such as demographics, or partly within it, such as staffing patterns.<ref name="Cline-12-27" /> | Cline (2008) uses the definitions of the US Governmental Accounting Standards Board (GASB) to create a common language.<ref name="Cline-12-26">Cline (2008), p. 12-26.</ref> An input is the amount of financial and non-financial resources applied to producing a service, and an output is a measure of the quantity of service provided, which may include a quality component.<ref name="Cline-12-27">Cline (2008), Table 12.2.1, p. 12-27.</ref> An outcome is a measure of progress towards an objective, and effectiveness is the degree to which goals and objectives are achieved.<ref name="Cline-12-27" /> Efficiency is the relation between inputs and outputs or outcomes, measured as the resources used or cost per unit of output.<ref name="Cline-12-27" /> Explanatory factors are conditions that affect performance, either largely outside the organisation's control, such as demographics, or partly within it, such as staffing patterns.<ref name="Cline-12-27" /> | ||
[[File:fse-fire-service-performance-measurement-v1.svg|thumb|upright=1.5|alt=Flow diagram from inputs to outputs to outcomes, with efficiency linking inputs to outputs and effectiveness linking outcomes to goals|Performance measurement terms: inputs, outputs and outcomes, with efficiency and effectiveness.]] | |||
The word benchmark has two meanings. It can mean the standard against which a measurement is compared, or the search for best practices in other organisations.<ref name="Cline-12-35" /> Krakeel (2008) describes benchmarking in the second sense, as emulating another agency's level of performance, and a standard as an accepted practice based on a consensus process.<ref name="Krakeel-13-152">Krakeel (2008), p. 13-152.</ref> Cline (2008) summarises the relation simply: the measurement is the count, and the benchmark is the yardstick against which the count is measured.<ref name="Cline-12-35" /> For example, the number of commercial buildings inspected becomes information when it is expressed as a percentage of all commercial buildings and compared with the previous year.<ref name="Cline-12-35" /> Benchmarks may come from technical standards, historical data, comparable organisations or local priorities.<ref name="Cline-12-35" /> | The word benchmark has two meanings. It can mean the standard against which a measurement is compared, or the search for best practices in other organisations.<ref name="Cline-12-35" /> Krakeel (2008) describes benchmarking in the second sense, as emulating another agency's level of performance, and a standard as an accepted practice based on a consensus process.<ref name="Krakeel-13-152">Krakeel (2008), p. 13-152.</ref> Cline (2008) summarises the relation simply: the measurement is the count, and the benchmark is the yardstick against which the count is measured.<ref name="Cline-12-35" /> For example, the number of commercial buildings inspected becomes information when it is expressed as a percentage of all commercial buildings and compared with the previous year.<ref name="Cline-12-35" /> Benchmarks may come from technical standards, historical data, comparable organisations or local priorities.<ref name="Cline-12-35" /> | ||
| Line 21: | Line 22: | ||
Time is the critical element when an emergency is reported, both for fire growth and for the survival of patients.<ref name="JohnsonPrice-12-217">Johnson and Price (2008), p. 12-217.</ref> For EMS systems, response time is the single performance measure most frequently used.<ref name="Krakeel-13-157" /> Johnson and Price (2008) divide the fire department total reflex time into dispatch time, turnout time, response (travel) time, access time and setup time.<ref name="JohnsonPrice-12-218">Johnson and Price (2008), p. 12-218.</ref> By understanding the objectives of each step, a department can measure its current performance against them and assess the cost of reducing each part.<ref name="JohnsonPrice-12-219">Johnson and Price (2008), p. 12-219.</ref> The components and their definitions are described in [[emergency response time]]. NFPA 450 gives a template of discrete time stamps, functional intervals and process intervals for EMS calls, so that agencies define response time uniformly.<ref name="Krakeel-13-157" /> | Time is the critical element when an emergency is reported, both for fire growth and for the survival of patients.<ref name="JohnsonPrice-12-217">Johnson and Price (2008), p. 12-217.</ref> For EMS systems, response time is the single performance measure most frequently used.<ref name="Krakeel-13-157" /> Johnson and Price (2008) divide the fire department total reflex time into dispatch time, turnout time, response (travel) time, access time and setup time.<ref name="JohnsonPrice-12-218">Johnson and Price (2008), p. 12-218.</ref> By understanding the objectives of each step, a department can measure its current performance against them and assess the cost of reducing each part.<ref name="JohnsonPrice-12-219">Johnson and Price (2008), p. 12-219.</ref> The components and their definitions are described in [[emergency response time]]. NFPA 450 gives a template of discrete time stamps, functional intervals and process intervals for EMS calls, so that agencies define response time uniformly.<ref name="Krakeel-13-157" /> | ||
In the standards described in the handbook, response objectives are stated as fractiles. As described by Johnson and Price (2008), NFPA 1710 sets time objectives for career departments, such as 60 seconds for turnout and 4 minutes or less for the first engine company to arrive.<ref name="JohnsonPrice-12-220" /> The fire department must set a performance objective of not less than 90 percent for achieving each time objective.<ref name="JohnsonPrice-12-220" /> NFPA 1221 requires that 95 percent of alarms be answered within 15 seconds, 99 percent within 40 seconds, and that dispatch be completed within 60 seconds at least 95 percent of the time.<ref name="JohnsonPrice-12-220" /> For ambulance services, | In the standards described in the handbook, response objectives are stated as fractiles. As described by Johnson and Price (2008), NFPA 1710 sets time objectives for career departments, such as 60 seconds for turnout and 4 minutes or less for the first engine company to arrive.<ref name="JohnsonPrice-12-220" /> The fire department must set a performance objective of not less than 90 percent for achieving each time objective.<ref name="JohnsonPrice-12-220" /> NFPA 1221 requires that 95 percent of alarms be answered within 15 seconds, 99 percent within 40 seconds, and that dispatch be completed within 60 seconds at least 95 percent of the time.<ref name="JohnsonPrice-12-220" /> For ambulance services, Krakeel (2008) reports a Commission on Accreditation of Ambulance Services measure of 90 percent compliance within 8 minutes 59 seconds for life-threatening calls.<ref name="Krakeel-13-155" /> The stated interval is for advanced life support ambulance arrival in urban and suburban communities, measured from notification of the responding unit.<ref name="Krakeel-13-157" /> As an illustration, the measured compliance fraction can be calculated and compared with the performance objective: | ||
<math display="block"> | <math display="block">p_\mathrm{meas} = \frac{N(t_i \le t_\mathrm{obj})}{N_\mathrm{tot}}, \qquad \text{goal met if } p_\mathrm{meas} \ge p_\mathrm{obj}</math> | ||
{| class="wikitable" | {| class="wikitable" | ||
| Line 36: | Line 37: | ||
| <math>N_\mathrm{tot}</math> || Total number of responses in the period || – | | <math>N_\mathrm{tot}</math> || Total number of responses in the period || – | ||
|- | |- | ||
| <math> | | <math>p_\mathrm{meas}</math> || Measured compliance rate || – | ||
|- | |- | ||
| <math>p_\mathrm{obj}</math> || Performance objective, for example 0.90 || – | | <math>p_\mathrm{obj}</math> || Performance objective, for example 0.90 || – | ||
| Line 42: | Line 43: | ||
A sample performance report in Cline (2008) shows why the fractile matters. The operational goal of a fictional emergency response programme was a response time of 4 minutes or less, 90 percent of the time.<ref name="Cline-12-36">Cline (2008), Table 12.2.2, p. 12-36.</ref> For July to December 2007, the reported average response time was 3 minutes 52 seconds, but only 69 percent of responses arrived within 4 minutes.<ref name="Cline-12-36" /> The average was 8 seconds inside the objective, yet compliance fell 21 percentage points short of the 90 percent goal. The report recorded an explanatory factor: freeway works had put detours on 20 percent of major traffic corridors in one part of the community.<ref name="Cline-12-36" /> As an illustration of the NFPA 1221 fractiles, a centre receiving 1,000 alarms would need to answer at least 950 within 15 seconds and 990 within 40 seconds. | A sample performance report in Cline (2008) shows why the fractile matters. The operational goal of a fictional emergency response programme was a response time of 4 minutes or less, 90 percent of the time.<ref name="Cline-12-36">Cline (2008), Table 12.2.2, p. 12-36.</ref> For July to December 2007, the reported average response time was 3 minutes 52 seconds, but only 69 percent of responses arrived within 4 minutes.<ref name="Cline-12-36" /> The average was 8 seconds inside the objective, yet compliance fell 21 percentage points short of the 90 percent goal. The report recorded an explanatory factor: freeway works had put detours on 20 percent of major traffic corridors in one part of the community.<ref name="Cline-12-36" /> As an illustration of the NFPA 1221 fractiles, a centre receiving 1,000 alarms would need to answer at least 950 within 15 seconds and 990 within 40 seconds. | ||
[[File:fse-fire-service-performance-measurement-v2.svg|thumb|upright=1.5|alt=Two-panel chart: average response time 3 min 52 s against a 4-minute target, and 69 percent compliance against a 90 percent objective|Sample performance report: the average response time (3 min 52 s) meets the 4-minute target, but only 69 percent of responses met it against an objective of 90 percent.]] | |||
These measures feed the [[standards of cover]] process of the Commission on Fire Accreditation International. Its historical response effectiveness study asks what percentage of compliance the existing system delivers against current goals.<ref name="JohnsonPrice-12-221">Johnson and Price (2008), p. 12-221.</ref> Its historical reliability study looks for call stacking and simultaneous calls in an area.<ref name="JohnsonPrice-12-221" /> The resulting statements are written as fractiles. One example: in 90 percent of incidents the first-due unit arrives within 4 minutes' travel, or 6 minutes' total reflex time.<ref name="JohnsonPrice-12-222">Johnson and Price (2008), p. 12-222.</ref> Commercial software can also analyse dispatch data to find problems in response patterns, such as neighbourhoods that receive subpar service.<ref name="Denlinger-12-71" /> | These measures feed the [[standards of cover]] process of the Commission on Fire Accreditation International. Its historical response effectiveness study asks what percentage of compliance the existing system delivers against current goals.<ref name="JohnsonPrice-12-221">Johnson and Price (2008), p. 12-221.</ref> Its historical reliability study looks for call stacking and simultaneous calls in an area.<ref name="JohnsonPrice-12-221" /> The resulting statements are written as fractiles. One example: in 90 percent of incidents the first-due unit arrives within 4 minutes' travel, or 6 minutes' total reflex time.<ref name="JohnsonPrice-12-222">Johnson and Price (2008), p. 12-222.</ref> The chapter defines total reflex time as including access and setup after arrival, although this example describes both times as arrival objectives.<ref name="JohnsonPrice-12-218" /> Commercial software can also analyse dispatch data to find problems in response patterns, such as neighbourhoods that receive subpar service.<ref name="Denlinger-12-71" /> | ||
== Outcome measures and their limits == | == Outcome measures and their limits == | ||
Outcome measures address the ultimate purpose of the fire service, but they are hard to interpret. The fact that most fires are suppressed with minimal losses does not show conclusively that service is adequate, and a large-loss fire does not necessarily mean that a department is inadequate.<ref name="Denlinger-12-71" /> Granito (2008) proposes the total cost of fire as an overall measure | Outcome measures address the ultimate purpose of the fire service, but they are hard to interpret. The fact that most fires are suppressed with minimal losses does not show conclusively that service is adequate, and a large-loss fire does not necessarily mean that a department is inadequate.<ref name="Denlinger-12-71" /> Granito (2008) proposes the total cost of fire as an overall measure. It can be compared with earlier years or with an average over three or more years; local officials can adjust for inflation and community growth or decline.<ref name="Granito-12-7">Granito (2008), p. 12-7.</ref> It is the sum of fire losses, the cost of prevention, the cost of suppression and the cost of fire insurance, and it can be tracked as prevention programmes are phased in.<ref name="Granito-12-15">Granito (2008), p. 12-15.</ref> An algebraic illustration of those components is: | ||
<math display="block">C_\mathrm{tot} = L + C_\mathrm{prev} + C_\mathrm{supp} + C_\mathrm{ins}</math> | <math display="block">C_\mathrm{tot} = L + C_\mathrm{prev} + C_\mathrm{supp} + C_\mathrm{ins}</math> | ||
| Line 64: | Line 66: | ||
|} | |} | ||
For public fire and life safety education, Hall (2008) treats fire deaths as the primary outcome, but they are rare, so random variation can hide or mimic a real effect. He illustrates this with the home smoke alarm campaign in Louisville, Kentucky, in the 1980s.<ref name="Hall-5-109" /> The fire death toll averaged 15 a year in 1980–1984 and 11 a year in 1985–1988, a decline of more than a quarter.<ref name="Hall-5-109" /> Treating all nine years as one set of annual death tolls, the difference between the two averages was only about 1.2 standard deviations.<ref name="Hall-5-109" /> An algebraic illustration of that comparison is: | |||
[[File:fse-fire-service-performance-measurement-v3.svg|thumb|upright=1.5|alt=Bar chart of annual fire deaths in Louisville 1980 to 1988 with averages of 15 before and 11 after the smoke alarm programme began in 1985|Fire deaths in Louisville, Kentucky, before and after a home smoke alarm programme began in 1985. The decline of about a quarter was only about 1.2 standard deviations.]] | |||
<math display="block">z = \frac{\bar{x}_\mathrm{before} - \bar{x}_\mathrm{after}}{s}</math> | <math display="block">z = \frac{\bar{x}_\mathrm{before} - \bar{x}_\mathrm{after}}{s}</math> | ||
| Line 75: | Line 78: | ||
| <math>\bar{x}_\mathrm{after}</math> || Mean annual number of fire deaths after the programme || deaths per year | | <math>\bar{x}_\mathrm{after}</math> || Mean annual number of fire deaths after the programme || deaths per year | ||
|- | |- | ||
| <math>s</math> || | | <math>s</math> || Population standard deviation of the nine annual death tolls || deaths per year | ||
|- | |- | ||
| <math>z</math> || Difference between the means in standard deviations || – | | <math>z</math> || Difference between the means in standard deviations || – | ||
|} | |} | ||
Statisticians usually look for at least 1⅔ standard deviations for significance in a one-sided test, so the Louisville result was not yet statistically significant.<ref name="Hall-5-109" /> Hall (2008) shows the nine annual tolls: 14, 13, 19, 15 and 13 before the programme, and 10, 9, 16 and 8 after it. | Statisticians usually look for at least 1⅔ standard deviations for significance in a one-sided test, so the Louisville result was not yet statistically significant.<ref name="Hall-5-109" /> Hall (2008) shows the nine annual tolls: 14, 13, 19, 15 and 13 before the programme, and 10, 9, 16 and 8 after it.<ref name="Hall-5-109" /> As a check using the population standard deviation of these nine values, the spread is about 3.3 deaths a year, so <math>z</math> ≈ 4.05/3.33 ≈ 1.2, reproducing the chapter's value. As an illustration, reaching 1⅔ standard deviations with that spread would have required a difference of about 5.6 deaths a year. | ||
Hall (2008) therefore recommends a baseline of at least five years, and more years for communities of fewer than 100,000 people.<ref name="Hall-5-109" /> For a population near 10,000 or less, a community's own fire deaths are too few to track success in any reasonable time. National or state data, or data from similar communities, should then be used.<ref name="Hall-5-109" /> For fire safety education aimed at prevention, as in many [[community risk reduction]] programmes, fire incidents are a better primary outcome measure, because fires are 50 to 100 times as common as fire deaths.<ref name="Hall-5-109" /> Hall and Weller (2008) distinguish four stages of programme evaluation: formative, process, impact and outcome evaluation.<ref name="HallWeller-5-120">Hall and Weller (2008), Table 5.8.1, p. 5-120.</ref> Outcome evaluation, which tests whether a programme reduces death and injury, requires a large study population and a comparable control group that did not receive the programme.<ref name="HallWeller-5-120" /> | Hall (2008) therefore recommends a baseline of at least five years, and more years for communities of fewer than 100,000 people.<ref name="Hall-5-109" /> For a population near 10,000 or less, a community's own fire deaths are too few to track success in any reasonable time. National or state data, or data from similar communities, should then be used.<ref name="Hall-5-109" /> For fire safety education aimed at prevention, as in many [[community risk reduction]] programmes, fire incidents are a better primary outcome measure, because fires are 50 to 100 times as common as fire deaths.<ref name="Hall-5-109" /> Hall and Weller (2008) distinguish four stages of programme evaluation: formative, process, impact and outcome evaluation.<ref name="HallWeller-5-120">Hall and Weller (2008), Table 5.8.1, p. 5-120.</ref> Outcome evaluation, which tests whether a programme reduces death and injury, requires a large study population and a comparable control group that did not receive the programme.<ref name="HallWeller-5-120" /> | ||
== Benchmarking between departments == | == Benchmarking between departments == | ||
Comparing a department with similar communities is a popular method, but Granito (2008) warns that the number of site-specific variables can be so high that comparisons prove confusing, so benchmarking requires cautious application.<ref name="Granito-12-7" /> Most fire departments benchmark against other fire departments | Comparing a department with similar communities is a popular method, but Granito (2008) warns that the number of site-specific variables can be so high that comparisons prove confusing, so benchmarking requires cautious application.<ref name="Granito-12-7" /> Most fire departments benchmark against other fire departments; large metropolitan departments give particular attention to neighbouring communities.<ref name="Cline-12-35" /> Prince William County, Virginia, compared itself with other combination departments, but organisational and cultural differences sometimes made this "comparing apples to oranges".<ref name="Cline-12-36" /> Not everything that is counted is a performance measure. Cline (2008) cites Ammons in noting that national or regional norms, such as firefighters per capita, show whether local practice lies within normal bounds but do not address performance.<ref name="Cline-12-35" /> | ||
Firefighters per capita also depends on the working week. In some US regions the scheduled week was 40 hours and in others 56 hours, so two communities with the same number of firefighters on duty could employ very different numbers.<ref name="Cline-12-37" /> As an illustration with roster numbers from Denlinger (2008), 58 positions on duty need about 216 members on a three-platoon | Firefighters per capita also depends on the working week. In some US regions the scheduled week was 40 hours and in others 56 hours, so two communities with the same number of firefighters on duty could employ very different numbers.<ref name="Cline-12-37" /> As an illustration with roster numbers from Denlinger (2008), 58 positions on duty need about 216 members on a three-platoon system with 24-hour tours, but about 289 in the chapter's separate 42-hour average workweek example.<ref name="Denlinger-12-73">Denlinger (2008), p. 12-73.</ref> For a city of 250,000 people, that is 0.86 or 1.16 firefighters per thousand residents for the same daily coverage. Such variables should be noted when choosing benchmark communities, even when a true like-for-like comparison is not possible.<ref name="Cline-12-37" /> | ||
Accreditation offers a more structured comparison. The Commission on Fire Accreditation International, established in 1996, provides a voluntary accreditation programme with 10 categories that include performance indicators.<ref name="Krakeel-13-157" /> It provides a standardised self-assessment with the help of a visiting team, and performance indications serve as its points of measurement.<ref name="Granito-12-7" /> The ISO Public Protection Classification is a separate grading for insurance purposes; in a January 2001 survey, 92 percent of 502 US fire chiefs and officials said their classification directly reflected improvements in their community.<ref name="Granito-12-15" /> | Accreditation offers a more structured comparison. The Commission on Fire Accreditation International, established in 1996, provides a voluntary accreditation programme with 10 categories that include performance indicators.<ref name="Krakeel-13-157" /> It provides a standardised self-assessment with the help of a visiting team, and performance indications serve as its points of measurement.<ref name="Granito-12-7" /> The ISO Public Protection Classification is a separate grading for insurance purposes; in a January 2001 survey, 92 percent of 502 US fire chiefs and officials said their classification directly reflected improvements in their community.<ref name="Granito-12-15" /> | ||
| Line 95: | Line 98: | ||
Within a department, Cline (2008) recommends a single, centralised database with a designated custodian as the only source of published performance information.<ref name="Cline-12-32">Cline (2008), p. 12-32.</ref> Even a simple measure such as the number of firefighters on duty can differ if one unit counts people and another counts positions, including vacancies.<ref name="Cline-12-33">Cline (2008), p. 12-33.</ref> Data entry errors in dispatch and incident systems can skew reports, so operational managers should review the data before public release.<ref name="Cline-12-32" /> Cline (2008) suggests a monthly reporting cycle for operational managers and a six-month cycle for senior managers.<ref name="Cline-12-34-35">Cline (2008), pp. 12-34–12-35.</ref> A performance report contains three parts: program descriptors, performance information and planning objectives.<ref name="Cline-12-35" /> The roles in this system follow the levels of the [[fire department organisation]], from the crews who deliver services to the managers who set policy.<ref name="Cline-12-26" /> | Within a department, Cline (2008) recommends a single, centralised database with a designated custodian as the only source of published performance information.<ref name="Cline-12-32">Cline (2008), p. 12-32.</ref> Even a simple measure such as the number of firefighters on duty can differ if one unit counts people and another counts positions, including vacancies.<ref name="Cline-12-33">Cline (2008), p. 12-33.</ref> Data entry errors in dispatch and incident systems can skew reports, so operational managers should review the data before public release.<ref name="Cline-12-32" /> Cline (2008) suggests a monthly reporting cycle for operational managers and a six-month cycle for senior managers.<ref name="Cline-12-34-35">Cline (2008), pp. 12-34–12-35.</ref> A performance report contains three parts: program descriptors, performance information and planning objectives.<ref name="Cline-12-35" /> The roles in this system follow the levels of the [[fire department organisation]], from the crews who deliver services to the managers who set policy.<ref name="Cline-12-26" /> | ||
[[File:fse-fire-service-performance-measurement-v4.svg|thumb|upright=1.5|alt=Funnel diagram: programme data flow to service areas, then divisions, then a centralised performance database|Flow of performance data in a fire department, from programmes to a centralised database.]] | |||
== Use in risk analysis and fire engineering == | == Use in risk analysis and fire engineering == | ||
Measured performance of fire brigades is also an input to engineering analysis. In fire probabilistic risk assessment for US nuclear power plants, earlier analyses used either the historical duration of fires or fire brigade response times as shown in unannounced fire drills.<ref name="Nowlen-3-5">Nowlen et al. (2005a), p. 3-5.</ref> NUREG/CR-6850 combines both: the brigade response is assessed from the scenario-specific response time and from suppression time curves derived from fire event data.<ref name="Nowlen-3-5" /> Its worked example uses a brigade response time of 7 minutes taken from fire drill records.<ref name="Nowlen-P-14">Nowlen et al. (2005b), p. P-14.</ref> With detection after 1 minute and target damage at 15 minutes, 7 minutes remain for manual suppression. The probability that the brigade then fails to suppress the fire follows from the suppression curve:<ref name="Nowlen-P-14" /><ref name="Hyslop-14-2">Hyslop et al. (2010), p. 14-2.</ref> | Measured performance of fire brigades is also an input to engineering analysis. In fire probabilistic risk assessment for US nuclear power plants, earlier analyses used either the historical duration of fires or fire brigade response times as shown in unannounced fire drills.<ref name="Nowlen-3-5">Nowlen et al. (2005a), p. 3-5.</ref> NUREG/CR-6850 combines both: the brigade response is assessed from the scenario-specific response time and from suppression time curves derived from fire event data.<ref name="Nowlen-3-5" /> Its worked example uses a brigade response time of 7 minutes taken from fire drill records.<ref name="Nowlen-P-14">Nowlen et al. (2005b), p. P-14.</ref> With detection after 1 minute and target damage at 15 minutes, 7 minutes remain for manual suppression. The probability that the brigade then fails to suppress the fire follows from the suppression curve:<ref name="Nowlen-P-14" /><ref name="Hyslop-14-2">Hyslop et al. (2010), p. 14-2.</ref> | ||
[[File:fse-fire-service-performance-measurement-v5.svg|thumb|upright=1.5|alt=Line chart of non-suppression probability falling exponentially with time available, with the example point at 7 minutes and 0.43|Probability of non-suppression in the 2005 electrical-fire example, P = exp(−0.12 t), with 7 minutes available giving 0.43.]] | |||
<math display="block">P_\mathrm{ns} = \exp(-\lambda \, t), \qquad t = t_\mathrm{damage} - t_\mathrm{det} - t_\mathrm{fb}</math> | <math display="block">P_\mathrm{ns} = \exp(-\lambda \, t), \qquad t = t_\mathrm{damage} - t_\mathrm{det} - t_\mathrm{fb}</math> | ||
| Line 117: | Line 122: | ||
|} | |} | ||
For | For the 2005 electrical-fire example with <math>\lambda</math> = 0.12 per minute, <math>P_\mathrm{ns}</math> = exp(−0.12 × 7) ≈ 0.43.<ref name="Nowlen-P-14" /> A 2010 supplement revised this method so that the brigade response time is already included in the distribution of suppression times and is no longer subtracted separately.<ref name="Hyslop-14-2" /> In Australia, the Fire Safety Verification Method handbook (2020) uses a related approach for buildings. If a distribution of fire brigade intervention times is not used, a slow response must be assumed, typically the 95th to 99th percentile of the time to application of water.<ref name="ABCB-104">Australian Building Codes Board (2020b), p. 104.</ref> The revised nuclear method uses a distribution of suppression times that includes brigade response; the Australian handbook instead discusses a distribution of intervention times or a slow-response percentile.<ref name="Hyslop-14-2" /><ref name="ABCB-104" /> | ||
== See also == | == See also == | ||
Latest revision as of 18:12, 14 November 2026
Fire service performance measurement is the systematic counting and judging of what a fire department does and achieves, compared with a defined standard of service. Cline (2008) defines evaluating as measuring and judging performance as progress towards a defined standard, and benchmarking as selecting the appropriate standard to use.[1] Together they turn raw counts into information that can be used to obtain resources, acknowledge success and improve services.[1]
Typical measures cover the resources used (inputs), the services delivered (outputs), the results for the community (outcomes), efficiency and effectiveness.[2] Response time is the measure most often used for emergency medical services, and standards state it as the share of calls that meet a time objective, such as 90 percent, rather than as an average.[3][4] Outcome measures such as fire deaths are harder to use, because the effect of a department's work is difficult to separate from chance and from factors outside its control.[5][6] This article describes the concepts and US practice as reported in the 2008 edition of the NFPA Fire Protection Handbook. It adds how measured fire brigade performance is used in risk analysis for US nuclear power plants and in the Australian performance-based building code.
Concepts and definitions
Cline (2008) uses the definitions of the US Governmental Accounting Standards Board (GASB) to create a common language.[7] An input is the amount of financial and non-financial resources applied to producing a service, and an output is a measure of the quantity of service provided, which may include a quality component.[8] An outcome is a measure of progress towards an objective, and effectiveness is the degree to which goals and objectives are achieved.[8] Efficiency is the relation between inputs and outputs or outcomes, measured as the resources used or cost per unit of output.[8] Explanatory factors are conditions that affect performance, either largely outside the organisation's control, such as demographics, or partly within it, such as staffing patterns.[8]
The word benchmark has two meanings. It can mean the standard against which a measurement is compared, or the search for best practices in other organisations.[2] Krakeel (2008) describes benchmarking in the second sense, as emulating another agency's level of performance, and a standard as an accepted practice based on a consensus process.[9] Cline (2008) summarises the relation simply: the measurement is the count, and the benchmark is the yardstick against which the count is measured.[2] For example, the number of commercial buildings inspected becomes information when it is expressed as a percentage of all commercial buildings and compared with the previous year.[2] Benchmarks may come from technical standards, historical data, comparable organisations or local priorities.[2]
Why fire departments measure performance
Denlinger (2008) notes that effectiveness in the fire service is difficult to measure, because the number of fires and losses that a department has prevented cannot easily be assessed.[5] Local fire losses do not necessarily reflect a department's performance, so managers have to rely on regional and national loss experience and on regular assessment of company effectiveness.[10] Measurement is therefore built into planning. Granito (2008) describes planning as a feedback loop, in which a continuing assessment shows how well a plan contributes to its goals and revised data are fed back into it.[11] A fire protection master plan should also establish a framework for measuring the effectiveness of fire protection expenditures.[12]
Performance data also support budgets. Many local governments require budget documents to include performance measures, so that funding is linked to its effect on service delivery.[13] To calculate efficiency, expenditures (inputs) must be linked to service delivery (outputs).[14] In a survey by Georgia State University and GASB, nearly 80 percent of local government respondents considered the weak link between performance and accounting databases to be a problem.[14] Duggan (2008) adds that a compelling case for more resources depends on the ability to extract and present quality data.[15] Cline (2008) stresses that results should not be punitive: a shortfall should prompt a review of resources, processes, training or equipment.[7]
What is measured
The measures a fire department collects follow its services. For emergency services, Cline (2008) lists calls and dispatched units by type, actual and average response times, and the percentage of responses within a set number of minutes.[16] The list also includes dispatch, turnout and travel time, patients transported, defibrillator use, training hours and certification levels.[16] Fire prevention measures include inspections and their turnaround time, fires contained to the room, unit or building of origin, and fire loss.[17] Civilian injury and death rates and the number of residents with smoke detectors or sprinklers are also listed.[17] Service dimensions such as population, area served, personnel, budget, stations and minimum staffing describe the context in which the results were achieved.[17]
The Ontario Fire Marshal's Office in Canada developed a performance measurement and benchmarking model that separates outputs from operational outcomes.[16] Outputs are the service deliverables, such as fires responded to, schools visited and homes inspected.[18] Operational outcomes are their effects, such as fires kept to the room of origin, homes with working smoke alarms and firefighters trained to handle fires without endangering their lives.[18] The model's key performance indicators include the time from the initial emergency call to the arrival of the first team, the containment of fire spread and the extent of fire-related injury and death.[17] For fire-based emergency medical services, the International Association of Fire Fighters developed measures for call processing, turnout, travel, staffing, deployment, protocol compliance and patient outcomes.[19]
Response time measures
Time is the critical element when an emergency is reported, both for fire growth and for the survival of patients.[20] For EMS systems, response time is the single performance measure most frequently used.[3] Johnson and Price (2008) divide the fire department total reflex time into dispatch time, turnout time, response (travel) time, access time and setup time.[21] By understanding the objectives of each step, a department can measure its current performance against them and assess the cost of reducing each part.[22] The components and their definitions are described in emergency response time. NFPA 450 gives a template of discrete time stamps, functional intervals and process intervals for EMS calls, so that agencies define response time uniformly.[3]
In the standards described in the handbook, response objectives are stated as fractiles. As described by Johnson and Price (2008), NFPA 1710 sets time objectives for career departments, such as 60 seconds for turnout and 4 minutes or less for the first engine company to arrive.[4] The fire department must set a performance objective of not less than 90 percent for achieving each time objective.[4] NFPA 1221 requires that 95 percent of alarms be answered within 15 seconds, 99 percent within 40 seconds, and that dispatch be completed within 60 seconds at least 95 percent of the time.[4] For ambulance services, Krakeel (2008) reports a Commission on Accreditation of Ambulance Services measure of 90 percent compliance within 8 minutes 59 seconds for life-threatening calls.[19] The stated interval is for advanced life support ambulance arrival in urban and suburban communities, measured from notification of the responding unit.[3] As an illustration, the measured compliance fraction can be calculated and compared with the performance objective:
| Symbol | Description | Unit |
|---|---|---|
| Measured time of response | s | |
| Time objective, for example 240 s for the first engine company | s | |
| Number of responses that met the time objective | – | |
| Total number of responses in the period | – | |
| Measured compliance rate | – | |
| Performance objective, for example 0.90 | – |
A sample performance report in Cline (2008) shows why the fractile matters. The operational goal of a fictional emergency response programme was a response time of 4 minutes or less, 90 percent of the time.[23] For July to December 2007, the reported average response time was 3 minutes 52 seconds, but only 69 percent of responses arrived within 4 minutes.[23] The average was 8 seconds inside the objective, yet compliance fell 21 percentage points short of the 90 percent goal. The report recorded an explanatory factor: freeway works had put detours on 20 percent of major traffic corridors in one part of the community.[23] As an illustration of the NFPA 1221 fractiles, a centre receiving 1,000 alarms would need to answer at least 950 within 15 seconds and 990 within 40 seconds.
These measures feed the standards of cover process of the Commission on Fire Accreditation International. Its historical response effectiveness study asks what percentage of compliance the existing system delivers against current goals.[24] Its historical reliability study looks for call stacking and simultaneous calls in an area.[24] The resulting statements are written as fractiles. One example: in 90 percent of incidents the first-due unit arrives within 4 minutes' travel, or 6 minutes' total reflex time.[25] The chapter defines total reflex time as including access and setup after arrival, although this example describes both times as arrival objectives.[21] Commercial software can also analyse dispatch data to find problems in response patterns, such as neighbourhoods that receive subpar service.[10]
Outcome measures and their limits
Outcome measures address the ultimate purpose of the fire service, but they are hard to interpret. The fact that most fires are suppressed with minimal losses does not show conclusively that service is adequate, and a large-loss fire does not necessarily mean that a department is inadequate.[10] Granito (2008) proposes the total cost of fire as an overall measure. It can be compared with earlier years or with an average over three or more years; local officials can adjust for inflation and community growth or decline.[26] It is the sum of fire losses, the cost of prevention, the cost of suppression and the cost of fire insurance, and it can be tracked as prevention programmes are phased in.[27] An algebraic illustration of those components is:
| Symbol | Description | Unit |
|---|---|---|
| Total cost of fire to the community | currency per year | |
| Fire losses (property, life and injury) | currency per year | |
| Cost of fire prevention | currency per year | |
| Cost of fire suppression | currency per year | |
| Cost of fire insurance | currency per year |
For public fire and life safety education, Hall (2008) treats fire deaths as the primary outcome, but they are rare, so random variation can hide or mimic a real effect. He illustrates this with the home smoke alarm campaign in Louisville, Kentucky, in the 1980s.[6] The fire death toll averaged 15 a year in 1980–1984 and 11 a year in 1985–1988, a decline of more than a quarter.[6] Treating all nine years as one set of annual death tolls, the difference between the two averages was only about 1.2 standard deviations.[6] An algebraic illustration of that comparison is:
| Symbol | Description | Unit |
|---|---|---|
| Mean annual number of fire deaths before the programme | deaths per year | |
| Mean annual number of fire deaths after the programme | deaths per year | |
| Population standard deviation of the nine annual death tolls | deaths per year | |
| Difference between the means in standard deviations | – |
Statisticians usually look for at least 1⅔ standard deviations for significance in a one-sided test, so the Louisville result was not yet statistically significant.[6] Hall (2008) shows the nine annual tolls: 14, 13, 19, 15 and 13 before the programme, and 10, 9, 16 and 8 after it.[6] As a check using the population standard deviation of these nine values, the spread is about 3.3 deaths a year, so ≈ 4.05/3.33 ≈ 1.2, reproducing the chapter's value. As an illustration, reaching 1⅔ standard deviations with that spread would have required a difference of about 5.6 deaths a year.
Hall (2008) therefore recommends a baseline of at least five years, and more years for communities of fewer than 100,000 people.[6] For a population near 10,000 or less, a community's own fire deaths are too few to track success in any reasonable time. National or state data, or data from similar communities, should then be used.[6] For fire safety education aimed at prevention, as in many community risk reduction programmes, fire incidents are a better primary outcome measure, because fires are 50 to 100 times as common as fire deaths.[6] Hall and Weller (2008) distinguish four stages of programme evaluation: formative, process, impact and outcome evaluation.[28] Outcome evaluation, which tests whether a programme reduces death and injury, requires a large study population and a comparable control group that did not receive the programme.[28]
Benchmarking between departments
Comparing a department with similar communities is a popular method, but Granito (2008) warns that the number of site-specific variables can be so high that comparisons prove confusing, so benchmarking requires cautious application.[26] Most fire departments benchmark against other fire departments; large metropolitan departments give particular attention to neighbouring communities.[2] Prince William County, Virginia, compared itself with other combination departments, but organisational and cultural differences sometimes made this "comparing apples to oranges".[23] Not everything that is counted is a performance measure. Cline (2008) cites Ammons in noting that national or regional norms, such as firefighters per capita, show whether local practice lies within normal bounds but do not address performance.[2]
Firefighters per capita also depends on the working week. In some US regions the scheduled week was 40 hours and in others 56 hours, so two communities with the same number of firefighters on duty could employ very different numbers.[16] As an illustration with roster numbers from Denlinger (2008), 58 positions on duty need about 216 members on a three-platoon system with 24-hour tours, but about 289 in the chapter's separate 42-hour average workweek example.[29] For a city of 250,000 people, that is 0.86 or 1.16 firefighters per thousand residents for the same daily coverage. Such variables should be noted when choosing benchmark communities, even when a true like-for-like comparison is not possible.[16]
Accreditation offers a more structured comparison. The Commission on Fire Accreditation International, established in 1996, provides a voluntary accreditation programme with 10 categories that include performance indicators.[3] It provides a standardised self-assessment with the help of a visiting team, and performance indications serve as its points of measurement.[26] The ISO Public Protection Classification is a separate grading for insurance purposes; in a January 2001 survey, 92 percent of 502 US fire chiefs and officials said their classification directly reflected improvements in their community.[27]
Data, records and reporting
Performance measurement depends on consistent incident data. Statistics from standardised reports reveal trends, allow comparisons across jurisdictions and play an important part in service and programme evaluations. In the United States, the National Fire Incident Reporting System (NFIRS) provides the format for this.[30] Hall (2002c) notes that NFIRS captures nearly a million fires a year, typically close to half of all fires reported to US fire departments.[31] It is not a true random sample, however, so national estimates are calibrated with the NFPA annual survey of fire departments, which is a stratified random sample.[31] National fire statistics of this kind supply the regional and national loss experience on which, according to Denlinger (2008), departments must rely to set their standards.[10]
Within a department, Cline (2008) recommends a single, centralised database with a designated custodian as the only source of published performance information.[32] Even a simple measure such as the number of firefighters on duty can differ if one unit counts people and another counts positions, including vacancies.[33] Data entry errors in dispatch and incident systems can skew reports, so operational managers should review the data before public release.[32] Cline (2008) suggests a monthly reporting cycle for operational managers and a six-month cycle for senior managers.[34] A performance report contains three parts: program descriptors, performance information and planning objectives.[2] The roles in this system follow the levels of the fire department organisation, from the crews who deliver services to the managers who set policy.[7]
Use in risk analysis and fire engineering
Measured performance of fire brigades is also an input to engineering analysis. In fire probabilistic risk assessment for US nuclear power plants, earlier analyses used either the historical duration of fires or fire brigade response times as shown in unannounced fire drills.[35] NUREG/CR-6850 combines both: the brigade response is assessed from the scenario-specific response time and from suppression time curves derived from fire event data.[35] Its worked example uses a brigade response time of 7 minutes taken from fire drill records.[36] With detection after 1 minute and target damage at 15 minutes, 7 minutes remain for manual suppression. The probability that the brigade then fails to suppress the fire follows from the suppression curve:[36][37]
| Symbol | Description | Unit |
|---|---|---|
| Probability that the fire is not suppressed in time | – | |
| Suppression rate for the fire type, from fire event data | min−1 | |
| Time available for manual suppression | min | |
| Time to damage of the target | min | |
| Time to detection | min | |
| Fire brigade response time | min |
For the 2005 electrical-fire example with = 0.12 per minute, = exp(−0.12 × 7) ≈ 0.43.[36] A 2010 supplement revised this method so that the brigade response time is already included in the distribution of suppression times and is no longer subtracted separately.[37] In Australia, the Fire Safety Verification Method handbook (2020) uses a related approach for buildings. If a distribution of fire brigade intervention times is not used, a slow response must be assumed, typically the 95th to 99th percentile of the time to application of water.[38] The revised nuclear method uses a distribution of suppression times that includes brigade response; the Australian handbook instead discusses a distribution of intervention times or a slow-response percentile.[37][38]
See also
- Public Protection Classification
- Fire company staffing
- Fire station location
- Fire-based emergency medical services
- Fire and life safety education
Notes
- ↑ 1.0 1.1 Cline (2008), p. 12-23.
- ↑ 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 Cline (2008), p. 12-35.
- ↑ 3.0 3.1 3.2 3.3 3.4 Krakeel (2008), p. 13-157.
- ↑ 4.0 4.1 4.2 4.3 Johnson and Price (2008), p. 12-220.
- ↑ 5.0 5.1 Denlinger (2008), p. 12-70.
- ↑ 6.0 6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8 Hall (2008), p. 5-109.
- ↑ 7.0 7.1 7.2 Cline (2008), p. 12-26.
- ↑ 8.0 8.1 8.2 8.3 Cline (2008), Table 12.2.1, p. 12-27.
- ↑ Krakeel (2008), p. 13-152.
- ↑ 10.0 10.1 10.2 10.3 Denlinger (2008), p. 12-71.
- ↑ Granito (2008), p. 12-18.
- ↑ Granito (2008), p. 12-20.
- ↑ Cline (2008), p. 12-34.
- ↑ 14.0 14.1 Cline (2008), p. 12-29.
- ↑ Duggan (2008), p. 12-82.
- ↑ 16.0 16.1 16.2 16.3 16.4 Cline (2008), p. 12-37.
- ↑ 17.0 17.1 17.2 17.3 Cline (2008), p. 12-39.
- ↑ 18.0 18.1 Cline (2008), p. 12-38.
- ↑ 19.0 19.1 Krakeel (2008), p. 13-155.
- ↑ Johnson and Price (2008), p. 12-217.
- ↑ 21.0 21.1 Johnson and Price (2008), p. 12-218.
- ↑ Johnson and Price (2008), p. 12-219.
- ↑ 23.0 23.1 23.2 23.3 Cline (2008), Table 12.2.2, p. 12-36.
- ↑ 24.0 24.1 Johnson and Price (2008), p. 12-221.
- ↑ Johnson and Price (2008), p. 12-222.
- ↑ 26.0 26.1 26.2 Granito (2008), p. 12-7.
- ↑ 27.0 27.1 Granito (2008), p. 12-15.
- ↑ 28.0 28.1 Hall and Weller (2008), Table 5.8.1, p. 5-120.
- ↑ Denlinger (2008), p. 12-73.
- ↑ Ahrens and Stewart (2008), p. 3-44.
- ↑ 31.0 31.1 Hall (2002c), p. 1-201.
- ↑ 32.0 32.1 Cline (2008), p. 12-32.
- ↑ Cline (2008), p. 12-33.
- ↑ Cline (2008), pp. 12-34–12-35.
- ↑ 35.0 35.1 Nowlen et al. (2005a), p. 3-5.
- ↑ 36.0 36.1 36.2 Nowlen et al. (2005b), p. P-14.
- ↑ 37.0 37.1 37.2 Hyslop et al. (2010), p. 14-2.
- ↑ 38.0 38.1 Australian Building Codes Board (2020b), p. 104.
References
- Ahrens, M., & Stewart, S. (2008). Fire data collection and databases. In A. E. Cote, C. C. Grant, J. R. Hall, Jr., & R. E. Solomon (Eds.), Fire protection handbook (20th ed., pp. 3-43–3-59). National Fire Protection Association.
- Australian Building Codes Board. (2020b). Handbook: Fire Safety Verification Method (Print version 1.1).
- Cline, D. (2008). Organizational benchmarking and performance evaluation. In A. E. Cote, C. C. Grant, J. R. Hall, Jr., & R. E. Solomon (Eds.), Fire protection handbook (20th ed., pp. 12-23–12-40). National Fire Protection Association.
- Denlinger, R. F. (2008). Managing fire-rescue departments. In A. E. Cote, C. C. Grant, J. R. Hall, Jr., & R. E. Solomon (Eds.), Fire protection handbook (20th ed., pp. 12-51–12-78). National Fire Protection Association.
- Duggan, B. P. (2008). Information management and computer technology. In A. E. Cote, C. C. Grant, J. R. Hall, Jr., & R. E. Solomon (Eds.), Fire protection handbook (20th ed., pp. 12-79–12-97). National Fire Protection Association.
- Granito, J. A. (2008). Planning for public fire-rescue protection. In A. E. Cote, C. C. Grant, J. R. Hall, Jr., & R. E. Solomon (Eds.), Fire protection handbook (20th ed., pp. 12-3–12-22). National Fire Protection Association.
- Hall, J. R., Jr. (2002c). Statistics. In P. J. DiNenno, D. Drysdale, C. L. Beyler, W. D. Walton, R. L. P. Custer, J. R. Hall Jr., & J. M. Watts Jr. (Eds.), SFPE handbook of fire protection engineering (3rd ed., pp. 1-193–1-202). National Fire Protection Association; Society of Fire Protection Engineers.
- Hall, J. R., Jr. (2008). Using data for public education planning and decision making. In A. E. Cote, C. C. Grant, J. R. Hall, Jr., & R. E. Solomon (Eds.), Fire protection handbook (20th ed., pp. 5-103–5-118). National Fire Protection Association.
- Hall, J. R., Jr., & Weller, M. W. (2008). Evaluation techniques for fire and life safety education. In A. E. Cote, C. C. Grant, J. R. Hall, Jr., & R. E. Solomon (Eds.), Fire protection handbook (20th ed., pp. 5-119–5-137). National Fire Protection Association.
- Hyslop, J. S., Kratchman, J., Siu, N., Afshar, C., Najafi, B., Joglar, F., Li, Y., Henneke, D., & Warner, M. (2010). Fire probabilistic risk assessment methods enhancements: Supplement 1 to NUREG/CR-6850 and EPRI 1011989 (NUREG/CR-6850, Supplement 1; EPRI 1019259). Electric Power Research Institute; U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research.
- Johnson, R., & Price, M. (2008). GIS for fire station locations and response protocols. In A. E. Cote, C. C. Grant, J. R. Hall, Jr., & R. E. Solomon (Eds.), Fire protection handbook (20th ed., pp. 12-215–12-231). National Fire Protection Association.
- Krakeel, J. J. (2008). Emergency medical services and the fire department. In A. E. Cote, C. C. Grant, J. R. Hall, Jr., & R. E. Solomon (Eds.), Fire protection handbook (20th ed., pp. 13-145–13-161). National Fire Protection Association.
- Nowlen, S. P., Najafi, B., Wyant, F. J., Forester, J., Kazarians, M., Kolaczkowski, A., Joglar, F., Funk, D., Hannaman, G., & Anoba, R. (2005a). EPRI/NRC-RES fire PRA methodology for nuclear power facilities: Vol. 1. Summary and overview (NUREG/CR-6850; EPRI 1011989). Electric Power Research Institute; U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research.
- Nowlen, S. P., Najafi, B., Wyant, F. J., Forester, J., Kazarians, M., Kolaczkowski, A., Joglar, F., Funk, D., Hannaman, G., & Anoba, R. (2005b). EPRI/NRC-RES fire PRA methodology for nuclear power facilities: Vol. 2. Detailed methodology (NUREG/CR-6850; EPRI 1011989). Electric Power Research Institute; U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research.