Program Details

Practice Type:	Promising
Program Name:	Capacity Assessment Tool for Public Health Preparedness Planning
Organization:	Institute for Community Health
Web site:
Overview:	The Institute for Community Health (ICH) has been working with the Cambridge Public Health Department’s Advance Practice Center (APC) for Emergency Preparedness since 2004 to evaluate its efforts in developing best practices for local public health emergency preparedness planning, including assessing its progress toward goals. The Capacity Assessment Tool was developed in collaboration with the Cambridge APC staff and conducted on annually with public health officials in each of the 27 communities that make up Emergency Preparedness Region 4b. The tool has provided APC staff with a systematic way to assess progress toward emergency preparedness planning goals and evaluate the quality of specific practices. The results of the annual assessments are used by local public health officials and APC staff to identify strengths and gaps in departmental plans, staff training and skills, infrastructure, and relationships among local public health officials, local government and first responders. The tool has also provided APC staff and the regional coordinator with historical data that have been used to advocate for local public health funding. The Capacity Assessment Tool provides data to inform progress toward preparedness planning goals and information to inform future strategic planning and activities to support future emergency preparedness goals.
Year Submitted:	2009
Responsiveness and Innovation:	After years of decreased funding for public health, an increase in responsibility for all-hazards planning has stretched the capacity of local public health departments (LPHDs) to plan for these and other health emergencies. LPHDs are recognized as playing critical roles in response to biological threats, including surveillance of unusual health events, dissemination of treatment protocols for communicable diseases, and implementation of risk communication plans. LPHDs in Massachusetts are required by the state and the CDC to develop and exercise plans for responding to biological and other public health hazards, develop surveillance systems to monitor infectious diseases, improve risk communications within and across communities, and educate and train the public health workforce. In 2004, many local public health departments had limited to no experience with emergency preparedness planning. Although state departments of public health were receiving funding to support the development of local public health capacity and infrastructure for emergency response, the learning curve for local, state, and federal public health officials was steep. The requirement for LPHDs to have plans and exercise them often does not include any component of evaluation or assessment, thus making it impossible for communities to know where they stand in their planning. The development of a Capacity Assessment Tool emerged as a key strategy to assess progress toward annual local emergency preparedness goals, identify strengths and gaps in preparedness planning and capacity, and inform strategic planning for future years. The results of the capacity assessment are available both to the participating communities and the regional leadership. For participating communities, the results provide a snapshot of where they are in the process of developing comprehensive plans for response during public health emergencies and building the capacity to respond. For regional leaders, the results enable them to see how well individual communities and the region as a whole are meeting deliverables and goals. Both local communities and regional leaders have used the results to identify priority areas for support in meeting expectations, capacity building, and workforce development. Finally, the capacity assessment tool has provided a mechanism to assess the innovative approaches that the Cambridge APC has used to help LPHDs meet complex and often abstract goals. A persistent challenge faced by federal, state, and local public health leaders has been defining what public health emergency preparedness is and how preparedness can be measured and assessed. Absent clear definitions and adequate performance metrics, it is difficult to assess the effectiveness of past investments, engage in continuous quality improvement of current efforts, or design and target future efforts (Nelson C., Lurie N., Wasserman J. (2007). Assessing Public Health Emergency Preparedness: Concepts, Tools, and Challenges. Annual Review of Public Health 28: 1–17). Using an annual survey to measure progress toward program or departmental goals and objectives is a relatively standard practice used by evaluators. However, at the time that the Cambridge APC was funded in 2004, few local public health departments or regions were using the practice to systematically assess progress toward emergency preparedness goals. This remains true today. One of the challenges faced by local and regional public health leaders has been that most of the national performance measures in this area were developed for state or large regional departments of public health. Debates on how best to measure preparedness at the local level are plentiful and not well-represented in the published literature. The CDC and HRSA (now ASPR) have been primarily responsible for distributing funds to build emergency readiness capacity and measuring performance in meeting goals and objectives. Since 2002, CDC’s and HRSA’s performance measurements have evolved from measuring capacity to assessing capability. Early in their programs, both agencies used markers or values that they called benchmarks to measure capacity-building efforts, such as purchasing equipment and supplies and acquiring personnel. Over time, the CDC has further developed performance measures to focus on measuring capabilities, such as training for the public health workforce on plans and essential equipment. Although important improvements in the measurement of readiness planning and capacity, the current metrics are not adequate to capture the full range of activities that local public health departments need to be prepared for public health emergencies. These activities include, but are not limited to improvements in the relationships between local public health officials and traditional first responders; participation of local public health officials in local emergency planning bodies; development and regular review and updating of emergency response plans; and communications between local and state public health staff, such as between public health nurses and state epidemiologists. The Capacity Assessment Tool was developed in the absence of national standards and metrics and reflects the goals that one APC hypothesized as being important in building local public health emergency preparedness capacity. Use of the Capacity Assessment Tool has helped the Cambridge APC assess its approach to developing local public health capacity for emergency preparedness and response and rethink some of their initial hypotheses about how best to help support capacity building, workforce training, infrastructure development, and relationship building.
Agency and Community Roles:	The initial instrument was developed as a collaborative effort between the evaluation team and various professionals within the Cambridge Public Health Department, including the lead public health nurse, the head of environmental inspection services, the regional emergency preparedness coordinator, and an epidemiologist. The Capacity Assessment Tool was developed with significant input from Cambridge APC and other local public health staff from 27 communities in Region 4b. Modifications to the instrument have occurred over time with input from the Cambridge APC staff and representatives from the 27 local public health departments that are in Region 4b. Input on changes to the instrument, administration strategy or reporting format are gathered at least annually through one-on-one meetings with select staff and during dissemination meetings with a wide spectrum of public health representatives.
Costs and Expenditures:	The costs associated with the development and annual implementation of the tool have decreased since its implementation in 2004. The first year was the most labor intensive and required the effort of one .5 FTE Senior Evaluator for two months to develop the instrument with APC staff (approximately three staff at .10 FTE for two months). Once the instrument was finalized, a database was developed (approximately .20 FTE for one month). Two members of the evaluation team spent approximately three months scheduling, conducting, and entering the data from the capacity assessment survey (approximately 1.25 FTE for three months). Finally, a data analyst spent one month analyzing the data and formatting the results for presentation to local public health officials. In all, the cost of the assessment during the first year was approximately $12,500. This does not include the in-kind costs provided by local public health officials in each city or town. These individuals contributed approximately 1.5 hours of time each year to the assessment process (1.5 hours * 27 = 40.5 in-kind hours from local public health officials). Each year the instrument has been modified to reflect shifts in terminology, addition of practices or activities, and recommended improvements based on the previous years’ experience. Although the staffing pattern remains roughly the same, the time it takes to update, conduct, and analyze the tool has decreased. We anticipate that currently the instrument costs approximately $7,000–$8,000 each year to conduct. However, the costs could be decreased even more if the instrument was self-administered using an online format. This is an option we are exploring for future use.
Implementation:	After years of decreased funding for public health, an increase in responsibility for all-hazards planning has stretched the capacity of local public health departments (LPHDs) to plan for these and other health emergencies. LPHDs are recognized as playing critical roles in response to biological threats, including surveillance of unusual health events, dissemination of treatment protocols for communicable diseases, and implementation of risk communication plans. LPHDs in Massachusetts are required by the state and the CDC to develop and exercise plans for responding to biological and other public health hazards, develop surveillance systems to monitor infectious diseases, improve risk communications within and across communities, and educate and train the public health workforce. In 2004, many local public health departments had limited to no experience with emergency preparedness planning. Although state departments of public health were receiving funding to support the development of local public health capacity and infrastructure for emergency response, the learning curve for local, state, and federal public health officials was steep. The requirement for LPHDs to have plans and exercise them often does not include any component of evaluation or assessment, thus making it impossible for communities to know where they stand in their planning. The development of a Capacity Assessment Tool emerged as a key strategy to assess progress toward annual local emergency preparedness goals, identify strengths and gaps in preparedness planning and capacity, and inform strategic planning for future years. The results of the capacity assessment are available both to the participating communities and the regional leadership. For participating communities, the results provide a snapshot of where they are in the process of developing comprehensive plans for response during public health emergencies and building the capacity to respond. For regional leaders, the results enable them to see how well individual communities and the region as a whole are meeting deliverables and goals. Both local communities and regional leaders have used the results to identify priority areas for support in meeting expectations, capacity building, and workforce development. Finally, the capacity assessment tool has provided a mechanism to assess the innovative approaches that the Cambridge APC has used to help LPHDs meet complex and often abstract goals. A persistent challenge faced by federal, state, and local public health leaders has been defining what public health emergency preparedness is and how preparedness can be measured and assessed. Absent clear definitions and adequate performance metrics, it is difficult to assess the effectiveness of past investments, engage in continuous quality improvement of current efforts, or design and target future efforts (Nelson C., Lurie N., Wasserman J. (2007). Assessing Public Health Emergency Preparedness: Concepts, Tools, and Challenges. Annual Review of Public Health 28: 1–17). Using an annual survey to measure progress toward program or departmental goals and objectives is a relatively standard practice used by evaluators. However, at the time that the Cambridge APC was funded in 2004, few local public health departments or regions were using the practice to systematically assess progress toward emergency preparedness goals. This remains true today. One of the challenges faced by local and regional public health leaders has been that most of the national performance measures in this area were developed for state or large regional departments of public health. Debates on how best to measure preparedness at the local level are plentiful and not well-represented in the published literature. The CDC and HRSA (now ASPR) have been primarily responsible for distributing funds to build emergency readiness capacity and measuring performance in meeting goals and objectives. Since 2002, CDC’s and HRSA’s performance measurements have evolved from measuring capacity to assessing capability. Early in their programs, both agencies used markers or values that they called benchmarks to measure capacity-building efforts, such as purchasing equipment and supplies and acquiring personnel. Over time, the CDC has further developed performance measures to focus on measuring capabilities, such as training for the public health workforce on plans and essential equipment. Although important improvements in the measurement of readiness planning and capacity, the current metrics are not adequate to capture the full range of activities that local public health departments need to be prepared for public health emergencies. These activities include, but are not limited to improvements in the relationships between local public health officials and traditional first responders; participation of local public health officials in local emergency planning bodies; development and regular review and updating of emergency response plans; and communications between local and state public health staff, such as between public health nurses and state epidemiologists. The Capacity Assessment Tool was developed in the absence of national standards and metrics and reflects the goals that one APC hypothesized as being important in building local public health emergency preparedness capacity. Use of the Capacity Assessment Tool has helped the Cambridge APC assess its approach to developing local public health capacity for emergency preparedness and response and rethink some of their initial hypotheses about how best to help support capacity building, workforce training, infrastructure development, and relationship building.
Sustainability:	Assessment of public health emergency planning will always be critical to its success. The Capacity Assessment Tool is developed and both the instrument and Access database are available to local public health officials. Evaluation instruments are more likely to be sustained beyond the life of any particular grant if the results they yield are useful for practitioners. The results from the Capacity Assessment Tool have been used each year to support continuation funding for the APC project. The results have also been used in various other grant applications to support additional emergency preparedness activities. Although the practice itself can be sustained with limited funds, the results of the practice also help sustain local emergency planning and capacity building efforts. Thus, the practice of using an annual assessment tool helps guide and sustain local public health efforts in this area. LPHDs may modify the instrument to incorporate additional goals and objectives. Such modifications would require changes to the Access database. LPHDs may have the capacity in-house to make these changes. If not, some small amount of funding may be required to support database modifications. The more cost-intensive aspect of the Capacity Assessment Tool in its current form is the analysis. Many LPHDs do not have the capacity to analyze survey data. They may need to hire an analyst to perform data analysis and table the results for dissemination. LPHDs could create an internship opportunity for public health students as one strategy for reducing costs and ensuring that the practice is affordable and sustainable. As noted earlier, the costs of administering the survey could be significantly reduced if it were translated into an online format, such as SurveyMonkey. The advantage that this offers is that the data are entered directly into a database and frequencies are readily available to local public health officials. The challenge of online surveys is that the response rate is typically not as good as if if were administered face-to-face, and the qualitative responses are generally shorter and less clear. However, we believe that LPHDs could easily adapt the instrument for their own use and reduce the costs of administration by using online survey technologies.
Lessons Learned:	Information was not provided in 2009

Back to main page.