THE USE OF OPERATIONS OBJECTIVES AND PERFORMANCE MEASURES IN PRIVATE AND PUBLIC ORGANIZATIONS

White Paper
February 15, 2010

Prepared for:

Federal Highway Administration Federal Transit Administration

FHWA-HOP-10-029

Quality Assurance Statement The Federal Highway Administration (FHWA) provides high-quality information to serve Government, industry, and the public in a manner that promotes public understanding. Standards and policies are used to ensure and maximize the quality, objectivity, utility, and integrity of its information. FHWA periodically reviews quality issues and adjusts its programs and processes to ensure continuous quality improvement.

Technical Report Documentation Page
1. Report No. FHWA-HOP-10-029 4. Title and Subtitle The Use of Operations Objectives and Performance Measures in Private and Public Organizations White Paper 7. Authors Jocelyn Bauer, Kelley Pecheux, Michael Smith, Zachary Fletcher 9. Performing Organization Name and Address Science Applications International Corporation (SAIC) 8301 Greensboro Drive McLean, VA 22102 12. Sponsoring Agency Name and Address United States Department of Transportation Federal Highway Administration 1200 New Jersey Ave., SE Washington, DC 20590 15. Supplementary Notes Mr. Richard Backlund, Federal Highway Administration, COTM 16. Abstract This paper examines lessons and insights from private companies and public organizations that may apply to agencies in the United States working to advance transportation planning for operations using a strategic approach. The use of specific objectives and performance measures to manage operational performance is common practice among self-sustaining private and public organizations that are responsible for generating sufficient revenue to meet costs and, in some cases, produce profit. Based on the information gathered, several useful practices were identified for consideration in transportation planning for operations including developing of a balanced set of objectives and performance measures and assigning weights to objectives based on their impact on customer satisfaction. 17. Key Words Performance management, quality improvement, operations objectives, performance measures, transportation planning, management and operations, private sector. 19. Security Classif. (of this report) Unclassified Form DOT F 1700.7 (8-72) 18. Distribution Statement No restrictions. This document is available to the public from: The National Technical Information Service, Springfield, VA 22161. 21. No of Pages 19 22. Price N/A 8. Performing Organization Report No. 10. Work Unit No. (TRAIS) 11. Contract or Grant No. DTFH61-06-D-00005 13. Type of Report and Period Covered 14. Sponsoring Agency Code 2. Government Accession No. 3. Recipient’s Catalog No. 5. Report Date February 2010 6. Performing Organization Code

20. Security Classif. (of this page) Unclassified

Reproduction of completed page authorized

Introduction
The use of specific objectives and performance measures to manage operational performance is common practice among self-sustaining private and public organizations that are responsible for generating sufficient revenue to meet costs and, in some cases, produce profit. The experiences of these organizations are a rich source of information to consider in the public sector. This paper examines lessons and insights from private companies and public organizations that may apply to agencies in the United States working to advance transportation planning for operations using a strategic approach. In particular, this paper examines the use of measurable objectives and performance measures by private and public organizations to improve service delivery to their customers over physical infrastructure such as toll roads or electrical lines. Information on the use of operations-related objectives and performance measures was gathered on the following organizations:      Federal Express (FedEx) TNT Express Delivery Services Illinois Tollway Austin Energy U.S. Power Company1

Small package delivery companies were researched because they must closely monitor and optimize their operations to provide good service, which enables them to attract and maintain customers in a highly competitive business environment. A tollway organization was included in this research to provide a closer point of comparison for public sector transportation agencies. An examination of electric utilities illustrated the use of objectives and performance measures for service delivery across wires instead of roads. Information for this paper was gathered from interviews with organization representatives and a literature review that focused on the methods, benefits, and challenges associated with:     Setting, communicating, and gaining commitment to measurable objectives and performance measures. Tracking progress. Making adjustments to processes and resource use based on performance information. Adjusting the objectives and performance measures themselves to meet current needs.

1

This company requested that its name not be used in this paper.

1



Demonstrating accountability to shareholders and customers.

Key Insights from Research
Based on the information gathered on private and public organizations that delivered services to customers, the following list of activities was developed for consideration in the use of objectives and performance measures in planning and managing transportation operations among public transportation agencies:  Develop a balanced set of objectives and performance measures. In 1992, Kaplan and Norton2 developed a set of measures known as the “balanced scorecard” during a year-long research project with 12 companies at the leading edge of performance measurement. This scorecard gives top managers a quick but comprehensive view of the business from four important perspectives: financial, customer satisfaction, internal processes, and the organization’s innovation and improvement activities. In the area of transportation operations, the idea of a balanced set of objectives and performance measures highlights the importance of examining operations in diverse ways: external outcomes (system efficiency/reliability), operator activities (internal processes), and customer satisfaction. Develop objectives for different levels or tiers in the organization based on responsibility. Austin Energy’s Electric Service Delivery business area and another U.S. power company interviewed both develop and use objectives based on organizational level. At the highest level of the organization, top-tier management focuses on broad objectives and associated measures that describe how the overall organization is performing. At the lower levels of the organization, the objectives are increasingly specific and related to the responsibilities of the personnel at that level of the organization. This helps employees better understand what is expected of them and how they can contribute to organizational objectives or goals. Assign weights to performance objectives according to their impact on customer satisfaction. FedEx developed a 12-component index known as the Service Quality Indicator and each item is weighted to reflect how significantly it affects overall customer satisfaction. FedEx uses customer satisfaction surveys to update its measures and weights.3 Measures of transportation operations may not all have equal importance or impact on desired objectives. It may be useful to give more weight to some aspects of performance than others to reach objectives more efficiently.





2

Kaplan, R.S. and D.P. Norton. “The Balanced Scorecard—Measures that Drive Performance,” Harvard Business Review, January-February, 1992, pp. 71-19. Morris, D.D. and R.J. Baker, “Measuring What Matters: [Want To Know How Successful Your Firm Is? Take the Customer’s Point of View],” California CPA, August 2003. Available at: http://findarticles.com/p/articles/mi_m0ICC/is_2_72/ai_107755417 (accessed March 2, 2009).

3

2



Set up a team for each objective or performance measure. FedEx set up a cross-functional action team for each component of its Service Quality Indicator. Each team is headed by a senior executive and assures the involvement of employees from all part of the company when needed.4 Likewise, in regional transportation operations cross-functional, cross-agency teams could be developed to focus on making progress and tracking a single operations objective. Communicate performance information regularly to staff. TNT Express Delivery Service in the United Kingdom uses a seven-indicator service performance report that is updated weekly and circulated among each package coordination and collection depot.5 The performance of each depot is indicated on the report and this creates competition among the depots for top performance. Similarly, giving transportation operations decisionmakers and operations staff performance information in a timely manner facilitates fact-based actions and decisions. Ensure objectives have a senior level champion. The package delivery companies of Federal Express (FedEx) and TNT both have senior executives that serve as champions within the organization for the performance measurement system. At FedEx, one senior executive leads an action team on each performance indicator. An assessment of TNT by Moon and Fitzgerald6 attributes the success of a performance management system to push the strategic direction of the company to five properties. One of them is the corporate champion for the performance measurement system. There are frequent corporate messages from the Head Office endorsing the system and attaching great importance to its results. In the area of regional transportation operations, a senior level champion can help to garner resources (staff, funding, etc.) and motivate others for the successful achievement of operations objectives. Maintain a high level of awareness of operational performance. One of the common features of the performance management systems in the organizations examined was a very high level of awareness of the performance of the system. In the case of the Illinois Tollway, the staff regularly tracks data on incident detection, response, and clearance times with time stamps and weekly reports. To manage congestion in construction zones, the Tollway installs sensors prior to construction to establish baseline operational performance; to monitor performance during construction; and, after construction is complete, to see how performance has improved.







4

National Institute of Standards and Technology, “Malcolm Baldrige National Quality Award 1990 Winner Federal Express Corporation,” October 17, 2002. Available at: http://www.quality.nist.gov/FederalExpress_90.htm (accessed February 25, 2009). Moon, P. and L. Fitzgerald. “Delivering the Goods at TNT: the Role of the Performance Measurement System,” Management Accounting Research, Vol. 7, 1996, pp. 431-457. Ibid.

5

6

3

Background
The use of objectives and performance measures to manage and improve the operational performance of an organization has been widely written about in literature in the areas of organizational management, strategic planning, and quality improvement. There are several well-known methodologies for improving organizational performance. The purpose of this section is not to explain these approaches, but instead to highlight ideas from performance management literature that may be instructive or provide context for the use of operations objectives and performance measures in planning for transportation operations. Necessity of a Measures-Based System for Organizational Improvement

Strategic planning helps to achieve the alignment of actions among the members of an organization toward a common set of goals, objectives, and actions. Howell7 notes that alignment helps Source: Marvin T. Howell, Actionable Performance organizations reduce unnecessary Measurement: A Key to Success, American Society or duplicative effort while for Quality, 2006. increasing the opportunities for significant improvement. In the strategic planning process described by Howell, “key result areas” are identified based on the organization’s mission and before the development of goals. The key result areas are the most important areas of improvement for the organization. Howell provides the example of Florida Power and Light and its key result areas of “quality, delivery, cost, safety, and corporate responsibility.” Out of the key result areas, broad, long-range goals are established. Howell recommends that at least one SMART (specific, measurable, actionable, relevant and realistic, and timeframed) objective be identified for each goal. The SMART objectives facilitate the development of performance measures that are used to assess and monitor organizational improvements. Performance measures serve to focus actions and investments on what is important to the organization. Performance measurement aims to support the implementation and monitoring of strategic initiatives. The selection of performance measures and setting targets for these measures are concrete formulations of a company’s strategic choices. Both financial and non-financial measures are needed to translate the strategy into specific objectives that provide guidelines for operational action for middle and lower management.8
7 8

“It is inconceivable that a world-class organization exists without a metrics-based management system. Metrics enable organizations to know where they are, assess the need for improvement, improve, and then monitor processes to ensure that they are under control and producing desired levels of quality.”

Howell, M.T. Actionable Performance Measurement: A Key to Success, American Society for Quality, 2006. Lohman, C., L. Fortuin, and M. Wouters, “Designing a performance measurement system: A case study,” European Journal of Operational Research, Vol. 156, Issue 2, 2004, pp. 267-286.

4

As noted by Lohman, the literature on performance measurement in operations describes different methods for developing performance measurement systems; however, a characteristic of many of the methods is the focus on developing performance metrics and a measurement system that is based on the company’s strategy and processes.

Malcolm Baldrige National Quality Award: Recognizing Organizations for Strategic Improvement Programs The Malcolm Baldrige National Quality Award was created in 1987 to help improve the quality and productivity of U.S. companies by recognizing those that have improved the quality of their goods and services and to advance the quality improvement efforts of organizations across private industry and the government through specific criteria and guidelines.9 The law authorizing this award recognizes that strategic planning for quality and quality improvement programs are necessary to compete in a global marketplace.10 Two of the core values, as outlined by Mark Graham Brown,11 that underlie the Baldrige Criteria for Performance Excellence are “management by fact” and “focus on results and creating value.” These values emphasize the importance of using outcomes-oriented objectives and performance measures to plan and manage an organization for improved performance. Baldrige Award winners are expected to have a systematic process for collecting data on a balanced set of performance measures and using that data to make decisions. The core value of “focus on results and creating value” reflects the need to manage both for quality in products and services as well as for profit. The 3rd Edition of Operations Management proposes five key performance objectives by which an operation can be evaluated:  “Quality – doing things right”  “Speed – doing things fast”  “Dependability – doing things on time”  “Flexibility – being able to change what you do”  “Cost – doing things cheaply”
Source: Slack, N., S. Chambers, and R. Johnson. Operations Management, 3rd Edition. FT Prentice Hall (2001).

9

The Malcolm Baldrige National Quality Improvement Act of 1987 - Public Law 100-107. Available at: http://www.baldrige.nist.gov/Improvement_Act.htm (accessed October 27, 2009). Also see National Institute of Standards and Technology, Baldrige National Quality Program. Available at: http://www.baldrige.nist.gov/index.html (accessed October 27, 2009). Ibid.

10 11

Brown, Mark Graham (2006). Baldrige Award Winning Quality: How to Interpret the Baldrige Criteria for Performance Excellence, 15th Edition. New York, NY: Productivity Press.

5

A Balanced Scorecard: A Comprehensive View of Performance During a year-long research project with 12 companies at the leading edge of performance measurement, Kaplan and Norton (1992) devised a set of measures, known as the “balanced scorecard,” that gives top managers a quick but comprehensive view of the business. The balanced scorecard allows managers to look at the business from four important perspectives. It includes financial measures as well as three operational measures that are the drivers of future financial performance: customer satisfaction, internal processes, and the organization’s innovation and improvement activities. As Kaplan and Norton (1992) explain: “…the scorecard brings together, in a single management report, many of the seemingly disparate elements of a company’s competitive agenda: becoming customer oriented, shortening response time, improving quality, emphasizing teamwork, reducing new product launch times, and managing for the long term.” In consideration of the customer’s perspective, Kaplan and Norton (1992) recommend that companies articulate goals for time, quality, performance, and service and then translate these goals into specific measures (e.g., on-time delivery). The internal measures for the scorecard should stem from the business processes that have the greatest impact on customer satisfaction (i.e., factors that affect cycle time, quality, employee skills, and productivity); measures that are influenced by employee actions. Measures related to a company’s innovation and improvement activities should describe the ability of the company to make continual improvements to its existing products and processes as well as its ability to introduce new products with expanded capabilities (e.g., time to develop next-generation product, process time to maturity). Kaplan and Norton (1992) note that as companies began applying the balanced scorecard, there was recognition that it represented a fundamental change in the underlying assumptions about performance measurement at that time. As those involved in the research project took the concept back to their organizations, they found that they could not implement the balanced scorecard without the involvement of the senior managers who have the most complete picture of the company’s vision and priorities. This was revealing because most existing performance measurement systems had been designed and overseen by financial experts.

6

Case Studies
This section provides a brief look at two delivery companies, a tollway organization, and two electric utilities. Each organization delivers a service over a physical infrastructure and closely monitors its respective operations and resulting customer service to maintain and expand its customer base. FEDERAL EXPRESS Prior to 1989, Federal Express (FedEx) assumed that on-time delivery was what its customers expected and valued most; however, input from customers showed that customers expected much more.12 In an effort to spur progress toward their ultimate target of 100 percent customer satisfaction, FedEx developed a 12-component index, known as the Service Quality Indicator (SQI). Each item in the SQI describes work process failures, and each is weighted to reflect how significantly it affects overall customer satisfaction. The SQI includes the following components / performance indicators, along with their weighting factors (shown in parentheses):13 – – – – – – – – – – – – Right day late service failures (1) Wrong day late service failures (5) Traces (1) Complaints reopened by customers (5) Missing proofs of delivery (PODS) (1) Invoice adjustments requested (1) Missed pick-ups (10) Lost packages (10) Damaged packages (10) Overgoods (5) Abandoned calls (1) International SQI indicator (1)

FedEx uses automated tracking systems to gather and track data. Rapid analysis of operations data yields daily SQI reports transmitted to workers at all FedEx sites. Management meets daily to discuss the previous day's performance, and weekly, monthly, and annual trends are tracked. Quality action teams (QAT) analyze data

12

Labovitz, G., Y.S. Chang, and V. Rosansky. Making Quality Work: A Leadership Guide for the Results-Driven Manager, John Wiley and Sons, 1994. Morris, D.D. and R.J. Baker, “Measuring What Matters: [Want To Know How Successful Your Firm Is? Take the Customer’s Point of View],” California CPA, August 2003. Available at: http://findarticles.com/p/articles/mi_m0ICC/is_2_72/ai_107755417 (accessed March 2, 2009).

13

7

contained in the company's major databases to identify the root causes of problems that surface in the SQI reviews. 14 To reach its aggressive quality goals, the company has set up one cross-functional team for each service component in the SQI. A senior executive heads each team and assures the involvement of front-line employees, support personnel, and managers from all parts of the corporation when needed.15 Two of these corporate-wide teams have a network of over 1,000 employees working on improvements.16 One of the ways in which FedEx gains commitment to its performance objectives is through performance support. FedEx encourages its employees to be innovative and to make decisions that advance quality goals and also provides employees with the information and technology they need to continuously improve their performance. Examples include the hand-held Tracker and the Digitally Assisted Dispatch System (DADS). The hand-held Tracker records activity information throughout the day and guides couriers through a series of performance measurements as they close out their day’s activities.17 The DADS communicates to some 30,000 couriers through screens in their vans. The system enables quick response to pick-up and delivery dispatches and allows couriers to manage their time and routes with high efficiency.18 In addition, the SQI measurements are directly linked to the corporate planning process, which begins with the CEO and COO and an executive planning committee. SQIs form the basis on which corporate executives are evaluated and individual performance objectives are established and monitored. Performance of the whole corporation in meeting performance improvement goals determines executive bonuses. If employees do not rate management leadership at least as high as they rated them the year before in the annual employee survey, no executive receives a year-end bonus.19 While the SQI measures internal process performance, FedEx relies on a customer satisfaction survey to measure satisfaction from the customer’s perspective. Not only can the customer satisfaction survey capture aspects of service quality that the SQI does not include, it can also capture the changing expectations of customers. This allows FedEx to recheck customer requirements and perceptions and to update its measures and weights accordingly. This ensures that the customer’s voice always drives FedEx’s actions and processes.20 The customer satisfaction survey consists of a quarterly telephone survey, a targeted customer survey, FedEx comment cards, a customer automation survey, and a Canadian customer survey.
14 15 16 17 18 19 20

National Institute of Standards and Technology (2002). Ibid. Ibid. Keyes, J. Implementing the IT Balanced Scorecard: Aligning IT with Corporate Strategy, CRC Press, 2005. National Institute of Standards and Technology (2002). Ibid. Howell (2006).

8

Since being placed in service in the late 1980s, the SQI has enabled FedEx to increase its on-time delivery performance from 95 percent to 99.7 percent in 2003 without adding significant costs.21 FORTUNE has ranked FedEx among the Global Most Admired Companies and America's Most Admired Companies lists since 2002 and 2001, respectively. The company has also been on the list of FORTUNE magazine's "100 Best Companies to Work For" for 12 of the past 13 years.22 The connection between what the company measures and rewards and their industry dominance is solidly linked.23 TNT EXPRESS DELIVERY SERVICES TNT provides global express delivery services, including parcels and freight. Moon and Fitzgerald24 conducted an in-depth study on the role of performance measurement at TNT Express Delivery Services in the United Kingdom (UK). This case study summarizes the relevant findings of that research. The operational system adopted by the company is structured like a giant wheel, with a central hub and a set of spokes. On the outer rim, there are 28 depots strategically situated around the country. On weekdays, each depot is responsible for the coordination and collection of all packages being sent by customers in their territory. These packages are sorted at the depot and those being sent outside of the territory are packed on trucks and sent to the hub. Departure times from and arrival times at the hubs are coordinated centrally in an extensive computerized scheduling exercise. After reaching the hub, packages are mechanically sorted by destination. Then the reverse process takes place with trucks being loaded with packages for delivery in each region by the morning delivery deadlines. Responsibility for the smooth operation of this system falls on the General Manager for network operations at the hub and on the 28 depot management teams at the depots. TNT’s primary objective is to deliver the packages to the right place at the right time, and TNT employs a variety of metrics to measure and track its operational performance. At the hub level, the company monitors the “end-of-sort” on a daily basis against a predetermined target. Updated predictions are made throughout the night and act as an early warning system indicating that the end-of-sort time may be delayed. At the depot level there are four separate performance measurement mechanisms which relate to:  
21 22

The depot overall. Sales and customer care.

Morris and Baker (2003).

Forbes.com, “FedEx Named Among FORTUNE Magazine's 2010 ‘Best Companies to Work For,” January 21, 2010. Available at: http://www.forbes.com/feeds/businesswire/2010/01/21/businesswire134307423.html (accessed February 15, 2010). National Institute of Standards and Technology (2002). Moon and Fitzgerald (1996), pp. 431-457.

23 24

9

 

Deliveries. Finance and administration.

Delivery performance is perceived to be fundamental to the success of TNT, and TNT has a clearly defined method for measuring its delivery performance. It does this using what it calls the Seven-Star Service Performance Report, which is measured weekly. The seven indicators or performance measures included on the report include:        Percent delivery on time. Failures. Percent deliveries that result in credit notes or that are unmatched with invoices. Percent misroutes. Number of late trucks (trucks arriving late at the hub). Loss claims as a percent of revenue. Damage claims as a percent of revenue.

A standard target is set for each of the indicators, and any depot achieving the target or better across all categories would gain a seven star rating for the week. Reports are generated weekly in the form of a league table, which ranks orders at the depots according to delivery on-time performance. Reports are circulated so that all depots know how they performed overall and in comparison to the other depots. Based on an assessment by Moon and Fitzgerald (1996), TNT has been successful at using its performance management system to push the strategic direction of the company into all aspects of its operations. Moon and Fitzgerald (1996) identify five properties of TNT’s performance management system that have allowed the company to do this:  Measuring the right things. The objectives and performance measures are well understood and communicated throughout the organization. The specific measures cover a range of dimensions thought to be central to corporate success. Key issues are translated into detailed action plans so that every individual is aware of their role and the requirements of that role. Internal benchmarking. Internal benchmarking is used by TNT to provide sets of absolute standards that the depots are expected to maintain, and there is a continual push from central management to reach the standards. Reward mechanisms. Incentive schemes are used throughout the business, linking the achievement of company targets with financial rewards. Each functional area at the depots has a bonus scheme that focuses on the key performance measures for that function.





10



League tables. League tables, generated weekly, display each depot’s performance relative to the others. These tables encourage competition between the depots. Corporate champion. There is a constant driving down of the corporate message from the Head Office. They believe in the performance measurement system and attach great importance to the results and “getting the service level right.”



ILLINOIS TOLLWAY The Illinois Tollway operates 286 miles of highway and serves approximately 1.3 million vehicles a day, about 40 percent of which is commercial vehicle traffic. The Tollway’s mission is to provide and promote a safe and efficient system of toll-supported highways while ensuring the highest possible level of service to its customers. To provide customers with a premium ride, the Tollway relies on a top-down support system that it considers critical to its overall performance and delivery of service. The Tollway began developing and monitoring performance in the early 1980s when traffic volumes were growing, and it started seeing congestion at its toll plazas. It also began looking at incidents that were resulting from the increased congestion. In the late 1980s to early 1990s, the Tollway began a concerted effort to manage traffic over the entire network, and this has evolved over time. It began with the use of a computer-aided dispatch (CAD) system and then started using an integrated approach that involved using dedicated police and maintenance staff to keep things moving. The Tollway created procedures and systems that were later integrated into a centralized traffic management center. The Illinois Tollway has worked to measure, monitor, and improve its performance in four key operations areas: overall traffic operations, toll collection, incident response, and construction. Overall Traffic Operations The Tollway’s overall objective is to minimize disruption to customers, and thus the Tollway places a high premium on mitigating congestion. Since the early 2000s, the Tollway has focused on measuring how the system is operating and performing (e.g., locations of back-ups, slow-downs, and segment travel times). Performance measures for overall traffic operations include traffic volumes, speed, travel times, and queue length. While there are stated performance targets related to congestion, the Tollway has established levels of tolerance that have been adjusted over time to better meet needs. For example, the tolerance threshold for queue length has been cut in half. Toll Collection Key performance measures for toll collection operations include congestion, number of incidents, and the percent of customers using I-PASS (electronic toll collection). Based on Tollway customer surveys, the Tollway learned that its customers place a high value on convenience. As a result, the Tollway began converting some lanes to low-speed IPASS only, and after receiving support from the governor in 2004, it made the decision to convert the entire system to open road tolling (ORT). As a result of the ORT, the 11

Tollway has eliminated congestion associated with toll plazas, and crashes have dropped dramatically. Incident Response The Tollway’s objective is to clear incidents as soon as possible. Key performance measures for incident response include detection, response, and clearance times. The Tollway tracks these measures regularly with time stamps and weekly reports and uses a computer aided dispatch system integrated with a traffic incident management system to facilitate the process. The Tollway has one police agency that handles its entire system. Crashes involving property damage are routinely cleared in less than 20 minutes, even inside work zones. Crashes requiring the involvement of the fire department are routinely cleared in 30 to 40 minutes. The Tollway also audits and monitors tow activity. The agency has 45 agreements with private firms that specify required response times. Internally, the Tollway communicates and gains commitment to improved incident response performance through training and awareness. All maintenance employees are trained in incident response and in 60 percent of incidents, Tollway employees are the first to respond. Externally, the Tollway gains commitments to performance through formal agreements that establish a framework for cooperation. The agency produces quarterly reports for the Governor’s office regarding non-recurrent congestion. Construction The Tollway’s focus is on knowing how its system is operating during construction. Key performance measures for operations during construction include traffic volumes, congestion, average travel times, and number of incidents. The Tollway installs sensors prior to construction to establish a baseline, measure performance during construction, and quantify improvements after construction is complete. Because of performance monitoring, Tollway operators know where back-ups are going to form, enabling the agency to get the word out to its customers. The Tollway management strives for a high level of communication with the public. It notifies the public a minimum of 10 days prior to construction or any major phase change, provides reminders, and then disseminates information during construction. ELECTRIC UTILITY COMPANIES This section highlights the use of objectives and performance measures at two electric utility companies across the United States. The first company featured wishes to remain anonymous and will be referred to as a “U.S. Power Company.” Information is also presented from Austin Energy of Texas. Both utility companies use worker safety and service reliability as primary categories of operational performance management. Generally, the companies use the standard industry performance measures for system reliability: System Average Interruption Duration Index (SAIDI) and System Average 12

Interruption Frequency Index (SAIFI). SAIDI is a measure of the average outage duration for each customer served whereas SAIFI is the average number of interruptions in service that a customer would experience typically over the course of a year. Austin Energy Austin Energy is a community-owned electric utility located in Austin, Texas. The utility provides a portion of its profits each year to help fund city services. Austin Energy produces and delivers energy to approximately 400,000 customers. It is the 9th largest municipal electric company in the United States. Austin Energy generates power through coal, gas-fired, and nuclear plants, captures renewable wind energy, and purchases energy from outside providers. Austin Energy also operates a transmission and distribution system.25 Austin Energy has adopted three overarching strategies to maintain a successful organization. In support of these strategies, Austin Energy developed five strategic objectives. The strategies and strategic objectives are as follows:26   Strategy: Risk Management. o Objective: Maintain Financial Integrity. Strategy: Excellent Customer Service. o Objective: Create and Sustain Economic Development. o Objective: Customer Satisfaction. o Objective: Exceptional System Reliability. Strategy: Energy Resource. o Objective: Renewable Portfolio Standard and Energy Efficiency.



Each objective is tracked using one or more performance measures and associated performance targets. Austin Energy measures system reliability with six reliability performance measures. Three measures focus on the delivery of electric services and the other three measures focus on power production. The performance measures that Austin Energy uses to gauge the delivery of electric services include:    SAIDI (System Average Interruption Duration Index): A common measure of the duration of power outages on the distribution system. SAIFI (System Average Interruption Frequency Index): A common measure of the frequency of power outages on the distribution system. SATLPI (System Average Transmission Line Performance Index): A measure of voltage sags or line faults on the transmission system.

25

Austin Energy, “Company Profile” web page. Available at: http://www.austinenergy.com/About%20Us/Company%20Profile/index.htm (accessed December 31, 2009).

26

Austin Energy, Strategic Planning Update, 2008. Available at: http://www.austinenergy.com/About%20Us/Newsroom/Strategic%20Plan/strategicPlanningUpdate_2008.pdf (accessed December 31, 2009).

13

The three performance measures that Austin Energy uses to assess the reliability of its power production rely on a metric called the equivalent availability factor (EAF), a measure of the availability of power for use. The three measures using EAF include the availability of power at two power generating facilities and the availability of power during peak season. The organization also aims to have renewable energy comprise 30 percent of its power generation portfolio and to improve energy efficiency by 15 percent by 2020. Austin Energy assesses stakeholder satisfaction with surveys of both customer and employee satisfaction. It aims to achieve a customer satisfaction target of 83 percent by fiscal year 2010, a measure based on the American Customer Satisfaction Index.27 In 2006, Austin Energy’s Electric Service Delivery business area began a formal quality management effort and in 2008 achieved certification under the ISO 9001:2000 standard. The ISO 9001:2000 standard specifies the requirements for a quality management system and emphasizes improvements in the effectiveness of processes through numerical performance measures. The Electric Service Delivery business area used a balanced scorecard approach to ensure service quality. It developed key performance indicators (performance measures and numerical targets) in the areas of cost, reliability/regulatory, customer satisfaction, safety, and employee satisfaction. With limited resources, the business area recognized the importance of balancing efforts to improve service with costs.28 The Electric Service Delivery business area divided its key performance indicators into three tiers according to organizational responsibility. The strategic performance indicators are used at the first tier by executives to lead the business area. These measures are aligned with the overall Austin Energy organizational strategies and feed into the organizational performance measures. The second tier key performance indicators are operational in nature and are used by managers to manage the business. Managers are held accountable for achieving the performance targets. The third tier of key performance indicators support the first two tiers and are focused on process and efficiency. They support action plans and are used by supervisors for day-to-day activities. The first-tier performance measures are reported to the community as part of the overall organization’s performance measures regularly through the Austin Energy website, bulletin boards, and a regular newsletter. Performance trends are reported monthly for cost, reliability/regulatory, and safety measures, whereas customer satisfaction and employee satisfaction trends are reported on an annual basis. Austin Energy strongly believes that tracking and reporting performance results is important for successfully managing and improving an organization.
27

Austin Energy, Strategic Plan Update, 2008. Available at: http://www.austinenergy.com/About%20Us/Newsroom/Strategic%20Plan/strategicPlanningUpdate_2008.pdf (accessed December 31, 2009). Telephone Interview with Mercedes Sanchez, Austin Energy, on December 3, 2009.

28

14

The Austin Energy Electric Service Delivery vice president and the business area management meet every 6 months to discuss the effectiveness of the quality management system. As part of these meetings, performance results are examined for the key performance indicators, and action plans are initiated for measures that are trending in the wrong direction. The meeting participants also re-evaluate the indicators and targets. Several benefits have been realized by the Electric Service Delivery business area because of its quality improvement effort, including improvements in communications and collaboration between operational work groups, the documentation of issues as they occur, identification of root causes, and action plans developed and carried out to address these issues. The quality improvement effort seems to be winning over many of the impacted employees. Austin Energy reported that the ISO 9000:2001 standard gave it a needed framework to help manage its business. For others working toward developing a quality management system, Austin Energy emphasized the importance of not trying to design a perfect system, but instead of developing a solid foundation that can be continuously improved over the long term.29 A U.S. Power Company A U.S. Power Company provides power over a 10,000 square mile area, covering several mid-sized cities with populations between 100,000 and 500,000. The company manages a radial transmission grid with very few interconnects between the lines. It controls or owns generating capacity in the United States, sells energy in U.S. markets, and delivers electricity to about 4 million customers. Highlights of the U.S. Power Company’s operational performance management process are below. The U.S Power Company:  Measures and tracks performance for internal processes and outcomes. In the area of process, it examines process improvement and worker safety. To assess outcomes, it measures reliability and customer satisfaction. Uses a hierarchy of performance goals and targets appropriate for each level of the organization. Broader goals relevant to the overall operation are established for high-level managers, whereas lower level personnel are given performance goals based on their responsibilities. Sets benchmarks against peers with similar resources and operating environments. The company sets performance targets slightly above the peer group benchmarks. Examines both the frequency and duration of service interruptions. Tracks performance measures using a 12-month rolling average; reliability performance data is compiled daily. Tracks measures that are believed to predict important outcomes; these are known as “forward indicators.”



   

29

Telephone Interview with Mercedes Sanchez, Austin Energy, March 2009.

15

Interestingly, the company also focuses on improving service problems during standard conditions rather than during storms or other major events. The company uses four categories of operational performance measures that are typically tracked using a rolling 12-month average: 1. 2. 3. 4. Safety Reliability Customer Satisfaction Process Improvement

The performance goals are developed according to tiers in the organization. Goals at tier 0, the top level, address the highest level organizational objectives and do not necessarily apply to all departments. A hierarchy of tier 0, tier 1, tier 2, and tier 3 goals address the objectives in increasing specificity, with performance measures being developed from the top (or most broad) down to the most detailed measures. The tier 0 goals are determined by defining the highest level organizational objectives. Tier 1 goals are developed to flesh out the tier 0 goals. Tier 2 goals are developed to support the tier 1 goals, and so on. There are numerous specific tier 3 goals that are managed at the working group level. The company’s performance measures were originally developed by benchmarking and analysis of best practices of comparable organizations. Identifying comparable organizations was difficult because many companies have more underground assets or more network interconnects that make them less prone to outages. Now that the company has found a peer group, this group has been helpful in sharing lessons learned and developing a best practices library. The U.S. Power Company holds an annual planning meeting to modify performance measures and determine performance objectives for the next year. All high-level managers attend and work to reach a consensus for tier 0 and tier 1 performance measures and objectives. The managers then meet separately with lower level staff to determine tier 2 and tier 3 measures and objectives. Typically the company takes about 4 months to finalize its performance goals for the year. Reliability is measured primarily through performance management indices that do not account for problems encountered during storms. The company also has an initiative underway to identify customers with frequent outages (5+ per year) and improve their service. It collects reliability data on an hourly basis and compiles the data daily. Outage information is collected from call centers and, increasingly, from the transmission grid itself. The company assesses customer satisfaction through quarterly phone interviews of residential customers. Company management sets a target percentage of customer satisfaction to reach based on other power companies’ customer satisfaction results. The company only recently began developing its process improvement objectives. 16

Senior managers of the U.S. Power Company communicate performance targets and data to staff by posting performance information on the company website and bulletin boards. Operations managers hold monthly meetings to coordinate their activities. Each working group within the company has milestones that it strives to achieve and tracks its progress against them. Most company performance measures are tracked publicly. The company has also faced challenges using performance measures. As people move away from land-line telephones, the company has had difficulty reaching an accurate cross section of customers for the satisfaction survey. Additionally, changes in leadership often translate into adjustments in performance measurement because of new goals and priorities. A utility company representative interviewed for this paper reported that the company had benefitted from the performance measurement approach. Despite staff reductions over the last few years, the company has remained above average in national rankings. He also stated that performance measurement has elevated the importance of customer satisfaction, and the company has also seen improvements in safety. Company budgets are sensitive to performance results, he noted. Because the company tracks service reliability, it was able to see that a reduction in the vegetation management budget corresponded to a decrease in system reliability. The company was then able to respond by restoring the vegetation management budget.

17