POLICE PROBLEMS: THE COMPLEXITY OF PROBLEM THEORY, RESEARCH AND EVALUATION by John Eck
University of Cincinnati

Abstract: Advancement of problem-oriented policing has been stymied by over-attention to police organizations and under-attention to police problems. This paper develops a research agenda for understanding police problems by addressing four fundamental questions: What are problems? What causes problems? How can we find effective solutions to problems? And how can we learn from problem solving? For each question a possible direction for theory, research, or evaluation is suggested. The variety of police problems, their non-linear feedback systems, the diversity of responses that can be applied to problems, and the difficulty of learning from problem-solving experiences highlight the complexity of police problems. The paper closes with a list of research questions designed to improve the science and practice of problem analysis and solution.

WHAT IS THE PROBLEM WITH POLICE RESEARCH?
Problem-oriented policing has become the victim of the disease it was meant to cure, the "means over ends syndrome." Symptoms of this disease include studies examining the internal workings of police organizations implementing problem-oriented policing, confusing problem-oriented policing with community policing, and generally failing to recognize that a new approach to policing requires a different approach to research. Though the number of police agencies applying problem solving has grown rapidly, particularly since 1987,

Crime Prevention Studies, vol. 15 (2003), pp. 79-113.

John Eck

the theory and practice of problem-oriented policing has grown slowly and fitfully. In the absence of leadership and goading from full-time researchers, it is understandable that practitioners did not explore unknown territories and push the boundaries of problem solving (Eck, 2003). Most police, and community members, when confronted with a problem immediately turn to notions of deterrence and incapacitation. Only when these fail to adequately address the problem, often failing repeatedly, do the police and the public explore something new. An important reason for the development of problem-oriented policing was the overreliance on the criminal law. Yet, to most police, and their publics, it is not immediately clear what the alternative is. In the United States, when alternatives are presented they often take the form of some method for reforming prospective or existing offenders — reaching out to disaffected youth, providing anti-drug education, managing sport and recreation opportunities for teens, and similar programs designed to thwart the development of criminal propensities. Using the law to sanction offenders or using social programs to forestall criminogenic tendencies is not always ineffective, though many such approaches are ineffective (Sherman et al., 2002). The difficulty is that both approaches are far too limited to have much impact. Further, overreliance on coercive authority has major negative effects on police legitimacy, and the police are singularly illequipped to provide effective assistance to potential offenders. The deficiencies of these approaches have been described elsewhere (Felson and Clarke, 1998). Other work describes why problemoriented policing has made limited progress and clarifies some of the confusions that have crept into writing and practice (Eck, 2003). This paper describes how research and evaluation can improve the theory and practice of problem-oriented policing. This paper does not discuss community involvement, multiagency collaborations, and related topics. These are important topics in their own right, they are quite valuable in practice, and they need to be examined. But a paper on the technical nature of problems is not the best place to address these issues. Indeed, most of what this paper describes could be applied by community organizations addressing problems or other governmental agencies, with or without the police. All concerned — not just the police — require a deeper understanding of problems. But a deeper understanding of problems will not come from studies of how police and communities work together (or fail to), or how local gov-

-80-

Police Problems ernment administration can be organized to create partnerships among government agencies. It is also important to understand that problems are real and have a "life" that is only loosely coupled with people's perceptions of them. Observations that beat cops are concerned with hard crimes, like robberies, but the public is concerned with incivilities, like litter, do not imply that there are not robbery problems, or that robbery problems are really litter problems (nor does it imply the opposite). This paper examines the diversity and causes of problems, not how priorities are set. How priorities are made and who makes them is not the subject of this paper. Ultimately, it will not matter how priorities are set, or who is involved in addressing problems, if we do not understand them and have useful ways of solving them.

Basic Questions
This paper is organized around four basic questions that parallel the SARA problem-solving process. The first section examines how to answer the question, "What are problems?" Until recently, this question has been answered rather simplistically. But recent developments in problem classification reveal an extraordinarily complex world of police problems. The second section examines the question, "What causes problems?" Until we can provide the police and the public with coherent and useful alternatives to common notions of deterrence and incapacitation, we cannot expect them to routinely solve problems. The thesis in this section is that problems are created by the breakdown of feedback processes that help people regulate potential offenders and potential problem situations. The third section asks the question, "How can we find effective solutions to problems?" This section introduces a question-based protocol for taking a problem solver through the steps from a description of a problem to applicable solutions. Even a rudimentary prototype problem-solving protocol suggests levels of complexity that neither police nor research have come to grips with. The fourth section examines the question, "How can we learn from problem solving?" Increasing systematic information exchange among police agencies is part of the answer. Another part of the answer is developing techniques for synthesizing low quality evaluations to provide real time "best" advice to problem solvers. This will be an ambitious undertaking, but there are some clues how we could proceed.

-81-

John Eck

The final section summarizes the implications from these suggestions and outlines a research agenda for problem-oriented policing.

WHAT ARE PROBLEMS?
Police problems are typically described as groups of related incidents of concern to the community. There are three elements here. First, problems are groups of incidents, not singular events. Second, the incidents in this group are connected in some meaningful way, not random or arbitrary. These two elements suggest that the events that make up a problem stem from the same underlying cause. The third element requires that the incidents be disturbing or harmful to members of the public, not just to the police (Goldstein, 1990; Office of Community Oriented Policing Services, 1998). This definition includes an extensive range of concerns and it provides limited guidance to the police or the public. The principal use of this definition is to define the outer boundaries of the problem territory — individual crimes are outside this territory, for example, as are police policies and procedures. Just as a problem is made up of similar events, so a problem type is made up of closely related problems (Figure 1). If we are confronted with a problem, and we have information from other problems, then we can apply this experience to the new problem. Currently police and researchers do this on an ad hoc basis. But to make greater progress more quickly, we need a problem classification scheme. A problem classification scheme building on Routine Activity Theory has been proposed (Eck and Clarke, in press). The Eck-Clarke problem classification scheme begins with a distinction between common problems and system problems. Common problems involve offenders coming into contact with their targets (human, animal, or thing). These make up the bulk of problems confronted by local police agencies. Two elements are particularly critical for understanding common problems: the behaviors of the people involved and the environment (or place) in which these behaviors occur. When offenders do not have to come into contact with their targets, we are dealing with a system problem. System problems are more typically the province of national police agencies and private organizations. Though the behaviors are similar, a system substitutes for the environment. A common example of a system problem is a destructive computer virus. The offender uses a worldwide system to vandalize targets at great distances. A series of mail bombings is another example of a system problem.1
-82-

Police Problems

Figure 1: Events, Problems, and Problem Types

To classify common police problems,2 Eck and Clarke identified six behaviors and eleven environments. The behavioral dimension is important because it draws attention to the way people act, the interaction among participants in a problem, and their motivations. The environmental dimension points to who owns the locations and has control over behaviors of people using the environment. The 66 problem types are shown by the cells formed by the intersection of the two dimensions in Table 1. This classification scheme requires a revision in the definition of common problems. Not only must a problem involve repetitive related events of concern to the public, but a definition of a particular problem must also include a description of a behavior and an environment. This eliminates from consideration such concerns as neighborhoods (they may contain problems but are not problems in themselves), status characteristics (like loitering or truancy), and other things that cannot be located in this grid. Removing vague concerns and requiring specificity enhances problem analysis and solution identification.

-83-

Police Problems

In the long run, we would expect to be able to link solution types to particular classes of problems with statements like, "If the problem is of type XY then solutions of forms A, B, and C are most likely to be helpful, but solutions of the forms D and E will be ineffective, and solutions of the form F will be counterproductive." This classification scheme was deliberately kept as simple as possible to make it useful (and it probably cannot be made much simpler and retain its utility). More importantly, it is maybe too simple. Here are some reasons why. First, consideration of system problems could easily double the count of problem types. We propose that the classification of system problems use the same behavioral dimension, but substitute a system dimension for the environments dimension (Eck and Clarke, in press). If there are more than 11 important systems, then the total number of problems will more than double. Second, there may be other dimensions not considered in this scheme. For example, elsewhere it has been suggested that all problems are of one of four types: repeat offender, repeat victim, repeat place, or a combination of these three repeats (Eck, 2001). It might be useful to add this or some other dimension. Third, within each of the two existing dimensions, the categories are rather large. These need to be subdivided further. Public ways, for example, consist of highways, roads, paths, and parking lots. And Eck and Clarke (in press) propose three subcategories for incivilities. So 66 problems is small compared to the number of problems we are likely to uncover if this classification scheme is developed further. As a thought experiment, consider only common problems. Begin with the current classification scheme, but imagine some simple modifications. First, assume that, on average, the column and row headings are divided into three subcategories each, and there is a new third dimension with only four categories. Now the number of possible common problems has grown to 2,376 (=33 x 18 x 4). This number sounds absurdly large. To see if this number is really absurd, let's look at one cell of Table 1 and see what happens if we probe deeper along each of its dimensions. Predatory-residential problems are reasonably common, so this is a useful example. First, let's subdivide predatory behaviors into: • • • • Breaking in and taking things; Breaking in and attacking people; Entering unsecured structures and taking things; Entering unsecured structures and attacking people;

-85-

John Eck

• • • • • • •

Deceptive entry and taking things; Deceptive entry and attacking people; Burning structures for profit; Burning structures for retaliation; Burning structures for intimidation; Vandalism for retaliation; Vandalism for intimidation;

• and others. These 11 types of predatory behavior are only some of the forms of the predatory behaviors we might find associated with residential environments. But 11 is enough to make the point that there are many. Next, let's look at possible types of residential environments. Again, we will only list some of the most obvious types and keep in mind that the list is probably much longer: • • • • • • • • • • • • Nursing homes; High-rise hotels in cities; Motels on highways; Single-family residential homes; Duplex family residential homes; Garden apartment complexes; High-rise condominiums; Single-room occupancy hotels; Mobile home parks; Rental vacation cottages; Summer homes; College dorms;

• and others. If we combine these two lists, we have 1 1 x 1 2 = 132 forms of residential-predatory problems. Some of these 132 problems may be rare "boutique" problems (deceptive entry into rental vacation cottages and attacking people, for example, is probably a rare incident and unlikely to be a frequent problem). Nevertheless, even if many police departments have few encounters with most of these problems, some
-86-

Police Problems

police department is likely to encounter some of them, and even the rare problems will be familiar to some agencies. Now, consider expanding each of the column and row elements in Table 1. Finding 10 or more discrete forms of each behavior, and 10 or more distinct forms of each environment is not unlikely. This suggests that each cell of Table 1 could contain, on average, over 100 problems. And this implies that there may be over 6,600 problems. And this number was calculated without consideration of systems problems and without adding any new dimensions. So rather than being absurd, 2,376 might be a low estimate of the number of problems. But the point of this exercise was not to come up with even a gross estimate of the number of problems. Rather, it was to provide an idea of the level of magnitude of the number of problems. And the reason for trying to get a level of magnitude is to demonstrate how little we know about problems. But even if there are only 66, it also is clear that we know very little about any of the problem types in the Eck-Clarke classification system, and for most problem types we have virtually no systematic knowledge. Any reasonable modification to the scheme shows that our ignorance is vast. This exercise demonstrates several important facts. First, there are many problems. Second, we know little about the vast majority of them. And third, taken as a whole, problem solving is extremely complex. So the first topic in a problem-oriented research agenda is the documentation and cataloging of different problem types. This includes identification of specific problems, their defining characteristics, and methods for usefully classifying them.

WHAT CAUSES PROBLEMS?
We get another hint of how little we know about problems when we examine the causes of problems. Not surprisingly, our ignorance of causes is even greater than our knowledge of the types of problems. This section summarizes and expands the current set of problem theories, once again by drawing heavily on Routine Activity Theory and related theories. By itself, Routine Activity Theory cannot explain problems; neither can theories that address fewer elements than Routine Activity Theory. Nevertheless, it can help use develop a framework for understanding problems.

-87-

John Eck

Routine Activity Theory is an explanation of crime events. Though it was developed originally to explain macro-level crime trends through the interaction of targets, offenders, and guardians (Cohen and Felson, 1979), it has been expanded over the years to include handlers of offenders (Felson 1986), places (Felson, 1987), and place managers (Eck, 1994, 1995). We can summarize a recent version of Routine Activity Theory (Felson, 1995) with the statement that: a crime is highly likely when an offender and a target come together at the same place at the same time, and there is no one nearby to control the offender, protect the target, or regulate conduct at the place. This is diagramed in Figure 2, where the inner triangle contains the elements necessary for a crime and the outer triangles contain the controllers sufficient for prevention.3 How can we move from this explanation of events to an explanation of problems? One possibility is to explain problems by way of offenders. Another is to explain problems by way of targets. So problems could be locations with many offenders or with many targets. Separately, neither of these explanations is adequate, however. First, we need targets and offenders together, at the same time, at the same place. So looking at either targets or offenders alone is inadequate. We need both. Second, we have to explain the absence of controllers — people who can intervene with the offender, target or place and keep the crime from occurring. Though we cannot explain problems solely on the basis of offenders or targets, examining offenders, targets, and places gives us a starting point in our search for an explanation. The empirical literature illustrates that offenders, targets, and places show highly skewed crime distributions. A few targets, places, or offenders are involved in a large proportion of the problem events, and all problems involve repeat offending, repeat victimization, repeat places, or some mixture of these repeats (Eck, 2001). A pure repeat-offending problem involves an offender attacking different targets at different places. A pure repeat-victimization problem involves a victim repeatedly attacked by different offenders at different places. A pure repeat-place problem involves different offenders and different targets interacting at. the same place. This is not a complete explanation (because it does not deal with the controllers) though it does address the first criticism above.

-88-

Police Problems

Figure 2: Routine Activity Theory's Crime Triangles

Controllers are at the heart of any useful theory of problems. Or to put it more precisely, problems are created when offenders and targets repeatedly come together and controllers fail to act. It is the breakdown of controllers that is the most important feature of this explanation, as offenders and targets often come together without any problem being created. To see how this breakdown occurs, let's look at Figure 3. Here we have the same elements as shown in the Routine Activity Triangles along with their lines of influence. Starting on the periphery and working inward we see that the controllers (manager, handler, and guardian) have influence over places, offenders, and targets. These elements together, in turn, directly influence whether a problem event will occur. Going in the other direction we see that the presence (or absence) of a problem event has a direct influence on places, offenders and targets. Problem events have direct, and indirect influence on controllers, depending on the nature of the element being controlled.

-89-

John Eck

Figure 3: Primary Lines of Influence and Feedback

These influences create a number of feedback loops. When a problem event occurs it influences how the offender behaves, which, in turn, affects the chance of another event. The positive signs on the arrows between the offender and event indicate a reinforcement process when the offender is successful and a suppressing effect when the offender is unsuccessful. But we also have to take into account the handler, if present. The offender's involvement in a problem stimulates the handler (positive sign) to suppress the offender's activity (negative sign). The handler might learn directly about the problem events, or from contact with the offender. But in either case, an effective handler will attempt to suppress the offender's problem behavior. Similar processes engage managers, guardians, and victims. With one exception, all of the loops act like thermostatic controls by dampening crime potential. The exception is the offender-crime loop, which aggravates crime potential when the offender is successful and dampens it when the offender is unsuccessful. So when the complete system of relationship is fully functioning, two things occur.
-90-

Police Problems

The offender is reinforced, but the countervailing forces from the controllers (and victim, if we are dealing with a human target) make the place and target less vulnerable. This decreases the chances of the next attack being successful and thereby reduces the number of repeat attacks. What happens when this system is not fully functioning? Let's begin with the offender. If the offender does not get positive rewards from offending, then the offender will cease to offend. But let's assume that the offender continues to get positive rewards. If any of the negative lines of influence diminish or disappear, then the system will be slower to respond to offender attacks, thereby increasing the chances of subsequent attacks, and increasing the number of repeats. An extreme breakdown is shown in Figure 4, where all of the negative feedback has disappeared so that the target is likely to be continually attacked at the same place. The offender is reinforced. The handler, in this example, is stimulated, but cannot influence the offender. The manager is unconnected with the place, and the place is unaffected by the crime. The absence of a negative influence on the target suggests that the target cannot take effective precautions. And, in this example, though there is a guardian, the guardian cannot influence the target. This might be a diagram of a serious domestic violence problem. The system just described is far simpler than exists in the real world. One obvious oversimplification is that it does not include learning from the experiences of others. This and other features could be added and would help our understanding. For now it's important to draw attention to the idea that problems occur because offenders continue to get some kind of reinforcement, but those who can do something to stop their troublesome behaviors: a) do not learn about the events, b) choose not to act, or c) cannot act effectively. Such feedback malfunctions are characteristic of all problems. Pure repeat-offender problems occur when offenders get reinforcement and are able to locate temporarily vulnerable targets and places. The controllers for these targets and places may act to prevent future attacks, but the offenders move on to other targets and places. It is the offender-handler breakdown that facilitates pure repeat-offender problems.

-91-

John Eck

Figure 4: Controller Collapse Due To Feedback Malfunction

Pure repeat-victim problems occur when victims continually interact with potential offenders at different places, but the victims do not increase their precautionary measures based on past encounters and their guardians are either absent or continually ineffective. The handlers may prevent the offenders from engaging in more of these events, and the managers may improve how they regulate conduct at their places, but the victim moves to other encounters with other offenders at other places. Pure repeat-place problems occur when new potential offenders and new potential targets encounter each other in a place where management does not change conditions. The setting continues to facilitate the problem events, even though handlers suppress offending and guardians suppress victimization. It seems highly unlikely that pure repeat problems are extremely common. Instead, there is likely to be some overlap, particularly for serious problems (Farrell and Sousa, 2001). The extent of this overlap is extremely difficult to measure because each repeat is best detected through a different type of data. Repeat offending is best detected
-92-

Police Problems

through offender interviews and arrest records that do not document the places or victims well. Repeat victims are best detected through victimization surveys, but these will have scant information on places and even less on offenders. Repeat places are best detected through police call-for-service and reported crime data. Even when these sources have information on victims, they seldom capture offender information, unless someone has been caught (Eck, 2001). For this reason, we should not expect precise measures of overlap for all three repeats, though pairs of repeats might be feasible in some circumstances (see, Everson and Pease, 2001). We should expect offenders, victims, and controllers to generalize from their personal experiences and to learn from the experiences of others. An offender, after successfully attacking a particular target, might seek out other targets that have similar characteristics. Pease (1998) identified the phenomenon of "virtual repeats," which occurs when offenders learn from successful attacks on one target and go on to attack nearby targets. Townsley (2000) has documented this phenomenon in an Australian community. This is similar to the process that gives rise to "hot products" (Clarke, 2000). But generalization may also operate to suppress crime. After unsuccessfully attacking a target an offender might avoid targets with similar characteristics. Offender generalization from their negative experiences helps explain diffusion of crime prevention benefits (Clarke and Weisburd, 1994). Generalization is probably not peculiar to offenders. Handlers can generalize from one experience to more closely monitor other similar situations in which the offender might get into trouble. Similarly, after being attacked in a particular place, a victim might avoid similar places and his guardian might be more vigilant in these places. A manager might more carefully monitor the behaviors of individuals who bear similar characteristics to prior offenders or victims. It is not difficult to see how these generalizations on the part of victims and controllers might prevent future problem events. But it is also easy to see how this very same behavior can lead to discriminatory practices aimed at individuals who are not likely to get into trouble. The less precise the information used to make the generalization, the more likely discrimination will arise.4 We might imagine each of these actors imbedded in a network of relationships that keeps them informed about the experiences of others. Offenders communicate with other offenders and with others who communicate with offenders. Gang membership is one form of such a network, and this might explain why gang members are more criminally active than offenders not in gangs (Thornberry, 1998). Among
-93-

John Eck

college males, social support for aggressive sexual encounters is associated with high rates of forced sex with women (Schwartz et al., 2001). Information about successful and unsuccessful attacks of others changes offender expectations about potential targets and places. Handler networks can help forestall the acts of potential offenders. Potential human targets and guardians might also be part of network that provides information about the attacks on others. Finally, managers can exchange information that helps update them on their places' vulnerability to future attacks. Professional associates often provide such information, and some, like local banking associations, may have institutionalized relationships with the police and other security professionals. The implication of generalization and networks is that the most successful offenders will have heightened abilities to generalize and learn from the experiences of others. Handlers, victims, guardians, and managers who have limited abilities to generalize and are least connected to networks will be the least successful in preventing problem events. The stability in repeat offending, places and victimization found by Spelman (1994a, 1994b, 1995) is due to the effect of these feedback loops breaking down around victims and places, but being reinforced for offenders. This discussion of generalization and networks requires us to expand the concept of feedback. The lines of influence shown in Figure 3 are only some of many possible connections that help the participants learn and adapt. For any participant, the less connected they are the less information they will receive, and the less successful they will be. The breakdown in controllers along with the stimulation of offenders can be readily appreciated in a number of business contexts. Large-scale merchants who do little to prevent theft because they can absorb the costs become repeat victims as offenders learn they are easy marks. Clarke and Goldstein (2002) provide another illustration of this phenomenon. New housing developers in the Charlotte area of North Carolina experienced a large number of thefts of appliances. The appliances were delivered to new housing developments prior to the houses being sold and as a consequence were unprotected after work hours, on weekends, and on holidays. To address this problem, the police tried to persuade the builders to alter the appliance installation schedule so that they would be delivered and connected just before or just after an owner occupied a residence. This approach was somewhat successful. The important point for this discussion is that it was the failure of the builders to change their be-94-

Police Problems

havior in the face of offending that allowed the problem to develop. The solution, like the solution to most problems, was to get one of the controllers to change their behavior in response to the offending behavior. In summary: • The existence of a problem suggests that some offenders learn how to take advantage of particular situations. • • • All problems require the breakdown of one of one or more control systems. The systems that break down lead to specific types of repeats — offenders, targets, places.

And how control systems break down and what can be done to repair the feedback system depend on the problem type and associated behaviors and environments. This group of propositions presents framework for research and action. It draws attention to problem features that are common across all problems. Still, it is not detailed enough to provide an indepth understanding of specific types of problems. When one combines this framework with the problem types it is apparent that the propositions are likely to manifest themselves differently in each type of problem. Even if this rudimentary theory of problems is helpful, considerable work is required to adapt it to various problem types. This system of relationships not only contains a set of feedback mechanisms, but also is nonlinear. The nonlinearity comes from Routine Activity Theory (Eck, 1995). Feedback and nonlinearity suggest that problems are highly complex and maybe chaotic. It is hard to predict the behavior of chaotic systems, though short-term predictions are sometimes possible. The reason for this is that small changes in the system get magnified in unexpected ways (Williams, 1997). The combination of feedback and non-linearity also suggests that the systems may be highly stable under some conditions (Mainzer, 1997). The possibility that some problems may be stable while others are chaotic presents another line of enquiry. But it is a line of enquiry that is very difficult to study in real world settings. Under these conditions, experimentation with simulated artificial problems can be extremely productive (Casti, 1997). For example, Liang (2001) has built a simulation of commercial robbery that can be experimentally manipulated based on the concepts of feedback elaborated above.

-95-

John Eck

This section describes problems as outgrowths of complex adaptive systems resulting from breakdowns in some feedback systems and reinforcement of offender feedback. Adaptation by all of the parties involved should be one of the most important topics of research. The goal should be a unified understanding of the four Ds: • • • Desistance — problem reduction; Defiance — problem amplification; Diffusion — problem contraction; and,

• Displacement — problem spread. Since these are not mutually exclusive outcomes of problems with multiple actors, research in this area will be difficult and will require us to expand the variety of analytical tools we use.

HOW CAN WE FIND EFFECTIVE SOLUTIONS?
One of the greatest difficulties for problem-oriented policing becomes apparent when one moves from analyzing a problem to responding to the problem. There is no obvious link between these two stages of the problem-solving process. In fact, there is a very large gap and we expect police problem solvers to leap easily across it. One bridge across this gap is the recent series of problem-specific guides developed by the Office of Community Oriented Policing Services of the U.S. Department of Justice (see, for example, Sampson, 2001). How do we know that the data collected point to a specific set of solutions? Or given a particular solution, how do we know that the data support its use? What features of a problem suggest that any one of the 16 types of interventions suggested by Situational Crime Prevention (Clarke and Homel, 1997) has a high likelihood of success, for example? Currently, we expect beat-level problem solvers (see Goldstein, in this volume) to identify a problem, collect information about it, and develop some insights leading to a solution. We call this professional judgment. As important as professional judgment is to developing a solution, no profession relies on it exclusively. Case studies of problem-solving efforts can improve professional problem-solving judgment. But in most professions more than case studies support professional judgment. It is also supported by theories and sets of protocols that link symptoms, diagnosis and action. With regard to problem solving, a protocol should have four characteristics.

-96-

Police Problems

First, the average beat-level problem solver should be able to use it in a variety of settings after some training. Second, the protocol should provide reasonably useful guidance to a broad range of problems. Third, the protocol should be theoretically-based. The previous section described a framework for examining problems. This framework can be used to direct problem solvers' attention to potential responses. Additional theoretical support comes from Offender Search Theory (Brantingham and Brantingham, 1981) and Situational Crime Prevention (Clarke and Homel, 1997). Fourth, the protocol should be question-based, rather than information-based (see also Townsley, in this volume). That is, the problem solver should be asked a series of questions, the answers to which point to possible solutions. A guide that is a compendium of data sources or analytical techniques is far less useful to practitioners, given the variability of data access across police agencies. Finally, the protocol should reveal multiple possible responses, rather than a single response. Figure 5 provides an example of what such a protocol might look like. It is an extract from a draft protocol (the questions and responses shown here deal with only place managers).5 To use this protocol requires some basic understanding of the theoretical perspectives noted above. Following a set of instructions, this prototype is divided into two parts. The first part (upper part of Figure 5) is a set of 155 questions about the problem, offenders, handlers, targets, guardians, places and managers. Additionally, it contains questions about tools that offenders, handlers, victims, guardians, and managers use. And it contains questions about movement patterns. A problem solver can attempt to answer some or all of the questions, depending on what she knows about the problem. Most of the questions point to a type of solution (when a "yes" answer is followed by a alpha-numeric code linked to a response). The solutions are listed in the second part of this prototype in 58 broad categories (bottom part of Figure 5). Importantly, these solutions are not detailed plans, but are instead pointers to classes of interventions. It is also possible to begin with a potential solution and work backward to determine if there is sufficient information to support its application. Here again we see a hint at the hidden complexity of problems, and the corresponding depth of our ignorance. This initial attempt at a general purpose beat-level problem-solving protocol suggests that there is a very broad array of potential solutions, each of which is potentially appropriate in some specific contexts, but few of which are generally applicable to most problems, regardless of context. It is also

-97-

John Eck

quite likely that for any particular problem in a specific context, there may be several alternative solutions. A protocol like this could be used by an individual or by a group, by police officers, community members, or by others involved in beatlevel problem solving. If encoded in software with the problem classification scheme described earlier, it could provide links to earlier problem-solving experience that could provide additional guidance. If connected to a problem-solving information system that tracks problem-solving efforts, the guide could facilitate management oversight and training improvements. This example protocol is based almost totally on theory. As greater empirical knowledge of the types of solutions that are effective (and ineffective) for particular problem types is acquired, the questions and responses can be updated, or even completely replaced. This requires that the police profession improve the way it learns from experience.

HOW CAN WE LEARN FROM PROBLEM SOLVING?
A problem-solving protocol like that described in the previous section is designed to suggest possible actions. Though it is theoretically based, the theoretical foundations applied here are far too general to provide more than a pointer to possible solutions. Two things are lacking. First, we have little empirically-based information about what responses are appropriate for particular problems. Second, there are few empirical tests of responses applied to specific problems. For these reasons, any problem-solving protocol must be supported by systematic evaluations of problem-solving responses. And these evaluations must be synthesized to improve our understanding of what solutions are most appropriate for each type of problem. Learning what types of solutions work for what problem is a daunting task. We can see how daunting if we make some unrealistically simple assumptions. First, based on the problem classification scheme, let's assume that there are exactly 66 types of problems. Second, based on the draft protocol described above, let's assume that there are only 58 solutions. Nevertheless, 58 solutions applied to 66 types of problem yields 3,828 possible applications of solutions to problems for testing. We have good reason to believe these figures for problems and solutions are low. But because we are multiplying, even small increases in either the number of problems or the number of solutions will dramatically increase the number of combinations.

-98-

Police Problems

Three things are readily apparent. First, gaining systematic knowledge about what types of solutions work best for which types of problems, and what types of solutions make things worse for which types of problems, is a task that is beyond the capabilities of any single police agency. However we approach this we know this must be a group effort involving many police agencies and many researchers. Second, given our experience with randomized experiments, it is clear that we cannot rely on them to make substantial progress at even a modest pace. Randomized trials cannot be applied to problems with small numbers of cases, unless one is willing to wait a very long time or involve large numbers of police agencies in a single experiment. Randomized trials require separable units of analysis — individual people or places that do not communicate with each other. Many problems do not have this characteristic. Randomized trials are often difficult to implement without organizational disruption. When the problem is common and serious and the tested solution is controversial, expensive, or has serious negative side effects, such disruption may be worthwhile. But few organizations will have the stomach to participate in such experiments on a regular basis. A single randomized experiment is insufficient; replication is required. But as the Spouse Abuse Replication Program demonstrated, multiple experiments in the absence of detailed theory may lead to interesting questions, but not necessarily to definitive conclusions (Maxwell et. al., 2001). Randomized experiments may be the gold standard of drug and medical procedure testing, but can it be the gold standard of crime science? The needs of crime science and the limits on randomized experiments might mean that we have to go off the gold standard.6 People still use gold as an investment tool, though most of us conduct most of our commercial transactions without it. Similarly we should continue to use randomized experiments in special cases, such as when large claims are made for expensive programs or when interventions may have strong negative side effects, or when a program is otherwise controversial. When program claims are sweeping, randomized experiments perform a very useful pruning function: deflating these claims by pointing out that the intervention does not work well for everyone or everywhere. This is the negative function of experiments (Eck, 2002b). Randomized experiments also have great utility when the following conditions can be met: (1) The problem is specific and well defined; (2) The problem is serious and widespread;

-101-

John Eck

(3) There is a well-defined and tested theory of the problem; (4) That theory clearly implies a coherent intervention; (5) The intervention is expensive, has strong side effects, or is controversial; (6) The theory clearly implies the context in which the intervention will work, and contexts in which it will not work; (7) Discrete isolated intervention units (people or places) exist in sufficient quantity that an experiment of reasonable power can be applied; (8) The base rate of the events for these units is high enough that a drop in the number of events can be detected; and, (9) The experimental intervention closely mimics the form the intervention would take when it is operationalized in everyday practice. When these conditions are met, a randomized experiment simultaneously tests the intervention and the theory. Conditions one and two limit experiments to clearly defined problems that are common and serious enough that experimentation is worthwhile. When conditions three and four are not met, then randomized experiments can provide a useful method for eliminating ineffective, expensive programs that have become entrenched. Nevertheless, we learn more from experiments if there is a sound theory behind the intervention. Condition five requires that there are real stakes in the outcome. A cheap intervention, with no side effects, that everyone likes is not worth the time and expense of a randomized experiment. Conditions six through eight assure that the experiment can provide meaningful results. The last condition assures us that the experimental conditions are not so artificial that they have no application to real world settings. This leads us to the third thing we know. We know that police agencies have trouble mounting complex evaluations on a routine basis. Some improvements can be realized, but we cannot expect the same level of rigor we would from a fully funded academic evaluation. In fact, there is a very sound reason for applying weak evaluation designs to problem-solving efforts. If the problem solver is more interested in whether the problem declined than in taking credit for the decline, and has little interest in promoting his particular solution, the problem solver is justified in using a simple evaluation design. Only when the problem solver expects to use the solution again, in a similar context, is a rigorous design, which eliminates most threats to

-102-

Police Problems

validity, justified (Eck, 2002b). But even in these cases, there are quasi-experimental designs that yield highly valid conclusions (Campbell and Stanley, 1963). We need to distinguish between two types of response assessments. A basic assessment tries to answer the question, "Did the problem go down?" These assessments do not attempt to determine what caused a drop in the problem. Simple pre-post designs are very practical for answering this question (Eck, 2002a). Extending the length of time for the post-intervention measures can help determine if the problem bounced back or stayed down. Advanced assessments address the question, "Did this treatment cause the problem to go down?" Interrupted time series and multiple time series designs are the most practical way to address this question. We need replications to answer the question, "Is the response generally effective or ineffective against this problem type in this context?" Replication requires that we have to compare multiple interventions to the same problem, all in a similar context. In principle, we could apply randomized experiments. But for the reasons listed above, this is not likely to be a practical solution in many situations. Generalization to other settings requires multiple interventions in multiple contexts. Meta-analysis also might be promising, but the instability of the results of such syntheses, due to unmeasured sources of variation, limit their utility (Lipsey and Wilson, 2001). It is as important to discover what does not work against a specific problem in a specific circumstance, as it is to discover what works. In general, many opportunity-blocking tactics appear to be very effective (Eck, 2002c), but we know much less about the specific contexts in which they are ineffective. It is far more useful to know that lighting, for example, is effective under conditions A, B and C but is ineffective in conditions D through G, than it is to come to some global assessment of lighting's effects on crime. We might be able to obtain positive and negative results from many simple evaluations of interventions, along with some contextual information about the problems and the responses. Unfortunately, most attempts to synthesize findings from multiple evaluations have focused on those evaluations with few methodological weaknesses and have difficulty making sense out of the many weak evaluations. From a methodological perspective, this makes a great deal of sense. But from a practical standpoint, it disposes of a great deal of information. To continue with the gold analogy: it is like only keeping the large nuggets from a gold mine and throwing the small ones and dust

-103-

John Eck

out with the tailings. If the gold seam is rich, this makes some sense. But we are working in a mine with few nuggets and a great deal of dust and there is little prospect of this changing. Consequently, we need to be able to glean bits of information from large numbers of weak studies. The task before us is to devise a method for sharing weak evaluation information among hundreds of police agencies, synthesizing this information, and coming up with robust results that can improve daily police practice. This will not be easy, but there does not seem to be an alternative. The minimal requirements are as follows. (1) A system linking hundreds of police agencies in North America, the United Kingdom, Scandinavia, Australia, and other countries where problem-oriented policing is being applied. The World Wide Web is probably the best platform for this system. (2) A centralized or distributed database of problem-solving efforts conducted by participating agencies. This requires a sponsoring agency or consortium of agencies to provide the staff to maintain the system. The system would have to be subsidized initially, but might be able to charge fees for services once it has demonstrated its utility. (3) Descriptions of problem-solving projects in two parts: a narrative and a set of quantitative descriptors. A narrative format based on the SARA process, or any similar process, would simplify submissions, collation, and dissemination. A problem-classification scheme would be the basis for indexing. (4) A standard reporting form for quantitative descriptors of problems, their contexts, how they were identified, analyzed, and responded to, along with descriptions of evaluation methods, measures of effectiveness, and results. A computer-based problem-solving protocol could automatically create such reports. (5) A process analyzing the database to create rank-ordered lists of possible responses to problems in particular contexts. It would also show what types of responses are unlikely to be effective, or even counterproductive, for particular contexts. Currently, multivariate models are used in meta-analysis, but these might not be suitable for this application. Other approaches may be possible, such as the use of artificial neural networks.
-104-

Police Problems

(6) A process for disseminating information about specific problem-solving efforts. This could be automated within a website and using e-mail. (7) A procedure for commissioning special reports from the database. None of these requirements appear to be beyond current capabilities. Development of an analytical process (item 5) to synthesize the findings may be difficult. But even here, it might be feasible to start with a crude process and upgrade it overtime as the analytical technology improves. The greatest challenge will be to assure that negative evaluation results are reported along with positive findings. Despite our interest in successful cases, they are of limited use without counterexamples to which they can be compared. This is true in biology (Dawkins, 1996; Mayer, 2001), engineering (Petroski, 1992), and in science in general (Popper, 1992). It is the unsuccessful cases that allow us see the limits of interventions, reveal where we are ignorant, stimulate us to look further, and provoke our creativity.

WHERE SHOULD WE GO?
This paper outlines a rudimentary theory of problems that draws substantially from Routine Activity Theory. This problem theory involves a hierarchy of events, problems, and problem types which is codified in a problem-classification scheme (Table 1). Events that occur in similar environments, involve similar behaviors, and are linked by the place, target, or offender, belong to the same problem. Problems comprised of behaviorally and environmentally similar events form problem types (Figure 1). Events cluster to form problems and problems cluster to form problem types. Events are of very short duration and usually occur in distinct locations. Problems are longer lasting, sometimes spanning years. Common problems are geographically-bounded. These boundaries include blocks, neighborhoods, cities, and, on occasion, regions. System problems are constrained only by the system's geography. Problem types of any sort have no temporal or geographic bounds. That is, any conclusion one draws about a particular problem type should be applicable to any problem of this type, wherever or whenever they are found. Linking problem types is a theory of problems. A theory of problems is applicable across all problem types. Problems arise from the repeated coming together of places, offenders, and targets when there

-105-

John Eck

are no capable controllers (Figure 2). The repetition stems from positive feedback to offenders and ineffective feedback to one or more of the controllers (Figure 3).

Figure 6: A Hierarchy of Interventions

The form of policing that problem-oriented policing seeks to replace — incident-driven policing — focuses most of its attention on events. To the extent that any problem solving is undertaken in incident-driven policing it is erratic and outside the day-to-day functioning of the police agency. Some agencies take on problems as they are identified (and as resources allow), but there is little consideration of the systemic processes that give rise to these problems and how problems are clustered by type. The overall goal of problemoriented policing has always been to build a knowledge base about types of problems that guides police action at the problem and event levels (Goldstein, 1979). This relationship among events, problems, problem types and the nature of police work is illustrated in Figure 6.

-106-

Police Problems

A truly problem-oriented police agency does not just do problem solving. It has systemic processes to learn from problem solving. This paper attempts to give a sense of how little we currently know, how difficult this undertaking is likely to be, and how many years must pass before we can legitimately claim we have made substantial progress. Police research should focus on the four questions examined here: What are problems? What causes problems? How can we find effective solutions? And, how can we learn from problem solving? A research agenda formed around these questions is rich with possibilities. Some of the questions that such a research agenda might address are:

What Are Problems?
(1) (2) (3) (4) (5) (6) (7) What are the appropriate dimensions? What problem types are possible? Are the types useful? What is the prevalence of problems by type? What are the symptoms of problems for each type? What causes problems? Do different problem types have different relationships among problem elements? (8) How do these relationships influence feedback? (9) How does environment influence feedback? (10) When does victim or controller feedback break down and why? (11) What preserves effective victim or controller feedback? (12) What promotes offender feedback? (13) When does offender feedback break down? (14) How do offenders adapt to victims and controllers and under what circumstances do desistance, defiance, diffusion, and displacement occur? (15) What problems are stable, and under what circumstances? (16) What problems are unstable, and under what circumstances? (17) What problems are chaotic, and under what circumstances?

-107-

John Eck

How Can We Find Effective Solutions?
(1) Do generic problem analysis protocols lead to effective solutions? (2) What questions should problem solvers ask to reach effective solutions for specific problems? (3) How do these questions vary by problem type? (4) What responses appear to be particularly suitable for specific problem types?

How Can We Learn From Problem Solving?
(1) How can a system to exchange problem-solving information be developed? (2) What form of database should this system use? (3) What is an effective way of describing problem-solving efforts that facilitates information exchange and learning? (4) Can we develop analytical process to synthesize basic assessment results and provide useful information to practitioners and researchers? (5) What would such an analytical process look like?

Address correspondence to: John Eck, University of Cincinnati, Division of Criminal Justice, P.O. Box 210389, 600 Dyer Hall, Cincinnati, OH 45221-0389 USA. E-mail: . Acknowledgments: Special thanks to my colleagues Bonnie Fisher and Graham Farrell for their stimulating conversations on many of the topics discussed here. I also owe debts to Johannes Knutsson and Ronald V. Clarke for their very useful comments, and to the other participants at the Kleivstua meeting for their help in thinking through these issues. All errors in this paper are my responsibility.

-108-

Police Problems

REFERENCES
Brantingham, P.L. and P.J. Brantingham (1981). "Notes on the Geometry of Crime." In: P.J. Brantingham and P.L. Brantingham (eds.), Environmental Criminology. Beverly Hills, CA: Sage. Campbell, D.T. and J.C. Stanley (1963). Experimental and QuasiExperimental Designs for Research. New York, NY: Houghton Mifflin. Casti, J.L. (ed.) (1997). Would-Be-Worlds: How Simulation is Changing the Frontiers of Science. New York, NY: Wiley. Clarke, R.V. (2000). "Hot Products: A New Focus for Crime Prevention." In: S. Ballintyne, K. Pease and V. McClaren (eds.), Secure Foundations: Key Issues in Crime Prevention, Crime Reduction and Community Safety. London, UK: Institute for Public Policy Research. and H. Goldstein (2002). "Reducing Theft at Construction Sites: Lessons from a Problem-Oriented Project." In: N. Tilley (ed.), Analysis for Crime Prevention. (Crime Prevention Studies, vol. 13.) Monsey, NY: Criminal Justice Press. and R. Homel (1997). "A Revised Classification of Situational Crime Prevention Techniques." In: S.P. Lab (ed.), Crime Prevention at the Crossroads. Cincinnati, OH: Anderson. and D. Weisburd (1994). "Diffusion of Crime Control Benefits: Observations on the Reverse of Displacement." In: R.V. Clarke (ed.), Crime Prevention Studies. (Crime Prevention Studies, vol. 2.) Monsey, NY: Criminal Justice Press. Cohen, L.E. and M. Felson (1979). "Social Change and Crime Rate Trends: A Routine Activity Approach." American Sociological Review 44(3):588-605. Dawkins, R. (1996). Climbing Mount Improbable. New York, NY: Norton. Eck, J.E. (2003). "Problem-Oriented Policing and Its Problems." In: W. Skogan (ed.), Community Policing: Can It Work? Belmont, CA: Wads worth. (2002a). Assessing Responses to Problems: An Introductory Guide for Police Problem-Solvers. Washington, DC: Office of Community Oriented Policing Services, U.S. Department of Justice. —- (2002b). "Learning From Experience in Problem-Oriented Policing and Situational Prevention: The Positive Functions of Weak Evaluations and the Negative Functions of Strong Ones." In: N. Tilley (ed.), Evaluation in Crime Prevention. (Crime Prevention Studies, vol. 14.) Monsey, NY: Criminal Justice Press.

-109-

John Eck (2002c). "Preventing Crime at Places." In: L.W. Sherman, D. Farrington, B. Welsh and D.L. MacKenzie (eds.), Evidence-Based Crime Prevention. New York, NY: Routledge. (2001). "Policing and Crime Event Concentration." In: R. Meier, L. Kennedy and V. Sacco (eds.), The Process and Structure of Crime: Criminal Events and Crime Analysis. New Brunswick, NJ: Transactions. (1995). "Examining Routine Activity Theory: A Review of Two Books." Justice Quarterly 12(4):783-797. (1994). "Drug Markets and Drug Places: A Case-Control Study of the Spatial Structure of Illicit Drug Dealing." Ph.D. Dissertation, Department of Criminology and Criminal Justice, University of Maryland, College Park. and R.V. Clarke (in press). "Classifying Common Police Problems: A Routine Activity Approach." In: M. Smith and D. Cornish (eds.), Theory for Practice in Situational Crime Prevention. (Crime Prevention Studies, vol. 16.) Monsey, NY: Criminal Justice Press. and W. Spelman (1987). Problem Solving: Problem-Oriented Policing In Newport News. Washington, DC: Police Executive Research Forum. Everson, S. and K. Pease (2001). "Crime Against the Same Person and Place: Detection Opportunity and Offender Targeting." In: G. Farrell and K. Pease (eds.), Repeat Victimization. (Crime Prevention Studies, vol. 12.) Monsey, NY: Criminal Justice Press. Farrell, G. and W. Sousa (2001). "Repeat Victimization and Hot Spots: The Overlap and Its Implications for Crime Control and ProblemOriented Policing." In: G. Farrell and K. Pease (eds.), Repeat Victimization. (Crime Prevention Studies, vol. 12.) Monsey, NY: Criminal Justice Press. Felson, M. (1995). "Those Who Discourage Crime." In: J.E. Eck and D. Weisburd (eds.), Crime and Place. (Crime Prevention Studies, vol. 4). Monsey, NY: Criminal Justice Press. (1987). "Routine Activities and Crime Prevention in the Developing Metropolis." Criminology 25(4):911-931. (1986). "Linking Criminal Choices, Routine Activities, Informal Control, and Criminal Outcomes." In: D. Cornish and R.V. Clarke (eds.), The Reasoning Criminal: Rational Choice Perspectives on Offending. New York, NY: Springer-Verlag. and R.V. Clarke (1998). Opportunity Makes the Thief: Practical Theory for Crime Prevention. London, UK: Research Development and Statistics Directorate, Home Office. -110-

Police Problems Goldstein, H. (1979). "Improving Policing: A Problem-Oriented Approach." Crime & Delinquency 25(2):236-258. (1990). Problem-Oriented Policing. New York, NY: McGraw-Hill. Leigh, A., T. Read and N. Tilley (1996). Problem-Oriented Policing: Brit Pop. London, UK: Home Office, Police Policy Directorate. Liang, J. (2001). "Simulating Crimes and Crime Patterns Using Cellular Automata and G1S." Ph.D. Dissertation, Department of Geography, University of Cincinnati. Lipsey, M.W. and D.B. Wilson. (2001). "The Way in Which Intervention Studies Have 'Personalities' and Why it is Important to MetaAnalysis." Evaluation and the Health Profession 24(3):236-254. Mainzer, K. (1997). Thinking In Complexity: The Complex Dynamics of Matter, Mind, and Mankind. New York, NY: Springer. Maxwell, C D . , J.D. Garner and J.A. Fagan (2001). The Effects of Arrest on Intimate Partner Violence: New Evidence from the Spouse Assault Replication Program. Washington, DC: U.S. National Institute of Justice. Mayer, E. (2001). What Evolution Is. New York, NY: Basic Books. Office of Community Oriented Policing Services (1998). Problem-solving Tips: A Guide to Reducing Crime and Disorder Through ProblemSolving Partnerships. Washington, DC: U.S. Department of Justice. Pease, K. (1998). Repeat Victimisation: Taking Stock. London, UK: Research Development and Statistics Directorate, Home Office. Petroski, H. (1992). To Engineer is Human: The Role of Failure in Successful Design. New York, NY: Vintage. Popper, K.R. (1992). Conjectures and Refutations: The Growth of Scientific Knowledge. New York, NY: Routledge. Sampson, R. (2001). Drug Dealing in Privately Owned Apartment Complexes. (Problem-Oriented Guides for Police Series, vol. 4.) Washington, DC: Office of Community Oriented Policing Services. Schwartz, M.D., W.S. DeKeseredy, D. Tait and S. Avi (2001). "Male Peer Support and a Feminist Routine Activities Theory: Understanding Sexual Assault on the College Campus." Justice Quarterly 18(3):623-650. Sherman, L., D. Farrington, B. Welsh, and D.L. MacKenzie (eds.), (2002). Evidence-Based Crime Prevention. New York, NY: Routledge. Spelman, W. (1995). "Once Bitten, Then What? Cross-sectional and Time-course Explanations of Repeat Victimization." British Journal of Criminology 35(3):366-383.

-111-

John Eck (1994a). "Criminal Careers of Public Places." In: J.E. Eck and D. Weisburd (eds.), Crime and Place. (Crime Prevention Studies, vol. 4.) Monsey, NY: Criminal Justice Press. (1994b). Criminal Incapacitation. New York, NY: Plenum. and J.E. Eck (1989). "The Police and the Delivery of Local Government Services: A Problem-Oriented Approach." In: J.J. Fyfe (ed.), Police Practice in the '90s: Key Management Issues. Washington, DC: International City Management Association. Thornberry, T.P. (1998). "Membership in Youth Gangs and Involvement in Serious and Violent Offending." In: R. Loeber and D. Farrington (eds.), Serious and Violent Juvenile Offenders: Risk Factors and Successful Interventions. Thousand Oaks, CA: Sage Publications. Townsley, M.K. (2000). "Spatial and Temporal Patterns of Burglary: Hot Spots and Repeat Victimization in an Australian Police Division." Ph.D. Dissertation, School of Criminology and Criminal Justice, Griffith University, Adelaide, Australia. Weisburd, D., C M . Lum and A. Petrosino (2001). "Does Research Design Affect Study Outcomes in Criminal Justice?" The Annals of the American Academy of Political and Social Science 578 (November):50-70. Williams, G.P. (1997). Chaos Theory Tamed. Washington, DC: Joseph Henry Press.

NOTES
1. As the events of September 11, 2001 show, the boundary between system and common problems is not clearly marked. In this series of incidents, an international crime organization was able to seize control of a system — commercial aviation — though to do so, some its members had to come into direct contact with their targets. 2. Eck and Clarke (2002) did not attempt to develop a comprehensive classification for system problems. Instead, they show how one can be developed. 3. There has been some speculation on the origins of the inner triangle (Leigh et al., 1996:18, footnote 1). It comes from the collaboration among William Spelman, Rana Sampson, and myself at the Police Executive Research Forum in the early 1990s. Sampson developed the triangle to teach police problem analysis based on earlier work (Eck and Spelman, 1989; Spelman and Eck, 1989).

-112-

Police Problems 4. This suggests that one form of discrimination can be characterized as prediction errors — false positives — that fall disproportionately on one group of people. 5. For more information on the draft protocol, contact the author at john.eck@uc.edu. 6. Comparisons of randomized experiments and non-randomized evaluations appear to show a systematic difference in results. Randomized experiments often show smaller treatment effects than their nonrandom cousins (Lipsey and Wilson, 2001; Weisburd et al., 2001). The one common explanation for this is that non-randomized studies are biased because they cannot control for all possible confounding variables. This leads to the "methods paradox." Premise 1. Randomized trials are the most valid method of drawing conclusions about interventions (from experimental theory). Systematic reviews of evaluations show meaningful differences between randomized experiments and nonrandomized studies, with the randomized results showing weaker performance for the interventions (from results of systematic analyses). Non-randomized studies are biased toward finding stronger effects than randomized studies. Systematic reviews are non-randomized studies (from examination of these studies). Differences between the randomized trials and the non-randomized studies are exaggerated by nonrandomized systematic reviews.

Premise 2.

Conclusion A. Premise 3. Conclusion B.

Conclusion B challenges the truth of premises 1 and 2 or the validity of conclusion A. Regardless of the source of the contradiction, the implication is that randomized experiments may not offer substantial improvements over non-randomized studies. The methods paradox is a variant of the ancient Greek "liar's paradox," so it can be summarized as "A nonexperimental study shows all non-experimental findings overestimate their results." Or, in the language of a recent large-scale systematic summary of evaluation research (Sherman et al., 2002), "A level 1 study shows that all studies below level 5 overestimate their results."

-113-