Chapter 17 Research and Evaluation

Abstract

The interdisciplinary field of victims' rights and services is continually developing. The "knowledge base" available through research and evaluation has seen tremendous advances. "Promising practices" recommendations are often developed and updated based on the research. Promising practices are significant to those working in the field and are of particular importance to victim service providers. A victim service provider must employ interventions with victims that are effective and efficient. Therefore, the provider has an ethical responsibility to update and acquire new skills that improve service delivery. Research provides the means of determining which interventions may have value and which may not.

This chapter reviews basic research issues and processes. Also, an extensive resource list is provided to help the reader locate materials to assist in designing and conducting research.

Learning Objectives

Upon completion of this chapter, students will understand the following concepts:

• How information about research findings can be obtained.
  
  
• The definition of basic research terms.
  
  
• Fundamental research and evaluation methods.
  
  
• How you can acquire information and technical assistance to conduct research.

Those who fall in love with practice without science are like a sailor who enters a ship without a helm or compass, and who never can be certain whither he is going.
--Leonardo da Vinci

Why Victim Advocates Need to Know More About Research and Science

Research is often viewed as a topic that is esoteric, the sole purview of "pointy headed intellectuals," and has no practical value to victim advocates. Nothing could be further from the truth. In the criminal justice and victim advocacy fields, almost everyone has strong beliefs about a host of topics from how much crime there really is, to the major causes of crime, to what types of services crime victims really need, to what is the best way to help crime victims, to whether crime victims should or should not have constitutionally protected rights. Much is known about each of these topics. Some of this informationis right; some is wrong; and unfortunately, there is often great difficulty distinguishing between which is which.

Research is nothing more than a systematic approach that is designed to help distinguish between beliefs and opinions that are supported by empirical data versus those that have no empirical support. The assumption is that if a technique has empirical support, it has a greater likelihood of being successful or helpful than one that does not. T. H. Huxley wrote: "The great tragedy of science--the slaying of a beautiful hypothesis by an ugly fact." Among other things, science and research are there to keep practitioners from falling prey to a charming idea that can actually cause people harm.

Research is necessary because it can and does address many issues in the field that have practical significance. Often there are no research articles or reports that offer immediate solutions to problems that arise or are specific to victim service providers' needs. However, quality research can provide alternative strategies, help us understand complex and puzzling problems, and help us not to fall into the trap of a beautiful opinion that has no relevance and may even make things worse for those who need help. Research also affects policy decisions that alter the way resources are allocated. Research is often used by leaders and administrators to change almost everything--the structure of organizations, the laws that govern us, and even the dollars that support our programs.

In a highly informative and entertaining book about science designed for the lay person, McCain and Segal (1969) describe science as a game that is informed by certain attitudes and played by certain rules. They make a distinction between science-based belief systems and belief systems based on dogma, and suggest that "it is the system of data-based explanation that distinguishes science from dogma." Scientists cannot accept statements unsupported by data and have the responsibility to decide, on the basis of evidence, the best explanation for a set of facts. In contrast, dogma is based on the pronouncements by people in political, religious, social, or even criminal justice authority. McCain and Segal capture the difference between science-based and dogma-based belief systems as follows:

One way of contrasting science and dogma is to say that a scientist accepts facts as a given and belief systems as tentative, whereas a dogmatist accepts the belief systems as given; facts are irrelevant (p. 31).

Victim advocates seek to learn more about crime victims and the best ways to help them. This chapter is designed to help victim service providers better utilize what scientists, researchers, and research have to offer. Since few victim advocates aspire to be scientists or researchers, the focus of this chapter is to help victim service providers become more critical consumers of research and form mutually beneficial partnerships with researchers. First, the issue of understanding research produced by others will be discussed. This will be followed by a primer on conducting research.

How VAW Research Can Inform Practice

(The following contains selected excerpts from the Executive Summary of Fostering Collaborations Between Researchers and Practitioners: Findings from Practitioner Focus Groups. A Report of the National Violence Against Women Prevention Research Center, May 2000.)

The National Violence Against Women Prevention Research Center (NVAWPRC) was established in 1998 by the Centers for Disease Control and Prevention. The Center includes a consortium of researchers and practitioners concerned with violence against women from the Medical University of South Carolina, National Crime Victims Research and Treatment Center, Charleston, SC; Wellesley College, Wellesley Centers for Women, Wellesley, MA; and University of Missouri-St. Louis, Center for Trauma Recovery, St. Louis, MO.

The fourteen practitioner focus groups were conducted in nine states and the District of Columbia and included the five sites of the 1999 National Victim Assistance Academy. Over 120 women and men participated in these two-hour long groups. While Caucasians represented the majority of participants, African Americans, Latina/Latinos, and Native Americans comprised 22 percent of the practitioner participants. The participants perform a variety of service, administrative, and training tasks. Ninety percent of the focus group participants serve domestic violence survivors; 83 percent serve sexual assault survivors; 30 percent serve batterers; and 16 percent serve sex offenders.

HOW CAN VIOLENCE AGAINST WOMEN (VAW) RESEARCH HELP PRACTITIONERS?

The focus group participants reported that they utilize research findings to help them identify "promising practices" that can help them to prevent, intervene with, and respond to victims of domestic violence and sexual assault. Practitioners said they consult research to identify models for prevention services that they can replicate in their own communities. They reported needing concise information that can help them in program planning, development and implementation. The focus group participants said their research needs were not limited only to victims. In addition, they identified a need to understand "what works" for batterer intervention and treatment programs. Practitioners stated that they need research to validate what otherwise is often considered to be "only anecdotal evidence" about violence against women.

Other important suggestions on how VAW research can help victim service practitioners included:

• Identify and meet the needs of traditionally underserved victim populations, primarily victims who are culturally diverse or who live in rural or remote areas.
  
  
• Provide vital tools to enhance funding opportunities.
  
  
• Improve victim outreach and community education efforts.
  
  
• Determine what is best for client services.
  
  
• Identify new problems, new directions and new solutions in efforts to eliminate violence against women.
  
  
• Improve school programs designed to educate children about domestic violence and sexual assault.
  
  
• Evaluate clients and programs.
  
  
• Support prevention initiatives.

WHAT QUESTIONS WOULD PRACTITIONERS LIKE TO SEE RESEARCHERS ADDRESS?

Focus group participants identified the need for research on the cycle of violence and, specifically, on the effects of witnessing violence on children. They expressed a need for understanding violence across the life-cycle and for understanding the inter-relationship of violence, substance abuse, and societal norms, values, and beliefs.

Participants suggested that research should address questions designed to help them best define and meet their clients' needs. They underscored how important it is for researchers to obtain victims' input about what needs are most important. For example, participants stated that many victims, but few researchers, were concerned about victim satisfaction with participation in the criminal justice system.

The focus group participants gave considerable attention to the question of why victims of color were less likely to access victim services. While there was a consensus that minority victims perceive a "cultural stigma" associated with seeking victim assistance, there was less knowledge as to why this might be true.

HOW WOULD ANSWERS TO THESE QUESTIONS HELP PRACTITIONERS?

Overwhelmingly, focus group participants indicated that the answers to their questions about violence against women would help them in organizational management, improving client services, and meeting victims' needs. Specifically, research could help in program planning and developing agency goals and objectives. Most evident was a strong emphasis on the need for research that will help determine "what works" to prevent and combat violence against women.

HOW CAN VAW RESEARCH STRENGTHEN THE CAPACITIES OF VICTIM ASSISTANCE PRACTITIONERS?

Advocates and practitioners wear myriad hats in addition to their primary roles as direct service providers. They are responsible for organizational management, victim outreach, community education, fund-raising, coalition building, and professional and allied professional training and education, to cite a few key duties. Many participants viewed research as a critical tool in helping them to make the best use of their limited time, and to improve their personal and professional capabilities in providing quality services to victims. Other capacity-strengthening outcomes include:

• Enhancing community education and public awareness, particularly in the area of media relations.
  
  
• Strengthening victim services by targeting populations who are in the greatest need of intervention and support.
  
  
• Strengthening efforts to educate allied professionals about violence against women.
  
  
• Enhancing personal career development.
  
  
• Strengthening efforts to raise critically needed funding, and providing a significant tool for grant writing.

FOCUS GROUP PARTICIPANTS' EXPERIENCE WITH RESEARCH

Approximately half of the focus group participants had participated in research projects relevant to violence against women, and all of the participants indicated that at one time or another they had been required to provide data on services related to violence against women.

Focus group participants had a wide range of research experiences--many had collected pre- and post-test data from clients or participants in training and educational programs. Most participants had experience with basic data collection, primarily for case management and funding purposes. There was unanimity across all focus groups that service providers disliked the processes of ongoing data collection for funding agencies. They agreed that such tasks took time away from direct services and did not seem to provide useful data for making policy decisions or convincing others of the need for more funding.

Practitioners' positive experiences resulted from participating in research projects that developed clear, mutually established goals that gave paramount consideration to victim safety and possible reactions, and that kept victim service providers informed and involved throughout the process.

Many negative experiences with research were described by the focus group participants. These negative experiences resulted primarily from what practitioners perceived as a degree of remoteness, or even arrogance, on the part of the researchers. They reported that often researchers paid insufficient attention to the effect of research on its participants (i.e., victims of violence), collaboration was non-existent, and practitioners' ideas and opinions were neither solicited nor respected.

PRACTITIONERS' PERCEPTIONS OF BARRIERS TO VAW RESEARCH

Focus group participations identified nine general barriers to VAW research:

1. Lack of resources (e.g., time and money).
   
   
2. Participation is too time consuming.
   
   
3. Lack of diversity in research topics and participants.
   
   
4. Difficulty identifying victims who are willing to participate in research projects.
   
   
5. Lack of trust between victims/service providers and the research community.
   
   
6. Individual researchers with whom practitioners had had bad experiences.
   
   
7. Need for greater collaboration among researchers so that efforts are not duplicative.
   
   
8. Need for practitioners to be actively involved in the conceptualization of research.
   
   
9. Varying "measurements" utilized by researchers that result in statistics that are "suspect."

ETHICAL ISSUES RELATED TO CONDUCTING VAW RESEARCH

By far, the most significant ethical issues that focus group participants identified were related to victim safety. Many practitioners declared that without strict guarantees of consideration to victim safety before, during and after participation in any research project, service providers would refuse to facilitate victim involvement. Several groups also identified the "timing" of victim participation in research. If a client was currently in a domestically violent environment, participants stated that special attention should be paid to protecting her privacy and promoting her sense of security.

Many focus group participants echoed one service provider's frustration with "researchers treating clients like lab rats." Similar underlying themes resonated through all the focus groups, based primarily on participants' feelings that many researchers ("unlike practitioners") lack understanding of, and empathy for, the plight of the victims involved in their research projects.

Three general recommendations emerged for researchers to minimize the risk of harming victims, service providers, and victim assistance organizations: (1) utilize practitioners as "middle persons" to provide liaison services between researchers and their clients, and to pay attention to victims' personal needs and provide support, as needed; (2) make victims feel safe and comfortable at all times in the research environment; and (3) use language that is understandable in research studies and instruments.

RESPECT FOR PRACTITIONERS

Although a researcher may respect a particular practitioner or all practitioners in theory, actions do speak louder than words. Many researchers have not followed through with behavior to indicate that they value the knowledge, skills, and experience of practitioners. The focus groups revealed many specific ways in which researchers could and should indicate their respect for practitioners, including:

• Approach practitioners as partners, valuing the knowledge they have about the needs and experiences of the women they serve.
  
  
• Understand that practitioners already have responsibilities in their work environment (i.e., becoming involved in a research project will involve added responsibility for them). Figure out how to compensate the practitioners or make the involvement in research rewarding to them prior to contacting them.
  
  
• Involve practitioners as collaborators in setting up the type of approach to research participants that will best serve the research, the agency, and the victims under study.
  
  
• Involve practitioners as much as possible in the design of the study, the interpretation of results, and the presentation or dissemination of findings.
  
  
• Offer monetary compensation to the practitioner and/or victims under study.

The findings from this NVAWPRC series of focus groups offer researchers and practitioners valuable insights into what violence against women practitioners perceive to be the benefits of, and barriers to, research. The violence against women community must examine these issues and create innovative methods to minimize barriers, maximize mutual benefits, and achieve successful collaboration.

For more information on this report, contact The National Violence Against Women Prevention Research Center (843-792-2945) www.vawprevention.org.

How to Understand Research

A comprehensive treatment of understanding empirical research is beyond the scope of this brief chapter. However, there are a few foundational tips to keep in mind about analyzing research. Victim service providers who do not feel that their current knowledge and skill level are sufficient in this area may wish to take (or re-take) a basic course in research methods and statistics. At the very least, reference should be made to text books in these areas that cover the basic terminology and techniques of empirical investigations. An extensive resource list is provided at the end of this chapter.

It is most typical to begin a research project by reviewing the work of others. This is most often found in the "Literature Review" contained in an article or report. When considering others' research, victim service providers who are less familiar with research methodology should keep the following in mind as they analyze research under consideration:

• Who is conducting or interpreting the research? It is important to know the discipline of the researchers/writers responsible for the report being reviewed. Even in this age of "interdisciplinarism," some researchers tend to find it difficult to truly think beyond their paradigms or "world views." Readers should know where the author is coming from academically, at the outset of this review of others' works.
  
  
• What are the research questions and/or the hypotheses underlying the research? Researchers begin their investigations by positing questions and hypotheses that they seek to analyze. Readers must have a good handle on what the researcher set out to find or support before they can understand the findings and conclusions.
  
  
• Victim service providers will need at least a working understanding of research methods and statistics. Victim service providers are not expected to become expert methodologists or statisticians. However, advocates should understand the basics of research methodology, variables, descriptive statistics, tests of significance, and some of the common limitations on a researcher's ability to capture true representations of actual experiences in the necessarily contrived confines of field studies.
  
  
• What can really be said about a study's findings? Victim service providers must be careful not to overstate the findings of the studies employed. For the most part, many findings can be refuted, at least to some extent by other studies. One factor in why this happens is the issue of generalization. The findings of a single study may not repeat or produce evensimilar effects in other settings or with other subjects. Obviously then, the strongest research is composed of those studies which have shown an intervention to have similar effects in other settings or situations with lots of different subjects.

BASIC RESEARCH TERMS

When people first begin to read research reports, they often encounter terms that are new. Even if readers are generally familiar with the terms in question, these terms may have a different or more refined usage in evaluation research. Spending some time learning these terms is worthwhile since they form the language of the scientific method and are used consistently to describe the results of empirical studies. The following are basic research terms.

Variable. A variable is anything that can have more than one value, that is, it is not a fixed item or event. A variable can change or vary. If something cannot vary, it is not a variable. It is usually the case that studies involve controlled observations of variables and their interrelationships. Variables can include a wide variety of factors such as victim satisfaction, different treatment outcomes, attitudes of officials toward victims, length of sentences.

There are two basic types of variables involved in research: dependent and independent. In general, an independent variable is something that influences or produces an effect on a dependent variable. The dependent variable, then, is one that depends on, or is influenced by, another variable. Generally speaking, an independent variable is the variable that is typically manipulated by the researcher to see what effects this manipulation has on the dependent variable. Of course, many times manipulation of variables is not possible, but the relationship between dependent and independent variables can be observed nonetheless in a naturally occurring manner (so called, naturalistic observation).

Study. A study is a very broad term covering just about all objective analyses of variables. Calling something a study does not necessarily imply it is a good one, however. Better studies comport with generally accepted rules regarding appropriate research methods as described below.

Subjects. Most typically, victim service providers will be interested in studies involving people. In such studies, the persons observed are called subjects. Subjects could be, for example, victims or survivors whose experience in the system or responses to treatment are being measured, or professionals whose service-providing activities are being evaluated.

Theoretical framework. All good studies begin with a theoretical framework, wherein researchers provide some insight into their general approach to the subject matter at hand. This is usually evident in the author's review of the literature where specific publications and research are cited and reviewed. From this, researchers develop a hypothesis.

Hypothesis. The hypothesis is an extremely important foundation upon which good research is conducted. A hypothesis is a declarative statement that typically expresses the relationship between variables. An example might be "Providing victim impact statements at sentencing significantly increases victim satisfaction with the criminal justice system regardless of sentencing outcomes."

Case study. A case study is a study containing observations about one subject. These studies are typically based on what is termed anecdotal evidence. A series of case studies typically provide more useful information that something of significance is happening that may merit further study. This further study may begin with a pilot study, which is a scaled down version of a major effort conducted for several purposes (for example, to test proposed measurement instruments, to hone the research methodology, and to see if there is a preliminary basis for supporting the hypothesis).

Sample study. More commonly, a sample study would be employed due to the increased inferential power of such studies. A sample study is one where only some of the individuals or events of interest to the researcher are studied so as to be able to draw conclusions about the population as a whole. The sample group is usually selected or assigned with some degree of randomness. This is done so that researchers can say that the sample is representative of the population they ultimately seek to speak about. For example, a group of individuals who have survived a significant traumatic event are randomly assigned to two or more treatment groups such as a traditional therapy approach and an eye movement and desensitization treatment group to see which ones respond better as a result of the treatment provided.

Randomized study. A randomized study is one in which subjects are assigned to different groups as randomly as possible. This may be done by flipping a coin or using a random number generator. In contrast, if the researcher decides which subjects go into which group, or if the subjects assign themselves, selection bias can cause the groups to no longer be comparable. The purpose of randomization is to represent, as best as is practicable, the entire universe of potential subjects, in this case, all crime victims. Since this is not possible, researchers attempt to assemble unbiased samples to study.

Controlled study. In a controlled study, at least two groups are compared. The experimental group receives the intervention or treatment, and the control group does not. The hypothesis is that if the samples were selected appropriately, the experimental group would be just like the control group, except for whatever the experiment provided (the intervention, sometimes called the treatment). The rationale is that any measurable differences between the groups can be attributed to the experimental intervention.

Generalizing. Assuming good research methods and appropriate statistics are employed, the results of these studies can often be generalized to larger groups with some level of confidence. As stated above, the basic rationale for a sample study is the impracticability, cost factors, or simply the impossibility of testing all potential subjects such as testing every rape victim in the country. Therefore, some smaller group is selected for study under controlled conditions andfor rigorous analysis that allows for inferences to be drawn from the sample. It is of the utmost importance that sample selection, or other methods employed, do not bias the outcomes.

Research questions-research design. The research design is based on research questions which develop from the underlying hypothesis. The research questions ask what variables can and will be manipulated and studied. A sound experimental design attempts to show a functional relationship or an interaction between two or more variables. A researcher sets out to show that changes in one variable influences or controls changes in another event. For example, do restraining orders issued on stalkers reduce violence to victims? Does having a restraining order, the independent variable, result in a reduced likelihood of the stalker hurting the victim, the dependent variable? When conducting research, the function of an experimental design is to control systematically the conditions surrounding how the independent variable and the dependent variable interact. An experimental design's primary purpose is to arrange conditions in order to rule out the possibility that some other event, rather than the independent variable, may have caused the changes to the dependent variable.

Using the above example to explore the idea of a good design over one that is not, a study could be conducted upon two groups of victims, those that have restraining orders on their stalkers and those that do not. To study the changes in rates of violence perpetrated by the stalkers on the victims, this experimental group design requires that the subjects in the group with restraining orders be identical to the group of subjects who do not have restraining orders. If all things are equal between the two groups with the exception of the restraining orders, it could be concluded that the differences in violence between the two groups is a function of the restraining order. If rates of violence are lower in the group with restraining orders, then it can be concluded that restraining orders help protect victims being stalked.

However, if the groups are not identical, then those features that make them different might account for the changes in violence and not the restraining order. For example, rather than assuring that the subjects are the same in both groups, the researcher just looks at those victims who have gotten restraining orders and those who have not. Suppose the researcher finds that the stalkers with a more violent history are the stalkers who are more likely to have restraining orders issued against them. Further, suppose that a violent history predicts future violent behavior. If the most likely stalkers to actually engage in violence are the ones who fall into the group with restraining orders, then it would not be surprising to find higher rates of violence and harm to victims in the group with restraining orders. The data would then suggest that having a restraining order increases the risk of violence to the victim. That would be an erroneous conclusion, however, because the research design was flawed--the groups were not the same. If instead, the history of the violence of the stalker was the same across both groups, then the data might have looked much different showing that restraining orders do indeed help protect the victim from harm.

Operational definitions. Research factors/variables must be clearly defined. For example, if the term "recidivism" is being used in a study, it should be defined, such as "committing another criminal or juvenile offense." Frequently, otherwise sound research is criticized dueto lack of precision in providing the operational definition of research variables. Moreover, how these variables are measured has a great impact on the success of the study. For example, is "committing another offense" measured by arrest data, conviction data, or interviews that may pick up additional violations?

Survey. A survey reports the results of a study where data were collected by way of questionnaires or interviews. Surveys can either be observational, if no intervention or treatment occurred, or can be used as pre-test and post-test measures before and after some intervention or treatment. A pre- and post-test design is among the simplest research designs. This approach simply means that some measurement is taken of a population before the experimental intervention, and then re-taken after this intervention to see if there is any significant difference. If other factors are well controlled, these differences can be, at least in large part, attributed to the experimental intervention (the introduced independent variable).

FUNDAMENTAL RESEARCH METHODS

Experimental Research Design. A survey's pre- and post-test approach is an experimental research design. The purpose of an experimental design is to provide controlled empirical comparisons when naturalistic observation alone is insufficient to answer the questions posed. Without experimental designs certain questions can never be reliably answered. There are many experimental designs used to control the interactions between the dependent and independent variables being studied. Most research texts describe these designs and how they can be used, such as Campbell and Stanley (1963) or Dixon, Bouma and Atkinson (1991).

Single subject designs. Of the many experimental research designs available, use of single subject designs has gained considerable attention in applied research over the last twenty or so years. The advantage of single subject designs is that a large number of subjects are not required to conduct highly reliable and valid research. Most victim advocates in most applied settings with proper training could conduct single subject design research studies. These types of studies focus on the effects of the independent variable as it is systematically delivered to a few subjects across time. These studies also focus on the variations that occur with each subject across time both before and after the independent variable or intervention is employed. For example, some measure of a crime victim's behavior with respect to their avoidance of events associated with the crime might be measured both before, during, and after treatment is delivered. The measurements often take place across many days or even weeks. The strength of this type of research is that it does follow the effects upon the individual subjects with repeated measures across a substantial period of days, weeks, and even months in some studies. Individual reactions and the pattern of their behavior can be assessed. Since behavior is a function of the relationship between the individual and their environment, it is not surprising that an intervention will have effects that are peculiar to each individual. Understanding these effects is important in understanding the very nature and desirability of using a particular intervention with a victim. Where the research question is actuarial in nature, large group studies are preferred. Where the research question relates to how something will affect an individual, then single subject experimental procedures may be more appropriate. If a researcher wants to talk about the population, such as the entire class of individuals with PTSD, then group experimental designs based on samples is more appropriate. However, if the researcher intends to draw conclusions about an individual, a sample is not appropriate.

Correlational studies. Correlational studies look for associations between variables. A positive correlation means that the greater variable X is, the greater one can expect variable Y to be. A negative correlation, also referred to as an inverse correlation, means that the greater variable X is, the less one can expect variable Y to be. It is important to note that correlations do not prove anything absolutely as much as they suggest a relationship. It is often said that "correlation is not causation," meaning that just because two items are associated does not mean that there is a cause and effect relationship. An example of a correlation study might involve measuring victim satisfaction with the criminal justice process and looking at the relationship between this measure and the specific courthouse or prosecutor's office that handled the victims case. The results might demonstrate that there is a relationship betweenvictim satisfaction and particular courts or prosecutor's offices. However, this does not in itself give us any real information about causation behind these results.

Prevalence/Incidence study. If the research in question is looking at the frequency of something at a particular point in time, this is called a prevalence study (such as the number of victims of violent crime per 100,000 people in the United States). If the study focuses on the frequency of something over a given period of time, it is called an incidence study (such as the number of violent crime victims in the last month). Often prevalence and incidence data are compared across time in what may be referred to as trend analysis, such as whether the number of violent crimes across certain years demonstrates a rising or falling trend.

Retrospective/Prospective study. A retrospective study looks to the past for information about the topic at hand. Often these studies involve reviewing archival data such as old arrest reports, etc. A prospective study is one which looks forward; a longitudinal (or longer-term) study may be prospective. For example, a longitudinal study of the recovery rates of victims exposed to different treatments that followed them into the future for several years would be prospective.

Blind study. A blind study means that the researchers and/or the subjects do not know which treatment group each subject is in. In a single-blind study, the subjects do not know but the researchers do. In a double-blind study, neither the researchers nor the subjects know which group the subjects are in; all information is coded, and the code is not broken until the end of the study. This helps avoid problems that occur when study participants and researchers deliberately or inadvertently contaminate study results.

Quantitative and qualitative research. So far the type of research described is known as quantitative research. However, the field of research lately has been broken into two approaches to the collection of meaningful information. Besides quantitative research there is an area of research now known as qualitative research. Quantitative research has been the predominant and most widely accepted methodology for collecting information on the world we live in during the last 100 years or so. Qualitative research has in the last twenty years become recognized as a legitimate and respected approach to understanding the relationship of humans to the world around them. This is evidenced by the dramatic increase in qualitative research publications in professional journals, the inclusion of qualitative research sections in revised editions of previously quantitative-only research textbooks, and the publication of numerous books on qualitative research methodology.

While the distinctions between the two approaches are difficult to define at certain levels, qualitative research is grounded in what Mason (1996) has described as three aspects. First, it is concerned with how the social world is interpreted, understood, and experienced. Second, it is based on data-gathering methods that are sensitive and flexible to the social context rather than rigidly standardized or structured. Third, qualitative research is based on methods of collecting data that attempt to discover the richness, complexity, and depth of the event within its social context.

For example, if a researcher was concerned about the influence of victims making impact statements to the court prior to sentencing, the researcher could investigate the problem differently using the two research approaches. From a qualitative approach, the researcher would be concerned with exploring each victim's experience during the court statement. Victims might be interviewed and asked to share everything about the experience from what they were feeling, to what thoughts they had at the time, to their own personal analysis of the experience. A quantitative researcher might explore the same event but do so with a standardized survey instrument or a set of precise questions asked exactly the same of each victim and scored in a precise and exact manner. The quantitative researcher would likely be concerned with a statistical analysis of the data afterwards while the qualitative researcher would look for similarities but typically would not use any statistics. The quantitative researcher would likely be concerned with reducing the data to representative numbers. The qualitative researcher would be concerned with describing each individual subject's experience in its depth and complexity. The quantitative researcher would not typically be concerned with more than a few variables and any individual subject data would be less important than those things that are similar across members of the group. The qualitative researcher would be concerned with a wealth of information, and the individual's experience would not be overlooked or lessened, even when commonalities between individual subjects were found; indeed, these might be seen as very instructive.

The strength of the quantitative research approach is its precision and concerns for reliable and generalizable research results. Its weakness is that it overlooks and fails to observe the context and plethora of variables affecting any situation. The strength of the qualitative approach is that it attempts to understand the individual's experience from the individual's perspective. It is less likely to overlook small but important variables. Its weakness is often its inability to be able to generalize findings to other people in other situations. However, qualitative research can explore the tapestry of experience and lead quantitative researchers to analyze, assess, and isolate variables that would never have been observed without the investigation of the qualitative researcher.

Descriptive and inferential statistics. Despite one's best efforts, it is inevitable that a discussion about research design and evaluation is likely to include some references to statistics. Often jokingly (or maybe not so lightly) referred to as "sadistics," statistics is the part of the research package that can cause the most concern to the uninitiated. However, many user-friendly statistical packages are currently available that may be loaded on most desktop PCs; often a basic understanding is enough to get the newcomer going. Indeed, only a few concepts are important to review here.

Two basic types of statistics are descriptive and inferential. Descriptive statistics describe or summarize information about a sample. Inferential statistics move beyond simple descriptions and are instructive as to what generalizations or statistical estimations can be made about the population.

The reader is already familiar, no doubt, with many basic descriptive statistics. There are three (generally known as measures of central tendencies), the mode, median, and mean. Themode is the number, item, score or other value that occurs most often. It is the most frequent occurrence in the sample. The median is the middle or midpoint of a distribution. Therefore, it is the number, item, score, or other value that has 50 percent of the others above and 50 percent of the values below it. The mean, perhaps the most often used measure of central tendency, is the average number, item, score, or other value in the distribution. It is, then, the arithmetic or mathematical center of the distribution.

There are many, many types of inferential statistics, and a full discussion is not possible here. A list of sources for obtaining more in-depth treatment can be found in Additional Resources at the end of the chapter.

Statistical significance. Statistical significance is a concept that is critical to an understanding of the generalizability of research findings. That is, how confident can one be about these findings, and how can or should these findings be used in the decision-making process? Understanding statistical outcomes is often a matter of degree of confidence in those findings, rather than an "absolute proof" versus "no proof" decision. Very often it is a matter of determining a comfort level with the "odds" that the results in question are due to the experimental manipulation (or the hypothesized naturally occurring relationship) rather than being due to some chance occurrence.

In keeping with this notion, statistical significance is expressed as the "probability" that the outcome was due to what the researcher hypothesized, versus a random outcome. This probability value is expressed in terms of p value. P values are typically <0.05 (less than the point 05 level) and <0.01 (less than the point 01 level). A value of <0.01 means that the probability that the results of the study occurred by chance is less than 1 percent. Or to phrase it another way, if one were to re-do ("replicate") the study 100 times, one would predict that in 99 cases the results would be the same. This is considered an excellent outcome. Perhaps the most often relied upon level is <0.05. This is considered solid statistical significance (the results would be replicated 95 out of 100 tries).

Sample size. Researchers are often unable to test the entire universe of subjects and must typically rely on smaller numbers of cases. A critical issue in both the research methodology and the power of any statistical findings is the size of the sample. Simply put, a larger sample helps to avoid what are called confounding variables. This simply means that there is always the possibility that something other than what was hypothesized actually produced the outcome. Careful methods, good variable measuring instruments, and other factors all contribute to a strong research design. Samples must also be of sufficient size to support the statistical significance of findings and generalizability. No doubt the reader is familiar with the phrase "statistically significant sample" being used in, for example, news reports that relay the results of national opinion polls. Some people may be surprised to learn that these samples are often in the low thousands, if that, and are being used to estimate the views of tens of millions of voters. The power of randomization and sizable samples, in concert with other methodological issues (such as whether or not the questions asked in the poll's questionnaire protocol are valid), combine to produce some strikingly dependable results.

EVALUATION RESEARCH METHODS

Among the most common applications of research methods in the victims services area is evaluating the effectiveness of a project or program. Indeed, evaluation research is not really a different or difficult area in and of itself. It is best thought of as simply research applied in the field or in a program setting. At its most fundamental level, evaluation research seeks to answer basic questions about whether or not the program is achieving its stated goals as measured in the research project.

There are many forms of evaluation research. Given the fact that many traditional experimental or "laboratory" research methods are not always possible in the "real world" setting of an ongoing victim program, a variety of innovative designs are utilized. Many of these are derived from Campbell and Stanley's seminal work Experimental and Quasi-Experimental Designs for Research (1963). This is a very important book to become familiar with, even on a basic level. In terms of specific evaluation research itself, there are several distinct categories. The reader will note that many of these are distinguished by what is being measured and when it is being measured.

Process/Impact or outcome evaluation. Service providers should understand certain distinctions between process evaluation, which investigates issues regarding the program's implementation, and impact or outcome evaluation, which looks more specifically at whether or not the program achieved its goals and had an effect on the issue at hand. For example, a process evaluation might look at how networks of service providers are formed and measure the number and intensity of these relationships. An outcome evaluation might focus on whether or not this networking actually helped victims in some way.

Empowerment evaluation. Empowerment evaluation is a model that is currently enjoying increased use in a wide variety of public and private settings. While still maintaining the utmost independence, the role of the evaluator evolves into one inclusive of collaborative functions within an open forum, and not merely one of expert-counselor. This approach involves both independent evaluation and the empowerment of management and program staff to continuously assure quality of services. This is also useful given the changing macro-contexts within which services are delivered. As Fetterman (1996) points out:

Empowerment evaluation is necessarily a collaborative group activity....As a result, the context changes: the assessment of a program's value or worth is not the end point of the evaluation--as is often the case in traditional evaluation--but part of an ongoing process of program improvement. This new context acknowledges a simple but often overlooked truth: that merit and worth are not static values. Populations shift, goals shift, knowledge about program practices and their value change, and external forces are highly unstable. By internalizing and institutionalizing self-evaluation processes and practices, a dynamic and responsive approach to evaluation can be developed to accommodate these shifts (Ibid., 5).

Staying on the Cutting Edge

Staying current in a developing field is both exciting and demanding. By virtue of its interdisciplinary nature, the crime victim area requires attention outside the primary fields of apractitioner's training. Among the fields involved in contributing to knowledge in this area are law enforcement, criminal justice, juvenile justice, criminology, corrections, psychology, social work, sociology, counseling, family studies, human services, public administration, medicine, nursing, and education.

With the ever increasing demands placed on service providers' time by heavy caseloads, staying current in a single primary area is oftentimes difficult. However, there are tools that may be employed to stay current and to better ensure the quality of crime victim services and advocacy. Much of the work in culling through the research and other literature is already being done, at least to some extent, by others. Many journals are published that contain this body of work. Some are specific to a field, such as Child Maltreatment published by the American Professional Society on the Abuse of Children, while others provide a variety of articles of interest to service providers, such as the Journal of Interpersonal Violence. One publication, Violence & Abuse Abstracts summarizes current research throughout areas of interpersonal violence and is a good starting point to see what journals are publishing materials of interest to the service provider. Victim service providers should draw upon these resources and not expend energies to re-create this work.

ACADEMIC INSTITUTIONS

Many relevant research activities may be ongoing in local colleges and universities. Victim advocates can pick up a school/course catalogue and read up on the work victim service providers are doing and the courses researchers are teaching. Victim service providers may not have taken the opportunity to reach out to learn what other victim services agencies in the area are doing, but keeping up with the latest research may reveal what others are doing in a related topic area. The following are several potential ways to work together to achieve a mutual benefit for service providers and researchers alike:

• A professor or academician may be asked to sit on an advisory board or board of directors.
  
  
• A professor or researcher may be willing to perform (or more likely have a student compile) a "literature search" or "literature review."
  
  
• Faculty members may be willing to review and critique a draft of an evaluation proposal.
  
  
• A victim services agency could offer an undergraduate or graduate student internship that would offer some quality volunteer work and provide access to the schools resources, such as on-line literature searches and academic journals.

PERIODICALS

Periodicals published by professional associations or publishing houses often have articles of current relevance. These include publications that are more substantial than the typical newsletter, but perhaps are not truly academic journals. The difficulty here typically involves the time needed to review these publications and the money needed to subscribe. Although these concerns are certainly real, the benefit to victim service providers and their agencies may well justify this resource allocation. It is important to invest these limited resources in the highest pay-off areas.

Victim advocates can begin by collecting suggestions from colleagues regarding what they are reading (or wish they had the time and money to read) and add to that list by talking to the professor(s) and their graduate student(s). Addresses should be obtained for the publications, and free sample issues requested.

Additional publications may be listed for review on a monthly or quarterly basis by visiting the library. To stay current across disciplines, victim service providers should look for periodicals that have a broad range of editors listed who represent the areas to be covered. Also, colleagues can be drawn upon to informally share information where articles of interest are brought to the attention of others to cut down on the initial work of each participant.

ON-LINE SERVICES

The power of the on-line services should not be underestimated. Specific information about on-line research is available in Chapter 21. The amount of time that can be saved in researching topics on-line can be astounding. The only caution here is to be particularly skeptical of sources found on-line if they cannot otherwise be verified as credible by the identification of author or institution such as when addresses end with <.edu> or <.gov>. While there is excellent information to be gathered from the Internet, there is a lot of pure nonsense there too. The Internet is a very powerful tool, but it is subject to abuse and manipulation. Information and references obtained from the World Wide Web should be cross-checked.

GOVERNMENTAL CLEARINGHOUSES

Various government agencies provide outstanding information clearinghouses, such as the National Criminal Justice Reference Service (NCJRS); the Office for Victims of Crime Resource Center (OVCRC) is part of NCJRS. In addition, departments such as Health and Human Services, Housing and Urban Development, and Education offer similar information services. Victim service providers should register with all applicable clearinghouses to assist in identifying innovative programs and current information.

EXPERIENCE

It is often noted that good experimental design is mastered by practice and not simply by being told the potential problems for which one should be on the lookout. The best way to keep up-to-date is to commit to conducting a small scale research project, or to writing a brief review article about some area of interest. Set reasonable, but strict, deadlines. Starting with the tips provided in this chapter, victim service providers should get input from a variety of sources and ask others to review and react to this work. No doubt the new researcher will be amazed at how much was already known, and a considerable array of additional material will probably be compiled. Victim service providers will learn much from an open-minded reception of methodological, content, and editorial feedback.

Some Final Research Reminders

Victim service providers should be mindful of a few important points:

• Correlation is Not Causation. Studies that demonstrate associations do not imply causative relationships. The fact that a correlation is found between two variables does not mean that one caused the other. The relationship may still be extremely important, depending upon the strength of the correlation, but other factors may not be measured by the study that contributed to the correlations found.
  
  
• A Significant Finding Is Not Necessarily Meaningful. Frankly, some of the research encountered will seem to go beyond its more obvious implications or otherwise seem quite esoteric. If the reader's reaction to the findings is "so what" or that the limitations on the study are so severe as to make it meaningless to work in the field, the reader is probably correct. However, victim service providers should be careful to take what may be useful from these studies that will help build on the field's knowledge.
  
  
• Be Alert to Clever Manipulations of Data. This is less of a concern in the more academic (or "refereed") journals that have high standards for their publications. However, as the publications that are being reviewed become less rigorous, or when more popular publications are being used, care must be taken to assure the validity of findings. There are many kinds of issues that arise in this way, a few of which follow:
  
  
  • Make sure that the reader has access to both the raw numbers as well as proportional representations. Readers should not rely heavily on, for example, percentage representations if there is not a good sense for the underlying data (which really should be made available). For example, two jurisdictions have claimed a 50 percent reduction in homicides in the same period. Jurisdiction A fell from 50 to 25, while jurisdiction B fell from 2 to 1. These may be equally significant depending upon the many circumstances involved, but they do represent quite different things such as a drop in the actual homicide frequency versus a percentage change in the homicide rate.

  • When data are provided graphically (for example, in graphs that show trend lines), look to see that the graph shows the zero point on the axis and, if it does not, then see if there is a good reason for this and if it is understandable as to what the data actually represent.

  • Be wary of trend data that make broad claims from either short spans of time or from two discrete points in time because manipulating the presentation of data is an easy way to limit the focus.

  • Readers should be very skeptical of claims made about the greater population at large from studies that have small sample sizes because there are limitations to the strength of estimating techniques. Studies that examine a few subjects with the purpose of examining the effects of an intervention on individual subject behavior are relevant and important. Even in these studies, however, one should be very careful in generalizing results to a larger or different population.

  • Victim service providers should be aware of misinterpretations that arise from mishandling proportions in population demographics. Even if group A and B seem to have the same absolute numbers of victims, if one group is many times the size of theother group, then their proportional representation should be stated in order to have a truer understanding of this phenomenon. (For example, two ethnic or racial groups may have the same number of homicide victims; however, only within the context of population proportion can these numbers be truly understood.)

  • Victim service providers should be skeptical, and not take research at face value. If the author is not convincing about the findings and conclusions drawn from the study, try to articulate what is wrong with the research or how it could have been done differently.
    
    

Victim service providers must be careful not to automatically discount research that simply does not happen to jive with their point of view. Research should be read to learn new things as well as to confirm current beliefs. Also, remember that no study is perfect. This is particularly true in the crime victim research area as the demands of ethical treatment of subjects and the limitations on data that can be gathered often conflict with the rigors of pure research.

As the victims' field expands, those who work in it need to keep up with an ever-increasing array of research and other published literature. It is important not to be anxious about delving into this area. Adopting the tips above will help victim service providers stay current and better ensure that their services to, and advocacy for, victims of crime will be of high quality.

Sound research should form the basis of developing sound practices that address the needs of the population of victims served. This research should be of good quality and study actual client populations in field settings whenever practical. Indeed, one's reputation, and the credibility of the field as a whole, relies to a significant degree on the field's collective ability to translate good research into quality service provision.

Research and Evaluation Self-Examination

1. Define the following:

a. Variable.

b. Operational definitions.

c. Randomized study.

d. Sampling bias.

e. Positive correlation.

2. Explain the difference between descriptive and inferential statistics.

3. List several ways in which your program could access or minimize the cost of research and evaluation services.

4. List and discuss three "clever data manipulations" to be wary of.

5. Describe the difference between the qualitative research approach and the quantitative research approach.

NVAA LEARNING EXERCISE

Chapter 17: Research and Evaluation

Brief Description of Exercise/Activity: In the United States, traditional measures of success for public safety and justice programs have primarily been measurements such as crime rates, reduction in crime, lower rates of recidivism, and numbers of cases processed. In this exercise, facilitators lead a group discussion about identifying new measurements for program and performance evaluations that directly relate to victim services and community involvement, using the facilitator probes for guidance. A student volunteer can assist the facilitator by recording students' responses on tear sheets.

Goals of Activity:

1. To analyze existing ways that program evaluation is conducted in the justice/public safety arenas.

2. To promote new thinking among students relevant to program evaluation and performance measures that are victim-specific.

NVAA LEARNING EXERCISE:

RESEARCH AND EVALUATION

FACULTY PROBES

VICTIM- AND COMMUNITY-SPECIFIC

PROGRAM EVALUATION MEASURES

• Does the agency's and/or programs's mission or value statement incorporate language that addresses victims as clients and victims' needs?
  
  
  
  
• Are there staff and/or volunteers designated to provide victims with assistance, information, and referral either in a full-time capacity or as a percentage of one's job requirements? Do job descriptions and/or duty statements clarify specific responsibilities to victims?
  
  
  
  
• Does the agency or initiative utilize victims and service providers in an advisory capacity to guide the development of policies, procedures and programs, and related victim services? What is the structure of the Victim Advisory Council or Community Advisory Council? Has its recommendations and efforts had a measurable impact on program implementation, the delivery of victim services, etc.?
  
  
  
  
• Does the agency or program provide orientation, continuing education, and training to its staff and volunteers about crime victims' needs, victims' rights, victim trauma, and supportive services available to assist victims? If so, how often are training programs held? How many professionals and volunteers are trained? Are participants' retention and application of what they learned measured at follow-on intervals?
  
  
  
  
• Are victims aware of their rights and available services? What are the scope and specific activities/products utilized to inform and involve victims? Are these services and products available in multiple languages, TTY, and Braille and in measures commensurate with victims' ages and cognitive development? Are they available in jurisdictions large and small, urban and rural?
  
  
  
  
• Can an increase in the number of victims who are accessing their core rights be determined? Such increases include the number of crime victims who request (and receive):
  
  
  • Notification of the case or offender status.
    
    
  • Restitution and/or other financial/legal obligations.
    
    
  • Protective orders or other measures to increase victim safety and security.
    
    
  • The right to submit a victim impact statement, written, oral, through audio/video tape, and/or by teleconference to the hearing site.
    
    
  • Information about and referrals to supportive services, offered either in the community or by the criminal or juvenile justice system.
    
    

• Does the agency or initiative measure victim satisfaction with its programs and services, either through victim satisfaction surveys, focus groups, or direct victim interviews? What are the cumulative results of victim satisfaction assessments?
  
  
  
  
• Can any reduction in victims' short- and/or long-term trauma be directly attributed to their participation in victim assistance processes and programs?
  
  
  
  
• How many community volunteers are involved in victim assistance initiatives? How many community volunteer hours are performed?