Contents	Introduction	Descriptive	Hypothesis	Tables	Appendix

Research Design

The objective of science is to explain reality in such a fashion so that others may develop their own conclusions based on the evidence presented. The goal of this handbook is to help you learn how to conduct a systematic approach to understanding the world around us that employs specific rules of inquiry; what is known as the scientific model.

The scientific model helps us create research that is quantifiable (measured in some fashion), verifiable (others can substantiate our findings), replicable (others can repeat the study), and defensible (provides results that are credible to others--this does not mean others have to agree with the results). For many the scientific model may seem too complex to follow, but it is often used in everyday life and should be evident in any research report, paper, or published manuscript. The corollaries of common sense and proper paper format with the scientific model are given below.

Corollaries among the Scientific Model, Common Sense, and Paper Format

Scientific Model		Common Sense		Paper Format
Research Question		Why		Intro
Develop a theory		Your answer		Intro
Identify variables		How		Method
Identify hypotheses		Expectations		Method
Test the hypotheses		Collect/analyze data		Results
Evaluate the results		What it means		Conclusion
Critical review		What it doesn’t mean		Conclusion

Overview of first four elements of the Scientific Model

The following discussion provides a very brief introduction to the first four elements of the scientific model. The elements that pertain to hypothesis testing, evaluating results, and critical review are the primary focus in Part III of the Handbook.

1) Research Question

The research question should be a clear statement about what you intend to investigate. It should be specified before research is conducted and openly stated in reporting the results. One conventional approach is to put the research question in writing in the introduction of a report starting with the phrase " The purpose of this study is . . . ." This approach forces the researcher to:

1. identify the research objective (allows others to benchmark how well the study design answers the primary goal of the research)
2. identify key abstract concepts involved in the research

Abstract concepts: The starting point for measurement. Abstract concepts are best understood as general ideas in linguistic form that help us describe reality. They range from the simple (hot, long, heavy, fast) to the more difficult (responsive, effective, fair). Abstract concepts should be evident in the research question and/or purpose statement. An example of a research question is given below along with how it might be reflected in a purpose statement.

Research Question: Is the quality of public sector and private sector employees different?

Purpose statement: The purpose of this study is to determine if the quality of public and private sector employees is different.

2) Develop Theory

A theory is one or more propositions that suggest why an event occurs. It is our view or explanation for how the world works. These propositions provide a framework for further analysis that are developed as a non-normative explanation for "What is" not "What should be." A theory should have logical integrity and includes assumptions that are based on paradigms. These paradigms are the larger frame of contemporary understanding shared by the profession and/or scientific community and are part of the core set of assumptions from which we may be basing our inquiry.

3) Identify Variables

Variables are measurable abstract concepts that help us describe relationships. This measuring of abstract concepts is referred to as operationalization. In the previous research question "Is the quality of public sector and private sector employees different?" the key abstract concepts are employee quality and employment sector. To measure "quality" we need to identify and develop a measurable representation of employee quality. Possible quality variables could be performance on a standardized intelligence test, attendance, performance evaluations, etc. The variable for employment sector seems to be fairly self-evident, but a good researcher must be very clear on how they define and measure the concepts of public and private sector employment.

Variables represent empirical indicators of an abstract concept. However, we must always assume there will be incomplete congruence between our measure and the abstract concept. Put simply, our measurement has an error component. It is unlikely to measure all aspects of an abstract concept and can best be understood by the following:

Abstract concept = indicator + error Because there is always error in our measurement, multiple measures/indicators of one abstract concept are felt to be better (valid/reliable) than one. As shown below, one would expect that as more valid indicators of an abstract concept are used the effect of the error term would decline: Abstract concept = indicator1 + indicator2 + indicator3 + error Levels of Data

There are four levels of variables. These levels are listed below in order of their precision. It is essential to be able to identify the levels of data used in a research design. They are directly associated with determining which statistical methods are most appropriate for testing research hypotheses.

Nominal: Classifies objects by type or characteristic (sex, race, models of vehicles, political jurisdictions) Properties:

1. categories are mutually exclusive (an object or characteristic can only be contained in one category of a variable)
2. no logical order

Ordinal: classifies objects by type or kind but also has some logical order (military rank, letter grades) Properties:

1. categories are mutually exclusive
2. logical order exists
3. scaled according to amount of a particular characteristic they possess

Interval: classified by type, logical order, but also requires that differences between levels of a category are equal (temperature in degrees Celsius, distance in kilometers, age in years) Properties:

1. categories are mutually exclusive
2. logical order exists
3. scaled according to amount of a particular characteristic they possess
4. differences between each level are equal
5. no zero starting point

Ratio: same as interval but has a true zero starting point (income, education, exam score). Identical to an interval-level scale except ratio level data begin with the option of total absence of the characteristic. For most purposes, we assume interval/ratio are the same. The following table provides examples of variable types:
 

Variable	Level
Country	Nominal
Letter Grade	Ordinal
Age	Ratio
Temperature	Interval

Reliability and Validity

The accuracy of our measurements are affected by reliability and validity. Reliability is the extent to which the repeated use of a measure obtains the same values when no change has occurred (can be evaluated empirically). Validity is the extent to which the operationalized variable accurately represents the abstract concept it intends to measure (cannot be confirmed empirically-it will always be in question). Reliability negatively impacts all studies but is very much a part of any methodology/operationalization of concepts. As an example, reliability can depend on who performs the measurement (i.e., subjective measures) and when, where, and how data are collected (from whom, written, verbal, time of day, season, current public events).

There are several different conceptualizations of validity. Predictive validity refers to the ability of an indicator to correctly predict (or correlate with) an outcome (e.g., GRE and performance in graduate school). Content validity is the extent to which the indicator reflects the full domain of interest (e.g., past grades only reflect one aspect of student quality). Construct validity (correlational validity) is the degree to which one measure correlates with other measures of the same abstract concept (e.g., days late or absent from work may correlate with performance ratings). Face validity evaluates whether the indicator appears to measure the abstract concept (e.g., a person's religious preference is unlikely to be a valid indicator of employee quality).

4) Identify measurable hypotheses

A hypothesis is a formal statement that presents the expected relationship between an independent and dependent variable. A dependent variable is a variable that contains variations for which we seek an explanation. An independent variable is a variable that is thought to affect (cause) variations in the dependent variable. This causation is implied when we have statistically significant associations between an independent and dependent variable but it can never be empirically proven: Proof is always an exercise in rational inference.

Association

Statistical techniques are used to explore connections between independent and dependent variables. This connection between or among variables is often referred to as association. Association is also known as covariation and can be defined as measurable changes in one variable that occur concurrently with changes in another variable. A positive association is represented by change in the same direction (income rises with education level). Negative association is represented by concurrent change in opposite directions (hours spent exercising and % body fat). Spurious associations are associations between two variables that can be better explained by a third variable. As an example, if after taking cold medication for seven days the symptoms disappear, one might assume the medication cured the illness. Most of us, however, would probably agree that the change experienced in cold symptoms are probably better explained by the passage of time rather than pharmacological effect (i.e., the cold would resolve itself in seven days irregardless of whether the medication was taken or not).

Causation

There is a difference between determining association and causation. Causation, often referred to as a relationship, cannot be proven with statistics. Statistical techniques provide evidence that a relationship exists through the use of significance testing and strength of association metrics. However, this evidence must be bolstered by an intellectual exercise that includes the theoretical basis of the research and logical assertion. The following presents the elements necessary for claiming causation:

External and Internal Validity

There are two types of study designs, experimental and quasi-experimental.

Experimental: The experimental design uses a control group and applies treatment to a second group. It provides the strongest evidence of causation through extensive controls and random assignment to remove other differences between groups. Using the evaluation of a job training program as an example, one could carefully select and randomly assign two groups of unemployed welfare recipients. One group would be provided job training and the other would not. If the two groups are similar in all other relevant characteristics, you could assume any differences between the groups employment one year later was caused by job training.

Whenever you use an experimental design, both the internal and external validity can become very important factors.

Internal validity: The extent to which accurate and unbiased association between the IV and DVs were obtained in the study group.

External validity: The extent to which the association between the IV and DV is accurate and unbiased in populations outside the study group.

Quasi-experimental: The quasi-experimental design does not have the controls employed in an experimental design (most social science research). Although internal validity is lower than can be obtained with an experimental design, external validity is generally better and a well designed study should allow for the use of statistical controls to compensate for extraneous variables.

Types of quasi-experimental design:

1. Cross-sectional study: obtained at one point in time (most surveys)
2. Case study: in-depth analysis of one entity, object, or event
3. Panel study: (cohort study) repeated cross-sectional studies over time with the same participants
4. Trend study: tracking indicator variables over a period of time (unemployment, crime, dropout rates)