12-Steps to a Research Design

Identify the resources available for the Study: time, money, expertise. We have $______ for this study. We have ____ staff days. We have ____ partner days. We have _____ participants days Key Decision CASE EXAMPLE1
STEP ONE: Determine the key research questions or hypotheses. What do you need to know? What relationships are you interested in investigating? STEP TWO: Determine very clearly what your dependent (the effect or impact you are interested in) and independent (the proposed cause of the effect) variables (are). It always easier and less costly to investigate a one-to-one relationship. However, it is often the case that we want to know either how multiple causes lead to a single effect, or to multiple effects. In this study, for example, we are looking at two dependent variables and one independent variable: HISSA/MJT groups economic security HISSA/MJT groups womens empowerment If you wish to investigate additional causal relationships, be forewarned that each additional independent or dependent variable will increase the load of burden of rigorous proof exponentially. How does this translate in terms of your plans: if you have the resources to actually assess the causal relationships between HISSA/MJT groups on two different dependent variables (effects, or impacts) among, say, 75,000 peoplethe addition of one dependent or independent variable will mean that you probably can only demonstrate such a relationship in around 20,000 people. The addition of a second variable would take it down to around 5,000 people (for the same resources you have available to you). STEP THREE: Identify crucial intervening or confounding variables. These are variables that may intrude themselves between your purported dependent and independent variables. Unless you attempt to control for them qualitatively or quantitatively you risk identifying false relationships. STEP FOUR: Define and identify specific and measurable (can be qualitative or quantitative) indicators for the dependent variable(s). Womens participation in village savings and loans will have a positive impact on womens empowerment Dependent variables: womens empowerment Independent variable: Participation in a village savings and loan program.

Urban vs. rural setting Length of time a VSL has been in existence Womens marital status Ethnicity Empowerment is defined broadly as the expansion of assets and capabilities of poor people to participate in, negotiate with, influence, control, and hold accountable the institutions that affect their lives2 and the expansion in [womens] ability to make strategic life choices in a context where this ability was previously denied to them.3 (See FY06 research framework for more specific evidence categories and core indicators for empowerment). Participating Women Husbands or male relatives of participating women Village headmen CARE staff Secondary studies

STEP FIVE: Determine data sources

This column shows a hypothetical example. It is intended to give a very concrete picture of the products from each step. However, it is important to keep in mind that the example is one in which mixed methods play an important research role, and in which quantitative data from a survey are needed to answer the research question. Purely qualitative research designs may involve a number of questions and design considerations that do not get covered in this example. 2 Deepa Narayan ed, Empowerment and Poverty Reduction: A Sourcebook. Washington, DC: World Bank, 2001. 3 Naila Kabeer, The Conditions and Consequences of Choice: Reflections on the Measurement of Womens Empowerment. UNRISD Discussion Paper 108. United Nations Research Institute for Social Development, Geneva, 1999.

STEP SIX: Determine the methods that you need in order to gather the information and data required and meet levels of rigor that will satisfy the intended audience(s) of the research. PRINCIPLE: adopt methods that are as complex as needed but simple as possible.

1.

2. 3.

4.
5.

Government statistical records (judiciary, participation in elections, pro-women policies) Local gender experts Nonparticipating men/women (comparison group) Quantitative survey that allows us to make statistically valid comparisons between participants and nonparticipants on the dependent variable (womens empowerment). Secondary data review (results of similar studies in similar contexts) Key informant interviews (on sociocultural and gender context) Semi-structured interviews with women and men (qualitative and participatory numbers4) Focus group discussions (qualitative and participatory numbers)

Think about your resources: Do you sense that they allow you to take on all four of these methods?
STEP SEVEN: Determine the overall research design strategy. Longitudinal (data will be collected at least twice over some period of time)? Cross-sectional (a single point in time)? To a very large extent, this decision is determined by your actual research questions from step one. It also can be influenced by the resources you have available. It can also be influenced by a longer term evaluation strategy that you have in mind. STEP EIGHT: Determine the appropriate sampling population. Who or what is the largest population that you wish to be able to describe and/or account for in relation to your hypothesis? Frequently, this decision is made not by the actual limits or extent of a particular program but by the resources you have available. The choice of the largest population that you wish to be able to speak authoritatively about vis--vis impact is frequently strategic and purposive. The key is to be very clear with ourselves and our stakeholders about who or what were leaving out, and whyand what population our research actually represents in terms of its findings. STEP NINE: Identify the critical sub-levels of analysis, from largest to smallest. Cross Sectional

9057 women and their households in 360 CARE village and savings and loans in one project in North Region.

Largest: CARE project Next: 360 VSLs Next: 9057 Households Smallest Analytical Unit: Individual women

Think about your resources: Do you sense that they allow you to take on all of this? Example: imagine, right now, that you know you have a 2-hour long household questionnaire that you hoped could be used in a random sample of the households. For it to be statistically valid, you would have to randomly sample around 500 of those households. Is that even possible, given your resource constraints? If not
STEP TEN: Select a sampling strategy for every level identified in step nine. There are basically two broad kinds of sample: probability samples and non-probability samples. Probability samples, also known as random samples, allow every analytical unit to have an equal chance of being selected. They allow you to generalize to a larger population. They also are best for avoiding researcher bias. Nonprobability samples, also known as purposive samples, cannot, on their own, allow you to generalize to a wider group. They are more subject to researcher bias although this can be minimized through
4

1. LARGEST: One CARE Project 2. Next Largest: 360 VSLs. Our resources do not allow us to do rigorous impact research in all of them. Well do a stratified random sample based on the intervening variables of a) length of time in program and b) urban/rural setting. Well classify all VSLs in the total population then randomly choose: -2 VSLs, urban, less than one year in program -2 VSLs, urban, more than three years in program -2 VSLs, rural, less than one year in program -2 VSLs, rural, more than three years in program

See Participation and Numbers, pla notes 47 (August 2003): 6-12.

establishing strict, objective criteria for choosing data sources. You can mix probability and nonprobability sampling strategies across methods, it needs to be emphasized. What remains very important, with nonprobability sampling (almost always used, for example, for identifying informants for in-depth interviews, focus group discussions, key informant interviews, etc.) is that you are clear and transparent about your selection criteria and that you follow those criteria to the letter. Being clear about your sampling strategy for qualitative methods is one very important form of impact research rigor frequently missing from impact assessments.

Organize one female and one male (i.e., spouse or HH head) focus group discussion in each of the eight. 3. Next largest: household. Among 8 VSLs 240 women randomly select the largest possible number that we can handle, given our budget and human resources, for the quantitative survey. The random selection of the women results in random selection, too, of their spouses or other male household head (if applicable). Sothe resource question needs to be thought of in light of the pair of interviews. For the sake of this case example, imagine weve decided we can handle 80 total quantitative surveys. We select, therefore, 5 women (and therefore their spouses or other HH head, if applicable), at random, from each of the eight VSLs. 4. Next largest: individual. Among the 8 VSLs 240 women Randomly select two per group for indepth, individual interviews. Avoid selecting the same women who participated in the quantitative survey. This random selection of women results in your random sample of men for in-depth interviews. In one of the urban and one of the rural villages, identify all HHs that are similar to participating HHs with regard to the intervening variables of marital status and ethnicity (purposive sample). Then, within that, randomly select 10 nonparticipating households for the quantitative questionnaire. Organize one FGD of nonparticipating women in each of the 8 VSL sites. Organize one FGD of nonparticipating spouses (why these numbers? Resource constraints. Its the most we can do in the time that we have).

STEP ELEVEN: Select a comparison group. Identify at least one comparison group sometimes called a control group that can represent the counterfactual, in other words, a group that has similar characteristics, contexts, and cultures as the main analytical units from STEP TEN. NOTE: formal comparison groups are frequently omitted from development impact research and the counterfactual is treated in a more qualitative manner, triangulating between data sources, using secondary data, etc. In the SII, we wish to include a comparison group if financially possible.

NOW: Calculate the actual costs of the research youve just described. Do you have the resources? Do you need to change any decisions? Frequently, at this stage, youll find you need to circle back as far as your research questions and make them more focused. Or, you might find that while you hoped to do impact research in a given project across 35 villages, you need to focus on a much smaller subset of those villages. Or, you might find that you need to reduce the scope of one or another particular method: for example, you might still go ahead with a stratified random sample HH questionnaire in all 35 villages, but only choose two for in-depth, qualitative methods.

TOTAL EFFORT FOR ABOVE STUDY A 1-hour long quantitative survey with 40 female participants = 40 hours (5 person days) A 1 hour long quantitative survey with 40 husbands or elder males in womens households = 40 hours (5 person days) 3. A 1 hour long quantitative survey with 20 comparison group women = 20 hours (2.5 person days) 4. A 1 hour long quantitative survey with 20 comparison group husbands/sr. males = 20 hours (2.5 person days) 5. In-depth interviews with 8 village headmen = three person days 6. Key informant interviews re., context = 3 person days 7. 8 focus group discussions with female participants = 3 person days 8. 8 focus group discussion with spouses of female participants = 3 person days 9. 2 focus group discussions with non-participating females = 1 person day 10. 2 focus group discussions with non-participating males = 1 person day 11. 16 in-depth interviews with female participations = 8 person days 12. 16 in-depth interviews with spouses of female participants = 8 person days 1. 2. TOTAL PERSON DAYS NEEDED FOR DATA COLLECTION: 45 BUFFER DAYS (to account for travel, logistical snags, etc.): 9 TOTAL PLANNED PERSON DAYS FOR THIS RESEARCH: 54 Size of research team: 5 # of days of field research needed FOR DATA COLLECTION: 11 Cost/day for data collection: $1200 TOTAL COST FOR DATA COLLECTION: $13,200 STOP: DO YOU HAVE ADEQUATE RESOURCES? IF NOT: YOULL NEED TO RETURN TO STEP ONE AND SCALE BACK YOUR AMBITIONS. STEP TWELVE: Apply the appropriate methods at 1. Secondary research on gender, power, politicalthe appropriate levels from step ten economic, and social context. 2. 8 VSLs and their villages: sociocultural context, secondary data, interviews with 8 customary leaders and eight key informants 3. In 8 VSLs, random sample of 40 women and their spouses/male HH heads: quantitative socioeconomic survey; related survey for husbands (if appropriate) = 80 HH surveys 4. In 8 VSLs, one focus group of VSL participants and one of their spouses = 16. 5. In 8 VSLs, individual interviews with two VSL participants = 16 in-depth interviews 6. In two of the VSL villages: 10 female and 10 male survey questionnaires as a comparison group = 40 surveys 7. In 8 VSL villages: one FGDs with nonparticipating females = 8 FGDs 8. In 8 VSL villages: one FGD with nonparticipating spouses = 8 FGDs