1.

Basic Elements of Research Design

Research design provides the plan and structure enabling the researcher ‘to answer
research guestions as validly, objectively, rately and economically as possible" [Kerlinger,
1973]. Therefore, the first and perhaps most important step is carefully to delineate what
questions are to be answered, and specifically, to identify the particular factor or variable
we want to understand (i.e., the dependent variable).
The second step involves selecting the independent variables that go into the development
of our hypothesis. The third step is to neutralize the effects of as many rival hypotheses as
possible, by controlling the effect of extraneous variables (i.e., variables that may have an
impact on the dependent variable but are not of immediate interest in the study. In other
words, we want to know the effect of only the independent variables to be causing any
variation and confounding our interpretation of the effect. The objectives of a good
research design are, in general, the following:
(a) to be able to determine the extent to which varying the independent variable(s) causes
variation in the dependent variable;
(b) to minimize the variation in the dependent variable attributable to variables not
included in the experiment or design;
(c) to control or minimize random errorr .
In addition, of course, a research design should include detailed specifications of the
observations to be made, the time when they should be made, the appropriate statistical
analysis, and considerations of the validity of the results and the inferences drawn from
those results.
But the research design that enables us to see most clearly the effects of the independent
variables on the dependent variable generally includes at least two groups which are
similar on all factors and di mensions except for the factor or dimension representing the
independent variable(s) of interest.
For example, consider a possible research problem concerning the evaluation of the effect
of qualified audit opinions ( the independ variable) on security prices (the dependent
variable). The researcher could select a sample of firms which qualified audit opinions
(this would be the experimental sample since contain independent variable upon which the
hypothesis is based), and also select an other sample of firms similar to the experimental
sample in every characteristic that could be relevant in determining security equilibrium
prices except that it did not have qualified audit opinions (this sample is usually called a
comparison or a control samp!e). A more complete definition of control samples and
major approaches to selecting such samples are explicated in the next section.
2. Experimental and Quasi-experimental Research Designs
The difference between a so-called ‘true’ experiment and a ‘quasi-experiment’ is not one
of value judgment, the first good, the second bad. It is a convenient way to differentiate
between two types of processes underlying the generation of data. In general, the
researcher is using an ex-post facto or quasi-experimental design data come from an
environment which existed or that occurred without his direct intervention. On the other
hand, he/she is said to be a conducting a true experiment if he is it able to manipulate
explicitly one or more independent variables, and (iii) to assign (if not select and assign)
subjects to control or experimental groups (basically to achieve randomization). As the
design departs from either of these two attributes, the nature of the investigation is shifting
more toward the quasi-experimental type.
The respective advantages, disadvantages and certain tradeoffs of true-experiments versus
quasi-experiment are discussed below:
First element ; The direct manipulative control of the experimenter over the variables
makes it easier to more fully satisfy the conditions of causality (discussed in the previous
chapter) under a true-experiment setting than a quasi-experimental one.
Second element: The method of controlling (neutralizing) the effect of extraneous (non-
experimental) variables on the dependent variable is generally quite true versus quasi-
experiments. Theoretically, a greater degree of control is possible in a true-experiment.
The most efficient method of minimizing (if not eliminating the influence of extraneous
variables is via a thorough randomization of experimental units (subjects) to experimental
conditions. In its purest form, randomization involves a random selection of subjects
(individ uals, firms, etc.) from the general population of interest and then as signing these
subjects, again randomly, to the various groups or conditions (e.g., bankruptcy versus non-
bankruptcy or qualified opinion versus unqualified opinion). Hence, a thorough
randomization (with a large enough total sample) can achieve a result that renders the
experimental group or sample to be similar to the control sample with respect to all
(known or unknown) extraneous variables (except the independent variable of interes).
Note that the above is a statistical (probabilistic) statement based upon the central limit
theorem. However, in fact, experimental and control samples may be slightly different
with respect to values of the extraneous variables, even under the best of circumstances.
Hence, even for true-experiments with adequate randomization, reproducibility and
triangulation are concepts that are still valid for the purpose of evaluating ultimate
causality. The type of randomization described above requires that the perimenter have
complete manipulative control over the independent variable(s) i.e., the setting should be a
true-experiment). Hence, this method of neutralizing the effects of extraneous variables
cannot be used when the research problem is concerned with variables in real en
vironments that can be observed but not manipulated. When operating under such a quasi-
experimental setting, different means are available for controlling the effects of extraneous
variables. One method consists of pair matching each unit (say firm) in the experimental
sample (say firms with qualified audit opinions) with another firm that is different on the
dimension of the independent variable )i.e.,a firm that does not have a qualified opinion),
but similar on all other dimensions identified as important (say industry membership. size
of firm, covariation of the firms security returns with those of the overall capital market,
rate of growth, etc.). Pair-matching is a deliberate attempt at neutralizing the effects of
extraneous variables that have been identified, hence important but unknown variables
may remain uncontrolled (unequalized) and contribute a confounding effect on the
dependent variable. Kerlinger (1973] cautions that matching is useful only if the
correlation between the matched variable and the dependent variable is ‘substantial,"
hence matching variables should pass the correlation test before they can be justifiably
used. In many studies, however, matching variables are identified on the basis of intuitive
appeal (that is, presumed correlation) .
A second method of control used in quasi-experimental research is some type of limited
randomization. The independent variable usually has already "occurred," that is, the
experimenter cannot influence value or category (e.g., a firm either has or has not received
a qualified opinion). The only degrees of freedom left to the experimenter are ‘randomly"
to select the experimental sample from the sub-population of firms that have received
qualified opinions and/or also ‘randomly’ to select the control sample from the other sub-
population of firms that did not receive qualified opinions. This type of limited
randomization can do absolutely nothing to equalize (hence neutralize) the effects of
extraneous variables, except by chance. Obviously, if are biases across the sub-populations
themselves, then random selection from within each sub-population cannot compensate for
such biases (usually termed as sample self-selection biases). Many experimenters seem not
to realize that such limited randomization is a very poor control procedure. Pair-matching,
if properly done, is a much better approach.
Given that limited randomization does not achieve much, and that exact pair matching may
not always be possible for all important identified dimensions, an alternative means of
statistical control is available. Here, the identified extraneous variables can be explicitly
included and made part of the formal model as additional independent variables. Thus
statistical techniques such as multiple regression, analysiss of variance or covariance, among
others, can be looked upon as effectively simulating the experimental procedure of
manipulating independent variables, controlling extraneous variables and probing the data for
causal dependencies. of course, the experimenter needs to have a good feel for the functional
relationship between the extraneous variable(s) and the dependent variable before using the
above approach Some generalizations of data collection cost and benefit are possible. Since
quasi-experimental research relies heavily on collecting and analyzing real world
observations, multiple comparisons or control groups are available at very little cost. By
contrast, generating additional comparison groups in a true-experimental setting is bound to
be more costly.

Examples of research from the accounting literature which come close to a true experiment
include the studies by Dyckman [1964] Pankoff and Virgil (1970), Barefield (1972],
Dickhaut l1973l and Ronen (1973, Libby (1975, Joyce [1976, Savich (1977l, Hofstedt and
Hughes (1977], and Uecker (1978].

Illustrations from the accounting literature which utilize quasi-experimental research include
the works authored by Beaver (1968) Deakin [1972), Gonedes, Dopuch, and Penman [1976],
Harrison Foster (1977, Ro (1978, Albrecht, Lookabill, and McKeown [1977], and Dhaliwal
[1978].

Third element: True-experiment designs typically rate higher on internal validity though
probably lower on external validity than quasi-experimental designs.

Internal validity, inbrief, is a measure of the extent to which variation in the dependent
variable can be really attributed to (or said to be caused by) variations in the independent
variables included in the study that is, how valid is the inference that causality exists.
External validity describes the extent to which the results of any one study are generalizable
to other samples, time periods, events, and so forth. Campbell and Stanley [1963 maintain
that internal validity is sine qua non research design, but that the ideal design should be
relatively strong on both internal and external dimensions. The need for strength in both areas
is especially true in research in applied fields such as accounting, since this and other
utilitarian disciplines place high priority on the transferability of the results to other persons,
programs, and situations. The importance of such generalizability may be less in other types
of research. Kerlinger [1973] asserts that in basic research, for example, the focus of interest
lies in discovering the rela tionships between a few fundamental variables, and transferability
is not important .

The nine threats to internal validily are generally stated in the following form (adapted from
Cafmpbell and Stanley (1963)

l. History: Events occurring between pretest and posttest, other than the experimental
treatment, may provide alternate explanations of effects

2. Maturation: Processes within the respondents or socio-economic units may produce

changes as a function of time per se, such as growth, fatigue, secular trends, learning, etc

3. Instability: Unreliability of measures, fluctuations in sampling persons or components,

autonomous instability of repeated or "equivalent" measures

4. Testing: The effect of taking a test upon the scores of a second test: the effect of
publication of a social or economic indicator upon subsequent readings ofthe indicator

5. Instrumentation: The non-equivalence of measures that are presumed to be equivalent;

changes in the calibration of a measuring instrument or changes in the observers may produce
changes in the obtained measurements

6. Regression artifacts: Psuedo-shifts occurring when persons or treatment units have been
selected upon the basis of their extreme initial scores

7. Selection: Biases resulting from differential recruitment of comparison groups or self-

selected" groups, producing different mean levels on the measure of effects

8. Experimental mortality: The differential biases resulting in differential rates of maturation

or autonomous change

We view all these conditions as different types of instability and we prefer to use more
simplistic terms such as

I. the lack of stability or neutrality of conditions prior to and following the occurrence of the
experimental event;
2. the instability of processes taking place within the experimental setting such as learning
and fatigue;

3. changes in the reliability of the measurements and of the calibration (measurement ability)
of the instrument used in the study and

4. changes in the conditions of the experimenter and/or the subjects, especially if the study is
extended over a long period of time and over varying conditions.

Major causes of threats to external validity (or what might be called external invalidity)
include:

1. changes in the representativeness of the sample due to performing pretests;

2. biases stemming from self selection;

3, the creation of unintended experimental effects due to contrived conditions (the Hawthorne
effect);

4. irrelevance of the independent variables (or treatments) or of their applications; and

5. the motivation of the subjects.

These problems with generalizability essentially question the adequacy and

representativeness of the experimental units, measures, treatments, and environment for the
purpose of extending the results of the research to larger populations of interest. It should be
emphasized that both internal and external validity problems are "design" problems, quite
distinct from other sources of possible invalidity such as logical inconsistencies that may
occur in the derivation of testable hypotheses from underlying theory, faulty definitions of
operational variables, inadequate strength of experimental manipulation or treat ment,
possible misuse of statistics, or incorrect or "dirty" data.

Another potential disadvantage of the true experiment design that must be mentioned in
passing is its possible ethical misuse. For example, income smoothing has been a favorite
topic in accounting re search, but we still do not know if managers do in fact take steps to
smooth accounting income artificially. Now, consider the ethical implications of placing top
executives in a laboratory under the control of a "lie-detecting" machine in order to observe
their reactions as they are asked the question: "Do you, Mr. Manager, manipulate accounting
policies in order to smooth accounting income?" Although this may sound like a far fetched
proposition, one of these authors has been asked once by another professor how he could
utilize the lie-detecting machine that his university just acquired in conducting accounting
research!

Undoubtedly, such etchical hazards often restrict the type of research design used. The
difficulty in performing this type of experimentation has led accountants to use only the
quasi-expermental type of methodology in evaluating the income smoothing hypothesis
3. Specific Forms of Design
in the previous section we saw that the basic distinction between the true experiment and
the quasi-experiment or ex-post-facto research design involve differing abilities for the
hmanipulation of independent variables, the selection of a comparison or control group,
the randomization of extraneous variables, and the trade-off between the strength internal
and external validity. Another difference, considered in a later section, concerns the
sources of data required by each. Nonetheless, despite these real differences, the basic
design framework is quite similar for both the true experiment and the quasi-experiment.
The four designs presented in Exhibit 1 capture the underlying structure needed for the
true experiment and its analogous quasi-experiment counterpart. To facilitate the
presentation, the symbols will be used throughout the discussion of all four designs.
As an example, consider the article by Libby (1975], which is analyzed in more detail in
Part II. Briefly, he used a sample of bankrupt firms as his experimental sample and a
sample of firms that did not fail as his control sample, and in both cases he requested loan
officers (subjects) to classify the firms as "failed" or "not failed" on the basis of five
financial ratios which were computed for each firm prior to failure dates. The
experimental and control samples were randomized before subjects made the judgments.
this study
X = financial ratios of bankrupt firms firms
X' = financial ratios of the no-failure firms (for the control group)
Ya = predictions of failure or no failure by loan officers after using the ratios (dependent
variable)
Yb= is not permitted in the study (it is a posttest only design discussed below)
Ob = observations of the classification as to failure or no failure as actually occurred in the
real world prior to performing the experiment. (Comparison of Ya and Ob evaluates quality
of judgments by loan officers.)
 Exhibit 1
Symbolic Notation
for the quasi- experiment or ex-post facto design
for the true experiment
Description
One level of the independent variable which
(a) was manipulated by the experimenter,
b) has occurred naturally without the intervention of the experimenter
Another level of the independent variable (including zero level) which
(a) was manipulated by the experimenter,
(b) has occurred naturally
Observations of the dependent variable (
a) in a true-experiment,
(b) in a natural setting
Random assignment (or selection and assignment) of subjects to experimental conditions
(i.e., control group formed via thorough randomization)
Pair matching or limited randomization to form the comparison or control group
indices designating the time referent of the dependent variable observations in relation to the
occurrence of the independent variable, i.e., before or aftcr
An example of a quasi-experiment with control is the study by Harrison [1977, which is also
analyzed in Part II. Harrison's study has four experimental subsamples consisting of firms
making accounting changes which were either discretionary or non-discretionary and which
resulted in an increase or in a decrease in reported income. He pair-matched each firm with
another that did not make accounting changes. He then used risk adjusted rates of return to
evaluate the hypothesis that the effects of accounting changes on security prices are
conditional on whether or not the changes are discretionary and on the impact of the change
in income.
In this study,
Z = accounting change as classified for each subsample (i.e., one level of the independent
variable)
Z ‘ = no accounting change for each of the control firms (i.e., another level of the
independent variable)
Ob = observed risk adjusted security returns before the accounting changes were made
(dependent variable)
Oa = observed risk adjusted security returns after the accounting changes were made
(dependent variable)
P = the nature of the control was pair matching of firms.
The above study is an example of the pretest-posttest design for quasi-experiments .
Example 1: The Pretest-Posttest Design:
In this design (displayed in Figure Four), the dependent variable is observed both prior and
subsequent to the occurrence or manipulation of the independent variable. The experimental
group is compared with a control (or comparison) group. Since the dependent variable of the
control group is also observed at periods or states corresponding to the observations made on
the experimental group (except for the occurrence of the independent variable), this design is
not sensitive to differences in events between pretest and posttest (e.g., history). Nor is it
sensitive to maturation effects associated with learning, fatigue, etc. Thus, history and
maturation types of instability do not threaten internal validity here since experimental and
control groups are equally affected by these factors. In the true-experiment version,
randomization considerably strengthens internal validity by neutralizing the effects of
extraneous factors. But the existence of the have an adverse effect by sensitizing to the nature
of the pre and thus possibly reduce the validity of the results. The need for a pretest arises in
situations where the temporal change in the behavior the dependent variable is of scientific
interest, or where time series properties themselves constitute the dependent variable. with
respect to the quasi-experimental versionof this designs in empirical studies. In many cases,
the term "control group" is uscd when it really implies only "comparison group such as a
sample derived through pair matching. The "comparison group becomes synonymous with a
true control group only when thorough so that random selection of the sample and the
random assignment of subjects to experimental conditions statistically "controls" against the
admissibility of rival hypotheses. Under quasi-experimental situations, the "comparison
group" can rarely be a full-fledged "control group." Nevertheless, the latter term is often used
for descriptive purposes.
It is difficult to generalize about the specific threats to internal validity in the quasi-
experimental design presented above without a feel for the specific research problem and the
variables involved. However, a common threat arises through the ‘self selection’bias
(referred to earlier) where experimental and comparison grou may have selected themselves
into groups on the basis of characteristics other than those of interest to the researcher. Self-
selection can lead to the two groups possessing different characteristics which are extraneous
to the research problem but which can differentially influence the dependent variable under
study. Matching cannot completely obviate this bias. Generally, one must exercise care in
anticipating threats to either external or internal validity resulting from biases caused by
selfselection, interaction between selection and maturation, and differental mortality rates
between the experimental and control groups. To the extent that some matching has been
achieved, the threats due to changing events (history), learning (maturation) and prior testing
can be partially obviated. Finally, note that the quasi-experimental design illustrated above
(and in subsequent illustrations) is the "best" counterpart of the true-experimental design
jointly presented with it . In actual studies, one may encounter more inferior versions where
there may be no attempt made to create a control or comparison.
Given the difficulty of justifying the pretest on a theoretical basis the design shown in Figure
Five does away with the pretest found in example one of both versions. For the true-
experiment version, if thorough randomization can be achieved, analysis of the posttest
observations (Ya) for both the experimental and control groups should provide sufficient
grounds for making inferences concerning the effect of the independent variable. For the
quasi-experiment version of the design, the comments on the pretest-posttest design presented
in Figure Four apply here fully. Most accounting experiments are of the posttest only type
presented in Figure Five.
example 3: the Solomon Four Group Empirical design
This is essentially a combination of the first two designs shown in Figures Four and Five and,
hence, has more potent controls and avoids many of the problems cited earlier. Control or
Comparison Group 2 is really more of a control group than another experimental group. Its
purpose is to check for possible invalidity that can result from sensitizing subjects by the
pretest, especially in the true-experiment version. Even though this design is logically
attractive, it is generally doubtful that the extra cost and effort can justify its use. We only
know of one study in accounting which has used this design (Keys [1978). Also, non-design
sources of threats to external validity are not controlled for.
Example 4: The Interrupted Time-Series Design
In socio-economic settings, the impact of abrupt, discontinuous events such as policy
proclamations and rule changes can be assessed with the design shown in Figure Seven.
The long-term past behavior of the dependent variable itself is used as the benchmark
against which to assess the discontinuity associated with a specific policy. The quasi-
experimental version is more likely to fit real situations of this type (the true-experimental
version is depicted here mainly for completeness of presentation). The question addressed
in this design is whether the event of interest had an intervention effect, that is, whether
the afterevent observations represent an undisturbed continuation of the time series as
observed prior to the occurrence of event or intervention. Knowledge of the structure
(parameters of the series is paramount in assessing whether or not an effect is real. Thus,
discontinuities may represent shifts in mean levels, trends, cycles or autoregressive
characteristics. A careful delineation of thc scries-model and the choice of the appropriate
statistical tests are essential in order to avoid spurious inferences. This type of analysis has
been extensively used in accounting and finance research, especially in the research on
information content of alternative disclosure policies and security prices. Most cumulative
average residual (CAR) studies, beta change studies, and time-series model change studies
(Gonedes and Dopuch [1977]) fall within this general category, except that many of them
do not adopt the comparison group concept. Most studies rely only on the pretest posttest
comparison of the experimental group. In some others, only the after-event series are
compared between experimental and control groups. Such truncations can only add to the
other threats to internal validity associated with quasi-experimental designs. It should be
noted that in certain cases, where the entire population is affected by the same treatment
(e.g., FASB or SEC mandated nondiscretionary disclosure requirements that affect
virtually all firms), searching for a control group may not be fruitful and the design
necessarily becomes confined to only one experimental group, with its heightened
potential for invalid inferences.
The four design examples presented above capture the basic ideas of design -
implementing the concept of control, and watching out for the threats to internal and
external validity as well as demonstrate the commonalities and distinctions between
experimental and quasi-experimental research .
Each of the four designs can be combined variously into other designs. For this purpose,
we refer the reader to Campbell and Stanley [1963], Keerlinger (1973, or other similar
books.
It is our belief that given the serious threats to valid generalizations or inferences from
experimental or quasi-experimental investigations,it is incumbent upon the researcher to
exercise care in assessing for himself or herself, as well as in communicating to the
readers, possible sources of threats to internal or external validity for which there was no
adequate control. This information will enable other researchers to devise and execute
replications and extensions that may improve the earlier design and perhaps more
importantly, it will enable the reader to form his or her own inferences suitably tempered
by such information.
 4. Methods of Generating Collecting Data .
One important distinction between true and quasi-experimental research lies in the manner
in which the observations were generated. Indeed, all research may be alternatively
classified by using this criterion along with the following related dimensions:
(a) Degree of realism of the stimulus setting (both task and environment) in which the
response occurs
(b) The degree of control which the researcher has over the presentation of the
independent variable.
(c) overt or covert participation of the researcher in gathering the data
(d) whether or not data is generated specifically for the study or is otherwise
independently collected from already existing material.
The four most common types of research studies, classified by the method of generating and
collecting data suggested in the above dimensions, are discussed briefly below.
I. Laboratory Experiments: The primary feature of laboratory experiments is the
possibility of nearly complete control over the setting and the experiment itself by
elimination or control of the influences of a large number of extraneous variables.
This is done by physical isolation of the research situation from the distractions of
real environments. Thorough randomization is possible and the researcher can
explicitly administer or manipulate one or more independent variables. Laboratory
experiments are the closest practical operationalization of the ideal of a true-
experimental design. Hence, internal validity is generally at its highest possible
level. But the realism given up in structuring the laboratory experiment makes
generalization of the results a problem, especially in social, psychological, or
economic investigations (including accounting). Typically, under such sterile
conditions, the independent variables lack strength and may induce only weak
responses. Representative examples of laboratory experiments in the accounting
literature are the studies by Barefield U1972) and Dickhaut (1973].
2. Field Experiments: Here there is an attempt to capture more of the realism m of
actual settings in order to improve the compared to labo. mental effects, and the
degree of generalizability as compared to laboratory experiments, by using more
representative subjects, tasks, environments, etc for conducting the research.
Independent variable(s) can be manipulated by the researcher and some degree of
randomization may be possible. The availability of these control devices makes
the field experiment different from the laboratory experiment only in degree and it
is sometimes difficult to distinguish clearly between the two. It is particularly
appropriate for researching problems directly concerned with complex socio-
economic and real-life situations. On the negative side, the practical exigencies
and limitations of the field situation may make it difficult to explicitly manipulate
some crucial independent variable(s), and randomization may not be as thorough
as that of the laboratory experiment. These latter may sometimes move field
experiments from the field experiments. These latter aspects may sometimes move
field experiments from the category of experimental to that of quasi-experimental.
Examples of field experiments include Libby (1975, and Abdel-khalik and
keller[1978]
3. Field Studies: Sometimes it is necessary to study real socio-economic systems
with all the attendant complex relations and interactions. Usually, explicit
manipulation of any independent variable is completely outside the researcher's
control and so is any scope for randomization. Such studies properly belong to the
quasi experimental class of designs. Also, the focus of such inquiries can
sometimes be different from the usual hypothesis testing concern of laboratory
and field experiments, in that field studies can be directed at either hypothesis
generation or hypothesis testing (Katz (1953]). The hypothesis generation type of
exploratory study attempts to discover significant and the relations among them.
As a step in grounded theory formulation it comes prior to later, systematic
testing. Since field studies are conducted in a more realistic environment, external
validty and the practical significance of the results are high. On the other hand, the
low level of explicit control (which may be somewhat mitigated by matched
comparison groups) makes it very difficult to draw theoretical and causal
inferences. The existence of large number of variables and uncontrolled "noise"
complicates the task of separating systematic variance from these other sources. It
may be noted that a large proportion of empirical financial accounting studies fall
within this category. Representative examples are studies by Patell U1976, Brown
and Rozeff [1978, and Gonedes [1978]
4. Survey Research. This type of study involves properly selecting and studying
samples chosen from populations to discover the relative incidence, distribution,
and interrelation of social, psychological and economic variables. The data
collected typically relate to facts, opinions and beliefs, attitudes, and behavior.
The most common data collection methods used are personal interview and mail
or telephone questionnaire. Even though control is not possible in terms of
manipulating independent variables, this may be somewhat compensated for by
the use of careful and rigorous sampling procedures and random selection of the
units surveyed. Also, certain adanced multivariate methods of statistical analysis
can be used in conjunction with the survey method to try to get at some aspects of
the causal relationships, even though unequivocal inferences are not possible
given the virtual lack of controls. Thus, survey research properly belongs in the
quasi experimental category, and its typical applications are in fact-finding and
hypothesis generation rather than theory testing or verification (though the latter is
possible if tentative hypotheses already exist).
Campbell and Katona U1953l have described the sequence of tasks involved in
conducting and analyzing a survey as follows: First, a statement of general
objectives is necessary to delineate the purpose and scope of the survey. Next, the
specific type of data to be collected is defined with a view to operationalize the
variables of interest. Then, the population to be sampled is defined and the method
of sampling is very carefully specified. A weighted or unweighted random
sampling procedure is used depending upon whether particular subgroups of a
population have special relevance to the objectives of the study. Once the
sampling procedure is determined, the method of approaching or contacting the
subject is decided and the questionnaire is developed. The characteristics of the
person making the contact, as well as the structure, focus, and phraseology of the
questions, can have a profound influence (bias) on the responses of the subjects.
Interviewers should be trained and the questionnaire should be extensively
pretested in order to climinate the noticeable effects of such biases. After the raw
data are collected, usually coding is necessary in order to classify the responses
into numerical or content categories prior to subjecting the data to statistical
summarization and analysis.
Survey research enables the collection of large amounts of infor mation that can
be highly accurate (provided biases and sampling error have been deliberately
controlled), hence, generalizability to the population is quite high. But because the
type of information that can be gathered accurately is of the surface or superficial
variety, surveys are usually not suited for the intensive study of a phenomenon. In
order to derive valid information from a survey, the investigator needs extensive
sophistication in sampling techniques, questionnaire construction and
 administration, and analysis of data. Examples of survey research in accounting
are the studies by Estes U1969] Brenner 1971], Chandra (1974, Benston and
Krasney (1978, and Mayer Sommer (1979].
5. Review of Quality Criteria
The two primary criteria for evaluating the quality of research have been described earlier.
These are:
(a) internal validity, which covers all conditions necessary to ensure that the observed
variations in the dependent variable a "caused" by variation in the independent variables
(or treatments) and
(b) external validity, which is concerned with the conditions necessary to generalize the
results of a particular research to other sample or settings in the same population or to
other populations.
As was mentioned earlier, whenever a research design emphasizes one type of validity, it
necessarily weakens the other. How does the re searcher go about deciding what trade-off
to make between the elements of internal and external validity? To answer this question,
let us look at the two extremes. If the research project is so weak on the elements of
internal validity that the independent variable explains a very tiny fraction of the variation
in the dependent variable, then generalization would not make any sense. Generalizing
almost "nothing" is a triviality. On the other hand, if the research project is very strong in
terms of the internal validity components but the results are not gener alizable, one can
still speak of the effects of the independent variables on the dependent variable. Although
internal validity is more scientific investigations than external validity, and judgement on
how to strike the proper balance between external and internal validity, and judgment on
how to strike the proper balance between the two is improved by ex perience.
A third criterion for quality of research design is reproducibility , presented earlier as one
level of triangulation. Generally, internally valid research should be reproducible by other
authors under different settings and for different samples. This concept falls under the
domain of multiplicity of methods (using different methods, designs, and researchers to
study a given problem) or triangulation. The higher the quality of triangulation, the
stronger the elements of internal and external validity. Hence, triangulation may be
considered as a means for gauging or assessing the strength of the two major quality
criteria internal and external validity.
Additionally, all researchers must meet certain basic quality criteria, such as honest data
collection and faithful reporting of facts and criteria. Such criteria are rooted in ethics and
morality, as well as imperative for evaluating good research. But some researchers may,
in addition to the criteria given above, adopt still others. Ijiri (1975, for example,
maintains that research should possess novelty, defensibilty, and availability. Although
ijiri discussed novelty in terms of newness of ideas," we feel that this characteristic can
encompass other elements such as research methods and approaches. Indeed, much of the
recent research in accounting is concerned with utilizing new methods on ideas and
problems that have been studied before. To this extent, novelty may not be as basic as
suggested by Ijiri, and it be comes in effect a manifestation of the concept of 'triangulation.
Similarly, the notion of defensibility includes reproducibility, and is therefore another
form of triangulation. Availability of research by disseminating it to others for evaluation,
extension, or adoption as a "policy," is an essential element of research if it is to be useful.
The preceding discussion is not exhaustive. Readers interested in an exhaustive treatment
of the less important quality criteria should consult textbooks in methodology. However,
the preceding presentation assumes that other obvious and minor quality criteria are met.
Summary This chapter conveyed information about three sections:
(i) The main criteria used in assessing the quality of research. These criteria were
presented in terms of internal validity, external validity and triangulation
(2) Examples of major research designs which emphasized (a) the concept of control (or
comparison) by creating control samples and/or performing a pretest and a posttest
analysis, and (b) the correspondence between designs of true-experiments and of ex-post
facto (or quasi-experimental research
(3) Methods of data collection, including laboratory and field experiments, field studies,
and survey research.
The chapter concluded by summarizing and highlighting the cri eria of research quality.