Running head: QUALITY: MODULE 3

Quality:

Module 3

Ryan Bradshaw

George Mason University

EDRS 822

Dr. Baily
QUALITY: MODULE 3

Validity and Bias

The validity of a qualitative research project’s findings take on a different meaning when

they come from a constructivist viewpoint. Humans create meaning of each situation based on Commented [SB1]: Why? Wouldn’t it be different for any
viewpoint to another viewpoint?

their own interpretations, which are shaped by their own experiences and cultures. Different

individuals will develop strikingly different understanding of the same event as a result (Crotty,

1998, p. 47). Cho and Trent (2006) believe articulate that researchers are only able to create Commented [SB2]: Would be a better word choice – since you
can’t really say that you know what they believe…

reconstructions and interpretations. Crotty (1998, pp. 47-48) makes the point that there is

therefore no one true or valid interpretation of a phenomenon, only useful interpretations. Hatch

(2002, p. 180) adds that “interpretations are constructed by researchers”, the purpose of which is

to give meaning to the data.

In the earlier example of Fish’s students interpreting the names on the chalkboard to be a Commented [SB3]: Remember that this example might not be
relevant to someone who does not know the text.

poem rather than simply a list of names of authors for the assigned readings, the student’s

interpretation of a poem was, at that time, true in their minds, even though it was ultimately false

(Crotty, 1998, p. 47). To them, based on the information that was available to them at the time,

they deduced that it was a poem, which was their interpretation of the information provided. In a

constructivist viewpoint, this example illustrates that it is not possible to confirm the findings as

true fact, but it is possible to demonstrate that the findings are the best interpretation available by

using as much information as possible to construct our reality on the subject. Cho and Trent

(2006) suggest that under a constructivist view, “validity can never be achieved, but instead

needs to be checked endlessly”. Crotty (1998), p. 58) adds that constructivism “suggests that

each one’s way of making sense of the world is as valid and worthy of respect as any other” (p.

58), adding validity to each individual’s interpretation and suggesting that there can be no one Commented [SB4]: Check for APA like this throughout.

ultimate ‘truth’.

2
QUALITY: MODULE 3

Polkinghorne (2007) brings a different perspective to the discussion of validity, stating

that the believability of a statement or claim of knowledge, based on the evidence and argument

offered in support of said statement or claim, is the general notion of validity. He adds that

occasionally validity is granted solely because of the individual who made the statement of

claim, such as a well renowned researcher in a community of practice. Previous knowledge and

beliefs may also play a role in a community’s willingness to believe in a knowledge claim

(Polkinghorne, 2007). I consider that the same is true for individuals, as their constructed past

experiences and personal opinions, as well as the source of the information, will play a role in

their ability to accept information as believable, valid, and true.

Interviews are used by qualitative researchers to “uncover the meaning structures that

participants use to organize their experiences and make sense of their worlds” (Hatch, 2002, p.

91). Hatch (2002, p. 23) believes that constructivist interviewers and the interviewees work Commented [SB5]: See above

together to co-construct understanding that is then reported as the interpretation of the data. He

also suggests that the constructivist viewpoint works well with informal interviews as they are a

means for the interviewer and interviewee to work together to create understandings (Hatch,

2002, p. 93).

I believe in both Hatch and Crotty’s views on constructivist validity in that there is no

one ‘truth’ to be found, however, it is possible for me to work with my research subjects to co-

construct an accurate interpretation of their reality. As Ravitch and Carl (2016, p. 186) state,

“validity, in qualitative research, refers to the ways that researchers can affirm that their findings

are faithful to participants’ experiences.” It is not about finding a ‘truth’, but about ensuring the

study was completed with rigor and quality in mind (Ravitch & Carl, 2016, p. 186).

3
QUALITY: MODULE 3

Four Aspects of Trustworthiness

In 1981, with the field of qualitative inquiry still in its infancy and still called

“naturalistic inquiry” (Guba, 1981, p?), Guba and Lincoln (1981, p. 104) suggested four

‘“translations’” of scientific terms related to validity into new “naturalistic” terms. Internal Commented [SB6]: Actually – I would italicize these terms

validity, related to the truth value of the project, became credibility. External

validity/generalizability, related to the applicability of the study, became transferability.

Reliability, dealing with the consistency of the study, became dependability. Lastly, objectivity,

the neutrality of the study, became confirmability.

Anfara Jr., Brown, and Mangione (2002) summarized Guba and Lincoln’s work by

offering suggested strategies to be used for each of the four aspects. For credibility,

recommended strategies included member checks, using peer debriefing, triangulation, and

prolonged engagement in the field. Transferability could be improved by using purposive

sampling, and providing thick description. Dependability is gained by creating an audit trail, peer

examination, and triangulation. Confirmability is also enhanced by using practice reflexivity and

triangulation.

Rodwell (1998) also looked at Guba and Lincoln’s work, but from a constructivist

perspective. Her views on credibility, dependability, and transferability are very similar to

Anfara Jr. et al.’s (2002). However, she makes the important constructivist addition to the

argument that confirmability does not equate to realizing a ‘truth’, only that the findings are tied

to data. An external auditor could identify and follow the logic used by the researcher to get from

raw data to final product. Yet, another researcher could use the same data and construct their

own, completely different, results (Rodwell, 1998).

4
QUALITY: MODULE 3

Cho and Trent (2006) created the label of ‘transactional validity’, which they define as

“an interactive process between the researcher, the researched, and the collected data that is

aimed at achieving a relatively higher level of accuracy and consensus by means of revisiting

facts, feeling, experiences, and values or beliefs collected and interpreted.” They highlight

member checking, as part of Guba and Lincoln’s (1981) credibility realm, as a key component of

the transactional process in order to engage participants to ensure “their realities correspond with

the interpretations brought forth by the researchers” (Cho & Trent, 2006). Cho and Trent (2006)

also identify triangulation as part of the transactionalist approach to bolstering the integrity of a

research project.

Ravitch and Carl (2016, p. 188) caution researchers that rigor in qualitative studies does

not need to be at the same level as a quantitative study, as it can be assessed in many ways. They

continue that “qualitative researchers should develop validity approaches that align with the

research questions, goals, and contexts of their studies” (Ravitch & Carl, 2016, p. 188).
Ensuring Quality and Rigor
Commented [SB7]: Ryan – this section does a good job
documenting the different camps and arguments – but you will want
to pull this together to how it impacts you and your study, but it also
has to take into account, what the reader needsto take away from all
this back and forth with the authors.

In order to ensure that quality and rigor that Ravitch and Carl (2016, p. 186) recommend

must be present in this work, I plan to utilize the following techniques to ensure that the

participants’ experiences are recorded as faithfully as possible, while heading the advice to use

only validity approaches that align with this study:

Quick transcription and immersion in the data. Hatch (2002, p. 116) recommends

quickly transcribing the interviews in order to complete an initial analysis as soon as possible. He

believes that this can assist in assessing the effectiveness of the interviews and identify any

problems that can be addressed in subsequent interviews. If done properly, it can reshape

5
QUALITY: MODULE 3

subsequent interviews and help identify potential gaps in the data before it is too late to address

them (Hatch, 2002, p. 116).

Hatch (2002, p. 181) also recommends immersing oneself in the data, by reading and re-

reading the data, to the point that “whatever impressions are formed throughout the analytic

process are considered within the context of the overall data set.” This is necessary to construct

an interpretation that makes the subject understandable for the reader (Hatch, 2002, p. 181). I

intend to use both of these techniques to add to the quality and rigor of the study. Commented [SB8]: So if you are going to use these techniques
– tell the reader how? How quickly will you transcribe? How much
will you re-read? Make these connections back to your research.

Member checks. Member checks, detailed in the Guba and Lincoln’s (1981) credibility

description, can take place at multiple junctures throughout the interview and analysis process, in

both formal and informal check-ins with the participants (Rodwell, 1998, p. 99). During

interviews, I plan to solicit participant feedback on my interpretations of their responses to the

questions in the semi-structured interview questions, as suggested by Rodwell (1998, p. 99).

Wolcott (as cited in Hatch, 2002, p. 116) also recommends asking interviewees direct questions

about the research process to conclude the interviews, for example “Are there topics we should

explore that I haven’t asked about?” Commented [SB9]: But is that really a member check? Would
you want to think about asking them questions about your
interpretations as you go through the interview as well?

I also intend to share my reconstructed analysis of the interviews with the participants, by

emailing the participants each a copy of my interpretation of the interview transcripts and

requesting their feedback via email on my interpretation. Guba (1981) and Hatch (2002, p. 188)
both identify this as a vital part of a constructivist researcher’s desire to co-construct meaning
with their participants as partners in the process. The email will include a series of guiding
Commented [SB10]: There a LOT of pros and cons about this –
so think about why and what you might get out of this…When they
see something in writing and it looks “not as polite” as they wanted
it to – what if they pull the data? What if they remove themselves
from the study at that point? What if you have to make negative
portrayals based on the information….This is a laudible thing to
want to do – but it should not be naively done either – it can have
huge ramifications on you work. Easy in theory – not so in practice.

questions for the participants to consider when reviewing the interpreted interview transcripts, as

suggested by Ravitch and Carl (2016, pp. 197-8), including:

6
QUALITY: MODULE 3

1. Is there anything that is not captured in this interpretation?

2. Is my interpretation authentic for you (in what ways yes or no)?

3. Is there anything that I am missing?

4. Are there specific areas you would like to clarify or add to?

5. Are there assumptions and /or biases that you see underneath anything I have

written or said that you feel I should challenge? Commented [SB11]: Good questions – but again – think about
it – and what it one goes south – what happens then.

Peer debriefing. As suggested by Guba and Lincoln (1981), utilizing a peer who

understands my methodology, but who is not involved with my research, can assist in limiting

my subjectivity on findings (Rodwell, 1998, p. 99; Ravitch & Carl, 2016, p. 231). As I have been Commented [SB12]: And can identify assumptions.

deeply involved in Club Sports as an administrator for almost a decade, my views on the subject

have been shaped by my previous experience. A peer reviewer can assist in the process by asking

tough questions, providing support, offering technical advice, and helping to work through

strong feelings that I may have towards one viewpoint or another (Rodwell, 1998, p. 99).

Rodwell (1998, p. 99) also advises that the researcher and the peer reviewer both keep reflexive

journals to track their discussions. Ravitch and Carl (2016, p. 231) add that utilizing a peer

reviewer throughout the data collection and analysis process can help challenge the researcher’s

“assumptions, biases, preconceived notions, and how each and all of these shape the ways that Commented [SB13]: 

[they] think about the data and the people in the study”. Utilizing a peer reviewer and a

journaling process will help reduce any subjectivity I bring to the project.

Triangulation. Guba (1981) considers triangulation a part of the credibility,

dependability, and confirmability aspects of trustworthiness. To Guba (1981), triangulation

involves the using a variety of data sources, to the extent that information should not be accepted

unless it can be verified and documented from at least two sources. Cho and Trent (2006)

7
QUALITY: MODULE 3

describe triangulation, as part of their transactional validity label, more bluntly as “verifying

facts through multiple sources”. Data should also be collected from various sources (Guba, 1981)

to provide for multiple voices to be heard in order to triangulate data and provide a “holistic

understanding of the situation and converging conclusions” (Anfara Jr, et al., 2002). Rodwell

(1998, p. 99) states that “triangulation should be used for comparison, distillation, or convergent

validation, or to see if the information holds up under comparison to something else.” She also

recommends using a reflexive journal to document the triangulation (Rodwell, 1998, p. 99).

Hatch (2002, p. 121) reminds readers that, to him, an important component of

triangulation is having researchers describe the data and sources they used carefully to allow

readers to make their own conclusions on the trustworthiness of the study. Ravitch and Carl

(2016, p. 195) highlight that the goal is not necessarily to seek convergence, but to seek out and

engage numerous views in answering the research question.

In this study, data triangulation, the act of collecting data from different people (Ravitch

& Carl, 2016, p. 195), will be used to collect data from a wide range of individuals who had

previously participated in club sports at the institution. Special attention will be paid to interview

participants from different genders, ages, income levels, and race/ethnicities, as well as

participants from various club sports, as to triangulate the data and enhance validity in the study.

These efforts will also be clearly described in the final project to allow readers to judge the

trustworthiness of the study themselves.

Audit trail. Guba (1981) suggests the establishment of an audit trail as part of the

dependability and confirmability aspects of trustworthiness. The goal of the audit trail is to

enable an external auditor to look at the data that was collected, the process that was used for

8
QUALITY: MODULE 3

data collection and analysis, and the process used to interpret the data in order to validate the

process used in the study, as part of a dependability audit (Guba & Lincoln, 1989, p. 242).

Rodwell (1998, p. 100) recommends keeping records of collected and analyzed data, including

process notes and journals of decisions made, to enable a dependability audit to be completed.

She also mentions that peer reviewers use a reflexive journal to document their meetings with the

researcher. An audit trail of all processes from data collection to final publication, including

reflexive journals from myself and my peer reviewer, will be compiled through the research to

allow for an external auditor to review and provide assurances of validity to my process.

Inherent Biases

Due to the fact that I have worked in collegiate recreation, specifically with club sports,

over the course of the last decade, my views on the subject have been constructed by my

experiences to date. I have inherent biases towards the value of club sports to the collegiate

experience of students and believe strongly students gain many transferable skills through their

participation in club sports, as identified by Hall, Forrester, and Borsz (2008) and Flosdorf, Carr,

Wallace Carr, and Pate (2016). I believe that students primarily identify that they acquired these

skills through participating in club sports post-graduation, when they are working in their

careers. In conducting this research, I will have to limit my preconceptions by utilizing the

aforementioned techniques of utilizing a peer reviewer and member checks, in order to enhance

the validity of this study.

Having worked at three institutions, George Mason University, Mount Royal University,

and Southern Illinois University Carbondale, in my time working in club sports, I have personal

connections to potential study participants at each of these schools. From my work experience, I

9
QUALITY: MODULE 3

also have additional biases and pre-existing views that I have constructed about each of these

colleges. In order to eliminate the risk of bias in this project based on the institution of study, it

will be conducted at a college that fits the requirements previously outlined in this paper at which

I have never had any direct involvement with the administration or operation of their club sports

program.

In working with the subjects of this study, I also need to recognize that the participants

are all donors to the institution that is being used in the study. These donors are of value to that

institution. In having the school’s alumni affairs office agree to have me contact some of their

donors for this project, I will need to assure the office that I will not discourage any of their

donors from donating again in the future and maintain a cordial and positive relationship with the

interviewees. This may play a factor in the ways the interviews are conducted and how far I am

able to push the subjects for descriptive responses, as to ensure that the experience of

participating in the study is a positive one for them that does not dissuade them from future

donations to the institution.

Lastly, it is important that I recognize my vested interest in the results of this inquiry. As

a collegiate recreation professional, identifying ways to increase revenue generation through

alumni is of value to me in my professional work. Campus recreation departments need

additional funds to support programs like club sports, which can be very expensive to operate.

Public institutions in the United States have collectively seen their state governments cut over

40% of all funding to higher education between 1980 and 2011 (Mortenson, 2012). This means

that additional funds must come from other sources, such as philanthropic donations, however,

only 8.7% of alumni typically donate to the alma mater (Council for Aid to Education, 2014).

Former club sports participants are a key demographic that have had more engaged experiences

10
QUALITY: MODULE 3

while at the institution, which typically translate into an increased willingness to donate (Monks,

2003; Weerts & Ronka, 2008). This could be the key group to bridge the funding gap for club

sport programs across the country. As a collegiate recreation and club sport professional, who is

an active member in NIRSA, the professional organization for collegiate recreation

professionals, the findings of this study may positively affect the future viability of having

institutions across the country provide club sports programs to students and help keep some of

my colleagues and myself employed. Member checks and peer debriefs will assist in minimizing

this potential bias.

Ryan,

This is an excellent paper and really well thought out. I hope you will be able to ensure the

quality you want for your study based on the ideas you are presenting here. I think that you have

applied a lot of this directly to your context and research project – which makes it for a strong

link to chapter 3 if you are going to use this as your dissertation study. I really enjoyed your

voice in class this semester and really look forward to seeing you move through the program!

My best to you!

dr. B

25/25

11
QUALITY: MODULE 3

Refrences

Anfara Jr, V. A., Brown, K. M., & Mangione, T. L. (2002). Qualitative Analysis on Stage:
Making the Research Process More Public. Educational Researcher, 31(7), 28-38.

Cho, J., & Trent, A. (2006). Validity in qualitative research revisited. Qualitative Research, 6(3),
319-340. doi:10.1177/1468794106065006

Council for Aid to Education. (2014). Voluntary Support of Education 2013. New York, NY:
Author.

Crotty, M. (1998). The Foundations of Social Research: Meaning and Perspective in the
Research Process. Thousand Oaks, CA: SAGE Publications.

Flosdorf, M. L., Carr, B. H., Wallace Carr, J., & Pate, J. R. (2016). An Exploration of the Sport
Club President's Experience. Recreational Sports Journal, 40(2), 106-119.
doi:http://dx.doi.org/10.1123/rsj.2016-0007

Guba, E. G. (1981). Criteria for Assessing the Trustworthiness of Naturalistic Iquiries.

Educational Communication and Technology Journal, 29(2), 75-91.

Guba, E. G., & Lincoln, Y. S. (1981). Effective Evaluation. Jossey-Bass Publishers: San
Francisco, CA.

Guba, E. G., & Lincoln, Y. S. (1989). Fourth Generation Evaluation. Newbury Park, CA: SAGE
Publications.

12
QUALITY: MODULE 3

Hall, S. L., Forrester, S., & Borsz, M. (2008). A Constructivist Case Study Examining the
Leadership Development of Undergraduate Students in Campus Recreational Sports.
Journal of College Student Development, 49(2), 125-140.

Hatch, J. A. (2002). Doing Qualitative Research in Education Settings. Albany, NY: State
University of New York Press, Albany.

Monks, J. (2003). Patterns of giving to one’s alma mater among young graduates from selective
institutions. Economics of Education Review, 22(2), 121-130. doi:10.1016/S0272-
7757(02)00036-5

Mortenson, T. G. (2012, Winter). State Funding: A Race to the Bottom. The Presidency: The
American Council on Education's Magazine for Higher Education Leaders. Retrieved
from http://www.acenet.edu/the-presidency/columns-and-features/Pages/state-funding-a-
race-to-the-bottom.aspx

Polkinghorne, D. E. (2007). Validity Issues in Narrative Research. Qualitative Inquiry, 13(4),

471-486. doi:10.1177/1077800406297670

Ravitch, S. M., & Carl, N. E. (2016). Qualitative Research: Brindging the Conceptual,
Theorettical, and Methodological. Thousand Oaks, CA: Sage Publications, Inc.

Rodwell, M. K. (1998). Social Work Constructivist Research. New York, NY: Garland
Publishing, Inc.

Weerts, D. J., & Ronca, J. M. (2008). Characteristics of Alumni Donors Who Volunteer at their
Alma Mater. Research in Higher Education, 49, 274-292. doi:10.1007/s11162-007-9077-
0

13