InternationalJournalofTrainingandDevelopment5:4 ISSN13603736

Casestudiesinjobanalysis andtrainingevaluation
JackMcKillip
Jobanalysisidentiesworkbehaviorsandassociated thatarerequiredforsuccessfuljobperformance.Job tasks iswidelyrecognisedasastartingpointinaneeds analysis fortraininganddevelopmentactivities.Thecontent analysis justicationforbasingtrainingonajobanalysiscanbe validity fortrainingevaluation.Effectivetrainingshouldonly used animpactonjobtaskssignicantlyrelatedtosuccessful have performance.Thisarticlepresentstwoexamplesoftheuse job jobanalysisfortrainingevaluation:evaluationofan of mationtechnologycerticationandevaluationoftraining infor aprofessionalassociationof for librarians. Ajobanalysisidentiestheworkbehaviorsandassociatedtasksthatare forsuccessfuljobperformance(Harvey,1990).Oncethetasksnecessaryfora required jobareanalysed,personnelselectioninstrumentscanbeconstructedand/or given programsdeveloped.Theutilityofajobanalysisdoesnotendwithselection training trainingneedassessment,butcanbeextendedtotrainingevaluation.This and illustratestheuseofajobanalysisforevaluationoftrainingintwosettings: article informationtechnologycertication,andapotentiallibrary an certication.

Stepsinjobanalysis
Althoughitsapplicationisexible,ajobanalysisusuallyinvolvesatleastfoursteps (Harvey,1990).First,workactivitiesoroperationsperformedonthejobare widelyfromjoboccupantsandsupervisors.Next,activitiesandoperations elicited groupedintotasksordutiestosimplifyandorganisestatementsandto are overlap.Third,theresultantlistofjobtasksisratedonmeasuresofoverall eliminate cancetosuccessfulperformanceonthejob.Joboccupantsorsupervisors signi accomplishthistask.Finally,theknowledgeandskillsrequiredforsuccessful also performanceareidentiedfordevelopmentofselectioninstrumentsor job trainingneeds. potential Asanexample,2,800computerprofessionalsfrom85countriesratedjobtask
JackMcKillipisProfessorofAppliedExperimentalPsychologyatSouthernIllinoisUniversity, Carbondale,Illinois629016502,USA.Email:mckillip siu.edu
BlackwellPublishersLtd.2001,108CowleyRoad,OxfordOX41JF,UKand350MainSt.,Malden,MA02148,USA.

Case studies

283

statementsforthejobofsystemengineeraspartofanInternetsurvey 1998).Overall,48statementsrelatedsixjobtaskstoeachofeighttechnical (McKillip, Ratingsincludedthreescales:importanceforsuccessfuljobcompletion,difculty areas. learnandperform,andfrequencyofperformance.Table1presentsexamplesof to taskstatementsandratingscales. job

Contentvalidityandevaluation
Contentvalidityreferstotheextenttowhichaselectioninstrumentora programcoverstheknowledgeandskillsrequiredtosuccessfullyperformajob. training useofjobanalysisasthestartingpointinselectionortrainingprovidesevidence The contentvalidityfortheseactivities.Goldstein(1993)andMcKillip(1997) of describe Table1:Systemsengineerjobanalysisstatementsandratingscales(McKillip,1999) Technicalareadescription: UnieddirectoryservicessuchasActive system andWindowsNTdomains. Directory Connectivitybetweenandwithinsystems, architecture components,andapplications.Examplesinclude system ExchangeServerconnectorsandSMS Datareplicationsuchasdirectoryreplicationand senders. databasereplication. Systemarchitectureconsiderationsinclude physicalandlogicallocationofresources. the Jobtasks statements Analysethebusinessrequirementsforthe architecture. system Designasystemarchitecturesolutionthat businessrequirements. meets Deploy,install,andcongurethecomponentsof system the Managethecomponentsofthesystem architecture. onanongoingbasis. architecture Monitorandoptimizethecomponentsofthe architecture. system Diagnoseandresolveproblemsregarding componentsofthesystem the architecture. Signicancerating: 1. OFLITTLEIMPORTANCEfortasksthat 5pointimportancescale verylittleimportanceinrelationtothe have completionofyour successful 2. OFSOMEIMPORTANCEfortasksthatare job. averageimportancerelativetoothertasksbutare of givenhigh not 3. MODERATELYIMPORTANTfortasksthat priority. ofsomeimportancebutgenerallyaregiven are priority. low 4. VERYIMPORTANTfortasksthatare forsuccessfulcompletionofwork.Thesetasks important higherprioritythanothertasks,butarenotthe receive importanttasks. most 5. EXTREMELYIMPORTANTfortasksthat essentialforsuccessfuljobperformance.Sucha are mustbecompletedandperformedcorrectlyinorder task haveasatisfactoryoutcome. to 284 InternationalJournalofTrainingand Development

BlackwellPublishersLtd.2001.

howvalidtrainingprocedurescanbebuiltonajobanalysis.Thelogicis inFigure1.TheXaxisdisplayssignicanceforjobsuccessandtheYaxisdisplays illustrated trainingemphasisorneed.Crucialtrainingactivitiesfocusonknowledgeand identiedinajobanalysisassignicanttosuccessonthejob,i.e.,theupper skills handquadrantofFigure1.Trainingonknowledgeandskillsthatarenot right resultsinawasteofresourcesthatarebetterdirected signicant Goldstein(1993)urgedthatthesamecontentvaliditylogicbeusedfor elsewhere. oftraining.Effectivetrainingshouldbefoundtoemphasisejobtaskssignicant evaluation successfulperformance,andnottocoverjobtasksoflowsignicance. for McKillipandCox(1998)usedanImportancePerformancematrixlikethatof Similarly, 1 toevaluatetraining.Jobanalysisratingsofimportancewerecombinedwith Figure trainingratingsoftrainingusefulnessorcoverageofjobtasks(performance).A post playoftasksalongthelowerlefttoupperrightdiagonalofthegure dis positivetrainingimpact.Thisuseofcontentvaliditylogictoevaluatetraining revealed illustratedintwoexamples. is

TheMicrosoftCompanydescribesonitscompanyswebpagetheMicrosoft SystemsEngineer(MCSE)credentialcerticationasrelevantforprofessionals Certied analyzethebusinessrequirementsanddesignandimplementtheinfrastructure who businesssolutionsbasedontheWindowsplatformandMicrosoftserver for Implementationresponsibilitiesincludeinstalling,conguring,and software. networksystems.Thecerticationisawardedafteranapplicantpassesaseries troubleshooting coreandelectiveexaminationsonMicrosoftproductsconcerningtechnical of ciencyandexpertiseinsolutiondesignand pro Onepartoftheevaluationofthiscerticationexaminedwhetherthose implementation. receivedthecerticationfounditusefulfortheirworkon24jobtasks. who ratingswerecorrelatedwithimportanceratingsfromapreviousand Usefulness conductedjobanalysis.Apositiveevaluationofthecerticationwouldbe independently byshowingthecerticationwasusefulforimportantjobtasks,butnot indicated otherwise. Methodolog y ThejobanalysisfortheMCSEcertication(EducationalTestingService, includedratingsoftheimportanceof91jobtaskscoveringeightjobareasby 1997) MCSEsfrom20countries.Avepointscalewasused,from(1)Notimportantto 415 (5)

Evaluatinganinformationtechnologycertication 1998) (McKillip,

Figure1:Contentvaliditymatrixforrelatingjobanalysisandtraining.Adapted McKillip(1997).Usedwith from permission. BlackwellPublishersLtd.2001. Case studies

285

Extremelyimportant.Fromthe91jobtasks,anexpertgroupofMCSEsselected subsetof24thatcoveredeachoftheeightjobareas,thatweremutually a andthatparalleledtherangeofimportanceratingsofthefullsetofjob exclusive, DuringApril1998,1,671MCSEsrespondedtotheInternetsurveyratingeach tasks. the24jobtasksonhowusefulthecerticationprocesswastotheirownjob of anceoneachjobtask,usinganinepointscalefrom(1)Notatallusefulto perform Extremelyuseful.Respondentshadbeencomputerprofessionalsforanaverage (9) 7.5 yearsandhadheldtheirMCSEcerticationfor1.5years.Theywere of inglymaleandthemajoritywasemployedintheUS(59percent).Fortyone overwhelm centoftherespondentscamefrom66other per countries. Results Figure2usesascatterdiagramtodisplaytheimportanceratingsof24job identiedfromthejobanalysis(Xaxis)withtheusefulnessratingsfromthe tasks survey(Yaxis).Twentythreeofthejobtasksfellalongtheimportantlower Internet 1 toupperright providingimpressivesupportfortheeffectivenessof left certication.MCSEsratedthecerticationasmostusefulforthosejobareas diagonal, the weremostimportant.Theydidnotratethecerticationasusefulforeverything that theydidunimportantjobareasgeneratedlowerusefulnessratingsforthe that cation.Theonejobtaskthatclearlydidnotfallalongthelowerlefttoupper certi diagonalwarrantedinvestigationforincreasedcoverageaspartofthe right process. certication

Evaluatinglibrarianpreparation 2
Mostprofessionallibrarianscompleteamastersinlibraryscience(MLS)degree partoftheirprofessionalpreparation.UnderstandingtheMLSasageneralist as aprofessionalassociationoflibrariansevaluatedwhetherfurthertraininginthe degree, ofacerticationwasneededtomeetthechallengeofkeepingpacewiththe form demandsoftheirparticulartypeoflibrarywork.Theevaluationinvolveda evolving job

Figure2:Useofjobtasksinevaluationoftraining.AdaptedfromMcKillip(1998). withpermission. Used

1

Thecorrelationbetweenimportanceasidentiedinthejobanalysisandusefulnessofcertication fromtheInternetsurveywas0.71.Thisisalargeandreliableassociation. 2 Dr. CarylCoxofProgramEvaluationforEducationandCommunities,705W.ElmStreet,Carbon daleIllinois62901coauthoredthisstudywithDr.McKillip.

286 InternationalJournalofTrainingand Development

BlackwellPublishersLtd.2001.

analysisthatidentiedthejobactivitiesanddutiesofmembersofthe associationandasurveyofitsmembersconcerning(1)theimportanceofthe professional iedjobtasksfortheirprofessionalworkand(2)theextenttowhichthetasks ident coveredaspartoftheirformalMLSeducation.Currenttrainingwouldbe were adequateifthetasksthatwereimportantfortheprofessionwerecoveredaspart judged formalstudyandifunimportanttaskswerenotcovered(i.e.,thelowerleftto of rightdiagonalofFigure1).However,ifcurrenttrainingdidnotcover upper tasks(i.e.,thelowerrightquadrantofFigure1),anassociationsponsored important mightbe certication needed. Methodolog y Reviewofassociationdocumentsandfocusgroupswithassociationmembers

leadersledtothedevelopmentof31jobtaskstatementscoveringpositionsheld and membersoftheprofessionalassociation.Thesejobtasks,alongwithother by wereincludedinamailedsurveyofmembers.Respondentsratedthejobtask questions, mentsonimportant[ce]toyourpresentjobas % librarianusinga9pointscale state labeled:(1)Notatall,(3)Minimally,(5)Somewhat,(7)Very,and(9) a Respondentsalsoindicatedwhetherornotthetaskwascoveredbyacademic Extremely. workfromanaccredited course Threecategoriesofassociationmemberswereselectedbyrandomsampling: institution. employedinprivatermsorcorporations,thoseemployedingovernmentor those libraries,andthoseemployedinuniversitylibraries.Some527association court (48percentcompletionrate)responded,closelymirroringcharacteristicsof members sampledongeographyandplaceofemployment.Eightysixpercentof those hadachievedanMLSorequivalentdegreeandanequalpercentageagreedwith respondents statement:[membersofthisprofessionof]librariansshouldhaveanMLSor the entlibrarydegree.Resultsreportedinthefollowingsectionwererestricted equival respondentswithanMLS to degree. Results Figure3usesascatterdiagramtodisplaytheimportanceratingsofthe31job (identiedbygeneralarea)bythepercentageofrespondentswhohadreceived tasks malacademictrainingforthetask.Clearly,mostjobtasksdonotfallinthe for leftandupperrightquadrantsofthegure!Closeinspectionshowsseveral lower fulandimportantpatterns.Forexample,tasksintheareaofReference, meaning andPatronServices(A)followtheeffectivetrainingpatternalongthelowerleft Research, upperrightdiagonal.Mostrespondentsreceivedformaltrainingonthe to importanttasksinthisareaanddidnotreceiveformaltrainingonthisareas most importanttasks.Bycontrast,tasksintheareasofLibraryManagement(B)and less InformationTechnology(C)areconcentratedinthelowerright of PotentialNeedquadrantofFigure1.Iftheprofessionalassociationwereto quadrantthe acertication,theseareasshouldbe launch explored.

Discussion
Neithertrainingneedsassessmentnortrainingevaluationareeasy.Thelogic effortusedonajobanalysistosupportvalidneedassessmentcanalsobeused and trainingevaluation.Asthetwoexamplesillustrate,Goldsteins(1993)suggestion for thecontentvaliditylinkbetweenjobanalysisandtrainingneedsassessmentcan that fruitfullybeextendedtolinkjobanalysisandtraining Requirementsforthelinkincludeathoroughjobanalysisofthepositionfor evaluation. trainingisconducted.Analysisshouldidentifyjobtasksthatareimportantaswell which thosethatareunimportantforsuccessfuljobperformance,andshouldnot as unimportanttaskspartwaythroughthejobanalysis,asisfrequentlythe abandon case BlackwellPublishersLtd.2001. 287 Case studies

Figure3:Useofjobtasksinevaluationoflibrarian Librarianjobtask training A Reference,ResearchandPatron areas B Library . Services C Information . Management D Technology . CollectionCareand E Teaching . Management F . SpecialKnowledge . (Harvey,1990).Importanceratingscanthenbecombinedwithposttraining mentsoftheusefulnessorthoroughnessofthetraining.Thepatternof assess associationoftaskimportanceandtrainingusefulnesswillnotresultfrom positive traineeenthusiasmwiththetrainingexperienceitself.Insuchacase,trainees simple tionsarelikelytobeindiscriminatelypositiveuncorrelatedwithtask reac Whentraineesjudgmentsfollowimportanceratings,persuasiveevidenceof importance. priatetrainingdesignandeffectivedelivery appro Inpractice,scatterdiagramdisplayslikeFigure2and3haveproven result. effectiveinpresentingthemessageoftrainingeffectivenessortrainingneed. particularly havequicklygraspedthelogicandmethodofevaluation,andeasilyinterpreted Clients results.Inaddition,thescatterdiagramsareeasilyconstructedbymost the programs,suchasMicrosoftExcel.Thepatternsthatresultfromthescatter spreadsheet notonlycangiveevidenceoftrainingeffectivenessbutcanalsobeusedto diagram wastedeffortsandpotentialneedsforfurther identify training. References
EducationalTestingService(1997),AnAnalysisoftheJobofaMicrosoftCertiedSystemsEngineer (Princeton,NJ:Author). Goldstein,I.L.(1993),TraininginOrganizations:NeedsAssessment,Development,andEvaluation, 3rded.(PacicGrove,CA:Brooks/Cole). Harvey,R.J.(1990),JobAnalysis,inM.D.DunnetteandL.M.Hough(eds),Handbookof IndustrialandOrganizationalPsychology,vol.2(PaloAlto,CA:ConsultingPsychologistsPress), pp. 71163. McKillip,J.(1997),NeedAnalysis:ProcessandTechniques,inL.BickmanandD.Rog(eds), HandbookofAppliedResearchMethods(ThousandOaks,CA:Sage),pp.26184.

288 InternationalJournalofTrainingand Development

BlackwellPublishersLtd.2001.