CredibilityIntervalsforOnlinePolling

IpsosReidismovingawayfromtheuseofclassicalmarginsoferrortodemonstrate
confidenceintheaccuracyofouronlinepollingresults.WehaveadoptedBayesianCredibility
Intervalsasourstandardforreportingourconfidenceinonlinepollingresults.

WhyistheClassicalApproachtoMarginsofErrorNotAppropriateforOnlinePolling?

Inordertoproduceanaccuratemarginoferror,onemustknowtheprobabilityofparticipation
foreachmemberofthesurveypopulationoreveryoneinthepopulationmusthaveaknown
chanceofparticipatinginthesurvey.

Onlinepolling,mostoftenconductedbytheuseofpanelsofconsumers/citizens,doesnotmeet
thisconditionbecauseoftwoeffects:nonresponsebiasandcoveragebias.Inordertouse
classicalmarginsoferrorforonlinepolling,onewouldhavetoassumethatnonrespondersto
onlinesurveysarecompletelyrandom,orthattheeffectofleavingoutnonusersissominimal
thatitcanbeoverlooked.Weknowthatbothofthesearefalse,aswecanmakethefollowing
claimsaboutonlinepollingandthosewhocompleteonlinesurveys:

Noteveryonehasinternetaccess;
Onlinepanelsarecreatedthroughanoptinprocess,ratherthanthrougharandom
fashionlikearandomdigitdialtelephonecall;
Lessisknownabouttheprofilesofindividualswhocompleteonlinesurveysversus
thosewhodonot,oraboutthelikelihoodofanonlinepersontocompleteanonline
surveyortoparticipateinasurveypanel.
Therefore,theprobabilityofbeingincludedinanygivenonlinesurveysampleisunknown,very
difficulttoascertain,orsimplyzero(noninternetusers).Further,thenatureofuseofthe
internetisnotuniformwithinthepopulation,sothislimitsonesabilitytocalculatethe
likelihoodofreachingapersonthroughanonlinepoll.Inshort,withoutthisknowledge,a
marginoferrorcannotbecalculated.

Despitethesechallenges,onlinepollingconductedinascientificmannerhasproventoyield
similarresultstoprobabilitysamplingconductedviatelephone.Putsimply,onlinepollingworks!
Thisposesachallenge,however,becauseitisalsoveryimportantforusasresearchersto
provideameasureofconfidencealongwithourdata,regardlessofthemethodofdata
collection.Assuch,Ipsosconsidersthemethodologyvalidandismovingtowardstheuseof
BayesianCredibilityIntervalstodescribetherelativerobustnessanduncertaintyofanonline
surveyestimate.

1
w ww .ips os- na. c o m


IntroductiontoBayesianCredibilityIntervals

BayesianCredibilityIntervalsoperatemuchinthesamewayasclassicalmarginsoferror,but
acknowledgetheuncertaintyofanestimate(inourcase,theprobabilityofanygivenpersonto
completeanonlinesurvey),andincorporateexternalfactors,suchaswhatweknowaboutthe
world,opinions,expertise,context,historyandotherdataintoitscalculationtocorrectforthe
unknown.Bayesianmodelsallowustogeneralizefromasampletoapopulation,sincethey
correctforunbalancedsamplesduetononresponse,coverageorotherbiases.Onedoesnot
needtoknowtheactualprobabilitiesofselection,which,asdescribed,arenearlyimpossibleto
ascertaininonlinepolling.

Inorderforthisapproachtoholdtrue,onemustassumethatthesampledesignis
conditionallyignorable,meaningthat,oncewecontrolforthebiasesmentionedabove,there
isnotarelationshipbetweenoneslikelihoodtoparticipateinanonlinesurveyandthevariables
wewanttomeasure.Ipsosistakingstepstoensurethatitsonlinesamplesareconditionally
ignorable,suchascombiningmultipleoptinonlinepanelandnonpanelsources.Inessence,by
combiningmultiplesamplesources,theholesinanyonesamplesourcecanbefilledbythe
othersource.

BayesianCredibilityIntervalsmeasurethedegreeofcertaintyonehasintheresultsbasedon
onesexperience,understandingandknowledgeofthepopulation,temperedbythedatathat
hasbeenobserved.Wecallthistheknowledgebase,andcanincludeotherpublishedsurveys,
pastsurveysbythesamepollster,historicalpoliticaloutcomedata,apractitionerspolitical
expertise,predictivemodelsandothersources.Thisknowledgebaseiscontinuouslyupdated
withnewinformation.

Anexampleofanoutputstatementofsuchanapproachreadsasfollows:
ThesearefindingsfromanIpsosReidpollconductedforCLIENTfromDATES.Forthesurvey,a
sampleofSAMPLESIZECanadianswasinterviewedonline.Theprecisionofonlinepollsis
measuredusingacredibilityinterval.Inthiscase,thepollhasacredibilityintervalofplusor
minusXXpercentagepoints.Formoreinformationaboutcredibilityintervals,pleaseseeclick
here.ThedatawereweightedtotheCanadianpopulationdatabyregion,gender,age,and
education.Statisticalmarginsoferrorarenotapplicabletoonlinepolls.Allsamplesurveysand
pollsmaybesubjecttoothersourcesoferror,including,butnotlimitedtocoverageerrorand
measurementerror.Wherefiguresdonotsumto100,thisisduetotheeffectsofrounding.

Insummary,IpsosisusingtheBayesianapproachtoframetheproblemofmarginsoferrorsfor
nonprobabilitysamplinginsuchawaytoallowustousethewealthofinformationavailable
outsideofthecurrentsurveytocalibratesurveyresultsandtoprovideastatisticalfoundation
forinferenceevenwhenaprobabilisticsampleisunavailable.

2
w ww .ips os- na. c o m


HowtoCalculateBayesianCredibilityIntervals

ThecalculationofcredibilityintervalsassumesthatYhasabinomialdistribu oncondi onedon

theparameter\,i.e.,Y|~Bin(n,),wherenisthesizeofoursample.Inthisse ng,Ycounts
thenumberofyes,or1,observedinthesample,sothatthesamplemean(y)isanatural
estimateofthetruepopulationproportion.Thismodelisoftencalledthelikelihoodfunction,
anditisastandardconceptinboththeBayesianandtheClassicalframework.TheBayesian1
statisticscombinesboththepriordistributionandthelikelihoodfunctiontocreateaposterior
distribution.Theposteriordistributionrepresentsouropinionaboutwhicharetheplausible
valuesforadjustedafterobservingthesampledata.Inreality,theposteriordistributionis
onesknowledgebaseupdatedusingthelatestsurveyinformation.Forthepriorandlikelihood
functionsspecifiedhere,theposteriordistributionisalsoabetadistribution((/y)~(y+a,n
y+b)),butwithupdatedhyperparameters.

Ourcredibilityintervalforisbasedonthisposteriordistribution.Asmentionedabove,these
intervalsrepresentourbeliefaboutwhicharethemostplausiblevaluesforgivenourupdated
knowledge base. There are different ways to calculate these intervals based on
. Since
wewantonlyonemeasureofprecisionforallvariablesinthesurvey,analogoustowhatisdone
withintheClassicalframework,wewillcomputethelargestpossiblecredibilityintervalforany
observedsample.Theworstcaseoccurswhenweassumethata=1andb=1and
.Using
asimpleapproximationoftheposteriorbythenormaldistribution,the95%credibilityintervalis
givenby,approximately:

Forthispoll,theBayesianCredibilityIntervalwasadjustedusingstandardweightingdesign
effect1+L=1.3toaccountforcomplexweighting2

Examplesofcredibilityintervalsfordifferentbasesizesarebelow.

Credibility
Samplesize
intervals
2,000
2.5
1,500
2.9
1,000
3.5
750
4.1
500
5.0
350
6.0
200
7.9
100
11.2
1

BayesianDataAnalysis,SecondEdition,AndrewGelman,JohnB.Carlin,HalS.Stern,DonaldB.Rubin,Chapman&Hall/CRC|
ISBN:158488388X|2003

2
Kish,L.(1992).WeightingforunequalPi.JournalofOfficial,Statistics,8,2,183200.

3
w ww .ips os- na. c o m


Ifyouhaveanyquestionsaboutcredibilityintervalsforonlinepolling,pleasecontact:
SeanSimpson
AssociateVicePresident
IpsosPublicAffairs
Tel:(416)5724474
sean.simpson@ipsos.com

2012, Ipsos Public Affairs

4
w ww .ips os- na. c o m