MEASURING ATTITUDES AND FEELINGS TOWARDS

DISCRIMINATION IN CROSS-NATION RESEARCH:

LESSONS LEARNED FROM THE EUROPEAN SOCIAL
SURVEY

Jaak BILLIET
Bart MEULEMAN**

Introduction

The dimensions that may be heuristically distinguished in the concept of discrimination

include motivation, discriminatory action, effects of discrimination, the relation between
motivation and action, the relation between action and effects, the immediate organisational
(institutional) context, and the larger societal context (Feagan & Eckberg, 1980: 2). Although
any proposal of the measurement of discrimination on the labour market should ideally cover
all these dimensions, this is clearly impossible within the limits of this study. We have
confined ourselves to the measurement of discrimination in the first round of the European
Social Survey. This means that our focus is exclusively on verbal expressions of
discrimination in general, and if possible, discrimination related to the allocation of jobs. This
is a first limitation of our study.
We are conscious of the complexity and the multitude of problems that are encountered in the
measurement of the behavioural aspect of discrimination at the different levels mentioned
because of i.e. the social undesirable and often illegal character of this kind of behaviour.
However, since our focus is on the attitudinal aspect of discrimination, these problems are
largely avoided. It is incorrect to assume that the response to a verbal question in a
questionnaire necessarily reveals an attitude which would become operative in the situation
depicted in the question (Linn, 1965: 353) as we all know from the classical study of Lapierre
(1934). This does however not mean that the study of verbal behaviour is meaningless.
Robert Merton has pointed out that it should not be assumed that overt behaviour is

Jaak Billiet is professor in social methodology, Centre for Sociological Research (CeSO), Katholieke Universiteit
Leuven. He is also member of the Central Coordination Team of ESS.
**
Bart Meuleman is preparing his PhD at CeSO as a fellow of the National Research Fund Flanders.

1
intrinsically any more real than verbal behaviour nor should it be considered as more
truthful. Overt actions do not necessarily reflect verbal actions and may deliberately
conceal or disguise them, but it is sometimes valuable to know a persons opinion even if it is
not directly related to his behaviour (Merton, 1940: 21). It is moreover possible to enhance the
prediction of behaviour by means of attitude measurement by not only measuring attitude
direction but also attitude strength independent of the direction (Billiet, 1993: 149-150).
Attitude strength refers to the importance the issue has for the respondent (Scott & Schuman,
1988).1
There also is a second limitation in this study. Since data from the European Social Survey
(ESS) is used in our study on the attitudinal dimension of discrimination, we are confined to
the measurements that are available in the ESS. Neither the cross-nation measurement of
discrimination on the labour market nor the measurement of discrimination in general were
aims of this survey. Instead, the first round of ESS in which 22 European countries are
involved contained an extensive module of 58 items drawn up to measure different aspects of
attitudes towards migration and immigrants (Meuleman & Billiet, 2006). This module is
theoretically relevant for the study of discrimination, since discrimination is usually related to
(ethnic or racial) prejudice (Allport, 1954/1979; Feagin & Eckberg, 1980; Sears et al., 1997),
stereotyping (Finkelstein et al., 1995), racial attitudes (Stewart & Perlow, 259), cultural
minorities (Simpson & Yinger, 1985), and in-group favouritism (Mummendy et al., 1992).
Thus, it is likely that this module holds some (indirect) measures of discrimination of ethnic
minorities or immigrants. Furthermore, the core module which appears in all ESS rounds
contains a question about the respondents (subjective) membership of a group that is
discriminated against. In a subsequent question it is asked on what ground this group has been
discriminated against. The question about the feeling of belonging to a discriminated group
does not measure the attitude towards discrimination as such. It is however relevant to have
not only a view on the acceptance of discrimination among the general population (attitude)
but also about the incidence of feelings of being discriminated. All in all, the measurements of
discrimination in ESS are very limited but three of these are more or less related to labour
market discrimination. The major advantage of the ESS round 1 data is the possibility of
comparing 22 countries.
This paper continues with a short presentation of ESS and an overview of the questions that
can be used as measurements of discrimination on the labour market. In a subsequent part, we

1
Other conceptions of attitude strength focus on other aspects of attitudes like high level of accessibility, internal
consistency, extremity, certainty, knowledge, intensity, and low levels of ambivalence (Abelson: 1988; Krosnick, 1988).

2
will discuss the main findings that are based on a strict methodology that takes care of the
equivalence of the measures over all countries. Finally, some reflections are made on the
limitations of the measurement of discrimination in a general social survey as ESS, and some
alternatives are proposed to improve a comparative measurement of discrimination in the
European countries.

Measurements of discrimination in ESS round 1

The European Social Survey (ESS) is a large-scale international survey that was designed to
measure attitudes, beliefs and behavior in several European countries. The survey was
conceived of as a series of cross-sections. The first round started in 2001 under the name
Monitoring attitude change in Europe. The data collection of this first round took place in
2002-03. In total, more than 40.000 respondents from 22 countries2 were interviewed face-to-
face (Jowell; et al. 2007). In 2004, a second round was organized. The number of participating
countries rose to 25.3 Data of the two rounds are freely available on the website of the ESS
data distribution office (http://ess.nsd.uib.no).
Two weight factors are delivered with the datasets, population weights and design weights.
Population weights take the size of each country into account and are only useful when one
wants to analyze the complete dataset or to compare subsets of countries. Design weights are
required for each kind of analysis since it takes design effects related to the sampling strategy
into account. For the first round, additional post-stratification weights (Holt & Smith, 1979)
are computed which are based on the joint distribution of gender, age, and education level in
each country (ps_weight). In this study, these weights are used in combination with the design
weights in order to adjust somewhat for nonresponse bias. Although the differences with
samples that are exclusively weighted by design weights are not significant in most cases, the
use of the additional weights gives a better view on real differences in the samples (Meuleman
& Billiet, 2005; Vehovar, 2006).4 A consequence of this decision is that France is not
included in the analysis since no post-stratification weights could be computed. Moreover,
Israel is dropped because of the very particular situation of this non-European country.
Consequently, not 22 but 20 countries are included in this study.

2
Austria (AT), Belgium (BE), Switzerland (CH), Czech Republic (CZ), Germany (DE), Denmark (DK), Spain (ES),
Finland (FI), France (FR), the United Kingdom (GB), Greece (GR), Hungary (HU), Ireland (IE), Isral (IL), Italy (IT),
Luxemburg (LU), the Netherlands (NL), Norway (NO), Poland (PO), Portugal (PT), Sweden (SE), Slovenia (SI)
3
Five new countries Estonia (EE), Iceland (IS), Slovakia (SK), Ukraine (UA), and Turkey (TU) - started participating.
On the other hand, Israel (IL) quitted the survey.
4
The gender by age by education weights for the first round are not yet centrally distributed. One of the aims of this study
is to check the quality of these weights before these are released.

3
The core module of ESS which appears in all ESS rounds, contains a question about the
respondents membership of a group that is discriminated against.
C16. Would you describe yourself as being a member of a group that is discriminated
against in this country (YES, NO).
In a subsequent question it is asked:
C17. On what ground is your group discriminated against?
PROBE What other grounds?
The pre-coded list of groups contains the following categories:
Colour or race, nationality, religion, language, ethnic group, age, gender sexuality,
disability, other (write in), dont know.
These questions about the feeling of being discriminated against are of a different nature than
the questions in the immigration module. Clearly, the latter are of greater interest for the
measurement of discrimination.
A subset of 58-item immigration module in ESS round 1 has proven to constitute several
reliable and valid measurement scales. Six latent variables, each measured by five items at
least. These scales can be labeled as followed: (1) the willingness to allow immigrants into the
country (D4-D9); (2) the evaluation of the consequences of migration (D25-D31); (3) support
for conditions for allowing immigration (D10-D17), (4) attitude towards asylum seekers
(D49-D45), (5) ethnic threat or prejudice (D18-D24); and (6) cultural homogeneity (D40-
D44). Moreover, the cross-cultural equivalence of most of these latent variables has already
been tested in all countries or in subsets of countries (Billiet & Welkenhuysen-Gybels, 2004;
Meuleman et al., 2006). The other items in the module do not form scales, but are singular (or
couples of) items that deal with equal rights, social distance, relations with immigrants, and
policy issues. In the second round (2004, 25 countries), the module of migration-related items
was dropped due to lack of space. However, two scales (willingness to allow immigrants and
evaluation of the consequences of immigration) were retained in a more limited version; each
consisting of three items (Meuleman & Billiet, 2006).
The module on immigration contains three items that may be useful for the measurement of
an attitudinal dimension of discrimination on the labour market (namely D36, D45, D50) and
one item that possibly belongs to the same latent concept since it deals with equal rights for
immigrants and citizens (D22). Two of these items belong to one of the sets that are
measurements of the aforementioned concepts (D22 and D45), the two others are singular.

Item Wording Response scales*

4
 People who have come to live here should be given the same rights as 1 (agree strongly)---
D22
everyone else. 5 (disagree strongly)
And now thinking of people who have come to live in [country] from
another country who are of a different race or ethnic group from most 0 (not mind at all) --
D36
people [country]. How much would you mind or not mind if someone - 10 (mind a lot)
like this was appointed as your boss?
How good or bad is a law against racial or ethnic discrimination in the 0 (extremely bad) ---
D45
workplace? 10 (extremely good)
While their applications for refugee status are being considered, 1 (agree strongly) ---
D50
people should be allowed to work in [country] 5 (disagree strongly)
* The response scales of items D22, D36, and D50 are reversed in the analysis. For that reason, high
scores indicate an attitude against discrimination and pro equal treatment.

Singular item D45 is in the context of labour market discrimination the most relevant opinion
statement since it is a direct measurement of an aspect of the attitude towards labour market
discrimination, i.e. discrimination in the workplace. Item D50 also deals with the allocation of
jobs, but it is less general since a specific category of people is mentioned (refugees). Item
D36 refers to the acceptance of somebody of another ethnic group in a leading position (in
relation to a job).
We will certainly pay attention to the particular items about a law against ethnic
discrimination in the workplace, but additionally we will try to find out whether these four
items are valid and reliable measurements of a latent underlying attitudinal variable we can
call attitude towards equal treatment of foreigners. This concept is at the one hand more
abstract (and larger) than discrimination on the labour market, and at the other hand certainly
does not cover all dimensions of labour market discrimination, but it is related to an important
aspects of ethnic discrimination e.g. unequal treatment based on group membership (Allport,
1954/1979: 51-52). At once, we investigate whether this concept has been measured
equivalently over all countries (Billiet, 2002). This is done by a test of factorial invariance by
means of Multi Group Structural Equation Modelling (MGSEM) (Jreskog, 1990; Jreskog &
Srbom, 1993).
One of the primary goals of this study is to make an international comparison of attitudes
towards discrimination. However, we cannot readily assume that scores on the measurement
scale can be compared in a valid, straightforward manner across countries. It is not sure
whether our measurement scale operates in exactly the same way in different countries (Horn
& McArdle, 1992). Observed differences might be due to the fact that people from different
cultural groups respond to the items in a different way (Harkness et al., 2003; van de Vijver &

5
Leung, 1997). The cross-cultural comparability of scores depends on the extent to which the
concepts have been measured equivalently.
In the literature, several schemes for ordering various forms of measurement equivalence were
proposed (Johnson, 1998). In this study, we rely on the hierarchical scheme that was proposed by
Steenkamp and Baumgartner (1998). According to these scholars, the lowest level of
equivalence is called configural invariance. This level of equivalence is reached when the
measurement instrument exhibits the same structure of salient and non-salient factor loadings
over all countries, and leads to the conclusion that the same concept is measured cross-culturally.
However, configural does not guarantee any score comparability. This level of equivalence is
often used as the baseline of equivalence testing. If the factor loadings of the items on the
underlying trait they are purported to measure are found to be invariant across countries, metric
equivalence is obtained. This second level of measurement equivalence implies that scale
intervals of the latent trait are equal across groups. Consequently, statistics that are based on
difference scores, such as regression coefficients and correlations, may be compared cross-
nationally. If also the intercepts of the functions that describe the relations between items and
latent factors are invariant, then one gets scalar invariance and full-score comparability is
guaranteed. Consequently, it becomes possible to compare the latent means between the groups.
Scalar invariance is most difficult to obtain since it assumes metric invariance.
However, Byrne et al. (1989) argued that full metric invariance is not necessary in order for
substantive analyses to be meaningful. Provided that at least two items -namely the items that
are fixed at unity to identify the model and one other item- per construct are invariant, cross-
national comparisons can be made in a valid way. Thus, this partial invariance only requires
cross-country invariance of the zero loadings and of some, but not necessarily all, of the
salient loadings. This idea is also supported by Steenkamp and Baumgartner (1998).
In the Structural Equation Modelling (SEM) approach, the scores on the observed variables
(items) are expressed as a linear function of a latent variable:

(1)
x (j g ) = (j g ) + (jg ) ( g ) + (j g )

According to this equation, each indicator xig is modeled as a regression function of latent

factor g , with intercept gj , regression slope or factor loading gj and stochastic error term jg .

The subscript j refers to the items and the superscripts (g) to the groups under study. In the
above model, the test of whether the indicator xj is an (scalar) invariant indicator of the latent

6
concept across the G groups involves testing whether the intercept and slope parameters of the
measurement model in (1) are invariant across these groups. Hence, in the context of the
structural equation model, the requirements for scalar equivalence can be written as follows:

(j1) (j g )
(1 ) = = (g) (2)
j j

To test whether the constraints in (2) hold in our dataset, a top-down approach is used. We start
from the most restrictive model, with all intercepts and slopes constrained to be invariant across
countries. In subsequent steps, it was assessed if the model fit could be improved substantially by
freeing some of the constrained parameters. The specifications of the fully invariant model for
the test of the invariance of the 4 indicators of the attitude towards equal treatment of
foreigners are therefore the following:

20 groups (countries) g {1, , 20}

Equal factor loadings over all groups: (j1) = (j2 ) = = (j20) for j {1, , 4}

All residual covariances zero: (ig, j) = 0 for g {1, , 20}, i,j {1, , 4} and i j

Variances of latent variable (g) not equal across the groups (completely free)

Equal intercepts over all groups: (j1) = (j 2 ) = = (j 20) for j {1, , 4}

Mean of latent variable for first group is fixed to zero and free in other groups: (1) = 0 ,
and ( g ) 0 for g {2, , 20}

The residual variances ( gj , )j were allowed to be different across all groups:

Since the variables are ordinal rather than interval measures, the estimation procedure for
ordinal variables as suggested by Jreskog (1990) is used. The analysis is performed with
LISREL8.7. Polychoric correlations are computed with PRELIS2 (Jreskog, 1993) after
constraining the thresholds equal for all groups to the thresholds in the joined datasets.
Weighed Least Squares estimation is used with an asymptotic (co)variance matrix. Given the
above described treatment of ordinal variables in MGSEM, the xj in the above model denotes
the continuous variable underlying the ordinal variable Oj that was actually observed.
Either the likelihood ratio test, the Lagrange Multiplier test, or the Wald test can be used to
test the equalities in equation (2). For large samples, - and the current sample is very large -

7
these three tests are equivalent. If the resulting test statistic indicates that the parameters in (2)
are not significantly different between the groups under study then we can conclude that the
indicator Xi has been measured invariantly across the groups under study (Billiet &
Welkenhuysen-Gybels, 2004).
Multiple fit statistics have been suggested for evaluating the fit of a structural equation model
(Bollen, 1989; Bollen & Long, 1992). Several of these fit statistics have been developed as
alternatives to the Chi-statistic (), which is very sensitive to the sample size. Since cross-
cultural survey research typically uses very large samples, we will not use the Chi-statistic as
a measure of fit. Instead we will rely on the Root Mean Square Error of Approximation
(RMSEA) and the Comparative Fit Index (CFI) to evaluate overall model fit. Values for the
RMSEA of 0.05 or less indicate good model fit; values up to 0.08 represent reasonable errors
of approximation (Browne & Cudeck, 1993). For the Comparative Fit Index, values of 0.90 or
higher are taken to indicate good fit (Bentler, 1990).
It was not possible to obtain a complete scalar invariant measurement model with the four
observed indicators. In subsequent steps, 31 parameters have to be set free before a model was
found that could not be improved substantially (see Appendix 1). According to our criteria,
the final model has an acceptable fit. The parameters of Model 31 are given in Table 1.

Table 1. Measurement model for the attitude towards equal treatment of foreigners in 20
countries of ESS Round 1.

Factor loadings (slopes) () D22 D36 D45 D50

Common slopes -0.55 -0.59 0.43 -0.42

FI -0.83 -0.26
GR 0.17
HU 0.13
NL -0.22
PT -0.27

Intercepts ()

Common intercepts 0.23 0.26 - 0.12 0.11

AT -0.26 -0.19
CH -0.29 -0.27
CZ -0.27

8
 DE -0.03
DK -0.54
ES 0.34
FI 0.85 -0.27
GB 0.66
GR -0.35 -0.25 -0.16
HU -0.07
IE 0.01
IT -0.13
LU -0.35
NO 0.53 0.49
PL 0.35
PT 0.43 0.51
SI 0.06 -0.12

As can be derived from the signs of the factor loadings, high values on the latent variable express
a positive attitude towards equal treatment, and low values express a negative attitude. The
measurement quality of this latent variable, evaluated by the strength of the relationship between
indicators and latent variable (the slopes), is of lesser quality than the other latent variables in the
immigration module.5 The reason for this is that the other sets were designed as indicators for the
constructs whilst the four indicators for measuring the attitude towards equal treatment of
foreigners are selected ad hoc from the immigration module. The meaning of the latent variable
is most determined by items D22 and D36. This means that the latent variable refers in first place
to immigrants (people who have come to live here) and to equal treatment (same rights,
appointed as your boss). The central items D45 about a law against discrimination in the
work place is not so strongly related to the latent variable. Because of this, we shall also analyse
it on its own.
As can be seen in table 1, the factor loadings (slopes) are not invariant over all countries. In
Finland, the loading of D36 is considerably higher and the loading of D50 considerably lower. In
Greece and Hungary, and in the Netherlands and Portugal, respectively item D45 (law on
discrimination) and item D50 (allowed to work in refugee status) are less strongly related to the

5
It was not possible to improve measurement quality by dropping item D45 or item D50. The loadings in the completely
invariant model are still low: 0.53 (D22), 0.36 (D36), 0.43 (D50) or 0.47 (D22), 0.61 (D36), -0.44 (D45). The quality is
even lower.

9
latent variable. For these five countries, full metric invariance is absent. However, we already
mentioned that only partial equivalence is needed in order to make cross-cultural comparisons
(Byrne et al., 1989; Steenkamp & Baumgartner, 1998). Given that we choose D22 and D45 to
calibrate the latent variable scale (see below), partial metric equivalence is obtained for 18 out of
20 countries. Indeed, only Greece and Hungary have deviating loadings on one of the calibration
items (namely D45).
Scalar invariance is more problematic. In 17 out of 20 countries, at least one intercept deviates
from the reference country (Belgium). However, latent mean comparisons are still possible if
we resort to partial equivalence. Items D22 and D45 are chosen to calibrate the scale as these
are the two items on which least deviations are present. According to this criterion of partial
invariance we may compare the latent means of all countries that have the intercepts and
slopes for the indicators D22 and D45 in common. The latent means of Greece, Hungary,
Czech Republic, Ireland, and Denmark are not comparable with the other countries since
these do not meet the criterion of two common invariant indicators.

10
Findings

BELONGING TO DISCRIMINATED GROUPS

What is the portion in each country of persons who feel that they belong to a minority that is
discriminated against for whatever reason it may be? Table 2 shows the percentages of
respondents who describe themselves as being a member of a group that is discriminated
against in total country samples (left part of the table) and the percentages of respondents who
describe themselves as belonging to such a group for reasons of colour, race, ethnic origin, or
nationality in a sub-sample of respondents of foreign origin6 (right part of the table). The
proportions of respondents of foreign origin in the total samples are displayed in the last
column.
The number of respondents that perceive themselves as belonging to a discriminated group
varies between 2.3% in Portugal and 13.0% in the United Kingdom. The portions are between
5% and 9% in ten other countries. These numbers are much higher when only the population
of foreign descent is considered. Between 1% (Italy) and 34% (Luxemburg) of the
respondents in the country samples report their foreign origin (see last column in Table 2).

Table 2. Percentages of respondents who described themselves as being members of a group

that is discriminated against in ESS Round 17

% belonging to a discriminated Proportion

% belonging to a discriminated group for reasons of colour, race, respondents
group ethnic origin, nationality of foreign
Country (total sample) (population of foreign origin) origin

Yes N (100%) Yes n (100%) (n/N)

UK 13.0 2043 23.2 171 0,084

SE 9.0 1992 17.2 104 0,052

FI 8.3 1995 17.6 47 0,024

NL 7.1 2356 27.7 128 0,054

GR 7.0 2546 38.6 174 0,068

6
Here, the term of foreign origin is used to refer to persons who are not a citizen of the country or who consider
themselves to be a member of an ethnic minority group.
7
The percentages of persons that consider themselves as belonging to a group discriminated against are weighted for
gender, age and education.

11
 BE 6.5 1884 11.4 122 0,065

IE 6.2 2037 16.0 96 0,047

AT 6.0 2212 20.6 192 0,087

SE 5.6 1712 40.3 84 0,049

SI 5.4 1499 6.7 47 0,031

HU 5.3 1682 29.5 128 0,076

NO 4.9 2036 11.5 95 0,047

DE 4.8 2908 16.7 182 0,063

DK 4.8 1490 16.0 55 0,037

CH 4.7 2034 9.1 275 0,135

PL 3.7 2087 0.9 58 0,028

LU 3.5 1537 3.3 523 0,340

CZ 3.4 1331 30.7 33 0,025

IT 2.3 1198 22.9 13 0,011

PT 2.3 1490 36.1 41 0,028

There are indications that these figures are downward biased because of nonresponse and
sample design characteristics, which led to an under-representation of the population of
foreign origin.
Within the sub-samples of respondents of foreign origin, the number of respondents that
perceive themselves as belonging to a group that is discriminated against because of ethnicity,
colour, race, or national belonging is more than 30% in Spain, Greece, Portugal, and Czech
Republic, and less than 10% in Switzerland, Slovenia, Luxemburg, and Poland (see column 4
in Table 2). Of the countries in our study, only four (namely Switzerland, Czech Republic,
Greece and Portugal) do not have a law against discrimination. Three of these are countries in
which more than 30% of the population of foreign origin perceived themselves as belonging
to a discriminated group because of colour, race, ethnic origin, or nationality.

ATTITUDE TOWARDS EQUAL TREATMENT OF FOREIGNERS

What are the between country differences in the attitude towards equal treatment of
foreigners? How is this attitude related to other attitudes towards immigrants? Among the
countries in which the measurements are scalar equivalent, there are serious differences in
latent means. In countries such as Sweden, Luxemburg, Finland, and Portugal, the attitude in

12
favour of equal treatment is highest, whilst in countries such as Switzerland and Slovenia the
mean attitude score is significantly lower. It is very likely that the scores are lowest in
Hungary and Greece but we cannot conclude this in a valid way since we restrict ourselves to
countries with scalar invariant measures. Our strict statistical and measurement criterion is
possibly rather conservative here, since we know also from other attitudes in ESS round 1,
and from other sources that negative attitudes towards foreigners are indeed most pronounced
in these two countries. The attitude towards immigration is most negative in Hungary and
Greece (Billiet & Welkenhusen-Gybels, 2004). In the European Value Study of 1999,
Hungary and also to a lesser degree Greece, count the largest proportions of respondents who
do not want to have immigrants, foreign workers, or Muslims, as neighbours (Halman, 2001:
40-41).

Table 3 Latent means of attitude towards equal treatment of foreigners .

Country Latent mean () t-value*

SE 0.68 31.01
LU 0.62 25.16
PT 0.58 18.67
FI 0.50 19.93
NO 0.40 16.00
ES 0.38 16.23
DK** 0.37 15.49
IE** 0.27 12.57
PO 0.23 11.39
UK 0.15 7.13
NE 0.14 6.97
AT 0.09 3.30
IT 0.04 1.64
DE 0.00 -0.03
BE 0.00 --
CZ** -0.01 -0.65
SI -0.13 -4.80
CH -0.16 -6.45
HU** -0.23 -8.29

13
 GR** -0.23 -8.45

* Parameter/SE
** Not scalar equivalent. Not compare with, or test against other means
The countries that obtain most support for equal treatment of foreigners by their inhabitants
appear also in several studies as the most open societies towards immigration. This is the case
for the Scandinavian countries (except Denmark), and for Luxemburg and Portugal.
Luxemburg seems to be a special case since this is - compared to the other countries - a very
small country. We should however take into account that it contains the largest proportion of
foreigners (34% according to our estimation in Table 2), mostly of Portuguese origin.
The attitude towards equal treatment is strongly negatively correlated with ethnic prejudice (r
varies between -0.49 in Germany and -0.26 in Luxemburg but in most countries stronger than
-0.30), and positively correlated with the view that the presence of immigrants in the country
gives a number of opportunities (r varies between 0.50 in Germany and 0.21 in Italy but
higher than 0.35 in most countries).

SUPPORT FOR A LAW AGAINST DISCRIMINATION

What is the support for a law against discrimination in the workplace? This question can be
answered by means of item D45, a variable that was also used as an indicator for the attitude
towards equal treatment of foreigners. Yet, we analyze this variable as a single item here
since discrimination at the work floor is the theme of the conference. What is the variation of
support for a law against discrimination at the workplace? Are firm differences observed
between countries that have passed a bill against discrimination and the level of acceptance of
such a law?
The mean scores at the 11-point scale expressing the support in favour of a law against
discrimination in the workplace are also highest in the countries that score highest at the equal
treatment variable, and are lowest in Greece, Switzerland, Italy, and Denmark. In order to
display somewhat more variation in the response, Table 4 displays the percentages of
respondents in each country who hold strong views in favour of equal treatment (scores 8, 9,
10 at the response scale), and the percentages of those who are at the other side of the scale
(scores 0, 1, 2, 3).8 Altogether, 50% of the Europeans who are included in ESS round 1
support a law against discrimination in the workplace. The highest support is observed in the
countries in which the citizens have most favourable attitudes towards equal treatment. This is

8
The decision of deviding the two sides not completely symmetrical (three versus four scores colapsed) is suggested by the
skewness of the observed variables with small numbers in the lowest categories (0,1)

14
not surprising because discrimination and unequal treatment are conceptually close.
Moreover, the item on a law against discrimination is one of the indicators of the attitude
towards equal treatment of foreigners as it is measured here.
Lowest support is in Greece, Czech Republic, Switzerland, three countries in which no bill
against discrimination was passed. Low levels of support for a law against discrimination are
also observed in Belgium and in Italy. The difference in support between the five countries at
the bottom of the distribution and the five on top is more than 20 percent points. This is a big
gap which can in our view go hand in hand with another cultural and political climate in
society. Much has still to be done in order to uncover the social, political, economic,
historical, and structural factors that can explain these differences.

Table 4. Percentages of respondents with a marked positive attitude (scores 8,9,10 and a
marked negative attitude (scores 0,1,2,3) towards a law against discrimination at the work
place because of ethnic origin.

Country % pro law % against law

SE 68.7 12.7
FI 65.6 11.6
LU 64.6 14.1
PT 63.2 4.3
IE* 62.6 11.7
NO 60.8 9.8
ES 56.8 11.6
GB 56.3 10.6
PL 55.1 13.4
NL 51.3 10.2
DE 49.8 13.7
DK* 47.5 25.4
HU 46.8 15.6
AT 46.4 14.5
SI 46.2 19.9
IT 43.6 21.5
BE 43.1 16.4
CH 42..6 18.9
CZ 41.3 14.7
GR* 40.8 25.8
* The intercepts for this indicator were not invariant in this country (see table 1),
consequently the meaning of the question may be not completely comparable.

15
DIFFERENCES IN MODELS EXPLAINING THE ATTITUDE TOWARDS EQUAL TREATMENT

What background and context characteristics explain the variation and the country differences
in the attitude towards equal treatment of foreigners? This rather complex question requires
information on a number of relevant variables at individual, region, and country level. Not all
relevant variables are available at the individual level, and much has still to be done in order
to collect data at the relevant context levels. In order to adequately answer the question, and
assumed that all relevant data is available, a multilevel model is in principle the most
adequate approach to study the simultaneous effects of individual and context level factors on
the attitude. However, a recent simulation study has revealed that the limited number of
countries in ESS is not large enough in order to obtain reliable and stable estimates in
multilevel structural equation models (Meuleman, Reeskens & Billiet, 2007). As we want to
stay in the environment of structural model testing (SEM) using the measurement model that
was found in the previous part of this study, we decided use multi group structural equation
modelling and to compare structural effects across countries.
In view of the restrictions mentioned, the estimated structural parameters of three social-
background variables (gender, age, education) are compared across countries in order to find
out whether the same (small) set of background variables explains the attitude towards equal
treatment of foreigners in the same way in all countries.

16
Table 5. Estimated structural parameters for the multi group structural equation model with
social-background variables (gender, education, and age) in ESS Round 1.

Effect on discriminatory attitudes

Country Gender* Education Age

Stand. par. t-value Stand. par. t-value Stand. par. t-value

AT 0.12 (2.90) 0.23 (5.68) -0.23 (-7.18)

BE 0.01 (0.19) 0.24 (6.30) -0.10 (-3.07)

CH 0.06 (1.54) 0.07 (2.13) -0.25 (-7.89)

SZ 0.18 (3.35) 0.33 (7.35) -0.20 (-5.43)

DE 0.13 (3.98) 0.17 (5.56) -0.28 (-10.73)

DK -0.02 (-0.39) 0.21 (5.56) -0.24 (-6.77)

ES 0.10 (2.19) 0.16 (3.26) -0.13 (-2.75)

FI 0.24 (6.58) 0.25 (7.55) -0.07 (-2.55)

UK -0.06 (-1.73) 0.22 (4.44) -0.31 (-8.46)

GR** 0.01 (0.26) 0.23 (6.36) -0.09 (-2.97)

HU** -0.05 (-1.20) 0.16 (4.47) -0.05 (-1.91)

IE -0.09 (-2.38) 0.23 (6.56) -0.12 (-3.92)

IT -0.02 (-0.48) 0.22 (4.45) 0.03 (0.54)

LU -0.07 (-1.57) 0.05 (1.07) -0.20 (-5.89)

NL 0.01 (0.35) 0.26 (8.39) -0.04 (-1.47)

NO 0.12 (3.38) 0.23 (6.07) -0.14 (-3.95)

PL -0.03 (-0.86) 0.20 (6.21) -0.09 (-3.28)

PL -0.02 (-0.53) 0.26 (5.10) 0.07 (1.60)

SE 0.20 (5.51) 0.17 (5.45) -0.35 (-7.58)

SI 0.07 (1.71) 0.20 (5.21) -0.07 (-2.01)

Fit indices = 2010.14 Df = 277 RMSEA = 0.060

* Reference = male
** Parameters not comparable with other countries

17
Still according to our rules concerning partial metric equivalence, the parameters of Greece
and Hungary are not comparable with the other countries. Education has the most solid effect
on the attitude towards equal treatment of foreigners. In all countries but one, a significant
effect (alpha = .05) is found in the sense that the higher educated are more supportive for
equal treatment. Only in Luxemburg, education seems to have no impact at all. Also with
respect to the age-effect, the country-specific effects are quite consistent. In 16 out of 20
countries, positive attitudes towards equal treatment seem to prevail especially among
younger persons. The evidence for gender is mixed. In some countries such as Finland,
Sweden, Germany, Czech Republic, Germany, Norway, Austria, and Spain, females are more
in favour of equal treatment of foreigners than their male counterparts. In the other countries
however, there is no effect of gender on the attitude, and in Ireland an effect in the opposite
direction is found.
The effect of education level on the attitude towards equal treatment of foreigners is
consistent, and in line with existing evidence about the effect of education on ethnocentrism
(Wagner & Zick, 1995). In nearly all countries, the score on the (positive) attitude towards
equal treatment increases significantly when the education level increases. This relation was
already found and explained by Allport who explains it by feelings and cognitive skills:
Perhaps it is because higher education lessens feelings of insecurity and anxiety. Or perhaps
education enables the individual to see the social scene as a whole, and to comprehend that
the welfare of one group is linked to the welfare of all groups (Allport, 1954/1997: 433).
More recent overviews explain this relation by attitudes as group relative deprivation,
perceived belief incongruence, political conservatism, and acceptance of inter-ethnic contact
as intermediate variables. The positive relation between education level and the attitude
towards equal treatment is most pronounced in the Netherlands, Czech Republic, and Finland.
However, there are two exceptions. This relation does not exist in Luxemburg, and is rather
weak in Switzerland. The specific situation with a large amount of highly educated immigrant
workers may be responsible for the absence of the relation with education in Luxemburg.
The effect of age is also in line with findings about the relation between education and ethnic
prejudice in general. Age is in most countries negatively related with the attitude towards
equal treatment. Older respondents are more likely to score lower on this attitude. This is most
pronounced in Germany, the United Kingdom, Switzerland, Sweden, Austria, and Denmark.
This relation is weak in Spain, Finland, and Slovenia and not significant in Hungary, the
Netherlands, Poland, and Ireland. It appears that the effect of age is higher in countries where

18
the effect of education is lower. This may be explained by the interrelation between education
level and age in most countries.

Discussion: some measurement problems

We will finalize this paper with a discussion of four measurement problems that we encounter
in the analysis of ESS data concerning aspects of attitudes towards immigrants; (1)
nonresponse bias for this kind of variables in comparative research and the restrictions in the
ways of reducing this bias; (2) the level of measurement invariance needed in view of the kind
of analysis to be executed; (3) a plea for the inclusion of split ballots and repeated
measurement within substantive surveys; (4) identifying the impact of contextual variables.

NONRESPONSE BIAS

We can expect nonresponse bias in both kind of questions that were asked concerning
discrimination and equal treatment in ESS Round 1. First, the detection of the proportion of
inhabitants in each country that feel that they belong to a group which discriminated against
for reasons of ethnic origin, race, colour, of national origin has very probably been
underestimated because the endorsement of this question is most likely among
nonrespondents. We have observed that the endorsement of that question is proportionally
much higher among the sub-populations of foreign origin (see Table 2). The selected sample
units in these populations are very difficult to reach because they are less likely to live at a
permanent address, are less likely to appear in sampling frames (administrative or commercial
lists), and are also less likely to cooperate in an interview because of language problems.
From a comparative viewpoint, there may be bias in the cross-country comparisons because
among other reasons the mentioned factors that effect cooperation in a survey depend on
countrys policy against immigrants, asylum seekers, and other kinds of foreigners.
Second, the scores on the latent variable attitude towards equal treatment of foreigners is
likely to be biased in the direction of an overestimation of those who support this attitude. In
other words, we assume that the estimation of the mean score on the latent variable is biased
in upward direction because of nonresponse, and that the bias is stronger in countries with low
response rates than in countries with high response rates. This assumption is based on
previous panel research in which it was found that respondents who have less interest in
politics, and who are more individualistic, are less likely to cooperate in subsequent panel
waves (Loosveldt & Carton, 2001a, 2002b; Pickery et al., 2001). Because of a strong relation

19
between these two variables and the ethnic prejudice, we may expect that respondents with
negative feelings towards immigrants are less likely to cooperate in surveys. Another
assumption is then that this relation also applies to refusals in cross sections. At basis of these
assumptions, we expected that cooperative respondents would have lower scores on ethnic
prejudice than reluctant respondents; these are respondents who first refused and later were
converted. This hypothesis was tested with the samples of Germany and the Netherlands in
which more than 480 reluctant respondents participated in round 1 of ESS. It was found that
in both countries the mean scores on ethnic threat were significantly higher among the
reluctant respondents than among the cooperative respondents (Billiet et al., 2007). The weak
point in this reasoning is that one must rely on the so-called continuum of resistance model
which assumes that the difficult to approach respondents and the reluctant respondents have
much in common with the sample units who finally refuse to cooperate. Because of this,
persuading reluctant sample units yields a more representative or balanced sample as response
rates increase (Stoop 2004; Teitler et al. 2003; Voogt 2004).
Given the findings mentioned, and the assumptions made, it is thus expected that the mean
scores on the attitude construct equal treatment of foreigners (scores 0 to 10), and on the
question measuring the support to a law against discrimination (D45) will be higher among
the cooperative respondents than among the reluctant respondents in Germany and the
Netherlands. The results in the Netherlands are in line with our expectations. The differences
in mean scores of the attitude towards equal treatment of foreigners are in the expected
direction since these scores are larger among the cooperative respondents (6.233) than among
the reluctant ones (6.087), and this difference is significant at the 0.05 level (t-value = 2.13).
There is also a significant difference (t-value = 2.86) in the mean score of the other variable
(D45): 7.058 for cooperative respondents versus 6.687 for reluctant respondents. There are no
significant differences in Germany. Table 6 shows the simple regression parameters ( in
Model 1), and the regression parameters after including the three background variables (age,
gender, education into the regression model ( in Model 2). The significant (unstandardized)
-coefficients in the simple regression model with type of respondents as a predictor
(Model 1) give the same information as the differences of means test. The probabilities that
the coefficient are zero are smaller than 0.05 (actually 0.03 for t = -2.17, and 0.004 for t = -
2.86). These coefficients are negative for the reluctant respondents since the reference
category refers to the cooperative respondents who are more in favour of equal treatment or
a law against discrimination. Assuming that regression parameters of type of respondent on

20
the attitudes are indications of bias, we may conclude that the bias in the law against
discrimination question (D45) does not completely disappear after introducing background
variables as age, gender, and education into the model (Model 2).

Table 6. Regression parameters as indications of bias in the attitude towards equal treatment
of foreigners, and support for a law against ethnic and ethnic discrimination (D45) in the
Netherlands and in Germany.

Simple and multiple regression s

Country Type of
respondent Equal treatment Law against discrimination (D45)

NL Reluctant
-0.149 -0.100 -0.370 -0.286
Ref: cooperative

t-statistic -2.17 -1.49 -2.86 -2.25

DE Reluctant
-0.061 -0.051 0.057 0.058
Ref: cooperative

t-statistic -0.78 -0.65 0.40 0.41

The differences are in Germany in the expected direction for equal treatment, but not for the
support of a law against discrimination (D45), and none of the parameters are significantly
different from zero.
In this study, the (compulsory) design weights that are correcting for unequal probabilities
because of differences in sampling designs, were completed by weights according to the joint
distributions of gender, age and education in the populations. We assume that large deviations
between the sample and population distributions are not so much affected by the sampling
frame and by the selection procedures but by nonresponse. According to this view (and
assumption), the applied weightings correct for bias. From our analysis it is clear that the bias
in substantial (attitude) variables is not completely under control by using these weights. In
the Netherlands where it was possible to estimate the bias under the assumption that reluctant
respondents are informative for final refusals, we found that the (assumed) bias still remains
after controlling for gender, age, and education. Other methods for further bias adjustment are
required.

CROSS-NATION EQUIVALENCE OF QUESTIONS AND CONCEPTS

21
In the analyses in this paper, metric as well as scalar invariance of the indicators for the
concept equal treatment of foreigners was tested. The conclusion was that the items are not
completely scalar invariant across all countries. Note, however, that this does not necessarily
imply that these items cannot be used for any comparisons across the countries. Equivalence
is not a black or white concept, but there are several shades of grey. The observation that not
complete metric and scalar invariance was obtained, does not imply that for most countries
the means of the latent variable are not comparable. Since complete scalar and metric
invariance is very difficult to obtain, it is acceptable to fall back on partial scalar equivalence
as defined by Byrne et al. (1989) and Steenkamp & Baumgartner (1998). That way it was
possible to compare the latent means between fifteen countries. Even when the latent means
are not comparable for all countries, the items can still be used for other kinds of analysis for
which only metric equivalence has been required. We have seen that metric equivalence was
realized for eighteen countries.
In the inspection of equivalence, we have confined ourselves by identifying the countries in
which the parameters are comparable. However, this is only one way of handling
inequivalence. There are several other ways of coping with invariant slopes and intercepts in
measurement models. It is possible to drop problematic items and to rely on a concept which
is smaller in meaning. We did not proceed that way since we had only four items to start with.
Another possibility is to leave countries completely out of the analysis. This was also not our
choice. All countries were kept in the tables, but the problematic cases are flagged, and by
doing this, we have opened a way for further research. Indeed, important questions that are
still open are: why are precisely these items not invariant in these countries? Several answers
are possible. The meaning of items can differ because of a specific cultural and political
context, or simply translation errors may have occurred as was already found in a number of
cases. Deviations from equivalence can also be caused by cross-national differences in the
amount of measurement error that is present (Saris, 1998). In other words, the detection of
invariance is not a finishing point but just a challenge for further research using additional
data that is not included in the surveys.

REPEATED MEASUREMENTS AND SPLIT SAMPLES

In order to assess the validity and reliability of the indicators, in each round of ESS a well
chosen set of repeated measurement in combination with split samples has been included.
This is done in both the pilots and the main surveys. At basis of a multitrait multimethod
approach (MTMM) in the pilots, the questions and response scales are selected that are least

22
sensible for random error and measurement error. The MTMM experiments in the main
surveys are designed in view of adjustments for measurement error (Saris, 1998; 2003). The
survey questions that are used in this study were not selected for inclusion in the MTMM
experiments. Consequently, it is not possible to estimate their reliability or to detect
systematic error. The only way we had for assessing the validity was the strength of the
relations between the indicators and the latent variable in the measurement model. However,
built in question wording experiments in substantive surveys may lead to considerable
improvement of the measurement quality of the questions, and can provide a much better
understanding of the phenomena that are studied (Sniderman, 1996).
An example of two questions about discrimination that are used in our own research may
illustrate this. In a study about the political knowledge of the Flemish (1996), we included two
test questions about symbolic racism:
(1) Suppose that there are two employees, one is Flemish, the other belongs to an ethnic
minority. Both are equal on all characteristics. When one of them has to be dismissed
because of the companys financial situation, who should be dismissed according to you,
the Flemish or the employee belonging to an ethnic minority?
(2) Suppose again there are two employees, one is Flemish, the other belongs to an
ethnic minority. Both are equal on all characteristics. When only one of them can have a
promotion, who should according to you get the promotion, the Flemish or the employee
belonging to an ethnic minority?
Two forms of the questions were assigned to two random samples, one sample received the
questions exactly as worded before (sample A), another with an additional alternative or
should this not make any difference? (sample B). The response distributions are below.

Table 7. Response distributions on two symbolic-racism questions in a split ballot

(percentages)

Response Dismissal of an employee Promotion of an employee

A B A B

Ethnic minority 45.1 25.9 4.0 0.8

Flemish 7.4 6.4 47.1 31.7
No difference 39.5 65.1 43.0 65.5
No opinion 4.0 0.6 2.8 0.6
No answer 3.9 2.0 3.2 1.4

23
 N 532 455 532 455

What question wording and response scale is most valid, A or B? Some would choose B
because the response scale is most complete with a reasonable middle alternative. Others
would prefer A because more social undesirable responses are obtained. Actually, the best
solution is using both questions in a split ballot (as is shown), or as repeated measurements at
different places in the questionnaire (with question wording A first), or a combination of both
repeated measurement and split ballot. Why? Because we obtain additional information and a
better insight when both question wordings are used. The questions without middle alternative
(A) provide an estimation of the portion in the population that is spontaneously in favour of
equal treatment; the questions with a middle alternative give a better estimation of the portion
in the population that supports unequal treatment, even when an acceptable solution is
suggested in the middle alternative. Additionally, we can estimate the portion with rather
unstable opinions.

IDENTIFYING THE IMPACT OF CONTEXTUAL VARIABLES

Perhaps one of the most appealing characteristics of comparative research is the possibility to
identify the impact of the context on individual characteristics, such as attitudes. The
theoretical relevance of this contextual approach is paramount. After all, the idea that
individuals are influenced by the broader context they are situated within, is a cornerstone of
sociological theory. From a practical point of view however, the study of context effects is far
from evident, as various methodological issues have to be dealt with. In this paragraph, we
would like to draw attention to two of these methodological problems.
Often, multilevel models are used for the purpose of studying national context effects (see for
example Coenders & Scheepers, 2003; Seymonov et al., 2004; Hooghe et al., 2006).
However, statistical inference with respect to the relation between context variables at the
national level and individual characteristics is problematic for several reasons. The number of
countries in cross-national surveys rarely exceeds 30 for budgetary and organisational
reasons. In the case of the ESS, between 22 and 26 countries participated in the various
rounds. Similar group sample sizes are too small for reliable statistical inference, a problem
also referred to as the small-N problem (Goldthorpe, 1997). A recent simulation study pointed
out that substantially larger number of countries are needed for multilevel structural equation
modelling to perform well. Data on at least 60 to 100 countries, depending on the expected
size of the context effect, would be required to guarantee accurate estimation (Meuleman et

24
al., 2007). This problem could be by-passed by using regional instead of national level data.
Statistical inference is also troubled by the fact that the countries in the study do not constitute
a random sample, which can introduce selection bias.
A second major challenge in the domain of national context effects is the availability of
reliable and comparable context variables. Despite efforts of international organisations that
have placed comparative databases at the disposal of the general public - such as OECD or the
European Union - the availability of context data is rather limited. The lions share of the
indicators is related to economics or demography. The comparability of variables that relate to
other domains such as immigration - is often questionable; because of international
differences is the definition or operationalisation of certain concepts (Lemaitre, 2005). As a
result, researchers often do not dispose of the theoretically most relevant indicators to test
their context hypotheses, but are instead obliged to use proxy variables. Of course, this can do
serious harm to the theoretical relevance of comparative research.

References

Abelson, R.P. (1988). Conviction. American Psychologist. Vol. 43: 267-275.

Allport, G. (1954). The Nature of Prejudice. NY: Doubleday & Co (25th Anniversary
Edidtion of 1979: Addison Wesley).

Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin,

107, 238-246.

Billiet, J. (1993). Stabiliteit en verandering in de attitude tegenover vreemdelingen. Pp. 147-

163 in: Swyngedouw, M., Billiet, J., Carton, A. & Beerten, R. (red.), Kiezen is verliezen.
Onderzoek naar de politieke opvattingen van Vlamingen. Leuven: Acco.

Billiet, J. (2002). Cross-cultural Equivalence with Structural Equation Modeling . Pp. 247-265
in: J. Harkness, F. Van de Vijver &. P. Mohler (Eds.). Cross-cultural Survey Methods. New-
York: Wiley.

Billiet, J. & Welkenhuysen-Gybels, J. (2004). Assessing cross-national construct equivalence in

the ESS: the case of six immigration items. In: Van Dijkum, C., Blasius, J. & Durand, C. (Eds.),
Recent Developments and Applications in Social Research Methodolog. Barbara Budrich Pub.
Proceedings of the Sixth International Conference on Social Science Methodology. August 17-
20 2004, Amsterdam. 19 pp.

25
Billiet, J., Phillipens, M., Fitzgerald, R. & Stoop, I. (2007), Estimation of Nonresponse Bias
in the European Social Survey: using Information from Reluctant Respondents. Journal of
Official Statistics, vol. 23 (2), pp. 1-29.

Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley.

Bollen, K. A., & Long, J. S. (1992). Tests for structural equation models. Sociological
Methods and Research, 21, 123-131.

Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A.
Bollen, & J. S. Long (Eds.), Testing structural equation models (pp.445-455). Newbury Park:
Sage.

Byrne, B.M., Shavelson, R.J. and B. Muthn (1989) Testing for the equivalence of factor
covariance and mean structures: the issue of partial measurement invariance. Psychological
Bulletin. Vol. 105 (3): 456-466.

Coenders, M., & Scheepers, P. (2003). The Effect of Education on Nationalism and Ethnic
Exclusionism: An International Comparison. Political Psychology, 24(2), 313-343.

Feagin, J.R. & Eckberg, D.L. (1980). Discrimination: Motivation, Action, Effects, and
Context. Annual Review of Sociology. Vol. 6: 1-20.

Finkelstein, L.M., Burke, M.J. & Raju. N.S. (1995). Age discrimination in simulated
employment contexts: an integrative analysis. Journal of Applied Psychology. Vol. 80 (6):
652-663.

Goldthorpe, J. H. (1997). Current issues in comparative macrosociology: a debate on

methodological issues. Comparative Social Research, 16, 1-26.

Halman, L. (2001). The European Values Study: A Third Wave. Source book of the 1999/2000
European Values Study Surveys. Tilburg: Tilburg University.

Harkness, J., Van de Vijver, F., & Mohler, P. (2003). Cross-cultural Survey Methods. New
Jersey: Wiley & Sons.

Holt, D. & Smith, T.M.F. (1979). Post Stratification. Journal of the Royal Statistical Society.
Series A, Vol. 142 (1): 33-46.

Hooghe, M., Reeskens, T., Stolle, D., & Trappers, A. (2006). Ethnic diversity, trust and
ethnocentrism and Europe: a multilevel analysis of 21 European countries. American
Political Science Association, Philadelphia (31 August-3 September 2006) pp. 26.

26
Horn, J.L. & McArdle, J. (1992). A practical and theoretical guide to measurement invariance
in aging research. Experimental Aging Research, Vol. 18, 117-144.

Johnson, T. P. (1998). Approaches to equivalence in cross-cultural and cross-national survey-

research. ZUMA Nachrichten Spezial, Cross-Cultural Survey Equivalence, Vol. 3: 1-40.

Jreskog, K.G. (1990). New developments in LISREL. Analysis of ordinal variables using
polychoric correlations and weighted least squares. Quality and Quantity, 24, 387-404.

Jreskog, K.G., & Srbom, D. (1993). LISREL 8 User's Reference Guide. Chicago:
Scientific Software International.

Jowell, R., Roberts, C., Fitzgerald, R. and Eva, G. (Eds.) (2007). Measuring attitudes cross-
nationally: lesions from the European Social Survey, London: Sage.

Krosnick, J. (1988). Attitude importance and attitude accessibility. Personality and Social
Psycghology Bulletin. Vol. 15: 297-308.

Lapierre, R.T. (1934). Attitudes versus Actions. Social Forces. Vol. 13: 230-237.

Lemaitre, G. (2005), The Comparability of International Migration Statistics. Paris: OECD

(Statistics Brief 9).

Linn, L.S. (1965). Verbal Attitudes and Overt Behaviour: a Study of Racial Discrimination.
Social Forces. Vol. 43 (3): 353-364.

Loosveldt, G. & Carton, A. (2001). An empirical tet of a limited model for panel refusals.
International Journal of Public Opinion Research. Vol. 13 (2), pp. 173-185.

Loosvelt, G. & Carton, A. (2002). Utilitarian Individualism and panel nonresponse.

International Journal of Public Opinion Research. Vol. 14 (4), pp. 428-437.

Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance.

Psychometrika. Vol. 58: 525-543.

Merton, R.K. (1940). Facts and Factiousness in Ethnic Questionnaires. American Sociological
Review. Vol. 5: 13-28.

Meuleman, B., Meireman, K., & Billiet, J. (2006). Immigratie en asiel. Opvattigen en
houdingen van Belgen, Duitsers, Nederlanders en Britten in het Europees Sociaal Survey . Pp.
63-88 in: Foblets, M.C., Vrielink, J. & Billiet, J. (Red.). Multiculturalisme ontleed. Een
staalkaart van onderzoek aan de K.U. Leuven. Leuven: Universitaire Pers Leuven.

27
Meuleman, B., Reeskens, T. & Billiet, J. (2007). How many countries are needed for
multilevel SEM? A simulation study. Paper presented at the Sixth International Multilevel
Confernce, Amsterdam, April 16-17, 24 pp.

Meuleman, B. & Billiet, J. (2005). Corrections for nonresponse in the ESS round 1 :
weighting for background variables . A simualtion. Research Report of the Center for
Sociological Research (CeSO). DA/2005-49, 17 pp

Meuleman, B. & Billiet, J. (2006). Quality and comparability of measures in the ESS:
attitudes towards migration and immigrants. Paper presented at the 25th Biennial Conference
of the Society for Multivariate Analysis in the Behavioural Sciences (SMABS), Prague, July
2nd-5th, 18 pp.

Mummendy, A. Simon, B., Dietze, C., Grnert, M. Haeger, G., Kessler, S., Lettgen, S. &
Schferhoff, S. (1992). Categorization is not enough: intergroup discrimination in negative
outcome allocation. Journal of Experimental Social Psychology. Vol. 28 (2): 125-144.

Phillips, S.T. & Ziller, R.C. (1997). Towards a theory and measure of the nature of
nonprejudice. Journal of personality and Social Psychology. Vol. 72 (2): 420-434.

Pickery, J., Loosveldt, G. & Carton, A. (2001). The effects of interviewer and respondent
characteristics on response behavior in panel surveys: a multilevel approach. Sociological
Methods and research. Vol. 29 (4), pp. 509-523.

Saris, W. (1998). The Effects of Measurement Error in Cross Cultural Research. ZUMA-
Nachrichten Spezial. January 1998, pp. 67-84.

Saris, W. (2003). Multitrait Multimethod Studies. Pp. 265-274 in: Harkness, J., Van de
Vijver, F., & Mohler, P. (2003). Cross-cultural Survey Methods. New Jersey: Wiley & Sons.

Scott, J. & Schuman, H. (1988). Attitude Strength and Social Action in the Abortion Dispute.
American Sociological Review, 53 (5): 785-793.

Sears, D.O., Van Laar, C., Carrillo, M. & Kosterman, R. (1997). Is it Really Racism? Public
Opinion Quarterly. Vol. 61 (1): 16-53.

Semyonov, M., Raijman, R., Yom Tov, A., & Schmidt, P. (2004). Population size, perceived
threat, and exclusion: a multiple-indicators analysis of attitudes toward foreigners in
Germany. Social Science Research, 33, 681-701.

28
Simpson, G.E. & Yinger, J.M. (1985). Racial and Cultural Minorities: an Analysis of
Prejudice and Discrimination. New York: Plenum Press (5th edition).

Steenkamp, J.-B. E. M., & Baumgartner, H. (1998). Assessing measurement invariance in

cross-national consumer research. Journal of Consumer Research. Vol. 25: 78-90.

Sniderman, P.M. (1996). Innovation in experimental design in attitude surveys. Annual

Review of Sociology. Vol. 22, pp. 377-399.

Stewart, D.L., Perlow, R. (2001). Applicant Race, Job Status, and Racial Attitude as
Predictors of Employment Discrimination. Journal of Business and Psychology. Vol. 16 (2):
259-275.

Stoop, I. (2004). Surveying Nonrespondents. Field Methods. Vol. 16, pp. 23-54.

Teitler, J.O., Reichman, R.E., & Sprachman, S. (2003). Costs and Benefits of Improving
Response Rates for a Hard to Reach Population. Public Opinion Quarterly. Vol. 26, pp. 126-138.

Vandenberg, R. J., & Lance, C.E. (2000). A review and synthesis of the measurement
invariance literature: suggestions, practices, and recommendations for organizational research.
Organizational Research Methods. Vol. 3: 4-70.

Van de Vijver, F. J. R. (1998). Towards a theory of bias and equivalence. ZUMA Nachrich-
ten Spezial, Cross-cultural survey equivalence, 3, 41-65.

Van de Vijver, F., & Leung, K. (1997). Methods and Data Analysis for Cross-Cultural
Research. Thousand Oaks: Sage.

Vehovar, V. (2006). Weighting in ESS Round 1. University of Ljubljana, Faculty of Social

Scienceshttp://vasja.ris.org.

Voogt, R. (2004). I Am Not Interested: Nonresponse Bias, Response Bias and Stimulus Effects in
Election Research. Amsterdam: University of Amsterdam (PhD Dissertation).

Wagner, U. & Zick, A. (1995). The relation of formal education to ethnic prejudice: its
reliability, validity and explanation. European Journal of Social Psychology. Vol. 25 (1), pp.
41-56.

29
Appendix. Subsequent steps in the test for scalar invariance

Model Df RMSEA CFI

Baseline: no equality contstraints 290.98 40 0.060 0.96
M0: invariant 's and 's 3735.65 154 0.120 0.45
M1: 4GB free 3296.50 153 0.110 0.51 439.15
GR
M2: 1 free 2930.38 152 0.100 0.57 366.12
FI
M3: 2 free 2600.14 151 0.096 0.62 330.24
PT
M4: 4 free 2369.41 150 0.092 0.66 230.73
LU
M5: 2 free 2240.14 149 0.089 0.68 129.27
DK
M6: 3 free 2083.50 148 0.086 0.70 156.64
CZ
M7: 1 free 1933.27 147 0.083 0.72 150.23
NO
M8: 4 free 1805.91 146 0.081 0.74 127.36
M9: 4PL free 1687.76 145 0.078 0.76 118.15
M10: 2CH free 1587.27 144 0.076 0.78 100.49
AT
M11: 2 free 1447.77 143 0.072 0.80 139.5
CH
M12: 4 free 1340.27 142 0.069 0.81 107.5
DE
M13: 2 free 1231.69 141 0.066 0.83 108.58
GR
M14: 4 free 1149.14 140 0.064 0.84 82.55
AT
M15: 4 free 1056.10 139 0.061 0.86 93.04
FI
M16: 4 free 976.98 138 0.059 0.87 79.12
M17: 1IT free 911.02 137 0.057 0.88 65.96
NO
M18: 2 free 852.57 136 0.055 0.89 58.45
HU
M19: 2 free 799.12 135 0.053 0.90 53.45
HU
M20: 3 free 760.60 134 0.052 0.90 38.52
IE
M21: 4 free 732.68 133 0.051 0.91 27.92
ES
M22: 4 free 715.86 132 0.050 0.91 16.82
SI
M23: 2 free 698.38 131 0.050 0.91 17.48
NL
M24: 4 free 670.44 130 0.049 0.92 27.94
M25: 2FI free 650.40 129 0.048 0.92 20.04
FI
M26: 4 free 632.97 128 0.047 0.92 17.43
GR
M27: 3 free 610.40 127 0.047 0.93 22.57
IE
M28: 3 free 588.12 126 0.046 0.93 22.28
PT
M29: 2 free 569.56 125 0.045 0.93 18.56
GR
M30: 3 free 552.79 124 0.044 0.93 16.77
PT
M31: 4 free 538.63 123 0.044 0.94 14.16

30