Educational Poverty as a Welfare Loss: Low Performance in the OECD According to PISA 2012

Show more

Received 21 March 2016; accepted 25 April 2016; published 28 April 2016

1. Introduction

The OECD’s Programme for International Student Assessment (PISA) provides the richest and most comprehensive database for the evaluation of the educational achievements of 15 year-old students in three different subjects: mathematics, reading, and science. The age of the students is very close to the end of compulsory education for most of the participating countries. Those results, therefore, are a good proxy of the basic knowledge ensured by the different countries to their citizens. Sixty-five countries and large economies participated in the 2012 wave of PISA. About half a million students participated in the study, representing 26 million 15 year-old students of those 65 countries and large economies [1] . The Programme establishes six levels of proficiency, parameterized in terms of the scores of the tests that students perform for each subject. Those levels can be used to determine some structural features of the different educational systems, particularly regarding equity (an aspect to which the OECD pays a lot of attention).

Level 2 is considered as the baseline level of proficiency so that those students who do not achieve this level are considered as low performers. ^{1}“The proportion of students who perform below this baseline proficiency level thus indicates the degree of difficulty countries face in providing their populations with a minimum level of competencies” [2] . There is evidence, particularly longitudinal studies developed in Australia, Canada, Denmark and Switzerland, showing that students who perform below Level 2 often face severe disadvantages in their transition into higher education and the labour force in subsequent years [3] .

Low performance is a problem that affects one fourth of the students of the OECD (and a larger proportion of those outside the OECD) with a sizeable variability between country members. It is therefore a widespread phenomenon and also a source of inequality between and within countries. Low performance has long-term consequences and affects the future of the individual and the society as a whole. As stated in the recent OECD report on this matter, “Students who perform poorly at age 15 face a high risk of dropping out of school altogether. When a large share of the population lacks basic skills, a country’s long-term economic growth is severely compromised” [4] .

The relevance of the problem makes it important measuring properly the extent of low performance, by considering not only its incidence (the share of low performers), but also its intensity (how far from the threshold students are), and the inequality among low performing students. The purpose of this paper is contributing to such a measurement by interpreting low performance as educational poverty and then applying the tools that are usual in the welfare analysis of inequality and poverty. From this perspective, measuring educational poverty regarding the competencies in mathematics, reading comprehension and science calls for a multidimensional measure that permits one identifying those students who can be deemed educationally poor, and also estimates how poor and unequal they are (the three poverty dimensions postulated by Sen [5] ). We shall derive here our poverty measure from a specific social evaluation function: that implicit in Atkinson’s inequality measure for a unitary degree of inequality aversion, which has been adopted by the United Nations in order to assess equity-adjusted human development. In this way we can interpret educational poverty as a welfare loss, as measured by that social evaluation function.

The work is organised as follows. Section 2 presents a methodology that permits one measuring educational poverty in terms of a welfare loss by applying the type of reasoning used in the analysis of income inequality and poverty (see for instance Chakravarty [6] ). Here the outcome variables are the test scores, rather than earnings or expenditures. The “poverty line” will thus be defined in terms of insufficient educational achievements, as given by the baseline level of proficiency. Interestingly enough the resulting poverty measure can be expressed as the product of the incidence and the equality adjusted intensity of educational poverty.

Section 3 applies this methodology to the measurement of educational poverty in the OECD countries. The data show that the variability of educational poverty among the OECD countries is much larger than that of average scores, in spite of the high (negative) correlation between both variables. They also show that OECD countries exhibit different mixes of incidence and intensity of educational poverty.

A few final comments in Section 4 close the work.

2. Measuring Educational Poverty

2.1. The Model

The measurement of educational poverty will be conducted in terms of a poverty index derived from an explicit welfare evaluation function. In that way the interpretation of educational poverty is that of a welfare loss due to the insufficient achievements of the students. ^{2}The relevant data for our evaluation problem refer to the OECD students’ test scores in mathematics, reading comprehension and science, using the 2012 wave of PISA [1] . For each given country those data will be arranged in the form of a matrix, Y, whose (i, j) entry tells the score obtained by student i in competence j. Each row of matrix Y, denoted by (), describes, therefore, the scores of student i in the three educational competencies evaluated. And each column, denoted by (j = 1, 2, 3), gives us the distribution of each competence across students.

A welfare evaluation function, relative to a society with n members (15-year old students of a given country), is in this context a mapping that associates to each positive n × 3 matrix, Y, a real number, W(Y),which provides an estimate of the welfare associated with that multidimensional distribution. We assume that our welfare evaluation function is defined for all population sizes and is homogeneous of degree 1. Homogeneity is a cardinal property that ensures a correct behaviour of the evaluation function in this context and incorporates a monotonicity feature [8] .

We shall define our poverty index as the relative welfare loss due to low performance, measured by the welfare evaluation function, W, with respect to a three-dimensional threshold z = (z_{m}, z_{r}, z_{s}), where sub-indices m, r and s refer to mathematics, reading and science, respectively. Those thresholds correspond to the minimum scores of level 2 in each competence (see below for details).

Given a n × 3 matrix Y of the students scores and a vector z = (z_{m}, z_{r}, z_{s}) of poverty thresholds, we define a student as educationally poor whenever. That is, a poor student is one whose individual welfare evaluation does not reach the minimum associated with the thresholds.

We now define our educational poverty index as the relative welfare loss of the poor, measured by the welfare evaluation function restricted to the set of the poor. Let p denote the number of poor students and let be the

unit vector of dimension m. is the minimum welfare that society would like to ensure for the poor students. Yet their actual welfare is given by, where is a matrix that describes the achievements of those poor students. The difference between those two values, , tells us how far away is this society from ensuring the minimum admissible educational welfare to all its members, in absolute terms. We shall identify our educational poverty measure, , with the ratio between that difference and the minimum welfare admissible for all students, (i.e. the relative welfare loss due to educational poverty). Formally:

(1)

in the understanding that if and only if there are no poor students.

The poverty index is defined, therefore, as the relative welfare loss due to the existence of students who do not reach the minimum admissible value of the reference variables. It is easy to check that this index moves into the interval (0, 1).

In order to give a precise expression to the generic Equation (1) we need to specify the welfare evaluation function W. We shall adopt here the one that has been used by United Nations in order to estimate the inequality adjusted human development index [9] . This function coincides with the implicit social welfare function that yields Atkinson’s inequality index for the unitary value of the inequality aversion [10] . It is the following:

(2)

where is the score of student i regarding competence j.

Note that is the geometric mean of the students’ scores regarding the jth competence, for j = m, r, s. Equation (2) can thus be rewritten as:

(2’)

Which says that the welfare measure corresponds to the geometric mean of Atkinson’s equally distributed equivalent values for.^{3}

Plugging this welfare evaluation function (2) into Equation (1) yields:

which in turn can be rewritten as follows:

(3)

where the first term, p/n, measures the incidence of poverty (the share of the poor in society) and the second term:

(4)

is the coefficient that measures the inequality-adjusted intensity of poverty.

Equation (3) is an intuitive and easy-to-handle formula, based on clear-cut assumptions, which corresponds to the poverty counterpart of the standard Atkinson’s welfare evaluation function for the unit value of the inequality aversion parameter. ^{4}Moreover, the two components of the index provide separate information on the extent and intensity of educational poverty (where the intensity measure is to be understood as adjusted by inequality). This is interesting because they permit uncover different patterns behind societies with similar values of educational poverty.

Note that when and when for some j = m, r, s. That is, the poverty measure coincides with the incidence of poverty only when there is some student with zero achievements.

2.2. From the Model to the Data

We now apply this methodology to the measurement of educational poverty using the results in the 2012 wave of the PISA report. Students’ performance is measured by a 0 - 1000 scale with mean around 500 and standard deviation 100. ^{5}The Programme establishes six levels of proficiency, parameterized in terms of the scores of the tests that students perform for each subject. It is understood that Level 2 is the baseline level of proficiency for an individual to be able to develop a reasonable integration in the labor market and, more generally, in society. Students who do not reach that level are considered to have insufficient knowledge (low performance). It is only natural to interpret insufficient knowledge as educational poverty and thus use the thresholds that define those minimum levels to set the corresponding poverty lines in mathematics, reading competence and science, respectively.

According to PISA 2012 the thresholds that define low performance in those competencies are: 420.1 test score points for mathematics (m), 407.5 points for reading competence (r), and 409.5 points for science (s). Therefore, our vector z of poverty lines is given by:. We consider that those three competencies are equally important.

The micro-data of the PISA report provide information about the test scores of individuals that conform the representative sample for each country. We take those individual micro-data as our starting point, focusing on the test scores on mathematics, reading comprehension, and science. Out of these data we are able to compute, for each student i in the sample of every OECD country, the corresponding individual poverty index. That is, the number:

A student is considered educationally poor whenever this number is strictly positive and non-poor otherwise. So our first step is computing this expression for each individual student in every country. We then select those students for which this expression is positive within each country, which gives us the set of poor students in that country. Once this set has been determined, we calculate all the elements required to compute the poverty index. That is, for each OECD country we compute:

. (5)

3. The Results

We present now the results that are obtained when analysing educational poverty in the OECD using the methodology described in Section 2.

The main results are reported in Table 1. The table contains both the value of the Educational Poverty Index (EPI) in absolute terms (i.e. the computation of Equation (5)) and the value of the index in relative terms (as percentages of the OECD mean). Using relative values is helpful because those numbers are easier to interpret since the EPI values are very small, as they correspond to the product of two numbers smaller than one. Figure 1 illustrates those values ordering the countries from best to worse.

Table 1 also provides information about the rank of the different countries regarding achievements (understood as the geometric mean of the test scores, ordered from more to less) and the poverty index (arranged in an increasing order). The comparison of those rankings shows that poverty analysis provides some information about the performance of the educational systems that is badly captured by the average scores. Even though the rank correlation is high, there are substantial differences in particular cases such as Belgium, Denmark, France, Luxembourg, New Zealand, Slovenia, Spain, Turkey, United Kingdom, and the USA.

A prominent feature of educational poverty, as clearly illustrated in Figure 1, is the large variability that exhibits among OECD countries: while the coefficient of variation of the test scores is very low, around 0.054, the coefficient of variation of educational poverty is ten times larger (0.537). The data exhibit a similar structure for the whole set of the countries participating in PISA, even though substantially amplified (the coefficient of variation of the test scores is 0.1 while that of educational poverty index jumps up to 0.9).

Educational poverty is highly correlated (negatively) with the average scores of the tests, with some 94% of common variance. There is a positive but relatively weak correlation between the Educational Poverty Index and the index of economic and socio-cultural status (ESCS). This is not surprising since it is already well established that the correlation between the ESCS index and average test scores is not very high (about one third of common variance).^{6}

Remark: It is worth noting that those data underestimate educational poverty in some countries, as all the information refers to the students who actually keep attending formal education. There are some countries in which the rate of 15-year olds who have abandoned the school is very high (in particular Mexico and Turkey), which implies that educational poverty would be substantially higher. For a discussion on how to combine data regarding access and achievement see Ferreira & Ginoux (2011), Ferreira, Ginoux & Aran (2011), Carvalho, Gamboa & Waltenberg (2012), Gamboa & Waltenberg (2012), and Tansel (2015) [15] - [19] .

Note that Equation (3) describes the Educational Poverty Index as the product of two terms. The first one, p/n, captures the incidence of educational poverty. The second one, ρ(Y, z), is a measure of the intensity of educational poverty adjusted by inequality. The correlation between both components is positive but moderate (a coefficient of 0.4), which indicates that OECD countries exhibit different mixes of both ingredients. The variability of those two components is also quite diverse. The coefficient of variation of the incidence is three times that of the inequality adjusted intensity (0.476 versus 0.153).

Table 2 provides the data regarding the values of the two components of the Educational Poverty Index, both in absolute and relative terms. Mexico, Chile, Turkey, Israel and Greece exhibit relative values of the incidence variable more than 40% higher than the OECD mean, whereas Korea, Japan, Finland, Poland and Ireland are at least 40% below the incidence average (40 is about one standard deviation of the relative incidence). Regarding equality-adjusted intensity, Israel, Sweden, France, Iceland and Belgium present values which are more than one standard deviation (15 points) above the OECD mean, whereas Estonia, Poland, Slovenia, Canada, Ireland and United States have values that are at least one standard deviation below the mean.

Table 1. OECD Educational Poverty Index (EPI), relative EPI (OECD mean = 100) and ranking of countries by EPI and mean test scores. PISA 2012.

Table 2. Incidence and inequality adjusted Intensity of educational poverty in the OECD (PISA 2012). Relative values take the OECD mean equal to 100.

Figure 1. Educational poverty in the OECD according to PISA 2012 (OECD mean = 100).

4. Final Comments

Educational poverty has been identified here the product of two different factors: the incidence of educational poverty, given by the share of students who do not reach level 2 of proficiency, and the inequality adjusted intensity, which measures how unequal they are and how far away are their scores from the threshold defining level 2.

The Educational Poverty Index is to be interpreted as a quantitative assessment of the welfare loss due to the failure of educational systems to provide a minimal knowledge to all citizens. This is so because the basic data (the PISA scores) are collected at the end of the period of compulsory education.

The data show that the OECD countries are much more diverse regarding educational poverty than with respect to the average scores. In all cases we find that most of the observed differences in the Educational Poverty Index are due to the differences in the incidence, whereas the impact of the equality adjusted intensity is smaller, with a positive but weak correlation between those variables (a common variance of about 0.17).

The high negative correlation between educational poverty and average scores tells us that reducing low performance appears as the most effective way of improving the overall educational outcomes. In other words, making the educational system more inclusive is the best strategy to get simultaneously higher equity and higher efficiency.

Finally, let us briefly comment on the possible causes of educational poverty, even if this question is outside the scope of this work. The analysis presented by the PISA team in OECD [4] identifies three main groups of factors that are associated with low performance, each one involving several variables (even though most of them are correlated). These groups of factors are:

1) Family environment: Low performance increases in socio-economic disadvantaged families, in students with immigrant background, in those who speak at home a language different from that in the school, those students living in rural areas, and those from single-parent families.

2) Individual characteristics: Attitudes and behaviours also influence the probability of low performance. The two main variables that seem to increase low performance are not having attended pre-primary education and having repeated a grade.

3) School-related factors: There is some evidence showing that more socio-economic diversity among students and less grouping by ability between classes tends to provide a better learning environment.

Identifying these elements helps designing measures to reduce educational poverty and increase the efficiency and equity of educational systems. Yet, as there are substantial differences in the profile of low performers among countries, as already pointed out, those measures have to be very much tailor-made. There are already some particular experiences illustrating that there are effective ways of reducing educational poverty in countries with very different characteristics [4] .

Disclaimer

The judgements contained in this paper express the views of the author and not those of the OECD or any of its state members.

NOTES

^{*}Thanks are due to José Antonio Robles for his help with the data and to Pablo Zoido and Álvaro Choi for their helpful comments. Useful comments from anonymous referees are also acknowledged. This work was realized when the author was visiting the OECD in Paris as a Thomas J. Alexander Fellow. Thanks are due to all facilities provided.

^{1}This convention is not universal, though. Some authors adopt level 3 as the baseline (e.g. Nonoyama-Tarumi & Willms [1] .

^{2}We follow here some of the ideas developed in Villar (2016) [7] .

^{3}This formula, which has the feature of penalising outcome dispersion both within and between dimensions, has a long tradition in welfare economics and has been characterized in a number of ways (see for instance Foster, López-Calva & Szekely [11] , Herrero, Martínez & Villar [12] , Seth [13] ).

^{4}This index can also be regarded as a derivation of Watts [14] poverty measure, under the assumption of equally important dimensions.

^{5}500 was the mean set in the 2000 edition of PISA. Actual mean values differ slightly from this one.

^{6}Note, however, that this correlation refers to the link between low performance and socio-economic conditions between countries. Things are different when we analyse low performance within countries by social groups.

References

[1] Nonoyama-Tarumi, Y. and Willms, D.J. (2010) The Relative and Absolute Risks of Disadvantaged Family Background and Low Levels of School Resources on Student Literacy. Economics of Education Review, 29, 214-224.

http://dx.doi.org/10.1016/j.econedurev.2009.07.007

[2] OCDE (2014) What Students Know and Can Do. Student Performance in Mathematics, Reading and Science. Volume I, Revised Edition, PISA, OECD Publishing, Paris.

[3] OECD (2012) Learning beyond Sixteen: Ten Years after PISA. PISA, OECD Publishing, Paris.

[4] OECD (2016) Low Performing Students: Why They Fall behind and How to Help Them Succeed. PISA, OECD Publishing, Paris.

http://dx.doi.org/10.1787/9789264250246-en

[5] Sen, A.K. (1976) Poverty: An Ordinal Approach to Measurement. Econometrica, 44, 219-231.

http://dx.doi.org/10.2307/1912718

[6] Chakravarty, S.R. (2009) Inequality, Polarization and Poverty. Springer, New York.

http://dx.doi.org/10.1007/978-0-387-79253-8

[7] Villar, A. (2016) Multidimensional Welfare-Poverty Indices.

[8] Blackorby, C. and Donalson, D. (1978) Measures of Relative Inequality and their Meaning in Terms of Social Welfare, Journal of Economic Theory, 18, 59-80.

http://dx.doi.org/10.1016/0022-0531(78)90042-X

[9] United Nations Development Program (2010) The Real Wealth of Nations: Pathways to Human Development. New York.

[10] Atkinson, A.B. (1970) On the Measurement of Inequality. Journal of Economic Theory, 2, 244-263.

http://dx.doi.org/10.1016/0022-0531(70)90039-6

[11] Foster, J.E., Lopez-Calva, L.F. and Szekely, M. (2005) Measuring the Distribution of Human Development: Methodology and an Application to Mexico. Journal of Human Development, 6, 5-25.

http://dx.doi.org/10.1080/1464988052000342220

[12] Herrero, C., Martínez, R. and Villar, A. (2010) Multidimensional Social Evaluation. An Application to the Measurement of Human Development. Review of Income and Wealth, 56, 483-497.

http://dx.doi.org/10.1111/j.1475-4991.2009.00375.x

[13] Seth, S. (2013) A Class of Distribution and Association Sensitive Multidimensional Welfare Indices. Journal of Economic Inequality, 11, 133-162.

http://dx.doi.org/10.1007/s10888-011-9210-3

[14] Watts, H. (1968) An Economic Definition of Poverty. In: Moynihan, D.P., Ed., On Understanding Poverty, Basic Books, New York, 316-329.

[15] Ferreira, F. and Gignoux, J. (2011a) The Measurement of Educational Inequality: Achievement and Opportunity. Ecineq Working Paper No. 240.

[16] Ferreira, F.H.G, Gignoux, J. and Aran, M. (2011) Measuring Inequality of Opportunity with Imperfect Data: The Case of Turkey. Journal of Economic Inequality, 9, 651-680.

http://dx.doi.org/10.1007/s10888-011-9169-0

[17] Carvalho, M., Gamboa, L.F. and Waltenberg, F.D. (2015) Equality of Educational Opportunity: Taking Both Achievement and Access into Account. Ecineq working Paper 2012-277.

[18] Gamboa, L.F. and Waltenberg, F.D. (2012) Inequality of Opportunity in Educational Achievement in Latin America: Evidence from PISA 2006-2009. Economics of Education Review, 31, 694-708.

http://dx.doi.org/10.1016/j.econedurev.2012.05.002

[19] Tansel, A. (2015) Inequality of Opportunities in Educational Achievements in Turkey over Time. IZA Working Paper DP 9005.