Research at the policy frontier in Latin America: Health, Education, Infrastructure and Housing and Climate Change
 Original Research
 Open Access
 Published:
Measuring the efficiency of public schools in Uruguay: main drivers and policy implications
Latin American Economic Review volume 24, Article number: 5 (2015)
Abstract
The aim of this research is to explore the existence of inefficient behaviors in public high schools in Uruguay and identify its potential drivers. To do so, we perform a twostage model using PISA 2009 and 2012 databases. In the first stage, we use Data Envelopment Analysis (DEA) to estimate efficiency scores, which are then regressed on school and student contextual variables. This second stage is carried out using four alternative models: a conventional censored regression and three different regression models based on the use of bootstrapping recently proposed in the literature. Our results show that educational efficiency in Uruguayan high schools significantly dropped in nine percentage points between 2009 and 2012. In terms of educational policy recommendations, in order to reduce the inefficiencies in the evaluated public schools in Uruguay, the focus should be put on reducing graderetention levels and promoting teaching–learning techniques that enhance student’s mathematics study skills and assessing students continuously through test and homework throughout the academic year. In this vein, our findings also show positive effects on public schools’ efficiency of providing the responsibility in the distribution of the school budget to school principals.
Introduction
There are basically two reasons why governments in developed countries have taken a strong interest in the determinants of educational quality over the last 50 years. First, improving academic outcomes have been proven to have a positive impact on economic growth (Barro 2001; Barro and Lee 2012; Hanushek and Kimko 2000; Hanushek and Woessmann 2012). Second, public expenditure on education is one of the largest public budget items, and the public sector is the main provider of education in most countries. Governments are not concerned solely with improving academic results, however, they mean to do so with the current educational resources, that is, through efficiency gains. The main reason is that public expenditure on education has grown over recent years in many countries, without leading to better academic results.
Particularly, the Uruguayan government has increased the country’s investment in education considerably over the last decade. Public expenditure on education accounted for 3.5 % of Uruguay’s GDP in 2000, whereas 10 years later it had risen to 4.5 %.^{Footnote 1} But this significant budgetary effort has not been accompanied by adequate reforms and public policies leading to better educational achievement in public schools. Conversely, the Uruguayan education system has entered into stagnation and recession in recent years, particularly at the public secondary education level, which has recorded high repetition and dropout rates as well as a steady decline in academic performance. For example, the repetition rate from 1st to 4th grades in public schools has increased between 2003 and 2012 from 21.3 to 27 % while the attainment rate was reduced from 72.7 to 67.4 % in the same period.^{Footnote 2} In addition, as evidenced by the latest results published in the PISA 2012 (Programme for International Student Assessment) Report from the OECD (Organisation for Economic Cooperation and Development), results in public schools remain steady across the first three waves in which Uruguay has participated, showing a downward trend in the last cycle (416, 420, 419 and 399 average points in 2003, 2006, 2009 and 2012, respectively).
As a consequence of these poor results, the Uruguayan public educational system problems are a recurring concern, not only for educational policymakers and the government, but also for teachers and families involved in the education process. In many cases, the discussion primarily still focuses on increasing public resources expended on education; however, there is no concluding empirical evidence in the economics of education literature to show that a higher level of resources leads per se to better results (Hanushek 2003).
These findings reveal that the solution to Uruguay’s educational problem is not simply to pour additional resources into the system; instead it is necessary to review and change some existing practices and educational policies that are not effective. In this sense, the main concern of educational policy makers in Uruguay should be to improve the quality of teaching and academic outputs with the currently available resources. To do this, it is clearly necessary to explore and address the main sources of educational inefficiencies.
Using the databases of different international programs,^{Footnote 3} many researchers have performed specific analyses of the main sources of inefficient behavior in the educational production process using student and school contextual variables (Wilson 2005; Afonso and St. Aubyn 2006; De Jorge and Santín, 2010; Cordero et al. 2011; Perelman and Santin 2011; CrespoCebada et al. 2014).^{Footnote 4}
Semiparametric twostage models were popularized by Ray (1991) and McCarty and Yaisawarng (1993) and are among the bestknown models for explaining the sources of inefficiency.^{Footnote 5} The first stage of this approach prescribes the use of a Data Envelopment Analysis (DEA) model to estimate a production frontier, which defines both the efficient and inefficient units. In the second stage, a regression technique is applied to explain the identified inefficient behaviors taking into account contextual variables. Twostage models differ primarily in the regression model specified in the second stage to explain efficiency scores. The most commonly applied methodology is the censored regression model (the socalled Tobit regression), followed by ordinary least squares (OLS) and truncated regression. Recently, Simar and Wilson (2007, 2011) proposed a new estimation methodology for the second stage based on the use of bootstrapping to overcome some drawbacks of these conventional estimation models. We apply the Simar and Wilson (2007) twostage approach as our baseline model in this research, but, as the discussion about which is the best model to be run in the secondstage regression is ongoing, we also run other secondstage specifications proposed in the literature in order to check the robustness of our conclusions.
Finally, it is noteworthy that even though there are several international educational efficiency studies for the OECD countries, research in the Latin American context is scant. To the best of our knowledge, there are no studies using this efficiency approach for the Uruguayan case. In Uruguay, interest has traditionally focused on education system coverage rates, the system’s redistributive effect and its impact on poverty and growth rather than the quality of the services provided and the academic outputs (Llambí and Perera 2008; Llambí et al. 2009; Fernández 2009).
Therefore, the main aim of this paper is to explore the sources of inefficiencies in Uruguayan secondary schools in order to provide new valuable and complementary evidence for the current national debate about which educational practices and policies could contribute to improving school academic results with the current resources. For this purpose, we apply a semiparametric twostage DEA approach to PISA 2009 and 2012 data in order to compare the results between the two periods. The paper is organized as follows. Section 2 presents the main methodological concepts. Section 3 briefly describes the Uruguayan education system, the PISA program and the variables included in the model. Section 4 reports the estimation results. Finally, Sect. 5 discusses the conclusions of this research and their implications for educational policy makers.
Methodology
The educational production function
The educational production function framework refers to the relationship between inputs and outputs for a given production technology. The theoretical approach used in this paper for linking resources to educational outcomes at school level is based on the wellknown educational production function proposed by Levin (1974), Hanushek (1979) and Hanushek et al. (2013):
where subindex i refers to school, and A _{ i } represents the educational output vector for school i. This output is normally measured through the students’ average scores in standardized tests. On the other hand, educational inputs are divided into B _{ i }, which denotes average student family and socioeconomic background, and S _{ i }, which are school educational resources.
The educational production function is frequently estimated considering the possible existence of inefficient behaviors in schools. Differences in efficiency may be due to multiple factors, such as poor teacher motivation, teaching and class organization issues, teacher quality or school management. Although all these factors are not direct inputs, they may affect student performance significantly. In this case, we estimate a production frontier where fully efficient schools would belong to the educational production frontier. These relatively efficient units achieve the maximum observed result given their resource allocation. However, inefficient units do not belong to the estimated frontier, and their inefficiency level is measured by the radial distance between each school and the constructed frontier. The production frontier to be estimated at school level would be
where 0 < u _{ i } ≤ 1 denotes the efficiency level of school i. Values of u _{ i } = 1 imply that the analyzed schools are fully efficient, meaning that given the initial input endowment and the existing technology, these schools are maximizing their outputs and managing correctly the available school inputs. Values u _{ i } ≤ 1 would indicate that the school is inefficient, and therefore the efficiency rate, θ _{ i } = 1/u _{ i } indicates the amount by which the actual output should be multiplied to reach the frontier in which case the school would be fully efficient.
In short, three types of variables are involved in the production process: educational outputs (A _{ i }), educational inputs (B _{ i }, S _{ i } ) and the estimated efficiency level (u _{ i }) for each school. Ray (1991) and McCarty and Yaisawarng (1993) were the first to propose applying a semiparametric twostage model to estimate efficiency scores and identify the main drivers. The first stage of this approach is to apply a DEA model which measures technical efficiency, whereas a regression analysis conducted in the second stage seeks out the main explanatory factors of efficiency. A more detailed description of the twostage methodology follows.
First stage: measuring efficiency through a DEABCC model
The measurement of efficiency is associated with Farrel’s concept of technical efficiency (Farrell 1957). Farrell defines the production frontier as the maximum level of output that a decisionmaking unit (DMU) can achieve given its inputs and the technology (output orientation). In practice, the true production frontier and the technology is not known and should be estimated from the relative best practices observed in the sample.
There are basically two main groups of techniques for estimating the production frontier: parametric, or econometric approaches (see Battese and Coelli 1988, 1992, 1995 for a review), and nonparametric methods based on mathematical optimization models. Although the use of parametric approaches has increased in education in the last decades,^{Footnote 6} nonparametric methods have been the most extensively applied methods for measuring educational efficiency.
Since the pioneering work by Charnes et al. (1978), 1981) and Banker et al. (1984),^{Footnote 7} the DEA model has been widely used to measure efficiency in many areas of public expenditure. The main reason for its widespread application is its flexibility, and the fact that it accounts for multiple outputs and inputs, unknown production technology and missing price information, which makes it to well suited to the peculiarities of the public sector. The technique applies a linear optimization program to obtain a production frontier that includes all the efficient units and their possible linear combinations. As a result, the estimated efficiency score for each DMU is a relative measure calculated using all the production units that are compared. The formulation of the outputoriented DEA program under variable returns to scale (DEABBC model) for each analyzed unit is
where for the ith DMU, θ _{ i } ≥ 1 is the efficiency score, y _{ i } is the output vector (q × 1) and x _{ i } is the input vector (p × 1), and thus X and Y are the respective input (p × n) and output (q × n) matrices. The (n × 1) vector λ contains the virtual weights of each unit determined by the problem solution. When θ _{ i } = 1 the analyzed unit belongs to the frontier (is fully efficient), whereas θ _{ i } > 1 indicates that the ith unit is inefficient, θ _{ i } being the radial distance between the ith unit and the frontier. In other words, θ _{ i } indicates the equiproportional expansion over outputs needed to reach the frontier. Therefore, the higher the score value θ _{ i } is, the greater the inefficiency level is.
In order to briefly illustrate a DEA model let assume the following simple twooutput singleinput setting where it is assumed that eight DMUs, A, B, C, D, E, F, G and H have an equal singleinput x to produce outputs y _{1} and y _{2}. The data and the efficiency index measured by DEA are showed in Table 1.
DEA runs eight linear programming problems (Eq. 3), one for each of the eight DMUs contained in our example in order to construct a piecewise linear frontier with best performers which envelops the other inefficient DMUs. Figure 1 shows this production frontier where A, B, C, D and E are efficient DMUs θ _{ A } = θ _{ B } = θ _{ C } = θ _{ D } = θ _{ E } = 1 because they lie on the boundary of the production frontier; whereas, being interior points, F, G and H are inefficient units θ _{ F } > 1; θ _{ G } > 1; θ _{ H } > 1.
DEA measures inefficiency as the radial distance from the inefficient unit to the frontier. For example, the performance of DMU F is measured projecting this unit upwards to point F′, a linear combination of DMUs D and E, with output 1 and output 2 equal to 2.4546 and 6.1365, respectively. DEA calculates the efficiency of DMU F as θ _{ F } = OF′/OF = 1.2273. This result means that DMU F could increase all its outputs proportionally multiplying its actual outputs level by 1.2273 with its input vector fixed.
Second stage: explaining educational efficiency scores
The estimated efficiency scores \(\hat{\theta }_{i}\) are regressed on a vector Z = (z _{1}, z _{2}, …, z _{ k }) of school and student contextual variables, which are not inputs but are related to the learning process:
The most used estimation method in this second stage is the censored regression model (Tobit), followed by ordinary least squares (OLS),^{Footnote 8} from which the main explanatory factors of the efficiency scores can be drawn^{Footnote 9}:
Xue and Harker (1999) were the first to argue that these conventional regression models applied in the second stage yield biased results because the efficiency scores estimated in the first stage (\(\hat{\theta }_{i}\)) are serially correlated. Accordingly, there has been a lively debate in recent years about which would be the most accurate model to apply in this second stage in order to provide consistent estimates. According to Simar and Wilson (2007), the efficiency rates estimated by the DEA model in the first stage are correlated by construction (as they are relative measures), and therefore estimates from conventional regression methods (Eq. 5) would be biased. Additionally, the possible correlation of the contextual variables Z _{ i } with the error term ɛ _{ i } in Eq. (5) is another source of bias.
Simar and Wilson (2007) state that bootstrapping can overcome these drawbacks. In their paper, the authors propose two algorithms^{Footnote 10} that incorporate the bootstrap procedure in a truncated regression model. They run a Monte Carlo experiment to examine and compare the performance of these two algorithms, and they prove that both bootstrap algorithms outperform conventional regression methods (Tobit and truncated regressions without bootstrapping), yielding valid inference methods. For small samples (problems with fewer than 400 units and up to three outputs and three inputs), Algorithm #1 fits results better than Algorithm #2, which is more efficient as of samples that exceed 800 units.^{Footnote 11} Since the samples analyzed in our research are made up of around a hundred schools, we apply the simple Algorithm #1, which is described below.^{Footnote 12}
Later, Hoff (2007), McDonald (2009), Banker and Natarajan (2008) and Ramalho et al. (2010) took up the discussion about the use of OLS, Tobit and fractional regression models in the second stage. Unlike Hoff (2007), who concluded that both (Tobit and OLS) models yield consistent estimations, McDonald (2009) showed that only the Tobit produces consistent results. Meanwhile, Banker and Natarajan (2008) provided a statistical model which yields consistent secondstage OLS estimations. Simar and Wilson (2011) again took part in the ongoing debate and concluded that only the truncated regression and, under very particular and unusual assumptions, the OLS model provides consistent estimates. Further, they proved that in both cases only bootstrap methods were capable of statistical inference.
From the above, we conclude that the research community does not yet totally agree about which is(are) the most consistent regression model(s) because this conclusion depends on previous assumptions about the data generation process. For this reason, we have chosen to estimate four alternative regression models in the second stage and compare the results. First, we specify the conventional Tobit (censored regression model), as it is the most commonly used in the literature. Then, for the sake of robustness, we estimate three regression models applying the bootstrap procedure: Algorithm #1 proposed by Simar and Wilson (2007) based on a truncated regression; and a Tobit regression and an OLS model with bootstrapping.
Data and variables
Brief description of the Uruguayan education system
The Uruguayan national education system is composed of four levels: 3 years of preprimary education (3–5 years old), 6 years of primary education (6–11 years old), 6 years of secondary education (12–17 years old), and college education at the end of secondary education. Secondary education is divided into 3 years of lower secondary education (Ciclo Básico Cómun) and 3 years of upper secondary education (Bachillerato). Compulsory education covers 14 years from the last 2 years of preprimary education (4 and 5 years old), through primary school, to the end of secondary education.^{Footnote 13}
In terms of public and private education production, the public sector takes absolute primacy over the private sector. In 2011, 84.5 % of high school students attended public schools (Education Observatory, National Administration of Public Education). This highlights how important the performance of public institutions is for national academic results, and therefore the need to benchmark schools and to assess both the management and the teaching practices implemented by these schools.
Uruguay has historically occupied a leading position in Latin America in terms of educational achievement, according to the main standard indicators and international studies. However, the Uruguayan education system (particularly the secondary and tertiary levels) is currently undergoing a phase of stagnation and recession. The major budgetary effort made by the government in the first decade of the twentyfirst century has not been accompanied by effective reforms and policies to improve educational outcomes.
The results of PISA 2009 and 2012 corroborate that Uruguay is still in an advantageous position within the region,^{Footnote 14} but also confirm that results have not improved compared to previous waves. In addition, test scores in the three analyzed areas (mathematics, reading and science) are more highly dispersed than in other countries, which mirror the high social segmentation of the education system. Comparing student’s performance by the schools socioeconomic context in PISA 2012, it is noteworthy that while almost 89 % of students who attended to schools in “very unfavorable circumstances” do not reach the minimum “competence threshold” defined by the OECD in mathematics,^{Footnote 15} this figure drops to 13 %^{Footnote 16} for students who attend to schools in “very favorable circumstances”.^{Footnote 17} By contrast, analyzing the percentage of topscoring students (performance levels four to six) defined by PISA analysts, we find that this proportion rises to almost 30 % of students in “very favorable circumstances”, whereas students from “very unfavorable circumstances” account for less than 1 %. This heterogeneity may be the consequence of differences not only in the initial resources endowment but also of efficiency. It is essential to explore the sources of such differences in order to improve academic outputs in more inefficient schools and to reduce inequalities in the education system.
PISA databases and model specification
PISA 2009 and 2012 are the fourth and fifth edition of an initiative that the OECD started up in the late 1990s to assess 15yearold students. The assessment focuses on measuring the extent to which students are able to apply their knowledge and skills to fulfill future reallife challenges rather than evaluating how well they have mastered a specific school curriculum. The evaluation addresses three knowledge areas: reading, mathematical and scientific literacy, and each wave tests in depth a major domain. In 2000 and in 2009 the major domain was reading, in 2003 it was mathematics, in 2006 science and finally, in 2012, it is again mathematics. In addition to academic achievement data, the PISA database contains a vast amount of information about students, their households and the schools they attend. Uruguay took part in PISA 2009 (2012) assessing 5927 (5315) students from (232) 180 public and private schools.
To perform the DEA model one of the main requirements is that the evaluated decisionmaking units should be as homogeneous as possible (Dyson et al. 2001). To estimate the production frontier and the efficiency indexes the technique assumes that all units operate under the same production technology and therefore under similar context and circumstances. In order to analyze homogenous schools, the original PISA databases were refined. Firstly, we assume that technologies in public and private sector are different due to different legal, organizational and curricular contexts and therefore, management drivers also differ in the two schools type. Two frontiers should be estimated, one for each sector. Unfortunately, the sample size of private schools is small to carry out a specific analysis of this sector so, we only analyze public schools. Secondly, we eliminate schools which only offer basic secondary education (1st, 2nd and 3rd year of high school) or only offer upper secondary education (4th, 5th and 6th year of high school). The cutoff age between the two cycles in Uruguay is just 15 years old and, since PISA evaluates students of this age, those students attending schools where only basic secondary education is offered are inevitably repeaters and, on the contrary, students attending schools where only upper secondary education is offered are all nonrepeaters. As a result, in schools where only basic secondary education is imparted, 100 % of the assessed students in PISA are repeaters in at least one previous course and, in those schools where only upper secondary education is imparted, 100 % of the assessed students are on the right course. Therefore, these institutions are not comparable when estimating the production frontier.
In sum, this analysis is carried out for 169 mixed public schools (98 from PISA 2009 and 71 from PISA 2012) which provide both cycles of secondary education. For comparative and robustness purposes, we perform the same analysis for PISA 2009 and PISA 2012 waves separately. Additionally, we run the model for both databases in a pool^{Footnote 18} including contextual variables simultaneously available in the two waves in order to check whether or not technical efficiency has changed significantly over the two periods.
Outputs, inputs and contextual variables
Outputs
It is difficult to empirically quantify the education received by an individual, especially when the focus is on analyzing its quality beyond the years of education. However, there is a consensus in the literature about considering the results from a standardized test as educational outputs, as they are difficult to forge and, above all, they are taken into account by parents and politicians when making decisions on education. In this research, we selected two variables as outputs of the educational process: the average results in reading (Read_mean) and mathematics (Maths_mean).^{Footnote 19}
Inputs
Regarding educational inputs, three variables were selected taking into account the educational production function in Eq. (1). They represent the classical inputs in education economics required to carry out the learning process: students (raw material), teachers (human capital) and infrastructure (physical capital).^{Footnote 20} A previous requirement for a variable to be considered as an input in an efficiency analysis is that it has to be positively correlated with all outputs. This monotonicity assumption (Coelli et al. 2005) implies that if we give more input to a DMU then we will expect to obtain equal or more quantity of outputs than in the previous situation or, in other words, additional units of an input will not decrease output. The following inputs were included in the first stage of the DEA:

Parental education (PARED): is an index that reflects the higher parental education expressed by the number of years of schooling according to the International Standard Classification of Education (ISCED1997, OECD).^{Footnote 21} It therefore represents the quality of the ‘raw material’ to be transformed through the learning process.

School educational resources (SCHRES): is an index of the quality of the school resources constructed from the school’s principal responses. It is therefore associated with the physical and human capital. The index was computed from the responses by principals to several questions related to the scarcity or lack of ten educational resources^{Footnote 22} including teachers, educational material and infrastructures. The school receives one point for each item for which the principal’s answer is that the school is not deficient ‘at all’. The maximum (minimum) score for each school is ten (zero) points, which indicates an excellent (dreadful) educational input.^{Footnote 23}

Proportion of fully certified teachers (PROPCERT): this index reflects the quality of teachers, and therefore the school’s human capital. The index is constructed by dividing the total number of certified teachers (with a teaching degree)^{Footnote 24} by the total number of teachers. This variable is especially relevant in the case of Uruguay since not all teachers have received the teaching training required to qualify as teachers.
As mentioned above, it is necessary to check the monotonicity assumption in order to ensure a correct DEA model specification. Table 2 presents the bivariate correlations of the selected outputs and inputs where all correlations are positive.
Contextual variables
The distinction between an educational input (first stage) and an explanatory variable of inefficiency or contextual variable (second stage) can be confusing. In this research we have considered that a variable is an explanatory factor of efficiency, and not an educational input, when it is not strictly essential to produce education but it can affect academic results through the efficiency term. These variables fulfill one or some of these conditions:

1.
The variable reflects some key aspect of school management and organization and/or the teaching–learning processes enacted in the classrooms.

2.
The variable is dichotomous, categorical or does not have a continuous measurement scale.

3.
The monotonicity assumption does not hold in practice, i.e., the selected variable does not show a positive correlation with academic outcomes.

4.
The variable is an indicator based on opinions with a high degree of subjectivity and difficult to contrast.
Building upon this criteria, we select fifteen contextual variables^{Footnote 25} (Z vector in Eqs. 4 and 5) associated with students and schools. Most of contextual variables appear in PISA 2009 and 2012; however, there are some variables that are only available in one wave. To be more precise we employ 12 (13) variables with PISA 2012 (2009) to run the second stage regression analysis. Finally, we only use the ten contextual variables that were collected in both waves.
Contextual variables included only in PISA 2009

TEST: a dummy variable that takes the value one when students are assessed by teachers through tests, quizzes or exams more often than once a month.

HOMEWORK: a dummy variable which refers to the assessment tools as well as the frequency with which they are applied. In this case, the variable takes value one when the students are assessed by means of homework every month. Both Tests and Homework are expected to have a positive effect on school efficiency.

EXTREADING: the percentage of students in the school who spend between one and 2 h per day reading for pleasure after school. It is understood that reading contributes to the student learning process, as it helps to improve spelling, reading comprehension and understanding skills. It is expected therefore to have a positive effect on school efficiency.
Contextual variables included only in PISA 2012

STUCHECK: percentage of students in the school that have answered yes to the statement ‘When I study mathematics, I make myself check to see if I remember the work I have already done’. This variable reflects the learning skills acquired along the student’s academic life.

STUIMPORT: percentage of students in the school that have answered yes to the statement ‘When I study for a mathematics test, I try to work out what the most important parts to learn are’. As in the previous case, this variable also reflects the learning skills acquired along the student’s academic life.
Contextual variables included in both databases and in the pool

PERIOD: dummy variable that takes value one if the student belongs to PISA 2012. This variable is only included in the pool.

PCTCORRECT: percentage of students assessed in the school who are in the academic year that a 15year student should really be in. This variable reflects the graderetention policy, and is another focus of attention in current educational discussions because there is no consensus about its net effect on educational results.

TEACHVOC: dummy variable that takes value one if the institution is a vocational technical school.

RURAL: dummy variable that takes value one if the institution is located in a town with less than 3000 inhabitants.

CITY: dummy variable that takes value one if the institution is located in a town with more than 100,000 inhabitants.

TEACHSTU: the number of teachers per hundred students. Some research includes class size as an educational input in the first stage, but we have decided to use it as an explanatory variable of efficiency since there is still no conclusive evidence about the real effect of this variable on student results.^{Footnote 26} Furthermore, this variable does not show a positive correlation with the analyzed outputs.
Finally, we incorporate a number of variables associated with school autonomy and management in terms of budget allocation, curriculum development, disciplinary policies and student assessment practices. There are no expected a priori positive or negative relationships between these variables and school efficiency, since empirical evidence emerging from international comparisons does not provide conclusions that are applicable to all education systems (OECD 2013b).

Curr_author: a dummy variable which takes the value one when the national authorities have a considerable responsibility for determining the content of the courses.

Disc_author: a dummy variable which takes the value one when the national authorities have a considerable responsibility for establishing student disciplinary policies.

Budget_ppal: a dummy variable which takes the value one when the school principal has a considerable responsibility for distributing the school budget.

Budget_author: a dummy variable which takes the value one when the national authorities have a considerable responsibility for distributing the school budget.

Asses_author: a dummy variable which takes the value one when the national authorities have a considerable responsibility for establishing student assessment policies.
Table 3 shows the main descriptive statistics of all selected variables: outputs, inputs and contextual variables.
Results
First stage analysis
Figure 2 illustrates the distribution of efficiency scores, θ _{ i } in Eq. (1), estimated by the outputoriented DEABCC model for the three data sets. Results show that between 2009 and 2012 the percentage of fully efficient schools in Uruguay dropped from 27 to 21 %. The average estimated efficiency score also decreases over this period from 1.087 in PISA 2009 to 1.116 in PISA 2012. This means that, on average, educational results in public schools could be increased in 2012 by 11.6 % given the available resources. Moreover, more than 16 % of the schools in 2012 could improve their results by over 20 % to reach the frontier while this percentage represented 13 % in 2009. Finally half of schools could improve outcomes by over 10 % with their current inputs in 2012 what represent an important decrease in results taking into account that barely one out of three schools belonged to this inefficient group in 2009.
Table 4 shows, as an example of the potential benefits of measuring inefficiency, the most inefficient schools with estimated efficiency scores greater than 1.20 for PISA 2012, i.e., where outputs could be increased by more than 20 % if they were fully efficient. All these schools perform far below the Uruguayan means in PISA 2012 (409 in mathematics and 411 in reading); whereas, if they were fully efficient, most of them would be above these national averages. In fact, some schools would even significantly exceed these averages in both subjects (e.g., schools 2, 4, 6 and 10).
An alternative way to evaluate the potential gains due to inefficiencies reduction is by comparing the actual distribution of students into PISA proficiency levels defined by the OECD with the potential distribution that would be observed once inefficient schools reached the frontier. Table 5 shows the percentage of students under (over) proficiency level 1 (level 4) and level 2 (level 5) in mathematics, reading and in at least one of the two evaluated disciplines in the two periods. Results show great potential gains derived from improving efficiency.
If all evaluated public schools were efficient in PISA 2012, the percentage of students below proficiency level 2 (the minimum ‘competence threshold’ defined by OECD) in mathematics (reading) could be reduced from 67.2 to 48.5 % in mathematics and from 58.8 to 42.9 % in reading. Moreover, the actual percentage of students who is below proficiency level 2 in at least one of the two evaluated areas would decline from 73.5 to 57.4 %. By contrast, analyzing the percentage of topscoring students (performance levels four to six) defined by PISA analysts, we find that this proportion could be doubled from 12.6 to 24.5 % and from 13.6 to 26.9 % in mathematics and reading tests, respectively. Indeed, these figures would be close to those actually observed in some OECD countries (e.g., United States 24.6 %, Sweden 24.4 % or Italy 26.7 % in mathematics). It is also important to note here the decline of PISA 2012 results with respect PISA 2009. Table 5 clearly shows how the actual number of top performers in at least one area was almost 11 percentage points higher in 2009 than in 2012. As a consequence, the potential percentage of students that could become top performers decreased too from 42.9 % in PISA 2009 to 35 % in PISA 2012. Results are closely related when we turn our attention to the comparison of students under the minimum competence threshold in mathematics or reading. In this case, we observe that the actual percentage of students without the minimum level of competences rose from 63.9 % in PISA 2009 to 73.5 % in PISA 2012 while the potential to lift students out of poor results decreased between the 2 years.
Secondstage analysis
In a second stage, the estimated efficiency scores are regressed over the contextual variables using four model specifications: the truncated regression with bootstrap proposed by Simar and Wilson (2007), the conventional Tobit, the Tobit regression with bootstrap and, finally, the OLS model with bootstrap. Results are shown for crosssections of PISA 2009, PISA 2012 and the pool in Tables 6, 7 and 8, respectively.
From the comparative analysis of the four specified models we can conclude that there are no major discrepancies between the results. The sign, magnitude and significance of almost all variables are similar in all models and databases, implying that any educational policy recommendations derived from them would be basically the same regardless the secondstage regression model finally chosen adding robustness to these findings. Taking into account this general conclusion, we will consider the specification proposed by Simar and Wilson (2007) as the baseline for discussing the results.
First, there is a set of variables that do not affect efficiency scores in any estimation. First, school location does not seem to affect the efficiency (RURAL and CITY). On average, schools in rural areas or small villages have worse educational outcomes than those located in bigger cities. The fact that the town size does not affect significantly the efficiency implies that the higher results are due to a greater allocation of educational resources and not to a better use of them. Likewise, the teacher–student ratio (TEACHSTU) does not affect either school’s efficiency.
Second, hardly any of the variables associated with school autonomy are significant (except for Budget_ppal). Decentralizing the responsibility of establishing the disciplinary policies (Disc_author) and assessment practices (Asses_author) or determining the content of the courses (Curr_author) does not seem to affect school efficiency. This is an interesting finding, since the decentralization issue is part of most current education discussions. International evidence shows that decentralization is successful in countries where there is also a school accountability practice properly regulated and with standardized criteria (Hanushek et al. 2013; OECD 2013b). This is not the case of Uruguay, where there is great heterogeneity in accountabilities and where, in many cases, there is not even a systematic way of presenting them.
Therefore, the results of this research could be associated with this international evidence, which points out that decentralization would only have positive effects on improving academic results if it is carried out accompanied by an appropriate accountability system. Another possible interpretation of this result lies in the fact that the autonomy indexes were computed from the principals’ responses and their perceived autonomy and therefore might not be reflecting the true degree of autonomy they actually have. In Uruguay, public high schools generally have low levels of autonomy; however, the variables included in this analysis show certain degree of variance (Table 3). This fact could suggest some distortion between reality and principals’ perceptions regarding their responsibility and autonomy.
By contrast, the fact that the school’s principal has a considerable responsibility for distributing the school budget (Budget_ppal) has a strong significant positive effect on efficiency in PISA 2012 and in pool estimations. Therefore, this result would suggest that to give the responsibility of allocating the school budget to the school’s principal would be an appropriate policy, at least in the case of secondary schools in Uruguay.
Third, there is a group of variables associated with students and teaching practices that are systematically significant and show the expected sign. Firstly, the percentage of students that are in the right year (PCTCORRECT) appears to be a positive and significant driver of efficiency in both databases separately and in the pool. This result calls into question the adequacy of current Uruguayan graderetention policies at all levels of the education system. Uruguay has one of the highest repetition rates in the region, which contrasts with international test results which show this country to be one of the region’s top performers. Therefore, it would perhaps be better to attempt to identify younger (primary education) students who are at risk of repeating and provide them with additional support early on in order to prevent grad retention. Secondly, the dummy variable that indicates whether the school is a secondary high school or a technical school (TECHVOC) is statistically significant in PISA 2009 and the pool pointing out that technical schools are more inefficient. Uruguayan high schools have on average better average academic results than technical schools. This result seems to point out that secondary high schools perform better due to a better management and not only because they have higher initial input endowments.
Thirdly, other interesting variables only appear in one PISA. On one hand, according to PISA 2009 estimations, student assessment methods and their frequency appear to positively influence efficiency. Indeed, schools where teachers assess their students continuously by setting conventional tests or exams (TEST) more often than once a month or by means of the homework made monthly (HOMEWORK) perform better than schools that do not make use of this tool or do so with a frequency other than once a month. At early ages homework needs to be set daily to establish students’ study habits, but 15yearolds should be set homework at less regular intervals to complement regular individual study. So, monthly homework to assess learning seems to positively affect students’ results.
On the other hand, regarding PISA 2012 both variables associated with student’s study skills in mathematics (STUcheck and STUimport) have a positive impact on efficiency. These variables reflect the students skills acquired over their academic life and thus, this ability could be associated with classroom teaching techniques adopted by teachers. Thus, it would be desirable to promote these learning techniques both in the classroom and at home. This means, not only to work at school but also to foster families’ commitment to support students work at home. Although this research is focused in secondary education, such practices should be encouraged from the beginning of the student’s academic life in previous cycles, when students are assimilating the learning techniques to be used throughout their academic life and when it is most effective to impact on their noncognitive skills (Heckman and Kautz 2013).
Finally, it is worth to highlight that the coefficient associated to the time period variable (PERIOD) points out to a significant drop in efficiency results in 2012 with respect to 2009 even after controlling for other contextual covariates also related with efficiency. From Table 3 it is straightforward to conclude that over this period mean outputs significantly decreased while mean inputs clearly increased (PARED and SCHRES) or remained almost constant (PROPCERT). This decline in performance cannot be easily explained but should alert the Uruguayan educational system how to invert this result to gain efficiency.
Discussion and conclusions
Modern countries agree about the need and importance of having a more and better educated population in order to ensure economic growth based on the high productivity of a skilled labor force. The high percentage of public spending on education is a reflection of this conviction. During the last decade the Uruguayan government has made a huge effort to increase educational resources; however, academic results have not improved. On the contrary, public education system (especially public secondary education) is in a deep crisis and the current educational national debate mainly focuses on the need to put more resources into the system instead of exploring how to make better use of available inputs, i.e., how to achieve a more efficient education system. This situation raises two open questions. Are Uruguayan public secondary schools efficient? Which policies and practices should be promoted in order to increase school efficiency? As far as we know, however, this issue has yet to be analyzed for the Uruguayan education system. This is the main aim of this research.
Our findings corroborate the presence of inefficient behaviors in public secondary schools. According to PISA 2012 results we conclude that with the current inputs schools could have increased their academic results on average by 11.6 % if adequate educational policies and practices had been designed by national authorities and implemented by schools. Furthermore, if schools were fully efficient, the percentage of students below proficiency level 2 (the minimum ‘competence threshold’ defined by OECD) could be reduced from 67 to 49 % in mathematics and from 59 to 43 % in reading. By contrast, the percentage of topscoring students (performance levels four to six), could be doubled from 12 to 24 % and from 14 to 27 % in mathematics and reading tests, respectively.
In addition, the secondstage analysis yields interesting evidence for planning and implementing effective policies to improve the efficiency of the Uruguayan public secondary education. The first noteworthy conclusion is that just increasing educational resources (e.g., reducing class size through recruiting more teachers) does not appear to be an appropriate policy because it does not have a positive and significant effect on school efficiency. By contrast, the results suggest that the national discussion and action on increasing education system efficiency should focus on reviewing the current graderetention policies and the teaching techniques.
Second, this research evidences that inefficiency is higher where there is a higher percentage of repeating students. So, students at risk of repetition should be identified at an early age and provided with extra support with the aim of preventing future school failure. Third, promoting teaching and learning techniques to enhance students’ study skills evidences positive effects on results. In addition, student assessment methods and their frequency appear to positively influence efficiency. Indeed, schools where teachers assess their students continuously by setting monthly homework or through test or exams more than once a month perform better than schools that do not make use of this tool or do so with other frequency. So, continuous monthly assessments seems to positively affect students’ results. Fourth, the fact that the school principal had a considerable responsibility for distributing the school budget (Budget_ppal) has a strong significant positive effect on efficiency. Therefore, this result suggests that it is a good practice to deliver the responsibility of allocating the school budget to the school principal.
Finally, we find a significant decline in efficiency results in Uruguay between the two analyzed periods. In other words, educational outputs in the last years have decreased despite the effort that Uruguayan authorities made putting more public expenditure in the system. Therefore it seems necessary to reach a large commitment from all stakeholders involved in the educational process in order to effectively seek and remove the inefficiencies.
In conclusion, this research offers a new perspective on how to tackle the current educational problem in public high schools in Uruguay from an efficiency viewpoint, providing some potential practices and policies that positively affect academic results. In this respect, this paper reports preliminary findings, and more research is, of course, still needed. For example, a qualitative and in depth analysis of the most efficient and inefficient schools could provide additional useful information about how to implement efficient practices and avoid the inefficient ones.
Notes
The GDP grew by 37 % in real terms over this period [Uruguayan Central Bank (BCU)].
Education Observatory, National Administration of Public Education (ANEP).
These programs include PISA, TIMSS (Trends in International Mathematics and Science Study), IALS (International Assessment of Literacy Survey) and PIRLS (Progress in International Reading Literacy Study).
See Simar and Wilson (2007) for a detailed review of twostage models.
The DEACCR model and DEABCC model, respectively, from now on.
Some authors actually estimate both models simultaneously to verify results robustness.
For a detailed review of estimation methods used in the second stage of semiparametric models, see Simar and Wilson (2007).
The authors propose a simple Algorithm #1 and a double Algorithm #2. The difference lies in the fact that Algorithm #2 incorporates an additional bootstrap in the first stage, which amends the estimates of the efficiency scores.
For a more detailed analysis of the results, see Simar and Wilson (2007, p. 45).
According to Simar and Wilson (2007) we define L = 2000 in this paper.
Art. 10 of the General Education Law N. 18,437 of December 12, 2008.
PISA 2009 showed that Uruguay was the Latin American country with the best results for mathematics and was second placed in science and reading (after Chile). In PISA 2012 Uruguay is placed in the third position in the three evaluated areas between all Latin American countries that participated in this wave.
PISA defines six competencies levels and states that basic skills are obtained at Level 2. In the case of mathematics Level 2 threshold is described as follows: “At Level 2 students can interpret and recognize situations in contexts that require no more than direct inference. They can extract relevant information from a single source and make use of a single representational model. Students at this level can employ basic algorithms, formulae, procedures, or conventions. They are capable of direct reasoning and making literal interpretations of the results”. For more details, see OECD (2013a).
National Administration of Public Education (ANEP), “Informe Ejecutivo Preliminar Uruguay en PISA 2012”. Available at http://www.anep.edu.uy/anep/index.php/presentaciones2012.
Schools are classified into five levels of socioeconomic context based on the quintile distribution of the average socioeconomic background of the students who attend to these schools (the average ESCS PISA index for each school). Levels are defined as “Very unfavorable” (the bottom quintile), “Unfavorable”, “Medium”, “Favorable” and “Very favorable” (the top quintile).
This analysis was suggested by a referee to explore extra sources of variation (especially temporal).
The result for science has been omitted since it provides little additional information to the reading and mathematical results. Besides, DEA becomes less discriminative as more dimensions are added to the problem (curse of dimensionality); therefore, we prioritize parsimony by choosing only two outputs.
We focus on the quality rather than just the quantity of these inputs.
In the case of Uruguay the equivalent scale used to compute the years of schooling is the following: ISCED 1 equals to 6 years; ISCED 2 equals to 9 years; ISCED Level 3A, 3B, 3C or 4 equals to 12 years; ISCED 5B equals to 15 years; and ISCED 5A or 6 equals to 17 years of schooling.
The item included are: ‘Qualified science teachers’, ‘Qualified mathematics teachers’, ‘Qualified reading teachers’, ‘Any other personal support’, ‘Science laboratory equipment’, ‘Instructional materials’, ‘Computers’, ‘Internet connectivity’, ‘Software’, ‘Library materials’.
This variable has been rescaled so the minimum value is one in order to avoid zero values in the empirical analysis.
Certified teachers in Uruguay are required to complete a 4year degree at the Instituto de Profesores Artigas (IPA), a higher education institution which provides specialized secondary teacher training.
Not all contextual variables appear in both PISA waves.
References
Afonso A, St Aubyn M (2006) Crosscountry efficiency of secondary education provision: a semiparametric analysis with nondiscretionary inputs. Econ Model 23(3):476–491
Banker RD, Natarajan R (2008) Evaluating contextual variables affecting productivity using data envelopment analysis. Oper Res 56(1):48–58
Banker RD, Charnes A, Cooper WW (1984) Some models for estimating technical and scale inefficiencies in data envelopment analysis. Manag Sci 30(9):1078–1092
Barro RJ (2001) Human capital and growth. Am Econ Rev 91(2):12–17
Barro RJ, Lee JW (2012) A new data set of educational attainment in the world, 1950–2010. J Dev Econ 104:184–198
Battese GE, Coelli TJ (1988) Prediction of firmlevel technical efficiencies with a generalized frontier production function and panel data. J Econom 38(3):387–399
Battese GE, Coelli TJ (1992) Frontier production functions, technical efficiency and panel data: with application to paddy farmers in India. J Prod Anal 3(1–2):153–169
Battese GE, Coelli TJ (1995) A model for technical inefficiency effects in a stochastic frontier production function for panel data. Empir Econ 20(2):325–332
Charnes A, Cooper WW, Rhodes E (1978) Measuring the efficiency of decision making units. Eur J Oper Res 2(6):429–444
Charnes A, Cooper WW, Rhodes E (1981) Evaluating program and managerial efficiency: an application of data envelopment analysis to program follow through. Manag Sci 27(6):668–697
Coelli T, Rao D, O’Donnell C, Battese G (2005) An introduction to efficiency and productivity analysis. Springer, New York
Cordero JM, CrespoCebada E, Pedraja F, Santín D (2011) Exploring educational efficiency divergences across Spanish regions in PISA 2006. Revista de economía aplicada 19(57):117–146
CrespoCebada E, PedrajaChaparro F, Santín D (2014) Does school ownership matter? An unbiased efficiency comparison for regions of Spain. J Prod Anal 41(1):153–172
De Jorge J, Santín D (2010) Determinantes de la eficiencia educativa en la Unión Europea. Hacienda Pública Esp 193:131–155
Dyson RG, Allen R, Camanho AS, Podinovski VV, Sarrico CS, Shale EA (2001) Pitfalls and protocols in DEA. Eur J Oper Res 132(2):245–259
Farrell MJ (1957) The measurement of productive efficiency. J R Stat Soc S A (Gen) 120(3):253–290
Fernández T (2009) La desafiliación en la educación media en Uruguay. Una aproximación con base en el panel de estudiantes evaluados por PISA 2003. Revista Iberoamericana sobre Calidad, Eficacia y Cambio en Educación (REICE) 7(4):32–49
Hanushek EA (1979) Conceptual and empirical issues in the estimation of educational production functions. J Hum Resour 14(3):351–388
Hanushek EA (2003) The failure of inputbased schooling policies. Econ J 113(485):64–98
Hanushek EA, Kimko DD (2000) Schooling, laborforce quality, and the growth of nations. Am Econ Rev 90(5):1184–1208
Hanushek EA, Woessmann L (2012) Do better schools lead to more growth? Cognitive skills, economic outcomes, and causation. J Econ Growth 17(4):267–321
Hanushek EA, Link S, Woessmann L (2013) Does school autonomy make sense everywhere? Panel estimates from PISA. J Dev Econ 104:212–232
Heckman JJ, Kautz T (2013) Fostering and measuring skills: interventions that improve character and cognition. National Bureau of Economic Research, No. w19656
Hoff A (2007) Second stage DEA: comparison of approaches for modelling the DEA score. Eur J Oper Res 181(1):425–435
Hoxby CM (2000) The effects of class size on student achievement: new evidence from population variation. Q J Econ 115(4):1239–1285
Levin HM (1974) Measuring efficiency in educational production. Publ Financ Q 2(1):3–24
Llambí C, Perera M (2008) La función de producción educativa: el posible sesgo en la estimación de efectos “institucionales” con los datos pisa. El caso de las escuelas de tiempo completo, Montevideo, Centro de Investigaciones Económicas (cinve)
Llambí C, Perera M, Messina P (2009) Desigualdad de oportunidades y el rol del sistema educativo en los logros de los jóvenes uruguayos. Centro de Investigaciones Económicas, Working paper, No. 4
Mancebón MJ, Muñiz MA (2003) Aspectos clave de la evaluación de la eficiencia productiva en la educación secundaria. Papeles de Economía Esp 95:162–187
McCarty TA, Yaisawarng S (1993) Technical efficiency in New Jersey school districts. In: Fried HO, Knox Lovell S. Smith CA (eds) The measurement of productive efficiency: techniques and applications. Oxford University Press, New York, pp 271–287
McDonald J (2009) Using least squares and tobit in second stage DEA efficiency analyses. Eur J Oper Res 197(2):792–798
OECD (2013a) PISA 2012 assessment and analytical framework: mathematics, reading, science, problem solving and financial literacy. PISA, OECD Publishing, Paris
OECD (2013b) PISA 2012 results: what makes schools successful? Resources, policies and practices, vol IV. PISA, OECD Publishing, Paris
Perelman S, Santin D (2011) Measuring educational efficiency at student level with parametric stochastic distance functions: an application to Spanish PISA results. Educ Econ 19(1):29–49
Ramalho EA, Ramalho JJ, Henriques PD (2010) Fractional regression models for second stage DEA efficiency analyses. J Product Anal 34(3):239–255
Ray SC (1991) Resourceuse efficiency in public schools: a study of Connecticut data. Manag Sci 37(12):1620–1628
Simar L, Wilson PW (2007) Estimation and inference in twostage, semiparametric models of production processes. J Econom 136(1):31–64
Simar L, Wilson PW (2011) Twostage DEA: caveat emptor. J Product Anal 36(2):205–218
Wilson PW (2005) Efficiency in education production among PISA countries with emphasis on transitioning economies. Mimeo, Austin
Worthington AC (2001) An empirical survey of frontier efficiency measurement techniques in education. Educ Econ 9(3):245–268
Xue M, Harker PT (1999) Overcoming the inherent dependency of DEA efficiency scores: a bootstrap approach. Mimeo
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Santín, D., Sicilia, G. Measuring the efficiency of public schools in Uruguay: main drivers and policy implications. Lat Am Econ Rev 24, 5 (2015). https://doi.org/10.1007/s4050301500195
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s4050301500195
Keywords
 Public secondary education
 Technical efficiency
 Performancebased reforms
 PISA
 DEA
JEL Classification
 I21
 C14
 H52