Gaming the (School) System

Every spring, parents face the trial of choosing the school that their child will attend in the fall—the K-12 institution that they hope will provide a nurturing learning environment and pave the way for success in college and the working world.

Many public school systems allow families to take their pick of schools in the broad community, rather than assigning students to attend a school in their immediate neighborhood. The goal is to match children to schools that best meet their academic and social needs.

Because some schools are in higher demand than others, school districts have devised ways to distribute seats among students—school choice, or student assignment mechanisms. As in any method of allocating scarce resources, there are winners and losers: Some children are assigned to their preferred school, others are relegated to a school ranked lower on their choice list and some are rejected by all their desired schools.

The widespread adoption of student assignment mechanisms in the United States and other countries has raised concerns about their efficiency and fairness. Economists who study these mechanisms observe that they’re imperfect instruments; some designs are prone to manipulation and bias. To improve their chances of getting their children into an acceptable school, many parents hide their true school preferences—subverting the intent of school choice. And unsophisticated players—those with little education and low income, for example—may be at a disadvantage in the school choice game.

Much economic research over the past decade on what’s known as the school choice problem has relied on theory or laboratory experiments to try to understand the strategies of households and the outcomes of different, commonly used student assignment mechanisms. But these approaches can go only so far in revealing the flaws of current school choice mechanisms and suggesting improvements.

Recent research by Maia Güell, an economics professor at the University of Edinburgh, and Caterina Calsamiglia, an associate professor at the Autonomous University of Barcelona, breaks new ground by using real-world data to analyze the issue of school choice.

In “The Illusion of School Choice: Empirical Evidence from Barcelona” (Minneapolis Fed Working Paper 712), Güell—who in July finished a one-year residency at the Minneapolis Fed—and Calsamiglia exploit a change in that city’s school administrative districts to gain fresh insight into student assignment mechanisms and their ramifications.

The results of their analysis upend the presumption that most parents act on their preferences in picking schools. In the widely used school choice mechanism they study, school assignment is largely determined by school district rules that induce parents to avoid the perceived risk of not enrolling their children in a school that is even minimally acceptable to them. To improve their odds of being allocated to an acceptable school, families systematically apply for schools in their neighborhood because the system’s rules give them priority in the local school. They make a “choice” as if they had no choice.

Just as college applicants include one school they’re confident will accept them, along with applying to less likely but more desirable colleges, parents of elementary school children go for a “safe” option. “The fact that you’ve implemented a choice system doesn’t mean that people are going to actually choose,” Güell said in an interview.

This implies that in communities with such a system, the benefits of offering school choice are limited, because not all families exercise genuine choice.

The economists also discover a new, subtle form of inequity in the student assignment method used in Barcelona and many other cities: Not only does the system harm less-educated families unversed in the rules of the school choice game; it also benefits some better-educated—and likely richer—parents because they can take greater risks, thereby gaining increased access to the best public schools.

Take your choice

Traditionally, public schools have assigned children to neighborhood schools close to where they live. But over the past 30 years, many school districts in the United States and other developed countries have adopted school choice systems that expand access beyond the neighborhood to other schools in the community or even outside it. (In 1987, Minnesota became the first U.S. state to authorize interdistrict school choice.)

By achieving a better fit between pupil and school in learning goals and teaching methods, for example, school choice is thought to contribute to academic success. Proponents say it also raises educational standards by fostering competition among schools. Educators consider school choice particularly important for low-income households in areas with subpar schools. “There are inequalities in our society,” Güell said, “so we want the kid who was born in a poor neighborhood to be able to go to a school outside that neighborhood.”

Capacity in each school is limited, requiring school districts to devise a means of fairly allocating seats in oversubscribed schools. In the typical student assignment mechanism, families submit a list of schools ranked according to preference. Then the school system applies a set of rules to those picks to determine how available seats are allotted to students.

The most common school choice mechanism in the United States is the “Boston mechanism,” named for the city that developed it after a 1974 court ruling enforced desegregation in Boston’s public schools. Variants of this system have been used in Seattle, Denver, Minneapolis and other communities across the country.

Under the Boston mechanism, students who list a school as their first choice are assigned to that school, with priority given to students who meet certain criteria, such as having a sibling in the school or living nearby. Priority points are awarded based on these criteria, and ties in points are broken through a random lottery. If no seats are left in that school, rejected students are considered for the school they ranked second, but only after children who ranked it as their first choice are assigned. As in the first round, students with priority go to the front of the line. The same procedure plays out in subsequent rounds until all students are placed in a school.

Other matching methods used by school districts include the “student deferred acceptance” mechanism, which makes tentative school assignments and reconsiders them at each step in the selection process based on students’ priority, and the “top trading cycles mechanism,” which allows students with priority for a school to trade places with other students with equivalent priority.

Flawed mechanisms

These and other proposed assignment mechanisms have generated lively debate among researchers and educators and have fostered a new area of research in what economists refer to as mechanism design theory. As explained by the Nobel committee that awarded its 2007 prize to pioneers in the field, mechanism design is “the art of producing institutions that align individual incentives with overall social goals.” Put otherwise, it’s about structuring the rules of a game to achieve the best outcome. A branch of game theory, mechanism design has been applied to real-life problems such as allocating broadcast spectrum, filling intern slots at hospitals and allotting kidneys to waiting transplant recipients.

School choice presents a similar challenge: identifying flaws in current assignment methods and finding ways to make them fairer, more efficient and less confusing for students and their parents.

There is no foolproof method; all systems currently used in schools have shortcomings, according to the tenets of mechanism design. But the Boston mechanism has elicited the severest criticism by researchers.

Maia Güell

In 2003, economists Atila Abdulkadiroğlu and Tayfun Sönmez published a seminal paper that revealed inherent flaws in the mechanism. Their theoretical analysis shows that it encourages families to lie about their true school preferences and apply to schools that are less popular though acceptable, out of fear of being assigned to a truly bad school (in their view) if they’re rebuffed by their first-ranked school. This is inefficient, because students forgo attending their preferred schools. And it puts families who state their true preferences—heedless of priority points—at a disadvantage.

After an article on the research appeared in the Boston Globe, the Boston school system reevaluated its student assignment mechanism, ultimately replacing it with the student deferred acceptance method. Subsequent economic research has confirmed these faults in the Boston mechanism and found it inferior to other methods.

Much of this research involves lab experiments that simulate various school mechanisms. For example, a 2006 paper co-authored by Sönmez found that only about 20 percent of subjects in a controlled experiment stated their true school preferences under the Boston mechanism. A 2010 experimental study by Calsamiglia and other researchers showed that imposing a limit on the number of schools that participants can list makes them more likely to try to manipulate the system by misstating their preferences.

Calsamiglia and Güell advance this line of research into the empirical arena by using real-world data to analyze choices made by parents in Barcelona. Drawing upon detailed application, admission and enrollment data, the economists focus on the role of priority points in shaping school decisions under the Boston mechanism.

There goes the neighborhood

In communities that use the Boston mechanism, over three-quarters of households pick a school in their neighborhood as their number one choice. But do families pick nearby schools because they genuinely prefer them, or for some other reason, such as a perception that they stand a better chance of entering those schools? Or, for families that can move to the neighborhood of their preferred school, are those motives intertwined?

Teasing out the truth empirically is no easy task. Simply observing the rankings families submit to the school district doesn’t reveal the motivations of families navigating the Boston mechanism; the strategies behind those selections are opaque. But changes in the way neighborhoods are defined in the Barcelona public school system afforded an opportunity for Calsamiglia and Güell to lift the veil on the decision-making process.

The city of Barcelona has used the Boston mechanism to place children in public schools since the mid-1990s. In the spring of 2007, school system administrators redrew the city map, abolishing old neighborhoods and establishing smaller ones based on the distance to schools from residents’ homes.

The researchers were aware of the changes; Calsamiglia lives in Barcelona and Güell was an associate professor at Pompeu Fabra University in Barcelona before leaving to join the faculty at Edinburgh. Both immediately realized the implications for investigating school choice. As a result of the change in neighborhood definition, the set of schools for each family that were “safer”—that is, they conferred more priority points than other schools, increasing the odds of acceptance—also changed.

So “what was safe yesterday is not necessarily safe today, and what was not safe yesterday now becomes safe,” Güell said. Because the change occurred right before the school application deadline, it also separated residential and school decisions; parents could not move to the neighborhood of their preferred school.

By observing demand for different sets of schools before and after neighborhoods were redrawn, the economists hoped to determine which impulse—genuine preference for a certain school or a desire for safety—is strongest under the rules of the Boston mechanism. “The key insight for us,” Güell said, “is that if parents choose according to their preferences, the [change in neighborhood definition] should not change their preferences—they still pick their top-ranked school.”

However, if families seek to avoid risk, the changeup in neighborhoods should lead them to alter their behavior, opting for safer schools in their new neighborhoods.

To prove the matter, Calsamiglia and Güell needed school application and assignment data for Barcelona children who entered elementary school between 2005 and 2010. Home addresses of these children were also required, to chart the effect of shifting neighborhood boundaries on family decisions. Working with the school district administration to allay any privacy concerns, the researchers obtained a rich data set chronicling school choices made on behalf of over 77,000 children in the Barcelona area.

They were ready to investigate how parents, constrained by the rules of the Boston mechanism, play the school choice game.

A rush to “safety”

The change in neighborhoods reshuffled the deck for Barcelona families, forcing them to rethink their school choices. Over 80 percent of households saw a change in the group of schools for which they had priority. For families living in the center of an old neighborhood, a new smaller neighborhood meant that many previously safer schools were no longer so. And some families living near the edge of an old neighborhood had new relatively safe schools to consider, because the new neighborhood included schools that previously lay outside it.

Calsamiglia and Güell map pupils’ addresses to tell which families applied to a school in their neighborhood—thus qualifying for priority points—both before and after 2007. (Because children are generally assigned to a school only once, the analysis necessarily compares the choices of different households over time.) The researchers examine the top choice of families—the school ranked first on their application form.^* If preferences are the main driver of school choice, demand for various schools in the system should not change with new neighborhood boundaries; in 2007 and in later years, parents should still apply to their favorite schools, regardless of location.

In fact, the neighborhood reshuffle prompted a marked shift between 2006 and 2007 in the proportion of families applying to different types of schools. Demand decreased for former neighborhood schools that now lay outside the new neighborhood and increased for schools once situated outside the neighborhood that now fell within the new neighborhood. From 2006 to 2007, the share of households that ranked first former nonneighborhood schools that were redefined as neighborhood schools rose from 9 percent to 17 percent. Also, for families living in the center of the old neighborhood, demand shifted from former neighborhood schools to schools that remained in the new neighborhood.

These findings point to a rush to safe choice schools in 2007; families shunned schools that no longer gave them priority, choosing instead schools that gave neighborhood residents a better chance of admission. This suggests that student assignment under the Boston mechanism with priority points is to a large degree determined by these points, not by parents’ predilection for certain schools. “To sum up, we find evidence that families’ preferences play a limited role in school choice because a change in the definition of neighborhood makes families change their choices,” Calsamiglia and Güell write.

Being relatively certain of getting into an acceptable school, though not their most preferred, is so attractive to families that the allocation of school seats in Barcelona is quite similar to the distribution that would result if children were simply assigned to schools in their neighborhoods. Previous theoretical studies or lab experiments overlooked the strong influence of priorities on parents’ choices. “The way priority points shape all these mechanisms was largely ignored, but they matter a lot,” Güell said.

For educators as well as economists, the implication is inescapable: If school choice is illusory under the Boston mechanism, so for the most part are the benefits believed to stem from permitting families to apply to the schools of their choice.

Not so naïve, after all

In an extension of their analysis, Calsamiglia and Güell shed light on another important question related to school choice—why some parents take big risks under the Boston mechanism by boldly stating their preferences for oversubscribed schools outside their neighborhood.

For each family in Barcelona, the chance of being assigned to a public school depends on the submitted choices, the priority points conferred by those schools and total demand for each school in the system. By calculating the odds of children being assigned to their first-ranked school—and if they are rejected, to their second-ranked school—the economists identify those applicants who could be considered “naïve”; that is, they apply to schools (usually outside their neighborhood) to which they have no chance of being assigned. About 23 percent of applicant families behave this way.

But further analysis shows that some “naïve” families are not so naïve, after all. Calsamiglia and Güell also obtained information on enrollment—the schools that children actually attended over the study period—from the Barcelona school district. The authors use these data to analyze the outcomes of risky behavior under the Boston mechanism.

It turns out that high risk takers comprise two types of households: families that seem truly unable to grasp the consequences of their choices and those that take a calculated risk because they have an alternative, an outside option in the parlance of game theory.

Like most school systems, Barcelona’s maintains a waiting list for oversubscribed schools, giving children who are rejected by those schools a second chance. However, even with waiting lists, families with fairly low priority for popular schools court disappointment: Almost one-third of such families are shut out of their top-ranked school. And, of that group, the naïve—families with less than the requisite points for their first-ranked school—fare the worst. Compared with households that incur less risk by applying to neighborhood schools, a smaller share of naïve families gain entry to any of their ranked schools, and a larger share fail to enter any selected school.

But about 14 percent of “naïve” families who miss out on their preferred school enroll their children in a school outside the Barcelona school system, either a public school in another city or a private school. These are households with an outside option.

Previous research had established the naïvete of some applicants under the Boston mechanism. But the role of the outside option in school decisions had not been explored. Calsamiglia and Güell’s analysis demonstrates that a significant share of outwardly naïve families are in fact sophisticated players who shrewdly weigh the odds.

Because parents have the option of enrolling their child elsewhere—most often in a private school—they can take the risk of applying to a preferred school outside their neighborhood. If this strategy works, the child is assigned to his or her first-ranked school from the waiting list (the researchers assume, based on the mathematical odds, that at least as many families win entry to their top choices as are rejected). If the risky bid fails, the family plays its get-out-of-jail-free card.

“Many of these so-called naïve parents know exactly what they’re doing,” Güell said. “If it doesn’t work, they go to a private school.” Thus, under the Boston mechanism, parents willing to roll the dice have a better chance of enrolling their children in the best public schools than families without an outside option.

Who are these savvy high rollers? Census data on the socioeconomic characteristics of Barcelona school applicants reveal that among families that exhibit naïve behavior, those that take the outside option are more highly educated than those that do not. These families are likely to have higher incomes, too. “Our empirical evidence suggests a new and important source of inequality that the [Boston mechanism] induces,” write Calsamiglia and Güell.

Not only are less-educated families more likely to choose unwisely and end up in lower-quality schools; those with higher education benefit the most from openly stating their preferences.

Priorities reconsidered

By divining the motivations of Barcelona families from the school system data, the economists show the extent to which a yearning for safety dominates the strategy of parents under the Boston mechanism. The pursuit of priority points trumps school preference, virtually eliminating school choice for all but a few. “The risk involved in stating preferences is not worth taking, leading most of the applicants to apply for one of the neighborhood schools,” the authors write.

Some families stay true to their school, but only better-educated applicants playing the outside option exercise informed choice; children in less-educated households that rank a nonneighborhood school number one are likely to wind up in the least-desirable schools.

Calsamiglia and Güell’s work has important implications for the design of student assignment mechanisms. Given the power of priorities in influencing school choice under the Boston mechanism, their findings suggest that children would benefit from other methods of allocating school seats that are fair to all families and encourage them to sincerely state their preferences.

In a follow-up paper, Güell teams with Calsamiglia and Chao Fu, an economist at the University of Wisconsin, to assess the potential gains of switching from the Boston mechanism to either the deferred acceptance or the top trading cycles mechanism, a method first proposed by Abdulkadiroğlu and Sönmez.

Once again tapping the Barcelona school system data, the researchers show that adopting top trading cycles—a method that allows families to pick their favorite schools without fear of losing priority for lower-ranked schools—increases the share of families whose children are assigned to schools outside their neighborhood.

However, neither study is likely to be the last word in the debate about the efficacy and fairness of school choice mechanisms. Some researchers, including Abdulkadiroğlu, have come to the defense of the Boston mechanism in recent years, comparing it favorably with other mechanisms. That may be one reason why many U.S. school districts have stuck with this method to govern the annual ritual of assigning students to their places.

Endnotes

References

Abdulkadiroğlu, A., and T. Sönmez. 2003. School choice: A mechanism design approach. American Economic Review 93 (3), 729-47.

Calsamiglia, C., C. Fu and M. Güell. 2014. Structural estimation of a model of school choices: The Boston mechanism versus its alternatives. Mimeo, University of Wisconsin.

Calsamiglia, C., and M. Güell. 2014. The illusion of school choice: Empirical evidence from Barcelona. Working Paper 712, Federal Reserve Bank of Minneapolis.

Calsamiglia, C., G. Haeringer and F. Klijn. 2010. Constrained school choice: An experimental study. American Economic Review 100 (4), 1860-74.

Chen, Y., and T. Sönmez. 2006. School choice: An experimental study. Journal of Economic Theory 127 (1), 202-31.