CARDUS

A Good Fit

How Matching Students and Schools by Religion Improves Academic Outcomes

December 13, 2021

Catherine Pakaluk

Nicholas Swanson

Education

Research Report

Education Outcomes Educational Pluralism

Executive Summary

Does fit between school and student matter for educational outcomes? More specifically, do religious schools provide any special advantage for students of the school’s religion? And if so, is this advantage the same for all affected students?

In this report we describe the first study that has assessed the value of a religious “good fit” in education. Using US data from the National Longitudinal Survey of Youth 1997 (NLSY97), the study estimated a religious match effect, which is the difference in standardized test scores for students paired to schools of their same religion, after controlling for other variables.

The findings of the study indicate statistically significant advantages for such pairings. Specifically:

  • Matched students in the NLSY97 outperformed unmatched peers in raw comparisons by 14 to 19 percentile points.
  • Matched students in Catholic schools outperformed unmatched students in Catholic schools by a sizable margin. (Previously observed “Catholic school effects” may have been confounded with “match effects.”)
  • Matched students in independent non-Catholic religious schools outperformed unmatched students in these schools by a sizable margin.
  • Controlling for other sources of variation, matched students in the NLSY97 outperformed their unmatched peers by 5 to 9 percentile points.

The magnitude of these findings invites further research. We discuss the implications for future research, public policy, and how we frame educational models for both.

This report introduces a completely novel perspective: Educational pluralism is of itself productive: matching religious students to religious schools that are a “good fit” for them leads to better outcomes and creates particular value for them, their own faith community, and the broader society. Accordingly, creating a public-policy environment that allows for and nurtures such pairing is in the public’s best interest and should be further explored.

Introduction

Is religious schooling any different from non-religious schooling? Religious schools are believed to contribute to students’ mental health, views of charity and morality, and spiritual development. 1 1 Cardus, “2018 US Cardus Education Survey: Spiritual Strength, Faithful Formation,” August 2019, https://www.cardus.ca/research/education/reports/2018-us-cardus-education-survey-spiritual-strength-faithful-formation/. But what about academic measures, such as board scores, graduation rates, and college matriculation? Do religious schools perform better, about the same, or worse than government and non-religious independent schools on academic measures? (For clarity, we refer to schools operated by the government as “government schools,” and independently operated, non-government schools as “independent schools,” whether taxpayer-funded or not.) On this question, the evidence is somewhat mixed. Some studies have found that religious schooling is correlated with advantages in academic outcomes, but after controlling for various background measures such as family structure the differences are not significant. Other studies have found significant advantages, but only for certain subgroups and certain outcomes. There is no line of research that finds religious schools produce disadvantages. Altogether, there is no overwhelming evidence in favour of or against religious schools purely on the academic measures.

For religious parents and proponents of educational pluralism, the mixed data on academic outcomes is unlikely to discourage placement in religious schools. Indeed, for many parents, the fact that their children will take religion classes at their religious independent school is enough to warrant the sacrifice that comes with paying tuition bills. For others, choosing a religious school may be appealing specifically because of the character and the non-cognitive benefits that often arise from schooling that is grounded in a faith community. But for those also seeking to know the academic impact, this report describes an exciting new development.

Here we summarize in accessible language the recent study “What Good Is a Good Fit? Religious Matching and Educational Outcomes.” 2 2 C.R. Pakaluk, “What Good Is a Good Fit? Religious Matching and Educational Outcomes,” Cosmos + Taxis 9, nos. 1 and 2 (January 2021): 3-30. https://cosmosandtaxis.files.wordpress.com/2021/02/pakaluk_ct_vol9_iss_1_2-2.pdf. The study asked, Do students matched to schools by religion perform better academically? Despite decades of research into religious schooling, especially into Catholic schools that dominate the United States’ independent-school landscape, the question about the effect of matching students to schools based on religion has never been considered—until now.

To understand this new angle, let us differentiate between two kinds of effects that a person or institution might have: one is a general effect, which extends to everyone who comes in contact with the person or institution. If Joe is a kind person, everyone who comes in contact with Joe will experience kindness. His kindness is a general effect that extends to everyone. The second kind of effect is a specific effect, which is experienced only by some of the people who interact with the person or institution. Let us suppose now that Joe is a fine foreign-car mechanic. If people who have domestic cars bring them to Joe, he may certainly extend his kindness, but he will not be able to distinguish himself as the fine mechanic that he is. Joe is a great mechanic when he is matched up with clients who own foreign cars. The specific excellence that Joe has is dependent on being matched with the right clients. Thus, we can understand the specific effect that Joe has on certain clients who own foreign cars as a match effect.

In the case of schools, a general effect would be a finding that a school has a similar positive (or deleterious) effect on all students who study there. A specific effect would be a finding that a school has a positive effect for particular students only, or that one aspect of its excellence is dependent on being matched with the right students. There are plenty of reasons to think that schools—like Joe in the example—have some strengths (or weaknesses) that extend to all students (general effects), and other strengths that are experienced by just some students (specific, or match effects). For example, schools that specialize in athletics may be better for students with a zeal for sports than for students with a devotion to the arts. Of course, some kinds of specific or match effects are subtle and hard to document. Parents of all types intuit this; as a result, they often use the language of a school being a “good fit” (or not) for their child. Surprisingly, there has been no research before now concerning whether “good fit” has measurable academic benefits or is simply a matter of personal satisfaction.

The paper that we summarize in this report does two important things: First, it makes the case for “good fit” as worth studying in itself, as a potential key to unlocking some of the unsolved questions in education research. Second, it looks specifically at whether religious matching—religious “good fit”—has any measurable effect on academic outcomes. The findings may shed new light on the role of religious schools in shaping academic outcomes and present new avenues for further research.

In previous research on religious schools, what we have called general effects have usually been labelled as school quality, understood as a fixed effect of a school or a class of schools that affects all students in a similar way. Previous research into Catholic schools, for example, has focused on whether Catholic schools yield better academic outcomes for students in comparison to government schools. 3 3 This study was conducted in the United States, where all Catholic schools are independent schools. These papers effectively ask the question, Are Catholic schools better than government schools (on some measure)? One might ask the same about schools of other Christian denominations or other faiths, or of homeschooling. But this is not the question that our report addresses. Instead, we ask whether Catholic schools are better for Catholic students (than are other pairings for Catholic students), and whether evangelical schools are better for evangelical students (than are other pairings for evangelical students), insofar as the pairings can be identified within schools.

A number of challenges arise in studying this question, chief of which is the limited number of religious “good fit” schools. For example, while there are about seven thousand K–12 Catholic schools in the US, there are less than three hundred Methodist schools. 4 4 National Center for Education Statistics, “Table 205.45 on Independent Education Enrollment and School Numbers,” Digest of Education Statistics, https://nces.ed.gov/programs/digest/d19/tables/dt19_205.45.asp?current=yes. This sort of numeric advantage for Catholic schools poses a difficulty in that large surveys that rely on cross-sections of students will “catch” many more Catholic students who are matched than will catch students of other faiths. Another challenge is the inability to randomly assign students to schools, a difficulty that plagues nearly all research on school effects. Studies of both general and match-specific school effects have to grapple with issues of correlation and causation by controlling for other sources of variation wherever possible. This study has pushed the limits of the data used here, but further study with a more expansive sample of students in religious and non-governmental schools could help eliminate some of the remaining uncertainty in pursuit of these questions.

Ultimately, the focus of these findings is on religious “good fit” or religious match effects rather than on the generic benefit of religious schooling compared to non-religious schooling. Since religious communities often confer a profound sense of belonging on their members, it is time to ask whether religious schools confer special benefits mediated through that channel of membership. Knowing the answer can help policy-makers and school-choice researchers come closer to understanding how schools as institutions work and may lead to expanded opportunities (vouchers, tax credits, etc.) for parents who value religious schools but cannot afford their cost. 5 5 As a matter of the theoretical question at hand, we take homeschooled students to be studying in a religious “good fit” environment. But because the data set used in this paper did not include any homeschooled students, we save this thought for future study.

Research Approach

Data

We use data from the 1997 National Longitudinal Survey of Youth (NLSY97) conducted by the United States Bureau of Labor Statistics. 6 6 US Bureau of Labor Statistics, “National Longitudinal Survey of Youth 1997,” https://www.nlsinfo.org/content/cohorts/nlsy97. A longitudinal study surveys a sample population at different moments in time across many years, sometimes even decades. The first installment of the NLSY (for which data is used in this study) comes from the 1997 round of the survey.

The NLSY97 contains data on 8,984 students. A modified sample of 8,817 was created by dropping those students for whom religion, or school type, was unknown. Of those 8,817 students, 51.1 percent were female and 48.9 percent were male. Students in the sample were 52.0 percent white, 25.9 percent black, 21.2 percent Hispanic (these minorities were oversampled), and 0.9 percent other. The geographic breakdown was 17.7 percent rural, 49.0 percent suburban, and 32.2 percent urban. As for the school breakdown of this sample, 92.2 percent attended government schools, 5.4 percent attended religious independent schools (breaking out to 3.5 percent Catholic schools and 1.9 percent other religious), and 0.8 percent attended non-religious independent schools. 7 7 Due to omitted categories, percentages do not sum to 100. The small percentage of students who attended religious schools poses an important limitation for this study: ideally, to test pairings of religious students and religious schools, we would want a large cross-section of students in religious schools, or a large cross-section of religious schools. This data set provides neither but was the best available data when the original study was undertaken.

To measure student academic outcomes, two test scores were considered: the PIAT Math standardized test score and the ASVAB composite math-reading standardized test score.

Methodology

To maximize (or “catch”) all the possible matches, we tallied students according to religious matching in two ways: strong match and weak match. We counted a strong match when a student reporting a particular religious affiliation was matched to a school of that exact reported affiliation—for example, a Catholic attending a Catholic school. We counted a weak match when a student of one reported affiliation was matched to a school of a similar reported affiliation, using data on religious branching from the Association of Religion Data Archives. 8 8 Association of Religion Data Archives, “Religion Family Trees,” http://www.thearda.com/Denoms/Families/trees.asp. Using this system, we organized the data using two match variables: alpha match, which comprises the strong-matched students (302, or 3.4 percent of the sample), and beta match, which comprises the strong-matched and the weak-matched students added together (430, or 4.9 percent of the sample).

Designing a variable for religious match is an imperfect task. The variable could be improved in a future study by gathering a better sample that allows for tallying only strong matches (e.g., Catholics at Catholic schools and Lutherans at Lutheran schools). There is also the question of the degree of practice that a family may have—do students who attend church only once a month benefit from a match to the same extent as students who attend church weekly? Another form of the match variable might focus on students who go to a school specifically attached to the church where they worship. Finally, future research should also be devoted to understanding the mechanisms through which “good fit” works, especially in terms of religion.

Findings

Matched Students Outperform Unmatched Peers in Raw Comparisons

Matched students outperformed unmatched peers in raw comparisons by about 14 to 19 percentile points. Raw effects are rendered in plain comparisons without controlling for other factors, such as family income, family structure, mother’s education, and location. 9 9 These factors are included in the multivariate regressions in the original paper, and results including them are covered in the section below titled “Controlling for Other Sources of Variation.” Figure 1 shows a significant match effect for both alpha (strong) and beta (strong + weak) match groups. In the PIAT Math, the alpha match has an advantage of nearly 15 percentile points, and the beta match has an advantage of over 14 percentile points. The ASVAB Math-Reading results suggest a similarly large match effect. Indeed, alpha-matched students’ advantage over their unmatched peers is over 19 percentile points, while beta-matched students have an advantage of over 18 percentile points.

Matched Students Outperform Unmatched Students in Catholic Schools

Because most religious schools are Catholic, and because the NLSY97 was not designed (sampled) to “catch” lots of students in religious schools, the majority of the matched students in our sample attend Catholic schools. Thus, we ask, Is the match advantage in figure 1 simply a quality advantage from attending Catholic schools in general? We address the answer in two parts.

First, in figure 2 we show the raw Catholic school effect for all students. The PIAT Math results reflect a 13-percentile-point advantage for Catholic school students over non-Catholic school students (i.e., government school, non-religious independent school, non-Catholic religious independent school). The ASVAB shows an almost 18 percentile-point (17.51) advantage for Catholic school students over their non-Catholic school peers.

Second, in figure 3 we compare alpha-matched and unmatched students within Catholic schools. If the match effect shown in figure 1 were simply a Catholic-school effect, then we would not expect to see a large difference between alpha-matched and unmatched students within Catholic schools. But in fact, conditional upon being in a Catholic school, there is a remaining 10-percentile-point advantage in the PIAT test and an 11-percentile-point advantage in the ASVAB test for alpha-matched students (Catholics at Catholic schools). Thus, figure 3 points to the possibility that the match effect shown in figure 1 is not a spurious effect of high-quality Catholic schools.

Matched Students Outperform Unmatched Students in Independent Schools

Because all religious schools in the US are independent, every matched student in our sample attends an independent school. Thus, we ask, Is the match advantage in figure 1 simply a quality advantage from attending independent schools in general? This question is analogous to the question about Catholic schools. We address this in three parts.

First, in figure 4 we show the raw independent-school effect for all students, by comparing scores for all government-school students to those from all independent-school students. The advantage is about 9 percentile points on the PIAT Math and 13.5 percentile points (13.45) on the ASVAB Math-Reading.

Second, in figure 5 we compare matched and unmatched students within independent schools. If the match effect from figure 1 were simply an independent-school effect, then we would not expect to see a large difference between matched and unmatched students in independent schools. But in fact, conditional upon being in an independent school, there is a roughly 15-percentile-point advantage in the PIAT test and a roughly 15- to 16-percentile-point advantage in the ASVAB test for alpha- and beta-matched students. These findings suggest that religious matching is doing some of the work. If the effect were just an independent-school effect, one would anticipate that unmatched independent-school students would score roughly the same as alpha- and beta-matched students. Here, that is not the case.

Third, we point out the difference between government-school students (light cream bars) and unmatched independent-school students (dark yellow bars). On the PIAT Math test, government-school students and unmatched independent-school students score roughly identical, around 36. On the ASVAB Math-Reading test, unmatched independent-school students have a modest four-point advantage over their government-school counterparts. Overall, unmatched independent-school students look more like government-school students than like their matched independent-school peers.

Controlling for Other Sources of Variation: Matched Outperform Unmatched Students

Thus far we have not accounted for the fact that matched families may have more household income, live in different neighbourhoods, have parents with more education, and have favourable family structures. Using multivariate regression analysis (ordinary least squares), we adjust for the contribution of these factors and present the modified results in figure 6.

While the effect sizes of matching are reduced as expected by controlling for other factors, the magnitude of the effect remains substantial: including only the alpha (strong) matches, we estimate an advantage of close to six points for the PIAT test, and nearly nine points for the ASVAB test. If we expand the sample to include both strong and weak matches, the regression estimate for the beta matches is about six points advantage for the PIAT Math and nearly nine points for the ASVAB.

Discussion and Implications

Summary of Findings

This study provides a novel approach to thinking about the value of religious schools, in which religious schools create particular value for members of their own faith community by virtue of belonging. In the first test of this hypothesis, presented in the findings here, we observe sizable advantages in one data set for students matched to schools based on their religion. Figure 1 showed that matched students scored between 14 and 19 percentile points higher than their unmatched peers on the PIAT Math and ASVAB Math-Reading tests. Figure 6 showed that, controlling for other sources of variation (such as family income, family structure, and level of mother’s education), matched students still scored better, with advantages of five to nine percentile points. The magnitude of these findings invites further research. Indeed, while using the NLSY97, we had to navigate a limited subsample of students who attended religious schools. Subsequent research with new data on religious students and schools will be critical to confirm and extend the findings in this report, as discussed below.

 

Two Educational Models

In addition to the novel approach outlined above, the research we describe is also the first to document the significance of a “good fit” in education. The results are broadly suggestive that current research in education data could benefit from new frameworks.

Consider two models for thinking about education. The first we might call a factory model (related to what we described as general effects). In a factory-model analogy, students are like raw materials that are shaped in general when the educational process occurs. The “mechanics” of the institution include structural factors such as the quality of the teachers, the size of classes, the types of classes, and the physical environment. The model assumes that students are acted on by the school environment, and results may vary depending on the mechanics of the institution and the type of raw materials (characteristics of the student).

A second approach might be called a garden model (related to what we described as specific effects). A garden combines the elements of soil, water, sunlight, and nutrients to provide the right environment for a specific plant to thrive. In this analogy, students are like specific plants that thrive in specific circumstances. Flourishing results not so much from the garden (or the school) acting on the plants but on the interaction of a specific garden environment with specific plants.

Both a factory model and a garden model are valuable in discussions about school policy, since schools unquestionably have some characteristics of each. But the lack of inquiry into educational match or “good fit” hinders policy goals that may depend on understanding of the latter model. These considerations provide hope that avenues for new discoveries have not been exhausted.

Implications for Public Policy

School-choice advocates tend to rely on constitutional arguments about freedom (such as religious liberty or parental rights), or on arguments about competition among schools improving efficiency and accountability. This study introduces a new argument—namely, that educational pluralism may have an objectively productive value if it allows for students to be matched to schools that are a “good fit” for them, and if these good-fit pairings lead to better academic outcomes. As this study documents, initial evidence indicates that matching may have positive effects for students who are paired with schools of their same or similar religion.

If a good fit between student and school by itself raises outcomes—independently of any other characteristics of the school or student, as indicated by the findings—then all of society would reap the benefits of such a match. Accordingly, creating a public-policy environment that allows for and nurtures such pairing is in the public’s best interest and should be further explored.

 

Implications for Future Research

This research suggests at least three areas for future work. First, the present study should be reproduced using an expanded data set that would allow us to differentiate the effects of school quality (a factory model) and the effects of fit (a garden model). To do this, we need a sample with many more religious schools and students.

Second, more specific analyses could occur with better data. For instance, instead of the binary strong- or weak-match tally we used in this report, we could examine degree of religiosity and strength of membership in a faith community on a spectrum, which might allow for further insights.

Third, the logic of this research might be fruitfully applied beyond education. Areas such as health care, welfare, job training, substance-abuse programs, and the broader class of public goods that may be characterized as social investments in human persons may also lend themselves to being understood in terms of good fit.

Conclusion

Fundamentally, the question underlying this analysis is whether goods that are intended to affect human beings affect them in the same way. We believe that while most would say that such goods have an objective component (the factory model) and a subjective component (the garden model), it is clear that for reasons of expediency research has been almost entirely focused on identifying the objective component. Our results indicate, however, that the subjective component of certain human goods (such as education and health care) may be significant. We look forward to furthering this research agenda.

References

Association of Religion Data Archives. “Religion Family Trees.” http://www.thearda.com/Denoms/Families/trees.asp.

Cardus. “2018 US Cardus Education Survey: Spiritual Strength, Faithful Formation.” August 2019. https://www.cardus.ca/research/education/reports/2018-us-cardus-education-survey-spiritual-strength-faithful-formation/.

National Center for Education Statistics. “Table 205.45 on Independent Education Enrollment and School Numbers.” Digest of Education Statistics. https://nces.ed.gov/programs/digest/d19/tables/dt19_205.45.asp?current=yes.

Pakaluk, C.R. “What Good Is a Good Fit? Religious Matching and Educational Outcomes.” Cosmos + Taxis 9, nos. 1 and 2 (January 2021): 3-30. https://cosmosandtaxis.files.wordpress.com/2021/02/pakaluk_ct_vol9_iss_1_2-2.pdf.

US Bureau of Labor Statistics. “National Longitudinal Survey of Youth 1997.” https://www.nlsinfo.org/content/cohorts/nlsy97.