Ofsted admits reliability problems

Ofsted has not done enough to ensure school inspections are reliable, one of Ofsted’s directors admits.  

Ofsted is the Office for Standards in Education, Children’s Services and Skills. They inspect and regulate services that care for children and young people, and services providing education and skills for learners of all ages. The Office carries out regular inspections of each school in England, resulting in a published evaluation of the effectiveness of the school.

Sean Harford, Ofsted’s national director for schools, responds to a critical blog from head teacher Tom Sherrington of Highbury Grove School in north London. Some inspectors use data as a “safety net” instead of making a professional judgement, Harford wrote. Tom Sherrington had complained of “enormous flaws and absence of proper validity trials” in the current inspection system.

In the blog, Sherrington  had rejected the idea that schools “can be judged in a meaningful way via inspections”. “By ‘judged’ I am not talking about an experienced visitor giving some insightful developmental feedback based on an analysis of the available data and their observations; no doubt there are some people out there who can do this well enough. “I am talking about the process of distilling this mass of qualitative and quantitative information into a simple set of final grades, with one overall Judgement Grade”.

In his response, Harford admitted Ofsted does not currently ensure “directly that different inspectors in the school on the same day would give the same judgement”. He also agreed “some inspectors and some schools focus too much on a narrow range of data”. He said Ofsted trained its inspectors to use data as a “signpost”, rather than making it a “pre-determined destination”. “But the weakest ones have been guilty of using the published data as a safety net for not making fully-rounded, professional judgements.”

He said inspectors should draw on information from the school itself, including pupils’ work over time, progress across year groups and classes, improvements to teaching, and pupils’ attitudes to learning.

Harford said Ofsted’s pilots for its new short inspections would include reliability testing. “If reliability is a problem, we will review the issues to see what we need to do to make the inspections reliable.” Last month Ofsted closed its consultation on a new inspection framework which will introduce shorter inspections for all schools rated as good in England.

School leaders have described Harford’s comments as a definite shift in tone. Sherrington told the Times Educational Supplement he was amazed by the response to his blog, arguing that inspectors “can’t just continue to assert authority – they have to demonstrate reliability”.

Better data would lead to better decision-making around school choice?

The American Enterprise Institute (AEI) recently published the report, Better Data, Better Decisions: Informing School Choosers to Improve Education Markets. The AEI argues that better data would lead to better decision-making around school choice for parents, presumably strengthening good schools and allowing families to steer clear of low-quality schools. This report was reviewed by the National Education Policy Center

The AEI report provides a thoughtful reflection on, and assessment of, school choice decision-making. It reviews a number of different efforts to engage stakeholders in the school choice process. Ultimately, while the report does offer concrete suggestions aimed at improving the quality of school choice decision-making, it does not successfully argue that increasing choice will increase school quality overall. The report does recognize that many school-choice decisions are made based on informal networks and criteria that researchers have had difficulty identifying and evaluating.

This 16-page report (Better-Data-Better-Decisions)  asserts that choice is seen as a mechanism, and at times a panacea, for better educational quality. Given the intense interest in choice as a reform strategy, the report promises to provide an “empirical look at how governments and other third-party organizations can help inform families about their school choice options” (p. i). The report considers how to better inform choosers. Recognizing that no choice is ever perfectly informed, the report provides background on choice theory, focusing on the notion of “bounded rationality.” What this means is “decision making happens in contexts in which people lack relevant information about their options, have finite time to commit to collecting that information and are limited in their abilities to process and use what information they have” (p.2). The author argues that school choice is not an entirely “rational” decision but is shaped by 1) the context within which the choice is made, 2) the background of the chooser, and 3) the available information.

The report provides an overview of what families want from schools, where they get information on schools, and how they use information to make decisions. This section of the report relies on research conducted by the author. He notes that most information on what families want from their children’s schools is based on survey data which, while useful in some respects, does not take into account desirability bias: the tendency of respondents to respond in ways they think they are “supposed to.” The report suggests that other factors (such as accessibility and information from social networks) are actually more influential in real-life decision-making. It further argues that preferences and desires are malleable: they can change over time and in light of available information.

The report asserts that research on school choice has described two main sources of information: social and formal. Parents talk to other parents and make decisions based on these conversations. Some parents also consider other sources of information, including school districts, governmental entities, and non-governmental third parties. Two critical points the report makes about families’ ability to gather information are: 1) the quality of information available through social networks is heavily influenced by social class, with more well-to-do families typically having access to better information, and 2) the way information is presented (timing, format, amount, source) influences the capacity of the student or parent to use the information effectively. The report notes that finding ways to cut through “data smog” to determine what information is reliable, good, and relevant is a primary challenge. Lastly, the report addresses the challenge posed by the fact that, especially for middle and high school students, there are multiple decision makers (parents and students) who come to the process with different informational needs and desires. The report addresses the challenge of helping parents and students make good decisions that prioritize academic quality.

The report concludes with a series of recommendations presented in the language of bounded rationality; that is, the author advocates solutions that take into account the imperfect human reality of decision-making and that improve the odds that parents and students will make “good” decisions by providing them with good information in a timely fashion in an accessible format. The report encourages families to actually make a choice— to consider the myriad options available to them rather than “choosing” those most readily accessible (convenient) to them or passively accepting the schools selected by others in their social network. Those interested in improving student and parental decision-making are encouraged to be accurate in their information, to provide it in an accessible format, and to accommodate the imperfect reality of real people making hard choices that balance many facWindow_en_www_aei_org_wp-content_uploads_2014_11_Better-Data-Better-Decisions-4_pdftors beyond a school’s academic quality.

AEI’s Better Data report is grounded in both the extant literature on school choice and the author’s own research. Beginning with a useful introduction to the concept of “bounded rationality,” the report goes on to review recent research on choice generally and school choice specifically. The report relies on the literature from psychology, economics, and education to provide the reader with an understanding of work on choice and decision- making generally as well as on school choice in particular.

The author presents findings from two studies he conducted: an Online Experiments Study and a Field Experiments Study. The methods and findings for both studies are presented in sepfieldarate text boxes in the report, while conclusions drawn from the research are embedded in the report.

The Online Experiments Study used an online survey of 1,000 adults to understand how individuals use numerical ratings of components of school quality and parent comments.