{"id":225179,"date":"2025-02-07T05:00:00","date_gmt":"2025-02-07T05:00:00","guid":{"rendered":"https:\/\/schoolsweek.co.uk\/?p=225179"},"modified":"2025-02-06T13:11:14","modified_gmt":"2025-02-06T13:11:14","slug":"report-card-reforms-run-critical-reliability-risks","status":"publish","type":"post","link":"https:\/\/schoolsweek.co.uk\/report-card-reforms-run-critical-reliability-risks\/","title":{"rendered":"Report card reforms run critical reliability risks"},"content":{"rendered":"\n

Ofsted\u2019s planned changes to school inspections<\/a> are wide-ranging. In the face of such change, there is a danger of losing sight of inspection\u2019s core purpose: to provide judgments of school quality. If those judgments are not sufficiently reliable<\/a>, they undermine the very granularity the changes are aiming to provide.<\/p>\n\n\n\n

One clear positive from a reliability and consistency<\/a> perspective is that ungraded inspections will no longer take place from next year. They didn\u2019t have a lot of resource devoted to them, and research suggested that \u2018Good\u2019 judgments based on graded and ungraded inspections were not comparable.<\/p>\n\n\n\n

That said, reliability is a single point of failure<\/a> for school inspections and it remains risky for the proposed new system. Here are four ways it could become a critical issue, and what Ofsted can do to mitigate some of these risks.<\/p>\n\n\n\n

The sum and the parts<\/strong><\/h2>\n\n\n\n

It is well-known that scores awarded to sub-domains are less reliable than subject grades overall. For example, we are less confident in pupils\u2019 skills in different aspects of mathematics (geometry, statistics, algebra) than in their mathematics ability overall.<\/p>\n\n\n\n

The same almost certainly holds true for school inspection. So, by moving the focus away from an overall effectiveness grade to (at least) eight evaluation areas, Ofsted may be inadvertently worsening the reliability of the information they are providing.<\/p>\n\n\n\n

If not measured with sufficient reliability, this could lead parents (and those responsible for school improvement) to focus their attention on the wrong things.<\/p>\n\n\n\n

Correlation matters<\/strong><\/h2>\n\n\n\n

One of the problems with the four sub-judgments in Ofsted\u2019s current model is that they are so highly correlated that they tend to be of little use. Ofsted should therefore make clear how correlated they believe the scores across their new evaluation areas should be, and then see how this stacks up when their changes are being tested.<\/p>\n\n\n\n

For instance, our view is that personal development scores should be more strongly correlated with behaviour and attitudes than (for instance) curriculum.<\/p>\n\n\n\n

Were this not the case, we think it might bring into question how inspectors are awarding scores across the different evaluation areas, and whether the proposed changes can reliably offer the nuanced detail they are meant to bring.<\/p>\n\n\n\n

Either way, now is the time for Ofsted to set out exactly what patterns they expect to see.<\/p>\n\n\n\n

Inconsistent reliability across areas<\/strong><\/h2>\n\n\n\n

It seems likely that some evaluation areas will be measured with greater reliability than others. For instance, if achievement and attendance draw more heavily on quantitative data, then we anticipate they will be rated more consistently than personal development.<\/p>\n\n\n\n

Ofsted need to explore and clearly document such matters. If parents are meant to use this information to inform their choices, they need to know which judgment they can have greater (and lesser) confidence in.<\/p>\n\n\n\n

Reliable enough for the job<\/strong><\/h2>\n\n\n\n

Lets be clear. No school inspection regime will be 100 per cent reliable. What matters is whether it is sufficiently reliable for the purpose that it is being used.<\/p>\n\n\n\n

For that reason, it is important for Ofsted to make clear what level of reliability they would deem to be acceptable versus what would cause them concern.<\/p>\n\n\n\n

This is likely to vary according to the consequences attached. As there are still sanctions associated with the lowest grades (particularly for leadership), the level of reliability here needs to be higher than \u2013 for instance \u2013 those at the margin of being rated \u2018Secure\u2019 versus \u2018Strong\u2019.<\/p>\n\n\n\n

In sum, it is unclear if the proposed new grading system will be reliable enough to be useful.<\/p>\n\n\n\n

Ofsted should therefore do three things:<\/p>\n\n\n\n

    \n
  • set out what they consider to be an adequate level of reliability for each of the criteria in their framework \u2013 particularly for those that are high-stakes.<\/li>\n\n\n\n
  • commit to an independent and transparent programme of research to test the reliability of inspection<\/li>\n\n\n\n
  • be prepared to make changes if the inspection grades are found not to meet Ofsted\u2019s standards for reliability.<\/li>\n<\/ul>\n\n\n\n

    We can spend months discussing the detail of the new framework. But unless the judgments inspectors produce are sufficiently reliable, it will all be for nought.<\/p>\n\n\n\n

    This article was co-authored with Dr Sam Sims, Associate professor, UCL Centre for Education Policy and Equalising Opportunities<\/em><\/strong>
    <\/p>\n","protected":false},"excerpt":{"rendered":"

    Introducing a new school grading system also brings new risks that outcomes won\u2019t reliably measure school quality<\/p>\n","protected":false},"author":311124,"featured_media":225192,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"om_disable_all_campaigns":false,"footnotes":""},"categories":[4],"tags":[],"post-theme":[3559],"class_list":["post-225179","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news","post-theme-opinion-accountability"],"acf":[],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t