There are “serious flaws” in the way the UK Government measures
educational performance, according to a new study led by
Professor John Jerrim at the UCL Social Research Institute.
The government rely on the Organisation for Economic
Co-operation and Development (OECD) PISA data which tests
15-year-olds in reading, science and mathematics in around 80
countries to see how pupils in the UK rank
internationally.
PISA results have consistently driven changes to
schooling systems across the globe and in the UK. PISA has also
become the main resource to compare outcomes across its four
devolved nations, making it the only cross-national assessment,
measured on a regular basis.
The study, published today online and in the
forthcoming edition of the Review of Education, analyses
results from the OECD PISA study and highlights serious concerns
about the 2018 edition data for England, Wales, Northern Ireland
and Scotland.
The peer-reviewed research also finds that key
limitations with the data have not been transparently reported –
particularly by the Scottish Government and Professor Jerrim is
now calling for the Office of Statistics Regulation to conduct an
urgent review of how this key piece of education data is
reported.
In addition, the study finds that low levels of
participation in countries such as England and Scotland may have
resulted in potential biases in the PISA data with low achieving
pupils being underrepresented. In the case of Wales, he finds
that PISA mathematics scores could be inflated by as much as 15
points. This could lead Wales to fall seven places in the world
rankings; significantly below the international average, taking
Wales on a par with countries such as Malta and
Belarus.
Report author, Professor Jerrim (UCL Social Research
Institute, part of UCL Institute of Education) said: “PISA is
meant to be a representative study of 15-year-olds across the UK.
But there are serious flaws with some children being excluded
from the study, schools being unwilling to participate, and some
pupils not showing up for the test.
“In England and Wales, there is clear evidence that
some lower-achieving pupils have been systematically excluded.
While what has happened in Scotland is, frankly, a bit of a
mess.”
Other key findings from the paper
include:
-
In the UK, around 40% of students are not included
in the PISA data. This is amongst the highest anywhere in the
world.
-
Scotland changed the PISA test date in 2018 to
later in the school year. This meant that half the sample were
in a later school year (S5) than in previous editions of the
test. Yet this was not commented upon by either the Scottish
government or by the OECD.
-
A number of other anomalies emerge with the
Scottish data – such as a very high number of “ineligible”
pupils – which the Scottish Government have not adequately
explained or discussed.
-
In England and Northern Ireland, bias analyses were
produced but not published. These were obtained by Professor
Jerrim via freedom of information request and revealed – in the
case of England – schools with lower historical GCSE grades
were more likely to refuse to participate in the
study.
-
The PISA sample for Wales seems to systematically
underrepresent those who achieve low GCSE grades. For instance,
according to official government data, 41% of young people in
Wales failed to achieve a GCSE C grade in mathematics in the
2018/19 academic year. Yet, the figure for the PISA 2018 cohort
was just 31%.
One limitation of the paper is that it has focused
upon the PISA 2018 data for the UK, and not how similar issues
may have affected other countries and previous PISA rounds. Other
countries, particularly those with low overall participation
rates, accompanying the UK such as Canada, Sweden, New Zealand,
Portugal and Hong Kong may also be experiencing similar
issues.
Professor Jerrim added: “The most pressing issue is
for the Office for Statistics Regulation to conduct an
independent review of the UK’s PISA data. Clear guidelines need
to be put in place to ensure more transparent reporting in the
future.
“Crucially, at an international level, the OECD needs
to reconsider its technical standards, the strictness of which
these are applied, and its data adjudication processes. The
processes currently in place are nowhere near robust enough to
support the OECD’s claims that PISA provides truly representative
and cross-nationally comparable data.”
J Jerrim, ‘PISA 2018 in England,
Northern Ireland, Scotland and Wales. Is the data really
representative of all four corners of the UK? will be
published on Thursday 22 April and is under strict embargo until
then. When the embargo lifts the paper will be available
here: https://johnjerrim.com/papers/
