PISA: A balanced analysis?

So the PISA results were released today and with them a flurry of online articles providing various analysis and conclusions from the data.   It is my intention to post a couple of times over the coming weeks in relation to PISA and standardized testing.    As my first post my inspiration comes from a twitter discussion from the weekend, as part of the weekly #sltchat, where recruitment was being discussed.    The below tweet highlights the particular strand of the chat which I would like to highlight:

This strand of the discussion revolved around this constant need to bang on about how UK education is failing, is poor, isn’t working and a variety of other less than positive descriptions.

So what does this have to do with PISA.   Well, during my usual browse through social media and the news I came across an article in the Guardian looking at the PISA results.   You can read the article in full here.   The title:

“UK schools fail to climb international league table “

The use of “fail” in the article title; Not exactly a positive start.    It become more interesting when you dig around in some of the figures.   Lets just take the Science results;  The results show a fall from 514 in 2012 down to 509 this year which seems to align with the less than positive reporting however this doesn’t tell the full story.   It should also be noted that a drop of 5 represents a less than 1% variation.   Could this variation be explained through uncontrollable random variation within the sample group?    Is this drop statistically significant?   I doubt it.

Ignoring the issue of statistical significance for a moment, the UK science position in the rankings rose 6 places between 2012 and 2015 which seems to be a slightly more positive picture.   Also looking at the average of all countries we find that this fell from 501 to 493, representing a drop of 8 yet the UK only dropped by 5.   This UK difference, the drop of 5, could be considered as an improvement against the average across the period.   Also we should note that UK score of 509 is above the average, which again sounds reasonably positive.   A US article on the TIMSS data from last week had proclaim merrily that US students were “above average” however in this negative article we in the UK make no such claim despite the fact it would be valid.   The article was quick to point out falling UK results but didn’t report changes in the OECD averages across the period for comparison.

The article also didn’t share any information with regards the numbers of students involved in testing within each country and how this sample compares to the overall population of study within each country.   From a statistical analysis point of view this information would help in establishing the reliability, or lack there of, for the data.

So all in all I feel the negativity of the article doesn’t truly tell the story plus there is a lot of information missing which may cause us to question or at least assign less weight to the findings.

And all of the above is before I start discussing the issue of using standardized testing as a way to direct how individual students are taught in individual schools within individual geographical areas each with their own individual needs and context (did I use “individual enough to get my point across?).    Not to mention possible discussions in relation to the statistical value of the findings and also the impact of natural random variation on the results.

Do I like or value the PISA finding?   No not really, but that isn’t the point here.   My point, and I may have gone the long way about it, is why are we allowing such a negative view to be projected onto our education system when even the data seems to have some possible positive indicators.   Lets have some celebration of successes for once, of first steps in a positive direction.   Lets have anything except finger pointing!


Author: garyhenderson2014

Gary Henderson is currently the Director of IT in an Independent school in the UK. Prior to this he worked as the Head of Learning Technologies working with public and private schools across the Middle East. This includes leading the planning and development of IT within a number of new schools opening in the UAE. As a trained teacher with over 15 years working in education his experience includes UK state secondary schools, further education and higher education, as well as experience of various international schools teaching various curricula. This has led him to present at a number of educational conferences in the UK and Middle East.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: