PISA: A balanced analysis?

So the PISA results were released today and with them a flurry of online articles providing various analysis and conclusions from the data.   It is my intention to post a couple of times over the coming weeks in relation to PISA and standardized testing.    As my first post my inspiration comes from a twitter discussion from the weekend, as part of the weekly #sltchat, where recruitment was being discussed.    The below tweet highlights the particular strand of the chat which I would like to highlight:

There's a wider issue here… the press have a lot to answer for! Any problem and the finger gets pointed at schools #SLTchat

— Mr B (@MrB_online) December 4, 2016

This strand of the discussion revolved around this constant need to bang on about how UK education is failing, is poor, isn’t working and a variety of other less than positive descriptions.

So what does this have to do with PISA.   Well, during my usual browse through social media and the news I came across an article in the Guardian looking at the PISA results.   You can read the article in full here.   The title:

“UK schools fail to climb international league table “

The use of “fail” in the article title; Not exactly a positive start.    It become more interesting when you dig around in some of the figures.   Lets just take the Science results;  The results show a fall from 514 in 2012 down to 509 this year which seems to align with the less than positive reporting however this doesn’t tell the full story.   It should also be noted that a drop of 5 represents a less than 1% variation.   Could this variation be explained through uncontrollable random variation within the sample group?    Is this drop statistically significant?   I doubt it.

Ignoring the issue of statistical significance for a moment, the UK science position in the rankings rose 6 places between 2012 and 2015 which seems to be a slightly more positive picture.   Also looking at the average of all countries we find that this fell from 501 to 493, representing a drop of 8 yet the UK only dropped by 5.   This UK difference, the drop of 5, could be considered as an improvement against the average across the period.   Also we should note that UK score of 509 is above the average, which again sounds reasonably positive.   A US article on the TIMSS data from last week had proclaim merrily that US students were “above average” however in this negative article we in the UK make no such claim despite the fact it would be valid.   The article was quick to point out falling UK results but didn’t report changes in the OECD averages across the period for comparison.

The article also didn’t share any information with regards the numbers of students involved in testing within each country and how this sample compares to the overall population of study within each country.   From a statistical analysis point of view this information would help in establishing the reliability, or lack there of, for the data.

So all in all I feel the negativity of the article doesn’t truly tell the story plus there is a lot of information missing which may cause us to question or at least assign less weight to the findings.

And all of the above is before I start discussing the issue of using standardized testing as a way to direct how individual students are taught in individual schools within individual geographical areas each with their own individual needs and context (did I use “individual enough to get my point across?).    Not to mention possible discussions in relation to the statistical value of the findings and also the impact of natural random variation on the results.

Do I like or value the PISA finding?   No not really, but that isn’t the point here.   My point, and I may have gone the long way about it, is why are we allowing such a negative view to be projected onto our education system when even the data seems to have some possible positive indicators.   Lets have some celebration of successes for once, of first steps in a positive direction.   Lets have anything except finger pointing!

Research based education

There has been a lot of talk over recent months and years about the importance of “research” based practice in teaching and about the importance of research evidence to back up any new technique, approach or fad.   The recent articles following the release of the TIMSS results and the articles which are likely to follow the PISA results due in a weeks time go to show the value which is being attributed to research findings, to quantifiable measures.

The issue is that the idea of a given approach or finding being validated by research make intuitive sense and therefore it seems logical if not common sense that such an approach be taken.     As such we fail to consider the full implications of research and in particular the importance of sample size within the research methodology.

We seek to identify approaches which will be transferable and applicable across the whole of education.   We seek to find those magical teaching methods and learning activities that can successfully be used independent of whether we are in a UK state school in a deprived area or a private school in the UAE.     We seek to make general statements in relation to the state of Maths education, or other subjects, in whole countries or even continents.   The sum total of all children currently in education therefore forms our overall target population.    Based on this any study of 10 schools or even 100 schools makes up a tiny, need I say insignificant, proportion of the overall target population.   Taken on face value the sample size of 600,000 students for TIMSS 2015 sounds impressive however as a percentage of all students within the age ranges covered by TIMSS across all countries involved I suspect it will be a small number.

Daniel Kahneman in his book Thinking fast and slow (2014) discusses the issue of “the law of small numbers” in that, where the sample size is small there is a greater tendency for variance to occur.    He specifically mentions education and how research evidence has suggested, and I am careful to say suggested as opposed to proved, that small schools perform better than larger schools.    He then mentions contradictory evidence which suggests small schools perform worse.    The reasoning behind these contradictory findings Kahneman suggests is the fact that the small sample size used in a small school involved in these studies allows for local variance within the sample which is not mirrored across the target population.   So a small number of high achieving students in one year can result in a significantly positive average, whereas the following year a small number of low achieving students in a year can result in a significantly negative average.   Where the sample size is bigger, such as in a bigger school, the impact of a small number of students is lesser as a result of the total number of students.   So there is a greater likelihood for small schools, those with a small sample size, to appear in either the top or bottom as a result of random variation.

Taking the above into account I wonder about TIMSS 2015 and the fact that Singapore and Hong Kong are both at the top.   These each have a total population according to google of 5.4 and 7.2 million people.   How can we compare these with the UK and USA with populations of 64 and 319 million people?    The smaller sample size allows for more random variation.   Now it might be claimed that the fact they have remained at the top across different years shows this isn’t random variation however as Naseem Taleb suggests in The Black Swan, it only takes a single set of data to refute findings which countless previous data might have appeared to confirm.   TIMSS so far has only seen 6 data sets, 1 every 4 years since 1995, so maybe the next TIMSS data will be the one which provides the Black Swan.

Having given this some thought I wonder if the issue is the viewpoint we are taking which is one of education on a macro level.    Maybe the intuitive pursuit of research based practices is as valid and worthwhile as it feels however the problem lies in trying to look holistically.      Looking at practices in our own school or in a small number of local or very similar schools and at things, practices and approaches that work may be more productive.    We could still use a research based approach however it would be at a micro rather than macro level.       I can also see some linkages here to the teachmeet movement as surely it has been about grassroots teachers getting together to discuss their approaches and what works in their classrooms.

Maybe we need to stop looking for “the” answers and start focusing our energy on looking for “our” answers to the question of how we provide the students in our individual schools with the best learning experience and opportunities possible.

 

Standardized Testing

I have written a number of times about my feelings with regards standardized testing.    (You can read some of my previous postings here – Some thoughts on Data , Building Test Machines).   Having worked internationally in schools in the Middle East I am particularly aware of standardized testing and the weight put on the results from such testing.   Within the UAE there is a focus on ensuring that education is of an international standard with the measure of this international standard being the results from PISA and also from EMSA testing regimes.    As a result individual schools and their teachers are expected to pore over the EMSA results and analyse what the results mean.    I feel that this focus on a standardized testing regime such as PISA is misplaced as how can we on one hand seek differentiated learning tailored to students as individuals while measuring all students with the a single standardized measure.

As such it was with great interest I read the article in the TES titled, “Ignore Pisa entirely,’ says world expert”.     The article refers to comments provided by Professor Yong Zhao who I was lucky to see at an SSAT conference event back in 2009.    Back then I found Professor Zhao to be both engaging and inspiring as a presenter, with some of his thoughts echoing some of my own plus also shaping some of the thoughts and ideas that I came to develop.    Again I find myself in agreement with Professor Zhao.    I particularly liked his comment regarding the need for “creativity, not uniformity”.

I feel the focus on PISA is the result of valuing what is measurable as opposed to measuring what is valued.      Measuring student performance in a standardized test is easy, with various statistical methods then allowing for what appears to be complex analysis of the data, therefore lending us to be able to prove or disprove various theories or beliefs.     Newspapers and other publishers then sensationalize the data and create causal explanations.   Education in Finland was heralded to be excellent recently as a result of the results from PISA testing.     Teaching in the UAE was deemed to be below the world average however better than most other Middle East countries.    Did PISA really provide a measure of the quality of education?    I think not!

Can education be boiled down to a simple test?   Is a students ability to do well in the PISA test what we value?    Does it take into consideration the students pathway through learning as the pathway differs from one country to another?   Does it take into consideration local needs?   Does it take into consideration the cultural, religious or other contexts within which the learning is taking place?    Does it take into account students as individuals?    Now I acknowledge that it may be difficult or even impossible to measure the above however does that mean that we accept a lesser measure such as PISA just because it is easier?

There may be some place for the PISA results in education however I feel we would be much better focusing on the micro level, on our own individual schools and on seeking to continually improve, as opposed to what Professor Zhao described as little more than a “beer drinking contest”.

 

Education: Time to consider the customer?

I have recently been listening to Sir Richard Branson’ s audio book, The Virgin Way, and it has got me thinking about a number of things.   In one particular chapter of the book a number of companies were discussed, where each had been highly successful however then went through a period of significant loss.   Sir Richard suggested that these companies lost significant amounts of money due to a mistaken focus on “challenging” financial targets as the key indicator and focal point for performance discussions, at the expense of the overall good of the company and its organisational culture.    Having recently done a study on culture within international schools operating in the UAE, this made me consider possible parallels between the business world and the world of education.   Deal and Peterson (2009) in their book suggested that schools could learn from the business world in terms of developing culture.

In Sir Bransons book the focus on financial targets is identified as an error contributing to losses in the financial results which are being examined.   He suggested that the finances are a bi-product of the business as opposed to its sole aim, indicating that within his organisations the aim is to deliver excellent customer service and to look after the customers first.   He goes on to explain that if he looks after his customers, they will look after him and that positive financial outcomes will be a bi-product of this.

Drawing comparisons between Sir Branson’s comments with regards business and education I would suggest that the financial results examined to assess the success of businesses may be similar in nature to the standardised test results being examined to assess the success of schools.    Within education in recent times there has been a significant amount said and a significant focus put on standardised test scores.  Based on PISA results for example, Finland has been proclaimed by some to be the best education system in the world.

The question is whether education, like the businesses which suffered significant losses, is too focused on these specific standardised measures of educational performance at the expense of the culture of schools or the good of education in general.     Has education lost its focus on why education, schools and teachers exists;  Our purpose?

Now I know the above is very much general in nature and therefore does not apply to all schools or education systems.   My point is that in general I believe we need to step back and relook at what is important and our overall reason for teaching.     We need to look at the cultures of schools rather than standardised test results.    Sir Richard repeatedly discusses the importance of a focus on customer service.    Should we do the same and re-evaluate what we see as most important, maybe showing a little less attention to the standardised test results and a little more to our customers; The students!

 

Image courtesy of cooldesign at freedigitalphotos.net