Now the A-Level and GCSE results are out the usual sets of analysis and observations based on the data have started making an appearance. As usual causal explanations have been developed to explain the data, using what Naseem Taleb described as the backwards process. The resulting judgments have been established to fit the available data without any consideration for the data which is not available.
The perfect example is an article in the guardian (Wales A-Level results raise concerns pupils falling behind rest of UK, Richard Adams, Aug 2016) discussing the A-level results in Wales as compared with the results in England. The overall drop in the percentage entries achieving A* and A dropped in England “only slightly to 25.8%” while in Wales I “fell more steeply to 22.7%”. The causal explanation apparently arrived at by one “expert” was that boys had been “possibly distracted by the national football team’s success at Euro 2016”. This fails to consider the total number of entries in England when compared with Wales; I suspect Wales would have less entries therefore resulting in increased variability in Welsh results versus English results. The data also fails to include any information in relation to the students GCSE results. Had the Welsh students achieved lower GCSEs results than their English counterparts it may be that their overall lower level of achievement could amount to “better” results given their lower starting point as measured by GCSEs.
Another possible conclusion, which is easy for me to draw as a Scotsman and most likely more difficult for an Englishman, is that the data shows something which wasn’t related to the Welsh football performance at all. The English A-Level results could be better due to English students throwing themselves into their work following England’s poor showing during Euro 2016. It’s the same data but a different conclusion which has been generated and made to fit the data available without any consideration for the data which isn’t available.
Having considered further this issue I think I am now more inclined than ever to agree with Talebs comments regarding the importance of the unread books in a library rather than the read ones. Talebs discusses how a home library filled with read books gives a person the illusion of knowledge; the person has read it all. A library filled largely with unread books however makes clear all that we do not yet know and have not read. Reading each of these commentaries and analysis in relation to the A-Level data isn’t making me more informed or more educated, in fact it may be blinding me to the “true” facts or to other possibilities. I think, therefore, that this will be my last post moaning about “expert” analysis or results as from now on I need to stop reading the analysis in the first place!