We want our educational efforts to be informed by research as research will be able to show us what works and what does not. Hattie’s longitudinal study for example indicated the effect size of various educational interventions, drawn from a large number of studies conducted over a significant period of time. From this research we can identify the activities that we need to do more of and the activities which have little effect and therefore we shouldn’t spend as much time on. This all seems simple. Taking a research informed approach seems logical so why wouldn’t you take this approach?
As is normally the case the world isn’t as tidy and simple as we would like it. Hattie’s study is a good example of research in that it gathered data from across a number of different studies and contexts, plus over a period of time. It therefore presented findings which could be more easily generalised across educational settings and contexts. The issue here is the generalisable nature of the findings. It means that the findings “generally” hold true. In specific contexts or situations it is therefore possible that the findings may not hold true. Looking at education in general this is all well in good but teachers are dealing with individual students in their classrooms and therefore should be seeking to find what works for each child. Holding too strong a view in relation to research findings may lead to practices that don’t work with certain students being applied because the research shows they “generally” work. Worse still it could lead to practices that do work in a given situation and/or context being labelled as “generally” inappropriate and not being tried. We need to see educational research as a guide but be careful to understand that in some situations, doing the opposite may equally be effective.
Hattie’s study is based on a thorough and large data set meaning its statistical reliability is reasonable high. One problem with educational research is that most studies are not based on such a large data set. They are often based on a very small sample of schools and students. Studies are often conducted within a specific context such as a certain geographical area, national or region culture, certain age range or curriculum subject. The validity of the findings when generalised outside the context of the study is often questionable. I remember my own masters level study when we were guided on the need to state that the findings “suggested” or “pointed towards” as opposed to “demonstrating” or “showing” something to be true. You will find in most good education research a similar language in the conclusions. Without a large amount of data gathered from different contexts across a period of time it is highly unlikely any research findings can be generally applied across all or even most educational contexts. Even where findings are generalizable this doesn’t mean they are replicable in an individual context.
I need to be clear, I am not saying we shouldn’t use educational research in directing practice in individual schools and classrooms. What I am saying is we should do so with an awareness of the limitations, and bear these in mind.