Is doing more and efficiency our aim?

I have long been concerned by the “do more”, and “be more efficient” narrative which seems to surround our everyday lives.   We are constantly seeking to improve in all we do, which I think is a fair endeavour, but at what cost?   This was recently brought further into focus as I started reading “Thank You for Being Late: An optimists Guide to Thriving in the Age of Accelerations” by T.L. Friedman as I found myself with an hour to spare while waiting to meet someone.   I found myself that bit more content and relaxed as I used the extra hour which had become available to start reading the book and to engage in a bit of people-watching, watching the world rush about its business.  But are these opportunities to stop and reflect reducing in frequency and length?

I look at teaching for example, where I qualified as a teacher back in the late 90’s.   Looking at teaching now, there are so many more things to consider and to do whether this relates to educational research that we are considering, safeguarding, well-being, health and safety, neurodiversity, and much more.  Now all of these things are important but each is another thing to consider, additional cognitive load, or an additional process or task which needs to be completed.  Is there an extra resource in terms of time or cognitive capacity to undertake these things?   The answer is No.   We simply fold them into our everyday workload, which invariably means that although our efforts are getting better, we are also doing more than we ever did before.

Now generative AI can help a little here in that it can help us with some of the heavy lifting and free up some time for us.    This particular post was edited with the help of AI although it wasn’t initially drafted with AI;  I didn’t draft it with AI as this is very much a brain dump of thoughts and as yet AI solutions can’t interface with the human brain, although that may become possible at some point.    But in editing it with AI, I was able to proofread and make changes quicker than I would have been able to do myself therefore reducing the time taken to produce the post.    The challenge here however is this still all exists against a backdrop of “do more”, so the time I may have gained through the help of AI may simply be swallowed up by the next task I need to undertake to continue down the road of continual improvement.   In effect, the net benefit of AI may be quickly nullified by our continued drive for efficiency and maximising output.

Circling back to teaching, this therefore means that generative AI may benefit teachers for a short period, but that eventually, the benefits may simply dissolve in the face of ever-increasing requirements.    But the benefits are so important, that extra time might allow for greater teacher reflection on teaching practice, student learning and student outcomes, it might support greater networking and sharing of ideas plus might support improved well-being for teachers, which I would suggest may result in better teaching, better student outcomes and also better student wellbeing as the students see their teachers modelling good wellbeing practices.   The time AI solutions will provide might support us in spending more time on focussing on what it means to be human and on “human flourishing”.

 Maybe we need to question to “continual improvement” and “efficiency” narratives in that they need to exist in balance and cannot be assumed to be the “right” path.   In relation to continual improvement, I often refer to MVP, minimum viable product and “good enough”.    In relation to efficiency, if I wanted to be more efficient maybe I should stop taking breaks or work through my lunch.    We also need to consider decreasing marginal gains, and maybe that is where we are now, that a lot of the improvements we are bringing about are minor, iterative improvements, but at the cost of cognitive load, time and other resources which may outweigh the resultant benefit.   The extra effort required for each incremental change remains the same, yet the resulting gain is reduced with each change. There is also the challenge of complexity, where more complex processes or systems often bring about greater risk of failure or greater reliance on particular people or tools. And I haven’t even mentioned the speed of change, which the book I am reading refers to in its title, in the “age of accelerations”.   So all of this is happening quicker than ever before which therefore suggests the amount of time we have available to adapt to changes is decreasing.

I don’t have any answers here, so the purpose of this post is not to share a solution, but to pose a question.   I think I know the answer to the question, but not necessarily the answer to the problem it hints towards, but I think the best thing we can do is to start to talk about it and consider it.   So what is the question:

Can we keep adding to the things we need to think about, the processes and the complexity of our lives, or is there a limit?   

AI and general knowledge

I recently was musing on the benefits of general knowledge.   A recent conference I attended involved Prof Miles Berry where he talked about Generative AI as being very well-read.   I had previously seen a figure of around 2000- 2500 years quoted in terms of the time it would take a human to read all of the content included in the training data provided to GPT 3.5, which in my view makes it very well read indeed.   So, I got to wondering if it is this broad base of knowledge which makes generative AI, or at least the large language models so potentially useful for us.

A doctor and AI

Consider, for instance, a medical practitioner. While their expertise lies in diagnosing and treating illnesses, plus their bedside manner and ability to interact with patients and other medical practitioners, their effectiveness as healthcare professionals hinges on a robust understanding of anatomy, physiology, pharmacology, and medical ethics—domains that draw upon general knowledge. Similarly, an engineer relies on principles of mathematics, physics, and material science to design innovative solutions to complex problems.    As a professional, we are required to study and learn from this broad body of knowledge through degree programmes and other qualification or certification requirements.   But we are inherently human which means just because we have learned something at some point, and successfully navigated a qualification or certification route, doesn’t mean that we will remember or be able to access this information at the point of need.    If the medical practitioner therefore uses the AI to assist them initially, they will therefore be drawing on a bigger knowledge base than a human is capable of consuming, plus a knowledge base that doesn’t forget, or fail to remember content at some point learned.     The medical practitioner will still apply their experience and knowledge to the resultant output, bringing their human touch to help address the challenges of generative AI (bias, hallucinations, etc) however the use of generative AI to assist would likely make diagnosis quicker and possibly more accurate.

My changing workflow

The above seems to align with my views in relation to workflows I have changed recently to include generative AI.  Previously I might have known what I wanted to write and therefore get to writing rather than seeking to use generative AI.    Now I realise that, although I know my planned outcome, something which generative AI cannot truly know, no matter how much I adjust and finesse my prompts, generative AI brings to the table a huge amount and breadth of reading I will never be able to achieve.    As such, starting out by asking generative AI is a great place to start.    It will give you an answer to your prompt but will draw upon a far bigger reservoir of knowledge than you can.   You can then refine your prompt based on what you want to achieve, before doing the final edits.    It is this early use of generative AI which I think is the main potential for us all.   If we use generative AI early in our workflows we both get to our endpoint quicker, plus it also opens us up to thoughts and ideas we might never have considered, due to generative AI’s broader general knowledge. I still point my own personal stamp on the content which is produced, making it hopefully unique to my personal style and personality, but AI provides me with assistance.

Challenges and Considerations

Despite its tremendous potential, the integration of generative AI into everyday life and specialized domains poses several challenges and considerations. Chief among these are concerns regarding the reliability and accuracy of AI-generated content, as well as issues related to bias, ethical considerations, and privacy concerns. I do however note here that the issues of reliability, bias, ethics and privacy are not purely AI problems and are actually human and societal issues, so if a human retains the responsibility for checking and final decision-making, then the issue continues to be that of a human rather than AI issue.

Conclusion

Generative AI stands as a transformative force in harnessing and disseminating general knowledge, empowering individuals with instant access to information, facilitating learning and comprehension, and augmenting domain-specific expertise.    It provides a vast repository of knowledge acquired from its training data, which can be used to assist humans and augment their efforts.   I note this piece itself was generated with the help of generative AI, and some of the text and ideas contained herein are ones I may not have arrived at myself, plus I doubt I would have completed this post quite so quickly.    So, if AI is providing a huge knowledge base and assisting us in terms of getting to our endpoint more quickly, plus opening up alternative lines of thinking, isnt this a good thing?   

For education though I suspect the big challenge will be in terms of how much of the resultant work is the students and how much is the generative AI platforms.   I wonder though, if the requirement is to produce a given piece of work, does this matter, and if AI helps us get there quicker, do we simply need to expect more and better in a world of generative AI?

I suspect another challenge, which may be for a future post, is the fact that Generative AI is a statistical inference model and doesnt “know” anything, so is it as well read as I have made out? Can you be well read without understanding? But what does it mean to “know” or “understand” something and could it be that our knowledge is just a statistical inference based on experience? I think, on that rather deep question, I will leave this post here for now.

Is Gen AI Dangerous?

I recently saw a webinar being advertised with “Is GenAI dangerous” as the title.   An attention-grabber headline however I don’t think the question is particularly fair.   Is a hammer dangerous?   In the hands of a criminal, I would say it is, plus also in the hands of an amateur DIY’er it might also be dangerous, to the person wielding it but also to others through the things the amateur might build or install.     Are humans dangerous, or is air dangerous?   Again, with questions quite so broad the answer will almost always be “yes” but qualified with “in certain circumstances or in the hands of certain people”.    This got me wondering about the dangers of generative AI and some hopefully better questions we might seek to ask in relation to generative AI use in schools.

Bias

The danger of bias in generative AI solutions is clearly documented, and I have evidenced it myself in simple demonstrations, however, we have also more recently seen the challenges in relation to where companies might seek to manage bias, where this results in equally unwanted outputs.   Maybe we need to accept bias in AI in much the same way that we accept some level of unconscious bias in human beings.    If this is the case then I think the questions we need to ask ourselves are:

1. How do we build awareness of bias both in AI and in human decision-making and creation?
2. How do we seek to address bias?   And in generative AI solutions, I think the key here is simply prompt engineering and avoiding broad or vague prompts, in favour of more specific and detailed prompts.

Inaccuracy

I don’t like the term “hallucinations”, which is the commonly used term where AI solutions return incorrect information, preferring to call it an error or inaccuracy.   And we know that humans are prone to mistakes, so this is yet another similarity between humans and AI solutions.   Again, if we accept that there will also be some errors in AI-based outputs, we find ourselves asking what I feel are better questions, such as:

1. How do we build awareness of possible errors in AI content
2. How do we build the necessary critical thinking and problem-solving skills to ensure students and teachers can question and check content being provided by AI solutions?

Plagiarism

The issue of students using AI-generated content and submitting it as their own is often discussed in education circles however I note there are lots of benefits in students using AI solutions, particularly for students who experience language or learning barriers.    I also note a recent survey which suggested lots of students are using generative AI solutions anyway, independent of anything their school may or may not have said.    So again, if we accept that some use of AI will occur and that for some this might represent dishonest practice, but for many it will be using AI to level the playfield, what questions could we ask:

1. How do we build awareness in students and staff as to what is acceptable and what is not acceptable in using AI solutions?
2. How do we explore or record how students have used AI in their work so we can assess their approach to problems and their thinking processes?

Over-reliance

There is also the concern that, due to the existence of generative AI solutions, we may start to use them to frequently and become over-reliant on them, weakening our ability to create or do tasks without the aid of generative AI.   For me, this is like the old calculator argument in that we need to be able to do basic maths even though calculators are available everywhere.    I can see the need for some basic fundamental learning but with generative AI being so widely available shouldn’t we seek to maximise the benefits which it provides?  So again, what are the questions we may need to ask:

1. How do we build awareness of the risk of over-reliance?
2. How do we ensure we maximise the benefit of AI solutions while retaining the benefits of our own human thinking, human emotion, etc?   It’s about seeking to find a balance.

Conclusion

In considering better questions to ask I think the first question is always one about building awareness so maybe the “is GenAI dangerous” webinar may be useful if it seeks to build relevant awareness as to the risks.  We can’t spot a problem if we are not aware of the potential for such a problem to exist. The challenge though is the questions we ask post-awareness, the questions we ask which try to drive us forward such as how we might deal with bias where we identify it, how we might ensure people are critical and questioning such that they sport errors, how we evidence student thinking and processes in using AI and how we maximise both human and AI benefits.  

In considering generative AI I think there is some irony here in that my view is that we need to ask better questions than “Is GenAI dangerous”.    In seeking to use generative AI and to realise its potential in schools and colleges, prompt engineering, which is basically asking the right questions is key so maybe in seeking to assess the benefits and risks of GenAI we need to start by asking better questions.

OABMG Conference

I was lucky enough to be invited to speak at the Oxfordshire Academies Business Managers Group (OABMG) annual conference earlier in the week where I was speaking on AI in education and the possible impact and implications on school business managers.    It was a lovely event and I really enjoyed Sarah Furness the keynote speaker, however, sadly I had to leave following my session in order to catch a train, one of a number of trains needed to get me to and from the event.

Be brave

Sarah was both insightful and entertaining and to be honest, I could likely write a whole blog post just on the stories she shared however let me just summarise my key takeaways from her presentation.    Her key message, which resonated for me, was the need to be brave, which aligns with the values of my school, and also is so very important where we have technology advancing at such a pace but with regulation lagging so far behind.   We have no choice but to be brave especially given both students and staff are already experimenting with the use of AI.  We need to be brave in engaging, we need to be brave in experimenting and we need to be brave in accepting where things don’t go quite as they planned, but learning from these experiences.   The need for sharing, asking difficult questions and accepting challenges also aligned with my thinking, and again looking to AI in education, if we are to find our way with AI in schools I think this all rings very true indeed.  We need to be sharing our thoughts, and both challenging and accepting challenges from others, if we are to move forward.    Sarah’s talk was about leadership, using her context as a military leader and pilot;  maybe this will be key in the use of AI in schools, the need for effective, brave leaders who value and encourage diversity, sharing and challenge.

AI in education

Going into my presentation my key aim was to discuss AI in education and some possible uses for school business leaders.   I don’t have all of the answers, and to be honest, I don’t feel anyone has all the answers when it comes to AI and education, as AI is advancing at a rapid pace where education has changed little and is under both funding and also workload challenges.   That said, as I shared in my presentation, “The smartest person in the room, is the room”.   This David Weinberger quote is one of my favourites and is often used, as it highlights the need to discuss and share, in doing so we hopefully engage others to think about the issue, in this case, AI in schools, and collectively our thinking, our ideas and experience is enhanced.

Now you can view my presentation slides here if you are interested.   

At the end of my presentation, a couple of questions were raised which I would like to just pick up on, namely school engagement in AI in education, policy and also regulation.  

School Engagement in AI

I would like to draw attention to the article in the Express which highlighted that 54% of the students they surveyed were using AI in relation to their homework.  The key thing here is that students are using AI independently of whether schools have considered or talked about AI.  And it isn’t just students, you will also likely have staff, both teaching and support staff who are using AI.   The AI genie is out of the bottle and attempts to block it will inevitably be futile so, in my opinion, it is key that we engage with the use of AI, we talk with students and staff about AI, and that schools experiment and share.    But the fact AI is already here isn’t the only reason to use it in education.   We talk about the need to support individual students, differentiation, English as a second language and also SEND barriers to learning; all of these can be addressed to some extent through the use of AI tools.   Now I will note here that the use of AI tools may also increase some challenges, such as that of digital divides, but that was a key part of my presentation in talking about the risks and challenges first, as we need to use AI but only from a position of an awareness of risks and challenges.

Policies

Linked to the above, I think it is very important that schools put in place an AI policy if they haven’t already done so.   This allows the school to set out its guardrails in relation to the use of AI in the school.  Now there is a brilliant template for this, as created by Mark Anderson and Laura Knight, which can be found here.   Looking to the future I suspect the AI policy might be eventually absorbed into the IT acceptable use and/or academic integrity policies however for now, while AI use in schools is so new, I think having it as a standalone policy makes sense.

Regulation

There will need to be some form of regulation in relation to AI tools including their use in education however we have already seen that the technology is developing very fast while the regulation is lagging so far behind and is slow to adapt.   As such I think we should hope for and support some form of regulation to protect people, including our staff and students, and their data, but I don’t believe we can wait for this to happen.    AI is already here and students and staff are likely using it.  We can’t stop this, so I think we need to run with it, to try and shape the use and hopefully in doing so shape the regulation which follows.  This will mean making risk v. benefit decisions but seldom do we see anything which is beneficial without any risks.

Conclusion

The OABMG conference was enjoyable even though my visit was brief.   It was good to get to share some thoughts on AI in education and I hope those in attendance found the session useful.   My two key thoughts from the event are, the need to be brave, remembering we learn most from our mistakes, and the need in this ever-busy and complex world to share as collectively we are all better for it. I think these are two things I will try do more actively in future.

Thinking about thinking (with AI)

Artificial intelligence (AI) is definitely the big talking point in educational circles at the moment.  You just need to look at the various conference programs and you will almost always find at least one session touching on AI or generative AI.   Now a lot of the discussion is focused on the possible benefits or the risks associated with AI and less so with the practical applications and need to experiment.   It was in thinking about the practical side of things, looking at tools like ChatGPT, Diffit, Gemini and Bing Image Creator among others, that I got thinking how AI might link to meta cognition.

Learning about learning

The idea of learning about learning, about meta cognition, has been around for quite some time.    The thinking being that if we educate students about how they learn and get them thinking about their learning preferences (eek, I almost said learning styles there!) then they can make informed decisions about their learning, and hopefully be better learners.   It seems to make sense.  But how does this link to AI and generative AI?

Learning with a learning assistant

I think the key issue here is how we see AI in terms of the learning experience.   Is it simply a tool to spark ideas?   Is it a tool to review content?   Is it a tool to surface information?   I would suggest it is all of these things and more, and in the case of generative AI can operate as an assistant to teachers or to students.   It is definitely more than a bit of technology or simply a tool as I suspect in its use its shapes our thinking and our processes, much as the simple tools like the hammer shaped human thinking and processes in the past.    We also need to consider that process when working with generative AI (GenAI) is often iterative or taking the form of a dialogue between the user and the genAI solution.  The user fields an initial prompt, to which the genAI responses.   The user then reviews the response against what they were hoping for, and if they are anything like me they realize that they haven’t been specific enough so therefore now provide further directives to the AI, which in turn returns a new, hopefully better response, and so the dialogue continues until an output which is satisfactory to the user is reached.     Now some of this dialogue can possibly be sped up through the use of various prompt frameworks such as the PREPARE framework shared by Dan Fitzpatrick, however even then it is still likely to be a dialogue with Dan also providing a framework for the review and iterative part of this process, his EDIT framework.

Meta AI supported cognition?

If we are looking to prepare students to work with generative AI as their always available assistant I think we also need to start exploring with students how best to use them.   Part of this is about looking at their learning and how their learning processes might be different with AI.   I suppose it’s a bit like if all your learning was done with a partner, with another human being.  Looking at the nature of the interaction, being very much a dialogue, makes this comparison feel all the more apt.   You would need to consider their approach, their emotions, social interaction, etc.   Now an AI doesn’t have emotions or the social side of things, or at least not yet or as we currently know these to exist, but it does have its own approach, its own biases, its own strengths and its own weaknesses.  So if we are using or encouraging students to use AI in learning, I think we need to work with student to unpick the processes rather than simply focusing on the tools.  If I am looking for ideas and to be creative, how best to I use AI?   If I am looking to review and improve my work, how best am I to use AI?    If I want to use AI for research, how best do I do this?    Is this where Meta AI supported cognition comes in?

Conclusion

In relation to technology use in education I have always said it isn’t about the technology but about what you are seeking to achieve.   With AI it might be using Gen AI to produce better coursework or to give you a starting point or some new ideas.    But if we think beyond the short term goals, isn’t it about being able to better use AI to suit our needs as they arise and as such do we then need to spend time with students unpicking the how of their use of Gen AI, understanding the processes, what works and what doesn’t in order to get better in working with our newly found AI assistant?

Might teaching about Meta AI supported cognition become a thing?

AI: Desirable Imperfection?

Might there possibly be benefits in generative AI solutions that hallucinate, make things up and show bias?

We live in a world of convenience;   Once upon a time we had to do research in a library, going through card indexes and looking at the bibliography from one book to identify further reading, which would then necessitate hunting in the library for additional books, and then you would need to summarise everything you read into your piece of work.     Then Google came along and we could do the search far faster, getting instant lists of articles or websites based on a search.   We still needed to look at the content which our searches yielded, before identifying the best source information and then moulding this into our own final piece of work.   Things had become more convenient which was good, but with this came some drawbacks.   As users we tended to look at the first set of results returned, at the first page of search results rather than at subsequent pages meaning we lost some of the opportunities for accidental learning where, in a library, your search for one book might lead you to accidentally find other books which add to your learning.   Also our searches were now being partially manipulated by algorithms as the search wasn’t just a simple search like that of a card index, it was a search which an algorithm used to predict what we might want, what is popular, etc, before yielding it as a search return.    And these algorithms reduced the transparency of the searching process, potentially meaning our eventual work had been partially influenced by unknown algorithmic hands.   Next we started the push for “voice-first” where rather than a list of search items our new voice assistant would boil down the answer to our requests to a single answer spoken with some artificial authority.

So roll in Generative AI and ChatGPT and Bard;  Now we have a tool which will search for content but will also then attempt to synthesise this into a new piece of work.   It doesn’t just find the sources it summarises, expands and explains.    Further convenience combined with further challenges or risks.   But what if there are benefits from some of these challenges such as the hallucinations and the bias?  Is that possible?

Lets step back to the library;   My search was based on my decisions as to which books to select, with my reading and book selections then influencing the further reading I did.    Now bias and error may have been in the books but I could focus on thinking about such bias and error, with error generally a low risk due to the editorial review processes associated with the publishing of a book.     In the modern world however my information might come to me via social media platforms where an algorithm is at play in what I see, choosing what to surface and what not to.   Additionally, content might be written by individuals or groups without the editorial process meaning a greater risk of error or bias.   And with Generative AI now widely available we might find content awash with subtle bias or simply containing errors and misunderstanding presented confidently as face.     As an individual trying to do some research I have more to think about than just about the content.  I need to think more about who wrote the content, how it came to me, what the motivation of the writer was, whether generative AI may have been used, etc.  In effect I need to be more critical than I might have been back in the library.

And maybe this is where the obvious hallucinations and bias is useful, as it highlights our need for criticality when dealing with generative AI content, but also with wider content available in this digital world such as the content which we are constantly bombarded with via social media.   In a world of ever increasing content, increasing division between groups and nations and increasing individuals contributing either for positive or sometimes malicious reasons, being critical of content may now be the most important skill.    

If it werent for these imperfections would we see the need to be critical, in a world where I suspect a critical view is all the more important? And can we humans claim to be without some imperfections? Could it therefore be that actually the issues or challenges of generative Ai, its hallucinations and bias, may be a desirable imperfection?  

Exams and AI: A look at the current system

I recently presented at a conference in relation to AI and assessment.   I think this was reasonably good timing given JCQ had just released further guidance in relation to student coursework and AI plus AQA had announced they were going to use online testing as part of their exam suite in the Italian and Polish GCSEs starting from 2016.    I think this is a positive step forward in both cases however I think it is important that we see this journey as more than simply replacing pencil and paper exams with a hall full of students completing the same exams but as an online/digital exam.   There is significant potential here to ask ourselves what are we seeking to assess, why are we seeking to assess it and how are we best to assess?

The SAMR model

The SAMR model is useful when looking at technology change programmes.   The first element of SAMR is that of simple substitution, similar to the example I gave above in the introduction.   The concern for me is that this might be the goal being aimed at where technology and AI present such significant potential beyond mere substitution, and where the world has moved at a fast technologically drive pace, yet our education system has changed little, and our key assessment methodologies, of terminal coursework and exams have barely changed at all.

In looking to progress beyond substitution it might be useful to unpick some of the limitations of the current system.  For this purpose I am going to focus purely on terminal exams given they are such a significant part of the current formal education system in the UK.   So what are the limitations of the currently accepted system?

Logistics

One of the key drawbacks in the current system, as I see it, is the massive logistical challenge it presents.   Students have to be filed into exams halls across the country and the world all at the same time, to complete exam papers which have been securely delivered to exam centres.    Its quite an undertaking and even more so when you consider trying to keep the papers and questions secure.   In a world of technology where content can quickly and easily be shared it doesn’t take much before questions are out in the open ahead of the exam, advantaging those who have seen the information when compared with those who have missed it.    Then you have the issue of gathering all the completed papers up, sharing them with assessors to mark, quality assurance of marking and then eventual release of results to students some months later.    This is a world where technology supports the sharing of information, written, audio, video and more instantly.  Why cant the exams process be quicker and more streamlined, making use of technology to achieve this?

Diversity

Another key drawback has to be that of diversity.  We, more than ever, identify the individual differences which exist in us all.    Discussion of neurodiversity is common at the moment but despite this we still file all students into a hall to complete the same exam paper.     Now there are exam concessions which can be provided to students but this barely scratches the surface in my opinion.    Where is the valuing of diversity in all of this?

Methodology

We also need to acknowledge that the current exams system very much values those students who are able to memorise facts, processes, etc.   Memorisation is so key to exams success however out in the real world we have access to ChatGPT and Google to find the information we need when we need it, with the key then being how we then interpret, validate and apply this information to the challenges or work in front of us.    Shouldn’t the assessment methodology align with the requirements of the world we live in?   Now I will acknowledge the important of key foundational knowledge so I not suggesting we stop teaching any basic knowledge, but knowledge and memorisation should be less of a focus than it is now.

Conclusion

I believe technology could address a lot of the drawbacks listed above.  Now I note the use of technology will present its own challenges but how often do we find the “perfect” solution?    Wouldn’t a solution which is easier for schools to administer, is quicker and more efficient, is more student centred and more in line with the world we now live in be a good thing?

AI and assessment (Part 2)

Following on from my last post looking at AI and assessment (see here) where I focussed very much on the high stakes world of terminal exams and coursework, I would now like to look towards formative assessment and the learning process.   As with my last post, this post aims at sharing some of the points I made at a recent conference where I spoke on AI and Assessment, presenting some questions which I believe we need to increasingly consider in a world of AI and generative AI solutions.

AI Supported Learning

Learning platforms and computer based learning have existed for some time.   And they havent and dont look like the image here. I remember having to do some Maths learning during my teaching degree using a computer based learning platform and that was in the mid to late 90’s.    At the time I wasn’t that fond of these learning platforms and this feeling stayed with me.  My issue was that the platforms although offering differing routes through the broad content, were largely linear in their offering in relation to each topic or even the smaller units of learning.    This couldn’t compare to a teacher delivering content where they could see students struggling and then instantly seek to adjust the learning content accordingly.

We have came a long way from there, with AI and generative AI now able to provide us with far superior learning platforms with my sense being that these platforms tend to break into two types, one where the AI is analysing usage and interaction data to direction learning content creators and one, the more recent and emerging type, where generative AI provides an AI based support, teaching or coaching agent.

In the model where the platform analyses usage and interaction data the key benefit is that this data is gathered from all users looking for those common patterns or anomalies, looking at issues such as general, language, nationality, and a variety of other factors to find which learning content works and which does not.    This allows creation of effective learning content based on a huge amount of data across many schools and many learners, far beyond the data that a teacher may have at their hands.   As such the content in these platforms progressively improves over time and based on data rather than intuition or other less tangible factors, which may be wrong, which a teacher may rely on.

Where generative AI is used students get a chat bot which prompts and support students as they work through the learning content, with the AI trying to mirror the supportive and coaching role of a teacher, but individualised for each student and available any time, anywhere assuming access to a device and internet connection.    I feel it is here that there is the greatest potential especially in relation to more fundamental skills and knowledge development, freeing up teachers to focus on more advanced concepts and also on wider issues such as resiliency, leadership, interpersonal skills, wellbeing, etc.    I note recently reading a post about a school which uses AI where they don’t have “teachers” instead having “guides”.    I suspect this sounds more radical that it is in practice especially the reported comment by the co-founder that “we don’t have teachers”.   My view is that AI learning platforms wont replace teachers, however through the use of AI learning platforms working with teachers we may be able to achieve more and quicker with our students.   I suspect the school is more akin to this partnership that the report would suggest however have no first hand experience of the school so cannot be sure.

Challenges

AI as a tool to assist and maybe guide and deliver learning delivers a number of benefits however I think it is important to acknowledge some of the challenges and risks.  We may not have a solution at this point however at the very least we need to be aware.

Bias is a clear challenge and something which has been widely reported in relation to AI.    In my session I asked a generative AI solution for a picture of a nurse and a picture of a doctor which the solution returning images where the doctor images were all of males and the nurse images all of females, and where all the images where of white people.    This experiment clearly shows bias however the challenge in AI powered learning platforms is that the bias may not be so easily visible.   What if the platform decides based on statistics that students from particular area, nation, gender, preference, age or other characteristic do generally worse than average.   The platform may then present them content it believes to be appropriate to this ability level, in doing so impacting their ability to achieve, the challenge they receive, and possibly causing a self-fulfilling prophecy.   And when a parent asks regarding a students learning path, is it ethical to use learning platforms if the use of a learning platform means we may not be able to explain the decisions taken in the child’s learning experience and journey, where these decisions were taken by AI?

Data is another challenge we need to consider here in the possible huge and growing wealth of data learning platforms might gather in relation to students.   This isnt just the data a school might provide such as name, email and age, but the data produced through each and every interaction with the platform, plus the data gathered as diagnostic data such as the device being used, IP address, etc.    And then there is the data a platform might be able to infer from the data gathered;   Could an IP address, which suggests a rough geographic location, a device type and internet speed allow you to infer the wealth of a user or users family?     I suspect it could.    Now consider the massive amount of data gathered over time, across different curriculum subjects and each use of the platform;   The potential for inference grows with each additional data point.   How do we manage the risks here in relation to data protection, cyber risk and also accidental or purposeful mis-use of the data?  If we are to use AI assisted learning solutions I think we need to ensure we have considered how we might do this safely.

Conclusion

Educations has had its challenges for some time including teacher recruitment, teacher workload and wellbeing, and equity of access to education.   Maybe AI can help with some of this and maybe AI risks making things worse in some areas;  It is difficult to tell, although the one thing we can tell is that AI is here and here to stay so I think we need to make the most of it and shape its use to be as positive and powerful as it potentially can be.   A difficulty here however is the slow pace with which education changes (little has changed in almost 100yrs!).   Now the pandemic did cause some change in my view, but some of that has rubber banded back to pre-covid setups.   The question now is, is AI the next catalyst for education change, will it impact education as much or more than the pandemic and will its impact be persistent beyond the initial “shiny new thing” period.   Only time will tell although my sense is there is potential for AI to answer in the affirmative to all three questions.

References:

A Texas private school is using AI technology to teach core subjects; A. Garcia (Oct, 2023), CHRON, Texas private school replaces teachers with AI technology (chron.com)

Should AI be held to higher standards than humans?

Darren White posted an interesting question on twitter the other day in relation to the standards we hold AI to.    Should AI be held to higher standards than humans? This is something I have been given some thought to due to both having an interest in human heuristics and bias, plus an interest in artificial intelligence. 

Discussions on AI

There is already a lot of discussion regarding issues and challenges related to AI including discussion of bias and inaccuracy or “hallucinations”.    I myself have been able to recreate these two issues reasonably easily within generative AI solutions, firstly asking an image generation solution to create a picture of a nurse in a hospital setting and then a doctor in a hospital setting.   In this case the images were all of white individuals with the nurses all female and the doctors all male.    The evidence of bias was clear to see.    And in a separate experiment with a tool to help with report writing, the developer forgot to provide any data in relation to the fictitious student for which a report was being created but the tool simply made the report content up.    These issues are therefore clear to see and it is easy to jump to a standpoint where bias needs to be removed and inaccuracies or hallucinations stopped.

A human view

One of the issues here is that I believe we need to take a cold hard look at ourselves, at human beings and how we might respond to prompts if such prompts were direct at us rather than an AI.   Would we fair so much better than and AI?    I have a lovely poster in my office in relation to the cognitive biases which impact on human decision making and there has been plenty written about this and heuristics, with Daniel Kahneman’s book, Thinking, fast and slow, being one of my favourites.   A key issue here is that we are often not aware of the internal or “fast” bias which impacts on us and therefore may assess our biased decisions as being absent of bias.     In terms of hallucinations, again we humans suffer the same issue often stating facts based on memory, and holding to these facts even when presented with contradictory evidence.   The availability and confirmation biases may be at play here.    Another challenge when comparing with AI is that our biases and hallucinations are not clear for us to see, albeit they may be clear to others, yet with AI bias and hallucinations, at least in the form of those raised as examples, it is clear for all to see.  

End point?

I would suggest that in both AI and in human intelligence our ideal would be to remove bias and inaccuracy.   I would also suggest although this is a laudable aim it is also impossible.    As such, rather than focussing on the end we need to focus on the journey and how we might reduce the bias and reduce the inaccuracies both in humans and in AI.    It may be that in reducing bias in humans this may benefit AI, however it may also be possible that things work the other way and discoveries to help reduce bias in AI may help with bias in humans.   I note that a lot of human thinking, especially our fast thinking, can be reduced to heuristics or “generalisations” or “rules of thumb”;  How is this much different to the quick processing of an generative AI solution?  Does generative AIs probabilistic nature not tend towards quick creation of generalisations but based on huge data sets?

The future

So far, I have avoided getting pulled into the future and artificial general intelligence and I mention it for completeness only.   This will likely arrive in the future and most who claim to be AI experts seem to agree with this however there is much disagreement as to the when.   As such our immediate challenge is that of the generative AI we have now and its advancement over the creation of an AI solution capable of more generally out thinking us across different domains;  That said I would suggest that in a number of ways generative AI can already out perform us across many domains.

Conclusion

So back to the question in hand and whether we should seek to hold AI up to higher standards?    We should seek to avoid outcomes which have a negative impact on humankind so bias and inaccuracy and also the other challenges in relation to intelligence, such as equality of access to education, etc, are all things we should seek to reduce.    This I think is a common aim and can be applied to humans and AI.   In terms of the accepted standard, I think it is currently difficult to hold AI up to a higher standard than we hold humanity given the solutions are created by humans, trained on human supplied data and used by humans.   It may be that in AI solutions you get a glimpse of how entrenched some of our human biases actually are.   That said I also think it might be easier to remove bias and inaccuracies in an AI solution as compared to doing the same with a human;  I doubt the AI will seek to hold onto its position or to counter argue a view point, at least not yet.

AI and assessment (Part 1)

I recently spoke at an AI event for secondary schools in which one of the topics I spoke on related to AI and its impact on Assessment.   As such I thought I would share some of my thoughts, with this being the first of two blogs on the first of the sessions I delivered..

Exams

Exams, in the form of terminal GCSE and A-Level exams still form a fairly large part of our focus in schools.  We might talk about curriculum content and learning but at the end of the day, for students in Years 10,11, lower 6 and upper 6 the key thing is preparing them for their terminal exams as the results from these exams will determine the options available to students in the next stage of their educational journey.   The issue though is that these terminal exams have changed little.   I provided a photo of an exam being taken by students in 1940 and a similar exam in recent terms and there is little difference, other than one photo being black and white and the other being colour, between the photos.   The intervening period has seen the invention of DNA sequencing, the mobile phone, the internet and social media, and more recently the public access to generative AI but in terms of education and terminal exams little has changed.

One of the big challenges in terms of exams is scalability.  Any new solution needs to be scalable to exams taken in schools across the world.  Paper and pencil exams, sat by students across the world at the same time accommodates for this.  If we found life on Mars and wanted them to do a GCSE, we would simply need to translate the papers into Martian, stick the exams along with paper and pencils on a rocket and fire them to Mars.   But just as it is the way we have done things and the most easily scalable solution doesn’t make paper and pencil exams the best solutions.   But what is the alternative?

I think we need to acknowledge that a technology solution has to be introduced at some point and the key point is the scalability based on schools with differing resources.   As such we need a solution which can be delivered in schools with only 1 or 2 IT labs, rather than enough PCs to accommodate 200 students being examined at once as is the case with paper based exams.  So we need a solution which allows for students to sit the exams in groups, but without compromising the academic integrity of the exams where student share the questions they were presented with.    The solution, in my view is that of adaptive testing as used for ALIS and MIDYIS testing by the CEM.   Here students complete the test online but are presented different questions which adapt to students performance as they progress.   This means the testing experience is adapted to the student, rather than being a one size fits all as with paper exams.    This helps with keeping students motivated and within what CEM describe as the “learning zone”.   It also means as students receive different questions they can sit the exam at different times which solves the logistical issue of access to school devices.   Taken a step further it might allow for students to complete their exams when they are ready rather than on a date and time set for all students irrespective of their readiness.

AI also raises the question of our current limited pathways though education, with students doing GCSES and then A-Levels, BTecs or T-Levels and then onto university.    I believe there are 60 GCSE options available however most schools will offer only a fraction of this.    So what’s the alternative?    Well CalTech may provide a possible solution;  They require students to achieve calculus as an entry requirement yet lots of US schools don’t offer calculus possibly due to lack of staff or other reasons.   CalTechs solution to this has been to allow students to evidence their mastery of calculus through completion of an online Khan Academy programme.   What if we were more accepting of the online platforms as evidence of learning and subject mastery?   There is also the question of the size of the courses;   GCSEs and A-Levels and BTec quals are all 2 years long but why couldn’t we recognise smaller qualifications and thereby support more flexibility and personalisation in learning programmes?   In working life we might complete a short online course to develop a skill or piece of knowledge on a “just-in-time” basis so why couldn’t this work for schools and formal education?  The Open University already does this through micro credentials so there is evidence as to how it might work.   I suspect the main challenges here are logistical in terms of managing a larger number of courses from an exam board level, plus agreeing the equality between courses;   Is introductory calculus the same as digital number systems for example?

Coursework

Coursework is also a staple part of the current education system and summative assessment.    Ever since Generative AI made its bit entrance in terms of public accessibility we have worried about the cheating of students in relation to homework and coursework.    I suspect the challenge runs deeper as a key part of coursework is its originality or the fact that it is the students own work but what does that look like in a world of generative AI.    If a student has special educational needs and struggles to get started so uses ChatGPT to help start, but then adjusts and modifies the work over a period of time based on their own learning and views, is this the students own work?   And what about the student who does the work independently but then before submitting asks ChatGPT for feedback and advice, before adjusting the work and submitting;   Again, is this the students own work?  

There is a significant challenge in relation to originality of work and independent of AI this challenge has been growing.   As the speed of new content generation, in the form of blogs, YouTube videos, TikTok, etc, has increased year on year, plus as world populations continue to increase it become all the more difficult to be individual.  Consider being original in a room of 2 people compared with a room of 1000 people;    The more people and the more content, the more difficult it is to create something original.   So what does it really mean for a piece of work to be truly original or a students own work?

The challenge of originally and students own work relates to our choice of coursework as a proxy for learning;   It isnt necessarily the best method of measuring learning but it is convenient and scalable allowing for easy standardisation and moderation to ensure equality across schools all over the world.   It is easy to look at ten pieces of work and ensure they have been marked fairly and in a similar fashion;  Having been a moderator myself this was part of my job visited schools and carrying out moderation of coursework in relation to IT qualifications.   If however generative AI means that submitted content is no longer suitable to show student learning, maybe we need to look at the process students go through in creating their coursework.    This however has its own challenges in terms of how we would record our assessment of process and also how we would standardise or moderate this across schools.

Questions

I don’t have a solutions to the concerns or challenges I have outlined, however the purpose of my session was to stimulate some though and to pose some questions to consider.    The key questions I posed during the first part of my session were:

1. Do we need an annual series of terminal exams?
2. Does there need to be [such] a limited number of routes through formal education?
3. Why are courses 2+ years long?
4. Should we assess the process rather than product [in relation to coursework]?
5. How can we assess the process in an internationally scalable form?

These are all pretty broad questions however as we start to explore the impact of AI in education I think we need to look broadly to the future.    In terms of technology the future has a tendency to come upon us quickly due to quick technology advancement and change, while education tends to be slow to adapt and change.    The sooner we therefore seek to answer the broad questions or at least think about them the better.