• Were a Training Manager from 1960 to take a trip on a time machine to the present day they would emerge to a scarcely recognisable world. Learner-centred training, user-generated content and on-line delivery would all be things of wonder. However, they could draw comfort from evaluation methodologies: the Kirkpatrick framework, first enunciated in the 1950s, is still the cornerstone of training evaluation.

    Why this is so is something of a puzzle, as it is criticised by both academics for not being based on any reliable model of causality linking learning and performance and by training professionals for not providing a useful set of practical tools for doing an evaluation. For me the biggest problem with relying on Kirkpatrick is that it asks me to work backwards: I seemed to be looking at changes in behaviour and workplace impact, treating the process of training as a black box and then making guesses as to how these changes were caused by the training.

    So I started to look for an alternative, and have started to develop a combined application of systems thinking methodologies and theory-based evaluation, or TBE. TBE has apparently been primarily used to evaluate social care initiatives, and does not seem to have been widely adopted within the training community which is a shame, because I have found it to be extremely useful, although given the way in which much training seems to be conceived, it is not without its challenges.

    TBE works by examining the theory of change underpinning an action, and exploring how its implementation works. It then becomes possible to draw conclusions as to the degree to which what is being done contributes to subsequent changes. Reflection on any differences between theory and practice helps to identify what is or is not working.

    The first challenge is often identifying what the theory of change is for delivering a training programme. This should come from a training needs analysis, but in reality TNAs are often not done in any systematic way. Instead, an assumption is often made that training is necessary, and little thought may be given to the operational context where the learning will be implemented. This can mean that the way in which the training is delivered is not necessarily appropriate or that obstacles stopping people from using new knowledge and skills are not taken into consideration.

    If there is no theory of change from a TNA, we need to think through what this might be. At this point systems thinking tools can help, in particular one called Soft Systems Methodology, or SSM. I don’t have the space here to describe the full SSM process, but a key output is a diagram showing a ‘conceptual model’, which represents an ideal situation, in our case how the training is supposed to work. The diagram below shows what a conceptual model for a typical training programme might look like.

     

    The conceptual model now represents our theory of change, and this is what we use as the basis for the evaluation. We can look at each stage in the model and use these to reflect on the actual situation. For example, the first stage in the whole process of training design and implementation is step 1, to set requirements for the learning system: obvious questions to ask here are whether these requirements exist, how realistic they are, who decided what they were, and so on. By working through each step in our model and asking these questions we can build up a holistic picture of how well the whole training process is working (when we come to steps 9 and 10), and how each stage contributes to the overall effectiveness of the learning system.

    Theory-based evaluation places a great deal of importance on the word just used, ‘contribution’. Trying to use Kirkpatrick as an evaluation tool works the other way, and we find ourselves trying to attribute training to observed changes. Contribution works the other way, and helps us to ask questions about how each stage enables the next, and how the overall model contributes to changes in behaviour and improvements in organisational performance. And it’s important to remember that we can never be 100% confident about causation: all we can ever say is that it seems likely that doing X has led to Y.

    The links with the Kirkpatrick framework are in steps 4 (Level 2), 6 (Level 3) and 9 (Level 4). There is no direct use of Level 1, Kirkpatrick’s ‘reaction’ level: there is very little evidence in research literature to suggest any direct link between liking training and subsequent learning. However, during an actual learning event it is important to gather information from learners about how useful they think the learning will be and to find out about what obstacles there may be in transferring this learning. This feeds into step 5, where we find out about how new knowledge and skill is being used. Assessing the so-called ‘learning transfer climate’ is a major omission in Kirkpatrick-based approaches to evaluation.

    While step 4 assesses whether people are learning, step 6 assesses if they are learning the right things, and so helps us to modify the content of the training as necessary. Step 9 helps us to see whether or not the overall concept of the training is right: should the training be delivered in another way or is training an answer at all?

    Since I have started to use theory-based evaluation and systems thinking approaches in training evaluation projects I have found them to be extremely useful in helping me to make connections between the delivery of training and subsequent workplace changes. By working forwards from the training to the performance rather than backwards from the performance, which is what a simple application of the Kirkpatrick framework implies that we should do, it is much easier to conduct evaluations which really show the value of a training programme.

    The explanation of TBE and systems thinking here has been necessarily short, but if anyone is interested they are welcome to look at my website for more information: .

    Bryan Hopkins is an independent consultant who has worked in training for nearly 30 years. He specialises in providing training needs analysis, design, delivery and evaluation services to international organisations working in the development and humanitarian sectors.

    Bryan has written a number of books on different aspects of learning and development, and his latest book, entitled "Learning and Development: A Systemic Model for Analysing Needs and Evaluating Training" will be published by Routledge in late 2016.

    He lives in Sheffield with his family, a cat and a number of bicycles.

Comments (1 comment)

Rich Parker, Training In Aid

"Thanks Bryan, as ever, for leading the way on systems-based training approaches in the humanitarian sector. Your adaptations of the various tools (especially soft systems methodology) have been extremely useful in the various evaluation projects our organisation has been involved with this year. Whilst myself and others are perhaps less radical in our criticism of Kirkpatrick-style approaches (or perhaps this is simply a willingness to accept the dominate paradigm of our times?), we've certainly witnessed the added value that a systems approach can bring - even if only blended alongside other evaluation tools. As the proverb goes: "All models are inherently wrong, but some can be quite useful if applied at the right moment...." "

12:00 - Friday October 28th, 2016

Add Your Comment

Type your message in the box below and click send. [Messages that include web links will be rejected]

 

 

 [we will not display this]

Related articles

  • Is Social Media Learning?

    If you spend time looking at what your friends and family are doing on Facebook it’ll be hard to call that a defined learning event and the same goes for reviewing updates on Twitter or Instagram.However, if you use a platform such as ‘Tweetdeck’, or the like, you can organise your Twitter Followers into specific lists and then you can use this to look up specific trends. If you are reading...

  • Employee Engagement

    Earlier this month we help a CPD seminar on the topic ‘Employee Engagement’ at the Training Foundation’s offices in Coventry. This month I am going to use this blog to discuss the main themes and points.Our First speaker was Wendy Dean from Strategi HR.She discussed some important points such as “Give your Employee Engagement Programme a name” – this allows employees to get behind the...

  • BILD CPD Events 2016

    Your CPD (Continuing Professional Development) is important as an L&D professional.Which is why the British Institute for Learning & Development hold 6 affordable CPD events per year. Alongside that we also hold CPD webinars.Our next CPD event is 25th April in Woking – The subject matter is Mindfulness.Our full Programme is listed below, click the links to book your place.25th April 2016 –...

  • Investment in Learning & Development

    Investing in learning and development is a sore subject for many. When the question is posed it’s often related to how much is spent on learning and development (or training) of employees.But how much invested in the L&D team?As L&D professionals we ensure that others are encouraged to develop and are also encouraged to keep up to date with CPD. However, how much CPD or L&D have L&D...

  • What makes TAP Training the Defacto Training provider of choice?

    Last week I had the pleasure of being able to attend The Training Foundation’s TAP Certificate in Training Delivery.The Training Foundation delivered the process in three distinct phases – Style, Structure and Activity.Without giving away too much of the Training Foundation’s ‘trade secrets’, over the 5 day course we learnt all about the Queen’s Award for Innovation – winning...

  • Demonstrate Quality In L&D

    How do you demonstrate the quality of your learning and development?If you are working as part of an L&D team, you might be asked this question. You can point to the qualifications of your team, you might have even found a way to measure the all illusive return on investment… but as an L&D team it is hard to benchmark yourself with similar companies or competitors.And if you are learning...

  • Most Read Social Media Posts of 2015

    As the year draws to a close we get the usual emails and social media posts about the top 1o events to follow in 2016 or the top 5 developments from 2015.We have to jump on that bandwagon… But with a slight twist.Our Social Media output is geared very much toward a curator of news stories so we thought we’d put together the top ten most popular news items from our Social Media.In reverse...

  • Is the Role of the Company Trainer Dead?

    Once apon a time there was a company trainer, his name was Johnny and his job was to make sure that all the employees of ACME Engineering company got all the training that the company decided that they needed.As it was an engineering company and Johnny used to be an engineer; his job was relatively easy, in fact, Johnny still considered himself an engineer, just one who’d become so skilled in...

  • Shiny new object

    We’re all familiar with the shiny object, something that people tell you about to get you to stop doing what you are doing and pick up something else. Often its worse in business as there are no shortage of critics telling you that you should look at this shiny new object or this one. The modern world is full of shiny new objects and learning and development sees its fair share of them....