"How do you know when real change has occurred?" This is a question our Chief Strategy Officer, Colin Sloman, has asked many times. Colin shares his thoughts on using A/B Testing and data to deliver meaningful results that track change in large organizations.
In my first two decades as a change management consultant, there was one question I dreaded my clients asking: "How can we ensure that we achieve this change you are proposing?" In other words, would it work? And how would they know whether real change had occurred? For many of those early years of my career, I employed the 'hit-and-hope' method, which involved doing all the comms and training, workshops, and engagement activities I thought were necessary, but without really knowing whether that would work or be enough. Also, effective measurement was hard to design. Often there was a lack of willingness to invest in longitudinal data-gathering and analytics.
So what, if anything, has changed? In 2016, when I first met the Cognician Team, their approach to applying behavioral science and nudges was gaining traction. I'd been following the rise of behavioral economics and watching The Behavioural Insights Team significantly influence the UK government via the Cabinet Office Nudge Unit. It made good sense to apply neuroscience and behavioral science to organizational change, but how could we use this thinking in our existing change management methods? How could 'hit-and-hope' become a scientifically based change process to ensure that business change landed and could be measured?
My first experience using the Cognician Activation Framework was for newly hired consultants at a global consulting firm. We knew that the existing face-to-face onboarding process was okay – and we got decent feedback – but we believed that we could do more. So we involved Cognician and designed a multi-day challenge (MDC) program to run alongside our existing onboarding process. Borrowing from design thinking methods, we created our first A/B test. And, after a few short months, we were able to measure time to proficiency, billable hours, and line manager effort with specific behavior measures and a financial business case. The results of the A/B test were evident: they included a Brandon Hall Gold Award and tens of millions of dollars of benefits compared to the traditional approach. The onboarding MDC – with some regular updates – has been running ever since, and thousands of new joiners have benefited from adopting the new behaviors and mindsets that they are 'nudged' into in those first few weeks.
At Cognician, we love A/B (and in some cases C!) testing. Since joining Cognician as CSO, I've seen the benefits of activating change using these methods firsthand. In a recent project for a banking client, we split the audience into three cohorts. Cohort A's agile learning process was supported by a standard Cognician MDC. Cohort B received the same MDC, but with enhanced behavioral drivers. Cohort C went through their program, but without Cognician support. The results revealed that group B's contributions were of a consistently high standard when compared with the other two groups. Taking these findings forward, we use what we've learned and apply the approaches we know will have the most impact. We can also test different types of challenges to test their effectiveness and different engagement strategies to optimize the impact of a change project.
Recently, we have enhanced our A/B testing in two new ways to measure change impact in multi-day challenge change programs.
First, we've introduced the Activation Index (AI), which measures whether your target population has fully engaged with the change outcomes you've set. We look at three key behavior change drivers which, taken together, indicate how much of the target population has positively taken action regarding the change. We measure intent (have they accepted the change challenge and scheduled a time by which to complete it?), action (did they act on the challenge?) and reflection (did they actively reflect on the challenge?).
We've also introduced the Commitment Index (CI), which represents the target population's investment in adopting the desired activation outcomes. CI is measured by using natural language processing (NLP) to analyze participants' written insights to understand their specific intent. We look at five categories to assess their commitment throughout the program: social, physical, intentional, cognitive, and emotional investment. With these new measurement indices, we also offer benchmarks as part of our Standard Activation Reports to see how your program is doing compared to equivalent change projects.
At Cognician, we are committed to delivering mindset and behavior change in every one of our programs. We pride ourselves on creating programs underpinned by over 30 years of behavioral science research. We have codified this learning into our 8 Activation Catalysts and 48 Activation Tactics which we have published here: Activation Tactics Cards, and we continue to innovate using continuous A/B testing. Our new Activation Reports, which focus on participant activation and commitment, will give you the confidence that your change is actually happening (Activation Index) and the level of buy-in to the new ways of working (Commitment Index).
Let Cognician be your proof positive that real change has happened and that behavior and mindsets are changing too. If you would like to find out more about our AI and CI measures, please get in touch here.
