Hello and welcome to one of a short series of podcasts which will investigate aspects of impact assessment from four different perspectives, or four difference lenses. Please keep the conversation going by tweeting any thoughts with the #impactframeworks. Thank you for listening.
I’m Sana Zakaria and I work for NIHR. Today I’m talking to Kathryn Graham about her views on the use of impact assessment for accountability from her personal and professional perspective. Kathryn is the Executive Director for Performance Management and Evaluation at Alberta Innovates and has a significant amount of experience in the field of innovation and evaluation. Taxpayers, donors of charities, funders, they all want to understand whether their investment decisions are realising the benefits that they had anticipated, so the accountability agenda has never been more crucial in light of restricted funds and the competitive nature of research funding. And the first question I’d like to ask Kathryn is that researchers often feel that they’re at the brunt of the responsibility in answering the evidence requirements around the accountability agenda. Do you think this is true and do you think this is how it should be?
Yes, I think that’s a very good question and I think particularly in light of the conference we’re having; I think definitely yes, I think it’s been true particularly in the past. But I think we are seeing changes and I think part of those changes is reflective in the conference that’s about to happen in terms of really trying to understand, have a deeper understanding of impact and also really trying to perfect the change for wanting to change. But I do think they have had a lot of the brunt and no, I don’t think that’s the way it should be. I think researchers as well as funders and other key stakeholders, government, academia, etc., are accountable to the public. Particularly when we’re talking about public Dollars being used responsibly and efficiently and effectively. But I also agree from a funder’s perspective that the purpose of accountability impact is crucial, particularly as you said in the context of reduced funding and increased competition for research funding. So from an impact and a funder perspective the purpose of accountability is to show that money has been used efficiently and effectively and in the past really focusing on having researchers to account. But I really believe that it’s not only researchers who need to be held to account, the other key stakeholders and the research eco system. I think that given the demand particularly in the use of the public Dollars for this demonstrating public values, researchers more than ever are still going to be asked to demonstrate, report, and communicate their impact. I think one of the reasons I think researchers have beared the brunt from generating evidence for accountability, is the challenge that accountability in measuring impact is difficult. So there is many challenging in accessing data, being able to integrate impact evidence across the research lifecycle. And I think there has been an over reliance on self reported data and so I think that’s fallen somewhat on the research community. We have a lot of impact reporting tools, a lot of them are self report, and as an eco system, we really need to think about how we can design the data collection across the research lifecycle. As we also know with impact, we have many contributors in the eco system that are helping in terms of contributing to that impact. So I think the accountability requirements won’t go away but I think more than the researchers are accountable. However, I think it’s incumbent on researchers to embrace impact, have the ability to evidence their impact, tell their own narrative and have the ability to not only report on it but communicate it. I think particularly as a funder, that we want researchers to really focus on the research and not over burden them with the administration and red tape. They’re not the only player in the eco system when it comes to impact but we want to make sure that we have representation and from other players in the eco system. And also we understand that impact is cumulative over time and really supporting researchers in building that skill. I think researchers skill set is excellent when it comes to impact and really helping support them in evidencing their impact and help communicating them. I think it’s very important.
So you mentioned quite a lot about reducing administrative burden, Kathryn, and you talked about trialling new approaches, innovative methodologies. Now you’ve already alluded to some of this but considering the very stakeholders and their needs around accountability, what approaches do you think can be taken that fulfil that accountability agenda, and ticks multiple boxes whilst reducing the administrative burden. Because we tend to try and manage multiple stakeholder’s needs at the same time. But it doesn’t always quite pan out that way.
Totally agreed. I think ultimately having a system’s view for impact is ultimately the solution and that that will tick many of the boxes. As I said impact affects many players, many in the eco system. So that system’s view, I think, is very important to really achieve the impact and not only achieve impact but really ensuring that we optimise impact in the change we’re trying to affect. And this kind of system’s view does require a long time view with the ability also to communicate and report on monitoring information short term, but it is very engaged, a strategic long term view, that is collaborative and really focused in terms of challenges and society, and addressing the needs.
Do you think at all that there’s a mechanism to address the cultural challenges, because quite often government stakeholders or funders might want to see very different kind of an impact assessment take place compared to researchers and the kind of matrix, if you will, that researchers might value, would be very different from what a government funder might value? And how do you get around that problem?
So I think particularly when we’re talking about impact, we have different stakeholders perspectives and stakeholders can value different things and as a result of valuing different things, they measure different things. So I think historically one of the challenges has been the lack of alignment in terms of stakeholders measures incentives around impact and they haven’t been aligned. And I think this has added also to the burden, so I think taking a system’s view with diverging stakeholders view and perspectives and values, is part of how we need to change points or word. And getting more alignment in terms of what the desired impact is. And doing that upfront which can be very challenging but building the system and that acknowledges that upfront and really articulating the desire change, the desired impact up front, as part of any large initiative, is kind of part of the systems’ view. And also I think the change that does need to happen as part of the system’s view is aligning investments very much accordingly, different kind of road maps. So at Alberta Innovates we’ve been working on a more system’s view towards accountability and impact. We’ve been refining it with a performance impact management system called PIM and really trying to accommodate a much more holistic and a system’s view where, you’re right, the stakeholders do have diverging values, and value different impacts and measures, and putting that within the framework. And in my experience there’s two ways to address sometimes when there’s divergence. One is getting the group stakeholders together to reach consensus on what is the targeted or intended impact. And then sometimes allowing more flexibility where you’re taking a more comprehensive view and you’re designing for measures that meet different stakeholder needs for their organisations. So I think both can be accommodated but really taking this system’s view right across the research lifecycle, right from the idea stage right to applied research and also if appropriate, to commercialisation. And so building in, and designing up front, what is needed I think in the long term is the most efficient and effective way to start thinking about impact. And really it does start with what is the demand or the challenge, what is the need, aligning the strategic plan, the investment framework and thinking of it right across the cycle. And then we use kind of the house that impact built, the graphical figure, for these different components that go into the impact system but essentially in the basement of the house is the idea of assessing and evaluating and monitoring, so we have that evidence all along. So this idea is using the evidence right across the cycle with many feedback groups and adjusting and adapting along the way. So then I think this is part of the cultural change that will be talked about in terms of the conference in how do you do this, how do you implement these kind of systems, and get engagement across a diverse group of stakeholders. So really it’s a living organism and shows that the evidence can be in real time, it’s informing decisions and there’s a feedback group for adopting and modifying and adjusting as you go. So very much in terms of a continuous performance improvement cycle. So that’s our leader’s thinking in the challenge particularly of the burden and costs but really trying to focus more systematically as well of impact.
What national or organisational level enablers are needed to facilitate these impact assessments for researchers that meet these multiple accountability needs of government and decision makers?
So in terms of platforms there are many at many levels in terms of the organisation at a different country level and so in England for example there’s the research excellence framework which is pretty much a policy framework and that’s at a national level. In Canada we have a more bottom up approach of impact, in terms of kind of engage with the community in moving impact forward, so different (unclear 15:43). But I think as a community of professionals in terms of researchers and impact professionals, I think we really need a common impact platform. And so what I mean by that is sometimes we talk a lot about the challenges and impact which are very real, but in practise there is a lot of agreement in practise, on how to measure impact, how to address it, and we still need to continue and advance that. But really acknowledging the challenges up front and working with the community with this idea of a common impact platform that is sharing of information in terms of the philosophy of open innovation. Somewhat similar to expanding on the international spool and research impact assessment website, but this common impact platform I think would have four legs, or four pillars in terms of the platform it would focus, in terms of policy advancing the science of science, and this would be harnessing and curating I suppose, academic evidence that can help informing policies and practises. Also really preparing the trainees of tomorrow, establishing impact fellowships. We have such a thing in Canada with the Institute of Health Services policy research so having that more internationally, more professional development academic programmes in terms of impact. And also the ability to really up the game in terms of doing more experiment or pilots or testing for learning and sharing data. So the policy in advancing the science of science would be pillar. Another pillar would be in terms of people capacity building and outreach, and I think the conference is one example of this so including outreach events and sharing lessons learned and how people are implementing in terms of impact, professionalising the area, but also really having professional training development courses as well. And so the third pillar I feel would be in terms of practise and this is having an implementation tool box to advance the practise. And really again, I think we’ll hear a lot about this at the conference but having a pot of practical tools and resources practitioners, should actually advance the practise of impact in the field, so that would be the third pillar. And then the fourth pillar would be in terms of impact performance and the common part really is agreeing on common standards and accreditation and the idea there would be this kind of continuous performance improvement. So as a community establishes consensus around quality of standards, having an accreditation body, in terms of for organisations to help them, in terms of passing these kind of orders and accreditations and also really professionalising impact professionals as well, in terms of more competencies. So one example Alberta Innovates is undergoing a performance audit as part of the auditor general here in Alberta, and I think it’s a really good practise and process for organisations to go through that and really reflect on impact that’s been achieved in terms of investment but really looking at value generation and or processes, so very practical ways of doing self assessments. And as part of undergoing that process we will really take that on I think as an organisation in terms of doing self assessment audits in terms of impact. But what I particularly like about the process is really a lot of emphasis in terms of the learning and the sharing of the learning in terms of what’s worked and what hasn’t worked.
Brilliant, that sounds like a really comprehensive framework that you’ve just outlined and I think the pillars will really resonate with our listeners. I especially like the point around professionalising the professionals because I think that’s a definite need in this sector, especially in the UK as well. It’s been wonderful listening to you, Kathryn, it’s been really, really insightful and I’m sure everybody listening in has found it as insightful as I have. Are there any last words you wish to say to people?
I would say in my experience that I’ve been in the impact space for the last 13 years, in strategic evaluation for over 25 years, but as a community I’ve been very touched by the generosity of the research and impact community in they’ve been very open and welcoming. I’ve been very impressed with the calibre of the people in our community in really wanting to go deeper in understanding not only the what if impact, but the how, and the why. And really challenging ourselves to really help achieve the change we’re trying to change, and using research as one of the core enablers for that. So I think my last word is that I think it’s a fantastic group of people trying to make a difference but if from a human perspective, and a personal perspective, a great bunch of people, a great community to be part of, and I’m just really excited about the upcoming conferences, having lively discussions and debates for really moving the science and the practise forward.
Thank you for listening to this podcast. It’s one of four in a series exploring different impact lenses, please return to the website to discover the others, and don’t forget to Tweet us your comments and questions at #impactframeworks. And once again, thank you for listening.
