Hello, and welcome to one of a series of podcasts exploring key issues or areas of interest in impact evaluation today. We hope you enjoy the podcase, and please don’t forget to tweet your thoughts at #impactframeworks. Thank you for listening. My name is Mark Taylor, I am Head of Strategic Partnerships at NIHR Central Commissioning Facility. I am here to talk about the history of the International School on Research Impact Assessment. And we will be hearing today from the three founders of ISRIA. If I can ask them to introduce themselves?
Hi, I am Jonathan Grant. I wear two hats nowadays. I am Professor of Public Policy at King’s College, London in the Policy Institute at King’s for half of my time. And for the other half of my time, I run a small company called Different Angles that focuses on the social purpose of universities and research.
Hello, my name is Paula Adam. I am from Spain, and I am the director of research at AQuAS. AQuAS, for those that don’t know, is an assessment agency of the government, depending on the department of health, and we do assess research programs, centres, and policies.
Hello, my name is Katheryn Graham. I am the Executive Director of Performance Management and Evaluation at Alberta Innovates. And I am based out of Edmonton, Alberta, Canada.
Thank you very much. Jonathan, could I start with you? Although it is an open conversation to everyone here in the discussion: What motivated you to set up the International School on Research Impact Assessment?
I have been thinking about this, Mark, since you asked us to do this podcast. To be honest, I can’t really remember. I do remember Paula very generously inviting me to visit AQuAS in Barcelona. Being a person who likes good food and good wine, I always say “yes” to invitations to go to Spain. And then over dinner. Paula sort of raised this idea: Can we do some training around research impact assessment? And she had also been speaking to Kathryn, who I also knew independently at that time. And this idea sort of slowly began to emerge. I don’t think it was one of those eureka things. It was just a series of conversations with friends over beer.
Always a good way to start things. Paula, what created that first discussion from your point of view?
Yes. Well, I do remember this magic moment. At that moment I was involved in two projects, one was research impact assessment, the other one was indicators for healthcare performance. And I was invited to a summer school in Venice in an island where I spent one week interacting with academicians and practitioners. And then, returning back home, I thought this interaction for these few days altogether, in a nice place, was perhaps something that was needed for the research impact assessment community. Which wasn’t really a community at that moment. So, that is how I thought I could talk to Jonathan and Kathryn, who were the two contacts I had at the moment. Two international contacts I had at the moment doing research impact assessment. And I must say there was no other contact locally in my country, so the isolation I felt trying to do research impact assessment was a motivation for trying to pull something together.
So, by definition, it had to be international because there was no one else? It had to be across countries?
Exactly.
So, Kathryn, what was impact assessment like before the International School was set up? What sort of training was available from your point of view?
I very much agree with Paula. I was feeling very similar to her. Very isolated. And it was very disparate. And there is really no one-stop shop where you could go either in terms of place, or in terms of [inaudible 00:04:23] that was comprehensive in terms of what the offering was. In terms of impact assessment. For me, Paula called me, it was a very seminal moment for me. Because I was really very – similar to Paula – quite lonely, actually, sitting in my little office. But really wanting the same aspirations and feeling the same principles of wanting the community to get together to move this forward. And as Jonathan said, I had also been working with him separately, and he was very generous with his time in terms of being very established in impact and advancing both the science and the practice.
So, overall, did anyone actually understand, outside a very small group of people, what impact actually meant?
I would say yes. And I think we, as a community, spend far too much time debating that question. Impact is dead simple. Do you improve people’s lives through research? Yes or no. And we can come up with fancy definitions and wordsmith definitions, but at the end of the day it is about improving lives. So, in my mind, yes, we have a very clear idea what impact is. I appreciate other people like to be more specific than my very, sort of flippant response. But I think if you keep clarity on this, is it actually about improving the lives of people outside academia? Then that clarity allows you to really understand what you are doing, why you are doing it, and how you do it.
So, there was a knowledge of what it meant, but there wasn’t a broad agreement on how people might get trained up or understand it from a practitioner point of view. Would that be a fair thing to say going back five or six years?
Yes. I think in one sense-, Kathryn and Paula can comment on this probably better than myself. I think it is fair to say that most large research funders would have one person who was involved in assessing the impact of the research they fund. And they may well be doing other stuff alongside that task. So, there was this lack of community and lack of network and peer support. But also, there was a risk of reinventing the wheel. And I think at that time, I was working around Europe, and so we would get commissioned to do evaluations for research funders, and we would find ourselves almost connecting people, saying, “Well actually, so-and-so in this fund over here did it this way. Would you like us to introduce you to them?” And I think one of the long-term benefits of ISRIA has been to create that community of individuals working in separate organisations – and indeed, in separate countries.
Very fair point. So, Paula, the first ever event was held in Barcelona. What were your hopes and expectations at that first event?
The hopes before doing it, were just the creation of new opportunities, connections, creating interaction and mutual learning between the attendees. The idea was really to create a learning experience. But I think we went beyond that, because – at least for me – it was the first time I understand what an interconnected community of practice was. And that is something that beforehand, I wasn’t really having this concept in my mind. But then I understood that this is what it was. So, I think-, also, the hope was-, there were very different regional [inaudible 00:08:33] around the world. Research impact development was more developed in some Anglo-Saxon countries, but in some other countries, it was still something that was completely unknown for many. So, another hope – at least for me – was spreading the culture of asking and questioning research impact in other international regions. For this we tried to also invite people from Latin America. We created some plans to have practitioners from Latin America. Also, we tried to get people from different parts of Europe and even Africa and the Middle East.
Did that work? Did you get the audience that you intended?
I think, Mark, I think the audience was great. I remember receiving, every morning, the list of new registrations, and being very excited because I think we managed to get a fairly good number of people. Not too big, not too small. With a variety of profiles and backgrounds. And also with variety in terms of experience in research impact assessment. But it appealed to the community of very much of mutual learning. I would like to believe that everybody learnt from this experience.
I remember. It was my first contact with the impact community per se. I remember going for a workshop and coming back with a community. So, I think it was a very powerful audience to interact with, and I think in that sense, it was an exceptionally powerful event. Kathryn, what about yourself? How did you find the audience in that first school?
For me, there was huge diversity in terms of representation internationally. I think we had representation across 14 to 17 different countries, Paula? If my memory serves me. So, the diversity was amazing internationally. And it probably spoke to this conversation, that people wanted to connect with other people in the community. It reflected that. For me, my role is very practitioner based. So, taking a lot of conceptual frameworks and implementing them and applying them in the field. So, as part of the course, we put together with real, with practical tools and resources. So, everyone commented on the weight of the binder. It was pounds and pounds, especially travelling in the plane. But really, you know, putting together that binder as a comprehensive reference for the community was really impactful for me. And as I said, literally the weight of it as well. And I think everyone there, with that first meeting, and as Jonathan said, a lot of it was very organic, and the connections that were forged at that very first meeting are still alive today. So, I agree with your comment, Mark.
Apart from the weight of the binders, and the fact that we all needed to be bodybuilders to pick them up and take them home, what were the lessons learnt from that first school?
I was just reflecting on Kathryn’s word there, ‘organic’. I think it is fair to say the first ISRIA was very organic. We wanted to embed the idea of being a learning community from the outset, so we rather rashly asked all participants to rate the school at the end of every day. The sessions from that day. And we used UNO cards. I don’t know if you know UNO cards, but they sort of have colour coding on them. So, red was bad, and green was good, and yellow was okay. So, we gave everybody a UNO card to stick on the wall. And on the first evening, about 80 percent of them – would it be fair to say, Paula? – were red. And we spent most of that evening rewriting the curriculum. And actually, with benefit of hindsight, whilst that moment was sort of heart in mouth, and an incredible amount of hard work, I think that community we brought together in Barcelona really strengthened the cause for subsequent episodes in the next four iterations of the school. But it was organic. And in one sense, that is what community building is all about, isn’t it?
Absolutely, I completely agree. Again, remembering that first school, watching it evolve over the number of days – not just the actual coursework itself, but how you, as founders, interacted with people on the course, which I thought was absolutely fantastic – the importance of community, I thought, was brought across very strongly. So, then it moved to Canada. Kathryn, what were you expecting as the conch was passed, the Olympic torch was passed, from Spain to Canada?
It was great because we had the experience of Barcelona. And we had lessons learnt. So, when it came to Banff, we refined the material, but really used the base of that and wanted to ensure that we kept the community feel. So, one of our challenges was we did want, and do want, smaller classes for our courses. Around 40 or 50. But the demand in Banff and Canada was so high it reached 95 and could have gone higher. But we had to cut it off. So, I think the experience was excellent. I was very happy with the Banff course in terms of both the material and the delivery as well, as with the sense of community.
And over time, some of the alumni have become faculty. Or did become faculty.
That is correct. And that was a part of the strategy, really. For the community to bring it forward and advance it. That is right.
So, after Canada, the International School on Research Impact Assessment moved around to Australia. Then Doha. And finally, in Denmark, before wrapping up in Denmark. What do you think the long-term impact of the International School has been?
I think one of the most important things in terms of long-term impact of the school is that currently, there are a number of attendees of these different five schools that are now working and leading research impact assessment in different organisations around the world. So, I think the long-term impact is the influence of these learnings into a number of organisations globally. But also the interconnection and this community of practice that was built at that moment. Personally, in my country, it definitely made a change. After the five ISRIAs, the five schools, and promoting different people from my country going to the schools, we have been able to promote an agenda of influencing local research policymakers, and preparing similar seminars oriented to the local audience. And also implementation.
So, what about yourselves, Kathryn, and Jonathan? What do you feel the long-term impact has been? Especially from a funder perspective, because an awful lot of funders have sent people to the International School in the five years it was going. Have you noticed any changes in the way funders perceive impact, or undertake impact assessment?
Yes. I think that is quite a difficult question to answer without doing a survey, if I may say so. So, my perceptions would be that, as Paula said, it has built up that community. I met lots of people who I didn’t know through those various courses and continue to interact with them on a relatively regular basis. That international network is really helpful as well, because quite often when you are doing this type of work, somebody will ask, “What do they do in such-and-such country?” And you suddenly realise you met somebody from that country a few years ago, and I wonder if they could help me out. So, I think that you cannot underestimate the power of that community and the soft network that is being developed as a consequence of ISRIA. I mean, there is a degree of professionalisation. At least I like to think there is a degree of professionalisation. I have also sadly – or whatever the right word is – been involved in measuring research impact since I took on my first ever job at the Wellcome Trust back in 1997. So, I have been doing it for a rather long time, and I think it was always seen as a bit of an amateurish activity. I think we have contributed to the professionalisation of research impact assessment, and I think that is really important because when you are assessing anything, if it is a good assessment, then decisions will be made based on that assessment, and it is important that those assessments are rigorous and fair. Because when decisions are made, there are usually winners and there are usually losers. So, I think that professionalisation is quite important as well.
Absolutely. I think transparency in all things, when it comes to impact assessment, is an extremely important thing. And I think that came out of the school over the time I was involved with it. Kathryn, any thoughts from you about whether you have noticed any changes from an academic, funder, or practitioner perspective?
As Jonathan knows, I like alliteration, so I am going to answer the question in terms of Ps. So, definitely in terms of people, in terms of the capacity, the number of people that were trained. In Canada, we continue on with the regional courses and actually will be doing one federally, all going well this year. So that has continued. So, capacity. But I would say one of the bigger shifts, that Jonathan referred to in terms of professionalisation of the ranks, is really the capabilities. Really preparing the competencies and skills of doing impact. In terms of the legacy, we do know from the participants, a lot of them got jobs with the title of impact. And advanced their careers. So, that is definitely great news. We did a lot of good work in terms of advancing standards and accreditation. So, in the area now you see much more white papers and manifestoes. ISRIA came out with the ten-point guideline, and I think that has really helped with the culture in terms of creating common standards.
I think that the ten-point guidelines are particularly powerful. And I think they have helped a lot of funders, the National Institute for Health Research specifically, when it comes to standard operating processes for impact assessment. But I am wondering, now that ISRIA has gone, were there any successors you are aware of that have emerged? What is ISRIA Mark Two? Is there an ISRIA Mark Two out there?
All the materials and resources of ISRIA are in a webpage and they are registered under Creative Commons, so they are available for anyone globally that would like to use them. As to whether they have been used or not, we don’t know how much. But we have some ideas of-, we are getting some feedback about it. So, the resources are in the webpage. There are also thematic areas that have been explored. For example, a group of people explored the gender dimension of research impact assessment and did a call for action to include gender in research impact assessment. Another group worked on what is the patient participation, what is the role of the patients in the area of biomedical research in terms of participating in, describing, and setting the agenda for research impact assessment. There are also a number of conferences called ‘In the Trenches’ that have been in different regions. As Kathryn said, there is the ten-point statement.
As I said, it has been very powerful. The community that has been produced has been quite successful in spreading the word, so to speak. But Kathryn, going back to your point about the accreditation and standards, who is going to uphold that in the future do you think? How are we going to ensure that we don’t slip back to the sloppy thinking and treating impact almost as a glorified audit process?
I see that advancing a lot across different areas. Sometimes different language is used, so I do see this as the great community intersect across some of those advances. So, we know with the universities there is a lot of work being done on societal impact and data-driven engaged scholarship. Also, across the world, we are looking at industry who are focusing on environmental, sexual, and [inaudible 00:24:23] models, who are pulling in investment ratings to ensure that impact. So, I do see the community as well as the broader community really advancing this. Sometimes, it may get a little confusing because everyone is coming at it, as I said, with different language. As Jonathan said earlier, really, ultimately, it is looking at the difference research innovation has made and citizens. So, I really see-, I see more work being done in terms of standards. And the other comment I see an opportunity is to work closer with performance audits. We have just undergone a performance audit and what was very interesting, and I think very promising, is a lot of the focus was in terms of processes around sharing lessons learnt. So, I think connecting up with the broader community in advancing these standards is a promising opportunity.
Thank you for that. Jonathan, if I could ask you a question. If you could go back in some form of time machine to before the first Barcelona event, is there anything you would have told yourself then that you might have done differently?
In one sense, no. Because I think what we got right was not, sort of, telling people what to do, but cocreating the curriculum as we went along. So, as I said earlier, I think all those people, especially at the Barcelona event, deserve credit for helping us strengthen and improve what the curriculum was. I guess my slight regret is that we can never continue to find a – if you like, for want of a better word – a business model to keep it going. Because the effective model was that it was massively subsidised by the host funder. And involved a lot of work by the three of us, and other people who were teaching on the course. So, I guess I might have spent some more time – if I had my life over again – thinking about how to sustain it in a financial sense. But even that said, I think the fact that we ran five courses in five countries and trained something like 400 people over five years, I wouldn’t sort of give up on that as a concept. I am deeply proud of that contribution that we have made. The impact of the impact, if you like. So, I am not sure there is anything I would do radically different. That may sound incredibly arrogant, but you know, I can’t really think of the answer to that.
Paula and Kathryn, but Paula first, any thoughts? That time machine? Anything you might have done differently yourself? Or a particular memory from that first event?
Well, more than saying what I would do differently, I prefer to say what I would surely do again, which is doing it with the same co-founders. I think doing this together with Jonathan Grant and Kathryn Graham was great. And in terms of lessons learnt, I learnt so much from them, so I would do it again with you. And what else? I wish, if we had to run it again, that it didn’t rain the first day. Because it was a pity that we couldn’t see Barcelona from the rooftop. The Pedrera and Gaudi house. But again, I think it was a great experience. I am sure, if we run it again, that it would turn out differently, because as Jonathan said, we co-created on the way. So, God knows how it would come out if we run it again.
Ah yes, the organic ISRIA. Kathryn, any final thoughts?
Yes. I would say it was a roller-coaster of a ride. And I would do it again in a heartbeat. I think the principles and that agnostic approach kept us solid. That was always clear to us. What I am missing, and if I had a magic wand, I think at the time, as I said, I was feeling very lonely. And now, what I am feeling, and I am assuming I am not alone, but in trying to advance both the science and the practice, and I will speak from the funders’ perspective, is really having a safe place to develop, experiment. And in terms of impact, really advance some of emerging development work. It is really challenging to find a safe place to fail. So, having the idea of an impact action lab extending out from the principles and the structures of ISRIA would be fabulous. If we had that or would have that going forward.
Thank you very much, Paula, Kathryn, and Jonathan, for all your thoughts and for joining us to create this podcast. But before we go, the resources that they have mentioned that are available, they are available at www.theinternationalschoolonria.com. There you will find the International School on Research Impact Assessment website with the history of the International School and all the resources that have been mentioned in this podcast.
Thank you for listening to this podcast. It is one of four in a series of exploring different aspects of impact culture. Please return to the website to discover the others. And don’t forget to tweet us your comments and questions to #impactframeworks. Once again, thank you for listening.
