Hello and welcome to this final podcast of impact frameworks and cultural change 2021. This podcast will reflect on the conference, how it all began and what we learnt. Please keep the conversation going on Twitter using #impactframeworks.
It’s fantastic that tonight I am joined by Jess Romo who was the originator of this conference with me after discussions in December 2019. And one of the organiser members, Shaun Leamon from the Health Foundation. Jess, do you remember the impact board meeting we had in December – it’s a long time ago when we first talked about this event?
It has been. But yeah, I do remember it very well. I think initially, Mark, we were both hoping these would be face-to-face in person events in London. And actually because of the pandemic it turned out to be easier online which I think actually worked really well because we went from potentially engaging a smaller number of people primarily based in the UK, to actually engaging a wide range of people across the globe. So I think that was an intended, but really positive affect, and I’m really happy to see people engage in our social media on both Twitter, Linked In, with a pre-recorded content available yesterday so it’s been amazing. What do you think?
I’ve loved it. I think you’re right. I think if we’d kept to the original schedule of holding it in October last year face-to-face, without a pandemic, I’m sure we would have got 50 or 60 people to come along. There would have been a nice conference dinner which is the one thing I think we all do miss. But we’ve had over 650 people joined the site today from the UK, Canada, America, across Europe, and in with the networking sessions, I had a fantastic conversation with a delegate from Kenya, that would not have happened without the pandemic. Sean, how have you found it on the international side?
Yeah, I would completely agree, there’s been people that have joined that just wouldn’t have been able to attend had we had an in person conference. Again people from all over the globe in the networking session that I was in. People internationally attending the sessions which I think is a real benefit. You may be pining a little bit for some of that in person contact during the networking, but I think all in all it’s been a great event in a great format.
I think the format has worked actually. I’m not quite sure whether either of you had any chance to listen to some of the podcasts that were made and the vodcasts, but the vodcasts went down very well. Jess I can say that yours is particularly popular, you got loads of views. Shaun you too. And some of the Tweets we’ve got over the pat 24 hours have recognised some of the learning from those podcasts themselves. Did you enjoy making your vodcasts?
Shaun, do you want to go first.
You can take the silence from that as you will. Yeah it’s interesting. I’m not entirely sure I’m ever cut out to be a newsreader but it’s interesting to share thoughts on the conference and your hopes and aspirations which is always nice to let people know in advance, and also carry on those conversations. So from that perspective yeah certainly.
I feel that we’re all flexing different muscles during the pandemic and especially with online conferences. Picking up new IT skills and learning to use a wide range of online platforms which is a welcome addition to our skillset.
I think so, I didn’t realise that Zoom had a webinar system until this conference came along. And the power of the webinar system I think has been proven over the past day. In fact from starting this morning, because Jess as you say, it’s become an international conference, we got Tamika Hide to join organising a video from Research Impact Academia in Australia which meant we started at 2pm Australian eastern standard time, but at 3am this morning which meant a very early start, 2.30am for a couple of colleagues. And nerves obviously, because at 2.30am if something goes wrong who do you call? But it was fantastic and I think Tamika’s early showing went down extremely well. Followed by my breakfast networking which was very bizarre, seven or eight people joined with virtual cups of tea and virtual bacon sandwiches, which was a lovely session before Shaun I think you were on at half eight?
Indeed, yes the early session for us based in the UK and it was great to have Annette Boaz from London School of Tropical Medicine, and James Wilsdon from Sheffield University, to join me for that early session. So thanks again to them. And some really interesting points put across particularly about Annette reflecting on her experience of working as part of the government office of science in the UK government. And her reflections on actually – a really positive experience and a lot of learning to take forward, but also a real willingness to engage between both the policy side, but also the research community, to help that policy research interaction. And then James posed, I believe it was five very interesting paradoxes of the use of research and evidence and particularly as we move through pre, but also the post pandemic period and how we maybe reframe some of our thinking around the way. Evidence is collected, shared, and used, but also the types of knowledge that people are looking to bring to some of those policy discussions.
I think James is a fascinating individual by the way because he also does a lot in the research and research field, with the Wellcome Trust if I understand. So he’s a big Wellcome Trust type person. And of course after half eight we had the impact framework session, the original idea anyway for this conference. Jess, how did you find your time?
Yeah it was really exciting and we had a nice panel. So I was joined by Anne Rivers from NIHR and Dr Alison Tweed as well as Dr Grace Sweeney from NHS England and Improvement directorate. And I had a (unclear 07:07) on the Wellcome Trust impact framework. And yeah I think the participants really engaged with the content, there were really good questions. I suppose what I will highlight at the moment is a couple of things that came up in the session that were also echoed somewhere else. And there is something about the implementation of frameworks as well but more specifically. I think there’s a very strong call for systems thinking and the use of system based approaches to allow us to better deal with complex long systemic change. And also the use of theoretical change, logic models or anything else that starts conversations by clarifying what is it that we want to achieve, what kind of impact change. And figuring out what is the best way to contribute to that change. A case of studies came up particularly as a way to deal with long term systemic change to kind of deal with quantitative and qualitative data in ways that create stories that people can really engage with. And really help communicate things that otherwise is very difficult to do, particularly through visuals as well. Links are kind of useful in terms of methodologies. Journal impact as a measure of quality of research was one of those markers you remark in one of your podcasts on public and (unclear 08:45) engagement using numbers to evidence outcomes, is not good enough, such as for example, just counting the members of the public participating in research. We really need to go beyond, so I think everyone was saying similar things, we really need to go beyond just (unclear 09:02) and finding better ways to engage with systems, so yeah I stay with that.
It was definitely a powerful session, I’ve tried to watch as much as possible today while doing some of the impact conference elves work in the background, so to speak. But it was very enjoyable. Equally to see the learning across the three organisations represented. The Wellcome Trust has a framework, NHS England has a framework, NIHR are developing one as we speak. And the overlap of understanding I thought, was extremely powerful. Shaun did you get a chance to join that session, or were you busy preparing for your second session of the day, the great debate at midday?
No I did get to join the session, again really interesting interview and to hear the different organisations how they, approaching developing the impact framework and health foundations, like a lot of organisations, looking at how we can improve the way we go about trying to capture the impact of our own work. So it was really interesting to hear how other organisations have been going through that journey. The great debate at high noon, GMT was asked – another interesting one for me. I remember when we had the conversation back, it would have been last year, about a policy session and I suggested a moral maze type sort of format. And yeah really pleased with that. And I think all credit to Jonathan, Ian, and Jude for joining that session when we came up with the title. I was a little bit apprehension about whether we would get everyone to join in, particularly on a proposers side. But fair play, that Ian was more than happy to be involved in that. And again the same with Jude and Jonathan chairing. And I think what was really encouraging is that there was an open friendly honest debate around whether we should be continuing to have dedicated impact sections within the funding application. Interesting to see the very small server before and after that we did, that Jude managed to hold onto the no’s that we shouldn’t be removing it, although the gap was significantly reduced, which is an interesting finding. And I think the key things from that, and you can’t argue with a lot of – Ian’s point about the need to reduce the burden in the application process for some people. There were questions about just moving it from one part of the form to the other, how does that reduce the burden? But I think, as a funder ourselves, we would need to recognise that probably more needs to be done to support people at that application stage. But it’s also interesting to hear about how it’s really important to support those that are actually reviewing the applications, to really support a meaningful assessment. And friendly challenge to the application so that the people that are putting in the application can actually see the most useful feedback that we’re confident that it’s the best proposal it can be. And also the importance of bringing in those wider audiences to that. So I know most funders now have patient service users and people’s lived experience, on their panels, but how we can actually bring that into a more meaningful way, ensure everyone involved in that process is fully on board with that. I think really useful and some really insightful conversations came out of that session.
I think it’s a fascinating point, it’s not just the application form, who actually reviews the application form. And I think does need to be taken into account. I think we have a BBC news …
Exactly, one of those moments. For me, Shaun, it could be my 18 year old coming in at minute, so I think yours is cuter. But I’d suggest it is an important thing because it’s not just about the application form, it’s the system overall. And it’s intriguing that you mentioned about support for people who actually look at application forms, and actually analyse them. Because there’s a piece of work that we’ve just started recently, looking at a prototype tool kit to help peer reviewers, when it comes to impact assessment itself, whether that peer reviewer is one of the great and the good academics or a patient or service user, which we’re doing with Julie Bailey from the university of Lincoln, there’s a bit of work that started and should be ready in a few months’ time to actually look at it. Because that’s exactly the point, it’s not one bit of the system, it’s the entire system, holistically I think you have to look at. I don’t know whether you have any thoughts, Jess, about impact sections and application forms?
Yeah, I joined the session and I really liked how thought provoking, and I like how controversial it was. I think it was great, and I think what I really found useful it’s almost better to ask questions like why do we want to know about impact from up front, and what is the best way to do that. And therefore the discussion around whether you have them in the application form and who uses it, for what? So I think they were all valid points. I suppose that’s something that we discuss in the networking sessions that follow from that debate, was again maybe they’re rolled out, that kind of information has, not necessarily as assessment for criteria in terms of deciding funding. But to which external – that type of commentary can be used by funders to figure out how we can continue supporting those researchers after their grant for example. Particularly your funding basic research and are asking possibly to go through translational research and then hitting the ground. I think just having those discussions up front, and again maybe turning the tables and saying, actually this is something that funders need to be more aware of and use for plugging in gaps and closing those gaps across the research pathway. For me it’s a powerful thought as well at the back of that discussion. Hopefully that’s making sense.
No, it does, I think all these thoughts seem to be coming together over the days’ events, because after the great debate we had a session of cultural change run by Paula Adam from AQuAS and I thought that was fascinating. I had to dip in and out occasionally of that, but some of the stuff I heard, I think Jordi Molas-Gallart was quoting Machiavelli at one stage which is most unusual from my point of view for an impact conference. But it was absolutely fascinating about the cultural change. One would expect or hope from impact assessment, or even having an impact system itself, and who it should be for, and I found that a very powerful session, and obviously why are we doing it in that sense. Again did either of you guys make that session? I’m assuming you did, but it’s been a long day.
I did, I was able to – I don’t know if you did, Shaun, do you want to start?
Yeah, I caught some of it and sort of probably towards the end of the sessions. And I think it was an interesting point that was made about the cultural value because that also links to some of the conversations I was having later on about the cultural value and measuring cultural impact and someone I was talking to was referring to the centre in Leeds. They’re looking at how we can measure the broader cultural change and cultural impact. And how we look at some of that wider, dare I say, the sort of, the more intangibles that often come out of some of the work that don’t neatly fit into some of the matrix that often are required and asked for by funders as well as other organisations. And so the importance of thinking about that as well I think was a really important point and an interesting one.
Yeah, definitely, Jess?
I think the only thing that I will add is, again a challenging thought was made that was really calling for a change, encourage a research system and really encouraging researchers to go beyond (unclear 18:03) and research, to more widely share the work, explain the value of it, and again I’m sure that it ultimately leads to impact but also there was quite a lot of talk about again the role that the public has in the engagement and whether it is really playing a meaningful role or how can we improve that. So those two things really stay with me from that session as well.
Derek Stewart is a fantastic speaker by the way, I’ve heard him before, I’ve always been very moved by his points about public engagement and patient engagement in research, about the fact that there’s a danger of it becoming business as usual and that not enough time is consistently spent about why we do it in the first place, the ethos of engagement and I think that came through very well in that particular session as well. And then we moved to Gert Balling from Novo Nordisk Foundation’s session in the fantastically titled, Lessons Learnt session, you know when things go wrong, which again I thought was a very brave way of approaching this particular issue. Because no one likes to admit they’ve been wrong. And I thought his session did touch on it quite nicely. Sadly, I had to dip out of that one more than any of the others, so I’m sort of relying, Jess and Shaun, on any recollections you may have had of Gert’s session if you had time yourself, because as I said it has been a long day.
People are going to think that I did nothing.
It’s a conference, you’ve got to go the webinars.
Yeah, no again I had the leisure to join the session and I think they did a really good job, at kind of articulating very similar messages. And I think one of them was just that it’s so difficult to decide and implement impact frameworks. And really requires a level of change management across the whole organisation, but also externally including your (unclear 20:08) like the (unclear 20:10) holders and the wider research community so that was very strong. And again just all the things that I think they all mentioned were particularly around avoiding a top down approach only. And really ensuring that you take time and have patience and take people on a journey which means that you need to do a lot of raising awareness and explaining what this is about, and not particularly because most people will tend to fear these type of impact frameworks and work, there’s a fear of being evaluated and what that might mean for your career, for the programmes, so really engaging, communicating and building trustworthy relationships was key in terms of all the panellists’ words of wisdom. And particularly when it comes to a stakeholder engagement, the importance of being transparent. Again particularly with external audiences. Yeah, over to you, Shaun, if you have any other thoughts.
Unfortunately, or fortunately, I was on networking during Gert’s session.
You were, you were doing the two networking sessions. But there is a connection actually, Shaun, between Jess’ observations and the Health Foundation. The over reliance on top down which Jess mentioned. The Health Foundation about three or four years ago published a really nice paper about why innovations don’t get adopted back into the National Health Services in England. And one of which was the reliance, over over reliance of the top down push and old fashion dissemination. So I think Jess’ observations are very apposite considering the work the Health Foundation’s done in this particular area a while back. I think we then moved on to the final webinar which is still, or may have only just finished, with Kathryn Graham which I had to nip out of so that we could do this podcast, partly because if we had to wait until that finished, having been up since 2.30 in the morning, I think this particular podcast host would be falling into his keyboard right now. But I left as David Sweeney was giving an extremely interesting historical review of impact in the sense of where he started in the 90s and almost this passing the ball back from one to another between Australia and the UK on impact assessment. But equally ironically going back to Tamika’s first session at 3am GMT, where there’s this passing the baby comment came out. That impact is all about passing the baby from one stage to another which is a comment that Kathryn Graham used to use and ironically she is doing a final session as we speak, so the baby has been passed all the way through to Kathryn. And again I don’t know whether you caught any of the first half hour before joining me on this podcast?
I managed to catch just a little bit of it, but I mean that reflection’s really saying it as well, Mark, because it was one of the things that I opened a session with first thing this morning is that this is a conversation that’s been going on for years indeed decades, about how we make better use of research knowledge and evidence to support policy. And I think just picking up your point there about some of the previous work that the Health Foundation’s done about being able to take what we know works in one area and support the transfer of knowledge and evidence to another area. It is really important and as everyone knows context often trumps everything, and so being able to look more deeply about how you can share impact, knowledge, and evidence so that you can continue that building of impact and change from maybe that more micro or media level up to that more sort of macro policy level I think’s a really important thing and a conversation that importantly will continue for probably some time to go.
Thank you for that. Guys it’s been a long day, a lot’s been discussed in different webinars and networking sessions. But what about your overall impressions of Lessons Learnt. Can I ask you first of all, your thoughts on this?
Oh god, I think we’ve done a good job at summarising in some of the main things that have percolated throughout this session. I guess things that we haven’t yet covered that might be pertinent to flag now, is that there’s still not a lot of consensus on what research impact is, and what we mean by outcomes. It kept coming back and back throughout the day. But actually once you start listening to what people suggest as answers to this question, there’s a lot of similarities and basically it’s something that we’re now describing as going beyond the academic and research sectors, having policy and health practise change. And it’s ultimately about improving health and people’s lives. So I think we do have a working kind of a finish for research impact and I suspect there’s probably a lot more convergence done we perhaps feel at any given point. I’d just like to say something else that is more about – I’m kind of looking forwards I suppose and into the future past Covid. I feel that all of this conference particularly the context in which we’re living at the moment, has really demonstrated that we can shorten time to impact, as long as we have very clear outcome oriented discussion in mind and that’s how we start working together, linking, and mobilising the diverse stakeholders across the systems that we need to make health impact. Particularly moving from the clinical research sector for example we’ve seen a lot of (unclear 26:21) partnerships that weren’t there before and have been crucial to the balance of research. But also securing private sector partnerships, again to kind of make sure that there’s not a gap, we’re stopped in the pipeline from research to production. But also early engagement with policy stakeholders and sustaining that relationship to encourage as much as we can the evidence based decision making. And last, but not least, I feel that there’s a real effort to translate science and research to lay audiences. Proactively addressing this information and finding ways to motivate behavioural change and maximum uptake. So I think all of these are very crucial ingredients that I hope we continue to use in all the areas in the research and health systems. And that we can continue great and transform large scale change at reduce timelines.
Excellent summary there, Jess. Shaun, anything you would add from your perspective of Lessons Learnt over the past couple of days?
Yeah, I would echo everything that Jess said there. I think things additionally to add into that is, and perhaps preaching to the converted, is just that the importance of giving it the space and the resource and sort of really recognising that the impact agenda isn’t a nice to have, it’s a specialist area, it requires some skill and some support and that needs to be built in and factored into the work rather than just on top of the day job that other people are doing, or the research projects that they’re managing. And I think probably the other thing to add to that, the reflections over the last couple of days, is just the sort of trans disciplinary learning that can be had. So the learning that health and care research can offer to other sectors, but also the learning that the other sectors education for example could offer to the health and care as a sector in the impact agenda there. They would be the other things that I would add onto the top of that.
Yeah, I think that’s true. The networking sessions that have been held today which worked incredibly well. I wasn’t quite sure whether just an open Zoom call would work, but those small groups, the discussions that occurred, and the fact that I met people who have nothing to do with health and social care, but they are interested in impact, and the fact there were things to learn across these I think is very powerful. And something and about the health and social care impact into that, we to make sure we are not inward looking. It’s a big area, but we could do double speak if we’re not careful, I thought the networking sessions were very powerful to remind us that there’s more than just us. I think that was very, very powerful to hear. And personally I liked the mix, the comment, the structure of what we’ve done in the past couple of days. The live webinars, the informal networking. The videos you guys pre recorded, which did work, had an effect, people enjoyed them, people were commenting about them, and the podcasts themselves. I think that meant that there was something for everyone as cliched as that sounds. But I do think it’s been an extremely interesting couple of days and I look forward to finishing off a legacy website. So all the webinars, and all the stuff we’ve learnt can be available for people for a few more months at least to download. I don’t know if there are any final comments. We’ve been talking for half an hour and it seems like this might be a good time to wrap up the podcast. Jess, you first, any final thoughts or comments before we turn in for a decent weekend one hops?
I like to finish by giving you a huge thanks, Mark, to you and your team, particularly Urvashi Sharma and Alice Duboin, you guys have done an amazing job of having this plan implemented throughout the conference. And honestly we couldn’t have done it without you so a huge thanks in appreciation and the help of the whole organising committee, that’s really what I wanted to say.
That’s incredibly kind.
I can say no more.
All I can say actually that the organiser committee were a fantastic bunch of people to work with, so obviously you, Jess, you Shaun, and to Tamikaand Paula and Kathryn and Gert, it’s been a pleasure. Once we nailed down the format and we began to therefore talk about what we were going to put into it, it’s been a blast. And it’s been a complete pleasure. I think everyone knows I’m stepping down as head of impact for NIHR to move onto a different position within the organisation. And this is definitely my last lap and it’s down to you guys, it’s been a fabulous last lap. So before the mutual appreciation slightly goes over the top, Jess and Shaun thank you very much for joining me for this final conference round up. I hope everyone who’s listened to this has enjoyed this podcast and also everything else that this wonderful organising committee has put together over the past two days. And that the conversations continue on Twitter with #impactframeworks. And that you will enjoy looking at the recorded webinars again, fingers cross on the IT that hopefully will be available end of March, beginning of April on the legacy website, with the same web address. www.impactframeworks.info.
Thank you for listening to this podcast, we hope you enjoyed it and all the other content produced and provided by NIHR, the Welcome Trust, the Health Foundation, Aquas, Novo Nordisk Foundation and the Research Impact Academy. Keep checking the website. Impactframeworks.info and the #impactframework on Twitter for all news and updates.
