Hello and welcome to one of a short series of podcasts which will investigate aspects of impact assessment from four different perspectives, or four difference lenses. Please keep the conversation going by tweeting any thoughts with the #impactframeworks. Thank you for listening.
I’m Sana Zakaria, and I work for NIHR. Today I’m talking to Thomas Christensen about his views on the use of impact assessment for analysis from his personal and professional perspective. Thomas is senior vice president of the Novo Nordisk Foundation and has been working with impact assessment for many, many years. So we’re talking about analysis as one of the underpinning impact assessments, and analysis as curiosity helps us in understanding what’s working, what isn’t and forms the basis of information the other aids like advocacy and allocation. The Novo Nordisk Foundation states that the purpose of the impact assessment is to monitor the foundation’s strategic goals, to enable fact-based communication to society and create new knowledge of the impact of research funding. How easy is it to identify which impact assessment to do when you fund close to four billion Krone a year in projects?
I would say it’s really difficult. I wouldn’t say it’s easy. It needs to take a long term perspective on this because you don’t do impact assessment in one year because you have awarded grants for four billion Danish grants, which is approximately 500 million pounds. It’s necessary to have a strategy and to prioritise what is important for the impact assessment work in the foundation, and as a guiding principle we have focused on our vision of the foundation. And the vision of the foundation is to contribute significantly to research and development that improves the life of people and sustainability of society. And based on this vision we had tried to formulate nine impact principles for how the Novo Nordisk Foundation contributes to society, and based on these nine principles we have identified indicators which is important for the foundation and which say something about our societal impact. Our ambition is always to employ the best practise to analyse and assess the impact of research and development in society. And in order to do that we are collaborating with the researchers at universities with other foundations and we are also having a very close dialogue to our key stakeholders. The key stakeholders are mainly internally the grant giving departments in the foundation, but also the grant recipients. So we have a close dialogue to all the grant recipients and our ambition is that we have 100 percent response rate from all our grant recipients when we ask them to report once a year to the foundation. It is actually the big challenge of all impact assessment to collect data and to analyse this data.
I guess the vast majority of your funding goes to biomedical health science research and allocations. The learning from how to do impact assessments for analysis in that area translate to some of the more niche areas that you fund. So we talked about obviously at a very strategic level, but things like patients end of care, social humanitarian, and development aid areas in which you actually fund work. How do you conduct impact assessments at that level?
We, as you mentioned, have had historical focus on medical science and the treatment of patients. This has given us a lot of information about what we should expect from our societal impact. But when we set up our impact assessment department five years ago, we actually did not have any data. We had some ideas about our societal impact. So we had been in a learning process since we established the impact assessment department. So I would not say that we had founded the perfect model, but we have paved the way, at the same time we have been driving. So we have learnt by doing. Having said that, of course we have tried to, as I mentioned, collaborate with researchers who have a long term knowledge on how to do impact assessment. So we had awarded some grants to researchers so they can do impact studies on the impact of research in health and diseases. But at the same time we have collected our own data and formulated the logic model for how we expect to have a societal impact. And this logic model follows the theory of change, what could be expected from when you have input of investment into health research, what could then be expected in terms of activities, in terms of output, outcomes and long term societal impact. So this theory of change is a key instrument for us because based on this theory of change and our logic model, we have indicators for all our activities and that could be for individual grants, but it could also be for a research area. This KPI system has been developed in close collaboration with our experts in health science. So they have helped us to identify what are the most important indicators for measurement of the societal impact of investing in research in health area. So I think this is something where we this year have been able to get an agreement with our key stakeholders, what is important and what is not important when you measure investments in health research. So that has been a long process, a learning process, but also in the impact department we have people who have experience in social science so they have an additional interest in looking beyond what is usually expected to be the societal impact of investing in health research. So we are able to look well beyond what is normally expected among health scientists and this is a big advantage because then we can identify impact in politics, impact in economic area, impact in education and many other impacts which I am not fore seeing in the grant awarding activity of the foundation.
It sounds like you have worked in a very collaborative way with the research community to inform these KPIs that you were talking about. Now does that mean that there is a very bespoke approach taken to various funding streams and assessing their impact, and if so, do you ever find that you need to compare analysis across different disciplines to understand what works overall for the foundation when it comes to making investment decisions for instance?
We can learn much from benchmarking the different research areas because if we look for instance in natural science one can expect different societal impact compared to health science and again one could expect also different societal impact when we looked at humanities or social science compared to natural science and health science. So this gives a good benchmark for what to look for and what not to look for when we try to measure the societal impact in a certain scientific area. But we really respect the societal impact in different research areas is, and should be, very different, so we do not compare the outcome in a certain area for instance in natural science of investing 100 million Danish Krones, we do not compare with this with the outcome for instance in social science or in health research. Simply because the impact of research are different in different scientific fields, and also the costs of research are different in different scientific fields. We have an ambition in the foundation to donate money to all scientific fields so what we are looking at is actually more different instruments within a certain field. So for instance we look at smaller research projects, is that better in health science than large research programmes in health science, or research centres in health science. So we can look across the different instruments within the same scientific field but we do not look so much across different scientific fields, here we focus more on getting inspiration for what we could learn in other fields.
You mentioned international benchmarking and some of the research centres. Now one of the many areas the foundation invests in is the standard diabetes centres. Now the grant is designed to improve treatment for diabetics and then move to an international benchmark but also increase research activity and support the construction work. So when you look at such a varied bunch of outputs and outcomes, and when you’re reviewing the impact of such funding, where do you start with that?
When we invest in big research infrastructure or in research centres then you must understand that Denmark is a small country, so if we need to get an idea of what is the impact of the big investments we are doing in research centres, then we do not have benchmarks inside in our own country. So we need to look beyond our borders and to have an idea of what is best practise in the world, where do we have research centres or research units or research hospitals which could be used as a benchmark for our activities. So the international benchmark is quite important for us to understand how well our initiatives are doing. Also since we want to build up world class initiatives in Denmark and also help our research hospitals to perform world class treatment of the patients, we need to have an idea of what do we mean by world class and in this connection we also need to have some international benchmark. Normally we do not define these international benchmarks ourselves, we do it in close collaboration with our key stakeholders. That could be our grant recipients, it could also be research teams at universities in Denmark or in Scandinavia. So this is very important for us to be (unclear 13:27) to the knowledge which are created elsewhere to better understand what is the added value of the things we are doing. With respect to the (unclear 13:39) centres, ambition has been to build up new high technology research hospitals so we are funding buildings, but we also fund research activities, we fund treatments of patients, so treatments should be one of the best in the world, and then we fund education of the health professionals. So in all these four areas, we also collect data and monitor the development and since we’re talking about very new initiatives then we are in the very beginning of our monitoring, so the focus has been on establishing what I would call a baseline for monitoring the progress of our initiatives. So the baseline discussion has been very important because that is so to speak, a control group for the initiative so we can measure the added value in the Danish society, against this control group, and the control group can be international benchmarking but it can also be benchmarking against other initiatives in Denmark.
That’s been really insightful, Thomas, listening to you about all of these topics. Are there any last words you wish to say to our listeners on analysis around impact assessments?
I only want to say that impact assessment is a learning by doing process. But the key issue will be to collect data and continue to collect data from your grant recipients and from other sources and combining this data and improving the data base you have in the foundation. Because without data you will never be able to conduct impact assessment. I know that for some foundations or for some people it is hard to report on data or to collect data in a systematic way, but that is necessary if you would like to take impact assessment seriously. So I can only recommend to start collecting data and then also to accept this is a long term investment that you need to collect data for many years in order to get a proper impact assessment.
Very wise words. Thank you so much for your time, and we hope everyone tuning in has found this as insightful as I have.
Thank you for listening to this podcast. It’s one of four in a series exploring different impact lenses, please return to the website to discover the others, and don’t forget to Tweet us your comments and questions at #impactframeworks. And once again, thank you for listening.
