Hello and welcome to one of a series of podcasts exploring key issues or areas of interest in impact evaluation today. We hope you enjoy the podcast and please don’t forget to tweet your thoughts at #impactframeworks. Thank you for listening.
Mark: My name is Mark Taylor. I’m Head of Strategic Partnerships at NIHR CCF and I’m joined today by Dr Danny Kingsley, Scholarly Communication Consultant and Visiting Fellow at the Australian National Centre for the Public Awareness of Science and Helen Bingham, who is head of Knowledge Services and Technology Enhanced Learning at Health Education England. During this podcast, we’re going to discuss the importance of libraries and impact assessments and the use of grey literature, that is literature including from government departments and agencies, civil society and non-governmental organisations, academic centres and departments and private companies and consultants, which is not produced or published through traditional commercial or academic routes, but which can be central to impact assessments and case studies. Danny, it’s been great to read the article, the paper you’ve just had published, the Impact Opportunity for Academic Libraries through Grey Literature and that’s really what led to this podcast that we’re doing today. I wanted to ask you what are the main barriers for libraries becoming more involved with research as a whole, but more specifically, research impact?
Danny: Yes, this is interesting. I think that one of the things that really defines an academic library as different to other types of library is that academic libraries are involved with the production of research as well as the consumption of it, and that is unique. Usually, libraries are focused on consumption and I think that the concept of a library in many people’s minds, including with the research community, is a quiet place where there’s lots of books and so people do associate libraries with consumption. The reason why I think that the research impact has an opportunity for research libraries is because over the last 20 years or so, as we’ve moved into open access and more scholarly communication research support across the research lifecycle, the libraries have started changing what they’re doing or adding to their offerings into support of that publication and production process and I feel that the impact aspect of it is part of that broader lifecycle. When we come to what are the barriers, I think the barriers tend to be perception of what a library is and does and so there is a bit of a marketing gap, I think, that libraries haven’t jumped into where they really need to sell themselves more as partners in research rather than servants to research.
Mark: Helen, from your point of view within the National Health Service in the UK, does that resonate with you? Do you find that there are similar barriers ?
Helen: Yes, to some extent. I would say that NHS library knowledge services are also involved in the production of research, albeit at a smaller scale possibly than their colleagues in academic libraries. Quite often, there may be a recognition from organisations that those libraries exist and exist to support the research cycle and often, there is no formal recognition through funding. NHS libraries, the focus is very much education, clinical care, management decision-making but research support is very much what they do day in day out and often they have really strong practical working relationships with the practitioners who are involved in research. Those links are all in place, but I think I would agree with you, Danny, that in the NHS as well at strategic level and organisational level there might not be that recognition and certainly there isn’t the funding flowing through, so there’s not necessarily an expectation that this is a key role for library services.
Danny: Of course, the NHS does do research. There is a distinction with a university library in that there is both the research and the teaching aspect and there is a sort of structural barrier to the involvement in research in that in almost all cases, libraries report into the Deputy Vice-Chancellor or Pro Vice-Chancellor for academic which is the teaching side of the university rather than the Pro Vice-Chancellor or Deputy Vice-Chancellor for research. There is this structural assumption around the primary role of the library that is based on the student side rather than the researcher side of the community.
Helen: Yes, so there’s a direct parallel, isn’t there because most funding for NHS library services flows through education, but also there’s clearly within the NHS a very strong focus on clinical care, management decision-making as well and research is we have to work harder to make those links and get that recognition that’s part of the role and remit.
Mark: I think one of the issues also would be from my time on the coalface, is that libraries within the NHS within medical schools are neither one thing nor the other in that particular sense. They are used obviously by staff for, Helen, as you say, for clinical reasons but equally by clinical researchers who inhabit the same space and have the same buildings, which does therefore lead on to the question I think we’re talking about both perspectives of whether funders could play a larger role in encouraging institutions to utilise libraries to their full potential. What do you guys think about whether that’s possible that the funders could play a larger role here?
Danny: Certainly funders have caused the libraries to really step up, particularly in the UK, in relation to being involved in the research endeavour because of the policies around open access and requirements to make research outputs openly available. I held a position at Cambridge University Libraries, which was responsible for meeting some of those policy requirements and when that position was created, I was answering to both the librarian and the head of the research office because the requirement was meeting policies which is normally in the purvey of the research office, but it involved dealing with artefacts which is something the library is used to doing, so it was really straddling both of those areas. What the funders require in their policies again makes a big difference about what the library’s involvement is and so if the funders are saying we’re going to give you lots of money to pay for articles to be made open access, that’s one thing, but if they say you need to collect this material and make it available in a repository, that’s a distinctive action and activity the library can do that no one else can do really in the university. Whereas if you’re just managing a fund to pay for open access, that could be managed out of the research office. The policy determines what needs to happen and whether then that is best placed within an institution. I think Helen’s response will be quite different because the funding dynamic is different in the NHS.
Mark: Helen, we’re all ears?
Helen: Your question could funders play a larger role in encouraging institutions to utilise libraries? I would say yes for sure, in the case of the NHS and that actually if research funders could have a role in ensuring that NHS organisations take a bigger role here, if only a small proportion of the funding that is made available for research across the NHS, the billions that is made available for research, could flow to libraries, they would be better placed to support the whole research cycle. At the moment, it’s quite patchy. Larger NHS organisations which may be affiliated to a university will be much better placed to support their researchers and would-be researchers in their organisation than those in other sorts of NHS settings. Because the funding tends to flow to individuals rather than to employers via block grants, then the kind of opportunity to ensure that some of that funding flows to the support infrastructure is often lost in the NHS.
Danny: There’s a very distinct difference between what was allowed to be done with money that was provided through block grants from Research Councils UK, which is now UK Research and Innovation, and the charities open access fund which included the Wellcome Trust. The RCUK funds, you were allowed to use part of those funds to pay for staffing and for infrastructure if it helped your compliance with the requirements to make work open and accessible. That was not the case with the CoF funds. Those funds were purely only able to be used for the payment of article processing charges. When this was brought up at a meeting, this was now several years ago, at the Wellcome Trust, the Wellcome Trust made the comment that they were already providing significant funds to the institutions for the purpose of research and that the institutions should then use some of those funds for the payment of staff for this purpose. I don’t know if that was a naïve comment not realising the way funds are actually distributed within an institution or whether it was just a steer, but certainly when research funds are provided into an institution, mostly they do not find their way towards places like the library to support this sort of area. It just doesn’t happen. Yes, you were absolutely right, Helen, in terms of allowing some of that funding to be used for this purpose makes an enormous difference to whether or not services can actually be provided because otherwise the institution will need to actually make other funds available, and I know from personal experience that extracting those funds from institutions is extraordinarily challenging.
Mark: If we go back to actually the first question that the perceptions of what libraries can and can’t do, we come back to that again as being a barrier for libraries becoming more involved. That in many ways people don’t perceive them in that sense that even when funds are available or when funds are available for research, there is a not a particular stream to support libraries when it comes to being a repository of information for monitoring and management of research outputs. There’s a stream here as far as I can see during this conversation about barrier after barrier. Would that be right?
Danny: There is a lack of understanding, I feel, amongst the academic community of the broader understanding and knowledge of scholarly communication as an entity in its own right. As something that is itself an object of research of knowledge of expertise and skill. I describe this as I know teaching; I’ve been to school. Academics feel that because they some practical experience of the scholarly publishing system because they are publishing. They may be doing peer review. They may be doing some editing, that they, therefore, have the knowledge about it. This meta broader understanding and so they don’t necessarily see that there is that broader understanding at all, and even if they do, they don’t necessarily think that that knowledge and understanding would be held within the library community because librarians aren’t researchers, so what would they know? It’s very difficult to get across to the research community that the people who work in this space are highly, highly trained and have an enormous level of expertise. I know that I’ve had arguments with the university administration over the years where there seems to be this sort of perception that you can just get another staff member from somebody working on the check-out at Tesco. Give them half an hour’s training and they’ll be okay when it’s really there’s a six-month opportunity loss when you’re bringing in a new staff member in this sort of area because of the complexities associated with much of it. There are barriers there about not even really comprehending how complex and skilled this area is.
Mark: Under those circumstances, who should be responsible for ensuring that all outputs, including grey literature, are managed within a repository? It’s not going to be the researchers themselves if the unconscious bias is, as you say, Danny, I know about teaching because I went to school.
Danny: Well, there’s a couple of questions in there because it’s who should do it, but also even should it be done? When I first started working in this area which was now many, many years ago, I was putting together and rebooting the repository of the university I was working for and so we were looking at policies around what could go into that repository and initially, the university was saying well we only want things that can be claimed for, the equivalent of ERA. ERA is Excellence in Research for Australia and REF is the Research Excellence Framework in the UK. Things that could be claimed in those sorts of processes which are all, of course, already in the public domain in some way. They might be behind a subscription, but they are published. We won that argument that we wanted a broader range of material in the repository, and this is where we start talking about grey literature because that is where the value of the repository really comes to the fore because this is material that would otherwise just sit either in a drawer or sit on departmental websites as things like working papers and when they do sit on departmental websites as working papers, they usually sit there with breakable URLs. They’re not properly indexed. You can’t actually find them through a search engine and so on and so forth. Even quite organised grey literature that is valued by the research community is poorly managed by it. Getting that material in is a sort of like a no-brainer, but then you start moving into wider of grey literature and that obviously thesis are considered to be grey literature, but it might be blog posts, or it could be an article written for a professional magazine that is related to your work but is discarded and not thought of as valuable by the academic community because it doesn’t count for promotion. Those are the things that we should be capturing because they are the discourse around the research process and at Cambridge University there’s much excitement and work done around the Darwin manuscripts. There’s a digitisation programme that is pulling together, it’s been going for many, many years, pulling together all of his manuscripts that he had written and collected over the years that led to his seminal works. Now the Darwin manuscripts of today is this grey literature. It’s the discussions that go on in blogs and in social media. It might be email correspondence. It could be various conference proceedings that aren’t formally then published and so on and we’re kind of letting that disappear and we really should be capturing it into repositories. I think this is kind of trust in technology that it’s alright, Google will look after it and that’s not a well-directed trust, I don’t think for the long-term.
Mark: Well, no, because by definition that would suggest no curation and I think anyone who’s used any form of search engine knows how unreliable they can be, I guess. Helen, what are your thoughts about the responsibility for all of this?
Helen: Okay, well, I do think that employers, certainly in the NHS, have a role in that they kind of shouldn’t be giving out money or giving people time off in order to participate in quality improvement project or service evaluation or research projects without committing to capturing at least a minimum dataset into a repository but I think that this is an area where, in the lack of funding or other formal recognition, NHS library services have already stepped into the space because grey literature is so important in healthcare and perhaps in contrast to the academic sector, it’s relatively much better valued. Structured investigations, quality improvement initiatives, practitioner research, local guidelines. The near research elements, if you will, poster presentations and that kind of thing and so, when NHS library staff have got involved in institutional repositories, an increasing number of them are, then quite often it is around that non-traditional research publications. They are proactive in that area because almost by definition, it’s more important to those who are working close the action, as it were, in healthcare settings and that type of research is more accessible to a larger proportion of healthcare workforce than those who are really clinical academic researchers affiliated to universities who might be very focused on ensuring that their publication gets published in a high impact journal and into a kind of high-profile repository.
Mark: So there are cultural acceptance issues around grey literature, I think, is what I’m hearing here. I think for anyone listening, a definition of grey literature might be information produced on all levels of government, academia, business and industry in electronic and print formats not controlled by commercial publishing, i.e. where publishing is not the primary activity of the producing body. I think it’s interesting hearing from both of you that there would be acceptance depending on the status of the grey literature, so Darwin, that’s one thing, blogs is something else, but also the culture that might be using it. Within healthcare, it’s always been that way, but outside there might be some sniffiness around the use of grey literature at all, let alone within impact assessment itself. Can I ask, over and above that very broad sweep, what would be the barriers around making grey literature be perceived as worthy of being included in library repositories in the first place?
Danny: I’m thinking of what the objections were when we were having those conversations all those years ago about what was allowed to be put into the repository and one of the arguments for keeping it within the material that we would be putting forward for the assessment process, was that there was a concern that if people were allowed to put whatever they wanted into the repository, then it might cause a reputational problem for the university if people put rubbish in. I felt that that was a spurious argument because the reputational risk was probably far closer to the actual author who was responsible for providing the material than to the institution. We had a working group where we tried to define what scholarly work was. There was this idea that we would only have scholarly work in the repository and so we had to talk about what was scholarly work and because we had a cross-section of research areas there, running from things like musical manuscripts through to areas where a big part of their output is newspaper articles because it’s important for them to get messages out into the general public and so we ended up with a definition that a scholarly work was something that a scholar thought was a scholarly work, which is sort of a non-definition in some ways. It defies defining and what looks like a scholarly work to one person is not to the other. Part of the issue is that there are very strong disciplinary differences. Even the concept of a conference is very different depending on the discipline. In some disciplines, a conference is ‘the’ place that you publish all your fancy work. Whereas in others, a conference is a nice weekend away on a desert island where you might have two hours of meetings and the rest of the time, you’re just drinking Piña Coladas or whatever. It becomes very difficult to define these things on an institutional level because if you’re an institution that covers multiple disciplines, then you’ve that academic tribes and territories issue about not understanding where somebody else is coming from. It gets very tricky. Universities are tricky places. The word university implies universal, when really, it’s just a sort of bunch of fiefdoms. A bunch of egos. I don’t know. I don’t know what the best way of describing a university is.
Mark: Fantastic. Fiefdoms, I think, is a very accurate description in so many ways. Helen, what about your thoughts about the barriers for making grey literature being perceived as worthy? I mean within the NHS it seems like that’s less of an issue. Are there times when it is?
Helen: Well, yes. I mean, there still is for many clinicians wishing to progress their careers there is a strong drive to publish in high impact journals. That has to be said, so I wouldn’t want to say that the culture elitism, academic kudos thing isn’t an issue because, in some areas, it is but probably less so in the NHS. There is a little bit of reticence for sharing. Danny, you mentioned there about reticence, reputational risk of an institution being seen to be affiliated to research that’s not that great and in the NHS, some of the reticence comes around sharing research that might not have proved successful or even might have exposed some flaws in patient safety as a for instance. It’s that kind of reputational risk that can sometimes come into play, but I think on a very practical basis, barriers include the fact that this type of content is more difficult to capture in the first place. That there can be metadata and repository limitations. You kind of have to push and squeeze to make some of these types of resource fit into repositories and then also be surfaced in a helpful way. When you look at the sorts of search discovery services that we want to be able to surface depository content and the way that AI is now being used, trained to search across this type of content. There’s a bit more involved in training AI to search across poster content and pulling out the relevant concepts and abstracts, for instance, as opposed to nice, highly structured traditionally published journal articles.
Mark: Do you think that people who work within impact assessments and those who believe in the power of grey literature are, in fact, natural allies? A lot of impact assessments aren’t published in traditional journals in that particular sense. They are themselves or could be perceived as grey literature themselves. Is there some form of grand alliance that’s possible here between those who are involved in the impact assessment and those who believe that the grey literature library repositories have value, not just within research and education, but within the impact assessment community itself?
Danny: One of the things that came up in conversations when I was working with one of the universities on this impact work, was we were using some software that collects impact evidence. You can put it into this software and then you can move it around in different ways and sort of create links between things. It’s effectively a repository, but it’s not public-facing. One group that was doing work on cities research and that ranges from things like coastal erosion to putting in bi-planes to towers creating shadows and all sorts of things, it’s quite broad. But what a lot of the people in the room got quite excited about was that the collection of the information that they needed to pull together to make the impact argument was also the material that they needed to collect together for their professional registration. They were members of professional organisations and their certification and so normally they wouldn’t collect the kinds of information that they need to for that certification within the university context because it’s the sort of thing that doesn’t normally count for things like promotions or different roles because it’s things like publication in a professional magazine as opposed to a journal. They were seeing that this collection of research impact material for the research impact purpose could help them in a professional sense as well, so there was another use for it. Then we started talking about the option of those articles in magazines themselves being put into the repository, which then allows them to have DOI associated with it a digilogic identifier, which allows them then to be found and a metric score to be attached to it so you can start to see who’s reading it or who’s talking about that article. Suddenly this whole other ability to give this work a new life and to be able to track it opened up. We spoke to the magazines about this, and they were quite happy for us to put the material into the repository and make it openly accessible. I was a bit surprised about that, but it turned out that this was a whole other unexpected aspect of doing this work. That’s one part of it that the repository can hold this material. The repository can share this material or could just hold it as a repository and not make it openly available, but there is another angle to this which is that librarians are very good at finding information. That’s their real expertise and so if you are a researcher and you want to demonstrate some research impact and you think okay, I know that we spoke at this conference and I know we gave some town hall meetings for the council and we did x, y and z but I don’t have the documentation for that, the library can help find that information. That’s what they’re really good at. There’s a kind of double part of involvement that the library potentially has in this impact space, I think.
Mark: A lot of impact assessment is looking for changes in reality, so not a case of something which is reported we’ve done something, there’s a new guideline, but evidence that that guideline is being used. From an NHS perspective, within NHS libraries, is there something more, is there an opportunity there to help with impact assessment because the evidence of theory into practice might be kept there. Might be within the grey literature that an NHS library would hold.
Helen: I think that this is a real opportunity for NHS library services. It fits very well with their role and remit. Basically, the reason for NHS library services existing is to mobilise evidence and knowledge into practice. That is their role. That is how Health Education England champion NHS library knowledge services, position their role and it’s very much about encouraging them to evidence the impact of their work. We’re collecting a huge amount of case studies, vignettes about where library knowledge services have worked with practitioners to get research evidence into practice and that makes a difference and that is how we are working with NHS libraries to justify and promote their value and that’s a win-win really for the researcher and the practitioner and the librarian who enabled that research to translate into practice with an outcome for patients and the NHS more generally.
Mark: I’ve got a final question for you both. Given the rise of digital libraries, is it an existential necessity for libraries to become key players in the impact infrastructure?
Danny: I think this is an enormous question in some ways because it does speak to the question of what is an academic library in an open-access world? One of the fun facts that we discovered at Cambridge, there’s two sets of digital material in the library in Cambridge. There is the collection of research that’s been developed within Cambridge and has been published by Cambridge researchers that’s in the repository and there’s also the digitisation of the collections held within the library and both of those are openly available to the public. Looking at the usage of those, 85% of the usage comes from outside the UK. There is a huge appetite for this material. As we move into a digital world and particularly an open world, the library changes its focus from being a gated collection of information for its own community to a shared information across it. We become one enormous library around the world instead of being individual libraries. If you’re thinking in that kind of way, then it absolutely has to be at the centre of this because that’s where all the information is held. There is no space for having it held in somebody’s back drawer or on somebody’s personal hard drive. That’s not where it needs to be. We need to have it in the one big library and that your node, is your node into that big library. Your local library and your institution is the node into the big global one. Yes, I think that absolutely it has to be central.
Helen: Yes, and I would agree from and NHS library perspective, but I think that this has been the case for NHS library services for a while. I think if they’d continued only to be focused on accessing information and finding information, they would have probably died out a little while ago, so their role, as I’ve mentioned previously, is very much about getting research into practice. It’s making sure that the research is kind of actionable and can translate into practice and its librarians’ role in supporting that process that is really, really critical in the NHS, and that’s where they really add value.
Mark: Can I just thank both Danny and Helen for their time today on this podcast. I’d also like to thank Alex Zabala Findlay and Natayla Chapman, graduate trainees at NIHR, who help me put the question set for today’s podcast. Thank you very much for listening.
Thank you for listening to this podcast. It’s one of four in a series exploring different aspects of impact culture. Please return to the website to discover the others and don’t forget to tweet us your comments and questions to #impactframeworks. Once again, thank you for listening.
