OpenLobby.EU
Explore

European Network for Research Evaluation in the Social Sciences and Humanities

ENRESSH

The purpose of the Association is to encourage, promote and coordinate studies and researches into the evaluation of the SSH, in all its aspects: quality representations, bibliometric studies, qualitative analysis, intra and extra-academic impact, societal role, uses and effects of evaluations, etc.

Lobbying Activity

Response to Ex-post evaluation of Horizon 2020

29 Jul 2022

ENRESSH, the European Network for Research Evaluation in the Social Sciences and Humanities, appreciates the consultation for the evaluation of Horizon 2020. We strongly suggest to consult with specalists in research evaluation of ALL disciplines. We are aware that the specialists in evaluation of Social Sciences and Humanities research were not enough consulted for the evaluation procedure of H2020. The indicators and databases used are not valid for SSH research (and their validity can be questioned for many disciplines in the STEM fields too). Most importantly, it is crucial that evaluations do not focus on measurable, manifest outcomes because such outcomes reflect at best short-term impact, while long-term impact is the more relevant goal. Also, there are manyfold pathways to impact, most of them not uni-directional and going beyond co-creation (Bulaitis, 2020; Muhonen et al.,2020). The cited document on Impact Evaluation in H2020 clearly is not in-line with the Research Assessment Reform principles (with regard to scientific impact) and needs to be adapted urgently as indicators are not suitable for most disciplines (see van Leeuwen, 2013). Also the indicators for societal impact in this document are often not valid as they include only manifest outcomes, economic return or technological advancements but do not account for their consequences and miss many forms of contribution (see, e.g. Bulatitis 2017; 2020). Meanwhile, research on impact assessment is developed much further as has research evaluation in general (ENRESSH, 2020). Note also, that a focus on measurable and manifest impact might also lead to strengthening the divide between European countries (see, e.g., de Jong et al. (2018). Research can not only have positive impact but also negative impact and the non-identification of such leads to a misrepresentation of societal impact of research (Derrick et al., 2018). Also, note that scientific endeavour is collective, meaning not only co-authorship is relevant but also competitors. Science is only science if there is a discourse. Often, societal impact cannot be attributed to single projects. Rather, research and disciplines as such have impact. The evaluation of societal impact is a very delicate endeavour and a simplistic conceptualisation such as presented in the H2020 key performance indicators can lead to long-term negative consequences for research and society. Note that even though societal impact is seemingly conceptualised in a complex way, including co-creation etc., it is still too simplistic as the interplay between research and society, evaluation and research practice are very complex. This asks for specialised evaluation of H2020 projects; we therefore call for a more complex conceptualisation of scientific and societal impact, accounting for ALL disciplines and using valid indicators responsibly based on reliable and valid data (which is until now very rarely the case). Research Quality (including its impact) is a complex construct (Ochsner, 2022) and its measurement delicate and not entirely measurable (Ochsner et al. 2012). Focusing on measurable and manifest aspects only, excluding negative impacts of research, the positive impact of failure in research, opportunity costs etc. comes with the risk of negative societal impact of research evaluation! We therefore call for an invitation of specialists in research evaluation in ALL domains in the design of the final evaluation of H2020 projects! Bulatitis 2017 http://doi.org/10.1057/s41599-017-0002-7 Bulaitis 2020 http://doi.org/10.1007/978-3-030-37892-9 Derrick et al. 2018. https://hdl.handle.net/1887/65230 de Jong 2018 http://doi.org/10.1093/reseval/rvy036 ENRESSH 2020. http://doi.org/10.6084/m9.figshare.12049314 van Leeuwen 2013 http://doi.org/10.5283/bpf.173 Muhonen 2020 http://doi.org/10.1093/reseval/rvz003 Ochsner et al. 2012. https://doi.org/10.5283/bpf.157 Ochsner 2022: https://doi.org/10.4337/9781800372559.00010
Read full response

Response to Interim evaluation of Horizon Europe

29 Jul 2022

ENRESSH, the European Network for Research Evaluation in the Social Sciences and Humanities, appreciates the consultation for the interim evaluation of Horizon Europe. We strongly suggest that an evaluation is carried out together with specialists on research evaluation of ALL disciplines. It is highly relevant that the SSH parts of the research projects are not only seen as dissemination to the public because most of todays societal challenges are, in fact, SSH-related topics for which research is available. There is also ample knowledge on evaluation of SSH research. In fact, most issues currently discussed and leading to the reform of research assessment put forward by the EC and other European organisations, have been identified and put forward decades ago by representatives of the SSH. Specialists in research evaluation in the SSH are concerned that moving from exclusively focusing on excellence in academia (measured by bibliometrics) to societal impact (measured mainly through short-term manifest outcomes) will lead to a similar situation, i.e. that existing knowledge of specialists in research evaluation in the SSH about the pitfalls of evaluation of societal impact of research will be ignored and the same mistakes repeated. We therefore strongly suggest to consult with specialists of all disciplines, including the SSH who have a long-standing tradition of interacting with society. ENRESSH (www.enressh.eu) is of course an experienced and competent partner, having already consulted COST, national governments across Europe, ERC and many other organisations. Most importantly, it is crucial that evaluations do not focus on measurable outputs as these reflect at best short-term impact, while long-term impact is the more relevant goal. Also, there are manyfold pathways to impact, most of them not uni-directional and going beyond co-creation (Bulaitis, 2020; Muhonen et al.,2020). The cited document on Impact Evaluation in H2020 clearly is not in-line with the Research Assessment Reform principles (scientific impact) and needs to be adapted as indicators are not suitable for most disciplines (see van Leeuwen, 2013), also the indicators for societal impact are often not valid as they include manifest outcomes, economic return or technological advancements only but do not account for their consequences and miss many forms of contribution. Meanwhile, research on impact assessment is developed much further. Note also, that a focus on measurable and manifest impact might also lead to strengthening the divide between European countries (see, e.g., de Jong et al. (2018). We thus strongly call for a better conceptualisation and operationalisation of societal impact for Horizon Europe; societal impact is a complex matter and a focus on short-term, measurable outputs might lead to misinformation regarding the actual long-term impact of research, and, consequently, might function as disincentivation for researchers to embark on future projects that focus on long-term sustainable societal impact. Furthermore, there has been a French Presidency conference on Research Evaluation in the SSH, organised by HCERES. The proceedings will soon be published and we strongly recommend to consult them. ENRESSH thus strongly asks to reconsider the evaluation methods for Horizon Europe interim evaluation. A special focus needs to be put on the integration of SSH in the projects, as SSH have their research questions in their own right. The mistakes made in the last decades to focus on simple metrics and some STEM disciplines must not be repeated. We all feel the consequences now. We therefore call for an integration of research evaluation specialists of ALL disciplines, including the SSH, to develop a sustainable societal impact evaluation of Horizon Europe Projects. Bulaitis http://doi.org/10.1007/978-3-030-37892-9 Muhonen http://doi.org/10.1093/reseval/rvz003 de Jong http://doi.org/10.1093/reseval/rvy036 van Leeuwen http://doi.org/10.5283/bpf.173
Read full response