Multi-Stakeholder Research Data Management Training as a Tool to Improve the Quality, Integrity, Reliability and Reproducibility of Research

To ensure the quality and integrity of data and the reliability of research, data must be well documented, organised, and described. This calls for research data management (RDM) education for researchers. In light of 3 ECTS Basics of Research Data Management (BRDM) courses held between 2019 and 2021, we aim to find how a generic level multi-stakeholder training can improve STEM and HSS disciplines’ doctoral students’ and postdoc researchers’ competencies in RDM. The study uses quantitative, descriptive and inferential statistics to analyse respondents’ self-ratings of their competencies, and a qualitative grounded theory-inspired approach to code and analyse course participants’ feedback. Results: On average, based on the post-course surveys, respondents’ (n = 123) competencies improved one point on a four-level scale, from “little competence” (2) to “somewhat competent” (3). Participants also reported that the training would change their current practices in planning research projects, data management and documentation, acknowledging legal and data privacy viewpoints, and data collecting and organising. Participants indicated that it would be helpful to see legal and data privacy principles and regulations presented as concrete instructions, cases, and examples. The most requested continuing education topics were metadata and description, discipline specific cultures, and backup, version management, and storage. Conclusions: Regarding to the widely used criteria for successful training containing 1) active participation during training; 2) demand for RDM training; 3) increased participants’ Multi-Stakeholder Research Data Management 2 Liber Quarterly Volume 32 2022 knowledge and understanding of RDM and confidence in enacting RDM practices; and 4) positive post-training feedback, BRDM meets the criteria. This study shows that although reaching excellent competence in a RDM basics training is improbable, participants become aware of RDM and its contents and gain the elementary tools and basic skills to begin applying sound RDM practices in their research. Furthermore, participants are introduced to the academic and research support professionals and vice versa: Stakeholders will get to know the challenges that young researchers and research students encounter when applying RDM. The study reveals valuable information on doctoral students’ and postdoc researchers’ competencies, the impact of education on competencies, and further learning needs in RDM.


Introduction
During the second decade of the 2000s, many international, national, and institutional principles and policies and an increasing number of funders and publishers started recommending or mandating researchers to write data management plans (DMPs) and share data (e.g., Academy of Finland, 2019; "Amsterdam call for action on open science", 2016; European Commission, 2018a,b;European University Association, 2017;National Science Foundation, 2011;UNIFI, 2016;Wellcome, 2017). Researchers need education, guidance, and support in research data management (RDM) to help fulfil this task. At the core of these principles, policies and demands is to obtain data of publicly funded research openly accessible and reusable, when possible. In principle, research data, or at least metadata, should be Findable, Accessible, Interoperable and Reusable (FAIR 1 ). Sound RDM practices advance the integrity of data, reliability of research results, transparency of the research process, and reproducibility of research (e.g., Chiarelli et al., 2021). However, research transparency and data reuse may only be fully realised if data is opened and shared (Borghi et al., 2018). Shared research data also avoids the gathering of duplicate data and enables combined efforts to find solutions to complicated interdisciplinary research issues like climate change and pandemics (Doucette & Fyfe, 2013;Shearer, 2009). Moreover, sharing research data can significantly shorten the time it takes to move from an initial scientific discovery to practical applications (Federer, 2016). Nevertheless, it is only useful to share well-documented, described, and organised data (Borghi et al., 2018;Rieser, 2018) that provides clear data sharing parameters, including intellectual property rights (IPR) and agreements. .
Though RDM 2 is perceived as important or very important by researchers and graduate students (Pasek & Mayer, 2019;, many researchers are not managing their data according to recommended RDM guidelines. For example, graduate students are often given substantial data management responsibilities in research projects though they usually have received little or no education in RDM (Goben & Griffin, 2019;Krahe et al., 2020;Maienschein et al., 2019;Wiley & Kerby, 2018). Thus, they tend to develop ad hoc solutions with the trial-and-error method (Thielen & Hess, 2017;Wright & Andrews, 2015). Therefore, RDM practices are often unstandardised, and IPR and contract issues may be unfamiliar. Also, documentation made to carry out the ongoing research that does not consider other uses and users does not enable data sharing and reusing, undermining research reproducibility .
In this article, our goal is to find how generic, multi-stakeholder training can improve participants' competencies and further comprehension of the relevance of sound research data management practices to the quality and integrity of data and reliability of the research.
In practice, we will report the outcomes of the Basics of Research Data Management (BRDM) course over three years (2019)(2020)(2021), held at two Finnish universities. The learning objectives and contents of BRDM were developed based on an interview study on doctoral students' RDM competencies and learning needs Rantasaari & Kokkinen, 2019), discussions with the leader of the biostatistician team of the University of Turku (UTU), and research literature and lessons learned from previous RDM trainings (e.g., Piorun et al., 2012;Qin & D'ignazio, 2010;Whitmire, 2015;Wright & Andrews, 2015).
We aim to answer the following questions: • RQ1: How did course participants self-rate their RDM competencies before and after BRDM course?
• RQ2: What kind of educational impact did the course have on participants' RDM competencies (knowledge, skills, and abilities) based on participants' self-ratings and collected and categorised feedback? • RQ3: What kind of further learning needs did the respondents express after the course?
After the introduction, we will discuss specific contents and lessons learned in previous RDM basic trainings directed specifically for graduate students or researchers. The methods section will describe BRDM's objectives, structure, and learning methods. Research methods used to answer research questions RQ 1 to 3 will be described. Section four contains the results of the study, and section five the discussion and conclusions.

Common Contents in RDM Education
We collected the information of 30 RDM trainings from research articles and conference proceedings directed at graduate students or researchers between 2010 and 2021. Using the trainings' descriptions, the author categorised their contents as RDM topics and listed the topics handled in each training (Table 1 in Appendix A). The number of trainings addressing each topic is charted below (Figure 1). The most common topics covered in over 50% of the trainings were "Planning data management and organisation" (27); "Sharing and reuse" (25); "Storage, backup, and security" (21); "Metadata and data description" (21); "Preservation" (21); "Legal and ethical issues" (17); and "Quality and documentation" (17).
Overwhelmingly, the most common topics were "Planning data management and organisation" and "Sharing and reuse" which is understandable as the need for RDM became widespread after big funders like Wellcome (2017) and National Science Foundation (2011) began mandating DMPs and recommending data sharing in funding applications. Finnish major research funder Academy of Finland has required DMPs and data sharing in principle since 2015 (Academy of Finland, 2019). In RDM educational programs, data have been noted as a validator of research. Also, the reuse of data, as well as the policies, permits, and licenses demanding the data sharing, and the importance of becoming familiar with data sharing culture and infrastructure, have been discussed (Piorun et al., 2012;Read et al., 2019; Research data service, n.d.; Wright & Andrews, 2015).
Besides the informational type of contents, some courses and workshops include more technically oriented RDM topics such as data analysing and visualising, wrangling, merging, cleaning, and publishing data sets, as well as building and using relational databases for data gathering, organising, and querying (Carpentries, n.d.;Pascuzzi & Sapp Nelson, 2018;Qin & D'ignazio, 2010;Read et al., 2019; Research data service, n.d.; Wright & Andrews, 2015).

Lessons Learned in RDM Education
Though a lack of comprehensive and specific reporting of the results of educational RDM efforts exists (Goben & Griffin, 2019;Perrier et al., 2017), the feedback and results appear to be satisfactory or good. Typically, the attendants have been reported to have given good feedback (e.g., Chew et al., 2021;Whitmire, 2015), with their satisfaction varying from medium to high (Muilenburg et al., 2014).
As a result of training, competencies usually improve by one step, typically from "no competency" to "little competency" or from "some competency" to "good competency" (Qin & D'ignazio, 2010;Schmidt & Holles, 2018;Wright & Andrews, 2015). According to Peters and Vaughn (2014), based on the participants' self-assessment (n = 65) after the NECDMC workshop, the competencies were mostly good. In a survey after five RDM courses held in 2013-2017, 77% of the respondents (n = 31) considered the course useful, and 58% said they were interested in advanced education when available (Wiljes & Cimiano, 2019). In the feedback of the four clinical RDM workshops, respondents (n = 113), who were mainly project coordinators, faculty members, and managers, expressed a need to learn RDM from many viewpoints and aspects like IPR, data security and privacy, and data curation (Read, 2019).
Participants have typically requested more practical exercises, disciplinespecific cases, hands-on learning, and interactivity to concretise generic RDM principles to develop the training and deepen their competencies (Adamick et al., 2013;Byatt et al., 2013;Chew et al., 2021;Pascuzzi & Sapp Nelson, 2018;Wiljes & Cimiano, 2019). Nevertheless, fictitious cases not closely connected to participants' own research have been stated as uninteresting in feedback (Peters & Vaughn, 2014). Participants were most interested in learning more about data types and formats, archiving and long-term preservation, and metadata in the post-NECDMC workshop survey by Peters and Vaughn (2014), as well as data sharing, IPR, and legal issues. Participants were also interested in gaining more information on metadata and data security issues in the post-course survey of the NECDMC application by Muilenburg et al. (2014). In general, interactivity, discussion, peer supporting, and letting students apply generic principles in their own data are ways of concretising RDM (Peters & Vaughn, 2014;Read et al., 2019;Wright & Andrews, 2015).
Educational interventions in RDM are usually coordinated and led by libraries in academic institutions. Ideally, they begin with contextualising the education and determining the researchers' practices and needs via interviews, surveys, work shadowing, or focus groups (Kafel et al., 2014;Oliver, 2017;Qin & D'ignazio, 2010). In some cases, there has been a multi-professional steering group or committee, under which library is leading, and usually also carrying out the implementation (Kafel et al., 2014;Piorun et al., 2012). The library has been the main, and many times, the only actor arranging and implementing education on RDM. However, in interviews and surveys with students and researchers, the fact that data management needs are unrestricted to informational and consulting services typically delivered by the library has become evident (Joo & Peters, 2020;Oliver, 2017). Examples are creating RDM guidance, helping with data management plans, and planning and implementing education. Librarians may lack expertise in technical RDM assistance or using data science tools for data analysing, visualising, coding, cleaning, and database building (Cerny, 2021;Read, 2019). Librarians are not necessarily the best advisors on ethical and legal issues or safe and secure storage, either (Cerny, 2021;Peters & Vaughn, 2014). Thus, many educators are planning an increased collaboration in data management training and support with researchers, libraries, research IT, legal services, research funding, and research offices (Castle, 2019;Cox & Pinfield, 2014;Joo & Peters, 2020;Latham, 2017;Oliver, 2017;Peters & Vaughn, 2014;Read, 2019;Revez, 2018;Verbaan & Cox, 2014;Wittenberg & Elings, 2017;Yu, 2017).

Course Backgrounds
Our research goal is to find how generic, multi-stakeholder training can improve participants' competencies and further comprehension of the relevance of sound research data management practices regarding the quality and integrity of data and reliability of the research. The methods section will describe how we aimed at these goals with the versatile expertise of the course designers and teachers, as well as the learning objectives, course structure, and contents. We will also describe how we analysed the results of the training.
The analysed BRDM course was developed and implemented at the University of Turku (UTU), the third-largest research-intensive university in southwestern Finland with eight faculties, five independent units, and 21,000 students including 2,000 doctoral students and 3,300 staff members. The data policy of UTU (2016) motivated the planning of the studied course, according to which researchers would be offered training and support for writing DMPs and data managing during the project's lifecycle. The OpenUTU project group, containing members from the research office, library, research computing services, legal affairs, and communications unit of UTU, created the data policy. The library was responsible for creating and coordinating trainings and support in RDM for researchers. Because developing education for all researchers Liber Quarterly Volume 32 2022 was impossible, the head of library services (the author) suggested starting with doctoral students (DSs) and postdoc researchers (PdRs) in a prime position to learn sound RDM practices from the beginning of their career. The author interviewed 35 doctoral students, supervisors, and biostatisticians in UTU to learn the perceived importance of RDM competencies and doctoral students' current competencies Rantasaari & Kokkinen, 2019). Data management planning, documentation of data processing, and managing IPR and contract issues contained the most profound skills gaps. However, participants also lacked knowledge of different issues throughout the data lifecycle. Therefore, the author, with the leader of UTU's biostatistician team, set up a working group and invited researcher-teachers from different faculties, a grant writer, data librarians, lawyers, a data security officer, and an IT computing specialist to plan and teach a course on RDM for DSs and PdRs. In 2020 we extended the course to Turku's other university -Åbo Akademi University (ÅAU) -the only Swedish language multi-faculty university with 5,500 students and 1,100 staff members in Finland and with whom UTU has a long tradition of joint projects.

Learning Objective, Course Structure and Data Management Plans in BRDM
A participant's learning objective was to familiarise themselves with RDM's central concepts and develop a high-class research plan and data management plan (DMP). After completing the course, a participant comprehends the significance of well-documented FAIR data for the ongoing study and other potential use and users, applying safe and secure practices in collecting, producing, handling, storing, sharing, and preserving the data, and acknowledging IPR, privacy, and sensitivity considerations when needed. 3 Though BRDM is a generic and introductory course, we separated the course for different tracks. The preliminary idea behind the track-based division was that the data management actions needed and applied depend partly on the type of the data, partly on research methods, and partly on discipline (Aker & Doty, 2013;Joo & Peters, 2020;Lefebvre et al., 2018;Scholtens et al., 2019;Weller & Monroe-Gulick, 2014). These underlying factors delineate what kind of contracts, usage rights, storing solutions, processing, reuse, and preserving is needed or possible. For example, the methods in the clinical health sciences are usually experimental or observational 4 ; data are often identifiable, confidential, and highly sensitive. In the natural sciences, methods are typically experimental, observational, or simulation-based 5 ; data is largely not confidential and sensitive. However, there can be other rigorous demands for handling, storing, and preserving large data sets. In survey and qualitative research, the data and its needed and possible actions can vary greatly, depending partly on discipline and partly on each respondent's or interviewee's answers, the study subject's activity, and so forth.
In 2019, the first year, the BRDM course consisted of three tracks (Clinical Health Sciences, Survey Research, and Natural Sciences), with seven face-toface modules in Finnish for DSs and PdRs at UTU. In each track, participants were to prepare a shared DMP together during the course. A DMP was based on a fictitious research plan delivered by the faculty teacher-researchers in Module One. The participants learned by familiarising themselves with pre-class materials and preparing assignments on Moodle, after which they attended a lecture on the module.
In 2020, the course began with a joint introductory lecture with all the four tracks (Clinical Health Sciences, Survey Research, Interview Research, and Natural Sciences). The course was developed for DSs and PdRs of UTU and ÅAU. Clinical Health Sciences and Survey Research tracks were held in Finnish, whereas Interview Research and Natural Sciences were held in English. The course was turned fully online via Moodle after the three first modules because of the COVID-19. Instead of preparing a fictitious research plan and DMP, everyone created their own research plan and a DMP. Course modules were linked by mapping each module with the sub-section(s) of the General Finnish DMP template 6 and adding an assignment to prepare and update a relevant section of the DMP before and after each module's workshop session. The last assignment was to return the DMP and give an anonymous peer review of another participant's DMP. Finally, the author of this article assessed and rated each DMP and gave a general level feedback of all the DMPs using Finnish DMP Evaluation Guidance (FDEG) (Aalto et al. 2021). Otherwise, the learning followed the 2019 pattern, consisting of preclass activities followed by a lecture on Zoom.
In 2021, the course was online from beginning to end and adapted a flipped classroom method for teaching. The course continued with the same four track structure used in 2020 except the Interview Research track was turned to the Qualitative Research track. In each module, participants introduced themselves with the modules' pre-class materials in Moodle and drafted a relevant section of their DMP for themselves. The participants also added questions to the discussion forum based on the pre-class materials and their own data. After pre-class activities, the module's Zoom workshop session was reserved for discussion based on the questions that participants had written beforehand or asked during the workshop. As in 2020, the modules' post-class assignment was to update a DMP's relevant section, informed by the discussion in the modules' workshop. Each participant returned their DMP and peer-reviewed another participant's DMP as a final assignment for the course, after which the author assessed and scored the DMPs (Table 1).

Formative Assessment: Feedback
Following each module, participants were asked to give formal feedback through an online form (Appendix B). Module-based feedback was used as a formative assessment to control the participants' learning, receive information on experienced challenges, and collect proposals for improving the course. Hence, feedback produced ongoing information for the teachers to edit and enhance the modules and their contents. Moreover, halfway through the course, the author compiled the feedback with answers and information about remedies that were made or would be made.
The author used the grounded theory-inspired approach to code and analyse the feedback for this study (Bryant & Charmaz, 2016;Cassidy, 2012;Timonen et al., 2018). Sub-categories were created based on the topics that emerged from the coded comments. Grounded theory as an analysing approach is well-suited for processing and analysing the feedback as the aim was to let the feedback data speak for itself and not use an existing theoretical framework and categories formulated according to the framework.

Summative Assessment: Survey
The participants were asked to participate in a survey to self-rate their competencies before and after the course. In 2019, the survey was carried out twice -before and after the course -on a scale from 1 to 5: 1 = no competence, 2 = some competence, 3 = good competence, 4 = very good competence, 5 = top competence (Appendix C). In 2020-2021, we performed a post-course survey in which participants were asked to self-rate their competencies before and after the course on a scale from 1 to 4: 1 = no competence, 2 = little competence, 3 = somewhat competent, 4 = very competent (Appendix D). Participants were also asked to give a course rating from 1-100, if they would recommend the course to other DSs and PdRs, and choose the topics about which they would like to have more education.
The survey served as a summative assessment for collecting participants' perceptions of their learning, the quality of the course, and further education needs.
The respondents' self-ratings of their competencies were analysed using JMP Pro 16 to produce descriptive and inferential statistics with medians, custom quantiles, Wilcoxon signed-rank test (one group), Wilcoxon ranksum test (two independent groups), and Steel-Dwass test (multiple comparisons). Frequencies and Chi-square test were used for announcing further learning needs. A significance level of 0.05 (two-tailed) was used. Also, module-and course-based feedback comments were coded and categorised in NVivo 12.

Summative Assessment: Data Management Plans
In Module One, each participant created their own research plan. During the course and based on their research plan, they wrote a DMP using a Finnish General DMP template and guidance. The course participants' DMPs will be analysed in a later study.

Participants
Of the 386 enrolled participants in 2019-21, 346 (90%) were DSs, 37 (10%) were PdRs, and 3 (1%) were university employees. Of those who completed the full course with 3 ECTS credits, 154 (91%) were DSs, 14 (8%) were PdRs, and 1 (1%) was a university employee. Of the participants who did not complete the full course but (on average) half the modules, 72 (80%) were DSs, 17 (19%) were PdRs, and 1 (1%) was a university employee. In 2019, participants who did not complete the full course performed (on average) 3 out of 7 modules; in 2020, 3 out of 8; and in 2021, 4 out of 8. Performing only part of the modules does not mean that participants interrupted the course but that the modules were performed evenly between modules 0 (introduction) and 8 (final assignment). PdRs, in particular, picked modules according to their interests, without needing to earn the 3 ECTS credits ( Table 2).

Feedback
We asked participants to fill in a feedback form after each module on the Moodle course platform. In 2019, the module-based feedback was a mandatory course assignment. This task was voluntary in 2020 and 2021, echoed in the number of feedback forms we received: 133 forms in 2019, 114 in 2020, and 69 in 2021. In 2019 and 2020, participants were given a time slot at the end of the classes or workshops to provide feedback; in 2021, we simply reminded participants to give feedback after the live Zoom workshop sessions.
The feedback form contained three main categories: 1. What are the three things you have learned? 2. How will the things you have learned change your practices? 3. How would you suggest the module be developed? Under these main categories, the author created sub-categories and sorted the comments using a grounded theory-inspired approach. The five biggest sub-categories stand for 90 to 100% of all comments in the main categories (Figures 3 to 5; Tables 3 to 5 in Appendix A). Because a respondent's comment in a feedback form could include several aspects, it could be placed accordingly in two or more sub-categories. For example, the comment "I am now more aware of IPR issues and GDPR, which enables me to plan my next research in more detail" has been placed in the sub-categories "I will pay notice to IPR, agreements and licenses", and "I will pay more notice to data privacy and data security". Hence, the total number of comments in different sub-categories is bigger than the number in the original four main categories.

What are the Three Things you have Learned?
Based on the feedback given in this main category, we identified 252 separate comments in 2019, 194 in 2020, and 119 in 2021. The sub-category "What, why and when in RDM" contains comments of RDM essentials such as the learning of the rationale and tools to plan data management, the central concepts of RDM, and the different phases such as storing, documenting, preserving, and sharing of data: "(I learned) ways to conceptually approach Research Data Management and the practices and perspectives related to it." (Module 2, Qualitative Research, ID 16/2021).
The number of the comments concerning data management planning and documentation grew in 2020 and 2021: "Documentation of data in a clear and readable form is a crucial step in the data management and processing." (Module 5, Natural Sciences, ID 91/2020).
At the same time, the percentage of the comments belonging to the sub-category "The importance of legal considerations" dropped from 25% in 2019 to 8% in 2021. (Figure 3; Table 3 in Appendix A).

How will the Things you have Learned Change your Practices?
Based on the feedback given in this main category, we identified 87 comments in 2019, 132 in 2020, and 63 in 2021. The comments concerning improving data management planning and documenting practices increased from 21% to 49%, whereas the comments about the intention to focus more on IPR, agreements, and license issues decreased from 29% to 8%. The following quotation illustrates the increased number of comments concerning documentation's importance:

"I learned a lot about the importance of documentation and metadata as well as publishing datasets. I will apply the FAIR principles when my research work needs to be checked and will review the data management all the time." (Module 7, Qualitative Research, ID 68/2021).
At the same time, the percentage of data privacy comments (e.g., data privacy notice, informed consent, and GDPR), and data security comments (safe and secure data storing platforms), increased from 15% to 25%. (Figure 4; Table 4 in Appendix A).

How would you Suggest the Module be Developed?
We identified 90 (2019), 136 (2020), and 101 (2021) proposals to develop the modules. Most of the respondents wished for practicality such as more discipline-specific instruction, checklists, and cases, along with the clarification and standardisation of course practicalities, schedules, and course platforms, and how to balance the workload between different modules. More practicality and concreteness were desired, especially in law-related modules three and four:

"All the law-related sections could explain things in less of a law-speech manner as law speech is generally really vague and does not provide any practical knowledge. In general, the wideness of topics was really good." (Post-Course Survey, Natural Sciences, ID 53/2021)
In 2019 and 2020 (but not in 2021), many comments expressed a desire for more interactivity and discussions. Unlike in 2020-2021 courses, participants preparing their own research plan and DMP in 2019 -visible in the 7 (8%) comments -was impossible. For the first time in 2021, we received 12 (12%) answers that the module is good as is. (Figure 5; Table 5 in Appendix A).

The Overall Score
In the post-course surveys after the 2020-2021 courses, participants were asked to score the course between 0 and 100. After the 2021 course, participants were also asked if they would recommend the training to other DSs or PdRs. Based on the survey respondents' general score of 68 out of 100 in 2020 (n = 53) and 74 out of 100 in 2021 (n = 64), the course lived up to the reasonable expectations of a general level introductory education. Equally, 92% of the post-course survey respondents in 2021 expressed they would recommend the course to other DSs and PdRs.

BRDM 2019
Participants were asked to rate their current RDM competencies on a fivepoint scale from 1 to 5 before and after the 2019 course (Appendix C). Hence, Liber Quarterly Volume 32 2022 45 (82%) enrolees answered the pre-course survey, and 17 (41%) of those who completed at least part of the modules answered the post-course survey. Before the course, participants' median self-rating of their RDM competence was 1.96 (Q1:1.82, Q3:2.09). After the course, participants' median self-rating of their competence was 2.32 (Q1:2.12, Q3:2.84). The improvement was statistically significant, p = 0.003 (Wilcoxon rank-sum test), or 0.36 points. (Figure 6; Table 6 in Appendix A).

BRDM 2020-2021
The surveys in 2020 and 2021 (Appendix D) differed from the 2019 survey related to the contents and execution: • Instead of pre-and post-course surveys, we only had a post-course survey. • The competencies were specified to respond more closely to the learning objectives of the modules . • The scale was 1 to 4 instead of 1 to 5. The combined response rate to the surveys was 49% (106 respondents out of 217 participants) after the 2020-2021 courses. On the 1 to 4 scale, the median self-rated competence before and after the courses was 1.97 and 3.03, respectively. Thus, the median self-rated competencies improved statistically highly significantly, p < 0.0001 (Wilcoxon signed-rank test), or 1.06 points (Figure 7; Table 7 in Appendix A).
Regarding the variance in the results between disciplines and course tracks, differences were statistically insignificant at the level of total medians, although some were found concerning a few specific competencies before the course. First, respondents in the "Qualitative Research" track and the "Humanities, psychology, and theology" discipline self-rated their competence higher than those in the "Clinical Health Sciences" track in identifying the data life cycle and recognising a DMP's components (p = 0.02, Steel-Dwass). Second, respondents in the "Qualitative Research" track and the "Humanities, Psychology, and Theology" and the "Social Sciences, Business, and Economics" disciplines self-rated their competence in applying anonymisation higher than those in the "Natural Sciences" track (p = 0.01,

Fig. 7: Based on median self-ratings, respondents' competencies related to the specified learning objectives before and after the BRDM 2020-2021 courses. Light blue bars represent the competencies before the courses and dark blue bars after the courses. Full descriptions of the learning objectives can be found in the survey form (Appendix D).
Steel-Dwass) and the "Science and Engineering" discipline (p = 0.02, Steel-Dwass). Third, respondents in the "Social Sciences, Business, and Economics" discipline and the "Qualitative Research" track self-rated their competence in applying data privacy higher than those in the "Science and Engineering" discipline and the "Natural Sciences" track (p = 0.02, Steel-Dwass). All differences after the course were insignificant.

Subjective Educational Needs in 2020-2021: What would you like to Learn more about?
Participants were asked to choose the topics they wanted to learn more about in the post-course surveys. As much as 102 respondents (96%) expressed interest in advanced training. Six topics receiving over half (261) of all mentions (471) were "Metadata and description" (55), "Discipline-specific cultures" (44), "Backup, version management, storage" (42), "Ethics and legal considerations" (40), "Quality and documentation" (40), and "Visualisation and representation" (40). However, interest for advanced training in "Discovery and acquisition" (21) and "Data curation and reuse" (26) were the lowest. Differences were statistically insignificant related to respondents' discipline or course track. The frequencies of mentions for further learning needs are illustrated in Figure 8.

How did the Course Succeed?
In this article, our goal was to find how generic, multi-stakeholder training could improve participants' competencies and further comprehension of the relevance of sound research data management practices to the quality and integrity of data and reliability of the research. Furthermore, the questions we aimed to answer were as follows: RQ1) How did the course participants selfrate their RDM competencies before and after the course? RQ2) What kind of educational impact did the course have on participants' RDM competencies (knowledge, skills, and abilities) based on participants' self-ratings and the collected and categorised feedback? RQ3) What further learning needs did the respondents express after the course? These questions will be discussed with the help of the criteria for successful training as created by Oo et al. (2021).
Based on the systematic review of 28 RDM trainings between 2012 and 2019, Oo et al. (2021) introduced a four-part criterion for successful training consisting of 1) active participation during training; 2) demand for RDM training; 3) increased participants' knowledge and understanding of RDM and confidence in enacting RDM practices; and 4) positive post-training feedback. How BRDM matched these criteria will be discussed below.
Concerning the participation during training, BRDM was based on active learning: Participants read and listened to course materials, completed assignments, developed their own research plan and a DMP, peer-reviewed each other's DMP, drafted questions based on course materials and their own data management issues, and participated in the workshop discussions. The activities sought to help participants link the principles and other theoretical contents to their research practices (see also Whitmire, 2015;Wiljes & Cimiano, 2019;Wittenberg & Elings, 2017). Judging by the feedback during and after the 2021 course, we succeeded in bringing interactivity and discussion to modules. However, there was still a demand for turning, especially legal and data privacy principles and regulations, into concrete instructions, cases, and examples when possible.

Liber Quarterly Volume 32 2022
After completing the course, almost all respondents expressed interest in further education for RDM training. The most frequently mentioned topics for further learning were "Metadata and description", "Discipline-specific cultures", "Backup, version management, storage", "Ethics and legal considerations", "Quality and documentation", and "Visualisation and representation". Metadata, ethics, and legal issues were also the most wanted topics for continued learning in the courses of Muilenburg et al. (2014) and Peters and Vaughn (2014). Conversely, despite emphasising FAIR principles, as well as data sharing and reuse throughout BRDM, advanced training in "Discovery and acquisition" and "Data curation and reuse" were the least preferred topics. Minor interest in these might be comprehensible concerning cultures of practices in many disciplines where researchers' primary interest is getting their current project through and obtaining results from the data rather than long-term preservation and the possible data reuse in future projects (Kowalczyk, 2017;. Concerning increased knowledge, understanding, and confidence in enacting RDM practices, participants highlighted that they had learned RDM essentials such as understanding the rationale and learning the tools to plan data management, RDM's central concepts, and storing, documenting, preserving, and sharing data. Moreover, participants learned legal and data privacy issues and how to use REDCap and NVivo in data collecting and organising. Correspondingly, they reported that the training would change their current practices in planning research projects, managing and documenting data, acknowledging legal and data privacy viewpoints, and using REDCap and NVivo in data collecting and organising. The median self-rated improvement in RDM competencies was 0.36 points in 2019 and 1.06 in 2020-2021 -one level up from "little competence" to "somewhat competent". One-step improvement during a generic RDM course is a typical result that has been documented in several post-course surveys (e.g., Qin & D'ignazio, 2010;Wright & Andrews, 2015).
As far as respondents' disciplines or course tracks are concerned, the differences were statistically insignificant at the level of total medians. However, some significant differences were found concerning a few specific competencies before the course. Respondents in the "Qualitative Research" track and the "Humanities, Psychology, and Theology" and "Social Sciences, Business, and Economics" disciplines self-rated their anonymisation competencies before the course as higher than those in the "Natural Sciences" track and the "Science and Engineering" discipline. Likewise, respondents in the "Social Sciences, Business, and Economics" discipline and the "Qualitative Research" track self-rated their data privacy management competencies before the course as higher than those in the "Natural Sciences" track and the "Science and Engineering" discipline. These differences are comprehensible because data in qualitative research and social sciences, more often than in the natural sciences and engineering, contain personal or even sensitive contents. However, the differences had disappeared after the course. This indicates that the course had bridged the gaps in respondents' competencies regarding their disciplines and course tracks. On applying the RDM principles in participants' own data management planning, the results of the assessment and rating of the returned DMPs in the BRDM 2020-2022 courses will be reported in another upcoming article.
Pertaining to feedback, the course was perceived as a solid and important introduction to RDM's different aspects. Teachers -including a grant writer, researchers, data librarians, lawyers, a data privacy officer, a data archive specialist, a biostatistician, and an IT professional -were appreciated as real domain experts. However, regarding propositions for course development, respondents asked for a more down-to-earth approach, concretising, and examples, especially in legal and data privacy issues. Clarification of the course platform and course practicalities were also requested.
BRDM can be determined as one of the few trainings (so far) that meet all parts of the four-part criteria for successful training as defined by Oo et al. (2021). However, because of BRDM's limited number of participants, we cannot generalise our study's results and the factors affecting them outside the studied group. Furthermore, we cannot know the long-term impact of the participants' self-rated competencies on their RDM activities without follow-up. Still, 319 returned module-based feedback forms, and 168 survey responses revealed valuable, indicative information of doctoral students' and postdoc researchers' competencies, the impact of the education on competencies, and further learning needs in RDM.

The Value of BRDM and Lessons Learned
BRDM is an educational effort bringing value to RDM training. So far, academic libraries have been the main, and many times the only, actor arranging Liber Quarterly Volume 32 2022 and implementing education on RDM in research-intensive universities. As a further development need, educators have often mentioned a need for collaboration with multiple stakeholders (Castle, 2019;Cox & Pinfield, 2014;Joo & Peters, 2020;Latham, 2017;Oliver, 2017;Peters & Vaughn, 2014;Read, 2019;Revez, 2018;Verbaan & Cox, 2014;Wittenberg & Elings, 2017;Yu, 2017). In BRDM, using versatile expertise in planning and teaching has been embedded from the beginning: Academic and research support experts planned and taught the course. Second, the contents of BRDM were wide-ranging containing most of the phases of data life cycle, beginning from the writing of a highclass research plan -which makes this course unique -to the sharing and long-term preservation of the data. However, limited resources excluded more technical data science contents such as analysing, visualising, cleaning, merging, and programming data. Third, participants applied sound RDM principles in their data management by writing a DMP during the course. Hence, assessing BRDM's results is based not only on the feedback and self-rating of the participants' competencies with further learning needs (typical measures of success in many previous trainings) but the returned DMPs. Fourth, a flipped classroom approach that is rarely used as a teaching method in previous RDM training (Griffin, 2020;Johnston & Jeffryes, 2015;Mithun & Luo, 2020), was adapted in the BRDM 2021 course. Fifth, many previous RDM trainings have been criticised for inadequate reporting (Goben & Griffin, 2019;Perrier et al., 2017). In this study, we aimed for extensive and precise reporting.
Next, we will present some concrete lessons that we have learned during planning, implementing, and analysing the results of BRDM in 2019-2021.
Planning and implementing training with multiple RDM stakeholders enable acknowledging all relevant aspects of the data life cycle. Participants will get an overall view of the numerous factors affecting RDM, while stakeholders' overall understanding of the RDM and the challenges doctoral students and postdoc researchers confront increase. The downside of a large working group and many teachers is the administrative burden in coordinating the training. Moreover, the pedagogical skills of multi-professional specialists can be diverse. Therefore, keeping the training coherent by reaching a consensus on the learning objectives, teaching methods and contents, course practicalities, and deadlines with all the teachers and working groups is paramount.
RDM is an organic part of a research project -from planning the goal and research questions and proceeding to the methods of collecting, producing, processing, storing, sharing, and preserving the data. Thus, recalling and updating or, preferably, rewriting a research plan is important when developing a DMP. Otherwise, research and data management plans can be asynchronous for example regarding data types to be collected, produced, and reused in a project.
Collecting feedback throughout training serves as a formative assessment to control the participants' learning, receive information from experienced challenges, and gather proposals to quickly improve the training.
Though planning and implementing the flipped classroom approach takes a lot of work from teachers, it pays back by increasing flexibility and helping activate participants. Still, quizzes or follow-up tasks are essential to show that participants learned the pre-class materials, as Mithun and Luo (2020) have pointed out.
Measuring the learning results should not be based solely on the participants' self-assessment or feedback, but on the assessment of assignments such as DMPs developed during training. Moreover, a follow-up intervention would be needed to collect empirical evidence on how the planned actions in DMPs have been applied in research practice (see also Perrier et al., 2017).
A modular structure enables cherry-picking the training, reducing the dropout rate. For example, PdRs do not necessarily need credits or certification by completing a training but want to bridge their knowledge gaps by choosing the modules that interest them.
Finally, according to this study and many others (e.g., Chew et al., 2021;Pascuzzi & Nelson, 2018;Wiljes & Cimiano, 2019), there is never too much IPA (Interaction, Practice, and Application) in training. Still, participants achieving excellent competence in a basic training are improbable. Instead, discipline, data type or research method specific workshops with fewer participants will help deepen the elementary skills (e.g., Petters et al., 2019;Read, 2019;Thielen & Hess, 2017). However, as highlighted in research literature, training without synchronised incentives, policies, processes, and infrastructure is insufficient to bring about behavioural change Perrier et al., 2020). A realistic target for a generic training could be that participants become aware of RDM and its contents and gain the elementary tools and basic skills to begin applying sound RDM practices in their research processes. Moreover, introducing participants to support services of multiple RDM stakeholders is important. That stakeholders learn what kind of challenges researchers and research students encounter when applying RDM is equally important.

<0.0001
Creating a storage and backup plan, and applying it to your data using the services of your organization, or the services of The IT Center for Science (CSC)