Meta-science summer school: Designing and conducting studies to improve research

Participants will learn more about meta-research by designing and starting to collect data for a meta-research study in a small team. Participants may choose to keep working after the summer school to complete and publish their project.

Concerned about the reproducibility and transparency of research? Interested in learning more about meta-research, or the science of science? Looking for hands-on experience? Excited to work with other early career researchers across disciplines?

What is meta-research?

Meta-research, or research on research, applies the scientific method to study the research process and ecosystem. This may include examining the design, conduct, reporting and dissemination of research, factors that influence the assessment of research or the composition of the research workforce. Meta-researchers may also develop, test, or implement interventions to improve research culture and practice.

Please note: This summer school is not focusing on meta-analysis. Meta-analysis refers to the process of systematically identifying all studies that address a particular research question (systematic review). While some meta-research uses similar techniques, there are clear differences to systematic reviews.

Description

Rather than focusing on a few pre-selected topics during the summer school, instructors will support participants in developing meta-reseach study proposals that are relevant to their own work and will provide value to the research community. An international group of experienced meta-researchers will work closely with each group to develop a study proposal.

Programme

Virtual Preparatory Phase

Before the summer school, participants will watch a series of videos introducing meta-research concepts and methods to maximize learning during the in-person sessions. In each 30-minute video, a meta-researcher will briefly describe a study that they conducted, and highlight important design considerations, possible challenges, and errors or design flaws that should be avoided. You can find recordings of past webinars on YouTube or register to attend live webinars.

Online Project Selection

During this interactive online workshop, participants will brainstorm and critically evaluate meta-research study ideas with support from experienced meta-research instructors. Participants will be divided into 3-4 small project teams based on their interests.

In-presence Summer School (near Frankfurt, Germany)

Each project team will work with experienced instructors (see below) to define their research question, identify pre-existing literature, design a study, prepare a proposal, develop a possible protocol, and perform feasibility testing. In the final session, each group will present their work, discuss feasibility, and decide whether to continue working with the instructors to complete their project after the summer school.

Hackathon

This half-day hackathon will take place during the in-presence summer school. Participants will learn how to curate data and what to consider when designing an automated tool to screen scientific papers or preprints for a common problem or beneficial practice (e.g. open data). Automated tools can be used to collect data, monitor uptake of good research practices, or as an intervention to improve reporting.

Instructors - Internationally-recognized meta-scientists who specialise in different fields and meta-research methodologies

Tracey Weissgerber (BIH QUEST Center): Tracey Weissgerber’s paper on the inappropriate use of bar graphs has been viewed more than 400,000 times and contributed to journal policy changes. Her meta-research specialities include data visualization, statistics, and open/reproducible methods reporting. Methods expertise: Systematic review/literature survey, automated screening, innovative educational formats.

Anita Bandrowski (University of California, San Diego): Anita Bandrowski is a founding member of the research resource identifier initiative, and founded a start-up that developed the SciScore automated screening tool. Many publishers use this tool to screen submitted manuscripts for MDAR checklist items. Methods expertise: Automated screening, meta-research using automated tools.

Malcolm Macleod (CAMARADES, University of Edinburgh): Malcolm Macleod is a clinician-scientist who has led or contributed to many transformational science improvement efforts, including the ARRIVE guidelines for animal studies, the Landis criteria, and the MDAR checklist. Methods expertise: Systematic review, meta-analysis, checklist/guideline development, interventional studies, preclinical and translational meta-research.

Malte Elson (META-REP, Ruhr University Bochum): Malte Elson examines the formation and perpetuation of research norms and procedural standards in psychology. He studies the transparency, reproducibility, robustness, and replicability of scientific findings, and quantifies the credibility, validity, and methodological adequacy of research literature. Methods expertise: Systematic review, content analysis, experimental design.

Vladislav Nachev (BIH QUEST Center): Vladislav Nachev is a data scientist with a background in behavioural biology and neurobiology. He programs automated screening tools and dashboards with metrics of open and responsible research. Methods expertise: Data analysis, automated screening, experimental design.

Rachel Heyard (Center for Reproducible Science, University of Zurich): Rachel Heyard is a biostatistician and meta-researcher. Her research interests include testing new approaches for awarding research funding, statistical methods in research, and replicability.

Costs

The Summer School is free to attend. A limited amount of travel funding is available to cover transportation costs from participants’ location to the venue. Please indicate whether you would like to apply for travel funding in your application.

ECTS

Students will receive a certificate after completing the summer school. BIH and Charité graduate students can obtain 2.0 ECTS for completing the summer school and preparatory work. As the preparatory work includes watching videos from the Meta-research Methods webinar series, students who participate in the summer school can not receive additional ECTS credit for attending the webinar series.

Scientific coordination and organisation

Dr. Tracey Weissgerber – University of Coimbra

Application

Interested in attending? The call for applications will open in 2025. Prior meta-research experience is not required. If you have an idea for a meta-research study that you might like to propose during the summer school, include this in your application. Remember that participants will come from many different fields; hence, projects that apply to many disciplines may resonate more with other participants than field-specific projects.

Evaluating Applications:

Each application will be anonymized and scored by two instructors. Scored applications will be divided into pools according to gender and career stage, and we will select the highest scoring applicants in each pool.

A total number of 29 participants (Master students, PhD students, and postdoctoral fellows) will be selected. We anticipate that we may receive more applications from psychology than from other fields, as meta-research has received more attention in psychology. We will limit the number of participants in psychology to 10 to ensure that participants come from many different disciplines.

Date: 15 – 20 June 2025

Language: English

Location: near Frankfurt, Germany

Number of participants: 29

Registration: will be opened in due time.

Contact: In case of questions or inquiries, please contact Tracey Weissgerber.