Pre-conference video and audio test

Check that your video and audio work before the first event starts.

Examining Questioned Data: Technology as a Detection Tool

Research misconduct in the federal regulation (42 C.F.R. Part 93) means fabrication, falsification, or plagiarism (FFP) in proposing, performing, or reviewing research, or in reporting research results (§ 93.103). This session will discuss the types of questioned data and tools that currently are used to identify and confirm FFP with special emphasis on the need for additional tools and automation. Examples of different types of falsified/fabricated (FF) data from closed misconduct cases at the Office of Research Integrity (ORI) and forensic tools used by ORI to detect and confirm intentional FF will be presented.

Computational research integrity and cancer research: building tools and narratives to improve the health of the research literature

Computational research integrity and cancer research are young and old research fields, respectively, and yet they have much in common. Cancer research involves the discovery of biological features that reliably distinguish cancer cells from normal cells. These features are targeted by drug developers to create cancer therapies that are tested by researchers and then applied by clinicians to patients. Similarly, computational research integrity involves the identification of publication features that reliably depart from established norms or standards. These features inform the creation of automated tools that are then tested and applied by researchers and publishers to manuscripts and papers. Based upon our experience of applying the semi-automated tool Seek & Blastn to the molecular cancer research literature, we will describe how the skill to employ automated literature screening tools needs to be matched by the will to apply these tools and then act upon their results. Beyond developing the skills to apply automated literature screening tools within different user groups, we propose that achieving the necessary willingness to tackle pervasive research integrity problems will require the development of positive narratives that speak to shared aspirations and values.

An inside look at real challenges in research misconduct investigations

Institutions face many challenges in managing complex research misconduct cases. From the increased public nature of allegations to the rapidly evolving technical landscape, research integrity officers have to develop expertise across many areas and identify tools to ensure the integrity of these complicated investigations. We will discuss common issues in institutional research misconduct proceedings and the growing need for technical solutions, for example record sequestration, forensic image and data analysis, and other variables that complicate institutional reviews. We will shine a light from the inside out, on how institutional investigations are more than spotting problem images and the critical work done to protect the integrity of the research record.

Contributed presentations

12:30 PM - 12:50 PM:
Ghazal Mazaheri, Kevin Urrutia Avila and Amit K. Roy-Chowdhury, "Learning to Identify Image Duplications in Scientific Publications"

12:55 PM - 01:15 PM:
Kyle Siler, Philippe Vincent-Lamarre, Cassidy R. Sugimoto and Vincent Larivière, "The Lacuna Database: Empirical Data to Identify Obscure, Unconventional, Questionable and/or Predatory Journals"

A System for Forensic Analysis of Scientific Images

In this talk I will describe a system that we are developing for the forensic analysis of images and other media extracted from a scientific publication. This system uses many modern media forensic methods to examine images and determine if the image has been likely altered or modified. The tools that are available include duplication detection, copy/move detection, provenance analysis and media forensics tools. The current system has methods for extracting images, figures, and captions and maintaining the relative relationships of the figures in a paper. The system has a simple and intuitive web-based user interface, a sophisticated database, and is easily extensible using Docker containers.

Panel: Institutional investigators

Institutional investigations are many times the first places where systematic problems are investigated. In many cases, these are the places where investigations are most consequential because they carry with them the potential for disrupting funding, collaboration networks, and labs. We hope to exchange ideas about how these institutional investigations work from the inside—not often understood from the outside. The panelists will discuss the kinds of issues and challenges they most often find. They will also discuss the role that technology could play in their work.

Panel: Publishers

Publishers play a crucial role as the last line of defense against scientific error and potential misconduct. This panel will discuss the role publishers have in deterring and detecting potential issues. They may also discuss how they seek clarification from authors, corrections, or retractions. This panel will discuss the role that human investigators play and the tools they use or do not use.

Pre-conference social

Session for open discussion among the attendees. Also, a time for checking that your equipment works.

Image duplication detection tools — insights from a human spotter

Despite peer-review and editorial screening, science papers can still contain images or other data of concern. A visual scan of 20,000 papers published in 40 biomedical journals showed that 4% contained inappropriately duplicated images. Papers containing incorrect or even falsified data could lead to wasted time and money spent by other researchers trying to reproduce those results. Thorough image screening before publication would be beneficial for editors, publishers, and readers, and act as a deterrent for fraudulent submissions. There is a great need for high-throughput computational tools to find image duplications and manipulations in scientific manuscripts, and to help detect the growing number of fabricated manuscripts produced by paper mills. Elisabeth Bik will present some case examples, insights, and challenges that she has encountered as a human visual duplication detector.

Responsible Use of Support Tools for Plagiarism Detection

Many academic institutions are of the opinion that they can simply solve the problem of plagiarism by purchasing the use of so-called plagiarism detection software. But as a recent test of such support tools shows, the systems don't find all plagiarism and will report text overlap that is not plagiarism as if it were. Institutions that rely only on some similarity measure for determining sanctions need to be aware of how meaningless the numbers these systems report are.

In this talk the results of the recent test of support tools for detecting plagiarism conducted by the European Network of Academic Integrity will be presented, followed by a discussion of what constitutes the responsible use of such tools.

Roles and Responsibilities for Promoting Research Integrity

Dr. Lauer, the NIH Deputy Director for Extramural Research, will discuss one funding agency’s perspectives on the roles and responsibilities of different stakeholders in addressing varying types of research and professional misconduct.

Challenges and Approaches to Media Integrity

Advances in machine learning technologies have led to the rapid proliferation of automated and semi-automated media (image, video, text, audio) manipulation and synthesis technologies. These technologies make it easier for unskilled individuals to create compelling media manipulations and may reduce our trust in many forms of information. DARPA has made significant investments in the development of media forensics tools that can help defend against such falsified media. This talk will discuss trends in media falsification technologies and the work that has been developed by DARPA programs to defend against falsified media.

Contributed presentations

12:30 PM - 12:50 PM:
Zubair Afzal, Marleen Sta, Marialaura Martinico, Daniel Gregory, Lekhraj Sharma, Ramsundhar Baskaravelu and George Tsatsaronis, "Improving reproducibility by automating key resource tables"

12:55 PM - 01:15 PM:
Colby Vorland, David Allison and Andrew Brown, "Semi-automated Screening for Improbable Randomization in PDFs"

Panel: Funders

This panel will provide a funder’s perspective on research integrity. One view is on the investigative role of research integrity issues. The panel may discuss how they investigate research integrity problems within their organization or organizations they oversee. Cases of scientific errors or misconduct not only severely impact science but also might produce significant losses to the funder. Some funders engage in direct research efforts to understand research misconduct and educate scientists and institutions about it. Other funders see research integrity and forensic science in general as a more fundamental long-term threat to the foundations of science and even as a tool used by adversaries. In this sense, research integrity applications go beyond scientific research and might percolate into other society areas, such as fake news generation and deception capabilities.

Competition launch: Artificial Intelligence for Computational Research Integrity

Numerous cases of scientific mistakes or even misconduct frequently appear in the media. Scientists, publishers, and institutions care about these issues but are overwhelmed by the number of submissions: reviewing them is mostly done manually in a time consuming and costly manner. This presentation will suggest that we can use artificial intelligence (AI) to improve how publications are filtered at an unprecedented scale and accuracy levels. The key to achieving these goals is to take a page from AI’s book: create high-quality, large-scale datasets of common scientific errors or potential misconduct that modern AI techniques can use to learn from and recognize. Akin to how AI has revolutionized object recognition, speech recognition, and information retrieval, this new research integrity era, which we call computational research integrity, promises to revolutionize how we detect errors before they get published. I will the research integrity competition to spearhead these new ideas.

Panel: Tool Developers

One concrete result of computational approaches in research integrity will depend on the development of tools. We all have become used to text plagiarism detection, but researchers could significantly expand these kinds of tools to other media modalities. This panel has experts who have identified errors and potential misconduct in images, genetic sequences, and statistical data. This panel will discuss current challenges and opportunities in tool development, especially regarding why we do not use them more widely and what's missing.

PubPeer, past, present and future

The PubPeer website, dedicated to facilitating public discussion of the scientific literature, launched in late 2012. Using open metadata, it creates a dynamic page for every publication with a Digital Object Identifier (DOI) or arXiv ID. Users can post and read comments about each publication. This short-circuits the many potentially conflicted actors stifling the correction of science: authors, journals and institutions. By offering strong anonymity, regulated by strict content guidelines and moderation, PubPeer has created a protected space where even serious criticism could be aired without risk of professional or legal reprisals. The site has helped reveal an unsuspected volume of research misconduct. Users can be alerted to discussion about articles as they work by installing the PubPeer browser and Zotero plugins (which you should all do). Taking stock after 8 years of operation, we stand by our sometimes-controversial decision to allow regulated anonymous commenting, which we believe has reinforced what we see as the primary function of PubPeer - to serve as an early-warning system for readers and users of articles. Official procedures for correcting science remain too slow, unreliable and inefficient; this was especially apparent during the pandemic. We aim to continue integrating sources of high-quality discussion, with recent innovations including comments from preprint servers, scientific Twitter and a sister site for overlay journals called Peeriodicals. Journals and institutions can receive tailored alerts via "PubPeer Dashboards". Our guiding philosophy for the future is to accelerate scientific progress by facilitating rapid, effective and public exchange of discussion about publications, while eschewing all metrics and focusing on the substance of scientific publications.

Understanding the Provenance of Visual Disinformation Targeting Science

The COVID-19 pandemic has attracted significant attention to scientific matters related to the cause, treatment, and prevention of the disease that has upended our lives. Alarmingly, not all of the information available on the Internet is what it appears to be. Deceptive memes, bogus ads, and fabricated infographics are proliferating, with all threatening to undermine the public's trust in science. Given the vast scale of the problem, an automated capability that can identify new instances of visual disinformation, trace its origin, and ultimately flag it as being problematic is needed. But compared to text, visual content presents unique challenges for media forensics. This talk presents an end-to-end processing pipeline for image provenance analysis, which works at real-world scale. It employs a cutting-edge image filtering solution that is able to find related images, as well as novel techniques for obtaining a provenance graph that expresses how the images, as nodes, are ancestrally connected. Building from provenance analysis, the talk goes on to introduce a scalable automated visual recognition pipeline for discovering meme genres of diverse appearance. This pipeline can ingest meme images from a social network, apply computer vision-based techniques to extract features and index new images into a database, and then organize the memes into related genres. Recent examples of visual disinformation targeting science will be highlighted, including repurposed imagery, parasitic advertising, and pandemic-related memes. Finally, the talk will conclude with thoughts on continued research in this direction.

Ignorance or mimicry? Lessons from the merchants of doubt

Agnotology – a new field dedicated to the study of ignorance -- has provided crucial insights into the corporate staging and management of scientific controversies on topics of great public importance ranging from the dangers of tobacco smoking to global warming, and others in between. (The book and eponymous film Merchants of Doubts or the work of Robert Proctor are examples). A shared goal of these corporate interventions, the agnotologists have argued, is the questioning of scientific evidence and claims so as to create the appearance that these issues are not matters of remarkably solid consensus but, to the contrary, the focus of substantial controversies among scientists themselves. The effect of such misinformation campaigns has been to splinter or confuse the public’s response, dangerously delaying urgent regulatory interventions.

But while the agnotologists’ goal has been to study the production of non-knowledge or the destruction of knowledge -- from the feeding of false beliefs to the public to actively creating gaps in the availability of knowledge by making it secret -- their most original and interesting contribution has been the thick description of the techniques, strategies, and rhetoric of the production of doubt about science and medicine – something that is exceptionally effective precisely because it is neither truth nor falsehood, neither knowledge nor ignorance. Its power is its elusiveness.

Adding new examples to those exposed by the agnotologists, I argue that the production of doubt is not the production of ignorance but an essentially different regime that, while coming to life in the traditional form of scientific controversies, it does not aim at producing or manipulating the content of knowledge but at strategically subverting the norms of science as a profession. The production of doubt is neither the negative mirror image of the production of positive knowledge, nor simply a scientific Potemkin village where politics is camouflaged as science. Given the extraordinarily high stakes involved, it may be comforting to believe that, while highly damaging to the credibility of science, the systematic production of doubt is also completely external to it. It is politics, not science. I share the agnotologists’ sentiments and commitments, and believe that science and the corporate production of doubt are driven by radically different goals. Sadly, however, they are not easily separable in a formal, conceptual sense. The strategic production of doubt is a very specific form of parasitism that cannot exist outside of its host. Its effectiveness derives only from mobilizing the authority of science against itself.

Image Manipulation Detection — From Visual Inspection to Technology Driven Procedures?

This presentation asks whether and how technical tools can help facilitate the inspection of images and what steps need to be taken to support tool development. We will discuss some of the factors that speak for and against the use of technology and have a look at the image integrity database (IIDB), which we have build in Berlin to provide training data for algorithm development.

Contributed presentations

12:30 PM - 12:50 PM:
Yury Kashnitsky, Vaishnavi Kandala, Egbert Wezenbeek Van, Ijsbrand Jan Aalbersberg, Catriona Fennell and Georgios Tsatsaronis, "How near-duplicate detection improves editors' and authors' publishing experience"

From Cancer to COVID-19, Does Science Self-Correct?

Rapid publication of results — particularly on preprint servers — has grown dramatically during the COVID-19 pandemic, and has forced researchers, health care professionals, journalists, and others to grapple with the concept of reliable and actionable information. The pandemic has given rise to more than 80 retractions at the time of this writing. Is that cause for concern? My lens for this talk will be ten years of experience reporting on retractions for Retraction Watch, including creating the world’s most comprehensive database of retractions, with close to 24,000 and counting.

Panel: Journalists

Journalists have uncovered many scientific integrity cases that have helped us understand the extent and gravity of research misconduct. The website Retraction Watch and PubPeer have become the place where these investigations are heavily discussed, and evidence presented. They have provided invaluable tools to the scientific community—without much official recognition. This panel discusses journalism's role in research integrity and how tools may help their investigations.

Panel: Investigators/whistleblowers

There is no doubt that the whistleblower and private investigators' role has become increasingly crucial for vetting and correcting scientific research. In this panel, the members will discuss their role in research integrity, their motivations, approaches, and threats they face. The panel may also discuss how bad actors are adapting and creating new types of misconduct that are becoming impossible to catch on time or entirely invisible.