FEVER 3 Workshop

Project Debater is the first AI system that can meaningfully debate a human opponent. The system, an IBM Grand Challenge, is designed to build coherent, convincing speeches on its own, as well as provide rebuttals to the opponent's main arguments. In February 2019, Project Debater competed against Harish Natarajan, who holds the world record for most debate victories, in an event held in San Francisco that was broadcasted live world-wide. In this talk I will tell the story of Project Debater, from conception to a climatic final event, describe its underlying technology, and discuss how it can be leveraged for advancing decision making and critical thinking.

Automatic fact checking is one of the more involved NLP tasks currently researched: not only does it require sentence understanding, but also an understanding of how claims relate to evidence documents and world knowledge. Moreover, there is still no common understanding in the automatic fact checking community of how the subtasks of fact checking — claim check-worthiness detection, evidence retrieval, veracity prediction — should be framed. This is partly owing to the complexity of the task, despite efforts to formalise the task of fact checking through the development of benchmark datasets. The first part of the talk will be on automatically generating textual explanations for fact checking, thereby exposing some of the reasoning processes these models follow. The second part of the talk will be on re-examining how claim check-worthiness is defined, and how check-worthy claims can be detected; followed by how to automatically generate claims which are hard to fact-check automatically.

Our society is struggling with an unprecedented amount of falsehood, hyperbole, and half-truths. Politicians and organizations repeatedly make false claims that jeopardize the integrity of journalism. Disinformation now floods the cyberspace and influences many events on and offline. To fight false information, the need for automatic fact verification has never been so urgent. Existing studies primarily focus on free-form text as evidence crawled from Wikipedia or News websites. The direction of using semi-structured knowledge as evidence like relational tables has yet to be explored systematically. In this talk, we will mainly focus on introducing a new benchmark dataset called TabFact, which allows us to systematically study the fact verification problem under semi-structured tables as evidence.

Significant progress has been made recently in using NLP techniques to identify facts and the relationships between them. In this talk, I will argue that in formulating problems related to the assessment of truth, it is important to take into account the human process of interpretation. This potentially motivates a shift in thinking about fact extraction, from a problem that is fundamentally about engineering and addressed by NLP and machine learning, to a richer combination of engineering and scientific inquiry that overlaps more significantly with questions in the social and cognitive sciences.

Majority of previous studies on task-oriented dialogue systems are restricted to a limited coverage of APIs related to the set of tasks considered in the application domain. However, users oftentimes have domain related requests that are not covered by these APIs, even for their task-focused intents. To enable natural interactions with machines, we propose to expand the coverage of task-oriented dialogue systems by incorporating external, unstructured knowledge sources, such as web documents related to the task domain. We recently introduced an augmented version of MultiWOZ 2.1 multi-domain task-oriented dialogue corpus, which includes sub-dialogues of out-of-API-coverage turns and responses grounded on external knowledge sources. In this talk, I’ll review our work in this area and summarize our initial findings, focusing on challenges related to factual accuracy.

Are fake fake news as bad as real fake news? Are all fakes news bad and all real news good? Is fact-checking all we need? In the first part of the talk, I will present our recent study that investigates the extent to which state-of-the-art neural language models can generate fake news and learn to sort them out from real news. Next, I will share our recent efforts to reason about malicious intents behind human manipulated images. I will conclude the talk by returning to the three questions posed above in light of three levels of analysis of fake news: perceptual (i.e., distributional fingerprints), semantic (i.e., fact-checking), and pragmatic (i.e., malicious vs benign intent) and open the floor for discussion.