Code and data on colour classification in Visual Genome: github
ManyNames: We were very happy to collaborate with the AMORE group for the collection of this dataset for object naming in real-world images github
Projects
INAS (starting soon, duration: 3 years)
Title: Interactive Argumentation Support in the Invasion Biology Domain
Project description: Developing a good, new argument is not an easy task. In real-world argumentation scenarios, arguments presented in texts (e.g. scientific publications) often constitute the end result of a long and tedious process. A lot of work on computational argumentation has focused on detecting, analyzing and aggregating these products of argumentation processes, i.e. argumentative texts. In this project, we adopt a complementary perspective: we aim to develop an argumentation machine that supports users in and during the argumentation process in a scientific context, enabling them to follow ongoing argumentation in a scientific community and to develop their own arguments. To achieve this ambitious goal, we will focus on a particular phase of the scientific argumentation process, namely the initial phase of claim or hypothesis development. In scientific argumentation, a carefully developed and thought-through hypothesis is often crucial for researchers to be able to conduct a successful study and, in the end, present a new, high-quality finding or argument. Thus, an initial hypothesis needs to be specific enough that a researcher can test it based on data, but, at the same time, it should also relate to and extend previous general claims made in the community. In this project, we investigate how argumentation machines can (i) represent concrete and more abstract knowledge on hypotheses and their underlying concepts, (ii) automatically compute semantic relations between hypotheses made in scientific publications, and between hypotheses and datasets, and (iii) interactively support a user in developing her own hypothesis based on these resources. This project will thus combine methods from different disciplines: natural language processing, knowledge representation and semantic web and – as an example for a scientific domain – invasion biology.
Funding: DFG, RATIO SPP
PIs: Sina Zarrieß, Tina Heger, Birgitta König-Ries
NLP4VIS (Nov 2020 - Oct 2023)
Title: A generic conversational interface for scientific data visualization
Project description: The goal of this project is to work towards a generic Natural Language Interface (NLI) that allows users to interact with data and visualizations of data in an intuitive way, via conversational language. Given such a generic NLI, users could enter formally complex queries on their data in natural language (e.g., in a data set on exam gradings: \textit{find me a question that Master students answers significantly better than Bachelor students}), without the need to extensively familiarize with the technical backend of the visualization tool at hand (e.g., programming in \textsf{python/matplotlib}). Compared to traditional interfaces for visualization, NLIs bear the potential to greatly improve usability of existing tools, simplifying and speeding up visual exploration and analysis of scientific data for different user groups.
Project description: In many historical sciences, photographs and other images of architecture serve as a source and basis for subject- and theory-specific investigations. Although AI-based computer vision methods have developed significantly in recent years, they can support the process of source research and criticism in a rudimentary way at best, e.g. for the exploration of image repositories or the retrieval of images. HistKI aims to explore the support and modeling of image source research and criticism as a complex and fundamental historiographical working technique by multimodal AI-based methods. Related sub-questions are: How do historians and other scholars find and evaluate image sources? What generic procedures and sub-problems can be identified for this purpose? How can this be promoted with AI-based approaches? How do AI techniques impact the humanities research process? These questions will be explored using selected scenarios in which images, texts, and 3D models interact synergistically to describe architectural objects and urban ensembles for a process of analysis. With the help of machine learning methods, object sources and text sources (e.g.: captions) will be linked in HistKI in order to allow a detailed contextualization and localization of photographs in the future, thus going a significant step beyond previous methods of distant viewing.
