Ильвовский Дмитрий Алексеевич

SynEL: A synthetic benchmark for entity linking

2026 · ARTICLE · en

Large language models (LLMs) offer significant potential for constructing commonsense knowledge graphs from text, demonstrating adaptability across diverse domains. However, their effectiveness varies significantly with domain-specific language, highlighting a critical need for specialized benchmarks to assess and optimize knowledge graph construction sub-tasks like named entity recognition, relation extraction, and entity linking. Currently, domain-specific benchmarks are scarce. To address this gap, we introduce SynEL, a novel benchmark developed for evaluating text-based knowledge extraction methods, validated using customer support dialogues. We present a comprehensive methodology for benchmark construction, propose two distinct approaches for generating synthetic datasets, and evaluate accumulated hallucinations. Our experiments reveal that existing LLMs experience a significant performance drop, with micro-F1 scores decreasing by up to 25 absolute points when extracting low-resource entities compared to high-resource entities from sources like Wikipedia. Furthermore, by incorporating synthetic datasets into the training process, we achieved an improvement in micro-F1 scores of up to 10 absolute points. We publicly release our benchmark and generation code to demonstrate its utility for fine-tuning and evaluating LLMs.

Enhancing FEVER-Style Claim Fact-Checking Against Wikipedia: A Diagnostic Taxonomy and a Generative Framework

2025 · CHAPTER · en

Fact-checking is a crucial yet challenging task that continues to gain importance. In an effort to address this issue, the FEVER large-scale dataset was developed to facilitate evidence-based fact-checking using Wikipedia as a reference. Despite numerous proposed approaches and evaluations on this dataset, a comprehensive understanding of the errors made by these approaches is still lacking. Here, we aim to bridge this gap. We introduce a diagnostic taxonomy and a generative framework to enhance FEVER-style fact-checking. We establish a taxonomy of errors and we construct a diagnostic dataset that enables the analysis of the errors made by state-of-the-art models as well as their distribution within the FEVER dataset. Additionally, we provide a set of prompts to generate examples within this taxonomy. Our experiments demonstrate promising results through the utilization of these generated examples for fine-tuning.

Enhancing RAG and Knowledge Graphs with Discourse

2025 · CHAPTER · en

We consider a number of Retrieval Augmented Generation (RAG) architectures to address a lack of specific information and hallucination issues of Large Language Models (LLM)—based question answering. We start with conformal prediction which acts on top of LLM and maintains a set of generations instead of a single one and attempts to find the best element of this set, which is assumed to be the “most average one”. We then proceed to LLM self-reflection series of RAG architectures predicting the multi-hop question answering session before actual search for an answer. After that, we propose a mechanism for LLM to filter out answers inappropriate with respect to style. All these components need discourse-level analysis for more robust functioning. Knowledge graph (KG) and Abstract Meaning Representation (AMR)-based knowledge graph construction follow. We evaluate the contribution of all of these components to overall answer relevance and also zoom in on the role of discourse-based subsystem in each of these components. There is a substantial improvement of performance due to the fourcomponent architecture introduced in this paper; the contribution of discourse-based subsystems is fairly modest.

Unleashing the Power of Discourse-Enhanced Transformers for Propaganda Detection

2024 · CHAPTER · en

GroundHog: Dialogue Generation using Multi-Grained Linguistic Input

2024 · CHAPTER · en

Recent language models have significantly boosted conversational AI by enabling fast and cost-effective response generation in dialogue systems. However, dialogue systems based on neural generative approaches often lack truthfulness, reliability, and the ability to analyze the dialogue flow needed for smooth and consistent conversations with users. To address these issues, we introduce GroundHog, a modified BART architecture, to capture long multi-grained inputs gathered from various factual and linguistic sources, such as Abstract Meaning Representation, discourse relations, sentiment, and grounding information. For experiments, we present an automatically collected dataset from Reddit that includes multi-party conversations devoted to movies and TV series. The evaluation encompasses both automatic evaluation metrics and human evaluation. The obtained results demonstrate that using several linguistic inputs has the potential to enhance dialogue consistency, meaningfulness, and overall generation quality, even for automatically annotated data. We also provide an analysis that highlights the importance of individual linguistic features in interpreting the observed enhancements.

ZenPropaganda: A Comprehensive Study on Identifying Propaganda Techniques in Russian Coronavirus-Related Media

2024 · CHAPTER · en

The topic of automatic detection of manipulation and propaganda in the media is not a novel issue; however, it remains an urgent concern that necessitates continuous research focus. The topic is studied within the framework of various papers, competitions and shared tasks, which provide different techniques definitions and include the analysis of text data, images, as well as multi-lingual sources. In this study, we propose a novel multi-level classification scheme for identifying propaganda techniques. We introduce a new Russian dataset ZenPropaganda consisting of coronavirus-related texts collected from Vkontakte and Yandex. Zen platforms, which have been expertly annotated with fine-grained labeling of manipulative spans. We further conduct a comprehensive analysis by comparing our dataset with existing related ones and evaluate the performance of state-of-the-art approaches that have been proposed for them. Furthermore, we provide a detailed discussion of our findings, which can serve as a valuable resource for future research in this field.

Truth-O-Meter: Handling Multiple Inconsistent Sources Repairing LLM Hallucinations

2024 · CHAPTER · en

Large Language Models (LLM) often produce text with incorrect facts and hallucinations. To address this issue, we developed a fact-checking system Truth-O-Meter which verifies LLM results on the Internet and other sources of information to detect wrong claims/facts and proposes corrections for them. NLP and reasoning techniques such as Abstract Meaning Representation and syntactic alignment are applied to match hallucinating sentences with truthful ones. To handle inconsistent sources while fact-checking, we rely on argumentation analysis in the form of defeasible logic programming, selecting the most authoritative source. Our evaluation shows that LLM content can be substantially improved for factual correctness and meaningfulness on an industrial scale.

Усовершенствование классификации научных событий с помощью кластеризации смежных областей исследований

2024 · CHAPTER · ru

Классификация научных событий – это сложная задача, требующая точного определения и назначения тематических категорий. Задача усложняется, когда встречаются общие или неточные категории, не отражающие суть конкретной научной дисциплины. В нашем исследовании представлена методология, базирующаяся на принципах кластеризации областей исследований (FOS). Этот подход позволил значительно улучшить процесс классификации научных мероприятий, обеспечивая более точное и полное представление каждого события. Таким образом, данная методология предлагает ценные возможности для ученых, научных сообществ и организаторов мероприятий, способствуя более эффективному управлению и анализу научной информации.

Transformer-based Multi-Party Conversation Generation using Dialogue Discourse Acts Planning

2023 · CHAPTER · en

Recent transformer-based approaches to multi-party conversation generation may produce syntactically coherent but discursively inconsistent dialogues in some cases. To address this issue, we propose an approach to integrate a dialogue act planning stage into the end-to-end transformer-based generation pipeline. This approach consists of a transformer fine-tuning procedure based on linearized dialogue representations that include special discourse tokens. The obtained results demonstrate that incorporating discourse tokens into training sequences is sufficient to significantly improve dialogue consistency and overall generation quality. The suggested approach performs well, including for automatically annotated data. Apart from that, it is observed that increasing the weight of the discourse planning task in the loss function accelerates learning convergence.

Multimodal Discourse Trees in Forensic Linguistics

2023 · CHAPTER · en

В работе исследуется концепция построения мультимодального дискурсивного дерева для структурированного представления текста, обогащенного дополнительной информацией из источников различной природы. В более ранних работах были введены понятия коммуникативных дискурсивных деревьев, расширенных с помощью теории речевых актов, а также расширенных дискурсивных деревьев, которые отражают структуру не одного текста, а набора связанных документов; в данной работе мы исследуем возможность расширения дискурсивной структуры за счет включения данных из дополнительных (нетекстовых) модальностей. Мы называем подобное дерево мультимодальным дискурсивным деревом и показываем, что отношения, которые можно установить между частями текста (дискурсивными единицами), также переносятся на данные, дополняющие текст, к которым можно отнести записи из баз данных (например, истории вебпоиска или финансовых операций и т.д.). Мы показываем, что построение мультимодального дискурсивного дерева помогает улучшить качество решения задач поиска на примере анализа судебных документов, которые в большинстве случаев сопровождаются информацией из различных дополнительных источников, по сравнению с поиском по ключевым словам или поиском по стандартному (текстовому) дискурсивному дереву.