2024年8月9日 · To address this challenge, we design a joint multimodal scene graph for charts to explicitly represent the relationships between chart elements and their patterns. Our proposed multimodal scene graph includes a visual graph and a textual graph to jointly capture the structural and semantical knowledge from the chart.

2024年12月21日 · 指南涵盖了环境搭建、数据集准备、模型推理和训练的流程： 1. 克隆项目代码. 首先从官方仓库克隆代码： cd MSG. 2. 配置运行环境. 项目提供了两种方式来设置运行环境，推荐使用 environment.yml 方法，以确保完整的环境依赖。 如果想搭建最小依赖环境： 如果想完全复现官方环境： 提示：第二种方法可以避免遗漏依赖项，推荐使用。 如果遇到需要激活. source ~/.bashrc. 3. 数据集准备. 官方数据集基于 Apple 的 ARKitScenes 转换而来，需要从 …

2024年10月17日 · MSG data is converted from Apple's ARKitScenes by transforming its 3D annotations to 2D. The converted dataset can be found at this Dataset Hub on Huggingface. We have also kept the code snippets for data convertion in data_preprocess. To use the data, download and unzip the data to ./data/msg TODO: specify the data usage.

2024年10月15日 · In this work, we propose to build Multiview Scene Graphs (MSG) from unposed images, representing a scene topologically with interconnected place and object nodes.

This research proposes a novel multimodal scene graph, including a visual graph and a textual graph, to capture the structure and semantic information from charts.

2024年10月21日 · To address this challenge, we design a joint multimodal scene graph for charts to explicitly represent the relationships between chart elements and their patterns. Our proposed multimodal scene graph includes a visual graph and a textual graph to jointly capture the structural and semantical knowledge from the chart.

2024年8月9日 · To address this challenge, we design a joint multimodal scene graph for charts to explicitly represent the relationships between chart elements and their patterns. Our proposed multimodal scene graph includes a visual graph and a textual graph to jointly capture the structural and semantical knowledge from the chart.

提出的多模态场景图通过视觉图和文本图共同捕捉图表的结构和语义知识，显著提高了对图表元素的理解，进而在图表问答基准测试中表现优异。 Automatic Chart Question Answering (ChartQA) is challenging due to the complex distribution of chart elements with patterns of the underlying data not explicitly displayed in charts. To address this challenge, we design a joint. 本研究解决了自动图表问答中图表元素的复杂分布及数据模式难以识别的问题。

2025年4月10日 · To tackle this challenge, we introduce a graph-enhanced LLM with multi-scale retrieval (MSG-LLM). It captures similar graph structures and semantics across graphs at different scales and bridges the graph alignment across multiple scales.

To address this challenge, we design a joint multimodal scene graph for charts to explicitly represent the relationships between chart elements and their patterns. Our proposed …