José Nascimento

2025

1. IEEE Access

   Multimodal Classification of Social Media Disaster Posts With Graph Neural Networks and Few-Shot Learning

   J. Nascimento, P. Bestagini, and A. Rocha

   IEEE Access, 2025

   Abs Bib Code

   Social media has emerged during the last decade as a potential information source in crisis scenarios, providing data in real-time or just after the occurrence of the event. Nevertheless, current social media data acquisition procedures result in datasets, presenting myriad non-informative content, which hinders posterior analyses. While previous studies have used machine learning to address this issue, they typically require many labeled examples, hardening their use in a real-world scenario. Moreover, social media posts tend to be multimodal, which adds complexity to how these data should be represented. This paper extends upon our previous work and presents a new method for identifying the most informative content related to an event in textual and visual data through few-shot learning. The results show that this method outperforms existing approaches in both performance and efficiency, offering a valuable solution for a timely analysis of crisis-related social media data and advancing research in this area.

   @article{11136138,
     author = {Nascimento, J. and Bestagini, P. and Rocha, A.},
     journal = {IEEE Access},
     title = {Multimodal Classification of Social Media Disaster Posts With Graph Neural Networks and Few-Shot Learning},
     year = {2025},
     volume = {13},
     number = {},
     pages = {150168-150178},
     keywords = {Social networking (online);Pipelines;Vectors;Principal component analysis;Graph neural networks;Disasters;Training;Visualization;Few shot learning;Feature extraction;Semi-supervised learning;crisis informatics;multimodal classification;few-shot learning;social media},
     doi = {10.1109/ACCESS.2025.3602364}
   }

2024

1. WIFS

   Interactive Event Sifting using Bayesian Graph Neural Networks

   J. Nascimento, N. Jacobs, and A. Rocha

   In 2024 IEEE International Workshop on Information Forensics and Security (WIFS), 2024

   Abs arXiv Bib Code

   Forensic analysts often use social media imagery and texts to understand important events. A primary challenge is the initial sifting of irrelevant posts. This work introduces an interactive process for training an event-centric, learning-based multimodal classification model that automates sanitization. We propose a method based on Bayesian Graph Neural Networks (BGNNs) and evaluate active learning and pseudo-labeling formulations to reduce the number of posts the analyst must manually annotate. Our results indicate that BGNNs are useful for social-media data sifting for forensics investigations of events of interest, the value of active learning and pseudo-labeling varies based on the setting, and incorporating unlabelled data from other events improves performance.

   @inproceedings{nascimento2024,
     title = {Interactive Event Sifting using Bayesian Graph Neural Networks},
     author = {Nascimento, J. and Jacobs, N. and Rocha, A.},
     booktitle = {2024 IEEE International Workshop on Information Forensics and Security (WIFS)},
     year = {2024},
     organization = {IEEE},
   }

2022

1. WIFS

   Few-shot Learning for Multi-modal Social Media Event Filtering

   J. Nascimento, J. P. Cardenuto, J. Yang, and 1 more author

   In 2022 IEEE International Workshop on Information Forensics and Security (WIFS), 2022

   Abs arXiv Bib Code

   When a forensic event of large scale happens, immediately visual content related to it is shared on social networks. These data might be potentially useful for a posterior forensic inspection, given that they can depict different views in different moments of the event. However, an analysis of social media imagery related to an event might be harmed by the abundant number of irrelevant items that are retrieved by a collection procedure, such as memes and images from previous events. Manually sanitizing the dataset at hand is unfeasible, since it might contain thousands of items. To tackle this problem, we study the employment of machine learning techniques to speed up the procedure and reduce the required human force. In detail, our work follows four paths. The first one aims to provide a good representation for images, exploring different pre-trained convolutional neural networks and feature fusion; the second one targets including humans to the loop of the machine learning pipeline, by employing instance selection and active learning techniques; the goal of the third one is to perform classification with few samples, using semi-supervised techniques that vary from graph-based methods to graph neural networks; and the last one aims to incorporate knowledge from previous events to the machine learning pipeline, using domain adaptation and available datasets from previous events. These four paths are promising, and can improve the performance of methods for this task.

   @inproceedings{nascimento2022,
     title = {Few-shot Learning for Multi-modal Social Media Event Filtering},
     author = {Nascimento, J. and Cardenuto, J. P. and Yang, J. and Rocha, A.},
     booktitle = {2022 IEEE International Workshop on Information Forensics and Security (WIFS)},
     year = {2022},
     organization = {IEEE},
   }