publications

2025

1. IEEE Access

   Multimodal Classification of Social Media Disaster Posts With Graph Neural Networks and Few-Shot Learning

   J. Nascimento, P. Bestagini, and A. Rocha

   IEEE Access, 2025

   Abs Bib Code

   Social media has emerged during the last decade as a potential information source in crisis scenarios, providing data in real-time or just after the occurrence of the event. Nevertheless, current social media data acquisition procedures result in datasets, presenting myriad non-informative content, which hinders posterior analyses. While previous studies have used machine learning to address this issue, they typically require many labeled examples, hardening their use in a real-world scenario. Moreover, social media posts tend to be multimodal, which adds complexity to how these data should be represented. This paper extends upon our previous work and presents a new method for identifying the most informative content related to an event in textual and visual data through few-shot learning. The results show that this method outperforms existing approaches in both performance and efficiency, offering a valuable solution for a timely analysis of crisis-related social media data and advancing research in this area.

   @article{11136138,
     author = {Nascimento, J. and Bestagini, P. and Rocha, A.},
     journal = {IEEE Access},
     title = {Multimodal Classification of Social Media Disaster Posts With Graph Neural Networks and Few-Shot Learning},
     year = {2025},
     volume = {13},
     number = {},
     pages = {150168-150178},
     keywords = {Social networking (online);Pipelines;Vectors;Principal component analysis;Graph neural networks;Disasters;Training;Visualization;Few shot learning;Feature extraction;Semi-supervised learning;crisis informatics;multimodal classification;few-shot learning;social media},
     doi = {10.1109/ACCESS.2025.3602364}
   }

2024

1. WIFS

   Interactive Event Sifting using Bayesian Graph Neural Networks

   J. Nascimento, N. Jacobs, and A. Rocha

   In 2024 IEEE International Workshop on Information Forensics and Security (WIFS), 2024

   Abs arXiv Bib Code

   Forensic analysts often use social media imagery and texts to understand important events. A primary challenge is the initial sifting of irrelevant posts. This work introduces an interactive process for training an event-centric, learning-based multimodal classification model that automates sanitization. We propose a method based on Bayesian Graph Neural Networks (BGNNs) and evaluate active learning and pseudo-labeling formulations to reduce the number of posts the analyst must manually annotate. Our results indicate that BGNNs are useful for social-media data sifting for forensics investigations of events of interest, the value of active learning and pseudo-labeling varies based on the setting, and incorporating unlabelled data from other events improves performance.

   @inproceedings{nascimento2024,
     title = {Interactive Event Sifting using Bayesian Graph Neural Networks},
     author = {Nascimento, J. and Jacobs, N. and Rocha, A.},
     booktitle = {2024 IEEE International Workshop on Information Forensics and Security (WIFS)},
     year = {2024},
     organization = {IEEE},
   }

2. AI Knows What You Did Last Summer: Applications in Digital Forensics

   J. Yang, J. Nascimento, G. Bertocco, and 5 more authors

   In Computer Vision: Challenges, Trends, and Opportunities, 2024

   Abs Bib

   The past decade saw an explosive advancement in smartphones and cameras, making them even more powerful and cheaper. As a result, not only can we record high-quality footage of the world around us, but their increased accessibility makes it rare that a significant event goes unnoticed. Furthermore, once something is captured, it can be uploaded instantly to social media and accessed by millions of people. Even though this connectivity has multiple positive effects, social media also raised new challenges in our society, such as the spread of fake news and misinformation. This scenario has created a range of new problems that Digital Forensics is concerned about, from analyzing major events in the world — such as, terrorist attempts, wars, and natural disasters — to understanding how these are perceived and discussed online. To cope with the scale of such problems, they are often approached by powerful artificial intelligence (AI) techniques developed during the last decades. This chapter presents modern applications of AI in the Digital Forensics field. Person re-identification (PReID), visual analytics, fact-checking, event filtering, and authorship attribution are just a few examples of forensic tasks that have been leveraged from recent advances in AI research in this new scenario. We discuss each task’s associated problems, challenges, current methods, and future research directions and hope to give the reader a general idea of the challenges of the field and its imperative demands for new tools and smart techniques to face the challenges of the 21st century.

   @incollection{inbook,
     author = {Yang, J. and Nascimento, J. and Bertocco, G. and Theophilo, A. and Padilha, R. and Soriano Vargas, A. and Andaló, F. and Rocha, A.},
     booktitle = {Computer Vision: Challenges, Trends, and Opportunities},
     publisher = {CRC Press},
     year = {2024},
     pages = {82-108},
     title = {AI Knows What You Did Last Summer: Applications in Digital Forensics},
     isbn = {9781003328957},
     doi = {10.1201/9781003328957-5},
   }

2023

1. Real-world-events data sifting through ultra-small labeled datasets and graph fusion

   D. Vega-Oliveros, J. Nascimento, B. Lavi, and 1 more author

   Applied Soft Computing, 2023

   Abs Bib

   The information on social media is vital, especially for events such as natural disasters or terrorist attacks, that might cause rapid growth of data sharing through social media networks. However, collecting and processing data of an event is a challenging task and essentially requires a great deal of data cleaning and filtering out what is relevant/irrelevant to the event. Data sifting task endeavors to identifying the related content to the depicted event data. We propose a learning strategy to dynamically learn complementary contributions from different data-driven features through a semi-supervised graph-fusion technique. Our proposed method relies upon minimal training labeled data samples —  ultra-small data learning. Learning through a small labeled set is also of particular interest to forensic investigators and medical researchers — concerning massive data labeling and minimizing energy-efficient computing to reduce redundancy and repetitions. We assess the effectiveness of the proposed semi-supervised method on five datasets from real-world events. Compared with prior-art (supervised and semi-supervised ones), experimental results show the proposed method achieves the best classification results and most efficient computational footprint.

   @article{VEGAOLIVEROS20231098652,
     title = {Real-world-events data sifting through ultra-small labeled datasets and graph fusion},
     journal = {Applied Soft Computing},
     volume = {132},
     pages = {109865},
     year = {2023},
     issn = {1568-4946},
     doi = {https://doi.org/10.1016/j.asoc.2022.109865},
     url = {https://www.sciencedirect.com/science/article/pii/S1568494622009140},
     author = {Vega-Oliveros, D. and Nascimento, J. and Lavi, B. and Rocha, A.},
     keywords = {Ultra-small classification, Data sifting, Semi-supervised learning, Graph-fusion learning}
   }

2022

1. WIFS

   Few-shot Learning for Multi-modal Social Media Event Filtering

   J. Nascimento, J. P. Cardenuto, J. Yang, and 1 more author

   In 2022 IEEE International Workshop on Information Forensics and Security (WIFS), 2022

   Abs arXiv Bib Code

   When a forensic event of large scale happens, immediately visual content related to it is shared on social networks. These data might be potentially useful for a posterior forensic inspection, given that they can depict different views in different moments of the event. However, an analysis of social media imagery related to an event might be harmed by the abundant number of irrelevant items that are retrieved by a collection procedure, such as memes and images from previous events. Manually sanitizing the dataset at hand is unfeasible, since it might contain thousands of items. To tackle this problem, we study the employment of machine learning techniques to speed up the procedure and reduce the required human force. In detail, our work follows four paths. The first one aims to provide a good representation for images, exploring different pre-trained convolutional neural networks and feature fusion; the second one targets including humans to the loop of the machine learning pipeline, by employing instance selection and active learning techniques; the goal of the third one is to perform classification with few samples, using semi-supervised techniques that vary from graph-based methods to graph neural networks; and the last one aims to incorporate knowledge from previous events to the machine learning pipeline, using domain adaptation and available datasets from previous events. These four paths are promising, and can improve the performance of methods for this task.

   @inproceedings{nascimento2022,
     title = {Few-shot Learning for Multi-modal Social Media Event Filtering},
     author = {Nascimento, J. and Cardenuto, J. P. and Yang, J. and Rocha, A.},
     booktitle = {2022 IEEE International Workshop on Information Forensics and Security (WIFS)},
     year = {2022},
     organization = {IEEE},
   }

2021

1. A Inteligência Artificial e os desafios da Ciência Forense Digital no século XXI

   R. Padilha, A. Theóphilo, F. A. Andaló, and 6 more authors

   Estudos Avançados, 2021

   Abs Bib

   A Ciência Forense Digital surgiu da necessidade de tratar problemas forenses na era digital. Seu mais recente desafio está relacionado ao surgimento das mídias sociais, intensificado pelos avanços da Inteligência Artificial. A produção massiva de dados nas mídias sociais tornou a análise forense mais complexa, especialmente pelo aperfeiçoamento de modelos computacionais capazes de gerar conteúdo artificial com alto realismo. Assim, tem-se a necessidade da aplicação de técnicas de Inteligência Artificial para tratar esse imenso volume de informação. Neste artigo, apresentamos desafios e oportunidades associados à aplicação dessas técnicas, além de fornecer exemplos de seu uso em situações reais. Discutimos os problemas que surgem em contextos sensíveis e como a comunidade científica tem abordado esses tópicos. Por fim, delineamos futuros caminhos de pesquisa a serem explorados.

   @article{padilha2021inteligencia,
     title = {A Intelig{\^e}ncia Artificial e os desafios da Ci{\^e}ncia Forense Digital no s{\'e}culo XXI},
     author = {Padilha, R. and The{\'o}philo, A. and Andal{\'o}, F. A. and Vega-Oliveros, D. A. and Cardenuto, J. P. and Bertocco, G. and Nascimento, J. and Yang, J. and Rocha, A.},
     journal = {Estudos Avan{\c{c}}ados},
     volume = {35},
     pages = {113--138},
     year = {2021},
     publisher = {SciELO Brasil}
   }

2. ICASSP

   Semi-Supervised Feature Embedding for Data Sanitization in Real-World Events

   B. Lavi, J. Nascimento, and A. Rocha

   In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021

   Abs Bib

   With the rapid growth of data sharing through social media networks, determining relevant data items concerning a particular subject becomes paramount. We address the issue of establishing which images represent an event of interest through a semi-supervised learning technique. The method learns consistent and shared features related to an event (from a small set of examples) to propagate them to an unlabeled set. We investigate the behavior of five image feature representations considering low- and high-level features and their combinations. We evaluate the effectiveness of the feature embedding approach on five collected datasets from real-world events.

   @inproceedings{lavi2021semi,
     title = {Semi-Supervised Feature Embedding for Data Sanitization in Real-World Events},
     author = {Lavi, B. and Nascimento, J. and Rocha, A.},
     booktitle = {2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
     pages = {2495--2499},
     year = {2021},
     organization = {IEEE}
   }