Sentiment Analysis for Software Engineering; A Study on the Effectiveness of Data Augmentation and Ensembling using Transformer-based Models

Abstract

Sentiment analysis for software engineering has undergone much research to efficiently develop tools and approaches to classify sentiment polarity for software engineering contents. It started with customized tools based on lexicon and supervised approaches like SentiStrength-SE, SentiCR, and Senti4SD. Pre-trained transformer-based models like BERT, RoBERTa, and XLNet have later outperformed the tools. These models give an improved classification of sentiment polarities for software engineering content when fine-tuned on SE-specific datasets. Although the performance of these models is much better than previously existing tools, there is still much room for improvement, and that is what we have demonstrated in this work. We use three pre-trained transformer-based models on four gold-standard SE-specific datasets and ensemble the models to show the improvement of the ensemble approach over the individual pre-trained transformer-based models. We use two key metrics to assess performance: weighted-average F1 scores and macro-average F1 scores. We also apply text augmentation on the datasets that have some issues like small size and class imbalance and then evaluate the performance of our approaches on the augmented datasets as well. Our results show that the ensemble models outperform the pre-trained transformer-based models on the original datasets and that data augmentation further improve the performance of all the approaches used in the work.