Abstract
This work proposes the use of a fine-tuned Transformers-based Natural Language
Processing (NLP) model called BERTimbau to generate the word embeddings from texts
published in a Brazilian newspaper, to create a robust NLP model to classify news in
Portuguese, a task that is costly for humans to perform for big amounts of data. To assess
this approach, besides the generation of embeddings by the fine-tuned BERTimbau, a
comparative analysis was conducted using the Word2Vec technique. The first step of the
work was to rearrange news from nineteen to ten categories to reduce the existence of class
imbalance in the corpus, using the K-means and TF-IDF techniques. In the Word2Vec step,
the CBOW and Skip-gram architectures were applied. In BERTimbau and Word2Vec steps,
the Doc2Vec method was used to represent each news as a unique embedding, generating
a document embedding for each news. Metrics accuracy, weighted accuracy, precision,
recall, F1-Score, AUC ROC and AUC PRC were applied to evaluate the results. It was
noticed that the fine-tuned BERTimbau captured distinctions in the texts of the different
categories, showing that the classification model based on this model has a superior
performance than the other explored techniques.