Index Terms: Sentiment analysis, hate speech detection, text classification, evaluation methodology, calibration, threshold selection, reproducibility, MLOps.

Introduction

Sentiment classification of short social-media text is one of the most frequently reported tasks in applied natural language processing, and comparisons of classical learners with neural networks on Twitter data are abundant [1, 2]. Such comparisons are, however, sensitive to evaluation choices that are rarely controlled. Fitting a vocabulary or an embedding model on the union of training and test data leaks test information into the features [17]; selecting a model on the same split that is reported overstates its score [18]; reporting accuracy on a task with 7 percent positives rewards the majority class; and evaluating probabilistic classifiers at a fixed decision threshold conflates the quality of the ranking they produce with the calibration of their scores [19, 22]. Each of these effects can exceed the differences between methods that papers then rank.

This article grew out of the author’s undergraduate thesis, which compared naive Bayes, logistic regression, a support vector machine, a random forest and gradient-boosted trees on two Twitter corpora in notebook form, with exactly the shortcomings listed above. We re-implement that study as a benchmark in which the evaluation protocol is fixed once and applied to every combination of dataset, representation and model, extend it with the neural models that the thesis proposed as future work and with a third, balanced corpus of long documents, and ship it as a maintained system rather than as a set of scripts. The contributions are:

1. A 56-cell benchmark (three corpora, five representations, eight model families) under one leakage-free protocol: features fitted per cross-validation fold, decision thresholds chosen on out-of-fold predictions, model selection by cross-validated F1, and a single scoring pass on a held-out split (Section 4).

2. An analysis that separates ranking ability from thresholding and calibration. On the imbalanced hate-speech task all eight families are within 0.013 ROC-AUC of each other, yet their positive-class F1 spans 0.076, and the family with the best cross-validated F1 transfers its threshold worst (Section 6).

3. Evidence across the three corpora that sparse \(n\)-gram features remain the strongest representation for classical learners on short text, that learned document embeddings are the weakest representation on every task, and that on balanced long documents linear models on TF-IDF match networks trained from scratch at a fraction of the cost.

4. A reference implementation in which every number in this paper is regenerated from the saved out-of-fold and test probabilities, each run is tracked, the selected model is exported with a model card [28] and a monitoring baseline, and a prediction service with database-backed drift detection turns the benchmark into an operable system [27].

Related Work

Sentiment analysis and opinion mining are surveyed by Pang and Lee [1] and Liu [2]. For Twitter specifically, Go et al. [3] introduced distant supervision with emoticons as noisy labels, a precursor of the lexicon-derived labels used in our vaccine corpus, and Waseem and Hovy [4] and Davidson et al. [5] established racist and sexist tweet detection as a task with strong class imbalance, in which F1 on the minority class is the metric of interest. Classical text classifiers remain competitive baselines: multinomial naive Bayes [6], linear support vector machines [7], random forests [8] and gradient boosting [9]. Wang and Manning [10] showed that linear models over bigram features are hard to beat on sentiment corpora such as IMDB [11], which is consistent with our IMDB results. Distributed representations for text, Word2Vec [12] and Doc2Vec [13], and the two neural architectures we include, the text CNN of Kim [14] and the LSTM [15], are standard. Pretrained transformers [16] set the current ceiling on these corpora; we deliberately restrict the grid to models that train from scratch on a CPU and quantify the remaining gap.

On methodology, Kaufman et al. [17] catalogue leakage, Cawley and Talbot [18] show how model selection on the evaluation data biases reported scores, Lipton et al. [19] prove that the F1-optimal threshold depends on the classifier and the class prior, and Platt [20], Niculescu-Mizil and Caruana [21] and Guo et al. [22] study the calibration of margin-based and neural classifiers; Davis and Goadrich [23] motivate precision-recall analysis under imbalance. On the systems side, Sculley et al. [27] describe the operational debt of machine learning, Mitchell et al. [28] propose model cards, and MLflow [29] provides experiment tracking; population stability indices for drift originate in credit scoring [30].

Datasets

Table 1 summarises the corpora. The first two are the thesis experiments; the third is an addition chosen to contrast short, imbalanced social-media text with long, balanced reviews.

Hate. The Analytics Vidhya Twitter sentiment practice set: 31,962 tweets labelled 1 when racist or sexist, 0 otherwise (2,242 positives). Handles are anonymised as @user in the source; the competition’s unlabelled test file is not used. We split stratified 80/20 (seed 42), giving 25,569 training and 6,393 test tweets.

Vaccine. Pfizer and BioNTech COVID-19 vaccine tweets from Kaggle (11,020 rows). As in the thesis, no human labels exist: after light cleaning and de-duplication (10,353 unique tweets), each tweet is labelled by the sign of its TextBlob [26] polarity as negative, neutral or positive. Scores on this corpus therefore measure agreement with a lexicon labeller rather than with human judgement; we keep it because it is half of the original study and because it exposes how representations interact with lexicon-defined targets.

IMDB. The Large Movie Review Dataset of Maas et al. [11]: 25,000 training and 25,000 test reviews, balanced, with negative defined as a rating of at most 4 and positive as at least 7. We use the official split.

No corpus is redistributed with the code. Download locations are read from the environment, and a manifest records the SHA-256 hash, row count and class balance of the files used for a run, together with a content fingerprint of the loaded frame that is written into every model card.

Methodology

Text normalisation

One transformer implements the thesis recipe and is the first step of every exported pipeline, so training, evaluation and serving apply identical cleaning: HTML unescape and tag removal, lower-casing, removal of URLs and @-handles, retention of letters only (hashtags optionally kept as tokens), a minimum token length, the scikit-learn English stop-word list, and Porter stemming. The hate corpus keeps hashtags and drops tokens shorter than three characters; the vaccine and IMDB corpora drop the hash symbol and keep two-character tokens.

Representations

Five representations are evaluated. Counts and TF-IDF use unigrams and bigrams with document-frequency bounds (minimum 2, maximum 90 percent), a 5,000-term vocabulary for tweets and 20,000 terms for reviews, and sublinear term frequency for TF-IDF. Word2Vec [12] is a skip-gram model (200 dimensions, window 5, 10 negative samples, 20 epochs on tweets and 8 on reviews) trained on the training fold only, with the document vector as the mean of its token vectors. Doc2Vec [13] is a distributed-memory model (200 dimensions, 15 epochs) in which both training and test documents are embedded by inference, so that the two are represented the same way. Sequence denotes padded token identifiers (vocabulary of at most 20,000 tokens for tweets, 30,000 for reviews; 48 tokens for tweets, 200 for the CNN and 160 for the BiLSTM on reviews) consumed by the neural models, whose embedding layer is initialised from a Word2Vec model trained on the same fold.

Models

Classical models use scikit-learn [24] and XGBoost [9]: multinomial naive Bayes (additive smoothing 0.5, count-based features only), logistic regression (L2, \(C{=}1\) on tweets and 2 on reviews, balanced class weights on the Twitter corpora), a linear SVM wrapped in sigmoid calibration with three internal folds [20] so that it exposes probabilities, a random forest (400 trees, balanced subsample weights; 300 trees on IMDB), gradient-boosted trees (histogram method, depth 6, 500 rounds at learning rate 0.05 on tweets; 400 rounds at 0.1 on reviews, subsample 0.9, column subsample 0.5), and a soft-voting ensemble of the logistic regression, the calibrated SVM and XGBoost with equal weights. The thesis’ boosted-tree parameters (depth 8, minimum child weight 6, \(\gamma{=}1.2\)) were tuned on Word2Vec features and under-fit sparse \(n\)-gram matrices; they are retained in the configuration for reference but not used.

The neural models are implemented in PyTorch [25] and wrapped as scikit-learn estimators so that they pass through the same cross-validation, threshold selection, tracking and export as the classical models. The CNN follows Kim [14]: 128-dimensional embeddings, kernel widths 3, 4 and 5 with 128 filters each, global max pooling, dropout 0.5. The BiLSTM [15] has 128 hidden units per direction (64 on reviews) with concatenated max and mean pooling over time. Both train with Adam (learning rate \(10^{-3}\)), batch size 64, a class-weighted cross-entropy on the imbalanced corpora, and early stopping on a 10 percent validation slice of the training fold (at most eight epochs on tweets, four and three on reviews).

Evaluation protocol

The same procedure is applied to every (corpus, representation, model) cell:

1. The training split is cleaned once. Each representation is fitted once per cross-validation fold on that fold’s training part and once on the whole training split; the resulting matrices are cached and shared by all models evaluated on that representation, so that models are compared on identical inputs and no representation ever sees validation or test documents during fitting.

2. Stratified \(k\)-fold cross-validation (\(k{=}5\) on tweets, \(3\) on reviews) produces out-of-fold class probabilities for every training document.

3. For binary tasks, the decision threshold on \(P(\text{positive})\) is the value in \([0.05, 0.95]\) that maximises F1 on the out-of-fold probabilities [19]. The three-class task uses the arg-max.

4. The selected cell per corpus is the one with the highest cross-validated F1 at that threshold; the test split plays no part in selection [18].

5. The model is refitted on the whole training split and scored once on the test split at the selected threshold: accuracy, precision, recall, F1 (positive class, or macro average), ROC-AUC, average precision, a per-class report and a confusion matrix. F1 at the default threshold of 0.5 is recorded alongside.

6. Out-of-fold and test probabilities are saved, so that every figure and table is regenerated from disk without retraining.

F1 on the positive class is the primary metric on the hate corpus because accuracy is dominated by the 93 percent majority class; macro F1 is used on the three-class corpus; both F1 and accuracy are reported on the balanced IMDB corpus.

System

Fig. 1 shows the implementation. One YAML file defines a benchmark (datasets, cleaning, representations, models, grid, folds, seed) and is logged with every run; each cell is an MLflow [29] run with parameters, cross-validated and test metrics, the threshold, timings and the confusion matrix, and the selected pipeline per corpus is exported as one serialised object (cleaner, representation, model) together with a model card [28] (data fingerprint, selection rule, metrics, per-class scores, software versions, commit) and a monitoring baseline. A Flask service exposes prediction, health and statistics endpoints over the exported pipelines and writes every prediction to a PostgreSQL store; a monitor compares recent traffic with the training baseline using the population stability index [30] of the token-length and predicted-class distributions, the out-of-vocabulary rate against the training vocabulary, and the share of inputs that become empty after cleaning, with warning and alert levels and a stored history. Continuous integration runs the unit tests against PostgreSQL, a one-minute version of the benchmark on a synthetic corpus, and a container build on every change; infrastructure definitions provision a container service, a managed database, object storage for artifacts and a scheduled drift task on a public cloud. No credential or dataset location appears in the code.

Experimental Setup

All results were produced on a laptop CPU (eight cores, no GPU) with pinned package versions (scikit-learn 1.5, XGBoost 2.0, Gensim 4.4, PyTorch 2.13). Seeds are fixed for the split, the folds, Gensim and PyTorch; Gensim’s multithreaded training introduces run-to-run variation of the order of \(10^{-3}\) in F1 for the embedding-based cells. The Twitter grid comprises 22 cells per corpus (naive Bayes on count-based features only, the ensemble on TF-IDF and Word2Vec, the networks on sequences); the IMDB grid is reduced to 12 cells for cost (no Doc2Vec, the random forest and SVM on TF-IDF only, shorter network schedules). The whole study runs in about two hours.

Results

Table 2 lists the selected model per corpus, Table 3 every family at its best representation, and Table 6 every representation at its best model. Figures 2 to 5 are drawn from the saved probabilities of the hate corpus, the task on which evaluation choices matter most.

Ranking ability versus thresholding

On the hate corpus every family reaches a test ROC-AUC between 0.939 (calibrated SVM) and 0.952 (XGBoost on Word2Vec), and average precision between 0.695 and 0.762 (Fig. 2); the eight families order the test tweets almost equally well. Their F1 on the positive class, however, spans 0.608 to 0.684, and the spread is explained by calibration and threshold transfer rather than by ranking. Fig. 3 shows that the class-weighted networks are strongly over-confident (mean predicted probability 0.87 in the top decile against an observed positive rate of 0.51), whereas the calibrated SVM, the forest, the boosted trees and the ensemble lie close to the diagonal; accordingly the networks’ F1-optimal thresholds are 0.85 and 0.90 (Fig. 4) while the classical models’ lie between 0.23 and 0.69.

Table 4 quantifies both effects. Tuning the threshold on out-of-fold predictions improves test F1 over the default of 0.5 for seven of the eight families, by 0.008 to 0.087 for the classical models (logistic regression on TF-IDF: 0.586 to 0.654) and by 0.09 to 0.14 for the networks; the random forest, whose optimum lies at 0.40, loses 0.005; a fixed threshold of 0.3, as used in the original notebooks, would have been near-optimal for naive Bayes and XGBoost and far from it for logistic regression and the networks. The last column shows threshold transfer: the threshold is chosen on probabilities of models trained on four fifths of the data and applied to a model refitted on all of it. For the calibrated SVM, logistic regression, the forest, the ensemble and XGBoost the transferred threshold reproduces the cross-validated F1 within 0.02; for the CNN it costs 0.064 (cross-validated 0.672, test 0.608) and for the BiLSTM 0.012, because early stopping ends the refitted network at a different epoch and its over-confident score scale shifts accordingly. The CNN is nevertheless the model selected by the protocol, since selection uses cross-validated F1 only; we report the outcome rather than override it, and note that averaging the fold models instead of refitting (cross-validation bagging) would keep the score scale consistent.

Fig. 5 places the differences in context: fold standard deviations are 0.01 to 0.02, so the top eight cells (cross-validated F1 0.650 to 0.672) are statistically indistinguishable on this corpus, and a claim that a network beats a linear model here would not survive the confidence intervals. Per-class scores of the selected models are given in Table 5: on the hate corpus the CNN at its tuned threshold trades recall (0.498) for precision (0.782) on the 448 positive test tweets.

Representations

Table 6 confirms across all three corpora that sparse \(n\)-gram features are the strongest representation for classical learners: TF-IDF or counts give the best cell for every model except XGBoost, which reaches its best hate-speech score (0.684) on mean-pooled Word2Vec. Doc2Vec is the weakest representation for every model on both Twitter corpora (0.42 to 0.56 on hate, 0.42 to 0.48 on vaccine): with documents of ten to twenty tokens, inferred paragraph vectors carry little beyond noise, and linear models on them fall below naive Bayes on counts. Mean-pooled Word2Vec sits between the two; it helps tree models, which can exploit interactions between embedding dimensions, and hurts linear ones. On the vaccine corpus the gap is largest: embedding features reach 0.42 to 0.55 macro F1 against 0.70 to 0.78 for \(n\)-gram models, because the lexicon labeller responds to the presence of specific words that mean pooling blurs.

Vaccine corpus

The CNN is clearly ahead on the three-class task: macro F1 0.812 (cross-validated 0.799 ± 0.012, accuracy 0.859) against 0.775 for logistic regression on counts, 0.757 for the TF-IDF ensemble and for XGBoost on counts, and 0.779 for the BiLSTM. The per-class report (Table 5) shows the minority negative class (205 test tweets) at F1 0.686 with recall 0.741, the neutral majority at 0.894 and positive at 0.856. Because the target is the sign of a lexicon polarity, a model that captures short patterns around lexicon words (negation, intensifiers) reproduces the labeller better than a bag of words can; these numbers should be read as agreement with TextBlob, not as human-judged sentiment accuracy.

IMDB

On balanced, long documents the picture inverts. Logistic regression (F1 0.886, accuracy 0.885), the calibrated SVM (0.884, 0.882) and their ensemble with XGBoost (0.888, 0.885) on 20,000 TF-IDF terms are within 0.004 of each other with fold standard deviations of 0.002; the BiLSTM (0.876) and CNN (0.875) trained from scratch do not reach them, and the tree models trail by 0.02 to 0.03. The tuned thresholds of the best cells lie at 0.45 to 0.50, so thresholding is immaterial on this task. Accuracies of 88.2 to 88.5 percent agree with the 88.9 percent reported by Maas et al. [11] for their learned word vectors with a linear classifier and sit below the 91.2 percent of NB-SVM over bigrams [10] and the 93 percent and above of fine-tuned transformers [16], which locates this grid precisely: it is the ceiling of models that train from scratch on a CPU in minutes.

Cost

Table 7 reports the time to fit the best cell per family on the full training split. The linear models train in well under a second on tweets and under a second on reviews; the forest and XGBoost in seconds to tens of seconds; the networks in 20 to 30 seconds on tweets and 80 to 90 seconds on reviews. On the hate corpus the resulting differences in F1 are within one to two fold standard deviations; on IMDB the cheapest models are also the most accurate. Whether the 0.04 to 0.05 macro-F1 advantage of the CNN on the vaccine corpus is worth a hundredfold increase in training time and an uncalibrated score scale is an application decision that the benchmark makes explicit rather than hides.

Comparison with the original study

The thesis notebooks evaluated the same classical models on the hate corpus with the vectorizer fitted on training and test documents together and a fixed threshold of 0.3, reporting validation F1 between 0.51 and 0.66 for logistic regression, the SVM, the forest and XGBoost on count, TF-IDF and Word2Vec features (and far lower on Doc2Vec), a public-leaderboard F1 of 0.70 for tuned XGBoost on Word2Vec, and 94 percent accuracy for naive Bayes on a split in which unlabelled test rows were assigned the negative class. Under the present protocol naive Bayes reaches F1 0.658 (accuracy 0.954, which is mostly the majority rate), XGBoost on Word2Vec 0.684 and the linear models 0.654: the ordering of the original study survives, its absolute numbers do not, and the one figure that looked best was an artefact of leakage and of reporting accuracy under imbalance.

Threats to Validity

Weak labels. The vaccine corpus is labelled by a lexicon; models are rewarded for learning its idiosyncrasies. The hate and IMDB corpora have human labels. Single test split. Test scores come from one stratified holdout (the official split for IMDB). Cross-validated means and standard deviations are reported alongside, and we have avoided claims that are smaller than one standard deviation. Threshold transfer. Thresholds selected on fold models are applied to refitted models; we quantified the cost (up to 0.064 F1 for the CNN) and identified cross-validation bagging as the remedy. Grid scope. No pretrained language model is included; the grid characterises what training from scratch on a CPU achieves, and the IMDB comparison with the literature bounds the gap. Temporal validity. Both Twitter corpora date from 2016 to 2021; vocabulary and hashtag usage have since moved, which is what the drift monitor in the released system is for. Hardware. Timings are from one laptop and are indicative only.

Conclusion

We fixed one evaluation protocol and applied it to 56 combinations of corpus, representation and model. Three findings hold across the study. First, on imbalanced short text the ranking ability of classical and neural classifiers is nearly identical, and the reported differences in F1 are differences in calibration and threshold handling; tuning the threshold on out-of-fold predictions is worth up to 0.09 F1 for classical models and 0.14 for networks and must be part of any comparison, and over-confident networks transfer such thresholds worst. Second, sparse \(n\)-gram features remain the representation of choice for classical learners on short text, learned document embeddings are the weakest choice everywhere, and on balanced long documents linear models on TF-IDF match networks trained from scratch at a hundredth of the cost. Third, the numbers of the original notebook study were dominated by leakage and by accuracy under imbalance; a rigorous protocol preserves its ranking but not its figures. The released system makes these results regenerable from saved probabilities and keeps the selected models operable, with tracked runs, model cards, a database-backed prediction service and drift monitoring. Future work is cross-validation bagging of the neural cells, a fine-tuned transformer as an upper reference, and human annotation of a subset of the vaccine corpus to replace the lexicon labels.

References

1. B. Pang and L. Lee, “Opinion mining and sentiment analysis,” Foundations and Trends in Information Retrieval, vol. 2, no. 1-2, pp. 1-135, 2008.

2. B. Liu, Sentiment Analysis and Opinion Mining. San Rafael, CA, USA: Morgan & Claypool, 2012.

3. A. Go, R. Bhayani, and L. Huang, “Twitter sentiment classification using distant supervision,” Stanford University, CS224N Project Report, 2009.

4. Z. Waseem and D. Hovy, “Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter,” in Proc. NAACL Student Research Workshop, San Diego, CA, USA, 2016, pp. 88-93.

5. T. Davidson, D. Warmsley, M. Macy, and I. Weber, “Automated hate speech detection and the problem of offensive language,” in Proc. 11th Int. AAAI Conf. Web and Social Media (ICWSM), Montreal, Canada, 2017, pp. 512-515.

6. A. McCallum and K. Nigam, “A comparison of event models for naive Bayes text classification,” in Proc. AAAI Workshop on Learning for Text Categorization, Madison, WI, USA, 1998, pp. 41-48.

7. T. Joachims, “Text categorization with support vector machines: Learning with many relevant features,” in Proc. 10th European Conf. Machine Learning (ECML), Chemnitz, Germany, 1998, pp. 137-142.

8. L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5-32, 2001.

9. T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, San Francisco, CA, USA, 2016, pp. 785-794.

10. S. Wang and C. D. Manning, “Baselines and bigrams: Simple, good sentiment and topic classification,” in Proc. 50th Annu. Meeting Assoc. Computational Linguistics (ACL), Jeju Island, Korea, 2012, pp. 90-94.

11. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in Proc. 49th Annu. Meeting Assoc. Computational Linguistics (ACL), Portland, OR, USA, 2011, pp. 142-150.

12. T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Advances in Neural Information Processing Systems 26, Lake Tahoe, NV, USA, 2013, pp. 3111-3119.

13. Q. Le and T. Mikolov, “Distributed representations of sentences and documents,” in Proc. 31st Int. Conf. Machine Learning (ICML), Beijing, China, 2014, pp. 1188-1196.

14. Y. Kim, “Convolutional neural networks for sentence classification,” in Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, 2014, pp. 1746-1751.

15. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.

16. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, Minneapolis, MN, USA, 2019, pp. 4171-4186.

17. S. Kaufman, S. Rosset, C. Perlich, and O. Stitelman, “Leakage in data mining: Formulation, detection, and avoidance,” ACM Trans. Knowledge Discovery from Data, vol. 6, no. 4, art. 15, 2012.

18. G. C. Cawley and N. L. C. Talbot, “On over-fitting in model selection and subsequent selection bias in performance evaluation,” J. Machine Learning Research, vol. 11, pp. 2079-2107, 2010.

19. Z. C. Lipton, C. Elkan, and B. Naryanaswamy, “Optimal thresholding of classifiers to maximize F1 measure,” in Proc. ECML PKDD, Nancy, France, 2014, pp. 225-239.

20. J. C. Platt, “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” in Advances in Large Margin Classifiers, A. J. Smola, P. Bartlett, B. Schölkopf, and D. Schuurmans, Eds. Cambridge, MA, USA: MIT Press, 1999, pp. 61-74.

21. A. Niculescu-Mizil and R. Caruana, “Predicting good probabilities with supervised learning,” in Proc. 22nd Int. Conf. Machine Learning (ICML), Bonn, Germany, 2005, pp. 625-632.

22. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proc. 34th Int. Conf. Machine Learning (ICML), Sydney, Australia, 2017, pp. 1321-1330.

23. J. Davis and M. Goadrich, “The relationship between precision-recall and ROC curves,” in Proc. 23rd Int. Conf. Machine Learning (ICML), Pittsburgh, PA, USA, 2006, pp. 233-240.

24. F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” J. Machine Learning Research, vol. 12, pp. 2825-2830, 2011.

25. A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems 32, Vancouver, Canada, 2019, pp. 8024-8035.

26. S. Loria, “TextBlob: Simplified text processing,” software, release 0.15, 2018. [Online]. Available: https://textblob.readthedocs.io

27. D. Sculley et al., “Hidden technical debt in machine learning systems,” in Advances in Neural Information Processing Systems 28, Montreal, Canada, 2015, pp. 2503-2511.

28. M. Mitchell et al., “Model cards for model reporting,” in Proc. Conf. Fairness, Accountability, and Transparency (FAT*), Atlanta, GA, USA, 2019, pp. 220-229.

29. M. Zaharia et al., “Accelerating the machine learning lifecycle with MLflow,” IEEE Data Engineering Bulletin, vol. 41, no. 4, pp. 39-45, 2018.

30. N. Siddiqi, Credit Risk Scorecards: Developing and Implementing Intelligent Credit Scoring. Hoboken, NJ, USA: Wiley, 2006.