VITAL

VirusTotal Integrated Toolkit for Aggregation and Labeling

A unified and reproducible framework for constructing, comparing, and longitudinally evaluating Android malware labels derived from VirusTotal antivirus verdicts.

Installation and execution

This project requires Python 3.11.

From the project root, install all dependencies:

python -m pip install -r requirements.txt

Then run the pipeline:

# Build the dataset, run all methods, and create all figures
python main.py

# Reuse the existing dataset, run all methods, and recreate the figures
python main.py --skip-dataset

# Build the dataset and run all methods without creating figures
python main.py --skip-plots

# Reuse the existing dataset and run all methods without creating figures
python main.py --skip-dataset --skip-plots

# Run only specific methods
python main.py --skip-dataset --methods kantchelian_em decision_tree

# Display all available command-line options
python main.py --help

The names accepted by --methods are:

threshold_1
threshold_4
majority_vote
calibrated_threshold
kantchelian_em
kantchelian_logistic
revmatch
weighted_average
bayesian_decision
decision_tree
cpm
dam
gam
charlton
du_derived

Multiple method names can be provided after --methods.

Main objectives

• Load and normalize longitudinal VirusTotal reports
• Construct a pseudo-ground truth using the latest VT report and ClarAVy
• Extract and normalize per-engine AV verdicts
• Split APKs into training, validation, and test partitions without data leakage
• Implement multiple AV aggregation methods through a common interface
• Calibrate score-based aggregation methods toward a common target false-positive rate
• Evaluate predictions on both first-scan and longitudinal VirusTotal reports

Ground truth

The pseudo-ground truth is constructed from the latest available VirusTotal report of each APK.

• Benign: no AV engine detects the APK as malicious
• Malicious: ClarAVy returns a non-SINGLETON malware family
• High-detection fallback: at least 10 AV engines detect the APK
• Uncertain: all remaining samples are excluded

The latest report is used only to construct the reference label and is not provided as input to the aggregation methods.

If ClarAVy cannot be executed or its output cannot be loaded, the complete ground-truth strategy falls back to detection-count labeling: zero detections are benign, at least four detections are malicious, and the remaining APKs are excluded.

Dataset construction

Only scans preceding the latest report used for pseudo-ground-truth construction are retained. An APK must also satisfy the following requirements:

• at least two historical scans;
• at least 30 days between its first and last retained scans;
• no more than 30 days between first submission and first scan.

After splitting, only AV engines present in at least 90% of the first training scans are retained. Engine selection therefore uses training data only.

Evaluation protocol

The default APK-level split is chronological and uses the date of each APK's first scan:

• training: the first year of data;
• validation: the following two quarters;
• test: all remaining data.

If these fixed windows produce an empty partition, the pipeline automatically falls back to a chronological 60%/20%/20% split. All scans belonging to the same APK always remain in the same partition.

Only the first scan of each training APK is used to fit the aggregation methods. Thresholds are then selected on the first validation scans by choosing the available operating point closest to the target FPR of 1%; equal-distance candidates are resolved in favor of the highest TPR.

The fixed threshold_1, threshold_4, and majority_vote baselines retain their native decision rules and are not calibrated. Moreover, an exact validation FPR of 1% is not always attainable when a method produces discrete scores or many ties. The 1% value is a validation target, not a guarantee on the temporally later test set, whose FPR may differ because of distribution shift.

Evaluation is performed in three complementary ways:

• metrics and ROC curves on the first scan of every test APK;
• separate first-scan metrics for problematic test APKs;
• longitudinal analysis of the time required to reach a stable correct verdict.

The problematic subset contains benign APKs with at least one detection on their first scan and malicious APKs with fewer than four first-scan detections. It is evaluated separately but is never used for fitting or calibration.

Aggregation methods

Voting baselines

• Threshold voting (t = 1)
• Threshold voting (t = 4)
• Majority voting
• Validation-calibrated detection-count voting

Kantchelian et al.

Better Malware Ground Truth: Techniques for Weighting Anti-Virus Vendor Labels

Two approaches are implemented:

• Generative EM model: unsupervised estimation of malware prevalence and scanner-specific detection and false-positive probabilities.
• Logistic Regression: supervised model that learns one coefficient for each AV engine.

RevMatch

Fung et al. — RevMatch: An Efficient and Robust Decision Model for Collaborative Malware Detection

Uses labeled historical AV-feedback patterns. Exact verdict patterns are searched first; when too few matches are available, low-quality scanner verdicts are progressively removed through feedback relaxation.

Weighted Average

A reliability-aware baseline derived from the models evaluated in the RevMatch work. Scanner weights are computed from their estimated false-positive and false-negative rates.

Bayesian Decision

A Bayesian aggregation baseline based on scanner-specific TPR and FPR estimates and a conditional-independence assumption.

Decision Tree

A supervised decision-tree aggregation model trained on AV verdicts. The implementation uses an entropy-based Scikit-learn decision tree as an approximation of the J48 model evaluated in the RevMatch study.

Sakib et al.

Maximizing Accuracy in Multi-Scanner Malware Detection Systems

Three probabilistic scanner-combination models are implemented:

• CPM — Combined Probability Model: estimates multi-scanner performance assuming independent scanners.
• DAM — Dependency Approximation Model: incorporates pairwise scanner dependencies.
• GAM — Greedy Approximation Model: provides a computationally cheaper approximation of scanner-combination performance.

Charlton et al.

A New Method for Inferring Ground-Truth Labels and Malware Detector Effectiveness Metrics

Uses agreement among scanners and a synthetic bellwether detector to estimate relative scanner reliability. These estimates are refined through an iterative weighted-voting procedure.

Du-derived aggregation

Du et al. — Statistical Estimation of Malware Detection Metrics in the Absence of Ground Truth

The original work estimates scanner-level effectiveness metrics rather than defining a per-sample aggregation rule.

In this framework, the adjusted FPR and FNR estimates proposed by Du et al. are used to derive scanner weights, which are then employed in a weighted-voting aggregation method.

This implementation is therefore explicitly considered a Du-derived aggregation method, rather than a direct reproduction of an aggregation algorithm proposed in the original paper.

Methods discussed but not included in the experimental comparison

The following methods are analyzed in the related work but are not implemented in the final experimental comparison:

• Maat: requires a specific longitudinal re-scanning protocol that cannot be faithfully reproduced with the available irregular VT histories.
• SIRAJ: requires large-scale temporal self-supervised pre-training and a dedicated neural architecture.
• Okazaki et al.: requires reliable malware-category labels that are not available in the current binary dataset.
• Choo et al.: targets phishing-versus-malware URL classification rather than benign-versus-malicious Android APK labeling.

License

This project is released under the MIT License.