April 2026
Classifier LLM - Fine-Tuned Phishing Detection Model
ML Research & Engineering - Subproject under Sweep Internship
Model: RoBERTa (Fine-tuned)
Categories
Machine Learning
Natural Language Processing (NLP)
Cybersecurity
A fine-tuned transformer-based classifier developed to distinguish phishing and legitimate email content, later integrated into the Sweep Mail threat-detection pipeline.
This model was integrated into Sweep Mail's threat detection pipeline. It worked alongside the other detection components to analyze incoming email and contribute to the final threat classification
RoBERTa Fine-Tuned Results & Evaluation Details
Accuracy99.78%
F1 Score0.9934
Training Set284k Emails
Test Set23k Emails
Epochs3
HardwareNvidia A100 80GB
Model Comparison
| Model | Accuracy | Precision | Recall | F1 (phishing) | F1 (macro) | Samples |
|---|---|---|---|---|---|---|
| LLaMA 3.1 8B | 0.7756 | 0.6130 | 0.9704 | 0.7514 | 0.7735 | 21,772 |
| Mistral 7B | 0.6252 | 0.4824 | 0.9953 | 0.6498 | 0.6233 | 21,772 |
| Qwen 2.5 7B | 0.8818 | 0.7605 | 0.9660 | 0.8510 | 0.8765 | 21,772 |
| RoBERTa (Baseline) | 0.3494 | 0.3494 | 1.0000 | 0.5179 | 0.2589 | 21,772 |
| RoBERTa (Fine-tuned) | 0.9954 | 0.9972 | 0.9896 | 0.9934 | 0.9949 | 21,772 |
Model Evaluation Pipeline
Interested in learning more about the process behind these results? Read the Technical Report for a detailed breakdown of this project