PhishLens: Calibration-Aware Selective Routing with Verified Local-LLM Evidence Selection for Phishing Website Detection
DOI:
https://doi.org/10.31033/IJEMR/16.5.2026.1951Keywords:
Phishing Website Detection, Probability Calibration, Selective Prediction, LLM Explanation, Evidence Verification, Dataset ShiftAbstract
Large language models (LLMs) can explain phishing verdicts in natural language, but using them as the detector adds latency and cost, and their explanations can cite evidence that is absent or wrong. We present PhishLens, a phishing website detector in which frozen classical models set every phishing label and probability, while a small local LLM is used only to select explanatory evidence for uncertain cases. A LightGBM model on 46 URL and static-HTML features, with a 20-feature URL-only Random Forest fallback, produces the label and a calibration-checked probability. Cases whose probability falls inside a frozen uncertainty band are routed to a local Qwen3 (4B) model that may return only ranked identifiers of verified evidence and one approved safety-action code; a deterministic verifier checks every output, and any failed output is replaced by a deterministic explanation. The LLM cannot change the probability or label, and submitted URLs are never visited. Detector features, models, calibration choices, classification thresholds and routing bands were selected on registrable-domain-grouped Phish360 partitions and frozen before the final detector evaluations. The later ID-selection protocol was assessed only in a 20-case development-validation pilot and did not alter any detector or reported test prediction. On the untouched Phish360 test set, the URL+HTML pathway reached 94.58% accuracy, MCC 0.8869, F1 0.9316, PR-AUC 0.9798 and ECE 0.0298. In a second-look external evaluation on CompPhish v3, it reached 87.64% accuracy, MCC 0.7517, F1 0.8672, PR-AUC 0.9441 and ECE 0.0635. The routing policy would avoid LLM invocation for 80.55% of external cases, although the routed share rose from the 10% validation budget to 19.45%. In a separate 40-case validation pilot, all 40 Qwen outputs passed deterministic verification, with a Wilson 95% confidence interval of 91.2%–100%. Exploratory feedback did not establish that Qwen explanations were more useful than deterministic explanations. Calibration and routing budgets must therefore be re-checked under dataset shift.
Downloads
References
J. Ma, L. K. Saul, S. Savage, & G. M. Voelker. (2009). Beyond blacklists: Learning to detect malicious web sites from suspicious URLs. In Proc. 15th ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining (KDD ’09), pp. 1245–1254. DOI: 10.1145/1557019.1557153.
O. K. Sahingoz, E. Buber, O. Demir, & B. Diri. (2019). Machine learning based phishing detection from URLs,. Expert Syst. Appl., 117, 345–357. DOI: 10.1016/j.eswa.2018.09.029.
G. Xiang, J. Hong, C. P. Rose, & L. Cranor. (2011). CANTINA+: A feature-rich machine learning framework for detecting phishing web sites. ACM Trans. Inf. Syst. Secur., 14(2). DOI: 10.1145/2019599.2019606.
A. Aljofey et al. (2022). An effective detection approach for phishing websites using URL and HTML features. Sci. Rep., 12(1), Art. no. 8842. DOI: 10.1038/s41598-022-10841-5.
T. Koide, H. Nakano, & D. Chiba. (2024). ChatPhishDetector: Detecting phishing sites using large language models. IEEE Access, 12, 154381–154400. DOI: 10.1109/ACCESS.2024.3483905.
J. Lee, P. Lim, B. Hooi, & D. M. Divakaran. (2024). Multimodal large language models for phishing webpage detection and identification. In Proc. APWG Symp. Electron. Crime Res. (eCrime), pp. 1–13. DOI: 10.1109/eCrime66200.2024.00007.
T. Cao et al. (2025). PhishAgent: A robust multimodal agent for phishing webpage detection. In Proc. AAAI Conf. Artif. Intell., 39(27), pp. 27869–27877. DOI: 10.1609/aaai.v39i27.35003.
W. Li, S. Manickam, Y.-W. Chong, W. Leng, & P. Nanda. (2024). A stateof-the-art review on phishing website detection techniques. IEEE Access, 12, 187976–188012. DOI: 10.1109/ACCESS.2024.3514972.
D. Sivaneswaran, C. T. E. R. Hewage, H. M. K. K. M. B. Herath, R. S. Rathore, V. K. Singh, & W. Jiang. (2026). A systematic literature review of large language models in phishing attack generation and detection. Array, 30, Art. no. 100775 DOI: 10.1016/j.array.2026.100775.
W. Li, S. Manickam, Y.-W. Chong, & S. Karuppayah. (2025). PhishDebate: An LLM-based multi-agent framework for phishing website detection. In Proc. IEEE Int. Conf. Big Data (BigData), pp. 6606–6615. DOI: 10.1109/BigData66926.2025.11401440.
K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, & M. Fritz. (20234). Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. In Proc. 16th ACM Workshop Artif. Intell. Secur. (AISec ’23), pp. 79– 90. DOI: 10.1145/3605764.3623985.
T. Koide, H. Nakano, & D. Chiba. (2026). Clouding the mirror: Stealthy prompt injection attacks targeting LLM-based phishing detection. arXiv:2602.05484. Available at: https://arxiv.org/abs/2602.05484
M. Turpin, J. Michael, E. Perez, & S. Bowman. (2023). Language models don’t always say what they think: Unfaithful explanations in chain-ofthought prompting. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 36, pp. 74952–74965. DOI: 10.52202/0752803275.
L. Huang et al. (2025). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2), 1–55. DOI: 10.1145/3703155.
S. Kuikel, A. Piplai, & P. Aggarwal. (2025). Evaluating large language models for phishing detection, self-consistency, faithfulness, and explainability. arXiv:2506.13746. Available at: https://arxiv.org/abs/2506.13746
C. Guo, G. Pleiss, Y. Sun, & K. Q. Weinberger. (2017). On calibration of modern neural networks. In Proc. 34th Int. Conf. Mach. Learn. (ICML), ser. PMLR, 70, pp. 1321–1330. Available at: https://proceedings.mlr.press/v70/guo17a.html
A. Niculescu-Mizil & R. Caruana. (2005). Predicting good probabilities with supervised learning. In Proc. 22nd Int. Conf. Mach. Learn. (ICML ’05), pp. 625–632. DOI: 10.1145/1102351.1102430.
R. M. Mohammad, F. Thabtah, & L. McCluskey. (2014). Intelligent rulebased phishing websites classification. IET Inf. Secur., 8(3), 153–160. DOI: 10.1049/iet-ifs.2013.0202.
Y. Zhang, J. I. Hong, & L. F. Cranor. (2007). Cantina: A content-based approach to detecting phishing web sites. In Proc. 16th Int. Conf. World Wide Web (WWW ’07), pp. 639–648. DOI: 10.1145/1242572.1242659.
A. S. Bozkir, F. C. Dalgic, & M. Aydos. (2023). GramBeddings: A new neural network for URL based identification of phishing web pages through n-gram embeddings. Comput. Secur., 124, Art. no. 102964. DOI: 10.1016/j.cose.2022.102964.
C. Opara, B. Wei, & Y. Chen. (2020). HTMLPhish: Enabling phishing web page detection by applying deep learning techniques on HTML analysis. In Proc. Int. Joint Conf. Neural Netw. (IJCNN), pp. 1–8. DOI: 10.1109/IJCNN48605.2020.9207707.
C. Opara, Y. Chen, & B. Wei. (2024). Look before you leap: Detecting phishing web pages by exploiting raw URL and HTML characteristics. Expert Syst. Appl., 236, Art. no. 121183. DOI: 10.1016/j.eswa.2023.121183.
L. Breiman. (2001). Random forests. Mach. Learn., 45(1), 5–32. DOI: 10.1023/A:1010933404324.
T. Chen, & C. Guestrin. (2016). XGBoost: A scalable tree boosting system. In Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining (KDD ’16), pp. 785–794. DOI: 10.1145/2939672.2939785.
G. Ke et al. (2017). LightGBM: A highly efficient gradient boosting decision tree. In Proc. Adv. Neural Inf. Process. Syst. (NIPS), 30. Available at: https://proceedings.neurips.cc/paper_files/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html
M. S. I. Ovi, M. H. Rahman, & M. A. Hossain. (2024). PhishGuard: A multi-layered ensemble model for optimal phishing website detection. In Proc. 6th Int. Conf. Sustain. Technol. Ind. 5.0 (STI). DOI: 10.1109/STI64222.2024.10951075.
S. Abdelnabi, K. Krombholz, & M. Fritz. (2020). VisualPhishNet: Zeroday phishing website detection by visual similarity. In Proc. ACM SIGSAC Conf. Comput. Commun. Secur. (CCS ’20), pp. 1681–1698. DOI: 10.1145/3372297.3417233.
Y. Lin et al. (2021). Phishpedia: A hybrid deep learning based approach to visually identify phishing webpages. In Proc. 30th USENIX Secur. Symp., pp. 3793–3810. Available at: https://www.usenix.org/conference/usenixsecurity21/presentation/lin
R. Liu, Y. Lin, X. Yang, S. H. Ng, D. M. Divakaran, & J. S. Dong. (2022). Inferring phishing intention via webpage appearance and dynamics: A deep vision based approach. In Proc. 31st USENIX Secur. Symp., pp. 1633–1650. Available at: https://www.usenix.org/conference/usenixsecurity22/presentation/liuruofan
L. Tang, & Q. H. Mahmoud. (2021). A survey of machine learning-based solutions for phishing website detection. Mach. Learn. Knowl. Extr., 3(3), 672–694. DOI: 10.3390/make3030034.
M. Alam et al., (2026). SoK: PHILTER: Uncovering security and functional gaps in AI-based phishing website detection literature via an LLMbased reasoning framework. In Proc. 35th USENIX Secur. Symp., pp. 4981–5000. Available at: https://www.usenix.org/conference/usenixsecurity26/presentation/ala m
Y. Li et al., (2024). KnowPhish: Large language models meet multimodal knowledge graphs for enhancing reference-based phishing detection. In Proc. 33rd USENIX Secur. Symp., pp. 793–810. Available: https://www.usenix.org/conference/usenixsecurity24/presentation/liyuexin
R. Liu, Y. Lin, X. Teoh, G. Liu, Z. Huang, & J. S. Dong. (2024). Less defined knowledge and more true alarms: Reference-based phishing detection without a pre-defined reference list. In Proc. 33rd USENIX Secur. Symp., pp. 523–540. Available: https://www.usenix.org/conference/usenixsecurity24/presentation/liuruofan
F. Trad, & A. Chehab. (2024). Prompt engineering or fine-tuning? A case study on phishing detection with large language models. Mach. Learn. Knowl. Extr., 6(1), 367–384. DOI: 10.3390/make6010018.
S. Saha Roy, & S. Nilizadeh. (2026). PhishLang: A real-time, fully clientside phishing detection framework using MobileBERT. In Proc. Netw. Distrib. Syst. Secur. (NDSS) Symp. DOI: 10.14722/ndss.2026.231037.
G. Goldenits, P. König, S. Raubitzek, & A. Ekelhart. (2026). Small language models for phishing website detection: Cost, performance, and privacy trade-offs. J. Cybersecur. Priv., 6(2), Art. no. 48. DOI: 10.3390/jcp6020048.
Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, & I. Mordatch. (2024). Improving factuality and reasoning in language models through multiagent debate. In Proc. 41st Int. Conf. Mach. Learn. (ICML), ser. PMLR, 235, pp. 11733–11763. Available: https://proceedings.mlr.press/v235/du24e.html
N. T. V. Nguyen, F. D. Childress, & Y. Yin. (2025). Debate-driven multiagent LLMs for phishing email detection. In Proc. 13th Int. Symp. Digit. Forensics Secur. (ISDFS), pp. 1–5. DOI: 10.1109/ISDFS65363.2025.11012014.
Y. Xue, E. Spero, M. W. Woo, W. Gao, & G. Russello. (2025). MultiPhishGuard: An explainable and adaptive multi-agent LLM system for phishing email detection. arXiv:2505.23803. Available: https://arxiv.org/abs/2505.23803
T. Yadav, & M. Masum. (2026). Explainable multi-agent LLM framework for phishing email detection via role-specialized evidence decomposition. Electronics, 15(12), Art. no. 2606. DOI: 10.3390/electronics15122606.
M. T. Ribeiro, S. Singh, & C. Guestrin. (2016). ‘Why should I trust you?’: Explaining the predictions of any classifier. In Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining (KDD ’16), pp. 1135–1144. DOI: 10.1145/2939672.2939778.
S. M. Lundberg, & S.-I. Lee. (2017). “A unified approach to interpreting model predictions. In Proc. Adv. Neural Inf. Process. Syst. (NIPS), 30, pp. 4765–4774. Available: https://proceedings.neurips.cc/paper_files/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html
M. C. Calzarossa, P. Giudici, & R. Zieni. (2024). Explainable machine learning for phishing feature detection. Qual. Reliab. Eng. Int., 40(1), 362–373. DOI: 10.1002/qre.3411.
B. Lim, R. Huerta, A. Sotelo, A. Quintela, & P. Kumar. (2025). EXPLICATE: Enhancing phishing detection through explainable AI and LLM-powered interpretability. arXiv:2503.20796. DOI: 10.48550/arXiv.2503.20796.
G. Desolda, F. Greco, & L. Viganò. (2025). APOLLO: A GPT-based tool to detect phishing emails and generate explanations that warn users. Proc. ACM Hum.-Comput. Interact., 9(4), 1–33. DOI: 10.1145/3733049.
Y. Wang et al. (2025). Can you walk me through it? Explainable SMS phishing detection using LLM-based agents. In Proc. 21st Symp. Usable Privacy Secur. (SOUPS), pp. 37–56. Available: https://www.usenix.org/conference/soups2025/presentation/wang
F. M. Cau, G. Desolda, F. Greco, L. D. Spano, & L. Viganò. (2025). Can large language models automate phishing warning explanations? A controlled experiment on effectiveness and user perception. arXiv:2507.07916. DOI: 10.48550/arXiv.2507.07916.
Y. Xu, & X. Luo. (2026). Auditable and robust LLM-based phishing detection via provenance-guided evidence contracts. Comput. Mater. Continua. DOI: 10.32604/cmc.2026.085276.
G. W. Brier. (1950). Verification of forecasts expressed in terms of probability. Mon. Weather Rev., 78(1), 1–3.
M. P. Naeini, G. Cooper, & M. Hauskrecht. (2015). Obtaining well calibrated probabilities using Bayesian binning. In Proc. AAAI Conf. Artif. Intell., 29(1). DOI: 10.1609/aaai.v29i1.9602.
H.-T. Lin, C.-J. Lin, & R. C. Weng. (2007). A note on Platt’s probabilistic outputs for support vector machines. Mach. Learn., 68(3), 267–276. DOI: 10.1007/s10994-007-5018-6.
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 S. Nandhini, M. Saranya, P.M.G. Jegathambal
This work is licensed under a Creative Commons Attribution 4.0 International License.
Research Articles in 'International Journal of Engineering and Management Research' are Open Access articles published under the Creative Commons CC BY License Creative Commons Attribution 4.0 International License http://creativecommons.org/licenses/by/4.0/. This license allows you to share – copy and redistribute the material in any medium or format. Adapt – remix, transform, and build upon the material for any purpose, even commercially.
