Enhancing Arabic Phonetic Learning through YOLO-Based Grapheme Recognition and Audio Feedback

Anisa Dwi Nurchayati; Eric Kunto Aribowo; Akhmad Fadjeri

doi:10.32332/an-nabighoh.v28i1.319-340

Authors

Anisa Dwi Nurchayati Sunan Kalijaga State Islamic University Yogyakarta https://orcid.org/0009-0002-0473-6317
Eric Kunto Aribowo Sunan Kalijaga State Islamic University Yogyakarta
Akhmad Fadjeri Universitas Ma'arif Nahdlatul Ulama Kebumen https://orcid.org/0000-0003-0195-6883

DOI:

https://doi.org/10.32332/an-nabighoh.v28i1.319-340

Keywords:

Arabic computer vision, YOLO algorithm, Arabic phonetics, Grapheme detection, Arabic language learning, Computer-Assisted Language Learning, CALL

Abstract

Background: Islamic religious education in Indonesia introduces Arabic script (hijaiyah) as the foundational step of Quranic literacy, primarily within non-formal settings such as Madrasah Diniah. Students at Madrasah Diniah in Kebumen face persistent difficulties distinguishing and pronouncing similar Arabic letters, such as ع (ain) and غ (ghain), due to phonological interference from their native Javanese and Indonesian languages, which lack equivalent guttural phonemes. Traditional methods, such as Iqra’ books and letter cards, fail to provide real-time phonetic feedback, limiting the accuracy of pronunciation, which is essential for Quranic recitation. Research Objectives: In this study, a computer vision application was developed that detects Arabic letters from uploaded images, displays Indonesian phonetic transliteration, and generates automatic audio pronunciation (image-to-text-to-voice) using the You Only Look Once Version 11 (YOLOv11) algorithm within a Waterfall development model. Methodology: Employing a research and development (R&D) approach, the study followed five waterfall stages: requirements analysis, system design, implementation, verification, and maintenance. Verification involved expert judgment, end-user evaluation, and confusion matrix-based algorithm assessment. A dataset of 360 annotated Arabic letter images was used for training in Google Colab. Results: The resulting application, Arabic as a Foreign Language (AFL), detects Arabic letters from images, displays phonetic transliteration, and generates automatic audio pronunciation. The YOLOv11 model achieved 99.5% accuracy (mAP50) and 91.12% accuracy (mAP50-95). Expert Black-Box testing yielded an average Likert score above 4.0 across all functional criteria, with the audio feature rated highest (4.6). Unique Contribution: This study provides empirical evidence that computer vision-based pronunciation support effectively bridges grapheme recognition and accurate phoneme production, addressing a critical gap in AFL pedagogy within Indonesian Islamic education. Conclusion: The study successfully developed a YOLOv11-based application (AFL) with high accuracy and positive expert evaluations, effectively addressing pronunciation difficulties among Madrasah Diniyah students in Kebumen and demonstrating the potential of computer vision to bridge grapheme recognition and phoneme production in Arabic learning. Recommendations: Future work should improve accuracy for letter pairs Kaf–Tsa (ك–ث), Nun–Tsa/Nun–Ta (ن–ث/ن–ت), and Alif–Ghayn/Za–Ghayn (ا–غ/ز–غ), as they possess fundamentally distinct Unicode encodings and vector embeddings, not merely diacritical dot variations.

Downloads

Download data is not yet available.

Author Biographies

Anisa Dwi Nurchayati, Sunan Kalijaga State Islamic University Yogyakarta

Anisa Dwi Nurchayati is a Master’s student in the Arabic Language and Literature at UIN Sunan Kalijaga Yogyakarta, Indonesia. She completed her Bachelor’s degree in Islamic Education Management at UIN Prof. KH Saifuddin Zuhri Purwokerto. Her academic work spans Arabic linguistics, educational technology, and ICT-based learning management systems. Among her published works are a systematic literature review on Arabic phonetic interference (Stilistika: Jurnal Pendidikan Bahasa dan Sastra, 2026), a study on the effectiveness of Learning Management Systems in administrative services at madrasah level (Jurnal Kridatama Sains dan Teknologi, 2023), a collaborative paper on ICT-based education management development at MTs Plus Nurul Falah (Jurnal Kridatama Sains dan Teknologi, 2024), and a study on ICT-based learning media validity testing (Jurnal Pendidikan Surya Edukasi, 2022). She is actively involved in community service activities and applied research on the intersection of digital technology and Islamic education. She can be contacted at [email protected].
Eric Kunto Aribowo, Sunan Kalijaga State Islamic University Yogyakarta

Eric Kunto Aribowo is a lecturer at UIN Sunan Kalijaga Yogyakarta, specializing in linguistics, sociolinguistics, onomastics, and educational technology. His latest research spans Arabic onomastics, language technology, genre-aware language modelling, and technology-mediated language learning. His relevant publications include Genre Aware Language Modeling for Indonesian Academic Writing: Building and Evaluating IndoSciBERT (Lingua: Journal of Linguistics and Language, 2025), studies on Arabic naming practices and Arab-Javanese linguistic identity (Arabi: Journal of Arabic Studies, 2019; Arabiyat, 2019; Langkawi, 2020), and multiple works on digital educational tools including gamification, mobile learning, and AI-assisted academic writing. He has delivered invited lectures on AI in research and publication, bibliometric analysis, and Arabic language learning technology at over 100 events. He can be contacted at [email protected].
Akhmad Fadjeri, Universitas Ma'arif Nahdlatul Ulama Kebumen

Akhmad Fadjeri is a lecturer at Universitas Ma'arif Nahdlatul Ulama Kebumen, Indonesia, specialising in computer science, machine learning, and computer vision. He is the author of “Computer Vision untuk Pemula” (Rumah Kreativ Wadas Kelir, 2024), a foundational textbook on computer vision for Indonesian practitioners. His applied research focuses on YOLO-based object detection systems, safety helmet detection using YOLOv8, physical fitness classification using Naive Bayes algorithms, and the development of ICT-based educational management systems. He has supervised multiple undergraduate and postgraduate research projects at the intersection of deep learning and educational technology, and has contributed to community service programmes in digital innovation and Islamic education institution management. He can be contacted at [email protected].

References

Al Farisi, Mohamad Zaka, Mad Ali, Zawawi Ismail, Hikmah Maulani, Nalahuddin Shaleh, and Shofa Mushtofa Khalid. “Investigation of the Pronunciation of the Voiceless Fricative Non-Sibilant Phoneme /θ/ in Arabic: An Acoustic Phonetics Comparative Analysis.” Dirasat: Human and Social Sciences 52, no. 6 (2025): 5447. https://doi.org/10.35516/Hum.2025.5447. DOI: https://doi.org/10.35516/Hum.2025.5447

Alghyaline, Salah. “A Printed Arabic Optical Character Recognition System Using Deep Learning.” Journal of Computer Science 18, no. 11 (2022): 1038–50. https://doi.org/10.3844/jcssp.2022.1038.1050. DOI: https://doi.org/10.3844/jcssp.2022.1038.1050

Alsharhan, Eiman, and Allan Ramsay. “Improved Arabic Speech Recognition System through the Automatic Generation of Fine-Grained Phonetic Transcriptions.” Information Processing & Management 56, no. 2 (2019): 343–53. https://doi.org/10.1016/j.ipm.2017.07.002. DOI: https://doi.org/10.1016/j.ipm.2017.07.002

Anugrah, Anugrah, Mantasiah Rivai, and Fatkhul Ulum. “Analysis of Regional Language Phonological Interference on the Pronunciation of Hijaiyah Letters of Madrasah Tsanawiyah Students in Gowa Regency.” Pinisi Journal of Art, Humanity, and Social Studies 4, no. 4 (2024). https://journal.unm.ac.id/index.php/PJAHSS/article/view/2882.

Arafat, Syed Yasser, and Muhammad Javed Iqbal. “Urdu-Text Detection and Recognition in Natural Scene Images Using Deep Learning.” IEEE Access 8 (2020): 96787–803. https://doi.org/10.1109/ACCESS.2020.2994214. DOI: https://doi.org/10.1109/ACCESS.2020.2994214

Birgün, Mehmet. “Integrating AI into Qur’an Learning: Technical Advances and Pedagogical Gaps.” Social Sciences & Humanities Open 13 (June 2026): 102499. https://doi.org/10.1016/j.ssaho.2026.102499. DOI: https://doi.org/10.1016/j.ssaho.2026.102499

Bochkovskiy, Alexey, Chien-Yao Wang, and Hong-Yuan Mark Liao. “YOLOv4: Optimal Speed and Accuracy of Object Detection.” Version 1. Preprint, arXiv, 2020. https://doi.org/10.48550/ARXIV.2004.10934.

Bouchakour, Lallouani, and Nadjla Bettayeb. “Arabic Text Detection and Recognition in Video Using Deep Learning.” Signal, Image and Video Processing 19, no. 8 (2025): 685. https://doi.org/10.1007/s11760-025-04295-1. DOI: https://doi.org/10.1007/s11760-025-04295-1

Bouchal, Hakim, Ahror Belaid, and Farid Meziane. “Towards Accurate Recognition of Historical Arabic Manuscripts: A Novel Dataset and a Generalizable Pipeline.” ACM Transactions on Asian and Low-Resource Language Information Processing 24, no. 10 (2025): 1–30. https://doi.org/10.1145/3744243. DOI: https://doi.org/10.1145/3744243

Burhaeın, Erick, Akhmad Fadjerı, and Ibnu Prasetyo Widiyono. “Application of Naive Bayes Algorithm for Physical Fitness Level Classification.” International Journal of Disabilities Sports and Health Sciences 7, no. 1 (2024): 178–87. https://doi.org/10.33438/ijdshs.1330745. DOI: https://doi.org/10.33438/ijdshs.1330745

Chabane, Mafaza, Fouzi Harrag, and Khaled Shaalan. “Advancing Low-Resource Dialect Identification: A Hybrid Cross-Lingual Model Leveraging CAMeLBERT and FastText for Algerian Arabic.” Expert Systems with Applications 284 (July 2025): 127816. https://doi.org/10.1016/j.eswa.2025.127816. DOI: https://doi.org/10.1016/j.eswa.2025.127816

Daman, Damanhuri, and Jamiluddin Yacub. “Mengenalkan Huruf Hijaiyah Pada Anak Usia Dini.” Azzahra: Jurnal Pendidikan Anak Usia Dini 3, no. 2 (2022). https://ejournal.staidarussalamlampung.ac.id/index.php/azzahra/article/view/374.

Fadjeri, Akhmad. Computer Vision Untuk Pemula. CV. Rumah Kreatif Wadas Kelir, 2024.

Khalifa, Salam, Abdelrahim Qaddoumi, Ellen Broselow, and Owen Rambow. “Picking Up Where the Linguist Left Off: Mapping Morphology to Phonology through Learning the Residuals.” Proceedings of The Second Arabic Natural Language Processing Conference, 2024, 258–64. https://doi.org/10.18653/v1/2024.arabicnlp-1.22. DOI: https://doi.org/10.18653/v1/2024.arabicnlp-1.22

Khan, Nisar, Riaz Ahmad, Khalil Ullah, et al. “Robust Arabic and Pashto Text Detection in Camera-Captured Documents Using Deep Learning Techniques.” IEEE Access 11 (2023): 135788–96. https://doi.org/10.1109/ACCESS.2023.3336404. DOI: https://doi.org/10.1109/ACCESS.2023.3336404

Khanam, Rahima, and Muhammad Hussain. “What Is YOLOv5: A Deep Look into the Internal Features of the Popular Object Detector.” Version 1. Preprint, arXiv, 2024. https://doi.org/10.48550/ARXIV.2407.20892.

Khanam, Rahima, and Muhammad Hussain. “YOLOv11: An Overview of the Key Architectural Enhancements.” Version 1. Preprint, arXiv, 2024. https://doi.org/10.48550/ARXIV.2410.17725.

Khlif, Wafa. “Multi-Lingual Scene Text Detection Based on Convolutional Neural Networks.” Doctorat ès Informatique et applications, PhD, La Rochelle, 2022. https://doi.org/10.70675/cb8b6b05z423ez4574z8799z6408ef415b50. DOI: https://doi.org/10.70675/cb8b6b05z423ez4574z8799z6408ef415b50

Kusumawan, Ardhian Tirta, and Fetty Poerwita Sary. “The Influence of Job Characteristics on Job Application Intention Through Employer Attractiveness as a Mediating Variable on Generation Z in the Indonesian Pharmaceutical Manufacturing Industry.” Enrichment: Journal of Multidisciplinary Research and Development 3, no. 3 (2025): 400–413. https://doi.org/10.55324/enrichment.v3i3.389. DOI: https://doi.org/10.55324/enrichment.v3i3.389

Lubis, Torkis, Sounia Rabhi, and Rahmat Linur. “Enhancing Arabic Pronunciation Learning through Contrastive Analysis: An Empirical Study with Indonesian University Students.” International Journal of Special Education 41, no. 2 (2026). https://www.internationalsped.com/index.php/ijse/article/view/3331.

Medina, Zahra Nida Al, Zukhaira Zukhaira, and Muhammad Yusuf Ahmad Hasyim. “The Arabic Adventure; Media Board Game Untuk Keterampilan Berbicara Bahasa Arab Siswa Kelas VII MTs Di Purbalingga.” Lisanul Arab: Journal of Arabic Learning and Teaching 8, no. 1 (2019): 1. https://journal.unnes.ac.id/sju/laa/article/view/32541.

Milo, Thomas, and Alicia González Martínez. “A New Strategy for Arabic OCR: Archigraphemes, Letter Blocks, Script Grammar, and Shape Synthesis.” Proceedings of the 3rd International Conference on Digital Access to Textual Cultural Heritage, May 8, 2019, 93–96. https://doi.org/10.1145/3322905.3322928. DOI: https://doi.org/10.1145/3322905.3322928

Mohamed, Yuslina, Zainurrijal Abd Razak, Tuan Haji Sulaiman Ismail, Nada Ibrahim Alribdi, and Mesbahul Hoque. “A Systematic Review Of Arabic Phonetic: Hijaiyyah’s Pronunciation Among New Learners.” Ijaz Arabi Journal of Arabic Learning 7, no. 2 (2024). https://doi.org/10.18860/ijazarabi.v7i2.24173. DOI: https://doi.org/10.18860/ijazarabi.v7i2.24173

Mosbah, Lamia, Ikram Moalla, Tarek M. Hamdani, Bilel Neji, Taha Beyrouthy, and Adel M. Alimi. “ADOCRNet: A Deep Learning OCR for Arabic Documents Recognition.” IEEE Access 12 (2024): 55620–31. https://doi.org/10.1109/ACCESS.2024.3379530. DOI: https://doi.org/10.1109/ACCESS.2024.3379530

Muaad, Abdullah Y., Md Belal Bin Heyat, Faijan Akhtar, et al. “Artificial Intelligence for Text Analysis in the Arabic and Related Middle Eastern Languages: Progress, Trends, and Future Recommendations.” International Journal of Intelligent Systems 2025, no. 1 (2025): 6091900. https://doi.org/10.1155/int/6091900. DOI: https://doi.org/10.1155/int/6091900

Murat, Ayşe Aybilge, and Mustafa Servet Kiran. “A Comprehensive Review on YOLO Versions for Object Detection.” Engineering Science and Technology, an International Journal 70 (October 2025): 102161. https://doi.org/10.1016/j.jestch.2025.102161. DOI: https://doi.org/10.1016/j.jestch.2025.102161

Muthmainnah, Hilyah Ahya, and Lina Marlina. “Kesalahan Fonologi Dalam Membaca Al-Qur’an Pada Siswi Penutur Bahasa Sunda Di Tahfiz Qur’an Al-Marjan.” Al-Lahjah : Jurnal Pendidikan, Bahasa Arab, Dan Kajian Linguistik Arab 8, no. 1 (2025). https://ejournal.unwaha.ac.id/lahjah/article/view/5936.

Nailurrahmi, Fitma, and Lina Marlina. “Analisis Interferensi Fonetik Bahasa Ibu Tehadap Pelafalan Fonem Bahasa Arab Dalam Pembelajaran Kosakata.” Moderasi : Journal of Islamic Studies 5, no. 1 (2025): 279–90. https://doi.org/10.54471/moderasi.v5i1.113. DOI: https://doi.org/10.54471/moderasi.v5i1.113

Naim, Ngainun, Abdul Aziz, and Teguh Teguh. “Integration of Madrasah Diniyah Learning Systems for Strengthening Religious Moderation in Indonesian Universities.” International Journal of Evaluation and Research in Education (IJERE) 11, no. 1 (2022): 108. https://doi.org/10.11591/ijere.v11i1.22210. DOI: https://doi.org/10.11591/ijere.v11i1.22210

O’Connell, Nicola. “Research and Development.” Nurse Prescribing 5, no. 5 (2007): 226–29. https://doi.org/10.12968/npre.2007.5.5.23745. DOI: https://doi.org/10.12968/npre.2007.5.5.23745

Politeknik Negeri Batam. “Hijaiyah Letters Computer Vision Model.” Roboflow. Accessed June 11, 2026. https://universe.roboflow.com/politeknik-negeri-batam-dtarv/hijaiyah-letters.

Rahayu, Ratih, and Sri Munawarah. “Investigation of Yogyakarta Dialect’s Vocabulary in Areas of Banyumas’ Ngapak Dialect.” In Sociolinguistics and Dialectological Studies in Indonesia. Nova Science Publishers, Inc., 2021. https://www.scopus.com/pages/publications/85116743336.

Redmon, Joseph, Santosh Divvala, Ross Girshick, and Ali Farhadi. “You Only Look Once: Unified, Real-Time Object Detection.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, 779–88. https://doi.org/10.1109/CVPR.2016.91. DOI: https://doi.org/10.1109/CVPR.2016.91

Slamaa, Amany A. “AI-Driven Semantic Information Retrieval for Arabic Language- Systematic Literature Review.” Computer Science Review 60 (May 2026): 100920. https://doi.org/10.1016/j.cosrev.2026.100920. DOI: https://doi.org/10.1016/j.cosrev.2026.100920

Tuama, Bilal Abdulrahman, and Farhan Mohamed. “A Systematic Literature Review of Deep Learning Methods for Handwritten Text Recognition in Historical Arabic Manuscripts.” Engineering, Technology & Applied Science Research 15, no. 4 (2025): 25772–82. https://doi.org/10.48084/etasr.12123. DOI: https://doi.org/10.48084/etasr.12123

Wardana, Ari Kusuma, Marti Widya Sari, and Kartikadyota Kusumaningtyas. “Developing Martial Art Championship Scheduling System Using the Waterfall Model.” Proceeding of the 7th International Conference of Science, Technology, and Interdisciplinary Research (IC-STAR 2021) (Bandar Lampung, Indonesia), 2023, 040006. https://doi.org/10.1063/5.0105697. DOI: https://doi.org/10.1063/5.0105697

Yafooz, Wael M. S. “Enhancing Arabic Dialect Detection on Social Media: A Hybrid Model with an Attention Mechanism.” Information 15, no. 6 (2024): 316. https://doi.org/10.3390/info15060316. DOI: https://doi.org/10.3390/info15060316

Yaseen, Muhammad. “What Is YOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector.” Version 1. Preprint, arXiv, 2024. https://doi.org/10.48550/ARXIV.2408.15857.

Zaatiti, Hadi, Hatem Hajri, Osama Abdullah, and Nader Masmoudi. “Towards Stable AI Systems for Evaluating Arabic Pronunciations.” NLP and Machine Learning Trends 2025, August 23, 2025, 23–33. https://doi.org/10.5121/csit.2025.151603. DOI: https://doi.org/10.5121/csit.2025.151603

Zhao, Qian, and Junhao Zhu. “An Improved YOLOv11 Architecture with Multi-Scale Attention and Spatial Fusion for Fine-Grained Residual Detection.” Results in Engineering 27 (September 2025): 107061. https://doi.org/10.1016/j.rineng.2025.107061. DOI: https://doi.org/10.1016/j.rineng.2025.107061