Peningkatan Performa Model Pengenalan Hitungan Jari (1–5) Berbasis Konteks Budaya Menggunakan Lokalisasi YOLO, Transfer Learning, dan Grad-CAM untuk Explainable AI
DOI:
https://doi.org/10.53696/venn.v5i3.493Keywords:
Finger Counting, Lokalisasi Tangan, YOLOv10x, Transfer Learning, Explainable AIAbstract
Pengenalan gestur hitungan jari (1–5) atau daktilonomi memiliki peran fundamental dalam pembelajaran matematika dasar sebagai representasi kognitif yang terwujud (embodied numerosity). Namun, model klasifikasi pada Natural Hand Digit Dataset masih menunjukkan akurasi terbatas (sekitar 66%) akibat gangguan latar belakang, variasi posisi, dan pencahayaan. Penelitian ini bertujuan untuk meningkatkan performa pengenalan gestur angka melalui lokalisasi tangan berbasis YOLOv10x yang dilatih pada dataset HaGRID sebagai tahap pra-pemrosesan, serta mengintegrasikan Grad-CAM untuk explainable artificial intelligence (XAI). Metode yang digunakan terdiri atas deteksi dan pemotongan wilayah tangan menggunakan YOLOv10x, kemudian klasifikasi jumlah jari menggunakan transfer learning ResNet50 dengan fine-tuning dan callback. Hasil eksperimen menunjukkan bahwa penggunaan citra hasil lokalisasi tangan (cropped) meningkatkan akurasi hingga 73,98% dengan macro F1-score 72,44%, jauh lebih baik dibandingkan citra asli (akurasi 41,66%). Visualisasi Grad-CAM mengonfirmasi bahwa model benar-benar fokus pada konfigurasi jari yang relevan pada prediksi benar, sementara pada prediksi salah, peta aktivasi mengungkap penyebab kesalahan (misalnya angka "2" dikenali sebagai "1" karena fokus pada satu jari). Penelitian ini menyimpulkan bahwa pendekatan lokalisasi tangan sebelum klasifikasi efektif meningkatkan performa dan ketahanan sistem, sementara Grad-CAM memberikan transparansi dan interpretabilitas yang penting untuk membangun kepercayaan dalam aplikasi pembelajaran matematika interaktif.
Downloads
References
Ahmad, K. A., Nurrifqhi, A., Fettyana, F., & Fadilah, N. (2025). Pembangunan Dataset Gestur Angka (1-5) yang Terkontekstualisasi Budaya sebagai inisiasi Basis Data Pengenalan untuk Alat Bantu Pembelajaran Matematika.
Ahmad, K. A., Silpani, D. C., & Yoshida, K. (2023). The Impact of Large Sample Datasets on Hand Gesture Recognition by Hand Landmark Classification. International Journal of Affective Engineering, 22(3), 253–259. https://doi.org/10.5057/ijae.ijae-d-22-00022
Balat, M., Awaad, R., Zaky, A. B., & Aly, S. A. (2025). Revolutionizing Communication with Deep Learning and XAI for Enhanced Arabic Sign Language Recognition. http://arxiv.org/abs/2501.08169
Bender, A., & Beller, S. (2012). Nature and culture of finger counting: Diversity and representational effects of an embodied cognitive tool. Cognition, 124(2), 156–182. https://doi.org/10.1016/j.cognition.2012.05.005
di Luca, S., & Mauro, P. (2011). Finger numeral representations: More than just another symbolic code. In Frontiers in Psychology (Vol. 2, Number NOV). https://doi.org/10.3389/fpsyg.2011.00272
Elharrouss, O., Akbari, Y., Almaadeed, N., & Al-Maadeed, S. (2022). Backbones-Review: Feature Extraction Networks for Deep Learning and Deep Reinforcement Learning Approaches. https://doi.org/10.1016/j.cosrev.2024.100645
Foteinos, K., Linardakis, M., Radoglou-Grammatikis, P., Argyriou, V., Sarigiannidis, P., Varlamis, I., & Papadopoulos, G. Th. (2026). Visual Hand Gesture Recognition with Deep Learning: A Comprehensive Review of Methods, Datasets, Challenges and Future Research Directions. http://arxiv.org/abs/2507.04465
He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep Residual Learning for Image Recognition. http://arxiv.org/abs/1512.03385
Kapitanov, A., Kvanchiani, K., Nagaev, A., Kraynov, R., & Makhliarchuk, A. (2024). HaGRID - HAnd Gesture Recognition Image Dataset. https://doi.org/10.1109/WACV57701.2024.00451
Nuzhdin, A., Nagaev, A., Sautin, A., Kapitanov, A., & Kvanchiani, K. (2024). HaGRIDv2: 1M Images for Static and Dynamic Hand Gesture Recognition. http://arxiv.org/abs/2412.01508
Pika, S., Nicoladis, E., & Marentette, P. (2009). How to order a beer: Cultural differences in the use of conventional gestures for numbers. Journal of Cross-Cultural Psychology, 40(1), 70–80. https://doi.org/10.1177/0022022108326197
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2019). Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. https://doi.org/10.1007/s11263-019-01228-7
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Khawaritmi Abdallah Ahmad, Rahmat Syam, Mariani, Ahmad Talib
This work is licensed under a Creative Commons Attribution 4.0 International License.
The authors certify that:
- if the manuscript is co-authored, they are authorized by their co-authors to enter into these arrangements.
- the work described has not been formally published before in a registered ISSN or ISBN media, except in the form of an abstract or as part of a published lecture, review, or thesis.
- it is not under consideration for publication elsewhere,
- its publication has been approved by all the author(s) and by the responsible authorities – tacitly or explicitly – of the institutes where the work has been carried out.
- they secure the right to reproduce any material that has already been published or copyrighted elsewhere (it does not infringe on the rights of others).
- they agree to the license and copyright agreement.
All articles published are licensed under Creative Commons Attribution 4.0 International License.
- Authors retain copyright and other proprietary rights related to the article.
- Authors retain the right and are permitted to use the substance of the article in their own future works, including lectures and books.
- Authors grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution License (CC BY 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post or self-archive their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
