Leakage-Aware Benchmarking of Lightweight Models for Robust Handwritten Hanacaraka Character Recognition

Abstract

Handwritten Hanacaraka character recognition is important for preserving Javanese script, but reported results can be difficult to compare because datasets, preprocessing procedures, and train-test separation protocols vary across studies. This study presents a leakage-aware benchmark of lightweight models on a public handwritten Hanacaraka dataset containing 20 basic character classes. A data audit removed 17 unreadable image files and retained 1,562 valid images. Two experimental settings were evaluated: a perceptual-hash grouped split for leakage-aware testing and a random-stratified split as an optimistic upper-bound scenario. The leakage-aware benchmark compared HOG with SVM, HOG with Random Forest, MobileNetV2 head-only training, fine-tuned MobileNetV2, and a confusion-aware MobileNetV2 variant. Fine-tuned MobileNetV2 achieved the best leakage-aware result with 53.82% accuracy and 49.59% macro-F1, while robustness testing under image distortions produced 47.85% accuracy and 44.53% macro-F1. In the optimistic random-stratified experiment, an ensemble of EfficientNetB0 and MobileNetV2 with test-time augmentation reached 74.11% accuracy and 74.24% macro-F1. The results indicate that stricter evaluation substantially lowers performance and that visually similar classes remain difficult. Therefore, future Hanacaraka recognition work should report leakage control, robustness, and confusion analysis, not only clean-set accuracy.