Design and Implementation of Artificial Intelligence-based Real-time Pothole Detection for IoT Smart Infrastructure

  • Rahardhita Widyatra Sudibyo Department of Electronic Engineering, Politeknik Elektronika Negeri Surabaya
  • Akhmad Alimudin Department of Multimedia Creative Technology, Politeknik Elektronika Negeri Surabaya
Abstract views: 88 , PDF downloads: 109
Keywords: IoT, AI, Pothole Detection, CNN, Mobilenet SSDV2


The Internet of Things (IoT) has been extensively deployed for Smart Cities due to its ability to process many different and heterogeneous end systems. The IoT innovation encourages artificial intelligence applications to process data. In Smart City infrastructure, the road is a critical component of transportation infrastructure that supports the economic, social, and cultural things of community life and various aspects of community life. Road conditions affect a variety of community activities. Good roads enhance comfort and support local businesses. However, many roads remain in bad condition, such as potholes. Various methods have been attempted to identify potholes, especially the two-dimensional imaging method. This paper proposes the real-time Artificial Intelligence detection of potholes using the Convolutional Neural Network (CNN), which leverages the Edge Tensor Processing Unit (TPU) with the MobileNet SSD v2. The system was set up on a Jetson Nano with a few extras, including a camera and GPS, to support the IoT infrastructure. Evaluation for the model consists of device implementation, model evaluation, GPS position deviation, and on-road implementation. The effectiveness is confirmed through experiments using a system test-bed that generates ideal mAP off 0.22 and recall values.


Author Biographies

Rahardhita Widyatra Sudibyo, Department of Electronic Engineering, Politeknik Elektronika Negeri Surabaya



Akhmad Alimudin, Department of Multimedia Creative Technology, Politeknik Elektronika Negeri Surabaya




B. B. V. Rao, H. G. Chandrakanth, "IoT based Machine Learning Automation Algorithm for Controlling the Industrial Loads", in Int. J. of Intelligent Engineering and Systems, Vol.14, No.4, pp. 146-156, Aug. 2021.

S. Deng, H. Zhao, W. Fang, J. Yin, S. Dustdar and A. Y. Zomaya, "Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence", in IEEE Internet of Things Journal, vol. 7, no. 8, pp. 7457-7469, Aug. 2020.

I. R. Putra, “Ministry of Public Works: 3.338 Holes along the North Coast Road in Java Island”, Accessed 18 Nov. 2022.

Md. S. Arman, Md. M. Hasan, F. Sadia, A. K. Shakir, K. Sarker, and F. A. Himu, "Detection and Classification of Road Damage Using RCNN and Faster R-CNN: A Deep Learning Approach", in Cyber Security and Computer Science. ICONCS 2020. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 325. Springer, July 2020.

R. Mukherjee, H. Iqbal, S. Marzban, A. Badar, T. Brouns, S. Gowda, E. Arani, and B. Zonoos, "AI Driven Road Maintenance Inspection," Proc. 27th ITS World Congress, June 2021.

S. Naddaf-Sh, M. M. Naddaf-Sh, A. R. Kashani, and H. Zargarzadeh, "An Efficient and Scalable Deep Learning Approach for Road Damage Detection," Proc. 2020 IEEE Int. Conf. Big Data, pp. 5602–5608, 2020.

Z. Liu, W. Wu, X. Gu, S. Li, L. Wang, and T. Zhang, "Application of combining yolo models and 3d gpr images in road detection and maintenance," Int. J. Remote Sens., vol. 13, no. 6, pp. 1–18, 2021.

Z. Pei, R. Lin, X. Zhang, H. Shen, J. Tang, and Y. Yang, "CFM: A Consistency Filtering Mechanism for Road Damage Detection," Proc. 2020 IEEE Int. Conf. Big Data, pp. 5584-5591, 2020.

D. Arya, H. Maeda, S. K. Ghosh, D. Toshniwal, A. Mraz, T. Kashiyama, and Y. Sekimoto, "Deep learning-based road damage detection and classification for multiple countries," Int. J. Autom. Constr., vol. 132, 2021.

Y. Liu, X. Zhang, B. Zhang, and Z. Chen, "Deep Network for Road Damage Detection," Proc. 2020 IEEE Int. Conf. Big Data, pp. 5572–5576, 2020.

F. Kortmann, K. Talits, P. Fassmeyer, A. Warnecke, N. Meier, J. Heger, P. Drews, and B. Funk, "Detecting Various Road Damage Types in Global Countries Utilizing Faster R-CNN," Proc. 2020 IEEE Int. Conf. Big Data, pp. 5563–5571, 2020.

T. Hascoet, Y. Zhang, A. Persch, R. Takashima, T. Takiguchi, and Y. Ariki, "Faster RCNN Monitoring of Road Damages: Competition and Deployment," Proc. 2020 IEEE Int. Conf. Big Data, pp. 5545–5552, 2020.

D. Arya, H. Maeda, S. K. Ghosh, D. Toshniwal, and Y. Sekimoto, "RDD2020: An annotated image dataset for automatic road damage detection using deep learning," Int. J. Data in Brief, vol. 36, 2021.

A. Mahardika, Y. Sari, and C. Dewi, "Sistem Temu Kembali Citra Lubang Jalan Aspal Berdasarkan Tingkat Kerusakan Menggunakan Ekstraksi Fitur Gray Level Co-occurrence Matrix", in Jurnal Pengembangan Teknologi Informasi dan Ilmu Komputer, vol. 2, no. 10, pp. 3811-3821, Feb. 2018.

H. Maeda, Y. Sekimoto, and T. Seto, "Lightweight Road Manager: Smartphone-based Automatic Determination of Road Damage Status by Deep Neural Network", in MobiGIS '16: Proc. of the 5th ACM SIGSPATIAL International Workshop on Mobile Geographic Information Systems, pp. 37–45, 2016.

I. D. Pratama, H. Mahmudah, and R. W. Sudibyo, "Design and Implementation of Real-time Pothole Detection using Convolutional Neural Network for IoT Smart Environment", in Proc. 2021 Int. Electron. Symp. IES pp. 675–679, 2021.

Z. S. Hernanda, H. Mahmudah, and R. W. Sudibyo, "CNN-Based Hyperparameter Optimization Approach for Road Pothole and Crack Detection Systems", in Proc. 2022 World AI IoT Congress pp. 549–554, 2022.

K. E. An, S. W. Lee, S.-K. Ryu, and D. Seo, "Detecting a Pothole using Deep Convolutional Neural Network Models for an Adaptive Shock Oserving in a Vehicle Driving", in Proc. IEEE International Conference on Consumer Electronics (ICCE), pp. 1-2, 2018.

V. Pereira, S. Tamura, S. Hayamizu, and H. Fukai, "A deep learning based approach for road pothole detection in Timor Leste", in Proc. IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI), pp. 279-284, 2018.

L. K. Suong and J. Kwon, "Detection of Potholes Using a Deep Convolutional Neural Network", in Int. J. UCS, vol. 24, no. 9, pp. 1244– 1257, 2018.

Y. Y. F. Panduman, A. R. A. Besari, S. Sukaridhoto, R. P. N. Budiarti, R. W. Sudibyo, and F. Nobuo, “Implementation of Integration VaaMSN and SEMAR for Wide Coverage Air Quality Monitoring”, in Int. Journal TELKOMNIKA, vol. 16, no. 6, p. 2630, Dec. 2018.

Z. Zhao, P. Zheng, S. Xu and X. Wu, "Object Detection with Deep Learning: A Review", in IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 11, pp. 3212-3232, Nov. 2019.

W. S. E. Putra, A. Y. Wijaya, and R. Soelaiman, “Klasifikasi Citra Menggunakan Convolutional Neural Network (CNN) pada Caltech 101”, in Jurnal Teknik ITS, vol. 5, no. 1, pp. A65-A69, 2016.

Aparna, Y. Bhatia, R. Rai, V. Gupta, N. Aggarwal, and A. Akula, "Convolutional Neural Networks based Potholes Detection using Thermal Imaging", in Journal of King Saud University - Computer and Information Sciences, Feb. 2019.

N. C. Thompson, K. Greenewald, K. Lee, and G. F. Manso, "The Computational Limits of Deep Learning", in arXiv:2007.05558v1, July 2020.

NVIDIA, Jetson Nano Developer Kit, embedded/jetson-nano-developer-kit. Accessed 18 Nov. 2022.

A. Reuther, P. Michaleas, M. Jones, V. Gadepally, S. Samsi, and J. Kepner, "Survey of Machine Learning Accelerators", in arXiv:2009.00993v1, Sep. 2020.

W. Li and M. Liewig, "A Survey of AI Accelerators for Edge Environment", in Trends and Innovations in Information Systems and Technologies. WorldCIST 2020. Advances in Intelligent Systems and Computing, vol 1160. Springer, Cham, Jun. 2020.

A. Basulto-Lantsova, J. A. Padilla-Medina, F. J. Perez-Pinal, and A. I. Barranco-Gutierrez, "Performance Comparative of OpenCV Template Matching Method on Jetson TX2 and Jetson Nano Developer Kits", in Proc. 10th Annual Computing and Communication Workshop and Conference (CCWC). pp. 0812-0816, Jan. 2020.

Google Coral, Build Beneficial and Privacy Preserving AI, https: // Accessed 18 Nov. 2022.

L. A. Libutti, F. D. Igua, L. Pi˜nuel, and L. D. Giusti, "Benchmarking Performance and Power of USB Accelerators for Inference with MLPerf", in 2nd Workshop on Accelerated Machine Learning (AccML), Spain 2020.

Coral Benchmark, Edge TPU Performance Benchmarks, Accessed 18 Nov. 2022.

J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, et al, "Speed/accuracy trade-offs for modern convolutional object detectors", in Proc. of the IEEE conference on computer vision and pattern recognition, pp. 7310–7311, 2017.

Y. Zhang, H. Pheng, and P. Hu, "CS341 Final Report: Towards Real-time Detection and Camera Triggering", in 2017.

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks", in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.

B. Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference", in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2018.

T. Vincenty, "Direct and Inverse Solutions of Geodesics on the Ellipsoid with Application of Nested Equation", in Survey Review Ministry of Overseas Development Surrey, vol. XXIII. no. 176, Apr. 1975.

Apache JMeter, Accessed 18 Nov. 2022.

How to Cite
Sudibyo, R. W., & Alimudin, A. (2023). Design and Implementation of Artificial Intelligence-based Real-time Pothole Detection for IoT Smart Infrastructure. Inform : Jurnal Ilmiah Bidang Teknologi Informasi Dan Komunikasi, 8(2), 125-131.