Title: SPASL_v1: A Versatile and Efficient Robustness Benchmark for Image Classification Algorithms
Abstract: Image classification is one of the most fundamental vision tasks. Remarkable progresses in prediction accuracy have been achieved with the development of deep neural networks and vision transformers in the last decade. However, in real-world applications and interdisciplinary studies, these models face many challenges such as data quality
issue. Unexpected behaviors are often observed under such degraded situations. Researchers began to comprehend the robustness design of classification models. But there is a lack of compact and comprehensive benchmarks that quickly and precisely evaluate the robustness of a model in multiple dimensions. In this work, we propose a versatile and efficient robustness benchmark called SPASL_v1. It contains 50,000 images over 1,000 classes and assesses the performance of a model in five aspects: accuracies on very difficult images, images with partial information, robustness against adversarial attacks, speckle noise, and low resolution degradations. There is no human decision intervention in the image inclusion process of SPASL_v1. Images are selected based on the vote of a group of selected models. A total of 80 renowned and widely utilized models are tested. Among them, ViT-L/16 secures the best SPASL score of 51.16 out of 100. On the other hand, 74 models (92.5%) achieved a score under 30.
Bio: Yiming Bian (Member, IEEE) received the B.S. degree in computer science from the Tianjin University of Technology, Tianjin, China, in 2018, and the M.S. degree in computer engineering from Iowa State University, in 2020, where he is currently pursuing the Ph.D. degree under the supervision of Prof. Arun K. Somani. His current research interests are computer vision, deep learning, and optimizations. Specifically, he is interested in low-quality image recognition, robustssness design of deep learning models and large model optimizations.
Please click this URL to start or join. https://iastate.zoom.us/j/96810972944?pwd=SVVLWlY2cVdZYXhxWWg4ZHF1cVdSZz09
Or, go to https://iastate.zoom.us/join and enter meeting ID: 968 1097 2944 and password: 334840
Join from dial-in phone line:
Dial: +1 309 205 3325 or +1 312 626 6799
Meeting ID: 968 1097 2944
Participant ID: Shown after joining the meeting
International numbers available: https://iastate.zoom.us/u/aqUgrVklM