Fostering Progress in Performance Evaluation and Benchmarking of Robotic and Automation Systems

Abstract

We have shared benchmarks for many engineering systems and products in the market that can be used to compare solutions and systems. We can compare cars in terms of maximum speed, acceleration, and maximum torque; computers in terms of flops, random access memory, and hard disk capacity; and smartphones in terms of battery life and screen dimensions. We also have shared usability metrics based on human factors, which are used to compare the ease of use of different software interfaces. When we come to the evaluation and the comparison of how intelligent, robust, adaptive, and antifragile the behaviors of robots are in performing a given set of tasks, such as daily life activities with daily life objects such as in a kitchen or a hospital room, we are in trouble.