Deep neural networks (DNNs) conquer more and more of the autonomous vehicle driving stack – up to end-to-end trained systems. Numerous challenges and benchmarks exist, inviting to push metrics higher and higher. But does this directly translate to a safer or improved automated vehicle operation? The practitioner asks: How good is good enough, and are we even measuring the right things?

Our workshop invites both scientific and industrial approaches to deeper understand and overcome what we call the “crisis of metrics”. We request contributions to the questions: Which metrics are the right ones for which problems? What are the dos and don’ts? Do singular metrics, e.g. of perception outputs, still have their place in end-to-end trained systems? How to combine metrics? Are metrics invariant enough towards diverse system outputs that could all be considered as safe and reasonable operation? Should metrics be designed to take downstream applications into account instead of being fully independent to prevent silo thinking across research groups or departments?

In the age of generative approaches, challenges regarding metrics go even beyond. How to measure quality without ground truth or usefulness for certain purposes? Is it necessary that generated videos/images are real-world like to be useful, or is it rather enough if specific aspects that the downstream application (like object detectors) focus on are accurately generated?

What are the right losses in DNN training and how is the interplay with the right metrics? Are intermediate losses useful in end-to-end learning systems, or do we only focus on the system’s outcome? Should metrics and losses be developed, which are invariant with respect to multiple “good” solutions?

With this workshop, we bring practitioners and researchers together to discuss and potentially advance the state of the art of metrics and benchmarks for automated driving and to obtain a common understanding about the challenges ahead.