“A new position paper argues that AI evaluations need item-level benchmark data to address systemic validity failures in how we assess generative AI systems. Current evaluation paradigms suffer from unjustified design choices and misaligned metrics, making granular diagnostic analysis essential before deploying AI in high-stakes domains.”