Abstract
With the development of Human-AI Collaboration in Classification (HAI-CC),
integrating users and AI predictions becomes challenging due to the complex
decision-making process. This process has three options: 1) AI autonomously
classifies, 2) learning to complement, where AI collaborates with users, and 3)
learning to defer, where AI defers to users. Despite their interconnected
nature, these options have been studied in isolation rather than as components
of a unified system. In this paper, we address this weakness with the novel
HAI-CC methodology, called Learning to Complement and to Defer to Multiple
Users (LECODU). LECODU not only combines learning to complement and learning to
defer strategies, but it also incorporates an estimation of the optimal number
of users to engage in the decision process. The training of LECODU maximises
classification accuracy and minimises collaboration costs associated with user
involvement. Comprehensive evaluations across real-world and synthesized
datasets demonstrate LECODU's superior performance compared to state-of-the-art
HAI-CC methods. Remarkably, even when relying on unreliable users with high
rates of label noise, LECODU exhibits significant improvement over both human
decision-makers alone and AI alone.