Abstract
Human-AI cooperative classification (HAI-CC) approaches aim to develop hybrid
intelligent systems that enhance decision-making in various high-stakes
real-world scenarios by leveraging both human expertise and AI capabilities.
Current HAI-CC methods primarily focus on learning-to-defer (L2D), where
decisions are deferred to human experts, and learning-to-complement (L2C),
where AI and human experts make predictions cooperatively. However, a notable
research gap remains in effectively exploring both L2D and L2C under diverse
expert knowledge to improve decision-making, particularly when constrained by
the cooperation cost required to achieve a target probability for AI-only
selection (i.e., coverage). In this paper, we address this research gap by
proposing the Coverage-constrained Learning to Defer and Complement with
Specific Experts (CL2DC) method. CL2DC makes final decisions through either AI
prediction alone or by deferring to or complementing a specific expert,
depending on the input data. Furthermore, we propose a coverage-constrained
optimisation to control the cooperation cost, ensuring it approximates a target
probability for AI-only selection. This approach enables an effective
assessment of system performance within a specified budget. Also, CL2DC is
designed to address scenarios where training sets contain multiple noisy-label
annotations without any clean-label references. Comprehensive evaluations on
both synthetic and real-world datasets demonstrate that CL2DC achieves superior
performance compared to state-of-the-art HAI-CC methods.