Flying drones is an increasingly popular activity. However, it is challenging due to the required perceptual and motor skills for following and stabilizing the drone, especially for people with special needs. This paper describes CandyFly, an application supporting people with diverse sensory, cognitive and motor impairments to pilot drones. We observed an existing accessible piloting workshop and evaluated CandyFly during eight additional workshops over three and a half years using a research-through-design process and ability-based design methods. We identified users' needs, formulated requirements and explored adaptive interactions such as using pressure-sensitive keys, adjusting controls to the pilots’ range of motion, or limiting the drone’s degrees of freedom to cope with a broad range of disabilities. Our results show that the pilots and their caregivers enjoyed flying and emphasized CandyFly’s ability to be tailored to specific needs. Our findings offer a framework for designing adaptable systems and can support the design of future assistive and recreational systems.