Flying drones is an increasingly popular but challenging activity. It is even more challenging for users with special needs. In this paper, we present our collaboration with a drone piloting workshop for people with motor and cognitive disabilities to design CandyFly, an application adaptable to the users' abilities. Several piloting workshops allowed pilots to have fun flying drones with CandyFly and allowed us to explore interactions to help pilots and their caregivers. We conclude with a discussion of the adaptations made along three axes: hardware, software and automation; which can be used for the design of future assistance and leisure systems.