This paper describes two studies with people with sensory, cognitive and motor impairments flying drones as a leisure activity. We found that despite several adaptations to existing technologies that match their abilities, flying remains very difficult due to the required perceptual and motor skills. We propose an adaptation space at hardware, software and automation levels to facilitate drone piloting. Based on this adaptation space, we designed HandiFly, a software for piloting drones which is adaptable to users' abilities. Participants were able to fly and emphasized its ability to be tailored to specific needs. We conclude with future direction to make drone piloting a more inclusive activity.