Assistive Tech with AR Way-finding
24 Sep 2018I joined a mini-batch at the Recurse Center last week, and spent my time researching computer vision for people with visual impairments. I came up with an app to find empty seats in a crowded space, like a classroom or conference hall. Here’s a breakdown of what I did and why.
This clip shows me panning the device around the room. Sighted users will see a box outlining the chairs in the viewport.
The Problem
A person with a visual impairment may find it annoying and uncomfortable to find a seat at an event. From talking with friends of mine who are blind, this is especially true because
Blind folks get unwanted attention when way-finding.
This can lead to disruptions, and sometimes unwelcome guidance.
Hypothesis
Mobile phones now have enough power to run real-time object recognition, using their built in cameras and processing power. We can use these advancements to augment traditional way-finding devices like canes and service animals.
Challenges
- Recording in public is still taboo and conspicuous.
- People with disabilities don’t have a lot of cash to spend, due to high unemployment rates.
Design
I built the initial design using Adobe XD, using auto-animate and audio output features. In the process I discovered that audio cues needed to be very succinct and in some cases replaced with vibration, and that I’d need to signal obstacles, distance, and placement, as well as just identifying the object.
Approach
There are many computer vision models for neural networks. Some have existing ports to mobile phones. iOS also has it’s own native object recognition in ARKit.
iOS is more popular than Android with people using screen readers, so I started there. I tried ARKit’s object classifier first, but it does not identify more that one object at once. It’s also limited to newer devices.
Instead of ARKit 2, I turned to Forge, an iOS toolkit for neural networks. It includes the YOLO neural network. This implementation is limited to 20 object categories, but they include chairs and sofas. Recognition is super fast, on the order of milliseconds, and works reliably in a handful of conditions. It’s also not restricted to newer devices.
There’s still work left to do, specifically adding placement and distance indication.
Credit
Matthijs Hollemans’s work porting YOLO to iOS made this possible. Also, thanks to Gus and Chancey for the usability feedback, and to the Recurse Center for being a great place to learn and experiment.