Image Describe Apple Shortcut
The Inspiration
While Android users recently gained the ability to get AI-powered image descriptions via TalkBack and Gemini, iOS users haven't had a native equivalent. VoiceOver can provide image descriptions, but they aren't as rich as the ones powered by large language models. Existing third-party apps, while extremely useful, require people to manually share images via the share sheet. Others in the community have created great shortcuts that work similarly to this shortcut that I created, but they use paid APIs. To bridge this gap, I developed a custom shortcut that leverages Apple Intelligence ChatGPT integration to provide seamless, free image descriptions.
How does the shortcut work?
Activated by a simple VoiceOver gesture, the shortcut streamlines getting image descriptions through the following steps:
- A screenshot is captured upon activation.
- The image is sent to ChatGPT with a prompt designed to identify the specific UI element currently focused by VoiceOver.
- A description is returned to the user, who can then ask follow-up questions.
- The entire process occurs without the user ever leaving their current app.
Why are image descriptions like this important?
Having image descriptions on demand opens up all sorts of possibilities for someone who is blind or visually impaired. It enables people to look at their photo library and memories, browse picture based apps like instagram, look at a graphic in an article, and much more.