THE LABHACKATHON · ACCESSIBILITY

Lumos

A camera-based assistant for blind and low-vision people. Point the phone at your surroundings, ask a question, and hear a description aloud.

THE HACKATHON DEMO

Point the camera.
Ask. Listen.

Lumos explores how a phone could help someone understand a scene through sound. The camera provides the visual context, the user can speak a question, and the assistant answers aloud.

In this recording, the camera looks at people outside a building. Lumos shows descriptions in Urdu and switches between listening, thinking and speaking as the scene and conversation change.

The demo also explores context across turns: the presenter introduces a friend by name, then asks about him later in the recording.

  • Camera contextUse the scene in front of the phone as the basis for a description.
  • Spoken interactionAsk about the surroundings and hear the response.
  • Visible feedbackListening, thinking and speaking states make the interaction easy to follow.
Read the demo transcript English translation · 1 min 23 sec

User What is happening in front of me?

Lumos Two young men are sitting on the steps, facing each other and talking.

User This is my friend Haris. Save him.

Lumos Haris is sitting on the steps, typing on his phone with both hands.

User Can you tell me whether my friend Haris is standing in front of me?

Lumos Haris is standing ahead on the left, looking at his phone.