iCan Eye

Problem

A useful assistive camera must do more than capture an image. The system has to move camera data across a constrained wireless connection, determine which vision service is actually available, and return information in a form that does not depend on sight.

iCan Eye addressed that complete path. A camera attached to the capstone hardware captures the scene, an iPhone receives the image, and the application speaks the resulting description.

Constraints

This work was part of a team capstone with shared hardware and protocol boundaries. Camera transfer had to operate over Bluetooth rather than a high bandwidth network connection. Cloud vision required connectivity and configuration. Local vision depended on the phone, operating system, bundled models, and runtime health.

The system was an engineering prototype, not a certified mobility aid. Navigation and other broader cane capabilities remained outside the strongest iCan Eye demonstration path.

Ownership

My responsibility covered the mobile application and the iCan Eye subsystem. I worked on the ESP32 camera firmware, Bluetooth image path, application integration, vision selection, diagnostics, and speech behavior.

Other team members contributed to the broader cane hardware. The case study keeps my application and camera work separate from that shared boundary.

Decisions

I treated Bluetooth image transfer as a protocol instead of assuming every notification would arrive correctly. The transfer carries size, sequence, completion, checksum, and error information so the application can distinguish a valid image from a stalled, incomplete, or corrupt one.

I also kept vision availability explicit. Runtime checks decide whether cloud or local processing can run. In automatic mode, the application reports an unavailable backend and selects another available path. A cloud only selection reports failure instead of silently changing modes.

Accessibility shaped the interaction rather than being added after the vision work. The primary result is concise spoken information supported by accessible controls, VoiceOver semantics, speech settings, and useful diagnostics.

Implementation

The ESP32 camera firmware captures JPEG frames and sends events plus image chunks through the shared Bluetooth protocol. The Flutter application validates and assembles each frame before passing it to the scene description service.

The scene description service selects Gemini when the configured cloud path is available or uses an available local vision path. Native iOS services and application level health checks expose backend status to the routing layer. The application then presents the description and speaks it through iOS text to speech.

The repository keeps the protocol definition, Dart implementation, firmware implementation, application services, and focused tests together so the boundary can be inspected from both sides.

Result

The result is an integrated prototype with a traceable camera to speech path. The public repository shows the firmware, protocol, mobile services, vision routing, and test surfaces behind that path.

Cloud vision is the most reliable demonstration path documented by the project. Bluetooth camera capture and local vision remain dependent on verified hardware and device capabilities.

Limitations

No claim is made that every local model runs on every iPhone, that live vision is ready for unsupervised outdoor navigation, or that the prototype replaces an established mobility aid or human guide.

The available public evidence does not support performance, accuracy, battery life, or user outcome numbers. Those claims would require controlled testing with preserved results.

Reflection

The hardest part was not any single model or interface. The challenge was making firmware, transport, mobile state, vision services, and audio behavior tell the truth about one another.

That work reinforced a principle I now carry into other systems. A graceful fallback is only useful when the software first knows and reports what failed.