Every day thousands of otolaryngology procedures rely on surgical loupes and bulky microscopes. Changing magnification means physically swapping lenses — which can add upwards of 30 minutes to a case — and the equipment crowds an already tight operating room.
Project Iris replaces that with a head-worn AR device that magnifies the surgical field, segments the anatomy in real time, and is controlled without breaking sterility. We built it as a proof of concept for vocal-fold surgery, with the architecture meant to generalize to other surgical domains.
- Segmentation mAP
- 87–95%
- Magnification
- up to 20×
- Training images
- 6,228
- Surgeons in user study
- ~20
Four systems, one device
Gesture & tactile control — adjustable magnification up to 20× driven by non-touch gestures and a sterile, wireless neck brace with tactile buttons, so the surgeon never breaks the sterile field or raises their hands overhead.
Instance segmentation — a YOLOv8 model that identifies and outlines the vocal folds live in the headset's view.
Optical-flow stabilization — anchored on the segmentation masks to cancel head tremor at high magnification, where tiny movements are massively amplified.
Android integration — an on-device app for OR parameter configuration and data collection, running fully standalone with no cloud dependency.
The segmentation model
To our knowledge, YOLOv8 had not previously been applied to vocal-fold segmentation. We trained on 6,228 images of the surgical field, augmented in Roboflow (Gaussian noise, motion blur, zoom, rotations) to improve generalization, and ran a Ray Tune random search over batch size, dropout, weight decay, and learning-rate schedule.
The best model reached 87–95% mAP, segmenting the vocal folds robustly across varied lighting and viewpoints — clear enough to sharpen perception for experienced surgeons and to aid training for new ones.
Hardware & prototype
Built on an Epson Moverio BT-40 AR headset paired with a ROK-3588s single-board computer — enough on-board compute for the ML, in a package smaller, lighter, and cheaper than a compatible smartphone. An adjustable-pitch camera bracket lets surgeons keep a straight, ergonomic posture.
We validated the prototype in a user study at Grady Memorial Hospital / Emory University School of Medicine: roughly 20 surgeons and 13 independent studies comparing the headset against microscopes and loupes on a suturing task. Feedback was strongly positive.
Built with