Laparoscopic Skills Trainer

A browser app that watches a laparoscopic box trainer through its camera, recognises the kit and the progress of the task in real time, and scores the exercise. All inference runs on the device.

The problem

Laparoscopic skills are built on box trainers: a camera, two instruments and a set of standard drills. Someone has to watch each attempt and judge it, and that someone is usually a surgeon.

What I built

A web application that turns the box trainer’s own camera into the examiner. Open a page, place the kit, and the app recognises it, walks the trainee through the task and scores the result.

• It sees the task. Real-time vision models analyse the live camera feed: the kit, the instruments and the objects being manipulated.
• It follows the steps. Each drill is a sequence of steps such as insert the instruments, cut the first half of the circle or tighten the loop on the black mark, and the checklist advances as they are completed.
• It gives feedback as you go. Overlays show where the instruments should be, instruction banners and reference clips explain each step, and a timer and counters keep score.

How it works

The vision models are developed in PyTorch and deployed to the browser, where inference runs fully client-side on the trainee’s own device. The interactive gameplay is written in TypeScript, and I built custom evaluation metrics to track how the models and the system as a whole perform.