I Trained a Sign Language Detection Transformer — and You Can Too

Nicholas Renotte returns to sign language detection four years after his first model, this time rebuilding the whole library around a transformer detector. His promise is bold: collect your own data and train a personal model in under forty minutes. The rebuild also answers viewers who hit dependency problems with the older codebase.

The core is a Detection Transformer in the style of the 2020 paper: a ResNet-50 backbone extracts image features, positional encodings are added, and the result passes through transformer encoder and decoder layers. A torchinfo printout shows 26.9 million trainable parameters, with encoder depth, decoder depth, attention heads and hidden size left configurable. Two prediction heads come out at the end, one naming the class among hello, I love you and thank you, the other drawing the box on the frame.

Training hinges on the Hungarian matcher, a linear assignment routine that pairs the model's fixed set of 25 object queries with the true boxes. Three loss terms are weighted and summed: a classification term, a box-coordinate regression term and a generalized overlap term that brings geometry into the objective. A small worked example in the utilities folder lets the curious poke at the matching logic directly.

The fastest win lands in the first five minutes: clone the repo, open it in VS Code, run the realtime script with the uv runner, and the shipped weights start predicting from the webcam immediately. Those shipped weights come from a 4,426-epoch run that took roughly a day on a MacBook with no external GPU. A custom logging panel prints the model name, class count, inference time and frames per second, and a device index fixes most camera errors.

Data collection is deliberately plain: a capture helper grabs thirty frames per class for training, then five per class for testing. You wave each sign inside the frame while the logger counts up, and you delete blurred or misframed shots afterward. The video is frank that bad frames in means bad boxes out.

Labeling moves off the retired desktop tool onto Label Studio, which runs locally and free. You pick the bounding-box template, add the three labels in the exact video order, and that order matters because training and live prediction align to it. Keyboard shortcuts speed up the boxing, uploads go in batches under the hundred-file cap, and the finished set is exported in YOLO format with images to replace the train and test folders.

The data module is a standard PyTorch dataset with a split personality: training applies random crops and color jitter so the small set stretches further, while the test split disables those tricks. Running the module in test mode renders normalized images with their boxes, which doubles as a readiness check before any GPU time is spent.

The training script offers two paths: fine-tune from the shipped weights or comment out one line and start from random initialization. The recommended route keeps the shipped weights, with learning rate and per-term loss weights adjustable. The full run targeted ten thousand epochs and was stopped past four thousand; the demo run writes a checkpoint every ten epochs. Loss falls from the 7-8 band into the 5-6 band with dips into the 4s, while flat test loss is attributed to the tiny test split.

Testing loads the final checkpoint file into the test script and runs it over held-out frames. The panel reports dataset statistics, the active transforms, the architecture summary and, per prediction, the class name, confidence and box coordinates. That row-by-row readout is the honest part of the demo: you see exactly what the model caught and how sure it was.

The finale swaps the personal weights into the live script and points it at the webcam. A confidence gate at eighty percent filters the 25 queries down to the keepers, and lowering that gate in a live experiment shows how noisy the raw query set can get. Even small on-air mishaps, like a microphone blocking the frame, get fixed in stride.

Taken together the video builds a complete loop in one sitting: collect, label, train, test and deploy live, all on a laptop with no datacenter hardware. The personal weights attuned to your own hands are the quiet payoff. A longer architecture deep dive is offered if viewers want it.

To steelman the other side: a three-gesture demo does not prove DETR is the right tool. Practitioners would reasonably reach for a single-stage detector or a lightweight pose pipeline, which train faster on tiny datasets and run cheaper on CPU. The video never benchmarks against those, so its architecture choice reads as pedagogical rather than proven.

What is missing is any serious robustness test. Around ninety training images and a handful of test frames come from one person, one room, one camera, so lighting shifts, skin tones, cluttered backgrounds and occluded hands are never exercised. Test loss staying flat is waved away as a data-size artifact instead of being investigated, and the 25-query, 80-percent-threshold configuration is never ablated.

On verifiability: the headline numbers — 4,426 epochs in about 24 hours on a MacBook, loss falling from the 7-8 band into the 5-6 band — come from the author's own logs with no independent repeat. There is no mAP, precision-recall curve or held-out evaluation, only loss traces and eyeballed frames. I would re-run the checkpoints on my own hardware before quoting any of these figures as facts.

My practical read, first person: I would recommend this repo as a concept prover and a teaching loop, not as assistive technology. Real sign language translation needs large vocabularies, signer independence and temporal modeling, none of which a three-class still-image detector claims. If I were starting, I would build this first for confidence, then compare against a pose-based baseline on public data before writing a single production claim.

sign language · detr · object detection