Object detection with focal loss, seven years later
The 2019 version of this post trained Fizyr’s Keras RetinaNet on a set of harbour photos, and spent its opening explaining focal loss. The argument goes: a one-stage detector proposes an enormous number of candidate boxes, nearly all of them empty water and sky, and if you treat every one of those as an equally interesting training example then the easy negatives bury the signal. Focal loss turns down the volume on the examples the model already gets right.
That repository now opens with a line saying it is deprecated in favour of torchvision. Last commit, March 2023. HyperLabel, which drew the boxes, does not resolve.
So I retrained on the same 750 frames with something current, expecting to write a short note saying the tooling had moved on. The numbers turned out to be more interesting than that.
The data is still up, in both formats. The Pascal VOC version is what the original post used, 750 images with an XML annotation each. The YOLO-format version carries the same boxes as plain text, and is the one everything below actually trains from. I checked the two against each other while writing this and they agree on all 750 frames, so it genuinely does not matter which you take.
The loss outlived everything around it
I used RT-DETR, a transformer detector, which is the accuracy-first end of the trade-off. The YOLO post covers the fast end, and both models saw identical data, an identical 80/20 split and 60 epochs on the same T4.
Here is the part I did not see coming. RT-DETR still uses focal loss. RetinaNet’s anchors are gone, its backbone is gone, non-maximum suppression is gone, the whole architecture has been replaced by an encoder-decoder that does set prediction. The loss function stayed. You can check rather than believe me:
1
2
3
4
from ultralytics.models.utils.loss import RTDETRDetectionLoss
c = RTDETRDetectionLoss(nc=5)
print(type(c.fl).__name__, c.fl.gamma, c.fl.alpha) # FocalLoss 1.5 tensor(0.2500)
print(c.vfl) # None
Gamma 1.5, alpha 0.25, against the 2017 paper’s gamma 2.0 and the same alpha. Seven years of architecture churn and it came through with one hyperparameter adjusted.
Training it is the same three lines as the YOLO post, with the batch halved because RT-DETR is about 33 million parameters against yolo26n’s 2.6 million and 16 will not fit on a T4:
1
2
3
4
5
from ultralytics import RTDETR
model = RTDETR("rtdetr-l.pt")
model.train(data="boats/data.yaml", epochs=60, imgsz=640,
device=0, batch=8, patience=20)
Fifty-four minutes, against thirteen for the small YOLO.
Both mAP curves flatten somewhere around epoch 45. The validation losses are still drifting down at 60 and none of them have turned back up, so nothing was overfitting and a longer run might have found a little more. Sixty was a number I picked to match the YOLO run, not one I tuned.
What it bought
| YOLO26n | RT-DETR-l | |
|---|---|---|
| mAP50 | 0.778 | 0.931 |
| mAP50-95 | 0.444 | 0.530 |
| precision | 0.617 | 0.879 |
| recall | 0.796 | 0.916 |
| training | 773 s | 3246 s |
| inference | 4.65 ms | 40.63 ms |
Better everywhere, at 4.2 times the training and 8.7 times the inference. That is the trade-off the old post described in the abstract, now with numbers attached to it.
The averages hide where the money actually went, though:
| Class | Boxes | YOLO26n | RT-DETR-l |
|---|---|---|---|
| Cruise | 1518 | 0.983 | 0.993 |
| Sailboat | 364 | 0.929 | 0.968 |
| Tug | 186 | 0.689 | 0.874 |
| Unknown | 83 | 0.792 | 0.824 |
| Ferry | 38 | 0.498 | 0.995 |
Cruise ships gained a hundredth of a point, because there was nothing left to win. Ferries went from 0.498 to 0.995 on thirty-eight training examples, which is to say the worst class in the set became the best one.
The same thing happens inside a single frame. Both models ran on a validation image whose labels list four objects. YOLO:
Three cruise ships. RT-DETR, same frame:
Four. The extra one is a sailboat maybe 44 pixels across, jammed against the left edge, at 0.82.
Why I am not going to tell you focal loss did this
It would round off beautifully. The post about focal loss discovers that the model with focal loss rescues the rare class. I do not think it holds.
Focal loss is aimed at the imbalance between foreground and background, which is a fact about how a detector proposes boxes. It does not know that ferries are rare, and it has no mechanism for knowing. What it suppresses is easy examples, and the easy examples here are empty water.
Then there is the more basic problem, which is that I changed everything at once. RT-DETR has thirteen times the parameters, a completely different architecture, set prediction with Hungarian matching, and no NMS step. Any of those could be doing the work. Comparing two entire models and then crediting one component is the kind of reasoning I would pick apart if somebody showed it to me, and I am not going to publish it because it makes a tidier ending.
What survives is narrower and still worth having: the extra capacity went almost entirely into the classes that were starved, and no aggregate number would have told you that.
Picking one
Anything with a latency budget, a camera feed, a device, and 4.65 ms against 40.63 ms decides it for you. Go back to the YOLO post.
Offline work where the frames are already sitting on disk and nobody is waiting, take the slow one. On this dataset that is the difference between a ferry detector and a thing that says “ferry” half the time.
And look at the per-class table either way. Both of these models will hand you one number, and on this data that number had a class scoring 0.498 hiding inside it.
Discussion
Powered by Disqus