Nginx, docker push and 404 errors

Recently after trying some more elaborate Docker image builds, I discovered I had an issue with my Docker Registry setup (as described in Deploying a secure public Docker registry with Nginx and Traefik), which was manifesting itself as random 404 page not found errors:

$ docker --debug  push my.private.registry/cant-think-of-a-container-name:latest

The push refers to a repository [my.private.registry/cant-think-of-a-container-name]
5ae877b9fd02: Pushing  3.164MB
34be67ece980: Layer already exists
9411f02cc27c: Pushing  1.092MB/296.6MB
5df38987acef: Pushing    290MB/290MB
683c7228321d: Layer already exists
e6562eb04a92: Pushing  122.9MB/122.9MB
596280599f68: Pushing  44.64MB/44.64MB
5d6cbe0dbcf9: Pushing  123.4MB/123.4MB
Error: Status 404 trying to push repository cant-think-of-a-container-name: "404 page not found\n"

If you check the docker push logs (in debug mode), the 404 page not found error actually relates to an attempt to access something in the /v1/... path on the registry - which is odd because the Docker registry protocol has been v2 for, like, ever now.

So whats going on?

Looking at the logs from Nginx, it becomes quite clear - Nginx itself is throwing a http 413 error because the PUT and POST bodies being sent by docker push are well in excess of the size allowed as "valid" request bodies by my Nginx setup.

docker push doesn't really understand the http 413 error, and tries to fall back to the v1 protocol in desperation - personally, I'd have called it a day and gone down the pub, but docker didn't, poor little thing.

So the 404 page not found error as shown by docker push is actually the last error in a chain, and that chain is caused by the first 413 error.

So how do you solve the http 413 error?

Quite simple, bump up the size Nginx allows for request bodies:

client_max_body_size 2000M;

This can go in the http, server or location configuration blocks, depending on how wide you want it applied.