Since November 2020, Docker Hub (https://registry-1.docker.io) has put into effect request rate limiting, primarily for Anonymous and Free user accounts. Anonymous and Free users are limited to 100 and 200 container image pull requests per six hours respectively. Docker Pro and Docker Team accounts allow for 5000 pulls in a 24 hour period. For further details on Docker Hub's rating limiting, see https://www.docker.com/increase-rate-limits/
As a result of this rate limiting, users of Red Hat's Quay.io may also be affected if the Quay image uses a base image that resides in Docker Hub. See https://access.redhat.com/articles/5531191 for further details.
What is the Impact to Nexus Repository users?
If you are proxying to Docker Hub especially with an Anonymous or Free Docker Hub account, then you may experience pull request failures with a "manifest unknown" error response e.g.
$ docker pull nexus-test.local/alpine/git:v2.34.2
Error response from daemon: manifest for nexus-test.local/alpine/git:v2.34.2 not found: manifest unknown: manifest unknown
In the Nexus Repo <data_dir>/log/nexus.log you may see WARN logging similar to the following:
DD/MM/YYYY:HH:MM:SS +0000 WARN [qtp1658624585-654545] admin org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: GET /v2/library/alpine/git/manifests/sha256:59ada2270cda98dd960d921297578f8c94cd66026110b4c6494d8fc97c966c28: 429 - org.sonatype.nexus.repository.docker.internal.V2Exception: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
And in the Nexus Repo <data_dir>/log/outbound-request.log file you may see a HTTP 429 response for the corresponding request to Docker Hub e.g.
If this observed, then you are hitting the request rate limiting.
How to handle rate limit failures?
Firstly, if you are an enterprise that is a regular user of Docker and pull images from Docker Hub, then you should strongly consider purchasing a Docker Hub Pro or Teams account which will allow you to perform 5000 pulls in a 24 hour period from Docker Hub.
You should then configure your Docker Hub proxy on the Nexus Repo side with this Pro/Teams user as follows:
- From the Nexus Repo UI --> Administration --> Repository --> Repositories page, open your Docker Hub proxy config page, then scroll down to the "HTTP" section.
- Enter your Docker Hub username and password and then click the [ Save ] button.
In addition to using a Pro Docker Hub account, you should also consider implementing the following settings in your Docker Hub proxy on the Nexus Repo side to try and minimise the number of outgoing requests being made to Docker Hub:
- Set an appropriate "Maximum metadata age" (e.g. 1440).
- Set an appropriate "Maximum component age" (e.g. -1). The example value "-1" prevents the proxy repository from checking for changes to already existing images.
- For Docker group repositories, move the Docker Hub proxy repository to the end of the Group Members list. This may not be appropriate, if you have a large members list and you rely on Docker Hub for most of your images. In that case, you should at least place the Docker Hub proxy below any hosted repos.