Sonatype Nexus Repository Manager is a great tool for storing/proxying artifacts.
It supports many types of artifacts (Maven, Docker, APT, etc.) as well as multiple types of repositories, hosted, proxy and group.
Recently, I experienced some storage problems with my hosted Docker repository. The blob store containing the hosted Docker repository, which contains my docker images, got really big and I couldn’t reduce its size.
Nexus provides you some tooling for maintenance of your repositories. Intuitively, cleanup policies seems to be the solution to the problem, but it wasn’t.
Most of my Docker images contain only latest and stable tags, one or two version tags exist which I don’t want to delete, so I couldn’t set any cleanup policy.
In fact, cleanup policy is supposed to be used to remove ephemeral or aging (and old) artifacts like snapshot JARs. You shouldn’t delete any of your versioned artifacts, unless you don’t have any other choice.
My problem was not the old artifacts. Although, I didn’t have many images stored inside repository, the disk usage was increasing and wouldn’t decrease.
Solution Link to heading
The solution included me reading the Nexus docs :D.
First thing to note, even if I set some cleanup policies, or delete all of my Docker images or any other artifacts manually, my disk usage wasn’t going to change, because Nexus would soft-delete them.
Nexus has Tasks feature that is used to run scheduled maintenance tasks, which I wasn’t aware of. For example, the cleanup tasks are controlled from a task called Cleanup Service.
Nexus docs say that all deletion operations are soft deletes, in order to hard delete them we need set Compact blob store task for each blob store.
However, this wasn’t the solution again, but I am close.
When I read further down the docs, I came to a section called Docker Cleanup Strategies. They made a section for Docker repository cleanup.
Docker images are made of layers, which are mapped to blobs in blob store in Nexus. When my CI pipeline create images, some of the layers change with every new image. When they got pushed to the Docker registry they create new blobs. My problem was that the old, orphaned blobs stay in the blob store and I don’t have any tasks to clean them. Even If there was only one image in my repository, every time a new image is pushed, orphaned blobs would be created and will stay in the repository.
This was the cause of my issue with the storage.
To prevent this situation, Nexus has Delete unused manifests and images task which can be scheduled to remove these orphaned/unused layers. This removal action is of course a soft-delete, in order to persist you need to run Compact blob store task after this task.
There is also Delete incomplete uploads task which soft-deletes incomplete uploads if any of them exist in your repository.
Conclusion Link to heading
After all tasks completed, blob disk usage was low and I was happy.
So keep in mind that if you are using Nexus Repository Manager, make sure that you setup maintenance tasks correctly according to your repository type. Read the docs.
Except the Cleanup Service task which doesn’t hard-delete anything, other three tasks are really essential but they are not to be setup by default. I think that Nexus Web UI should include a warning about the necessity of these tasks or a shortcut to add these tasks when creating repository or blob store.
I will later add these tasks to my automated Nexus installation. If I do I would leave a note here.