15

I'm a writing a docker registry API wrapper to pull images from one private registry and push them to another.

Based on the documentation first I need to pull the manifest and the layers for an image:tag. Following Puling An Image I've successfully downloaded all the layers for a particular image:tag and the manifest.

Following Pushing An Image I've followed the steps:

  1. POST /v2/<name>/blobs/uploads/ (to get the UUID i.e. Location header)
  2. HEAD /v2/<name>/blobs/<digest> (check to see if it already exists in the registry)
  3. PUT /v2/<name>/blobs/uploads/<uuid>?digest=<digest> (Monolithic Upload )

What's not clear to me are the following:

  1. Is the UUID unique to each individual layer I push or is that reused for all layers (e.g. Do I need to run a new POST for each layer an get a new UUID before I attempt to upload it?).
  2. The Completed Upload section indicates

For an upload to be considered complete, the client must submit a PUT request on the upload endpoint with a digest parameter

However, as mentioned I'm using the Monolithic Upload which uses a PUT and would be the same request as what shows in the Completed Upload section. So by doing a monolithic upload am I also completing the upload at the same time?

Problem

  1. When I go through all the steps above I receive the BLOB_UNKNOWN error when uploading a digest, e.g.

    { "errors:" [{ "code": "BLOB_UNKNOWN", "message": "blob unknown to registry", "detail": { "digest": } }, ... ] }

According to the docs this error is produced when pushing a manifest and one of the layers in the manifest are unknown:

If one or more layers are unknown to the registry, BLOB_UNKNOWN errors are returned. The detail field of the error response will have a digest field identifying the missing blob. An error is returned for each unknown blob. The response format is as follows:

What confuses me about this is

  1. I'm pushing a digest (aka a layer) not the manifest so why is this error returning?
  2. I would expect the blob to be unknown because I'm pushing a new image into the registry

For now I'm going to use the docker client, but I haven't found any wrapper examples on-line to see how this is pulled off. Presumably I'm missing some logic or misunderstanding the docs, but I'm not sure where I'm going wrong?

Kuya
  • 7,280
  • 4
  • 19
  • 31
mdo123
  • 1,757
  • 3
  • 16
  • 34
  • A bit late to the party and by no means a solution: Have you considered using the docker client (as you said) but going through a proxy which would allow you to inspect the traffic? Because the client talking to the v2 registry is done as plain text over http the traffic could be read and understood by a human – Justin Tamblyn May 30 '19 at 20:20
  • Did you manage to succeed? I got the same issue – Ayrton Werck May 29 '20 at 18:47

1 Answers1

4

Wow, it's nice to know I'm not the only one lost in the void with the V2 API ...

I'm implementing a similar library and ran across the same issue. From my understanding of the documentation there are two Monolithic uploads: The single exchange POST variant (mentioned at the bottom of the docs), and the two exchange POST + PUT variant (mentioned at the top of the docs).

I wasn't able to get the POST-only method working. In my case, I was using it to upload the image manifest after the layer blobs, and before the registry manifest. While the POST appears successful and returns 202, the debug logging on the registry shows that it's never replicated from the staging location into the data store (as happens after chunked uploads). The subsequent attempt to upload the manifest then fails with 400, and debug logging "blob unknown to registry".

I was, however, able to work around this issue by using the POST+PUT method.

The key sections in the docs for me were:

Though the URI format (/v2//blobs/uploads/) for the Location header is specified, clients should treat it as an opaque url and should never try to assemble it.

and

A monolithic upload is simply a chunked upload with a single chunk ...

Following those two instructions I created a new Location header (and UUID) using POST, appended the digest value, and completed the upload by PUTting the blob to the modified location.

Side note: Looking at the registry debug logs, the docker CLI checks the existence of the blobs before starting a new upload (and after completing the uploads too -- assuming as a double check on the status code).

Update: Found myself working on this again, and figured I'd update you on what I found ...

The registry only supports handling the response body during PATCH and PUT operations; the copyFullPayload helper is not invoked for POST. Additionally, all uploads appear to be treated as monolithic uploads (in the sense that they stream the blob from a single request body) as handling of the Content-Range header does not appear to be implemented.

Side Note: I conducted this analysis under the scope of increasing test coverage of the V2 API during an overhaul; here is a working example of the POST+PUT method. In general I've found the official documentation to be out-of-sync with the current implementation with regards to headers and status codes. I have tested this against a local V2 registry and DockerHub, but not against other registries such as DTR, quay, or MCR.

Richard Davis
  • 41
  • 4