Syncing some images from docker hub ends with MANIFEST_INVALID error

Support
1

So I have a docker hub product with some repositories that correspond to different images.
Most are working fine but for two of them I get a “MANIFEST_INVALID” error.
Sync Settings
Registry URL: https://registry-1.docker.io/

Upstream Repository Name: keycloak/keycloak
Include Tags: latest

Upstream Repository Name: timescale/timescaledb
Include Tags: latest-pg14, latest-pg15

From production.log:

2023-04-24T14:23:35 [I|app|b106cbec] Started POST "/katello/api/v2/repositories/bulk/sync?organization_id=1" for 11.11.11.11 at 2023-04-24 14:23:35 +0000
2023-04-24T14:23:35 [I|app|b106cbec] Processing by Katello::Api::V2::RepositoriesBulkActionsController#sync_repositories as HTML
2023-04-24T14:23:35 [I|app|b106cbec]   Parameters: {"ids"=>[223], "organization_id"=>"1", "api_version"=>"v2", "repositories_bulk_action"=>{"ids"=>[223]}}
2023-04-24T14:23:35 [I|bac|b106cbec] Task {label: , execution_plan_id: 50fa3fc7-0286-47a5-975b-6ee1ddd53242} state changed: pending
2023-04-24T14:23:35 [I|bac|b106cbec] Task {label: Actions::BulkAction, id: 4fb0979b-97df-4f75-8497-348357066b07, execution_plan_id: 50fa3fc7-0286-47a5-975b-6ee1ddd53242} state changed: planning
2023-04-24T14:23:36 [I|bac|b106cbec] Task {label: Actions::BulkAction, id: 4fb0979b-97df-4f75-8497-348357066b07, execution_plan_id: 50fa3fc7-0286-47a5-975b-6ee1ddd53242} state changed: planned
2023-04-24T14:23:36 [I|app|b106cbec]   Rendered /usr/share/gems/gems/katello-4.7.4/app/views/katello/api/v2/common/async.json.rabl within katello/api/v2/layouts/resource (Duration: 15.1ms | Allocations: 8281)
2023-04-24T14:23:36 [I|app|b106cbec]   Rendered layout /usr/share/gems/gems/katello-4.7.4/app/views/katello/api/v2/layouts/resource.json.erb (Duration: 19.3ms | Allocations: 10928)
2023-04-24T14:23:36 [I|app|b106cbec] Completed 202 Accepted in 110ms (Views: 17.7ms | ActiveRecord: 16.8ms | Allocations: 32388)
2023-04-24T14:23:36 [I|bac|b106cbec] Task {label: Actions::BulkAction, id: 4fb0979b-97df-4f75-8497-348357066b07, execution_plan_id: 50fa3fc7-0286-47a5-975b-6ee1ddd53242} state changed: running
2023-04-24T14:23:36 [I|bac|b106cbec] Task {label: , execution_plan_id: 37409233-bc96-49b7-89fe-dd083759786a} state changed: pending
2023-04-24T14:23:36 [I|bac|b106cbec] Task {label: Actions::Katello::Repository::Sync, id: f1bbac67-24c4-4c50-830a-31ce02f5b742, execution_plan_id: 37409233-bc96-49b7-89fe-dd083759786a} state changed: planning
2023-04-24T14:23:36 [I|bac|b106cbec] Task {label: Actions::Katello::Repository::Sync, id: f1bbac67-24c4-4c50-830a-31ce02f5b742, execution_plan_id: 37409233-bc96-49b7-89fe-dd083759786a} state changed: planned
2023-04-24T14:23:36 [I|bac|b106cbec] Task {label: Actions::Katello::Repository::Sync, id: f1bbac67-24c4-4c50-830a-31ce02f5b742, execution_plan_id: 37409233-bc96-49b7-89fe-dd083759786a} state changed: running
2023-04-24T14:23:39 [E|bac|b106cbec] {'errors': [{'code': ErrorDetail(string='MANIFEST_INVALID', code='parse_error'), 'message': ErrorDetail(string="layers.0.mediaType: 'application/vnd.in-toto+json' is not one of ['application/vnd.oci.image.layer.v1.tar', 'application/vnd.oci.image.layer.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+zstd', 'application/vnd.oci.image.layer.nondistributable.v1.tar', 'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip', 'application/vnd.oci.image.layer.nondistributable.v1.tar+zstd']", code='parse_error'), 'detail': {'digest': ErrorDetail(string='sha256:df934da693450954bce21f0c3fc93ca3481e4a6598efe0e0a55ffaa9ca750657', code='parse_error')}}]} (Katello::Errors::Pulp3Error)
 b106cbec | /usr/share/gems/gems/katello-4.7.4/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
 b106cbec | /usr/share/gems/gems/katello-4.7.4/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
 b106cbec | /usr/share/gems/gems/katello-4.7.4/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
 b106cbec | /usr/share/gems/gems/katello-4.7.4/app/lib/actions/pulp3/abstract_async_task.rb:160:in `poll_external_task'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/polling.rb:22:in `run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/cancellable.rb:14:in `run'
 b106cbec | /usr/share/gems/gems/katello-4.7.4/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 b106cbec | /usr/share/gems/gems/katello-4.7.4/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 b106cbec | /usr/share/gems/gems/katello-4.7.4/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 b106cbec | /usr/share/gems/gems/katello-4.7.4/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 b106cbec | /usr/share/gems/gems/katello-4.7.4/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 b106cbec | /usr/share/gems/gems/katello-4.7.4/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 b106cbec | /usr/share/gems/gems/katello-4.7.4/app/lib/actions/middleware/remote_action.rb:16:in `run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 b106cbec | /usr/share/gems/gems/activesupport-6.1.7.2/lib/active_support/execution_wrapper.rb:91:in `wrap'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:17:in `run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 b106cbec | /usr/share/gems/gems/foreman-tasks-7.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `catch'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `block in execute_run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:575:in `execute_run'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:94:in `execute'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 b106cbec | /usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 b106cbec | [ sidekiq ]
 b106cbec | [ concurrent-ruby ]
2023-04-24T14:23:41 [I|bac|b106cbec] Task {label: Actions::Katello::Repository::Sync, id: f1bbac67-24c4-4c50-830a-31ce02f5b742, execution_plan_id: 37409233-bc96-49b7-89fe-dd083759786a} state changed: stopped  result: warning
2023-04-24T14:23:41 [E|bac|b106cbec] A sub task failed (Dynflow::Action::WithSubPlans::SubtaskFailedException)
 b106cbec |
2023-04-24T14:23:41 [I|bac|b106cbec] Task {label: Actions::BulkAction, id: 4fb0979b-97df-4f75-8497-348357066b07, execution_plan_id: 50fa3fc7-0286-47a5-975b-6ee1ddd53242} state changed: stopped  result: warning

Can someone else please verify if this fails also on other Foreman installations?

ex for one that does work fine:
Registry URL: https://registry-1.docker.io/
Upstream Repository Name: grafana/grafana-oss
Include Tags: latest

2

On Foreman 3.5.2, Katello 4.7.4.

3

Just updated to Foreman 3.6.1, Katello 4,8,0 but same problem.

4

Out of curiosity I tried it myself now, and yep the same issue on Katello 4.8,
also was looking at the image meta for short, and I didn’t see the application/vnd.in-toto+json mediaType file up to now, wonder where it gets that from :thinking:

5

Oh! now I see it, Docker Hub doesn’t show it, but Quay does in the frontend! (the keycloak image)

image
image2379×225 28 KB

It’s the 4th entry here

But with skopeo it’s also visible on Docker Hub:

{
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:f3071e991039cb3b28c9bb449cf092b2c325433c7d01b21e3031c166c800408b",
      "size": 1056,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:d5bc6a0f9bc044dad87141af9d22294f0d2f17acb43635b07aaae1b64a13428d",
      "size": 1056,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:ea52aa9bc1c8df251cccc41b5c6f9fba91dc7d19f4b579f26d4c0348f6fda4ef",
      "size": 566,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:f3071e991039cb3b28c9bb449cf092b2c325433c7d01b21e3031c166c800408b",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:210209fa5570513654521367a27972ba4dd0bcc00a6ee9066a4c037796096a15",
      "size": 566,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:d5bc6a0f9bc044dad87141af9d22294f0d2f17acb43635b07aaae1b64a13428d",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}
6

A little bit more tinkering with skopeo, the whole faulting thing looks like this:

image
image1405×772 172 KB

So it’s just metadata from the build, and nothing else, would either need to be trusted/allowed in pulp or just thrown away.

7

And looks like this already got fixed in pulp_container last month, so it’s just a matter of the fix getting to Katello: Add support of in-toto mediaType · Issue #1227 · pulp/pulp_container · GitHub

1 Like
8

Nice find. Hopefully this get some attention from the Katello devs and get added soon.

9

@katello this needs updating of pulp_container to 2.14.4+ (we are currently at 2.14.3), what is the correct way to request such a bump?

1 Like
10

@evgeni we can just create a packaging PR to bump up that version, we don’t typically re-record VCRs for Pulp z-updates.

1 Like
11

Sounds good!

Can you take care of this? Or shall I?

1 Like
12
2 Likes
13

The PR has been merged and the package is now live on the CDN, so you should be able to upgrade and enjoy all your containers again :slight_smile:

2 Likes
14

Just upgraded to python39-pulp-container-2.14.5-1.el8.noarch and can indeed confirm this solved the issue!
Many thanks for the quick fix! :medal_sports:

1 Like