Hammer repository upload-content succeeds on Foreman 3.1.1/4.3, content not in repo

Problem:
Using ‘hammer -u user repository upload-content --organization org --product ‘product’ --name ‘repo’ --path ./uploaded.rpm’ or the ‘–id’ method of the command or the UI upload, the command is successful, Tasks all show success, logs all show success - but content does not appear in target repository. This is a custom repo only with no upstream or sync.

Expected outcome:
Uploaded packages should appear in the listed repository.

Foreman and Proxy versions:
3.1.1, Katello 4.3

Foreman and Proxy plugin versions:

Distribution and version:
CentOS 7.9.2009

Other relevant data:
This worked without issue on previous 2.x releases. For testing, I have created new products, with new repos (no upstream URL of course, no sync) - same result each time. What is really strange is that every time I create a new, empty repo, one package does appear after creation - python36-dmlite. Yet, if I search for this package, it only comes up where it should, in EPEL7 repo. Production.log is below with the snippet pertaining to upload. There is a ‘delete’ message, but I assume that is for the upload process, not the file…or maybe I am wrong.

2022-03-25T11:29:54 [I|app|d6bc9c97] Completed 200 OK in 263ms (Views: 31.6ms | ActiveRecord: 34.0ms | Allocations: 49593)
2022-03-25T11:29:54 [I|app|7a1614ea] Started POST "/katello/api/repositories/236/content_uploads" for 1.2.3.4 at 2022-03-25 11:29:54 -0400
2022-03-25T11:29:54 [I|app|7a1614ea] Processing by Katello::Api::V2::ContentUploadsController#create as JSON
2022-03-25T11:29:54 [I|app|7a1614ea]   Parameters: {"size"=>15215948, "checksum"=>"6d48d467ce9e76156167afd247fe773c3171136a93fbca1079b7235f1e861735", "content_type"=>nil, "api_version"=>"v2", "repository_id"=>"236", "content_upload"=>{"size"=>15215948, "checksum"=>"6d48d467ce9e76156167afd247fe773c3171136a93fbca1079b7235f1e861735", "content_type"=>nil}}
2022-03-25T11:29:54 [I|app|7a1614ea] Authorized user testuser(Test User)
2022-03-25T11:29:54 [I|app|7a1614ea] Completed 200 OK in 357ms (Views: 0.3ms | ActiveRecord: 17.5ms | Allocations: 65237)
2022-03-25T11:29:55 [I|app|23d646e9] Started PUT "/katello/api/repositories/236/content_uploads/81e72f00-3370-48c9-b3d1-1d391c1130be" for 1.2.3.4 at 2022-03-25 11:29:55 -0400
2022-03-25T11:29:55 [I|app|23d646e9] Processing by Katello::Api::V2::ContentUploadsController#update as JSON
2022-03-25T11:29:55 [I|app|23d646e9]   Parameters: {"offset"=>"0", "content"=>"[FILTERED]", "size"=>"15215948", "api_version"=>"v2", "repository_id"=>"236", "id"=>"81e72f00-3370-48c9-b3d1-1d391c1130be"}
2022-03-25T11:29:55 [I|app|23d646e9] Authorized user testuser(Test User)
2022-03-25T11:29:55 [I|app|23d646e9] Completed 204 No Content in 300ms (ActiveRecord: 17.8ms | Allocations: 35929)
2022-03-25T11:29:55 [I|app|515657a4] Started PUT "/katello/api/repositories/236/content_uploads/81e72f00-3370-48c9-b3d1-1d391c1130be" for 1.2.3.4 at 2022-03-25 11:29:55 -0400
2022-03-25T11:29:55 [I|app|515657a4] Processing by Katello::Api::V2::ContentUploadsController#update as JSON
2022-03-25T11:29:55 [I|app|515657a4]   Parameters: {"offset"=>"2500000", "content"=>"[FILTERED]", "size"=>"15215948", "api_version"=>"v2", "repository_id"=>"236", "id"=>"81e72f00-3370-48c9-b3d1-1d391c1130be"}
2022-03-25T11:29:55 [I|app|515657a4] Authorized user testuser(Test User)
2022-03-25T11:29:55 [I|app|515657a4] Completed 204 No Content in 292ms (ActiveRecord: 15.2ms | Allocations: 35912)
2022-03-25T11:29:56 [I|app|e541fd94] Started PUT "/katello/api/repositories/236/content_uploads/81e72f00-3370-48c9-b3d1-1d391c1130be" for 1.2.3.4 at 2022-03-25 11:29:56 -0400
2022-03-25T11:29:56 [I|app|e541fd94] Processing by Katello::Api::V2::ContentUploadsController#update as JSON
2022-03-25T11:29:56 [I|app|e541fd94]   Parameters: {"offset"=>"5000000", "content"=>"[FILTERED]", "size"=>"15215948", "api_version"=>"v2", "repository_id"=>"236", "id"=>"81e72f00-3370-48c9-b3d1-1d391c1130be"}
2022-03-25T11:29:56 [I|app|e541fd94] Authorized user testuser(Test User)
2022-03-25T11:29:56 [I|app|e541fd94] Completed 204 No Content in 312ms (ActiveRecord: 21.2ms | Allocations: 36161)
2022-03-25T11:29:56 [I|app|dac8726b] Started PUT "/katello/api/repositories/236/content_uploads/81e72f00-3370-48c9-b3d1-1d391c1130be" for 1.2.3.4 at 2022-03-25 11:29:56 -0400
2022-03-25T11:29:56 [I|app|dac8726b] Processing by Katello::Api::V2::ContentUploadsController#update as JSON
2022-03-25T11:29:56 [I|app|dac8726b]   Parameters: {"offset"=>"7500000", "content"=>"[FILTERED]", "size"=>"15215948", "api_version"=>"v2", "repository_id"=>"236", "id"=>"81e72f00-3370-48c9-b3d1-1d391c1130be"}
2022-03-25T11:29:57 [I|app|dac8726b] Authorized user testuser(Test User)
2022-03-25T11:29:57 [I|app|dac8726b] Completed 204 No Content in 293ms (ActiveRecord: 14.9ms | Allocations: 35889)
2022-03-25T11:29:57 [I|app|49782b37] Started PUT "/katello/api/repositories/236/content_uploads/81e72f00-3370-48c9-b3d1-1d391c1130be" for 1.2.3.4 at 2022-03-25 11:29:57 -0400
2022-03-25T11:29:57 [I|app|49782b37] Processing by Katello::Api::V2::ContentUploadsController#update as JSON
2022-03-25T11:29:57 [I|app|49782b37]   Parameters: {"offset"=>"10000000", "content"=>"[FILTERED]", "size"=>"15215948", "api_version"=>"v2", "repository_id"=>"236", "id"=>"81e72f00-3370-48c9-b3d1-1d391c1130be"}
2022-03-25T11:29:57 [I|app|49782b37] Authorized user testuser(Test User)
2022-03-25T11:29:57 [I|app|49782b37] Completed 204 No Content in 295ms (ActiveRecord: 16.3ms | Allocations: 35952)
2022-03-25T11:29:58 [I|app|cc37d534] Started PUT "/katello/api/repositories/236/content_uploads/81e72f00-3370-48c9-b3d1-1d391c1130be" for 1.2.3.4 at 2022-03-25 11:29:58 -0400
2022-03-25T11:29:58 [I|app|cc37d534] Processing by Katello::Api::V2::ContentUploadsController#update as JSON
2022-03-25T11:29:58 [I|app|cc37d534]   Parameters: {"offset"=>"12500000", "content"=>"[FILTERED]", "size"=>"15215948", "api_version"=>"v2", "repository_id"=>"236", "id"=>"81e72f00-3370-48c9-b3d1-1d391c1130be"}
2022-03-25T11:29:58 [I|app|cc37d534] Authorized user testuser(Test User)
2022-03-25T11:29:58 [I|app|cc37d534] Completed 204 No Content in 308ms (ActiveRecord: 20.7ms | Allocations: 36076)
2022-03-25T11:29:58 [I|app|04069122] Started PUT "/katello/api/repositories/236/content_uploads/81e72f00-3370-48c9-b3d1-1d391c1130be" for 1.2.3.4 at 2022-03-25 11:29:58 -0400
2022-03-25T11:29:58 [I|app|04069122] Processing by Katello::Api::V2::ContentUploadsController#update as JSON
2022-03-25T11:29:58 [I|app|04069122]   Parameters: {"offset"=>"15000000", "content"=>"[FILTERED]", "size"=>"15215948", "api_version"=>"v2", "repository_id"=>"236", "id"=>"81e72f00-3370-48c9-b3d1-1d391c1130be"}
2022-03-25T11:29:58 [I|app|04069122] Authorized user testuser(Test User)
2022-03-25T11:29:58 [I|app|04069122] Completed 204 No Content in 271ms (ActiveRecord: 16.7ms | Allocations: 34507)
2022-03-25T11:29:58 [I|app|46636ba5] Started PUT "/katello/api/repositories/236/import_uploads" for 1.2.3.4 at 2022-03-25 11:29:58 -0400
2022-03-25T11:29:58 [I|app|46636ba5] Processing by Katello::Api::V2::RepositoriesController#import_uploads as JSON
2022-03-25T11:29:58 [I|app|46636ba5]   Parameters: {"uploads"=>[{"id"=>"81e72f00-3370-48c9-b3d1-1d391c1130be", "content_unit_id"=>nil, "name"=>"sample1.noarch.rpm", "size"=>15215948, "checksum"=>"6d48d467ce9e76156167afd247fe773c3171136a93fbca1079b7235f1e861735"}], "publish_repository"=>true, "sync_capsule"=>true, "api_version"=>"v2", "id"=>"236", "repository"=>{}}
2022-03-25T11:29:58 [I|app|46636ba5] Authorized user testuser(Test User)
2022-03-25T11:29:58 [I|bac|46636ba5] Task {label: , id: bee5baec-bb91-43b2-b518-529a9c560970, execution_plan_id: d71627a0-4381-4019-850a-dad5250d970a} state changed: pending
2022-03-25T11:29:58 [I|bac|46636ba5] Task {label: Actions::Katello::Repository::ImportUpload, id: bee5baec-bb91-43b2-b518-529a9c560970, execution_plan_id: d71627a0-4381-4019-850a-dad5250d970a} state changed: planning
2022-03-25T11:29:59 [I|bac|46636ba5] Task {label: Actions::Katello::Repository::ImportUpload, id: bee5baec-bb91-43b2-b518-529a9c560970, execution_plan_id: d71627a0-4381-4019-850a-dad5250d970a} state changed: planned
2022-03-25T11:29:59 [I|bac|46636ba5] Task {label: Actions::Katello::Repository::ImportUpload, id: bee5baec-bb91-43b2-b518-529a9c560970, execution_plan_id: d71627a0-4381-4019-850a-dad5250d970a} state changed: running
2022-03-25T11:30:01 [I|aud|46636ba5] Katello::Repository (236) update event on publication_href /pulp/api/v3/publications/rpm/rpm/5bd6daff-900c-44d8-832c-d94fab117714/, /pulp/api/v3/publications/rpm/rpm/faf23363-70bd-4a1a-b069-d9620544c45c/
2022-03-25T11:30:02 [I|bac|46636ba5] Task {label: , id: b1a5cec7-d7de-46c5-b6c9-2c3c35870760, execution_plan_id: 69ac8304-9da7-45d8-baeb-28d6b7bdd0b0} state changed: pending
2022-03-25T11:30:02 [I|bac|46636ba5] Task {label: Actions::Katello::Repository::CapsuleSync, id: b1a5cec7-d7de-46c5-b6c9-2c3c35870760, execution_plan_id: 69ac8304-9da7-45d8-baeb-28d6b7bdd0b0} state changed: planning
2022-03-25T11:30:02 [I|bac|46636ba5] Task {label: Actions::Katello::Repository::CapsuleSync, id: b1a5cec7-d7de-46c5-b6c9-2c3c35870760, execution_plan_id: 69ac8304-9da7-45d8-baeb-28d6b7bdd0b0} state changed: planned
2022-03-25T11:30:02 [I|bac|46636ba5] Task {label: Actions::Katello::Repository::CapsuleSync, id: b1a5cec7-d7de-46c5-b6c9-2c3c35870760, execution_plan_id: 69ac8304-9da7-45d8-baeb-28d6b7bdd0b0} state changed: running
2022-03-25T11:30:02 [I|bac|46636ba5] Task {label: Actions::Katello::Repository::CapsuleSync, id: b1a5cec7-d7de-46c5-b6c9-2c3c35870760, execution_plan_id: 69ac8304-9da7-45d8-baeb-28d6b7bdd0b0} state changed: stopped  result: success
2022-03-25T11:30:02 [I|bac|46636ba5] Task {label: Actions::Katello::Repository::ImportUpload, id: bee5baec-bb91-43b2-b518-529a9c560970, execution_plan_id: d71627a0-4381-4019-850a-dad5250d970a} state changed: stopped  result: success
2022-03-25T11:30:06 [I|app|] Rails cache backend: File
2022-03-25T11:30:08 [W|app|] Setting remote_execution_workers_pool_size has no definition, please define it before using
2022-03-25T11:30:08 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2022-03-25T11:30:08 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2022-03-25T11:30:08 [W|app|] Setting host_tasks_workers_pool_size has no definition, please define it before using
2022-03-25T11:30:09 [W|app|] Creating scope :exportable. Overwriting existing method Katello::Repository.exportable.
2022-03-25T11:30:10 [W|app|] Scoped order is ignored, it's forced to be batch order.
2022-03-25T11:30:26 [I|app|46636ba5]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/views/katello/api/v2/common/async.json.rabl within katello/api/v2/layouts/resource
2022-03-25T11:30:26 [I|app|46636ba5]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/app/views/katello/api/v2/common/async.json.rabl within katello/api/v2/layouts/resource (Duration: 8.1ms | Allocations: 5497)
2022-03-25T11:30:26 [I|app|46636ba5] Completed 202 Accepted in 27452ms (Views: 10.1ms | ActiveRecord: 36.6ms | Allocations: 173854)
2022-03-25T11:30:26 [I|app|d1300146] Started DELETE "/katello/api/repositories/236/content_uploads/81e72f00-3370-48c9-b3d1-1d391c1130be" for 1.2.3.4 at 2022-03-25 11:30:26 -0400
2022-03-25T11:30:26 [I|app|d1300146] Processing by Katello::Api::V2::ContentUploadsController#destroy as JSON
2022-03-25T11:30:26 [I|app|d1300146]   Parameters: {"api_version"=>"v2", "repository_id"=>"236", "id"=>"81e72f00-3370-48c9-b3d1-1d391c1130be", "content_upload"=>{}}
2022-03-25T11:30:26 [I|app|d1300146] Authorized user testuser(Test User)

You might be hitting this issue: Bug #34202: upload-content results in wrong RPM being added to product - Katello - Foreman which was fixed in 4.3.1.

1 Like

That’s exactly it! What a crazy bug. I don’t know why that bug didn’t pop up in my searching, obviously my Google-fu just didn’t cut it. Thanks for pointing me to this.