Error when adding Puppet Forge as repository

katello
puppet

#1

I’m following the documentation for importing modules from Puppet Forge as a repository and running into some trouble.

When I sync the repository, it ends with a warning result:
Total module count: .
Failed to download 5 modules.

PLP0034: The distributor e3ee7706-7ed9-4c45-aa3a-ba4679e969a2 indicated a failed response when publishing repository e3ee7706-7ed9-4c45-aa3a-ba4679e969a2.

I can see that it has created many thousands of files in /var/lib/pulp/published/puppet/https/repos/e3ee7706-7ed9-4c45-aa3a-ba4679e969a2, which look to be the modules, however, no Puppet Modules ever appear in the new repository.

This is a new Foreman install that I setup last week (Foreman 1.20.1, Katello 3.10.0). We’re trying to move from Spacewalk and everything is great so far. I’m just trying to replicate the “Configuration” functionality of Spacewalk now and from the documentation, it sounds like this is how I can accomplish it. Please forgive that I’m new to Foreman and Puppet.

I found the following in the production.log file:

Response: 200: {"exception": null, "task_type": "pulp.server.managers.repo.sync.sync", "_href": "/pulp/api/v2/tasks/2c575f17-22d1-4677-9551-40b4920a92c0/", "task_id": "2c575f17-22d1-4677-9551-40b4920a92c0", "tags": ["pulp:repository:e3ee7706-7ed9-4c45-aa3a-ba4679e969a2", "pulp:action:sync"], "finish_time": "2019-01-29T23:10:29Z", "_ns": "task_status", "start_time": "2019-01-29T23:09:15Z", "traceback": null, "spawned_tasks": [{"_href": "/pulp/api/v2/tasks/8fde719a-8010-4736-aacf-0bfb30cb96a3/", "task_id": "8fde719a-8010-4736-aacf-0bfb30cb96a3"}, {"_href": "/pulp/api/v2/tasks/1fea6a10-2857-4f69-bb75-03e3b61592d6/", "task_id": "1fea6a10-2857-4f69-bb75-03e3b61592d6"}], "progress_report": {"puppet_importer": {"modules": {"error_message": null, "execution_time": 66, "total_count": 40, "traceback": null, "individual_errors": [{"exception": "Could not find metadata file inside Puppet module", "traceback": [["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", 266, "_do_import_modules", "self._add_new_module(downloader, module)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", 301, "_add_new_module", "self.repo.working_dir)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/metadata.py", 52, "extract_metadata", "metadata = _extract_json(filename, temp_dir)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/metadata.py", 118, "_extract_json", "raise MissingMetadataFile()"]], "module": "druid-0.2.1", "author": "jowy"}, {"exception": "Could not find metadata file inside Puppet module", "traceback": [["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", 266, "_do_import_modules", "self._add_new_module(downloader, module)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", 301, "_add_new_module", "self.repo.working_dir)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/metadata.py", 52, "extract_metadata", "metadata = _extract_json(filename, temp_dir)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/metadata.py", 118, "_extract_json", "raise MissingMetadataFile()"]], "module": "test-1.0.0", "author": "felixcarmona"}, {"exception": "ValidationError (Module:189e1e4b-87a7-420f-89a6-bde6ffcdcf18) (StringField only accepts string values: ['checksum'])", "traceback": [["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", 266, "_do_import_modules", "self._add_new_module(downloader, module)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", 310, "_add_new_module", "module.save_and_import_content(downloaded_filename)"], ["/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", 921, "save_and_import_content", "self.save()"], ["/usr/lib/python2.7/site-packages/mongoengine/document.py", 304, "save", "self.validate(clean=clean)"], ["/usr/lib/python2.7/site-packages/mongoengine/base/document.py", 413, "validate", "raise ValidationError(message, errors=errors)"]], "module": "glusterfs-0.0.5", "author": "maxadamo"}, {"exception": "Could not find metadata file inside Puppet module", "traceback": [["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", 266, "_do_import_modules", "self._add_new_module(downloader, module)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", 301, "_add_new_module", "self.repo.working_dir)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/metadata.py", 52, "extract_metadata", "metadata = _extract_json(filename, temp_dir)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/metadata.py", 118, "_extract_json", "raise MissingMetadataFile()"]], "module": "strings-999.999.999", "author": "puppetlabs"}, {"exception": "Could not extract Puppet module.", "traceback": [["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", 266, "_do_import_modules", "self._add_new_module(downloader, module)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", 301, "_add_new_module", "self.repo.working_dir)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/metadata.py", 52, "extract_metadata", "metadata = _extract_json(filename, temp_dir)"], ["/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/metadata.py", 98, "_extract_json", "tgz.extractall(path=extraction_dir)"], ["/usr/lib64/python2.7/tarfile.py", 2047, "extractall", "self.extract(tarinfo, path)"], ["/usr/lib64/python2.7/tarfile.py", 2084, "extract", "self._extract_member(tarinfo, os.path.join(path, tarinfo.name))"], ["/usr/lib64/python2.7/posixpath.py", 80, "join", "path += '/' + b"]], "module": "demo-0.1.1", "author": "tobiashochguertel"}], "state": "success", "error_count": 5, "error": "None", "finished_count": 35}, "metadata": {"query_finished_count": 1, "traceback": null, "execution_time": 5, "query_total_count": 1, "error_message": null, "state": "success", "error": "None", "current_query": "https://forge.puppetlabs.com/modules.json"}}}, "queue": "reserved_resource_worker-1@foreman.dq2", "state": "finished", "worker_name": "reserved_resource_worker-1@foreman", "result": {"result": "success", "importer_id": "puppet_importer", "exception": null, "repo_id": "e3ee7706-7ed9-4c45-aa3a-ba4679e969a2", "traceback": null, "started": "2019-01-29T23:09:15Z", "_ns": "repo_sync_results", "completed": "2019-01-29T23:10:29Z", "importer_type_id": "puppet_importer", "error_message": null, "summary": {"total_execution_time": 71}, "added_count": 35, "removed_count": -29, "updated_count": 0, "id": "5c50dd65ec90f51bb1a3e10a", "details": {"finished_count": 35, "total_count": 40, "error_count": 5}}, "error": null, "_id": {"$oid": "5c50dd1be78f26a227d185ac"}, "id": "5c50dd1be78f26a227d185ac"}
2019-01-29T18:10:58 [I|kat|] GET: https://foreman.mydomain.local/pulp/api/v2/tasks/8fde719a-8010-4736-aacf-0bfb30cb96a3/: {"content_type"=>"application/json", "accept"=>"application/json"}

Response: 200: {"exception": null, "task_type": "pulp.server.managers.repo.publish.publish", "_href": "/pulp/api/v2/tasks/8fde719a-8010-4736-aacf-0bfb30cb96a3/", "task_id": "8fde719a-8010-4736-aacf-0bfb30cb96a3", "tags": ["pulp:repository:e3ee7706-7ed9-4c45-aa3a-ba4679e969a2", "pulp:action:publish"], "finish_time": null, "_ns": "task_status", "start_time": "2019-01-29T23:10:29Z", "traceback": null, "spawned_tasks": [], "progress_report": {"e3ee7706-7ed9-4c45-aa3a-ba4679e969a2_puppet": {"modules": {"error_message": null, "execution_time": 18, "total_count": 5984, "traceback": null, "individual_errors": null, "state": "success", "error_count": 0, "error": "None", "finished_count": 5984}, "publishing": {"http": "not-started", "https": "not-started"}, "metadata": {"execution_time": null, "state": "running", "error_message": null, "error": "None", "traceback": null}}}, "queue": "reserved_resource_worker-1@foreman.dq2", "state": "running", "worker_name": "reserved_resource_worker-1@foreman", "result": null, "error": null, "_id": {"$oid": "5c50dd65e78f26a227d1894b"}, "id": "5c50dd65e78f26a227d1894b"}
2019-01-29T18:10:58 [I|kat|] GET: https://foreman.mydomain.local/pulp/api/v2/tasks/1fea6a10-2857-4f69-bb75-03e3b61592d6/: {"content_type"=>"application/json", "accept"=>"application/json"}
Response: 200: {"exception": null, "task_type": "pulp.server.managers.repo.publish.publish", "_href": "/pulp/api/v2/tasks/1fea6a10-2857-4f69-bb75-03e3b61592d6/", "task_id": "1fea6a10-2857-4f69-bb75-03e3b61592d6", "tags": ["pulp:repository:e3ee7706-7ed9-4c45-aa3a-ba4679e969a2", "pulp:action:publish"], "finish_time": null, "_ns": "task_status", "start_time": null, "traceback": null, "spawned_tasks": [], "progress_report": {}, "queue": "reserved_resource_worker-1@foreman.dq2", "state": "waiting", "worker_name": "reserved_resource_worker-1@foreman", "result": null, "error": null, "_id": {"$oid": "5c50dd65e78f26a227d1894e"}, "id": "5c50dd65e78f26a227d1894e"}

#2

Decided to give up on syncing the whole repository and just upload our files manually. Works great. Sorry for the trouble.


#3

@flakshak I’m going to take a look and see if I can come up with an explanation for what you’re seeing. Stay tuned!


#4

@flakshack unfortunately I couldn’t reproduce this problem on my Katello 3.10 box. Out of curiosity did you try syncing that repo again once it failed the first time?