Pulp_deb with celery Cannot allocate memory and Out of memory

Support
1

Problem:

I can not Promote content view pulp_deb

Expected outcome:

Promote content view of .deb packages works with .rpm

Foreman and Proxy versions:

rpm -qa | grep foreman

foreman-1.19.0-1.el7.noarch
foreman-postgresql-1.19.0-1.el7.noarch
foreman-libvirt-1.19.0-1.el7.noarch
tfm-rubygem-foreman_ansible-2.2.9-1.fm1_19.el7.noarch
foreman-selinux-1.19.0-1.el7.noarch
tfm-rubygem-foreman_docker-4.1.0-1.fm1_18.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.1.3-6.el7.noarch
rubygem-foreman_maintain-0.2.4-1.el7.noarch
foreman-installer-1.19.0-1.el7.noarch
foreman-release-scl-7-1.el7.noarch
uat-l-srv-203.uat.domain.corp-foreman-proxy-1.0-4.noarch
tfm-rubygem-foreman_ansible_core-2.1.1-1.fm1_19.el7.noarch
foreman-compute-1.19.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.4-3.el7.noarch
foreman-installer-katello-3.8.0-1.el7.noarch
foreman-release-1.19.0-1.el7.noarch
uat-l-srv-203.uat.domain.corp-foreman-client-1.0-4.noarch
uat-l-srv-203.uat.domain.corp-foreman-proxy-client-1.0-4.noarch
tfm-rubygem-foreman_discovery-13.0.1-1.fm1_19.el7.noarch
tfm-rubygem-foreman-tasks-0.13.4-1.fm1_19.el7.noarch
tfm-rubygem-hammer_cli_foreman-0.14.0-1.el7.noarch
tfm-rubygem-foreman_remote_execution-1.5.6-1.fm1_19.el7.noarch
tfm-rubygem-foreman-tasks-core-0.2.5-1.fm1_18.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.12-2.fm1_17.el7.noarch
foreman-proxy-1.19.0-1.el7.noarch
foreman-cli-1.19.0-1.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.1.3-1.el7.noarch
foreman-debug-1.19.0-1.el7.noarch

rpm -qa | grep celery

python2-celery-4.0.2-5.el7.noarch

rpm -qa | grep katello

katello-repos-3.8.0-4.el7.noarch
katello-default-ca-1.0-1.noarch
katello-client-bootstrap-1.4.2-1.el7.noarch
tfm-rubygem-katello-3.8.0-1.el7.noarch
katello-common-3.8.0-5.el7.noarch
foreman-installer-katello-3.8.0-1.el7.noarch
tfm-rubygem-hammer_cli_katello-0.14.1-1.el7.noarch
pulp-katello-1.0.2-1.el7.noarch
katello-debug-3.8.0-5.el7.noarch
katello-certs-tools-2.4.0-1.el7.noarch
katello-installer-base-3.8.0-1.el7.noarch
katello-server-ca-1.0-1.noarch
katello-selinux-3.0.3-1.el7.noarch
katello-service-3.8.0-5.el7.noarch
katello-3.8.0-5.el7.noarch

Other relevant data:

I believe the problem may be in the celery, sometimes I received ā€œOut of memory: Kill process 7013 (celery) score 449 or sacrifice childā€

My server has 4core 16GB

Nov 12 20:50:39 uat-l-srv-203 pulp: debpkgr.debpkg:WARNING: [d57893c7] While processing /var/lib/pulp/content/units/deb/43/0e7a8312ed2cfa8ec14c1ea83d1b17ffe804100f4051e96803aeba3b14d1de/tk-doc_8.6.0+9_all.deb: 'md5sums' file not found, can't list MD5 sums
Nov 12 20:50:53 uat-l-srv-203 pulp: debpkgr.debpkg:WARNING: [d57893c7] While processing /var/lib/pulp/content/units/deb/d5/6002327a063468b954f23a9ef2bbc11e3d7aee09417487946a6218cb7d938c/hannah-data_1.0-3build1_all.deb: [Errno 12] Cannot allocate memory
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528) Exception caught from plugin during publish for repo [1-CV_BIONIC-uat-f30e6543-30aa-486d-af28-6d0d32c00264]
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528) Traceback (most recent call last):
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1258, in _do_publish
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 737, in wrap_f
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     return f(*args, **kwargs)
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py", line 80, in publish_repo
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     return self._publisher.process_lifecycle()
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 572, in process_lifecycle
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     super(PluginStep, self).process_lifecycle()
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 163, in process_lifecycle
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     step.process()
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 256, in process
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     self._process_block()
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 303, in _process_block
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     self.process_main()
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py", line 329, in process_main
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     with_symlinks=True)
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py", line 478, in create
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     component=component, architecture=architecture)
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py", line 449, in add_packages
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     pkg = debpkg.DebPkg.from_file(filename, Size=sz)
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debpkgr/debpkg.py", line 416, in from_file
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     control = debpkg.control.debcontrol().copy()
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debian/debfile.py", line 222, in debcontrol
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     return Deb822(self.get_content(CONTROL_FILE))
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debian/debfile.py", line 171, in get_content
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     f = self.get_file(fname, encoding=encoding, errors=errors)
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debian/debfile.py", line 145, in get_file
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     fobj = self.tgz().extractfile('./' + fname)
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debian/debfile.py", line 92, in tgz
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528)     raise DebError("%%s" %% e)
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.controllers.repository:ERROR: [d57893c7] (3040-42528) DebError: [Errno 12] Cannot allocate memory
Nov 12 20:50:53 uat-l-srv-203 pulp: pulp.server.async.tasks:INFO: [d57893c7] Task failed : [d57893c7-519c-49e0-ad76-aac2c901465c]
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528) Task pulp.server.managers.repo.publish.publish[d57893c7-519c-49e0-ad76-aac2c901465c] raised unexpected: DebError('[Errno 12] Cannot allocate memory',)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528) Traceback (most recent call last):
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     R = retval = fun(*args, **kwargs)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     return super(Task, self).__call__(*args, **kwargs)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     return super(PulpTask, self).__call__(*args, **kwargs)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     return self.run(*args, **kwargs)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1109, in publish
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1206, in check_publish
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1258, in _do_publish
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 737, in wrap_f
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     return f(*args, **kwargs)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py", line 80, in publish_repo
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     return self._publisher.process_lifecycle()
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 572, in process_lifecycle
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     super(PluginStep, self).process_lifecycle()
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 163, in process_lifecycle
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     step.process()
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 256, in process
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     self._process_block()
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 303, in _process_block
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     self.process_main()
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py", line 329, in process_main
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     with_symlinks=True)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py", line 478, in create
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     component=component, architecture=architecture)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py", line 449, in add_packages
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     pkg = debpkg.DebPkg.from_file(filename, Size=sz)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debpkgr/debpkg.py", line 416, in from_file
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     control = debpkg.control.debcontrol().copy()
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debian/debfile.py", line 222, in debcontrol
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     return Deb822(self.get_content(CONTROL_FILE))
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debian/debfile.py", line 171, in get_content
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     f = self.get_file(fname, encoding=encoding, errors=errors)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debian/debfile.py", line 145, in get_file
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     fobj = self.tgz().extractfile('./' + fname)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)   File "/usr/lib/python2.7/site-packages/debian/debfile.py", line 92, in tgz
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528)     raise DebError("%s" % e)
Nov 12 20:51:00 uat-l-srv-203 pulp: celery.app.trace:ERROR: [d57893c7] (3040-42528) DebError: [Errno 12] Cannot allocate memory
2

There is definitely a memory leak in pulp_deb. (The problem stems from how pulp_deb interacts with the python-debpkgr library dependency).

I am finalizing a pull request (https://github.com/pulp/pulp_deb/pull/57) that solves these memory issues.

With my patch I have had no problems syncing very large repositories (I tested Ubuntu xenial with 53837 packages) on a system with 12 GiB of RAM.

Without my patch there is currently no solution other than adding lots and lots of memory (and/or SWAP if you have lots of time) to the system you are using. I have known for this error to affect systems with 32GiB of RAM for repositories with ~50000 packages.

3

Tks Quba42, Iā€™ll try to apply your patch and see if it solves the problem, does any tip or recommendation suggest me?

4

Be careful applying the patch on systems you care about since it involves a db migration.
(It should only be used on test systems for now.)

Until my patch makes it into a release the only available solution is lots and lots of memoryā€¦

(In case you are really keen to get this working without a db migration, the bottom commit on the issue branch contains what should be a functional version of my patch that does not require a db migration.)

5

I do not want to modify the database at this time. Could you tell me which files to change and the lines? or send me the link with just these procedures to not change the bank?

6

Right, I am going to put in one last disclaimer:

The word ā€œpatchā€ may have been a bit of a mischaracterization.
With or without the data base changes this is a major change.
A major change that is still undergoing review and frequent changes.
I am also not entirely sure what version of pulp_deb is associated with ā€˜pulp-katello-1.0.2-1.el7.noarchā€™.
(My test setup uses Katello 3.7, but I donā€™t think pulp_deb changed from Katello 3.7 to Katello 3.8).

The short version: I canā€™t be held responsible if applying my patch breaks anything.
That being said, if applied correctly the patch should work.

  1. To apply the patch without database changes you are going to need the following three files:

https://raw.githubusercontent.com/ATIX-AG/pulp_deb/74b47601a5e2f6dce159c72536f9f0a064a6b1ca/plugins/pulp_deb/plugins/distributors/distributor.py

https://raw.githubusercontent.com/ATIX-AG/pulp_deb/74b47601a5e2f6dce159c72536f9f0a064a6b1ca/plugins/pulp_deb/plugins/distributors/configuration.py

https://raw.githubusercontent.com/ATIX-AG/pulp_deb/74b47601a5e2f6dce159c72536f9f0a064a6b1ca/plugins/pulp_deb/plugins/distributors/metadata_files.py

  1. Before you change anything make sure there are no running tasks and stop your Foreman/Katello installation:

    katello-service stop

  2. You should backup the existing files on your Foreman/Katello installation:

    cd /usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/
    mv distributor.py distributor.py.backup
    mv configuration.py configuration.py.backup

  3. You can now place the three new files from step 1 in the following folder:

    /usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/

  4. Restart you katello-services:

    katello-service start

  5. Verify pulp restarted successfully:

    systemctl status pulp_streamer.service -l

    (This should include something like ā€œActive: active (running)ā€ in the output.)

If anything went wrong go back to the old pulp_deb files:

cd /usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/
rm metadata_files.py
mv distributor.py.backup distributor.py
mv configuration.py.backup configuration.py
katello-service restart
1 Like
7

On second thought, a version level incompatibility between the three files above and your pulp installation is certainly possible.

However, you should still be able to go back to your old files if that happens (as described above).

1 Like
8

It worked! In my environment it took 1 hour to promote.

Now Iā€™ll test, promote between Locations

9

This pull request is a very nice surprise.

Iā€™ve been struggling with this for the last couple of months. I ended up splitting off a new Katello instance to handle pulp_deb. I also threw 40GB of ram at it and it works well enough to be useable. So Iā€™m pleased as punch to see this optimization under development.

Thanks for tackling this!

10

PLP0000: [Errno 12] Cannot allocate memory

Patch provided by quba42 is necessary also with foreman 1.21 and Katello 3.11

when you try to create a CV for Debian Stretch !