Foreman plugin dependency problem with "deface"

Doug_Scoular · November 19, 2016, 3:08am

Hi All,
I know this has been discussed before in the thread "Install foreman
plugins on RPM based installation" but that was resolved back in 2013. I'm
new to ruby and foreman so I couldn't quite follow the ins and outs of that
resolution. So I thought I'd better re-ask in a new thread.

I'm trying to follow the steps outlined in:

http://projects.theforeman.org/projects/foreman/wiki/How_to_Create_a_Plugin#RPM-installations

I've managed to install what I suppose is a development instance of foreman
from the git branch 1.12-stable and successfully added my plugin to the
foreman instance. It adds the extra menus and appears on the About page.

I then built my gem as instructed with:

*gem build my_foreman_plugin.gemspec *

which gave me a my_foreman_plugin-0.0.1.gem… all good! My first ever gem.

It's the next step that's causing me dramas…

I hop on to the "production" machine I've been asked to use for my plugin
testing (it's running 1.12), I copy the my_foreman_plugin-0.0.1.gem across,
but the server doesn't have an "scl ruby193" software collection however,
it does have the following collections:

[root@fman-vgrnt-01 ~]# scl --list
rh-ruby22
sclo-ror42
tfm

So I gave the following a whirl and it seemed to be happy (strange that the
tfm software collection seems to install into the rh-ruby22 software
collection):

[root@cpos-fman-vgrnt-01 vagrant]# scl enable tfm "gem install --verbose
–ignore-dependencies foreman_my_plugin-0.0.1.gem"
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/LICENSE
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/README.md
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/Rakefile
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/app/controllers/foreman_my_plugin/hosts_controller.rb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/app/helpers/concerns/foreman_my_plugin/hosts_helper_extensions.rb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/app/models/concerns/foreman_my_plugin/host_extensions.rb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/app/overrides/dashboard/index/sample_override.html.erb.deface
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/app/views/dashboard/_foreman_my_plugin_widget.html.erb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/app/views/foreman_my_plugin/hosts/hosts/new_action.html.erb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/app/views/foreman_my_plugin/hosts/new_action.html.erb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/app/views/foreman_my_plugin/layouts/layouts/new_layout.html.erb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/app/views/foreman_my_plugin/layouts/new_layout.html.erb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/config/routes.rb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/lib/foreman_my_plugin.rb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/lib/foreman_my_plugin/engine.rb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/lib/foreman_my_plugin/version.rb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/lib/tasks/foreman_my_plugin_tasks.rake
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/locale/Makefile
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/locale/en/foreman_my_plugin.po
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/locale/foreman_my_plugin.pot
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/locale/gemspec.rb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/test/factories/foreman_my_plugin_factories.rb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/test/test_plugin_helper.rb
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/test/unit/foreman_my_plugin_test.rb
Successfully installed foreman_my_plugin-0.0.1
Parsing documentation for foreman_my_plugin-0.0.1
Parsing sources…
100% [ 4/ 4] lib/tasks/foreman_my_plugin_tasks.rake
Done installing documentation for foreman_my_plugin after 0 seconds
1 gem installed

However, when I restart the apache server, I'm told the plugin's dependency
on the deface gem is a problem:

Web application could not be started

Gem loading error: Could not find 'deface' (>= 0) among 91 total gem(s)
Checked in
'GEM_PATH=/opt/theforeman/tfm/root/usr/share/gems:/opt/rh/sclo-ror42/root/usr/share/gems:/cpos-local/usr/share/foreman/.gem/ruby:/opt/rh/rh-ruby22/root/usr/share/gems:/opt/rh/rh-ruby22/root/usr/local/share/gems/opt/rh/sclo-ror42/root/usr/share/gems:/opt/rh/rh-ruby22/root/usr/local/share/gems',
execute gem env for more information (RuntimeError)

/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.3.0/lib/bundler_ext/bundler_ext.rb:30:in
`strict_error'

/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.3.0/lib/bundler_ext/bundler_ext.rb:56:in
`rescue in block in system_require'

/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.3.0/lib/bundler_ext/bundler_ext.rb:37:in
`block in system_require'

/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.3.0/lib/bundler_ext/bundler_ext.rb:35:in
`each'

/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.3.0/lib/bundler_ext/bundler_ext.rb:35:in
system_require' /cpos-local/usr/share/foreman/config/application.rb:16:in<top
(required)>'

/opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in
`require'

/opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in
require' /cpos-local/usr/share/foreman/config/environment.rb:2:in<top
(required)>'

/opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in
`require'

/opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in
require' config.ru:3:inblock in <main>'

/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/builder.rb:55:in
`instance_eval'

/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/builder.rb:55:in
initialize' config.ru:1:innew'
config.ru:1:in `<main>'

/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb:105:in
`eval'

/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb:105:in
`preload_app'

/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb:150:in
`<module:App>'

/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb:29:in
`<module:PhusionPassenger>'

/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb:28:in
`<main>'

As I said, I'm new to ruby. I don't quite understand why the instruction
tell me to use "–ignore-dependencies" when installing the gem nor do I
understand why "deface" isn't already installed as a dependency of the
production installation which is already using various plugins and smart
proxies which I assume may require "deface".

I can find no evidence of a deface module already present on the system.
Just the deface view overrides in my own plugin:

[root@cpos-fman-vgrnt-01 vagrant]# locate deface
/home/vagrant/foreman_my_plugin/app/overrides/dashboard/index/sample_override.html.erb.deface
/opt/foreman_my_plugin/app/overrides/dashboard/index/sample_override.html.erb.deface
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/foreman_my_plugin-0.0.1/app/overrides/dashboard/index/sample_override.html.erb.deface

Any help hugely appreciated…

Cheers,

Doug

Doug_Scoular · November 21, 2016, 9:45am

Hi All,

            Good news… I think I&#39;ve fixed this...



            I decided to attempt to install “deface”on to our

pseudo-"production" foreman server:

[root@fman-vgrnt-01 vagrant]# scl enable tfm "gem install deface"

Fetching: polyglot-0.3.5.gem (100%)

Successfully installed polyglot-0.3.5

Fetching: colorize-0.8.1.gem (100%)

Successfully installed colorize-0.8.1

Fetching: mini_portile2-2.1.0.gem (100%)

Successfully installed mini_portile2-2.1.0

Fetching: nokogiri-1.6.8.1.gem (100%)

Building native extensions. This could take a while…

ERROR: Error installing deface:

    ERROR: Failed to build gem native extension.



/opt/rh/rh-ruby22/root/usr/bin/ruby -r

./siteconf20161120-4146-wy5wdc.rb extconf.rb

mkmf.rb can't find header files for ruby at
/opt/rh/rh-ruby22/root/usr/share/include/ruby.h

extconf failed, exit code 1

Gem files will remain installed in
/opt/rh/rh-ruby22/root/usr/local/share/gems/gems/nokogiri-1.6.8.1 for
inspection.

Results logged to
/opt/rh/rh-ruby22/root/usr/local/lib64/gems/ruby/nokogiri-1.6.8.1/gem_make.out

So, obviously, I needed to install the relevant “devel” package for the
rh-ruby-22 software collection so that native extensions could be built. So
I ran this:

[root@fman-vgrnt-01 vagrant]# yum install rh-ruby22-ruby-devel

            But I got the follow error:

http://mirror.centos.org/centos/6/sclo/x86_64/rh/repodata/repomd.xml:
[Errno 12] Timeout on
http://mirror.centos.org/centos/6/sclo/x86_64/rh/repodata/repomd.xml: (28,
'Connection time-out')

Trying other mirror.

Error: Cannot retrieve repository metadata (repomd.xml) for repository:
centos-sclo-rh. Please verify its path and try again

I tried again, and it worked, the centos repo mirror must have been having
a transient issue! I was then able to install deface and its dependencies:

[root@fman-vgrnt-01 vagrant]# scl enable tfm "gem install deface"

Building native extensions. This could take a while…

Successfully installed nokogiri-1.6.8.1

Fetching: deface-1.0.2.gem (100%)

Successfully installed deface-1.0.2

Parsing documentation for nokogiri-1.6.8.1

Installing ri documentation for nokogiri-1.6.8.1

Parsing documentation for deface-1.0.2

Installing ri documentation for deface-1.0.2

Done installing documentation for nokogiri, deface after 8 seconds

2 gems installed

So “deface” appears to be a run-time dependency required by plugins based
on the foreman plugin example template which, likely, need to be
incorporated into our foreman provisioning script.

I’m surprised none of the other plugins had the “deface” requirement.

Finally, after restarting apache, I was able to login to the foreman server
and see my plugin’s menu additions.

Now plugin development can start in earnest!

Cheers,

Doug

Gwmngilfen · November 21, 2016, 8:13pm

Glad you got it working! For reference, here's a few things to know

It's generally not recommended to do development stuff on a production
server, since (as you've found out) it's remarkably tricky to get the SCL
dependencies right. As you've seen, it can be done, but it's probably
better making a proper RPM out of it. You can use our spec files as a base

here's an example for another deface-using plugin:

github.com

theforeman/foreman-packaging/blob/rpm/1.13/rubygem-foreman_column_view/rubygem-foreman_column_view.spec

# This package contains macros that provide functionality relating to
# Software Collections. These macros are not used in default
# Fedora builds, and should not be blindly copied or enabled.
# Specifically, the "scl" macro must not be defined in official Fedora
# builds. For more information, see:
# http://docs.fedoraproject.org/en-US/Fedora_Contributor_Documentation
# /1/html/Software_Collections_Guide/index.html

%{?scl:%scl_package rubygem-%{gem_name}}
%{!?scl:%global pkg_name %{name}}

%global gem_name foreman_column_view

%global foreman_dir /usr/share/foreman
%global foreman_bundlerd_dir %{foreman_dir}/bundler.d
%global foreman_pluginconf_dir /etc/foreman/plugins

Summary:    Column View Plugin for Foreman
Name:       %{?scl_prefix}rubygem-%{gem_name}
Version:    0.3.0

This file has been truncated. show original

Deface is indeed needed by many plugins - but we provide each gem
dependency as a separate package. You can find deface here:

http://yum.theforeman.org/plugins/latest/el7/x86_64/tfm-rubygem-deface-1.0.2-3.el7.noarch.rpm

This would obviously be an RPM dependency of the package created by (1).

Indirectly, this is why the gems are installed with --ignore-dependencies,
since it's intended that production systems get their plugins from yum/dnf
and as such, they can all depend on the deface rubygem package.

Hope that helps!
Greg

Doug_Scoular · November 22, 2016, 8:29am

Hi Greg,

That helps heaps!

I'm curious as to why the "production" 1.12 machine which uses a remote
puppet CA, puppet Compile Master and DB didn't already have a dependency on
"deface". I guess maybe the foreman_setup plugin doesn't use it.

Anyway, I'm now furiously trying to learn ruby so I can understand the
example plugin and the "column_view" plugin you referenced.

Many, many thanks,

Doug

···

On 22 November 2016 at 07:13, Greg Sutcliffe wrote:

Glad you got it working! For reference, here’s a few things to know

It’s generally not recommended to do development stuff on a production
server, since (as you’ve found out) it’s remarkably tricky to get the SCL
dependencies right. As you’ve seen, it can be done, but it’s probably
better making a proper RPM out of it. You can use our spec files as a base

here’s an example for another deface-using plugin:

https://github.com/theforeman/foreman-packaging/blob/rpm/1.
13/rubygem-foreman_column_view/rubygem-foreman_column_view.spec

Deface is indeed needed by many plugins - but we provide each gem
dependency as a separate package. You can find deface here:

http://yum.theforeman.org/plugins/latest/el7/x86_64/tfm-
rubygem-deface-1.0.2-3.el7.noarch.rpm

This would obviously be an RPM dependency of the package created by (1).

Indirectly, this is why the gems are installed with --ignore-dependencies,
since it’s intended that production systems get their plugins from yum/dnf
and as such, they can all depend on the deface rubygem package.

Hope that helps!
Greg

–
You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/foreman-users/idtLy5U0jyI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–

The big print giveth and the small print taketh away.

Gwmngilfen · November 22, 2016, 10:44am

> Hi Greg,
>
> That helps heaps!
>
> I'm curious as to why the "production" 1.12 machine which uses a remote
> puppet CA, puppet Compile Master and DB didn't already have a dependency on
> "deface". I guess maybe the foreman_setup plugin doesn't use it.
>

Correct (not in the Requires here:
https://github.com/theforeman/foreman-packaging/blob/rpm/1.13/rubygem-foreman_setup/rubygem-foreman_setup.spec)

setup adds whole new views but doesn't need to modify existing ones, so
no Deface required.

> Anyway, I'm now furiously trying to learn ruby so I can understand the
> example plugin and the "column_view" plugin you referenced.
>

Shout if you get stuck - I'm always happy to chat/hang out with the
community (man, I really need to start running some community office hours
:P) and column_view is one of my plugins

Cheers
Greg

···

On 22 November 2016 at 08:29, Doug Scoular wrote:

Doug_Scoular · November 28, 2016, 1:02am

Hi Greg et al,
I'm afraid I'm stuck again…

I managed to install foreman 1.12-stable from source and got the example
plugin working after I manually installed the deface gem with "gem install
deface".

I then realised that I couldn't really start exploring what my plugin
needed to do without some hosts to work with.

So I ensured that a puppet master and a puppet agent were running on the
same VM as foreman and were able to communicate.

However, this is where I got a bit confused. How do I get the puppet
master's hosts to appear in foreman?

At first I thought I just had to add a few lines to the
/etc/puppet/puppet.conf file referencing the script mentioned in the manual:

https://theforeman.org/manuals/1.12/index.html#3.5.5FactsandtheENC

So I added the following:

external_nodes = /etc/puppet/node.rb
node_terminus  = exec

Downloaded the node.rb script from
https://raw.githubusercontent.com/theforeman/puppet-foreman/master/files/external_node_v2.rb
and placed that at /etc/puppet/node.rb.

Script usually acts as an ENC for a single host, with the certname

supplied as argument

if 'facts' is true, the YAML facts for the host are uploaded

ENC output is printed and cached

···

# # If --push-facts is given as the only arg, it uploads facts for all hosts and then exits. # Useful in scenarios where the ENC isn't used.

However, the node.rb script just doesn’t seem to communicate with the
master:

Debug: Executing ‘/etc/puppet/node.rb theforeman.example.com’
Warning: Unable to fetch my node definition, but the agent run will
continue:
Warning: Failed to find theforeman.example.com via exec: Execution of
’/etc/puppet/node.rb theforeman.example.com’ returned 1:

When I try running the node.rb command by hand I see it’s getting a 400
error from the master:

[root@theforeman puppet]# which ruby
/opt/rh/rh-ruby23/root/usr/bin/ruby
[root@theforeman puppet]# /etc/puppet/node.rb theforeman.example.com
During the fact upload the server responded with: 400 Bad Request . Error
is ignored and the execution continues.
No support for plurality singular for POST operations
/etc/puppet/node.rb:378: warning: constant ::TimeoutError is deprecated
Error retrieving node theforeman.example.com: Net::HTTPBadRequest
Check Foreman’s /var/log/foreman/production.log for more information.

There’s nothing in foreman’s log but there is in the puppet master log:

Nov 27 16:02:19 localhost puppet-master[9921]: No support for plurality
singular for POST operations
Nov 27 16:02:19 localhost puppet-master[9921]: The indirection name must be
purely alphanumeric, not ‘theforeman.example.com’

If I remove the external_nodes and node_terminus lines my puppet agent
starts happily talking to the puppet master again (I have http_debug true
set):

[root@theforeman puppet]# puppet agent -t
opening connection to theforeman.example.com…
opened
<- “GET
/production/node/theforeman.example.com?fail_on_404=true&transaction_uuid=c689a6bb-6539-4371-a841-e9d1c1c8e542
HTTP/1.1\r\nHost: theforeman.example.com:8140\r\nAccep
t: pson, yaml, b64_zlib_yaml, raw\r\nAccept-Encoding: identity\r\n\r\n”
-> “HTTP/1.1 200 OK \r\n”
-> “Date: Mon, 28 Nov 2016 00:09:10 GMT\r\n”
-> “Connection: close\r\n”
-> “X-Puppet-Version: 3.8.7\r\n”
-> “Content-Length: 5291\r\n”
-> “Content-Type: text/pson\r\n”
-> “Server: WEBrick/1.3.1 (Ruby/1.8.7/2013-06-27) OpenSSL/1.0.1e\r\n”
-> “\r\n"
reading 5291 bytes…
-> “”
->
”{“data”:{“name”:“theforeman.example.com”,“parameters”:{“ipaddress_eth1”:“192.168.35.5”,“interfaces”:“eth0,eth1,lo”,“operatingsystem”:“RedHat”,“rubys
itedir”:"/usr/lib/ruby/site_ruby/1.8",“blockdevice_sr0_model”:“CD-ROM”,“lsbmajdistrelease”:“6”,“system_uptime”:“hours34seconds125942days1uptime1
day”,“fqdn”
:“theforeman.example.com”,“blockdevices”:“sda,sr0”,“kernelmajversion”:“2.6”,

… lines omitted for brevity …

I then wondered if I needed a smart proxy? I cloned the 1.12-stable version
and tried running that too but I fear I’m getting hopelessly confused
between the external node script, the puppet smart proxy and whatever a
foreman proxy is. My guess is that “foreman proxy” is just another term for
"smart proxy" and there are a number of smart proxies available.

All I really want is the absolute simplest 1.12-stable foreman plugin
development setup with a few test hosts.

I’m happy to try to document the steps needed. I have a vagrant RHEL6 VM as
my starting point, I have the puppet agent talking to the puppet master on
that VM, I have foreman 1.12-stable loading my clone of the example foreman
plugin starter project but I find myself completely confused.

Any words of wisdom, clarification or encouragement much appreciated.

Cheers,

Doug