Can't import Puppet classes after update

Problem:
After updating to Foreman 2.3.2 and Katello 3.18 I had to restore our puppet.conf because the update process reset it. We use custom path for our environment because we manage our puppet code with git. I reconfigure it but when i try to import classes from my server in the web ui I get an error message: Oops something goes wrong unnumbered(1) mixed with named

when I execute the suggested command from the error page foreman-rake errors:fetch_log request_id=46399b8c --trace
I’m also getting an error:

 > foreman-rake errors:fetch_log request_id=46399b8c --trace
> ** Invoke errors:fetch_log (first_time)
> ** Invoke environment (first_time)
> ** Execute environment
> ** Execute errors:fetch_log
> rake aborted!
> NoMethodError: private method 'puts' called for nil:NilClass
> /usr/share/foreman/lib/tasks/errors.rake:30:in 'block (2 levels) in <top (required)>'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in 'block in execute'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in 'each'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in 'execute'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in 'block in invoke_with_call_chain'
> /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in 'mon_synchronize'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in 'invoke_with_call_chain'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in 'invoke'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in 'invoke_task'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in 'block (2 levels) in top_level'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in 'each'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in 'block in top_level'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in''run_with_threads'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in 'top_level'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in 'block in run'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in 'standard_exception_handling'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in 'run'
> /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in'<top (required)>'
> /opt/rh/rh-ruby25/root/usr/bin/rake:23:in 'load'
> /opt/rh/rh-ruby25/root/usr/bin/rake:23:in '<main>'
> Tasks: TOP => errors:fetch_log

Can someone could help me with this?

Expected outcome:
Import puppet classe from server

Foreman and Proxy versions:
Foreman 2.3.2-1
Katello 3.18.1-1

Distribution and version:
CentOS 7.9.2009

Other relevant data:

Looks like there is a bug in the errors:fetch_log task when the layout of the log isn’t set to multiline_request_pattern that is preventing you from seeing the actual error. Please search /var/log/foreman/production.log for request 46399b8c and look for the full stack trace there. I will open an issue to correct the rake task.

This should fix the rake task itself:

1 Like

Here is the stack trace:

# cat production.log-20210216 | grep 46399b8c
2021-02-15T13:40:10 [I|app|46399b8c] Started GET “/environments/import_environments?env=test&proxy=1-myproxy.hostname” for 127.0.0.1 at 2021-02-15 13:40:10 +0100
2021-02-15T13:40:10 [I|app|46399b8c] Processing by EnvironmentsController#import_environments as HTML
2021-02-15T13:40:10 [I|app|46399b8c] Parameters: {“env”=>“test”, “proxy”=>“1-myproxy.hostname”}
2021-02-15T13:40:15 [I|app|46399b8c] Rendering common/_puppetclasses_or_envs_changed.html.erb within layouts/application
2021-02-15T13:40:15 [I|app|46399b8c] Rendered common/_puppetclasses_or_envs_changed.html.erb within layouts/application (Duration: 12.9ms | Allocations: 9701)
2021-02-15T13:40:15 [W|app|46399b8c] unnumbered(1) mixed with named
2021-02-15T13:40:15 [I|app|46399b8c] Backtrace for ‘unnumbered(1) mixed with named’ error (ActionView::Template::Error): unnumbered(1) mixed with named
2021-02-15T13:40:15 [I|app|46399b8c] Rendering common/500.html.erb within layouts/application
2021-02-15T13:40:15 [I|app|46399b8c] Rendered common/500.html.erb within layouts/application (Duration: 6.7ms | Allocations: 6000)
2021-02-15T13:40:15 [I|app|46399b8c] Rendered layouts/_application_content.html.erb (Duration: 6.5ms | Allocations: 5759)
2021-02-15T13:40:15 [I|app|46399b8c] Rendering layouts/base.html.erb
2021-02-15T13:40:15 [I|app|46399b8c] Rendered layouts/base.html.erb (Duration: 4.0ms | Allocations: 6909)
2021-02-15T13:40:15 [I|app|46399b8c] Completed 500 Internal Server Error in 4712ms (Views: 21.4ms | ActiveRecord: 282.6ms | Allocations: 1053487)

Thanks for the fix, I will do the changes manually asap.

rake task successfully patchted! Now returning the log trace and Forman / Plugins versions

1 Like

You should also change the logging layout to multiline_request_pattern so the stack trace is also prefixed with the request id (as you can see, the grep you posted does not include the actual back track for the error, which you can find by manually viewing the log file right after the Backtrace for... line)

Ok… How do I do that? :slight_smile:

If you are using the installer, you can use the --foreman-logging-layout option to configure it. If not, it should be defined in /etc/foreman/logging.yaml. once you modify it you’ll need to restart foreman for the change to take affect.

I have no /etc/foreman/logging.yaml but I have finded /usr/share/foreman/config/logging.yaml which contain:

:default:
:colorize: false
:console_inline: false
:log_trace: false
:level: info
:type: file
:layout: multiline_request_pattern
:json_items:
- logger
- timestamp
- level
- message
- mdc
- ndc
:pattern: “%d [%.1l|%.3c|%.8X{request}] %m\n”
:sys_pattern: “%m\n”
:facility: LOG_LOCAL6

But I’m still getting the warning “Logging layout is not multiline_request_pattern.”

Looks like it is defined in /etc/foreman/settings.yaml for systems configured by the installer

Indeed, but run foreman-installer --foreman-logging-layout didn’t update this setting.

I noticed this error during foreman-installer was running:

Upgrade Step 2/3: katello:correct_puppet_environments. This may take a long while.
Processing Puppet Environment 1/9: 1-Affichage_Centos_8-v1_0-puppet-a7d90fab-abf5-4b2e-be06-38fd4528a0a0 (1)
Content View Puppet Environment 1 Missing, Creating.
Failed upgrade task: katello:correct_puppet_environments, see logs for more information.

Any idea how can i manage this?

you need to give it a parameter, foreman-installer --foreman-logging-layout=multiline_request_pattern

I’m not sure about this one, perhaps one of the @katello team will know?

1 Like

@GC29 there’s very likely a stack trace in /var/log/foreman/production.log for that - can you share it here?

I found this:

# cat production.log | grep 1-Affichage_Centos_8-v1_0-puppet-a7d90fab-abf5-4b2e-be06-38fd4528a0a0
2021-02-17T11:11:54 [I|kat|] GET: https://myserver.domain/pulp/api/v2/repositories/1-Affichage_Centos_8-v1_0-puppet-a7d90fab-abf5-4b2e-be06-38fd4528a0a0/?: {"content_type"=>"appli
404 Not Found: {"http_request_method": "GET", "exception": null, "error_message": "Missing resource(s): repository=1-Affichage_Centos_8-v1_0-puppet-a7d90fab-abf5-4b2e-be06-38fd4528antos_8-v1_0-puppet-a7d90fab-abf5-4b2e-be06-38fd4528a0a0/?", "http_status": 404, "error": {"code": "PLP0009", "data": {"resources": {"repository": "1-Affichage_Centos_8-v1_0-puppet-aMissing resource(s): repository=1-Affichage_Centos_8-v1_0-puppet-a7d90fab-abf5-4b2e-be06-38fd4528a0a0", "sub_errors": []}, "traceback": null, "resources": {"repository": "1-Affichag0a0"}}

But I have no repository called “Affichage Centos 8” it’s my content-view name for CentOS 8 client.

# hammer content-view list
----------------|---------------------------|---------------------------|-----------|---------------------|----------------------------------------------
CONTENT VIEW ID | NAME                      | LABEL                     | COMPOSITE | LAST PUBLISHED      | REPOSITORY IDS
----------------|---------------------------|---------------------------|-----------|---------------------|----------------------------------------------
3               | Affichage Centos 7        | Affichage_Centos_7        | false     | 2021/02/11 12:58:50 | 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20
2               | Affichage Centos 8        | Affichage_Centos_8        | false     | 2021/02/11 13:11:16 | 1, 3, 5, 6, 4, 8, 2, 112, 123, 124
5               | Affichage Ubuntu 20.04    | Affichage_Ubuntu_20_04    | false     | 2021/02/16 10:55:12 | 270
1               | Default Organization View | Default_Organization_View | false     | 2020/11/24 14:54:20 |
----------------|---------------------------|---------------------------|-----------|---------------------|----------------------------------------------

I’m afraid there isn’t enough context to make a determination. The name of that repo is valid because a copy is created from Library with the name of the CV within its own name. Can you run foreman-rake katello:correct_puppet_environments and share the stack trace?

Here is the full trace of foreman-rake katello:correct_puppet_environments

# foreman-rake katello:correct_puppet_environments --trace
** Invoke katello:correct_puppet_environments (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:check_ping (first_time)
** Invoke environment
** Execute katello:check_ping
{:services=>
  {:candlepin=>{:status=>"ok", :duration_ms=>"24"},
   :candlepin_auth=>{:status=>"ok", :duration_ms=>"26"},
   :foreman_tasks=>
    {:status=>"FAIL",
     :message=>
      "The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer"},
   :katello_events=>
    {:status=>"ok", :message=>"2 Processed, 0 Failed", :duration_ms=>"0"},
   :candlepin_events=>
    {:status=>"ok", :message=>"1 Processed, 0 Failed", :duration_ms=>"0"},
   :pulp3=>{:status=>"ok", :duration_ms=>"49"},
   :pulp=>{:status=>"ok", :duration_ms=>"56"},
   :pulp_auth=>{:status=>"ok", :duration_ms=>"26"}},
 :status=>"FAIL"}
rake aborted!
Not all the services have been started. Check the status report above and try again.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/tasks/reimport.rake:10:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:217:in `block in invoke_prerequisites'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:215:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:215:in `invoke_prerequisites'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:194:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => katello:correct_puppet_environments => katello:check_ping

Linked to this post ?

@GC29 nice find. I do believe that is related to the issue you’re hitting. Let us know if that does the trick for you!

Unfortunately no. It’s even worst… Here the result after changing accordingly to my conf:
(/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/tasks/pulp3_content_switchover.rake)
and restarted the services with hammer service restart

\# foreman-rake katello:correct_puppet_environments --trace
** Invoke katello:correct_puppet_environments (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:check_ping (first_time)
** Invoke environment
** Execute katello:check_ping
{:services=>
  {:candlepin=>{:status=>"ok", :duration_ms=>"462"},
   :candlepin_auth=>{:status=>"ok", :duration_ms=>"70"},
   :foreman_tasks=>
    {:status=>"FAIL",
     :message=>
      "The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer"},
   :katello_events=>
    {:status=>"FAIL", :message=>"Not running", :duration_ms=>"1"},
   :candlepin_events=>
    {:status=>"ok", :message=>"27 Processed, 0 Failed", :duration_ms=>"0"},
   :pulp3=>{:status=>"ok", :duration_ms=>"338"},
   :pulp=>{:status=>"ok", :duration_ms=>"129"},
   :pulp_auth=>{:status=>"ok", :duration_ms=>"57"}},
 :status=>"FAIL"}
rake aborted!
Not all the services have been started. Check the status report above and try again.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/tasks/reimport.rake:10:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:217:in `block in invoke_prerequisites'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:215:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:215:in `invoke_prerequisites'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:194:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => katello:correct_puppet_environments => katello:check_ping

Now the Katello_events is failling too!

I also try this change with no success:

In this file:/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/tasks/reimport.rake

change
task :check_ping => :environment do
to
task :check_ping => "dynflow:client" do

Result:
{:services=>
{:candlepin=>{:status=>“ok”, :duration_ms=>“600”},
:candlepin_auth=>{:status=>“ok”, :duration_ms=>“142”},
:foreman_tasks=>{:status=>“ok”, :duration_ms=>“5”},
:katello_events=>
{:status=>“FAIL”, :message=>“Not running”, :duration_ms=>“1”},
:candlepin_events=>
{:status=>“ok”, :message=>“0 Processed, 0 Failed”, :duration_ms=>“0”},
:pulp3=>{:status=>“ok”, :duration_ms=>“310”},
:pulp=>{:status=>“ok”, :duration_ms=>“121”},
:pulp_auth=>{:status=>“ok”, :duration_ms=>“159”}},
:status=>“FAIL”}
rake aborted!
Not all the services have been started. Check the status report above and try again.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/tasks/reimport.rake:10:in `block (2 levels) in <top (required)>’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute’