YAML value error after Upgrade to 1.17.3

Support
#1

Problem:

All my YAML smart-class parameter values are wrongly formatted after upgrading from 1.16.2 to 1.17.3.

They look like this now:

--- '{"800-http_ALLOW"=>{"chain"=>"INPUT", "dport"=>80, "state"=>"NEW", "proto"=>"tcp",
  "action"=>"accept"}, "800-https_ALLOW"=>{"chain"=>"INPUT", "dport"=>443, "state"=>"NEW",
  "proto"=>"tcp", "action"=>"accept"}}'

Expected outcome:

800-http_ALLOW:
  chain: INPUT
  dport: 80
  state: NEW
  proto: tcp
  action: accept
800-https_ALLOW:
  chain: INPUT
  dport: 443
  state: NEW
  proto: tcp
  action: accept

Foreman and Proxy versions:
1.17.3

I’m aware of Bug #24640: 1.17 migration causes array/hash values for parameters to turn into strings with escaped quotes - Foreman, however I think this must be something different (the workaround doesn’t work either)

#2

It looks pretty much as the linked issue. Try running following

LookupKey.unscoped.select { |k| !k.valid? }

does it print [] or something different? If something different, try running this

LookupKey.unscoped.select { |k| !k.valid? }.map(&:errors)

and upload the result

#3

Hi,

This is the output of the 2. command:

irb(main):004:0> LookupKey.unscoped.select { |k| !k.valid? }.map(&:errors)
=> [#<ActiveModel::Errors:0x000000000be8ae08 @base=#<PuppetclassLookupKey id: 188, key: "fw_ruleset", created_at: "2016-04-05 16:03:28", updated_at: "2018-08-18 11:45:19", puppetclass_id: nil, default_value: "${basis_firewall::params::fw_ruleset}", path: nil, description: "", validator_type: "", validator_rule: nil, key_type: "yaml", override: true, required: false, merge_overrides: false, avoid_duplicates: false, omit: true, type: "PuppetclassLookupKey", merge_default: false, hidden_value: false>, @messages={:lookup_values=>["is invalid"]}, @details={:lookup_values=>[{:error=>:invalid, :value=>#<ActiveRecord::Associations::CollectionProxy [#<LookupValue id: 1, match: "fqdn=l9901022.vdst.vbaintern.de", value: "{\"800-http_ALLOW\"=>{\"chain\"=>\"INPUT\", \"dport\"=>80,...", lookup_key_id: 188, created_at: "2018-08-18 11:47:18", updated_at: "2018-08-19 10:22:17", omit: false>]>}]}>]

I’ve used my Vagrant test environment to produce this output. There is currently only one YAML value defined to simplify things …

Cheers,
Martin

#4

That’s the reason why workaround (and proposed fix) wouldn’t work in you case. It seems there’s some invalid override value. If you delete the invalid value, it should save fine with fixed default value.

#5

not sure about this … the parameter values were working fine in 1.16.2. I’m using Foreman as Puppet ENC.

I suspect, that the values became invalid during the upgrade to 1.17.3, which lead to the findings when executing LookupKey.unscoped.select { |k| !k.valid? }

Is there a way to check if the parameter values are valid before starting the update? As I using my test environment I can quickly revert to the old foreman version to test this.

#6

you can run the same code before the upgrade, if the value is invalid it will return the same message.
You can also try selecting one of the invalid ones after the upgrade and do something like:

LookupValue.unscoped.detect { |v| !v.valid? }.send(:validate_and_cast_value)

Which will hopefully show the exact error causing the casting to fail

#7

It seems that there are no unscoped values when using 1.16.2:

   irb(main):001:0> LookupKey.unscoped.select { |k| !k.valid? }
    => []

After the Upgrade to 1.17.3 it looks like this (like I said, I currently have only 1 YAML value defined in my test environment …)

irb(main):001:0> LookupKey.unscoped.select { |k| !k.valid? }
=> [#<PuppetclassLookupKey id: 188, key: "fw_ruleset", created_at: "2016-04-05 16:03:28", updated_at: "2018-08-19 12:44:33", puppetclass_id: nil, default_value: "${basis_firewall::params::fw_ruleset}", path: nil, description: "", validator_type: "", validator_rule: nil, key_type: "yaml", override: true, required: false, merge_overrides: false, avoid_duplicates: false, omit: true, type: "PuppetclassLookupKey", merge_default: false, hidden_value: false>]

irb(main):003:0* LookupValue.unscoped.detect { |v| !v.valid? }.send(:validate_and_cast_value)
=> ["is invalid yaml", "is invalid yaml"]

However, like I stated in my first post, the value of the YAML parameter changed to this after the Upgrade:

--- '{"800-http_ALLOW"=>{"chain"=>"INPUT", "dport"=>80, "state"=>"NEW", "proto"=>"tcp",
  "action"=>"accept"}, "800-https_ALLOW"=>{"chain"=>"INPUT", "dport"=>443, "state"=>"NEW",
  "proto"=>"tcp", "action"=>"accept"}}'

Which is clearly not a valid YAML value after all …

#8

I think I figured out what happened. When the migration ran the YAML value got loaded (which turned it into a ruby hash) and then dumped as a string:

[14] pry(main)> v = "
[14] pry(main)* ---
[14] pry(main)* 800-http_ALLOW:
[14] pry(main)*   chain: INPUT
[14] pry(main)*   dport: 80
[14] pry(main)*   state: NEW
[14] pry(main)*   proto: tcp
[14] pry(main)*   action: accept
[14] pry(main)* "
[15] pry(main)> YAML.load(v).to_s
=> "{\"800-http_ALLOW\"=>{\"chain\"=>\"INPUT\", \"dport\"=>80, \"state\"=>\"NEW\", \"proto\"=>\"tcp\", \"action\"=>\"accept\"}}"

I’ll continue to figure out how to turn that back into a valid YAML.

#9

I have opened a PR with an attempted fix for the issue here:
https://github.com/theforeman/foreman/pull/5956

If you can try testing it on your upgraded test environment it would be awesome. all you need to do is to download the new migration to /usr/share/foreman/db/migrate and run foreman-rake db:migrate (the changes to the old migration is to prevent people who still haven’t upgraded from hitting the issue in the future).

#10

I’ve tested the fix on my test environment and everything looks good so far. Also tested the fix on a copy of our production environment with several hundred smart-parameter associations and it also looks quite promising. I was not able to find new invalid smart-class parameter values after the upgrade …

Thank you very much for your help.

1 Like