Warning "disk is 12% full"

Support
1

Problem:

When I log into foreman is see the red dot at the bell symbol. There is a notification “Proxies - 1 New Event - foreman.example.com’s disk is 12% full. Since this proxy is running Pulp, it needs disk space to publish content views. Please ensure the disk does not get full.”

I don’t really see the point of this notification when there is still plenty of disk space available.

Expected outcome:

No notification, but only when disk space is maybe 90% full.

Foreman and Proxy versions:

1.17.1

Foreman and Proxy plugin versions:

Katello 3.6.0

Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]

2

looking at the code (
https://github.com/Katello/katello/blob/master/app/services/katello/ui_notifications/pulp/proxy_disk_space.rb#L9)
it seems that it should only warn over 90%, perhaps there is a bug in that
code…?

3

Upon first installing 1.17.1 I got the same error saying that the filesystem was 1% used.

4

+1
Same version same error

5

I could be reading that code wrong but I swear that it alerts on anything less than 90% like I think filled that directory to 91% you wouldn’t have an alert. But there is a conditional at the end of the if statements and the evaluation there might be wrong.

1 Like
6

Can you guys run this in foreman-console:

SmartProxy.unscoped.with_content.map {|x| x.statuses[:pulp].storage }

This should show JSON coming from your proxies. Git blamed /CC @dLobatog and @Justin_Sherrill

7

Here is the output:

irb(main):001:0> SmartProxy.unscoped.with_content.map {|x| x.statuses[:pulp].storage }
=> [{“pulp_dir”=>{“filesystem”=>"/dev/mapper/vg_pulp-lv_pulp", “1k-blocks”=>62879748, “used”=>7624556, “available”=>55255192, “percent”=>“13%”, “mounted”=>"/var/lib/pulp", “path”=>"/var/lib/pulp", “size”=>“kilobyte”}, “pulp_content_dir”=>{“filesystem”=>"/dev/mapper/vg_pulp-lv_pulp", “1k-blocks”=>62879748, “used”=>7624556, “available”=>55255192, “percent”=>“13%”, “mounted”=>"/var/lib/pulp", “path”=>"/var/lib/pulp/content", “size”=>“kilobyte”}, “mongodb_dir”=>{“filesystem”=>"/dev/mapper/vg_mongodb-lv_mongodb", “1k-blocks”=>25149444, “used”=>11959228, “available”=>13190216, “percent”=>“48%”, “mounted”=>"/var/lib/mongodb", “path”=>"/var/lib/mongodb", “size”=>“kilobyte”}}]

8

This indeed seems to be a logic error in the code - it should check if percentage is over 90, not under.

9

Plus the conversion from string to integer is clunky. Wanna file a patch? Should be relatively easy.

10

The check for < 90% is correct, the issue is different. In the context of our user here, here’s what happens:

  • When the code runs, percentage 12% is < 90%. The notification does not exist, therefore:
percentage[0..2].to_i < 90 && notification_already_exists?(proxy)
# evaluates to
true && false
# which is
false # so it does not skip the loop

  • The next line tries to update old notifications. Since there are no notifications at this point, update_notifications(proxy).empty? is true.

  • Finally it creates the notification (at this point we have a notification for proxy ‘foreman.example.com’ full at 12%).

After that, the job is scheduled, and when it runs (12h later):

  • The percentage is less than 90, 12%, and the notification already exists. At this point Foreman is happy and proceeds to remove the notification.
 percentage[0..2].to_i < 90 && notification_already_exists?(proxy)
 # evaluates to
 true && true

For the next 12h, we don’t have the notification. Now, 12h later… GOTO beginning of the post :wink:
Basically right before creating the notification I will add a check for > 90% full. The rest of the logic looks sound IMO.

11

@RBW @Arsene @gvde Fix at https://github.com/Katello/katello/pull/7476