Block in create_with_logging_context >> Completed 500 Internal Server Error in 15ms

Support
1

Problem:

Error message in production.log

93e369c1 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-07-18T18:39:45 [I|app|93e369c1]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (Duration: 5.2ms | Allocations: 6627)
2023-07-18T18:39:45 [I|app|93e369c1]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 8.8ms | Allocations: 13113)
2023-07-18T18:39:45 [I|app|93e369c1] Completed 500 Internal Server Error in 15ms (Views: 9.2ms | ActiveRecord: 0.7ms | Allocations: 15023)
2023-07-18T18:39:55 [I|app|99cc3696] Started GET "/notification_recipients" for 10.251.217.170 at 2023-07-18 18:39:55 +0200
2023-07-18T18:39:55 [I|app|99cc3696] Processing by NotificationRecipientsController#index as JSON
2023-07-18T18:39:55 [W|app|99cc3696] Action failed
2023-07-18T18:39:55 [I|app|99cc3696] Backtrace for 'Action failed' error (TypeError): incompatible marshal file format (can't be read)
 99cc3696 |     format version 4.8 required; 123.34 given

Expected outcome:

Foreman and Proxy versions:
Version 3.6.1 © 2009-2023 Paul Kelly and Ohad Levy
Foreman and Proxy plugin versions:

Installed Packages
ansible-collection-theforeman-foreman-3.10.0-1.el8.noarch
ansible-collection-theforeman-operations-1.3.0-1.el8.noarch
ansiblerole-foreman_scap_client-0.2.0-2.el8.noarch
candlepin-4.2.13-1.el8.noarch
candlepin-selinux-4.2.13-1.el8.noarch
foreman-3.6.1-1.el8.noarch
foreman-cli-3.6.1-1.el8.noarch
foreman-client-release-3.6.1-1.el8.noarch
foreman-debug-3.6.1-1.el8.noarch
foreman-dynflow-sidekiq-3.6.1-1.el8.noarch
foreman-installer-3.6.1-1.el8.noarch
foreman-installer-katello-3.6.1-1.el8.noarch
foreman-ovirt-3.6.1-1.el8.noarch
foreman-postgresql-3.6.1-1.el8.noarch
foreman-proxy-3.6.1-1.el8.noarch
foreman-redis-3.6.1-1.el8.noarch
foreman-release-3.6.1-1.el8.noarch
foreman-selinux-3.6.1-1.el8.noarch
foreman-service-3.6.1-1.el8.noarch
foreman-vmware-3.6.1-1.el8.noarch
katello-4.8.1-1.el8.noarch
katello-certs-tools-2.9.0-1.el8.noarch
katello-client-bootstrap-1.7.9-1.el8.noarch
katello-common-4.8.1-1.el8.noarch
katello-debug-4.8.1-1.el8.noarch
katello-repos-4.8.1-1.el8.noarch
katello-selinux-4.0.2-3.el8.noarch
pulpcore-selinux-1.3.2-1.el8.x86_64
python39-pulp-ansible-0.16.0-1.el8.noarch
python39-pulp-certguard-1.5.6-1.el8.noarch
python39-pulp-cli-0.14.0-4.el8.noarch
python39-pulp-container-2.14.5-1.el8.noarch
python39-pulp-deb-2.20.2-1.el8.noarch
python39-pulp-file-1.12.0-1.el8.noarch
python39-pulp-ostree-2.0.0-0.8.a6.el8.noarch
python39-pulp-python-3.8.0-1.el8.noarch
python39-pulp-rpm-3.19.2-1.el8.noarch
python39-pulpcore-3.22.2-4.el8.noarch
qpid-proton-c-0.37.0-1.el8.x86_64
rubygem-foreman-tasks-7.2.1-2.fm3_6.el8.noarch
rubygem-foreman_acd-0.9.4-1.fm3_6.el8.noarch
rubygem-foreman_ansible-11.2.0-1.fm3_6.el8.noarch
rubygem-foreman_azure_rm-2.2.9-1.fm3_6.el8.noarch
rubygem-foreman_bootdisk-21.0.4-1.fm3_6.el8.noarch
rubygem-foreman_column_view-0.4.0-6.fm3_3.el8.noarch
rubygem-foreman_discovery-22.0.4-1.fm3_6.el8.noarch
rubygem-foreman_expire_hosts-8.1.0-1.fm3_6.el8.noarch
rubygem-foreman_hooks-0.3.17-3.fm3_3.el8.noarch
rubygem-foreman_kubevirt-0.1.9-5.fm3_5.el8.noarch
rubygem-foreman_leapp-0.1.13-1.fm3_6.el8.noarch
rubygem-foreman_maintain-1.3.0-1.el8.noarch
rubygem-foreman_openscap-6.0.0-1.fm3_6.el8.noarch
rubygem-foreman_remote_execution-9.1.0-1.fm3_6.el8.noarch
rubygem-foreman_remote_execution-cockpit-9.1.0-1.fm3_6.el8.noarch
rubygem-foreman_scap_client-0.5.0-1.el8.noarch
rubygem-foreman_setup-8.0.1-2.fm3_3.el8.noarch
rubygem-foreman_snapshot_management-2.0.3-1.fm3_6.el8.noarch
rubygem-foreman_statistics-2.0.1-4.fm3_5.el8.noarch
rubygem-foreman_templates-9.3.0-2.fm3_5.el8.noarch
rubygem-foreman_virt_who_configure-0.5.13-1.fm3_6.el8.noarch
rubygem-foreman_webhooks-3.0.5-1.fm3_5.el8.noarch
rubygem-hammer_cli-3.6.0-1.el8.noarch
rubygem-hammer_cli_foreman-3.6.0-1.el8.noarch
rubygem-hammer_cli_foreman_ansible-0.5.0-1.fm3_6.el8.noarch
rubygem-hammer_cli_foreman_azure_rm-0.2.3-1.fm3_6.el8.noarch
rubygem-hammer_cli_foreman_discovery-1.1.0-1.fm3_3.el8.noarch
rubygem-hammer_cli_foreman_host_reports-0.1.0-1.fm3_3.el8.noarch
rubygem-hammer_cli_foreman_kubevirt-0.1.5-1.fm3_1.el8.noarch
rubygem-hammer_cli_foreman_openscap-0.1.13-2.fm3_5.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
rubygem-hammer_cli_foreman_ssh-0.0.3-1.fm3_5.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.18-1.fm3_5.el8.noarch
rubygem-hammer_cli_foreman_templates-0.2.0-3.fm3_5.el8.noarch
rubygem-hammer_cli_foreman_virt_who_configure-0.0.9-2.fm3_5.el8.noarch
rubygem-hammer_cli_foreman_webhooks-0.0.4-1.fm3_5.el8.noarch
rubygem-hammer_cli_katello-1.8.1-1.el8.noarch
rubygem-katello-4.8.1-1.el8.noarch
rubygem-pulp_ansible_client-0.16.0-1.el8.noarch
rubygem-pulp_certguard_client-1.5.7-1.el8.noarch
rubygem-pulp_container_client-2.14.3-1.el8.noarch
rubygem-pulp_deb_client-2.20.2-1.el8.noarch
rubygem-pulp_file_client-1.12.0-1.el8.noarch
rubygem-pulp_ostree_client-2.0.0-0.1.a1.el8.noarch
rubygem-pulp_python_client-3.8.0-1.el8.noarch
rubygem-pulp_rpm_client-3.19.0-1.el8.noarch
rubygem-pulpcore_client-3.22.2-1.el8.noarch
rubygem-qpid_proton-0.37.0-1.el8.x86_64
rubygem-smart_proxy_pulp-3.2.0-3.fm3_3.el8.noarch

Distribution and version:

cat /etc/os-release 
NAME="Rocky Linux"
VERSION="8.8 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.8 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2029-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.8"

Other relevant data:

2

I’m missing some details here. What was the request that you sent? Could you please provide the complete output of the production.log? (since the beginning of the request)

4 
2023-09-21T14:44:27 [I|app|5ac84fd3] Started GET "/notification_recipients" for 1.2.3.4 at 2023-09-21 14:44:27 +0200
2023-09-21T14:44:27 [I|app|5ac84fd3] Processing by NotificationRecipientsController#index as JSON
2023-09-21T14:44:27 [W|app|5ac84fd3] Action failed
2023-09-21T14:44:27 [I|app|5ac84fd3] Backtrace for 'Action failed' error (TypeError): incompatible marshal file format (can't be read)
 5ac84fd3 |     format version 4.8 required; 123.34 given
 5ac84fd3 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/core_ext/marshal.rb:8:in `load'
 5ac84fd3 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/core_ext/marshal.rb:8:in `load'
 5ac84fd3 | /usr/share/gems/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:598:in `deserialize_entry'
5

exactly same trouble as incompatible marshal file format (format version 4.8 required; 60.33 given) · Issue #1589 · rubygems/rubygems · GitHub

6 
# gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 3.1.6
  - RUBY VERSION: 2.7.8 (2023-03-30 patchlevel 225) [x86_64-linux]
  - INSTALLATION DIRECTORY: /usr/share/gems
  - USER INSTALLATION DIRECTORY: /root/.gem/ruby
  - RUBY EXECUTABLE: /usr/bin/ruby
  - GIT EXECUTABLE: /bin/git
  - EXECUTABLE DIRECTORY: /usr/bin
  - SPEC CACHE DIRECTORY: /root/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /usr/share/gems
     - /root/.gem/ruby
     - /usr/local/share/gems
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
     - "gem" => "--install-dir=/usr/local/share/gems --bindir /usr/local/bin"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /usr/local/sbin
     - /sbin
     - /bin
     - /usr/sbin
     - /usr/bin
     - /opt/puppetlabs/bin
     - /root/bin
7

Hi,

I had the same issue, slightly different error messages; but mail reports stopped working at a time we had upgraded to RHEL 8.9. I did a rollback for rubygem-json package and it started to working again.