I’m trying to get telemetry working, but there seems to be an issue somewhere.
The statsd exporter is started with listening on UDP 8111:
statsd_exporter -statsd.listen-udp :8111
And Foreman is configured to report to it as well:
:telemetry:
:enabled: true
:logger:
:enabled: true
:level: 'DEBUG'
:prometheus: false
:type: 'statsd'
:prefix: 'fm_rails'
:statsd:
:host: '127.0.0.1:8111'
:protocol: 'statsd'
Is the configuration correct, if so what else could be interfering?
1 Like
lzap
April 6, 2018, 7:53am
#2
Hey Seb! I don’t see statsd enabled in the YAML snippet, you are missing the second line “enabled: true” here, it defaults to false.
:statsd:
:enabled: true
:host: '127.0.0.1:8111'
:protocol: 'statsd'
Is this missing from the example yaml template? I believe not (haven’t checked tho).
1 Like
lzap
April 6, 2018, 7:59am
#3
Also “type” is extraneous, you had to copy paste some WIP version or something. Here is the final version:
# Foreman telemetry has three destinations: prometheus, statsd and rails log.
:telemetry:
# prefix for all metrics
:prefix: 'fm_rails'
# prometheus endpoint is at /metrics
# warning: ruby client library currently does not supprt multi-process web servers
:prometheus:
:enabled: false
# works with statsd_exporter too, use the rake task to generate config
:statsd:
:enabled: false
# IP and port (do not use DNS)
:host: '127.0.0.1:8125'
# one of 'statsd', 'statsite' or 'datadog'
:protocol: 'statsd'
# Rails logs end up in logger named 'telemetry' when enabled
:logger:
:enabled: false
# logging level as in Logger::LEVEL
:level: 'DEBUG'
Thank you! I did indeed copy from a WIP snippet of the PR.
That did activate telemetry, but now Foreman throws 500s
cannot load such file -- statsd-instrument (LoadError)
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
/usr/share/foreman/lib/foreman/telemetry_sinks/statsd_sink.rb:4:in `initialize'
/usr/share/foreman/lib/foreman/telemetry.rb:23:in `new'
/usr/share/foreman/lib/foreman/telemetry.rb:23:in `setup'
/usr/share/foreman/config/initializers/5_telemetry.rb:4:in `<top (required)>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:286:in `load'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:286:in `block in load'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:286:in `load'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/engine.rb:655:in `block in load_config_initializer'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/engine.rb:654:in `load_config_initializer'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/engine.rb:612:in `block (2 levels) in <class:Engine>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/engine.rb:611:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/engine.rb:611:in `block in <class:Engine>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `instance_exec'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `run'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:59:in `block in run_initializers'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:421:in `block in each_strongly_connected_component_from'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:48:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:48:in `tsort_each_child'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:415:in `call'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:415:in `each_strongly_connected_component_from'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:347:in `each'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:347:in `call'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:347:in `each_strongly_connected_component'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:226:in `tsort_each'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:205:in `tsort_each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:58:in `run_initializers'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/application.rb:353:in `initialize!'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `public_send'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `method_missing'
/usr/share/foreman/config/environment.rb:5:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
config.ru:5:in `block in <main>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/builder.rb:55:in `instance_eval'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/builder.rb:55:in `initialize'
config.ru:1:in `new'
config.ru:1:in `<main>'
/usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `eval'
/usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `preload_app'
/usr/share/passenger/helper-scripts/rack-preloader.rb:158:in `<module:App>'
/usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<module:PhusionPassenger>'
/usr/share/passenger/helper-scripts/rack-preloader.rb:28:in `<main>'
I’m running Foreman nightly packages.
ekohl
April 6, 2018, 10:45am
#5
Very recently I fixed packaging around this. Since yesterday should have the foreman-telemetry package available that installs the needed dependencies.
It would be good if the application simply refused to start if the dependency is unavailable but needed according the config. I’d encourage rails devs to look into this
2 Likes
Once nightly packages are ok again I’ll give it a try!
atm. yum complains about tfm-rubygem(dynflow) < 0.9.0 not being available.
ekohl
April 6, 2018, 11:09am
#7
lzap
April 6, 2018, 11:15am
#8
I will prepare a patch - if dependency is not found, telemetry stack will issue a warning and turn off telemetry.
lzap
April 6, 2018, 11:16am
#9
You can simply:
scl enable tfm -- "gem install statsd-instrument"
Then it will work.
Sucess! \o/ Telemtry is reported to statsd_exporter and Prometheus is collecting them.
Maybe one thing to note is that the statsd_exporter's default port seems to be 9125 rather then 8125
lzap
April 9, 2018, 8:43am
#11
Yeah statsd is just “defacto standard” and it seems to have 8125 as the port while prometheus tends to put everything into 9xxx “space”. We can change our example yaml to reflect that.
