Which logging files to send to ELK

Development
1

Hello,

I am working on documentation on how to setup Foreman with journald/rsyslog to send all our RoR logs (with custom fields) into ELK (ElasticSearch basically). I have this WIP configuration:

Now, this only capture logs which are in journal/syslog, but we have some configuration files in Foreman or Katello deployment. Since rsyslog is capable of importing also logs from external files via imfile module, which files should I include in the initial version of the document? I won’t have time to include them all, but I would like to hear what should be the priority.

Smart proxy can be configured to send logs to syslog, so this will work out of box (no extra fields tho).

Candlepin has candlepin.log and couple of others (audit.log, errors.log) as well as tomcat.log, I will likely configure only the first one. Will try to configure candlepin with syslog first, that should work according to docs. If not, imfile is the fallback.

Pulp has multiple files, some components log to syslog, some into files. I need an advice here, struggle here a bit. And then there’s qpid and other stuff related to Pulp.

Bare metal provisioning services TFTP and DHCP log into syslog by default, this will appear in ELK automatically as well.

Dynflow to my knowledge logs via RoR stack, that should go into journal when configured, it used to create some extra log file (dynflow_executor.{log,output}) tho. Also our foreman installer might be useful, but that’s imfile only and with time/date parsing - I will likely leave this to later.

To sum up, I am thinking this:

  • production.log (via journal - with custom/structured fields)
  • proxy.log (via syslog)
  • candlepin.log (via syslog)
  • cron.log (via imfile)
  • db_migrate.log and db_seed.log (via imfile)

Any comments on how you integrate with ELK/ES appreciated.

1 Like
2

I think there’s still /var/log/foreman/dynflow_executor.{log,output} being used by dynflowd that executes Foreman core jobs, it uses RoR log stack only in dev env, where the executor runs in the same process. Another service is smart_proxy_dynflow_core which logs to /var/log/foreman-proxy/smart_proxy_dynflow_core.log, that’s dynflow running in smart proxy. So there are probably good candidates for imfile. @aruzicka please keep me honest here.

3

dynflowd indeed has a couple additional log files:

logfile contents
dynflow_executor.log Output from the process monitoring the executor (see daemons[1])
dynflow_executor.output Logging from Dynflow before the rails environment gets initialized
production.log Everything after rails env is initialized should end up here

For smart-proxy_dynflow_core, everything should end up in /var/log/foreman-proxy/smart_proxy_dynflow_core.log

[1] - https://www.rubydoc.info/gems/daemons

4

@aruzicka do you have example entries by chance of both dynflow files so I can create imfile importer? I need just to see few lines to align timedate

5

Sure, take a look at the attachment
dynflow_executor.tar.gz (4.9 KB)

1 Like