A recent Redmine issue raised the question of why does Katello have its own
custom logger. This got me thinking about whether Katello maintaining this
difference from the core of Foreman still made sense today as to try to
align Katello closer.
The original custom logging comes from using the logging gem [1] and
creating a set of configurable loggers. For example, a logger for REST
calls to Pulp, one for Candlepin, one for logging SQL output or our Dynflow
actions. This provided a way to turn on or off specific logging for debug
purposes rather than simply dumping everything to the Rails debug logger.
The logging gem also provides configuration for things like log rotation,
colorization, formatting, etc (see [2] for our current setup). A few
thoughts I had on this:
Remove logging gem, and just use Rails logger for parity with Foreman
and other plugins
Add logging gem to Foreman proper and allow plugins to create different
loggers via configuration file; this could mean that plugins are able to
have their own loggers, and potentially denote this within the logs. For
example,
2015-04-23 07:45:35 [D] [Katello] Repository was created
+1 remove from katello. Loggging is a core feature, no need for plugins
to care about it.
+1 to open an RFE which makes ELK integration easier for core. Lets not
get all fancy on logging configuration when there are good ecosystems
being built around that.
– bk
···
On 04/23/2015 08:24 AM, Eric D Helms wrote:
> All,
>
> A recent Redmine issue raised the question of why does Katello have its
> own custom logger. This got me thinking about whether Katello
> maintaining this difference from the core of Foreman still made sense
> today as to try to align Katello closer.
>
> The original custom logging comes from using the logging gem [1] and
> creating a set of configurable loggers. For example, a logger for REST
> calls to Pulp, one for Candlepin, one for logging SQL output or our
> Dynflow actions. This provided a way to turn on or off specific logging
> for debug purposes rather than simply dumping everything to the Rails
> debug logger. The logging gem also provides configuration for things
> like log rotation, colorization, formatting, etc (see [2] for our
> current setup). A few thoughts I had on this:
>
>
> 1) Remove logging gem, and just use Rails logger for parity with Foreman
> and other plugins
> 2) Add logging gem to Foreman proper and allow plugins to create
> different loggers via configuration file; this could mean that plugins
> are able to have their own loggers, and potentially denote this within
> the logs. For example,
>
> 2015-04-23 07:45:35 [D] [Katello] Repository was created
>
> 3) Do nothing
>
>
> Thoughts?
>
> Eric
>
>
>
> [1] https://rubygems.org/search?query=logging
> [2]
> https://github.com/Katello/katello/blob/master/config/katello_defaults.yml#L98-L169
>
> --
> You received this message because you are subscribed to the Google
> Groups "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to foreman-dev+unsubscribe@googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
> All,
>
> A recent Redmine issue raised the question of why does Katello have its own
> custom logger. This got me thinking about whether Katello maintaining this
> difference from the core of Foreman still made sense today as to try to
> align Katello closer.
>
> The original custom logging comes from using the logging gem [1] and
> creating a set of configurable loggers. For example, a logger for REST
> calls to Pulp, one for Candlepin, one for logging SQL output or our Dynflow
> actions. This provided a way to turn on or off specific logging for debug
> purposes rather than simply dumping everything to the Rails debug logger.
> The logging gem also provides configuration for things like log rotation,
> colorization, formatting, etc (see [2] for our current setup). A few
> thoughts I had on this:
>
>
> 1) Remove logging gem, and just use Rails logger for parity with Foreman
> and other plugins
> 2) Add logging gem to Foreman proper and allow plugins to create different
> loggers via configuration file; this could mean that plugins are able to
> have their own loggers, and potentially denote this within the logs. For
> example,
>
> 2015-04-23 07:45:35 [D] [Katello] Repository was created
Probably way too naive but can't plugins simply do:
plugin_logger = Logger.new(app.root.join('log', "tracking.log"))
config.I18n.default_locale = :en
plugin_logger.debug("plugin just set its custom locale to :en\n\n")
in the engine.rb initializers?
I'd vote for 1) unless absolutely necessary, what areas absolutely need
Logging because the Rails logger does not suffice?
> 1) Remove logging gem, and just use Rails logger for parity with Foreman
> and other plugins
> 2) Add logging gem to Foreman proper and allow plugins to create different
> loggers via configuration file; this could mean that plugins are able to
> have their own loggers, and potentially denote this within the logs. For
> example,
>
> 2015-04-23 07:45:35 [D] [Katello] Repository was created
>
> 3) Do nothing
Rails logger is way too limited and simple, we should adopt anything
that is:
fine grained logging level names (per core component, per plugin)
has TRACE level (on top of DEBUG - very useful for debugging specific
stuff)
is packaged in Ruby SCL already
is backward-compatible with Ruby/Rails logging service
works at least with syslog (journald structured logging being plus)
Not sure if logging gem fulfills this, but both Rails and built-in Ruby
loggers are very limited. Having a decent logging solution is important
for us and I found myself several times praying for improvement in this
area.
Therefore I'd suggest to adopt Katello approach in Core, if it fulfills
items above which I think are important. If not, let's design logging
NG.
If we are going to touch this domain, please design everything with
journald support in mind. Specifically, journald offers structured log
input, which can be quite useful (for example application can be
configured to send DEBUG or TRACE output of a level name into journald
while administrator can still effectively work with the output and
constraint output using level or level names).
This also means we must avoid referencing files totally. If you see
"log/" + "something.log" anywhere, it smells and must go away in the NG
version.
(By "level name" I mean those "named loggers" we can leverage for each
plugin or core component. Not sure if this is the right term.)
> > All,
> >
> > A recent Redmine issue raised the question of why does Katello have its
> own
> > custom logger. This got me thinking about whether Katello maintaining
> this
> > difference from the core of Foreman still made sense today as to try to
> > align Katello closer.
> >
> > The original custom logging comes from using the logging gem [1] and
> > creating a set of configurable loggers. For example, a logger for REST
> > calls to Pulp, one for Candlepin, one for logging SQL output or our
> Dynflow
> > actions. This provided a way to turn on or off specific logging for debug
> > purposes rather than simply dumping everything to the Rails debug logger.
> > The logging gem also provides configuration for things like log rotation,
> > colorization, formatting, etc (see [2] for our current setup). A few
> > thoughts I had on this:
> >
> >
> > 1) Remove logging gem, and just use Rails logger for parity with Foreman
> > and other plugins
> > 2) Add logging gem to Foreman proper and allow plugins to create
> different
> > loggers via configuration file; this could mean that plugins are able to
> > have their own loggers, and potentially denote this within the logs. For
> > example,
> >
> > 2015-04-23 07:45:35 [D] [Katello] Repository was created
>
> Probably way too naive but can't plugins simply do:
>
> plugin_logger = Logger.new(app.root.join('log', "tracking.log"))
> config.I18n.default_locale = :en
> plugin_logger.debug("plugin just set its custom locale to :en\n\n")
>
> in the engine.rb initializers?
>
I suppose so, although then every plugin is logging to it's own separate
file and generating a timeline is not as easy? And there again, a custom
logger is being used. Whereas I think the logging gem lets you create
different configurations and switches on/off but a central log.
>
> I'd vote for 1) unless absolutely necessary, what areas absolutely need
> Logging because the Rails logger does not suffice?
>
I don't this was originally about whether the logger sufficed so much as
the amount of information that would be put into the log per call. For
example, you don't need to see the Pulp REST calls we make in all cases.
However, when you are trying to debug an issue, being able to flip that
switch to start seeing them can be nice. If we switch to pure Rails logger
for everything, this means you will see Pulp, Candlepin, Elasticsearch, and
Dynflow Actions throughout when in debug mode. I am fine with that
personally, just want to make sure the implications of the change are
understood.
Eric
···
On Thu, Apr 23, 2015 at 10:15 AM, Daniel Lobato Garcia wrote:
> On 04/23, Eric D Helms wrote:
>> All,
>>
>> A recent Redmine issue raised the question of why does Katello have its own
>> custom logger. This got me thinking about whether Katello maintaining this
>> difference from the core of Foreman still made sense today as to try to
>> align Katello closer.
>>
>> The original custom logging comes from using the logging gem [1] and
>> creating a set of configurable loggers. For example, a logger for REST
>> calls to Pulp, one for Candlepin, one for logging SQL output or our Dynflow
>> actions. This provided a way to turn on or off specific logging for debug
>> purposes rather than simply dumping everything to the Rails debug logger.
>> The logging gem also provides configuration for things like log rotation,
>> colorization, formatting, etc (see [2] for our current setup). A few
>> thoughts I had on this:
>>
>>
>> 1) Remove logging gem, and just use Rails logger for parity with Foreman
>> and other plugins
>> 2) Add logging gem to Foreman proper and allow plugins to create different
>> loggers via configuration file; this could mean that plugins are able to
>> have their own loggers, and potentially denote this within the logs. For
>> example,
>>
>> 2015-04-23 07:45:35 [D] [Katello] Repository was created
> Probably way too naive but can't plugins simply do:
>
> plugin_logger = Logger.new(app.root.join('log', "tracking.log"))
> config.I18n.default_locale = :en
> plugin_logger.debug("plugin just set its custom locale to :en\n\n")
>
> in the engine.rb initializers?
>
> I'd vote for 1) unless absolutely necessary, what areas absolutely need
> Logging because the Rails logger does not suffice?
I would say that the foreman logger is really annoying when you want to
raise the log level of one aspect but not another.
As i understand it today, the only way to raise logging is to change:
config.log_level = :debug
in production.rb.
By doing this it opens up the firehose of logging all sql queries along
with any app code debugging. Sure I can log those to another file via:
But this isn't well documented, and also brings me to my other issue.
Configuration of the logger via editing application code seems broken, I
would expect this to be set in a configuration file.
-Justin
···
On 04/23/2015 10:15 AM, Daniel Lobato Garcia wrote:
> On 04/23, Eric D Helms wrote:
–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
Daniel Lobato Garcia
All valid comments, but I would still consider doing this in the core…
not in a plugin.
– bk
···
On 04/23/2015 10:47 AM, Justin Sherrill wrote:
> I would say that the foreman logger is really annoying when you want to
> raise the log level of one aspect but not another.
>
> As i understand it today, the only way to raise logging is to change:
>
> config.log_level = :debug
>
> in production.rb.
>
> By doing this it opens up the firehose of logging all sql queries along
> with any app code debugging. Sure I can log those to another file via:
>
> config.active_record.logger = Logger.new("log/development_sql.log")
>
> But this isn't well documented, and also brings me to my other issue.
> Configuration of the logger via editing application code seems broken, I
> would expect this to be set in a configuration file.
>
> -Justin
>
> All valid comments, but I would still consider doing this in the core…
> not in a plugin.
>
> – bk
The thing is, logging was in core for katello but now resides external to foreman, the new core.
For myself, I still use the katello logger and its granular config quite often. For example, enabling elasticsearch (tire) api call logging. In fact, one of the things I miss most since foreman merge is the lack of api call logging done against the server itself (eg. running a hammer command and seeing the exact api url being hit with parameters).
My vote would be to bring the katello detailed logging into core foreman with at least as much granular control as in the past. Whether this is done with the existing gem or integrating elk (are they at odds or would elk benefit from improved logging?).
···
----- Original Message -----
On 04/23/2015 10:47 AM, Justin Sherrill wrote:
I would say that the foreman logger is really annoying when you want to
raise the log level of one aspect but not another.
As i understand it today, the only way to raise logging is to change:
config.log_level = :debug
in production.rb.
By doing this it opens up the firehose of logging all sql queries along
with any app code debugging. Sure I can log those to another file via:
But this isn’t well documented, and also brings me to my other issue.
Configuration of the logger via editing application code seems broken, I
would expect this to be set in a configuration file.
We don't require this of any other new dependency, as packaging new gems
is not a problem, provided it isn't a huge dependency tree. "logging"
is already packaged, which may indeed be useful, but that should not
restrict choosing the best solution for Foreman.
···
On 27/04/15 13:35, Lukas Zapletal wrote:
> Rails logger is *way* too limited and simple, we should adopt anything
> that is:
>
> - is packaged in Ruby SCL already
> > - is packaged in Ruby SCL already
>
> We don't require this of any other new dependency, as packaging new gems
> is not a problem, provided it isn't a huge dependency tree. "logging"
> is already packaged, which may indeed be useful, but that should not
> restrict choosing the best solution for Foreman.
>
>
> > All valid comments, but I would still consider doing this in the core…
> > not in a plugin.
> >
> > – bk
>
> The thing is, logging was in core for katello but now resides external to
> foreman, the new core.
>
> For myself, I still use the katello logger and its granular config quite
> often. For example, enabling elasticsearch (tire) api call logging. In fact,
> one of the things I miss most since foreman merge is the lack of api call
> logging done against the server itself (eg. running a hammer command and
> seeing the exact api url being hit with parameters).
>
> My vote would be to bring the katello detailed logging into core foreman with
> at least as much granular control as in the past. Whether this is done with
> the existing gem or integrating elk (are they at odds or would elk benefit
> from improved logging?).
+1. Rails logger is too basic for using in a platform-like application
that the Foreman is becoming. Just creating a separate logger for every plugin
does not help much, as every plugin then needs it's own configuration for it,
and sooner or later we would end up with writting our own Logging like functionality.
I like the Logging gem a lot, as it's nicely designed and has an active community around it.
As Tom mentioned, the Foreman itself would deserve revisiting the way we use the
logger to provide more friendly output for the developer (and since there is so much
things happening while performing a request, being able to filter the logs
based on the area I'm interested is readlly valuable)
– Ivan
···
----- Original Message -----
> ----- Original Message -----
On 04/23/2015 10:47 AM, Justin Sherrill wrote:
I would say that the foreman logger is really annoying when you want to
raise the log level of one aspect but not another.
As i understand it today, the only way to raise logging is to change:
config.log_level = :debug
in production.rb.
By doing this it opens up the firehose of logging all sql queries along
with any app code debugging. Sure I can log those to another file via:
But this isn’t well documented, and also brings me to my other issue.
Configuration of the logger via editing application code seems broken, I
would expect this to be set in a configuration file.
>
> >
> > > All valid comments, but I would still consider doing this in the core…
> > > not in a plugin.
> > >
> > > – bk
> >
> > The thing is, logging was in core for katello but now resides external
> > to
> > foreman, the new core.
> >
> > For myself, I still use the katello logger and its granular config quite
> > often. For example, enabling elasticsearch (tire) api call logging. In
> > fact, one of the things I miss most since foreman merge is the lack of
> > api call logging done against the server itself (eg. running a hammer
> > command and seeing the exact api url being hit with parameters).
> >
> > My vote would be to bring the katello detailed logging into core foreman
> > with at least as much granular control as in the past. Whether this is
> > done with the existing gem or integrating elk (are they at odds or would
> > elk benefit from improved logging?).
>
> +1. Rails logger is too basic for using in a platform-like application
> that the Foreman is becoming. Just creating a separate logger for every
> plugin does not help much, as every plugin then needs it's own
> configuration for it, and sooner or later we would end up with writting our
> own Logging like functionality.
>
> I like the Logging gem a lot, as it's nicely designed and has an active
> community around it.
>
> As Tom mentioned, the Foreman itself would deserve revisiting the way we use
> the logger to provide more friendly output for the developer (and since
> there is so much things happening while performing a request, being able to
> filter the logs based on the area I'm interested is readlly valuable)
>
> – Ivan
>
> > > > I would say that the foreman logger is really annoying when you want
> > > > to
> > > > raise the log level of one aspect but not another.
> > > >
> > > > As i understand it today, the only way to raise logging is to change:
> > > >
> > > > config.log_level = :debug
> > > >
> > > > in production.rb.
> > > >
> > > > By doing this it opens up the firehose of logging all sql queries
> > > > along
> > > >
> > > > with any app code debugging. Sure I can log those to another file
via:
> > > > config.active_record.logger = Logger.new("log/development_sql.log")
> > > >
> > > > But this isn't well documented, and also brings me to my other issue.
> > > > Configuration of the logger via editing application code seems broken,
> > > > I
> > > > would expect this to be set in a configuration file.
> > > >
> > > > -Justin
I might be slightly biased as I added the logging to Katello but anyway, +1
for revisiting Foreman logger and move logging from Katello to Foreman core.
IIRC it took me few days to integrate logging there, in Foreman it might be
even smaller task (we already have packages, configuration layer is already
implemented etc).
Benefits I miss in Foreman: more appenders (combined log and separate logs per
service at the same time), rolling files built-in, better control of messages
formatting (colors per levels for custom messages), different message
structure per environment (thread ID, timestamp in production as part of
message, while only log level in development).
I don't say we can't improve existing logger but why if we already have all of
this, proven itself for quite long time in Foreman plugin?
···
On Friday 24 of April 2015 06:05:28 Ivan Necas wrote:
> ----- Original Message -----
> > ----- Original Message -----
> > > On 04/23/2015 10:47 AM, Justin Sherrill wrote:
Per the rough outcome of this discussion, I have opened a PR to add the
rubygem 'logging' with support for the base logger and a sql logger. This
PR also includes support to allow plugins to create their own loggers via
their settings files [1].
···
On Tue, Apr 28, 2015 at 8:02 AM, Marek Hulan wrote:
On Friday 24 of April 2015 06:05:28 Ivan Necas wrote:
----- Original Message -----
----- Original Message -----
All valid comments, but I would still consider doing this in the
core…
not in a plugin.
– bk
The thing is, logging was in core for katello but now resides
external
to
foreman, the new core.
For myself, I still use the katello logger and its granular config
quite
often. For example, enabling elasticsearch (tire) api call logging. In
fact, one of the things I miss most since foreman merge is the lack of
api call logging done against the server itself (eg. running a hammer
command and seeing the exact api url being hit with parameters).
My vote would be to bring the katello detailed logging into core
foreman
with at least as much granular control as in the past. Whether this is
done with the existing gem or integrating elk (are they at odds or
would
elk benefit from improved logging?).
+1. Rails logger is too basic for using in a platform-like application
that the Foreman is becoming. Just creating a separate logger for every
plugin does not help much, as every plugin then needs it’s own
configuration for it, and sooner or later we would end up with writting
our
own Logging like functionality.
I like the Logging gem a lot, as it’s nicely designed and has an active
community around it.
As Tom mentioned, the Foreman itself would deserve revisiting the way we
use
the logger to provide more friendly output for the developer (and since
there is so much things happening while performing a request, being able
to
filter the logs based on the area I’m interested is readlly valuable)
– Ivan
On 04/23/2015 10:47 AM, Justin Sherrill wrote:
I would say that the foreman logger is really annoying when you
want
to
raise the log level of one aspect but not another.
As i understand it today, the only way to raise logging is to
change:
config.log_level = :debug
in production.rb.
By doing this it opens up the firehose of logging all sql queries
along
with any app code debugging. Sure I can log those to another file
via:
But this isn’t well documented, and also brings me to my other
issue.
Configuration of the logger via editing application code seems
broken,
I
would expect this to be set in a configuration file.
-Justin
I might be slightly biased as I added the logging to Katello but anyway, +1
for revisiting Foreman logger and move logging from Katello to Foreman
core.
IIRC it took me few days to integrate logging there, in Foreman it might be
even smaller task (we already have packages, configuration layer is already
implemented etc).
Benefits I miss in Foreman: more appenders (combined log and separate logs
per
service at the same time), rolling files built-in, better control of
messages
formatting (colors per levels for custom messages), different message
structure per environment (thread ID, timestamp in production as part of
message, while only log level in development).
I don’t say we can’t improve existing logger but why if we already have
all of
this, proven itself for quite long time in Foreman plugin?
