RFC: File audit for template rendering


#1
Hello,

Foreman does have a nice audit trail for many operations, what's missing is ability to find how a template (e.g. kickstart) was rendered. Storing whole template text in audit table is probably not the best thing to do, production.log is also not a good fit, so here is a proposal.

I want to create a small API called File audit (*) with ability to store arbitrary files into
/var/log/foreman/file_audit/id/text-timestamp where "id" is record id (or multiple ids concatenated with dots), "text" is arbitrary alphanum text and "timestamp" is unix epoch number. Example:

/var/log/foreman/file_audit/1.33.7/my.host.lan-pxelinux-1512492603

That API will be used to store contents of all templates rendered, so users can easily go "back in time" and see how templates are being rendered. The directory would be root-only reading and files will be created with restricted permissions (foreman user rw only). On system with SELinux, security would be more tightened allowing writing only to foreman_t domain and reading to nobody. For the example above, this would mean:

1.33.7/my.host.lan-pxelinux-1512492603

1 - file audit type (static list, 1 stands for "template audit entry") 33 - host id
7 - template id
my.host.lan-pxelinux - extra data so users can work and search from command line 1512492603 - UNIX epoch timestamp

Everytime new record is added, a log entry is created into
production.log containing file path. By default, there will be a cron job deleting all files older than one month. In documentation, we will ask users to rsync the directory to different location if long-term archival is needed.

This API could be used for other audit logging, for example when user uploads a manifest ZIP file in katello or new version of RPM/Puppet file. This will be the first step to improve audit around templates, later on we can create a plugin showing the data in audit/host pages if needed. But in the first phase, administrators could easily search/grep/diff those files when necessary.

(*) if you have a better name, please do propose


··· --
Later,
  Lukas @lzap Zapletal

#2
I don't think, this is a good idea. Storing files on the local filesystem is generally consideres an anti-pattern imho. Both for web applications and expecially in a container world.

In a clustered setup, you'd have to attach some kind of shared-storage to the foreman hosts to make this work. Possibly via NFS or something. This is meant to cause trouble. I generally advise not to have any state on an application server.

I think, this can and should be done with a database. I agree, that the audit model might not be the best place for this. But a database definitely is.

Just my 2 cents.

- Timo


··· Am 05.12.17 um 18:01 schrieb Lukas Zapletal:
Hello,

Foreman does have a nice audit trail for many operations, what's
missing is ability to find how a template (e.g. kickstart) was
rendered. Storing whole template text in audit table is probably not
the best thing to do, production.log is also not a good fit, so here
is a proposal.

I want to create a small API called File audit (*) with ability to
store arbitrary files into
/var/log/foreman/file_audit/id/text-timestamp where "id" is record id
(or multiple ids concatenated with dots), "text" is arbitrary alphanum
text and "timestamp" is unix epoch number. Example:

/var/log/foreman/file_audit/1.33.7/my.host.lan-pxelinux-1512492603

That API will be used to store contents of all templates rendered, so
users can easily go "back in time" and see how templates are being
rendered. The directory would be root-only reading and files will be
created with restricted permissions (foreman user rw only). On system
with SELinux, security would be more tightened allowing writing only
to foreman_t domain and reading to nobody. For the example above, this
would mean:

1.33.7/my.host.lan-pxelinux-1512492603

1 - file audit type (static list, 1 stands for "template audit entry")
33 - host id
7 - template id
my.host.lan-pxelinux - extra data so users can work and search from command line
1512492603 - UNIX epoch timestamp

Everytime new record is added, a log entry is created into
production.log containing file path. By default, there will be a cron
job deleting all files older than one month. In documentation, we will
ask users to rsync the directory to different location if long-term
archival is needed.

This API could be used for other audit logging, for example when user
uploads a manifest ZIP file in katello or new version of RPM/Puppet
file. This will be the first step to improve audit around templates,
later on we can create a plugin showing the data in audit/host pages
if needed. But in the first phase, administrators could easily
search/grep/diff those files when necessary.

(*) if you have a better name, please do propose


#3
I like this.

I agree the renderred template should not be stored along audit records we have today, it does not represent any data change. Also it would be a lot of data in our db and would make searching of other audit records hard. Having long kickstart files in log files is neither a good idea.

One small comment, Foreman must remain operational if directory can't be written to and a directory must be configurable (developer setups). The only downside is related to HA Foreman setups. Users must share the dir among all Foreman instances.

We should probably limit this to provisioning templates and partition tables only in terms of templates rendering. Job templates can grow rapidly, thousands of files per one invocation. The job template result is different for each host.

Btw this would be also great for host classification auditing. That answers the question, what parameter values were available for the host when ENC/info method was called during provisioning.

It'd also be great if foreman-debug would be gathering this, but I suppose that's already on your radar.


··· --
Marek

On December 5, 2017 6:02:14 PM Lukas Zapletal <lzap@redhat.com> wrote:

Hello,

Foreman does have a nice audit trail for many operations, what's
missing is ability to find how a template (e.g. kickstart) was
rendered. Storing whole template text in audit table is probably not
the best thing to do, production.log is also not a good fit, so here
is a proposal.

I want to create a small API called File audit (*) with ability to
store arbitrary files into
/var/log/foreman/file_audit/id/text-timestamp where "id" is record id
(or multiple ids concatenated with dots), "text" is arbitrary alphanum
text and "timestamp" is unix epoch number. Example:

/var/log/foreman/file_audit/1.33.7/my.host.lan-pxelinux-1512492603

That API will be used to store contents of all templates rendered, so
users can easily go "back in time" and see how templates are being
rendered. The directory would be root-only reading and files will be
created with restricted permissions (foreman user rw only). On system
with SELinux, security would be more tightened allowing writing only
to foreman_t domain and reading to nobody. For the example above, this
would mean:

1.33.7/my.host.lan-pxelinux-1512492603

1 - file audit type (static list, 1 stands for "template audit entry")
33 - host id
7 - template id
my.host.lan-pxelinux - extra data so users can work and search from command
line
1512492603 - UNIX epoch timestamp

Everytime new record is added, a log entry is created into
production.log containing file path. By default, there will be a cron
job deleting all files older than one month. In documentation, we will
ask users to rsync the directory to different location if long-term
archival is needed.

This API could be used for other audit logging, for example when user
uploads a manifest ZIP file in katello or new version of RPM/Puppet
file. This will be the first step to improve audit around templates,
later on we can create a plugin showing the data in audit/host pages
if needed. But in the first phase, administrators could easily
search/grep/diff those files when necessary.

(*) if you have a better name, please do propose

--
Later,
Lukas @lzap Zapletal

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

#4
You are right that in general, this is an anti-pattern, but the data we will be sending is write only logging data. It's essentially a log sink, but for arbitrary data (blobs, files, name it you want). In clustered setup, you simply keep the logs on nodes along with other logging data. On the other hand, we aim at using syslog for everything in the future, this could be additional thing to keep in mind.

We also discussed storing this in database schema in a formal way (new table rendered_template with references to host/template_kind) but arguments against are this is already quite complex part of our schema, keeping history could easily grow the database and performance could be problem for Remote Execution when performing large-scale script runs. We ended up discussing actually removing audit trail data from database, but this turned to be problem as we need to keep reference to taxonomy, users and records.

This brings me to alternative proposal:

I plan to work on full syslog support for Rails and Smart Proxy in January next year. The goal is to achieve structured logging using some standard format (CEE/CEF). We would send those rendered templates to Rails log (syslog) and there could be an optional syslog module to extract those log entries and put them into similar directory structure. This could be the default setup for small deployments, but for large-scale we will recommend to send logs over network to central logging point anyway, so the module could be installed there or depending on setup this could be loaded into ELK stack. (*)

In short, let's send the rendered templates to syslog, it deals with multiple lines just fine, we can simply filter these from
production.log and put them into a separate file or even directory structure.

(*) http://www.rsyslog.com/writing-rsyslog-plugins-in-2-minutes/

··· On Tue, Dec 5, 2017 at 6:34 PM, Timo Goebel <mail@timogoebel.name> wrote:
I don't think, this is a good idea. Storing files on the local filesystem is
generally consideres an anti-pattern imho. Both for web applications and
expecially in a container world.

In a clustered setup, you'd have to attach some kind of shared-storage to
the foreman hosts to make this work. Possibly via NFS or something. This is
meant to cause trouble. I generally advise not to have any state on an
application server.

I think, this can and should be done with a database. I agree, that the
audit model might not be the best place for this. But a database definitely
is.

Just my 2 cents.

- Timo

Am 05.12.17 um 18:01 schrieb Lukas Zapletal:

Hello,

Foreman does have a nice audit trail for many operations, what's
missing is ability to find how a template (e.g. kickstart) was
rendered. Storing whole template text in audit table is probably not
the best thing to do, production.log is also not a good fit, so here
is a proposal.

I want to create a small API called File audit (*) with ability to
store arbitrary files into
/var/log/foreman/file_audit/id/text-timestamp where "id" is record id
(or multiple ids concatenated with dots), "text" is arbitrary alphanum
text and "timestamp" is unix epoch number. Example:

/var/log/foreman/file_audit/1.33.7/my.host.lan-pxelinux-1512492603

That API will be used to store contents of all templates rendered, so
users can easily go "back in time" and see how templates are being
rendered. The directory would be root-only reading and files will be
created with restricted permissions (foreman user rw only). On system
with SELinux, security would be more tightened allowing writing only
to foreman_t domain and reading to nobody. For the example above, this
would mean:

1.33.7/my.host.lan-pxelinux-1512492603

1 - file audit type (static list, 1 stands for "template audit entry")
33 - host id
7 - template id
my.host.lan-pxelinux - extra data so users can work and search from
command line
1512492603 - UNIX epoch timestamp

Everytime new record is added, a log entry is created into
production.log containing file path. By default, there will be a cron
job deleting all files older than one month. In documentation, we will
ask users to rsync the directory to different location if long-term
archival is needed.

This API could be used for other audit logging, for example when user
uploads a manifest ZIP file in katello or new version of RPM/Puppet
file. This will be the first step to improve audit around templates,
later on we can create a plugin showing the data in audit/host pages
if needed. But in the first phase, administrators could easily
search/grep/diff those files when necessary.

(*) if you have a better name, please do propose

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Later,
  Lukas @lzap Zapletal

#5

Sorry for the late reply.
I like the solution to treat this as any other log file. However, I’m not sure, if syslog is the way to go. The 12-factor app just logs to STDOUT. That also helps to view the logs with journald.
We have quite a large ELK setup for our logs. The best solution has always been to output the logs directly in JSON as you don’t need to transform the data (which can be quite expensive, depending on the number of logs you need to process).


#6

I haven’t fully started on the design, but so far it looks like that syslog
is the only way that works for all backend systems (pulp and candlepin). I
mean local or UDP syslog endpoints.

That does not mean that Foreman must use it, on RHEL7 there is actually
both syslog and journald running, the latter has only small volatile buffer
forwarding everything to syslog. The key thing is that both services are
usually present, so we can use both as they can be configured both ways
(syslog->journald or journald->syslog). So I expect that we can leverage
journald structured logging in Foreman core app and have backend systems to
log into syslog in the first phase (unstructured). The correlation id,
which is being passed into logs by Candlepin today (!) can be extracted at
later stage.

Can you elaborate on the ELK stack you have? What you mean by JSON logging,
I don’t exactly understand how JSON can be used for that. Can you share
your vision how common structured logging should be implemented in Foreman?


#7

It seems to me it might be more useful to track these templates in a git repository. Would there be an easy way to wrap git for this? This would allow you version control, history and auditing without having to do too much work.


#8

Hello, thanks for the feedback. It is good idea, the audit template logger
exporter could do this as well, or provide a hook script to do any action.