This has been discussed before and presented, however, I'd like to pop this
again and get your comments and insights - especially with the discussion
going on this thread
<https://github.com/theforeman/foreman_openscap/pull/126>
This is our plan to refactor the ArfReport:
New design
···
-
XML file is parsed on the proxy, which tears down all the results as
arf_report
proxy posts arf_report to Foreman
arf_report is a Foreman report subclass
Details:
foreman_scap_client runs on the client and generates XML report (in the
form it is done today) and sends it to the proxy
2.
In the proxy:
1.
Save the XML file locally
2.
extract metrics and extract all the ArfReport results and post them
to Foreman as a complete report (metrics and logs) In Foreman
3.
Proxy sends the ArfReport HTML which is saved as ArfReportRaw (which
currently saves the xml file)
3. In Foreman:
1.
ArfReport < Report
2.
ArfReport has metrics and logs
3.
ArfReport Show in the same manner of Foreman report . Full HTML is
available via ArfReportRaw. XML is available via URL to proxy.
Regarding the discussion in the thread:
There are several pros and cons to stop using Scaptimony as the parsing and
persistence engine: Pros
ArfReport should be a subclass of Report. (so this is too Foreman
specific)
ArfReport parsing should be done on the proxy (so we don’t need the
scaptimony parsing part)
We wish to save ArfReport xml file on the proxy’s file system, today
it is saved on the foreman’s db
We wish to save the parsed report data and it’s html on ArfReport and
ArfReportRaw
Cons
We miss Scaptimony and OpenSCAP team contributions
We may miss changes in OpenSCAP that may be reflected in Scaptimony.
Way in the middle:
Create a gem “openscap_parser” which takes the report xml and returns
it as json (basically what I am trying to do with the proxy plugin). Then
anyone can use it to it’s own purposes: Use it on the proxy to send data to
foreman, use it on another Rails app to persist it in the db. Maybe this
should have been the initial purpose of Scaptimony?
Migrate Scaptimony to foreman_openscap (move models to foreman_openscap)
Process reports on proxy:
Extract metrics
-
Collect data for logs (Parse the xml ??)
-
Send reports from proxy to foreman immediately
On success, move xml file to ‘openscap/reports/<date_digest>.xml
-
Add to queue on failure
-
if sending to Foreman failed, the reports remain in
/var/spool/foreman-proxy/openscap
-
refactor current ‘smart-proxy-openscap-send’ to parse the xml,
send to foreman and move it on success
-
ArfReport < Report (sti)
Migrate existing ArfReport to new ArfReport
-
Copy metrics from arf breakdowns for reports
-
Parse XML and extract data for logs
-
Move XML to be HTML
-
html is saved as ArfReportRaw
-
Client sends full date (today: 2015-08-18, needed: 2015-08-18 12:31, or
better Time.now.to_i)
Packaging
Remove openscap dependency from foreman_openscap
-
Add openscap dependency to smart_proxy_o`penscap
On Monday, August 31, 2015 at 2:26:23 PM UTC+3, Shlomi Zadok wrote:
This has been discussed before and presented, however, I’d like to pop
this again and get your comments and insights - especially with the
discussion going on this thread https://github.com/theforeman/foreman_openscap/pull/126
This is our plan to refactor the ArfReport:
New design
XML file is parsed on the proxy, which tears down all the results as
arf_report
proxy posts arf_report to Foreman
arf_report is a Foreman report subclass
Details:
foreman_scap_client runs on the client and generates XML report (in
the form it is done today) and sends it to the proxy
2.
In the proxy:
1.
Save the XML file locally
2.
extract metrics and extract all the ArfReport results and post them
to Foreman as a complete report (metrics and logs) In Foreman
3.
Proxy sends the ArfReport HTML which is saved as ArfReportRaw
(which currently saves the xml file)
3. In Foreman:
1.
ArfReport < Report
2.
ArfReport has metrics and logs
3.
ArfReport Show in the same manner of Foreman report . Full HTML is
available via ArfReportRaw. XML is available via URL to proxy.
Regarding the discussion in the thread:
There are several pros and cons to stop using Scaptimony as the parsing
and persistence engine: Pros
ArfReport should be a subclass of Report. (so this is too Foreman
specific)
ArfReport parsing should be done on the proxy (so we don’t need the
scaptimony parsing part)
We wish to save ArfReport xml file on the proxy’s file system, today
it is saved on the foreman’s db
We wish to save the parsed report data and it’s html on ArfReport
and ArfReportRaw
Cons
We miss Scaptimony and OpenSCAP team contributions
We may miss changes in OpenSCAP that may be reflected in Scaptimony.
Way in the middle:
Create a gem “openscap_parser” which takes the report xml and
returns it as json (basically what I am trying to do with the proxy
plugin). Then anyone can use it to it’s own purposes: Use it on the proxy
to send data to foreman, use it on another Rails app to persist it in the
db. Maybe this should have been the initial purpose of Scaptimony?
I see additional value of keeping Scaptimony in the stack, it's not just
reports storage and report renderer, there's logic that tells us what hosts
(or any asset in general) are compliant etc. The question is how much of this
logic depends on raw XML parsing and if we can create report breakdowns
without full parsing on Foreman side.
If it's possible to create breakdowns on proxy I think we'd only need to
change some of its internals so when we need to render HTML version of report
it can download the raw XML from proxy, but otherwise it can remain as it is.
Also this should be configurable so Scaptimony would remain generic enough for
other projects.
And maybe the gems could be broken apart to something like scaptimony and
scaptimony_renderer and rendering would be only available if the renderer is
installed. The advantage I see is the HTML rendering would be optional and
therefore foreman_openscap would not (even indirectly) depend on openscap and
ffi gems unless user needs the full HTML report.
On the other hand If we can't split the parsing logic and reuse the scopes for
assets status then I don't see much benefit of having Scaptimony in Foreman
and maybe we should look what we can reuse from it on smart proxy side. I
guess that refers to what you've mentioned in "Way in the middle".
···
On Monday 31 of August 2015 04:37:25 Shlomi Zadok wrote:
> One more thing. User stories:
>
>
> -
>
> Migrate Scaptimony to foreman_openscap (move models to foreman_openscap)
> -
>
> Process reports on proxy:
> -
>
> Extract metrics
> -
>
> Collect data for logs (Parse the xml ??)
> -
>
> Send reports from proxy to foreman immediately
> -
>
> On success, move xml file to ‘openscap/reports/.xml
> -
>
> Add to queue on failure
> -
>
> if sending to Foreman failed, the reports remain in
> /var/spool/foreman-proxy/openscap
> -
>
> refactor current ‘smart-proxy-openscap-send’ to parse the xml,
> send to foreman and move it on success
> -
>
> ArfReport < Report (sti)
> -
>
> Migrate existing ArfReport to new ArfReport
> -
>
> Copy metrics from arf breakdowns for reports
> -
>
> Parse XML and extract data for logs
> -
>
> Move XML to be HTML
> -
>
> html is saved as ArfReportRaw
> -
>
> Client sends full date (today: 2015-08-18, needed: 2015-08-18 12:31, or
> better Time.now.to_i)
> -
>
> Packaging
> -
>
> Remove openscap dependency from foreman_openscap
> -
>
> Add openscap dependency to smart_proxy_o`penscap
>
> On Monday, August 31, 2015 at 2:26:23 PM UTC+3, Shlomi Zadok wrote:
> > This has been discussed before and presented, however, I'd like to pop
> > this again and get your comments and insights - especially with the
> > discussion going on this thread
> >
> > This is our plan to refactor the ArfReport:
> > New design
> >
> > -
> >
> > XML file is parsed on the proxy, which tears down all the results as
> > arf_report
> > -
> >
> > proxy posts arf_report to Foreman
> > -
> >
> > arf_report is a Foreman report subclass
> >
> > Details:
> > 1.
> >
> > foreman_scap_client runs on the client and generates XML report (in
> > the form it is done today) and sends it to the proxy
> > 2.
> >
> > In the proxy:
> > 1.
> >
> > Save the XML file locally
> > 2.
> >
> > extract metrics and extract all the ArfReport results and post them
> > to Foreman as a complete report (metrics and logs) In Foreman
> > 3.
> >
> > Proxy sends the ArfReport HTML which is saved as ArfReportRaw
> > (which currently saves the xml file)
> > 3. In Foreman:
> > 1.
> >
> > ArfReport < Report
> > 2.
> >
> > ArfReport has metrics and logs
> > 3.
> >
> > ArfReport Show in the same manner of Foreman report . Full HTML is
> > available via ArfReportRaw. XML is available via URL to proxy.
> >
> > Regarding the discussion in the thread:
> > There are several pros and cons to stop using Scaptimony as the parsing
> > and persistence engine:
> > *Pros*
> >
> > - ArfReport should be a subclass of Report. (so this is too Foreman
> > specific)
> > - ArfReport parsing should be done on the proxy (so we don't need the
> > scaptimony parsing part)
> > - We wish to save ArfReport xml file on the proxy's file system, today
> > it is saved on the foreman's db
> > - We wish to save the parsed report data and it's html on ArfReport
> > and ArfReportRaw
> >
> > *Cons*
> >
> > - We miss Scaptimony and OpenSCAP team contributions
> > - We may miss changes in OpenSCAP that may be reflected in Scaptimony.
> >
> > Way in the middle:
> > - Create a gem "openscap_parser" which takes the report xml and
> > returns it as json (basically what I am trying to do with the proxy
> > plugin). Then anyone can use it to it's own purposes: Use it on the
> > proxy
> > to send data to foreman, use it on another Rails app to persist it in
> > the
> > db. Maybe this should have been the initial purpose of Scaptimony?
> >
> > I would love to hear your comments and insights.
How much of the comparison logic do we get "for free" from scaptimony.
– bk
···
On 09/02/2015 05:18 AM, Marek Hulan wrote:
>
> On the other hand If we can't split the parsing logic and reuse the scopes for
> assets status then I don't see much benefit of having Scaptimony in Foreman
> and maybe we should look what we can reuse from it on smart proxy side. I
> guess that refers to what you've mentioned in "Way in the middle".
>
> How much of the comparison logic do we get "for free" from scaptimony.
not that much [1]. Another thing that Scaptimony could eventually bring is
waivers implementation. On the other hand I see our own waivers implementation
as a reusable thing for other report kinds (which I'd really like).
