Pulpcore Telemetry and Katello

[rbremer]

I am usually not providing much feedback on those issues, as I am not involved in neither Pulp nor K/F development. Being a software developer myself, I hope you can put my comment better into context.

A few things stick out to me in your comment.

While I totally understand the interest of the Pulp project in any kind of telemetry data, I mean we seem to live in a world today were this might be the norm, I do have many reservations here. I absolutely do not share your belief, that “most users don’t care”. My experience is quite the opposite.

The point I don’t rally understand here, is the feeling of pressure being put on the F/K team and ultimately their end users. To quote you:

Also it doesn’t speak to my core concern which is Katello would be way underrepresented in Pulp project decision making by defaulting to off.

So basically, if you don’t get telemetry from F/K users, things might happen. I leave the interpretation of your statement to the reader.

You develop a set of APIs, which is used by probably hundreds of projects every day. You want to make decisions about moving forward, informed decisions. As I stated above, that is absolutely needed and I applaud you for basing those decisions on actual usage data.

However, why does that need to involve runtime data from the user base of F/K? Shouldn’t the APIs being used in the code base be a perfect baseline for your analysis and decision making? I would assume, that taking away any single API in use by F/K will have an impact, as all its functionality is covered under the hood and not exposed to end users in any way.

I do not see any single reason, why F/K should even provide the option to include telemetry for any kind of plugin in their code. But as I stated above, this is my personal opinion, as a “privacy and security” first minded developer.

[iballou]

Hi @rbremer, thanks for the feedback.

@rbremer

However, why does that need to involve runtime data from the user base of F/K? Shouldn’t the APIs being used in the code base be a perfect baseline for your analysis and decision making? I would assume, that taking away any single API in use by F/K will have an impact, as all its functionality is covered under the hood and not exposed to end users in any way.

Let me share some of the potential data points that I’m excited about from a Katello perspective.

Katello is a small development team. If you look at the number of developers that regularly participate on GitHub, it’s a handful of folks. Katello is also a large toolbox with lots of moving parts. So, I believe Pulp Telemetry could eventually give us helpful data to help us prioritize our efforts. I could spend a week fixing a challenging bug that only 1% of our users will appreciate. If there is anonymous API data sent back from (consenting) users, we could get a better idea about what features are in use.

For one random example (don’t read too much into it), one piece of our software stack that gets a lot of engineering effort is dependency solving. We don’t have a great idea about how many people actually use it. So, if the data could have a simple graph of how many people tell Pulp to copy content via Katello with dependency solving (content view publishing), we would finally know how much the community relies on it.

It would be helpful for new features as well. We recently introduced Alternate Content Sources as a tech preview feature in Katello 4.6. In many ways we could continue improving the Alternate Content Sources integration further than what we are soon to present in Katello 4.7. If we had some basic data about how many people create ACSs in Pulp, we would have an idea about how much more effort we should put into making using ACSs even easier.

[bmbouter]

Thank @rbremer for the feedback. Yes this is controversial. It sounds like we disagree on what is best. Maybe I should have started with a discussion of a shared goal. My goal here is to have Pulp better serve the F/K community through data driven decision making.

It’s not my intention to pressure in any way, my apologies if that’s how it feels. It is my goal to point out that this decision has natural consequences. The Pulp project is increasingly looking to data driven decision making. If F/K decides not to participate in providing that data then Pulp can’t make decisions as well with respect to serving the F/K community.

So basically, if you don’t get telemetry from F/K users, things might happen. I leave the interpretation of your statement to the reader.

Yes the Pulp project might make a decision that unknowingly impacts F/K users. Here are two examples:

• Tech-preview APIs mostly/only used by F/K likely just stay tech-preview. See comment 7.

• Pulp upgrades the database minimum requirement to use DB feature X, which unknowingly causes F/K to also have to raise the minimum DB version. With F/K represented in the data, Pulp would easily know “wow this would be really significant to our user base” and without it we might incorrectly conclude “this will affect only a small percentage of installations”.

However, why does that need to involve runtime data from the user base of F/K?

Pulp doesn’t have insight into how F/K uses the Pulp APIs. Also, an API being used “somewhere in the Katello codebase” does not at all mean “actually used”. Even if it did, we don’t have data about what versions of F/K are actually in use.

[rbremer]

Thank you Ian for your feedback. I fully understand the need of getting as much insight as possible into an application’s usage, something I also have to deal with on a daily basis.

As old-fashioned as it might sound, we usually fall back to surveys as an open and transparent way of collecting the necessary feedback. This might not work in the case of F/K, as you might not even know who to ask.

One of the things I have seen implemented in the wild and which - from a pure privacy perspective - can relate to is an integrated survey/telemetry function in the product, which needs to be actively triggered by the end user/administrator. It should show the data being collected and sent and can also provide an overview of how that info is used. Think about it as “report to Apple” kinda dialog when an app crashes. It is so much more transparent from VMware “Technology Improvement Program”, where you agree once and would never know what data is actually sent and when.

Now the data you want to collect might look very detailed and technical, but I definitely would invest the time to present it to the end user in an understandable fashion, so they can decide whether to send it or not. The easier they can understand it, the more likely they would click on Send. And there can always be the option “Send automatically from now on”…

Having said all that, F/K is an open source software and all the work and effort you put in is not directly compensated by the end user. You have clearly the right to enforce telemetry data, if you opt to do so. I just wanted to chime in with the lessons learned from our customer base when it comes to telemetry.

[rbremer]

Brian, thank you for responding.

There is no need to apologize, as I am very happy to see different opinions being discussed in an open and professional way. At the end, we all learn something from it and can provide better products or services.

I wrote most of my thoughts in the reply to Ian (which I could not address to both of you, unfortunately), so I won’t repeat them here. There is just one thing which I would like to fully understand, as it also relates to the way we use libraries and frameworks internally.

As a developer of the Pulp framework, you provide a stable set of APIs, I assume within a certain release window. Whenever the release window closes, let’s say with a new major version, you might not be able to guarantee full compatibility with the previous set of APIs. Dependencies might change, the minimum DB version or any other constraints. You might also drop features or tweak the way existing features work.

So from my perspective it is the challenge of the F/K team as a - definitely heavy - consumer of your API set to keep up with your changes. They would expect the API to remain stable within a release window, with mostly bug or security fixes being introduced. If they use any experimental APIs it on their own risk, as it might be dropped at any time.

A major version change would also require a large amount of code being re-evaluated, rewritten or optimized. I can imagine the change from Pulp 2 to 3 was such a huge undertaking. Where would any API usage data help you in any of this? Changes to system requirements are usually driven externally anyway, like the chaos we had with RedHat dropping CentOS 8. The end user is forced to keep up with the ever-evolving landscape and staying behind with older releases is really no longer an option.
As said before, this is for me to understand the potential use of API telemetry, more out of curiosity than anything else. I hope it is ok to ask, as it is definitely not F/K related.

[iballou]

Your idea on a report that shows what data is sent back does resonate well with me. It would be awesome to see something like that since it gives the user no doubts about what is being sent. I’m hoping to strike a safe medium between that idea and the VMware “Technology Improvement Program” that you mentioned.

With how the Pulp team has implemented Telemetry I’m guessing it would be difficult to make this “push model” work, but I’m aiming to make the Telemetry data’s JSON “template” well documented from the Katello side. The user shouldn’t have doubts about what is sent based on this documentation that will be continuously updated as Pulp’s Telemetry evolves.

Our current rough plan is to make the opting in/out doable from the Foreman/Katello UI as well, so finding the documentation should be easy.

@rbremer

As old-fashioned as it might sound, we usually fall back to surveys as an open and transparent way of collecting the necessary feedback. This might not work in the case of F/K, as you might not even know who to ask.

This “old-fashioned” way is indeed how we’ve gotten feedback so far, besides with the RSS feed that Foreman has that doesn’t give a whole lot. That, and chatting with users at conferences. The sample size is pretty small though, which is the issue. I suppose community engagement could have some improvements as well, but I’ll leave that for another thread

[bmbouter]

Thanks for the questions Ronny. I’ll try to share some info, but let me know if its not getting at your question.

For a major version like Pulp3 we adhere to semver.org for our APIs for any API that is not in “tech preview”. If you look at Pulp2 it had roughly a decade long lifecycle and I suspect Pulp3 would be similar (we’re roughly at the end of year 3 currently).

New APIs are launched as tech preview first because once they leave tech preview, they are with us for the decade long window. This is where things kind of get complicated for Pulp and likely where analytics data on usage can provide likely the highest value. Answering the question: when is a new API stable enough to be accepted as around for 10 years is a question many folks (myself included) have trouble answering. When we don’t have a good answer we tend to just leave things in tech preview which is contrary to our goals of offering stable, long-term APIs.

Then there are the other things that change within that 10 year long lifecycle, e.g. the support of various OS versions or database versions will change. For example, the dropping of EL6 support as a supported OS for Pulp; we kind of had to at that point due to the impracticality of continuing to support it, but we had no idea of what the actual impact would be. This made it a hard choice, especially in terms of timing. Or more recently the dropping of Python 3.6 and 3.7 in favor of 3.8 as the minimum version. We kind of had to because our own dependencies were dropping support for it, but it was difficult to access the impact. If we knew the impact from data, we could focus on upgrade campaigns, upgrade tooling, etc even if we can’t do much in terms of changing the decision. Also knowing to wait N months while we focus on upgrades is an option too that improved data about impact can help us understand, even if the decision itself is unavoidable.

Hopefully this helps paint a picture of the kinds of things this data will be used for. More questions and other perspectives are welcome! Ultimately we’re here to help serve the F/K community.

[rbremer]

Thank you @bmbouter , this is a very good insight. I now understand much better the internal release management at Pulp.