Proposing to squash gh-pages for foreman-documentation

Development
1

Heya, I just did a clean checkout of foreman-documentation and this monster has over 2.4 GiB. Apologies, it was me who set it up this way in the very beginning :smiley: The offender is the gh-pages branch where we deploy all the content. I propose to squash the history of this branch to just one commit, since people typically do not work with this branch it should be okay, but there is a possibility that some test or something makes a full checkout of that branch and might break. But this is imho very unlikely, who would do full checkout on CICD, that is why shallow clones are for.

For you, nothing would change if you do not work with the branch actively. You can optionally reclaim 2 GiB of your space with:

git checkout main && \
git fetch origin --prune && \
git branch -D gh-pages 2>/dev/null || true && \
git gc --prune=now --aggressive

The commands to execute the proposal are:

git checkout gh-pages && \
git pull origin gh-pages && \
git checkout --orphan temp-gh-pages && \
git add -A && \
git commit -m "Reset gh-pages history to single snapshot" && \
git branch -D gh-pages && \
git branch -m gh-pages && \
git push -f origin gh-pages && \
git gc --prune=now --aggressive

We could even make it a scheduled job and only keep N commits.

  • Life is short, do it!
  • I want more, make it a job.
  • I don’t know about that…
0 voters
2

Here is a draft PR with automated job: Add weekly gh-pages history squash workflow by lzap · Pull Request #5267 · theforeman/foreman-documentation · GitHub

I realized that the GHA we use for publishing has an option that completely nukes all history and keeps just one commit all the time. This is also a solution, but I think it is better to have some history to validate deployment times and diffs.

3

Any reason for the 100 commits? 100 still sounds like a big number.

Would it make sense if the execution is time-based to have also the result time-based? Like keeping only the history of the last month or something.

1 Like
4

Alternative: Ignore remote refs with git — musicmatzes blog explains how to exclude the gh-pages: add fetch = ^refs/heads/gh-pages to the remote’s config. When I follow that:

$ du -sh .git
1,9G	.git
$ git gc --aggressive --prune=now
...
$ du -sh .git
74M	.git

Though I must also admit I previously had pulled in all the PRs which I no longer have now.

I don’t know how to specify this when cloning. Best I could come up with:

git clone --single-branch https://github.com/theforeman/foreman-documentation
cd foreman-documentation
git config set remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
git config set --append remote.origin.fetch "^refs/heads/gh-pages"
git fetch

Agreed that IMHO time based is better.

1 Like
5

Good idea, so last month? Two months? This is hard :slight_smile:

I only typically want to see when or if a change was deployed right away, for anything else we have the main branch itself.

But I am fine with status-quo, in that case, I would just update the README with @ekohl 's instructions on how to checkout the repo. Or even provide a script that will drop the content for folks with existing checkouts.

6

So I rebased my PR to have a configurable default value of one month of history kept (90 days).

I checked with @maximilian on Matrix, we have a good people wanting the job.

There is no need to rush with this, so let’s wait at least for a week. Cheers.

7

We merged the upstream PR by lzap and ran it today: Add weekly gh-pages history squash workflow by lzap · Pull Request #5267 · theforeman/foreman-documentation · GitHub :white_check_mark:

Thanks for implementing it.

2 Likes
8

All: please run the following command in your git checkouts:

git checkout master
git pull
git gc --aggressive --prune=now

My checkout went from 2.6 GiB to just 0.27 GiB.

New clones will still clone the big repo until GitHub schedules garbage collection, this is done automatically and it cannot be forced. We should see the total repository size under Settings > General go down (I think).