I started my travel on Wednesday June 4th afternoon and at about 9 in the evening I reached the hotel in Prague, Andel and went to grab some dinner. Returning to the hotel I met folks from my team (Ant, Michal, Patrik, David and Fabian) in the lobby and was happy to say hi to them.

I met Fabian and some other folks at the breakfast Thursday morning. We were talking about how the konflux pipelines could be used for rpm build instead of koji and Fabian shared some pitfalls, to be aware of and to avoid like signatures during the build process. After the breakfast we moved to the event venue, where we met Greg who gave us his cool 3d printed Fedora badges.

The first talk of the day were Opening Remarks with the event welcome and official passing of the torch of the FPL from Matthew Miller to Jef Spaleta dressed as a hotdog, followed by Jef’s intro talk, where he shared with us a little bit of his impressive background (I’m still reading about SuperDARN radars) and most importantly his vision for the future of Fedora (one of which is ethical AI use). Jef attended some of our CLE meetings before, but it was nice seeing him live. Although we didn’t manage to talk directly, he was very active joining talks, asking questions and commenting on stuff he found important, so I saw him around a lot.

The next talk was FESCO AMA with many of the current FESCO members answering questions about AI and new contributors and other stuff. Stephen Gallagher and others voiced their focus on ethical and sustainable use of AI with responsible use of resources within the Fedora community. Other members were talking about preparing Fedora for AI with the right tooling and platform, possibilities to leverage AI in Fedora Infrastructure, either as a coding assistance or automated actions. The opinion that AI should never be allowed to make decisions, but only present choices was also voiced.

The talk was followed by the Forgejo talk presented by the Forgejo upstream contributor and was very interesting since I’m currently in the Forgejo Deplyoment team. Otto Richter was introducing some of the new features, from which I was particularly interested in forgejo actions, which is something I was working on the last coupe of weeks. He also mentioned how the protected branches work, permissions and dependency issues.

After lunch I had hard time deciding which talk to attend. I finally went to hear more about Log Detective, where Tomas Tomecek and Jiri Podivin explained how logdetective works right now and how to use it - and also how to sanitize the data before uploading any logs, since once uploaded, the logs stay out there in public. I was surprised by the amount of users logdetective has since it’s available for the community (running on Communishift!). Jiri also explained how the reviews and scoring system work for those that would be interested in training the model.

The next talk I attended for about RISC-V and Fedora, presented by David Cantrell. I don’t know anything about RISC-V, so I took it as an opportunity to learn something new. The next talk I was looking forward to was the ROCm and AMD GPUs in Fedora. I wanted to hear about what’s new there because some time ago I was trying to train a model to detect penguins on pictures of t-shirts (might be that Troy Dawson inspired me) and I was using the same setup with AMD GPU with ROCm and trying to run a jupiter notebook with tensorflow and I remember struggling a lot with the GPU not being detected by ROCm.

After the break I joined CLE team AMA, where it was mostly us CLE folks :) Anyways, some interesting questions were asked and answered. In the evening I watched Greg’s talk Build it and they will come and other myths. I must say I enjoyed this talk a lot. Greg is a great presenter (the teaching experience shows) and as a former community architect built a lot of knowledge around the topic. Later, when I caught him on the hallway, he recommended me some of the books he read on this.

When the talks ended I went to visit an open air book festival organized by Museum of Literature in Praha-Bubenec and allowed myself to buy just two books, because my suitcase space was very limited.

In the evening we had the CLE dinner in a nearby traditional restaurant, and the food was great. He had some nice talks with Ant, Samyak, Michal and Aurelien, who I tried to convince to taste the traditional czech stinky cheese “Olomoucke tvaruzky” (which I love!), and only Samyak was brave enough, kudos to him :)

On Friday I volunteered at the registration desk with Emma, and we spent good amount of time talking about stuff, which was very nice. I missed whole morning of talks and I intend to catch up with some later. After that I attended Alexandr Bokovoy’s talk about the security risks of voluminous packages with many bundled dependencies where he drafted a potential workflow for upstream development.

We spent Friday evening on the boat trip on Vltava and enjoyed excellent sightseeing of iconic views of Prague, while eating dinner and then walked to the hotel with bunch of Flock folks.

Last day was the day of workshops, hackfests and discussions. I had hard time deciding which of them to attend, since they were not recorded and I could not watch the ones I missed later. I opted for Log detective workshop. Jiri Podivin demonstrated use cases of what logdetective.org webpage can do and I tried to upload logs from a package that I maintain and it’s build is failing to see the AI model’s response.

After the refreshment break I joined Dan Walsh’s talk about containers. He made a brief intro talking about container layering and the good practice of using not-too-light base images. He said that using small base images like alpine makes very small base layer that end up having very big layers on top of it (having to install many packages that could have been in the base layer already) and that it can slow down the build. It made me think and search for articles written by Dan to get more insights about good practices in writing image definitions. Dan suggested it to look like a pyramid or a wedding cake, with biggest base layer, followed by smaller layers on top. He also briefly mentioned new commands in podman, like podman artifact extract, that is extracts an OCI artifact to a local path and podman machine cp that securely copies the content between the host and the virtual machine. He then gave word to present experts in container technology to share their knowledge.

In the afternoon there was finally the long awaited discussion moderated by Tomas Hrcka I was looking forward to attend: Distgit implementation in Forgejo. The discussion was split into three areas: ACLs and ownership, lookaside cache and migration from Pagure.

The ACLs and ownership

d There was a discussion about the setup of rights and their granularity (admin, user, organization/teams, issues, pull requests) and whether the new forgejo distgit instance should follow the old rules or revise them and how the handle the use case of provenpackagers, sysadmin-mainers, releng people, whether to allow nesting org => team. Distgit now is for example allowing only teams, not organizations. The discussion also touched the need of a stable API, ownership of the repos in distgit, whether the packagers should own their repos and how to manage settings to accommodate all this granularity. Some people mentioned preference to not allowing force push. It was discussed which setting should a package maintainer have right to access, for example access branches, commit access, issues, pull requests and that’s about it and how to manage emergencies that are now handled by provenpackager members. Aleksandra Fedorova mentioned there should be one other person from the provenpackagers that would approve the changes and it should be the responsibility of the packager to contact them. There were ideas about how to go about it, whether to mark those that would be willing to be contacted or if FESCO should find one. There was a resolution to discuss the details on the devel-list. There were strong voices demanding strict policy for mass changes and handling the cases where mass changes are necessary with care.

The Lookaside Cache

Tomas presented three options of how to approach the lookaside substitute:

1. package registry in forgejo
2. git -lfs
3. or just use the ols lookaside cache

There were voices that favor consolidation of the sources and have the Lookaside Cache as part of the Forgejo. Aleksandra mentioned that the package registry does not fit the purpose Forgejo originally meant. Tomas said maybe we could make another registry and run as another service. It’s only for storing tarballs and sources. It is important that the fedpkg stays the same. Stephen Gallagher disagrees and says self hosting is the best option for the Lookside Cache. To have a reasonable pipeline, upload sources and perform an rpm build. Package registry should have the same flow as the Lookaside Cache. Kevin Fenzi said we would like to filter out the binary stuff and wanted to rule out the git -lfs option, since it changes hashes which would lead to confusion and packages might not get built. Jiri Podivin mentioned a kerberos ticket is now needed for the upload and Tomas replied that it’s gonna be addressed. Aoife asked who is gonna be responsible for for which chunks and who will make the decisions, Tomas said that it should be FESCO. There were some concerns about the terminology, since now the package registry does not make it obvious that it would include the tarballs as well. UI changes would be needed. It was also clarified that the Pagure instance will be moved first, since the work is much easier, and that discussions about distgit are necessary to move the effort forward. There’s also a question whether to have two instances - like now there is pagure and distgit or if to consolidate them. There are pros and cons to both. More discussions and experiments have to happen. Voices from the audience demanded they would be visually different not to confuse them.

The Migration from Pagure

Tomas presented a plan that counts with moving the general forge to Forgejo. We have staging Forgejo up and running and we focus on doing dev work of missing features during F43 release. During F44/F45 then iterative changes on staging and performance testing. For F46 there could be Forgejo deployed in production.

From the discussion: Forgejo currently does not support private issues. When this feature is added, FESCO and Council should switch to private issues first and test them. The discussion also touched whether the kernel groups could use Forgejo as well, to have all the fedora code there on one place. Aleksandra replied that kernel groups have different rules and it’s not certain if the packager rules would apply. Tomas said kernel code could be hosted by another instance of Forgejo with their specific set of rules. Aleksandra asked what could make the groups that moved to Gitlab change to Forgejo, for example CoreOS. CoreOS had to move to Gitlab for the Gitlab CI. There was a discussion how to help the teams to make it easy for them to switch to Forgejo. The resolution was to keep getting more feedback. Another question: whether it should be easy to make the git forge easily replaceable, to facilitate easy migration to a different git forge. Tomas said that the Lookaside Cache is easily portable being just and http.

It was exciting to be part of discussion where the very foundations of the new git forge were drafted.

As the last session of the day I attended Aurelien’s Improve the Infrastructure applications, where he introduced the structure of the Infra apps, like Authentication, FASJSON, Fedora Messaging and the Package update and Badges workflow. He also included an overview of how tiny-stage works, simulating the Fedora Infrastructure environment with many interacting apps, which I used many times in local testing and which is irreplaceable. There was a discussion between Aurelien and Jef whether moving tiny-stage to containers would be possible and how to solve the networking between them.

As always, Flock was really motivating for me, and I was quickly filling my notebook with notes about interesting things I wanted to learn more about, books and articles to read, podcasts to listen, stories from the ecosystem which are important for having context of what is happening in the community over time. I enjoyed meeting people of my team face to face, discussing work stuff and also just having fun. I also met many people from the community, some of which I knew by name, but never met and had the opportunity to talk to for the first time and some of which were completely new in the community.