Cloud Foundry
Keynote: Cloud Native Security: Rotate, Repair, Repave - Justin Smith, Director, Pivotal
updated
Attendees
TOC: Beyhan, Chris, Ruben
Ram Iyengar, CFF
Review actions from last week
Review PRs
RFC: TOC can stand in for WG leads
Any other business
Do we want to enable push protection for all repositories: docs.github.com/en/code-security/secret-scanning/push-protection-for-repositories-and-organizations
We decided to first collect feedback about this from the WGs and if there is interest we will look into options on how to automate it.
Action
Collect feedback for the secret scanning option provided by Github from the WGs: cloudfoundry.slack.com/archives/C026XJGNC2U/p1693925619689979
We’re planning to cut a new release once the Docker epic is complete.
Support for more user roles would come in v0.10.
Attendees
TOC: Amelia, Chris, Ruben
Ram Iyengar, CFF
Chris Clark, CFF
Anita Flegg
Sylvain Kalache
Review actions from last week
Review PRs
CloudFoundry GitHub community board
New docs wg github.com/cloudfoundry/community/pull/677
RFC: TOC can standin for WG leads github.com/cloudfoundry/community/pull/676
Any other business
No updates from Canonical - to reschedule next meeting due to US Holiday on 9/4
Action
Recording
Attendees
TOC: Chris, Ruben, Stephan
Ram Iyengar, CFF, Chris Clark CFF
Review actions from last week:
Ruben manually fixed the PR merge - in future PRs should be merge commits and not rebase
Review PRs
CloudFoundry GitHub community board
Final Approval of RFC for readiness checks
Any other business
Incorporating Supply Chain Files github.com/cloudfoundry/bosh/discussions/2466
Action Items
Ruben to run script for RFC
Script for future ref github.com/cloudfoundry/community/blob/main/toc/rfc/assign-rfc-number.sh
Not at the moment.
After Docker is complete it might make sense, coordinate via slack.
[JH] 2FA Requirement
US team to pick this up and create an open-source issue for it.
Connecting with other CF cross-company teams and learn how they’re doing it.
Docker Epic
Started it and working through the stories.
Filling in Roles in the future
Consider using “kubectl auth can-i” to validate RBAC rules for each role.
Provisioned Services
Potential future proposal if we end up doing this.
Don't miss this opportunity to hear from an experienced practitioner as he talks about navigating the intricacies of Kubernetes, while gaining invaluable insights, and generally fortifying and organization's engineering prowess in the context of cloud-based platform development.
Expect answers to questions such as
? How to choose the right infrastructure for Kubernetes
? Where to begin when implementing a scalable and reliable platform
? How best to secure a Kubernetes platform
? What's the best way to automate platform operations
? How can we measure and manage platform performance
This event is for engineers who are interested in learning more about building and operating platforms on Kubernetes. Whether you are new to Kubernetes or you are an experienced engineer, we expect that you will have actionable takeaways.
TOC: Ameilia, Beyhan, Chris, Ruben, Stephan
Ram Iyengar, CFF
Review actions from last week
Review PRs
CloudFoundry GitHub community board
[PR 653] Find the right place for bosh-package-java-release and bosh-package-cf-cli-release
Review RFCs
[RFC 665] User Account Management in Cloud Foundry with Multiple Identity Zones
On hold: authors on vacation. Peter gave feedback.
[RFC 640] for integrating pcap-release with BOSH
Today is the end of the commenting period
[RFC 630] add readiness healthchecks for apps
Maybe we can start the final comment period?
Any other business
Dan Mikusa gives update on Paketo WG
Continuous focus: timely release delivery. Want users to have confidence in us! Often overlooked, but very important.
Ubuntu Bionic: EOLed. Not as smooth as we would like. Started RFC to do this work a month before. Could’ve been more proactive. People were caught by surprise, maybe a blog post next time. Most of the work is communication, archiving repos, turning off CI.
First steering committee election! Filled all seats! Dan will share links with more details in slack.
Community sponsored UBI stack. Lots of contributions from red hat. Expands potential customers.
Default java version bumped to 17! There is an RFC for how to do it going forward re communication etc.
Contributions from oracle recently.
In progress
Working on ARM64 support. Challenging, big effort. Will probably be working on it the whole year. First output: ARM64 stack.
Decoupling buildpacks from dependencies. Idea: buildpacks will look at a path to get the dependencies. RFC is open.
Ruben: Where will compatibility testing belong to?
Ruben: Will CF still be able to use versions that have all of the dependencies there?
Amelia: general concerns about security. Dan: yes, we are thinking about it.
Language Family Builders: traditionally we have had an “everything” builder. It is big. A lot of people don’t use all of the bits of the one builder. Thinking about making smaller builders (eg java, node).
Make it easier for people who select java vendors. Currently have 11 buildpacks for different java providers. RFC approved, work pending.
Most asked for items/problems
ARM64 support.
Download caching: long standing issue. If you change image name, it invalidates the cache.
Ruben: we have an open CF RFC about integrating some paketo buildpacks. Do you have thoughts? How could we better collaborate?
Dan: fyi: some technical changes coming soon from CNCB.
Actions
Ruben: run some script for RFC 640
Recording
Docker proposal discussion: docs.google.com/document/d/1vJLBL1f_ebZm7YWJ9iiNa3DHhrhTOUIyBg1DzIpyuDA
Attendees
TOC: Ameilia, Beyhan, Chris, Ruben, Stephan
Ram Iyengar, CFF
Chris Clark, CFF
Dominik Froehlich, SAP
Alexander Lais, SAP
Review actions from last week
Update from FI WG
Already delivered or under development
Improvements in disaster recovery (Discussion)
Support the azure blobstore as external blobstore for bosh (PR)
Remove blobstore info from IaaS metadata endpoints (PR)
Support instance group tags (PR)
BOSH CLI provides linux arm64 version with release v7.2.3
Assume role support in the AWS CPI (PR)
Major release v9.0.0 of bosh-bootloader with breaking changes:
moved from terraform 0.11.x to 1.4.x with no upgrade scenario
Consolidated bosh-packages and bosh-io into cloudfoundry github org (PR)
Possible next things:
Network tracing support with the pcap-release (RFC)
Update strategy for BOSH variables (Discussion)
FIPS Stemcell exploration
GPU support (Draft PR)
Move BOSH Director pipeline to the community CI
Review PRs
CloudFoundry GitHub community board
github.com/cloudfoundry/community/pull/661
Review RFCs
FCP candidate: github.com/cloudfoundry/community/pull/640
New: github.com/cloudfoundry/community/pull/665
Attendees
TOC: Ameilia, Beyhan, Chris, Ruben, Stephan
Ram Iyengar, CFF
Sylvain Kalache, CFF
Georgi Sabev, CF on K8s WG
Review actions from last week
Update from CF on K8s WG
Two new releases but 0.8.0 introduced major increment with:
Kpack image sharing which makes re-pushing of app faster
Buildpack selection support with “-b”
No support for remote buildpacks
Support for service selection by labels
Support for rolling updates for push, restage and restart
This mean the CF API v3 deployment has been implemented
Deletion jobs for CF resources jobs, tasks
Improved finalizer for the resources which make sure that the resource is gone
Additional CF APIs as renaming apps, getting app domains etc. have been implemented
Next thing looking into:
Docker image support
User management
CF API v3 for getting users and deleting users
It could be implemented in a similar way like CF
Manage Services support
Exploration needs to happen
Team structure update
Giuseppe Capizzi is rotating out from the project. New execution lead Andrew Wittrock and new technical lead Dave Walter
Review PRs
CloudFoundry GitHub community board
github.com/cloudfoundry/community/pull/658
github.com/cloudfoundry/community/pull/656
FYI: github.com/cloudfoundry/community/pull/659
FYI: github.com/cloudfoundry/community/pull/653
Review RFCs
github.com/cloudfoundry/community/pull/630
github.com/cloudfoundry/community/pull/637
github.com/cloudfoundry/community/pull/640
Any other business
The CF community looks into options to add FIPS stemcell support. Discussions with Canonical are ongoing to clarify the required license options to get a FIPS license for the CF community so that validation can happen in the community. Next steps after the clarification could be:
RFC for how to produce the FIPS stemcell
Most probably also an RFC about how to consume the FIPS license in the CF community
Difficult to test
Might be more expensive than at first glance
Perm project retirement: lists.cloudfoundry.org/g/cf-dev/topic/proposal_to_retire_the_perm/74842395?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,74842395
This should still be easier than Perm since we don’t have to deal with a then 7 year old metaprogrammed codebase, but may still be more complicated than it seems
Is manual testing adequate? What is the cost of automated testing?
Pick glaring omission role and fill out slowly.
[GC] Branch protection PR
[RI] Why can’t Korifi default to list all Buildpacks available from Paketo? (Reference conversation)
At least DotNet Core, Python, and PHP.
Expose list of buildpacks in values file?
Simple configuration as list value in the Helm chart.
Experiment to see how much bloat from adding extra buildpacks.
[RI] v0.8.0/v0.8.1 is out. Could we go through the release notes and see what the implications are of each of those items?
UPSI service label proposal UPSI Service Label Enhancements has more info on that item
Attendees
TOC: Ameilia, Beyhan, Chris, Ruben, Stephan
Chris Clark, CFF
Ram Iyengar, CFF
Sylvain Kalache, CFF
Jochen Ehret, ARD WG
Arsalan Haider, Cloud.Gov
Review actions from last week
Update from App Runtime Deployments WG
Complete: Migrated all infrastructure to new GCP/AWS/dockerhub account from vmware accounts to community funded accounts.
New release: Bbl v9.0.0
Branch protection rules: activated.
Cflinuxfs4 migration: in new cf-d release, cflinuxfs3 has been removed. Only stack and stemcell on the latest cf-d are the new jammy ones.
Cflinuxfs4-compatable release is also integrated into the cf-deployment now
Continuous work to stabilize the CATs
Ongoing discussions about new stacks from the Paketo WG to be integrated into cf-d
Future: Update the upload release pipeline to use new bosh 256 sha, instead of sha1
Review PRs
CloudFoundry GitHub community board
github.com/cloudfoundry/community/pull/643
github.com/cloudfoundry/community/pull/654
github.com/cloudfoundry/community/pull/655
Review RFCs
github.com/cloudfoundry/community/pull/641
github.com/cloudfoundry/community/pull/630
There is still a discussion open and Amelia will look into it.
Any other business
Plan the vacation time
Amelia: I’m out next week. How do we report when we plan to be out?
Action
Amelia: Start something in slack to gather vacation time to decide when we need to cancel meetings.
Everyone: fill in your vacation time on slack
The goal of this talk is to gain community awareness and collect valuable feedback. We feel that the pcap-release project can provide lots of value to Cloud Foundry, because it makes it very easy for both operators and developers to capture network traffic from multiple hosts without any overhead.
Speakers
Dominik Froehlich
Senior Developer, SAP SE
Maximilian Moehl
Developer, SAP SE
Host:
Ram Iyengar
Chief Evangelist, Cloud Foundry Foundation
TOC: Andy, Beyhan, Eric, Ruben, Stephan
Ram Iyengar, CFF
Paul Warren, Vulnerability Management Working Group
Arsalan Haider, Cloud.Gov
Update from Vulnerability Management WG
Paul Warren presenting
Quite busy recently on the vulnerability front. Managing 8 vulnerabilities at the high point. Ticked down again to about 4 active vulnerabilities. 8 is a lot of vulnerabilities to have active at one point. Don’t think anything caused that uptick.
On USN side, continue to publish those monthly on the #security channel. No issue publishing those. Continue to work with Davos team to try to make updates to the system. We don’t think we should be publishing anything about Xenial anymore, for example.
Tracked the cflinuxfs4 through from experimental through to production. We’ll probably follow up with a post on #security.
PW stop for questions
No questions
We have become aware of a vulnerability in Concourse. I’ve got a Google doc I want to share.
PW to share doc with TOC members.
Worth addressing quickly.
We’ve had 2 or 3 instances where details of the vulnerability have been leaked out in the commit messages and release notes before we’ve had time to release that vulnerability. We probably want to remind the team that if they’re included in a #sec- channel they should be careful with what they put in public places, until it’s been reported to Mitre etc.
Review PRs
github.com/cloudfoundry/community/pull/591 (Paketo buildpacks)
github.com/cloudfoundry/community/pull/622
Any other business
Next week’s meeting (20th June)?
Action
PW to share Concourse vulnerability details with TOC members
github.com/cloudfoundry/korifi/blob/main/helm/korifi/values.schema.json#L182-L185
It appears that we set this in every CFBuild object, but then we don’t use it for anything?
Julian to add a story. We should mark the CFBuild CR property as unused also, since we never used it.
Staging resources settings are under an `api` key. Could we move them to a more global location?
Julian to add a story to refactor. Make sure we don’t leave the old properties references in the docs for any properties we move. We should make sure to get proper review for the PR also.
[JH] korifi/tools unit tests: are we running them in CI? Should we be?
Yes we should. JH to add a chore. Also add it to the test-everything script?
[TD - can’t attend] What is left for 0.8?
Missing anything in v0.8?
github.com/cloudfoundry/korifi/issues/2604
github.com/cloudfoundry/korifi/issues/2605
github.com/cloudfoundry/korifi/issues/2645 - bug about cf push not retriggering a build when the buildpacks change after a failure. (It just gets stuck) Probably we should not block on this issue though.
TOC: Amelia, Andy, Beyhan, Chris, Ruben, Stephan
Ram Iyengar, CFF
Chris Clark, CFF
Review PRs
github.com/cloudfoundry/community/pull/642
github.com/cloudfoundry/community/pull/636
github.com/cloudfoundry/community/pull/632
TOC chair selection
Beyhan Veli nominates himself
TOC selects Beyhan Veli as new TOC chair.
Contributor docs
Yearly review from VMWare of docs they contribute to. They identified it being difficult for newcomers to work out how to contribute, what projects are part of.
We thought we didn’t have many new contributors, but we do. It would be nice to have a better entrypoint on the website which explains how things work; guidelines on what should be submitted as what.
[DB/GC] Memory limits (and limits in general) for build pods
[GC] Branch protection exception on korifi-ci
Need a PR to github.com/cloudfoundry/community/blob/main/org/branchprotection.yml to override the auto-generated default configuration
[GS] Team reorg
[RI] (Pardon the multi-part question) What’s the roadmap like for the next releases? What are the major press-worthy features in the pipeline? When can we expect support for a cf marketplace command? What about support for deploying Docker containers?
Supporting existing Docker containers is an easier incremental step to introduce Korifi to new audiences
TOC: Amelia, Beyhan, Ruben, Stephan
Ram Iyengar, CFF
Arsalan Haider, Cloud.Gov
Review actions from last week
Review PRs
CloudFoundry GitHub community board
Review RFCs
github.com/cloudfoundry/community/pull/641
Any other business
How should we continue with the WG updates?
We will go with the “Last Update” date
Action
Check on final period in github.com/cloudfoundry/community/pull/641
New members
Amelia Downs, VMware
Chris McGowan, cloud.gov
Outgoing members
Andy Hunt, GOV.UK PaaS
Eric Malm, VMware
Cast for 2023-2024:
Amelia Downs, VMware
Beyhan Veli, SAP
Chris McGowan, cloud.gov
Ruben Koster, VMware
Stephan Merker, SAP
Attendees
TOC:
Arsalan Haider, Cloud.Gov
Review actions from last week
Update from App Runtime Platform WG
CF Day
Slides for ARP update talk
(docs.google.com/presentation/d/1KloUgoE4D9LTb0RemEPgy-HlMieDFhRs0fB-D9ANdf0/edit?usp=sharing)
Readiness Healthchecks
Original proposal (github.com/cloudfoundry/cf-deployment/issues/1094) in cf-d. Closed in favor of an RFC (github.com/cloudfoundry/community/pull/630).
AZ Aware Routing
Taking a step back to think about it more before developing anything.
Looking for feedback from any users, and asked the following questions of the audience in CF Day talk…
❓ Do you have smaller, single-zone CF deployments that you would like to consolidate to multi-AZ ones but are holding back for some reason?
❓ Do you have problems with cross-AZ behavior inside of an existing multi-AZ CF deployment
Upcoming
Go 1.21 - coming in August!
SAP is operating Cloud Foundry at very large scale and in a multi-tenant setup. Thousands of customers deploy apps, leading to highly heterogeneous workloads, client applications and a staggering number of requests on ingress and egress. With so many endpoints and apps there is more opportunity for things to go wrong.
In this talk Alex & Vladimir, on behalf of the SAP BTP CF Routing and Runtime team respectively, present the most significant past and ongoing challenges and how they were addressed so far. Their stories and encounters running large CF landscapes with varied workloads include the following:
Running Diego cells in separate deployments and with more NAT Gateways
How to overcome Hyperscaler bandwidth limits during landscape updates
Improving the interaction between Gorouter, Envoy and Diego with crashing apps and crashing cells
Our contribution to safely re-enable Gorouter route service hairpinning
Benefits for the Ecosystem:
The goal of this talk is to raise awareness for the scale at which CF can be operated and to share the challenges that come with such large deployments. This presentation is specifically focussed on the most scalable parts of the architecture, the routing stack and Diego runtime.
For the ongoing challenges, specific areas are identified in need of solutions. Those areas will require ideas from and alignment with the community. This talk aims to share gained knowledge and ideas to spark further collaboration within the wider community.
The BOSH director is built as a singleton that is only available as a single unit in an environment. As BOSH is normally only used during deploy time, it is not critical. During a small Availability Zone Outage, it is not possible to deploy new deployments, but the running system is unaffected and will work as long as it is deployed over multiple availability zones. But what happens if an availability zone is unavailable for several hours or days? Or if an AZ is destroyed? How do we restore a BOSH director in such a situation? The BOSH team at SAP spent some time working on a solution for such a scenario and will share the results in this talk with you.
On April 21st 2020, a blog post announced the availability of the Paketo Buildpacks. On April 21st 2020, our architect asked us to take a look. And on April 21st 2020, we started to adopt them. :-) In this session, we'll briefly talk about what Shipwright as the framework that builds container images on Kubernetes. We'll share how Paketo can be used with Shipwright, and how we productized this in Code Engine. We'll present how our users experience Paketo. We'll talk about where Paketo is really strong, but also where we struggled, and what we wish for in the future.
This session will explore the remarkable updates by the Application Runtime Platform Working Group, known for their pivotal role in developing key Cloud Foundry components such as Diego, Gorouter, Garden, Logging, Metrics, and more. We will discuss our progress towards swiftly and safely upgrading Go versions. Then we will demo the brand-new distributed tracing feature that will aid operators in more easily navigating through the vast logs of Cloud Foundry. And that's not all! Brace yourself for a sneak peek of readiness healthchecks, which are coming to revolutionize a Cloud Foundry near you soon.
Don't miss this opportunity to discover the exciting advancements presented in this session, brought to you by Amelia Downs, the Tech Lead of the Application Runtime Platform Working Group.
In this session, the current state of supply chain security in the Cloud Foundry ecosystem will be explored. We will look into the mechanisms used by Bosh to guarantee build reproducibility. In this context, we will also take a more in-depth look at Concourse. In the second part of the talk, we will be looking at future improvements that are planned in this space, like SBOM generation for Bosh releases, Concourse hermetic builds, and how all these things will eventually allow us to achieve SLSA compliance.
Platform Engineering has rocketed into popularity as the successor to DevOps and SRE. That discipline’s focus is on building and improving developer experience. Sounds familiar, right? Indeed, this has been the focus of the Cloud Foundry community from the beginning and - it works! This talk will explain what platform engineering is and suggest ways the two communities can help each other. Examples include: a product management mindset, internal advocacy & marketing, services and developer tools to build. The talk draws on years of real world experience at organizations. Past version of this presentation: I've given the content of this presentation several times, but I haven't framed it in this way at all. I'd reuse the same best practices, but the format of the presentation would be as I outline above, speaking to the two communities rather than giving end-users some best practices and advice to follow.
Join Stephan Klevenz, CFF board member, and Beyhan Veli, TOC member, at CF-Day in Heidelberg to explore the beautiful city of Heidelberg and draw parallels with Cloud Foundry. Let us discover together synergies between technology and urban landscapes. Additionally, Beyhan will provide updates from the CF community, highlighting collaboration within the Cloud Foundry ecosystem. Don't miss this insightful talk on innovation and community-driven progress.
As part of Cloud Foundry, Loggregator, the logging and metrics stack, offers forwarding application logs and metrics to third party logging providers via user-provided services. Authentication to the logging providers was only possible when using http, but not with the syslog protocol which allows for a much higher throughput. To improve log forwarding performance and provide secure connections to third party logging providers, mutual tls authentication was introduced. The goal of this talk is to showcase the feature and to outline the faced challenges during research and development. Felix will give a technical overview of the secured and improved syslog-drains and their introduction in a zero downtime environment.
In this talk, we will discuss the process of migrating from the legacy stack cflinuxfs3 based on Ubuntu 18.04 to the new stack cflinuxfs4 based on Ubuntu 22.04 in Cloud Foundry. We will begin with the integration of the new stack into cf-deployment, the canonical Cloud Foundry distribution. We will then cover the challenges rolling out the new cflinuxfs4 stack into large, multi-tenant, public PaaS foundations including planning, effective communication, and stakeholder management. Then, we will share insights about the adoption rate of cflinuxfs4 and discuss ways how to help customers getting their applications migrated to cflinuxfs4 so that eventually the old cflinuxfs3 stack can be retired. Finally, we will look ahead discussing how a stack migration could be improved in future - because the next stack migration will come in four years latest.
Platform engineering move aside! The Cloud Foundry community has been doing this for over a decade, and one could argue that the Cloud Foundry community has truly defined what a platform and platform engineering teams should look like. This session will detail how platform engineering teams at VMware run Cloud Foundry internally to host mission-critical applications. Furthermore, we will break down stories and metrics from our customer base on the incredible outcomes and scale they have reached with their Cloud Foundry deployments! This session will demonstrate how our everyday lives continue to be powered and impacted by Cloud Foundry in some way or another and continue to set the gold standard of platform engineering.
An outstanding property of Cloud Foundry has always been the ability to work at large scale. Therefore, it is no surprise that there are many Cloud Foundry environments containing thousands and ten thousands of apps. For various reasons, such as changing the infrastructure or the Cloud Foundry distribution, it might be necessary to move the workload from one Cloud Foundry environment to another. Such a migration usually needs to occur with a minimum of CF app downtime. In this talk you will learn about the resulting challenges and how to overcome them. At the end of the you will have learned about migration strategies for both applications and their data.
As the CF V3 API is feature complete and available since 2021 it's time to ramp down the V2 usage. This talk aims to address critical aspects of this transition, ensuring a smooth and efficient experience for users. We will delve into the introduction of a dedicated rate limit for V2 endpoints, designed to mitigate the impact on the overall performance of the system while encouraging users to migrate to the more advanced V3 API. To accommodate the diverse needs of our user base, we will discuss the user exemption process, which provides a temporary reprieve for those who require more time to adapt their applications and workflows to the new API version. In addition, we will explore the significant performance improvements offered by the Cloud Controller in terms of scalability and resource management. Lastly, we will address open topics, including the implications of the API V2 ramp down on various CF API clients and the necessary modifications to the CF CLI plugins to ensure seamless compatibility with the updated system.


