Peter Birkholm-Buch

Stuff about Software Engineering

Tag: Carlsberg (Page 1 of 2)

Gaia: How We Built a Platform to Transform Infrastructure Creation

2024-11-08 /

The Evolution of Infrastructure Management

In Software Engineering in Growth Products at Carlsberg, our journey towards modern infrastructure management began with a familiar challenge: as the number of development teams grew, the traditional approach of manually provisioning and managing infrastructure became a significant bottleneck. The DevOps team, tasked with building and maintaining infrastructure for multiple development teams, found themselves overwhelmed by the increasing demand for infrastructure resources.

Each new project required careful setup of networking, permissions, and cloud resources, all of which had to be manually configured by a small DevOps team. As development velocity increased and more teams came onboard, this model proved unsustainable. New projects faced delays waiting for infrastructure provisioning, while the DevOps team struggled to keep pace with mounting requests.

Reimagining Infrastructure Creation

The solution emerged when the DevOps team envisioned a different approach: what if developers could create their own infrastructure while adhering to organizational standards? The challenge was to enable self-service infrastructure without requiring developers to understand the complexities of building secure, scalable, and compliant cloud resources.

This vision led to the creation of Gaia, a platform that automates infrastructure creation while maintaining strict security and compliance standards. Built by the DevOps team, Gaia represents a fundamental shift in how infrastructure is provisioned and managed at Carlsberg.

The Platform Engineering Approach

Infrastructure as Code Evolution

Gaia elevates infrastructure creation beyond basic scripting by providing a comprehensive platform engineering solution. The platform utilizes Terraform for infrastructure provisioning but abstracts its complexity through a higher-level interface. This approach allows developers to focus on their applications while ensuring infrastructure deployments follow organizational best practices.

Standardized Module Library

The platform provides an extensive library of pre-built, production-ready modules covering the complete spectrum of AWS infrastructure components:

  • Compute Services: EC2, ECS, EKS, Lambda
  • Data Stores: Aurora, RDS, DynamoDB, DocumentDB, Redis, Elasticsearch
  • Networking: VPC, Load Balancers, API Gateway, Route53
  • Security: IAM, ACM, Secrets Manager
  • Messaging: SQS, Kafka
  • Monitoring: CloudWatch, Managed Grafana

Each module encapsulates best practices, security controls, and compliance requirements, ensuring consistent infrastructure deployment across the organization.

Developer Experience

Simplified Workflow

Gaia integrates seamlessly with existing development workflows through GitHub. Developers request infrastructure by:

  1. Creating a configuration file with simple key-value pairs
  2. Submitting a pull request
  3. Awaiting automated validation and deployment

Example configuration for a serverless function with a storage layer and an API Gateway:

The API Gateway configuration for the API is very simple:

Once the developer is ready to create the infrastructure a Pull Request is created and a “code owner” (in this case a Platform Engineer Team Member) approves the request and the infrastructure is deployed automatically.

Automated Compliance

The platform automatically enforces organizational standards and security policies. Developers don’t need to worry about:

  • Network configuration
  • Security group settings
  • Access control policies
  • Compliance requirements

All these aspects are handled automatically by Gaia’s pre-configured modules.

Technical Architecture

Terragrunt Integration

Gaia leverages Terragrunt as a wrapper around Terraform to provide enhanced functionality:

  • Automatic variable injection based on environment context
  • Template-based module generation
  • Configuration reuse across environments
  • Simplified state management

Monitoring and Observability

The platform includes native integration with monitoring tools:

  • Automated Datadog dashboard creation
  • Standardized monitoring configurations
  • Built-in health checks and alerts
  • Custom metric collection

Organizational Impact

DevOps Transformation

  • Reduced manual infrastructure work by approximately 80%
  • Shifted focus from repetitive tasks to platform improvements
  • Enabled scaling of development operations without proportional increase in DevOps resources

Development Velocity

  • Eliminated infrastructure provisioning bottlenecks
  • Reduced time-to-deployment for new projects
  • Enabled consistent implementation across teams

Governance and Security

  • Centralized policy enforcement
  • Automated compliance checking
  • Infrastructure drift detection and remediation
  • Standardized security controls

Future Directions

Multi-Cloud Strategy

While currently focused on AWS, Gaia is being extended to support Azure. This expansion presents unique challenges due to fundamental differences in how cloud platforms implement similar services. The team is working to maintain the same simple developer experience while adapting to Azure’s distinct architecture.

Platform Evolution

Planned enhancements include:

  • Enhanced monitoring capabilities
  • Expanded multi-cloud support
  • Deeper integration with development tools
  • Advanced automation features

Conclusion

Gaia represents a successful transformation from traditional DevOps to platform engineering. By providing developers with self-service infrastructure capabilities while maintaining security and compliance, the platform has eliminated a major organizational bottleneck. The success of this approach demonstrates how well-designed abstractions and automation can make infrastructure management accessible to development teams while maintaining enterprise-grade standards.

The platform has fundamentally transformed how Carlsberg manages cloud infrastructure. As cloud infrastructure continues to evolve, Gaia’s modular architecture and focus on developer experience position it well for future adaptations and enhancements. The platform serves as a testament to how modern platform engineering can effectively bridge the gap between development velocity and operational excellence.

Gaia was conceived by https://www.linkedin.com/in/josesganjos/ and built with the help from:

Beyond DevOps: The Rise of Full-Stack Platform Engineering

2024-11-04 /

The Evolution of Infrastructure Management

DevOps promised to bridge the gap between development and operations, aiming to deliver infrastructure faster and more efficiently. However, in many organizations, the reality often fell short of this ideal. DevOps frequently became a practice where operations teams learned to script infrastructure without fully embracing key software engineering principles. It became more about scripting than true engineering.

The Need for a Higher Abstraction

As infrastructure needs grew more complex, it became clear that traditional DevOps approaches were not scaling effectively. Tools like Terraform, while powerful, often proved to be terse and not particularly developer-friendly. They got the job done, but they weren’t providing the streamlined experience that developers needed. A new approach was necessary – one that would raise the level of abstraction and make infrastructure more accessible.

The Golden Path as a Product

Enter the concept of the “golden path” – a set of pre-built, standardized infrastructure solutions that developers can easily use and customize. This approach treats infrastructure as a product, designed with the end-user – the developer – in mind. 

The golden path isn’t just a set of scripts or configurations; it’s a carefully crafted product that encapsulates best practices, security considerations, and organizational policies. It automates infrastructure creation while maintaining alignment with company standards, allowing developers to provision cloud resources without needing to worry about governance, security, or configuration inconsistencies.

Raising the Abstraction Level

To understand the significance of this shift, consider this analogy: Terraform, while powerful, is often like the assembly language of infrastructure. Platform engineering, and the golden path approach, is about raising that abstraction, creating reusable and maintainable infrastructure solutions that developers can work with seamlessly. 

Just as high-level programming languages made software development more accessible and efficient compared to assembly language, the golden path aims to do the same for infrastructure management. By creating higher-level abstractions, we’re making infrastructure more understandable, manageable, and aligned with modern software development practices.

The Role of Full-Stack Platform Engineers

This new approach requires a new kind of professional: the full-stack platform engineer. These engineers think like developers while solving infrastructure challenges. They build scalable, reliable, and developer-friendly infrastructure that empowers teams.

Full-stack platform engineers focus on creating robust, scalable infrastructure solutions that directly support business needs, rather than getting bogged down in low-level configuration details. They apply the same rigor expected in software development to infrastructure design, treating infrastructure truly as code.

Enhancing Developer Experience and Security

The golden path approach significantly enhances the developer experience. By integrating infrastructure provisioning directly into familiar development workflows (like those in GitHub), it allows developers to request and manage infrastructure as part of their normal process, without delays or context switching.

This approach also allows for the seamless integration of security practices. By baking security considerations into the golden path from the start, organizations can shift security left in the development process, addressing vulnerabilities at their source without compromising developer productivity.

A New Era of Infrastructure Management

The rise of full-stack platform engineering and the golden path approach represents a significant evolution in how we think about and manage infrastructure. It’s not just DevOps 2.0; it’s a fundamental shift in mindset that treats infrastructure as a product designed for developer success.

By raising the abstraction level, applying software engineering principles to infrastructure, and focusing on creating reusable, maintainable solutions, this approach promises to make infrastructure more accessible, secure, and aligned with modern development practices. As organizations continue to grapple with increasing complexity, the golden path offers a way forward – empowering developers, enhancing security, and ultimately accelerating innovation.

At Carlsberg, this approach has been embodied in Gaia, our golden path platform built by full-stack platform engineers. Gaia exemplifies how treating infrastructure as a product can transform development processes, making them more efficient and developer-friendly. It stands as a testament to the power of full-stack platform engineering in creating solutions that truly serve the needs of modern development teams.

As more organizations embrace this shift, we can expect to see a new landscape of infrastructure management emerge – one where the golden path, crafted by skilled full-stack platform engineers, leads the way to more innovative, secure, and efficient software development practices.

DevEx Analysis: Software Engineering in Growth Products, Carlsberg vs Gartner Benchmarks

2024-10-15 /

Introduction

This report was created by having ChatGPT (Auto) and Claude individually compare and evaluate the initiatives on DevEx that I’ve blogged about in the past against the official Gartner DevEx Best Practices. I then had ChatGPT combine the two results into the report below.

Report

This report evaluates the Developer Experience (DevEx) initiatives within the Software Engineering department in Growth Products at Carlsberg, comparing them against Gartner’s The State of Developer Experience Initiatives report. The initiatives reviewed apply specifically to the Software Engineering team within Growth Products and do not reflect Carlsberg’s global operations.

1. Focus on Tooling and Integration

The Software Engineering team in Growth Products demonstrates strong alignment with Gartner’s recommendations in the area of tooling and integration:

Key Highlights:

  • Emphasis on integrated platforms and tooling, such as the Gaia platform
  • Focus on automation to streamline developer workflows
  • Adoption of AI-driven tools like GitHub Copilot to enhance developer efficiency

The initiatives outlined in “From DevOps to Platform Engineering: How Gaia Transformed Our Approach to Infrastructure Alignment and Developer Experience” highlight how integrated tooling supports smoother workflows, a priority emphasized in Gartner’s best practices.

Final Rating: 4.5/5. The focus on improving developer workflows through integrated platforms and automation closely aligns with Gartner’s recommendations.

2. Embedding Security into the Developer Workflow

The approach taken by the Software Engineering team aligns well with Gartner’s guidance on embedding security into daily developer processes:

Key Highlights:

  • Security integration as part of the daily development workflow
  • Use of tools such as GitHub Advanced Security
  • Inclusion of security roles within the broader DevEx initiative

The best practices detailed in “GitHub Advanced Security Enables Shifting Security Left” demonstrate how security has been embedded into the development pipeline, a practice highly recommended by Gartner for mature DevEx programs.

Final Rating: 4.5/5. The integration of security into the workflow, combined with tooling that supports shifting security left, positions the Software Engineering team’s DevEx initiatives at an advanced level in this area.

3. Emphasis on Professional Growth and Autonomy

The Software Engineering team’s emphasis on professional growth, particularly in relation to senior roles, aligns strongly with Gartner’s recommendations:

Key Highlights:

  • Empowering senior developers to act as facilitators and influencers
  • Promoting autonomy while ensuring organizational alignment
  • Balancing team independence with a controlled framework

The initiatives described in “Seniority, Organizational Influence, and Participation” provide a clear view of how senior developers are encouraged to take on leadership roles that foster cross-team collaboration and influence, aligning with Gartner’s focus on supporting professional growth and autonomy.

Final Rating: 5/5. The focus on autonomy and professional growth, especially for senior developers, represents an exemplary alignment with Gartner’s DevEx principles.

4. Cultural and Organizational Alignment

The initiatives to foster a supportive and collaborative developer culture align closely with Gartner’s recommendations for cultural and organizational alignment:

Key Highlights:

  • Clear definitions of expected behaviors across the Software Engineering department
  • Promotion of psychological safety within teams
  • Fostering a collaborative and inclusive work environment

The best practices outlined in “A Manifesto on Expected Behaviors” establish clear behavioral expectations that create a psychologically safe environment for developers. Gartner stresses the importance of a supportive culture in any comprehensive DevEx program, which these initiatives directly address.

Final Rating: 4.5/5. The Software Engineering department has established a strong cultural foundation that promotes collaboration and safety. Further emphasis on psychological safety could further enhance this area.

5. Measuring DevEx

While the Software Engineering team demonstrates strength in many areas, there is room for improvement in how DevEx outcomes are measured:

Key Highlights:

  • Focus on productivity and automation metrics
  • Some discussion of tooling-related metrics, such as in “GitHub Copilot Metrics”
  • Limited use of broader DevEx metrics, such as DORA or job satisfaction surveys

The current focus on productivity and efficiency aligns with Gartner’s recommendations. However, Gartner also recommends a broader range of metrics, including velocity, developer satisfaction, and retention. Expanding the measurement approach, as suggested in these practices, would provide a more comprehensive understanding of DevEx success.

Final Rating: 3/5. Increasing the use of specific DevEx outcome metrics, such as DORA and job satisfaction, would enhance the ability to measure and track the effectiveness of DevEx initiatives within the Software Engineering department.

Overall Rating: 4.3/5 (Strong Alignment)

The Software Engineering department’s DevEx efforts demonstrate strong alignment with Gartner’s best practices, particularly in the areas of tooling, security integration, autonomy, and culture. The main area for improvement is in the explicit measurement and tracking of DevEx outcomes through broader metrics.

Recommendations

Incorporating Gartner’s benchmarks, the following recommendations are suggested for the Software Engineering department in Growth Products:

  1. Implement More Specific DevEx Metrics: Expanding the use of DORA metrics, job satisfaction surveys, and other developer feedback mechanisms would provide deeper insights into the effectiveness of the DevEx initiatives.
  2. Continue Refining Integrated Tooling: Maintaining the momentum around tools like Gaia and GitHub Copilot will ensure the DevEx program continues to drive productivity and reduce friction in the development workflow.
  3. Sustain Focus on Security Integration and Culture: The strengths in security integration and developer culture should remain a focus, ensuring that these foundational pillars of the DevEx program continue to evolve.
  4. Consider Formalizing the DevEx Program: If not already formalized, creating a structured DevEx program with clear goals, metrics, and accountability would allow for better tracking of progress and outcomes.
  5. Regularly Assess and Iterate on DevEx Initiatives: Using feedback loops and metrics to continuously iterate on the DevEx approach will ensure that it evolves in line with both developer needs and organizational goals.

Conclusion:

The DevEx initiatives within the Software Engineering department in Growth Products at Carlsberg are well-aligned with Gartner’s best practices, particularly in terms of integrated tooling, security, professional growth, and cultural alignment. By expanding the metrics framework to include broader DevEx outcome measurements, the program could reach even higher levels of maturity and effectiveness.

Seniority, Organizational Influence and Participation

2024-10-15 /

The table below is meant to better explain how seniority, organizational influence and behaviors are connected. Anyone wanting to move up the career ladder must master our expected behaviors and participate accordingly.

StrengthEntryIntermediateExperiencedAdvancedExpert
ScopeEntry level professional with limited or no prior experience; learns to use professional concepts to resolve problems of limited scope and complexity; works on assignments that require limited judgment and decision making. Developing position where an employee is able to apply job skills, policies and procedures to complete tasks of moderate scope and complexity to determine appropriate action.Journey-level, experienced professional who knows how to apply theory and put it into practice with full understanding of the professional field; has broad job knowledge and works on problems of diverse scope.Professional with a high degree of knowledge in the overall field and recognized expertise in specific areas.Leader in the field who regularly leads projects of criticality to company and beyond, with high consequences of success or failure.  This employee has impact and influence on company policy and program development. Barriers of entry exist at this level.
AnalogyLearning about rope and knotsCan tie basic knots, learning complex knotsCalculates rope strength, knows a lot about knotsUnderstands rope making, can tie any knotKnows more about rope than you ever will, invented new knot
Influence & ImpactSelfPeersTeamDepartmentCompany
Coding100%100%80%-100%<25%<10%
ParticipationActively participate in discussions and team activities. Seek guidance from more experienced team members. Be open to receiving feedback and learning from mistakes.Contribute to team discussions and share knowledge gained from learning basic concepts. Offer assistance to junior team members. Seek feedback on performance and actively work on improving skills.Actively contribute expertise to team projects and discussions. Mentor junior team members and facilitate knowledge sharing sessions. Regularly seek out new learning opportunities and share insights with the team.Lead collaborative learning initiatives within the team. Actively contribute to the development of best practices and processes. Mentor and coach less experienced team members, fostering a culture of continuous learning.Serve as a subject matter expert, providing guidance and direction on complex projects. Spearhead innovative learning initiatives and contribute to industry knowledge sharing. Act as a mentor and coach for both technical and professional development.

A Manifesto on Expected Behaviors

2024-10-15 /

Introduction

In Software Engineering at Carlsberg our collective success is built on the foundation of individual behaviors that foster a positive, innovative, and collaborative environment. This document outlines the expected behaviors that every team member, irrespective of their role or seniority, is encouraged to embody and develop. These behaviors are not just guidelines but the essence of our culture, aiming to inspire continuous growth, effective communication, and a proactive approach to challenges. As we navigate the complexities of software engineering, these behaviors will guide us in making decisions, interacting with one another, and achieving our departmental and organizational goals. Together, let’s build a culture that celebrates learning, teamwork, and excellence in everything we do.

Learn together, grow together

“We embrace collaboration, share knowledge openly, and celebrate both individual and team success.”

  • Contribute and support: Actively participate in discussions, offer help, and celebrate each other’s successes.
  • Give and receive feedback: Regularly seek and provide constructive feedback for improvement.
  • Share expertise openly: Willingly share knowledge and expertise to benefit the team.

Communicate clearly, connect openly

“We foster understanding through respectful, transparent, and active communication using the right tools for the job.”

  • Listen actively, engage thoughtfully: Pay close attention, ask questions, and respond thoughtfully to diverse perspectives.
  • Clarity over jargon, respect in tone: Communicate with clarity, avoid technical jargon, and use respectful language.
  • Prompt and appropriate: Respond efficiently and tailor your communication to fit the situation and audience.
  • Choose the right channel: Utilize appropriate communication methods based on the message and context.

Continuous Learning and Improvement is the Way!

“We value continuous learning, actively seek opportunities to improve, and celebrate progress together.”

  • Quality First, Every Step of the Way: Never pass a known defect down the stream. If you see something which will cause problems for others then you should stop the work.
  • Challenge yourself and learn: Regularly seek new experiences and reflect on your experiences to improve.
  • Experiment and share: Be open to trying new things and share your learnings with the team.
  • Track your progress: Regularly measure your progress towards goals and adjust your approach as needed.

Own your work, drive results

“We take responsibility, proactively solve problems, and seize opportunities to excel.”

  • Embrace challenges, deliver excellence: Aim for impactful work and go the extra mile for outstanding results.
  • Be proactive problem-solvers: Actively seek, address, and prevent the escalation of challenges by ensuring solutions not only fit within established boundaries but also uphold the highest quality standards.
  • Learn and bounce back: Embrace mistakes as learning opportunities and quickly recover from setbacks.

The Intersection of DevEx and DevSecOps: We need a New Way Forward

2024-10-14 /

Developer Experience (DevEx) is critical for productivity, impact and retaining talent. In a world where software engineers are constantly asked to deliver more, faster, and more securely, companies can’t afford to treat DevEx and DevSecOps as separate priorities.

When these areas are siloed, we end up with fragmented workflows, frustrated developers, and disjointed experiences—counteracting the benefits of initiatives like unified development platforms. To move forward, we need an integrated approach to DevEx and DevSecOps, making security a seamless part of the development process while avoiding the fragmentation that current approaches have caused.

The current fragmented approach to DevSecOps is undermining Developer Experience. DevEx and DevSecOps serve different purposes, but poorly implemented DevSecOps practices can harm DevEx, reducing efficiency and developer satisfaction. It’s about ensuring security practices support developer productivity rather than interfere with it.

The Fragmentation Problem: A Warning for Growing Complexity

As organizations scale, it’s easy to fall into the trap of adding more tools to address new challenges—especially in security. Each new vulnerability or compliance requirement often results in adopting yet another tool. On the surface, this might seem like progress, but in reality, it adds complexity.

Each new platform comes with its own requirements, logins, and signals. Developers must toggle between different tools, piecing together information from multiple sources. This disrupts their workflow and increases the risk of errors. The very tools intended to improve security end up creating friction.

This fragmented approach seems common in many organizations. As more platforms are introduced, workflows become disjointed, and maintaining a unified process becomes harder. The result? Security becomes reactive, and developers spend less time building and improving software.

We need to rethink how we integrate security into the development process. A consolidated approach can help avoid these pitfalls while enhancing both security and productivity.

Our Success with Platform Consolidation: Improving Security and Developer Experience

At Carlsberg, we took a deliberate approach to consolidating our software development tools onto a single platform—GitHub—and used GitHub Advanced Security (GHAS) to shift security left into the developer workflow. This allowed us to address security vulnerabilities at their source, directly within the tools developers are already familiar with.

By integrating security into the developer workflow, our developers could use AI-powered tools like GitHub Copilot to write more secure code as they worked. This approach streamlined the process, reducing the need for developers to toggle between multiple platforms and ensuring that the code we deployed was free from known security vulnerabilities at the time of writing. The impact on Developer Experience (DevEx) has been significant—security is now a natural part of the development process, not an afterthought.

This consolidation not only raised our security posture but also improved developer productivity. By reducing context-switching and embedding security into the natural flow of work, we created a more cohesive, efficient development environment where developers felt empowered to take ownership of both the code and its security.

The Opposite Trend in DevSecOps: Tool Fragmentation and Complexity

While we’ve seen success in consolidating our platform and raising both security and Developer Experience, it’s the norm for many organizations to face the opposite challenge. When implementing DevSecOps, the introduction of more security tools often leads to a fragmented workflow. Developers are required to interact with multiple platforms, each with its own set of logins, signals, and processes, which disrupts their focus and lowers productivity.

Research has shown that this tool-centric approach to DevSecOps can lead to operational gaps, inefficiencies, and a disjointed developer experience. The very tools designed to improve security end up creating friction, making it harder for developers to get their work done. In addition, the immaturity of some automated DevSecOps tools further complicates integration into continuous delivery pipelines, undermining both security and efficiency.

This fragmentation isn’t specific to any one organization; it’s a widespread challenge as security teams strive to keep up with growing threats and compliance demands. The proliferation of tools, however, often leads to more silos and increased complexity—exactly the opposite of what we’ve achieved through platform consolidation.

A Call for Streamlining DevSecOps: Learning from Consolidation

The lesson here is clear: adding more tools to the mix isn’t the answer. To fully realize the potential of DevSecOps, we need to move away from tool fragmentation and focus on embedding security into the developer workflow, as we did with our consolidated platform on GitHub. By simplifying the development process and integrating security from the start, we can achieve better outcomes for both security and Developer Experience.

Security needs to be central, not an afterthought. Rather than bolting on security measures after the fact or adding layers of complexity with new tools, security should be a seamless part of how developers work. By making security a core aspect of the development process, we ensure that it is baked in from the very beginning. This approach not only improves security itself but also enhances the overall Developer Experience by reducing the friction and overhead often associated with traditional security processes.

References

1. DevSecOps People: “Identifying the Primary Dimensions of DevSecOps: A Multi-vocal Literature Review,” discusses the fragmentation of DevSecOps and the challenge of integrating multiple tools into a seamless workflow. https://www.sciencedirect.com/science/article/pii/S0164121224001080

2. AI for DevSecOps: A Landscape and Future Opportunities: This paper outlines the potential of AI in automating and enhancing security tasks within DevSecOps pipelines, but also highlights challenges around tool complexity and immaturity. https://arxiv.org/abs/2404.04839

From DevOps to Platform Engineering: How Gaia Transformed Our Approach to Infrastructure, Alignment, and Developer Experience

2024-10-05 /

Introduction

In the world of cloud development, managing infrastructure effectively while maintaining alignment across teams is a constant challenge. Historically, our DevOps team played a pivotal role in provisioning and managing cloud resources, ensuring developers had what they needed to build and deploy solutions. However, this model wasn’t sustainable as the number of projects grew and cloud environments became more complex. We needed a way to streamline infrastructure management without losing sight of alignment across teams and solutions, while also improving the overall Developer Experience (DevEx).

This realization led us to shift our DevOps team from a traditional support role into a platform engineering team, focused on building and maintaining tools that provide a golden path for developers. The result? Gaia—a platform that has radically transformed how we manage cloud infrastructure, maintain alignment throughout the organization, and drastically improve Developer Experience by embedding infrastructure creation into developers’ existing workflows.

The Evolution from DevOps to Platform Engineering

When we started, our DevOps team handled infrastructure provisioning manually and on a request basis. While this ensured quality control, it created bottlenecks as the number of requests grew, leading to slower project deliveries. Developers were often left waiting for infrastructure to be set up, while the DevOps team struggled to keep up with the workload.

This wasn’t a scalable model, so we pivoted. Rather than manually provisioning infrastructure, we built Gaia—a platform that automates infrastructure creation while maintaining alignment with company policies. Gaia represents our “golden path”—a set of pre-built modules that allow developers to provision cloud resources without needing to worry about governance, security, or configuration inconsistencies.

Not only did Gaia eliminate bottlenecks, but it also integrated directly into the GitHub workflow developers were already using, significantly improving Developer Experience. Developers now interact with the same tools they use for coding, making infrastructure requests feel like a natural extension of their development work.

The Remarkable Impact of Gaia on Developer Experience

Gaia’s impact has been nothing short of remarkable. By automating the infrastructure creation process, we’ve effectively removed the need for the DevOps team to manually create infrastructure for developers. Developers now have a self-service capability to quickly and easily provision what they need on their own, directly from within their existing GitHub workflows, without waiting for approval or intervention from the DevOps team.

This seamless integration has significantly improved Developer Experience in several key ways:

  • Familiarity: Developers don’t have to learn new tools or processes to request infrastructure. They use GitHub, the platform they are already familiar with, ensuring minimal friction when interacting with infrastructure.
  • Speed and Efficiency: With Gaia, infrastructure requests are submitted via GitHub pull requests (PRs), allowing developers to spin up resources quickly. This eliminates the lag time that often occurs when requests are handled through manual ticketing systems.
  • Embedded Governance: Developers no longer have to worry about compliance or governance rules. Every infrastructure resource created via Gaia is automatically aligned with company policies, freeing developers to focus entirely on building solutions without getting bogged down in regulatory details.

By embedding infrastructure creation into the developer workflow through GitHub, Gaia significantly boosts DevEx. Developers are empowered to take control of infrastructure setup, while still benefiting from built-in quality and governance checks that ensure alignment with the company’s standards.

The New Focus of Our Platform Engineering Team

With manual infrastructure creation largely eliminated, the role of the DevOps team has shifted to that of a platform engineering team. Their primary focus is now on maintaining Gaia and the shared modules that are used to provision infrastructure. Whenever new infrastructure resources or cloud services are introduced, the team ensures they are incorporated into Gaia in a way that adheres to company policies, ensuring alignment as our cloud architecture evolves.

This centralized approach allows the platform engineering team to ensure that the development process is as smooth as possible, enhancing the overall Developer Experience by constantly improving the tools developers rely on. Developers no longer need to spend time learning about the intricacies of cloud infrastructure or worry about whether their configurations meet governance requirements.

Integrating Infrastructure Creation into the Developer Workflow

One of the most significant achievements of Gaia is how seamlessly it integrates into the developer workflow. As mentioned, we built Gaia to work within a central repository in GitHub, where developers create pull requests to request infrastructure. These PRs are then reviewed and approved by the platform engineering team, ensuring that every infrastructure change aligns with company policies and best practices.

By embedding infrastructure creation into the PR process, we’ve achieved several goals:

  • Speed: Developers can request infrastructure as part of their normal workflow, without delays or waiting for separate approvals.
  • Quality Control: The PR process provides a natural checkpoint for the platform engineering team to ensure consistency and alignment across all teams and solutions.
  • Alignment: Centralizing infrastructure requests in a single repository ensures that all teams are working from the same set of standards, preventing silos and ensuring that every team follows best practices.
  • Enhanced Developer Experience: Since developers no longer need to switch between tools or wait for external teams, the process feels fluid and integrated. This reduces the cognitive load on developers and enables them to focus more on writing code and building features rather than managing infrastructure logistics.

Gaia’s GitHub-based process has streamlined how developers interact with infrastructure, further aligning infrastructure creation with developer workflows and enhancing their experience by reducing friction and improving productivity.

Conclusion

The transition from a traditional DevOps model to a platform engineering team centered around Gaia has been a game changer for us. By providing developers with a golden path for creating infrastructure, we’ve freed up their time to focus on what they do best: building innovative software solutions. At the same time, we’ve ensured that every infrastructure deployment is aligned with our policies and governance frameworks, without the need for constant oversight.

Gaia has made our infrastructure provisioning faster, more reliable, and more scalable, while allowing our platform engineering team to focus on higher-level work—maintaining the tools that enable this. By embedding infrastructure creation into GitHub workflows, we’ve also enhanced Developer Experience, making infrastructure provisioning a natural extension of the development process.

The future of DevOps, for us, lies in platform engineering, where teams enable developers rather than managing infrastructure requests. Alignment and Developer Experience are no longer afterthoughts—they’re built into the process, ensuring that as we scale, we do so efficiently, consistently, and with a developer-centric approach.

Gaia was built by:

Balancing Autonomy and Alignment in Engineering Teams

2024-10-01 /

The Spotify model has often been referenced as a way to structure engineering teams for agility and independence. It promotes business-owned product teams, where engineers report into product owners, and uses guilds to ensure that teams stay aligned on best practices. However, guilds often become more like “book clubs,” where participation is optional and relies on personal time. This happens because business line managers prioritize deliverables over cross-organizational collaboration, making it difficult to maintain alignment at scale.

Meanwhile, Team Topologies offers a different focus, looking at how different types of teams interact and organize. It doesn’t rely on guilds but instead emphasizes reducing dependencies and clarifying team responsibilities.

One of the main reasons I organize engineers into a single reporting line, rather than under product ownership, is to avoid these pitfalls. By centralizing the reporting structure, I can prioritize time for engineers to focus on cross-organizational standards and collaboration, ensuring alignment across teams without relying on optional participation.

The Importance of Alignment and Shared Processes

While models like Spotify emphasize team independence, they sometimes miss the mark on alignment. It’s critical that teams don’t end up siloed, each solving the same problems in different ways, or worse, working against established company practices. This is where alignment on best practices, methods, and tools becomes crucial.

Take the US Navy SEAL teams as an example. They are known for their ability to operate independently, much like Scrum teams. However, what people tend to overlook is that all SEAL teams undergo the same training, use the same equipment, and follow standardized methods and processes. This shared foundation is what allows them to operate seamlessly when they come together, even though they work independently most of the time.

In the same way, my approach ensures that engineering teams can solve problems on their own, but they’re aligned on best practices, tools, and processes across the company. This alignment prevents the issues often seen in the Spotify model, where teams risk becoming too focused on their own product work, losing sight of the bigger organizational picture.

Scrum Teams Need Independence from Authority

In Scrum teams, the issue goes beyond just estimation—it’s about the entire collaboration model. Scrum is designed to foster equal collaboration, where team members work together, estimate tasks, and solve problems without a hierarchy influencing decisions. When someone on the team, such as a Product Owner, is also the boss, this balance is broken. The premise of Scrum, which relies on collective responsibility and open communication, collapses.

If the Product Owner or any other leader on the team has direct authority over the others, it can lead to a situation where estimates are overridden, team members feel pressured to work longer hours, and decisions are driven by power dynamics rather than collaboration. This undermines the core principles of Scrum, where the goal is for teams to self-organize and be empowered to make their own decisions.

By keeping authority structures out of the Scrum team, we ensure that collaboration is truly equal, and that decisions are made based on the team’s expertise and collective input—not on the directives of a boss.

How We Balance Autonomy and Alignment

Instead of organizing engineers strictly around product owners and budgets—like in the Spotify model—we’ve created a framework where engineers report through a central engineering line. This keeps everyone on the same page when it comes to methods and processes. Engineers still work closely with product teams, but they don’t lose sight of the bigger picture: adhering to company-wide standards.

This approach solves a problem common in both the Spotify and Team Topologies models. In Spotify, squads may go off and build things their way, leading to inconsistencies across the organization. In Team Topologies, stream-aligned teams can become too focused on optimizing their flow, which sometimes means diverging from company-wide practices. By maintaining a central engineering line, we keep our teams aligned while still giving them the autonomy they need to innovate and move quickly.

The Result

Our approach strikes a balance. Teams are free to innovate and adapt to the challenges of their product work, but they aren’t reinventing the wheel or deviating from best practices. We’ve managed to avoid the pitfalls of silos and fragmented processes by ensuring that every team operates within a shared framework—just like how SEAL teams can work independently, but they all share the same training, tools, and methods.

At the end of the day, it’s not about limiting autonomy; it’s about creating the right kind of autonomy. Teams should be able to act independently, but they should do so in a way that keeps the organization moving in the same direction. That’s the key to scaling effectively without losing sight of what makes us successful in the first place.

GitHub Copilot Probably Saves 50% of Time for Developers

2024-08-14 /

Introduction

Recently GitHub released the GitHub Copilot Metrics API which provides customers the ability to view how Copilot is used and as usual someone created an Open Source tool to view the data: github-copilot-resources/copilot-metrics-viewer.

So let’s take a look at the usage of Copilot in Software Engineering in Carlsberg from end of May to end of June 2024.

I’m focusing on the following three metrics:

  • Total Suggestions
  • Total Lines Suggested
  • Acceptance Rate

As I think they are useful for understanding how effective Copilot is and I would like to get closer to an actual understanding of the usefulnes of Copilot rather than the broad statement offered by both GitHub and our own developers that it saves 50% of their time.

The missing data in the charts is due to an error in the GitHub data pipeline at the time of writing and data will be made available at a later stage.

The low usage in the middle of June is due to some public holidays with lots of people taking time off.

Total Suggestions

Total Lines Suggested: Showcases the total number of lines of code suggested by GitHub Copilot. This gives an idea of the volume of code generation and assistance provided.

Total Lines Suggested

Total Lines Accepted: The total lines of code accepted by users (full acceptances) offering insights into how much of the suggested code is actually being utilized incorporated to the codebase.

Acceptance Rate

Acceptance Rate: This metric represents the ratio of accepted lines to the total lines suggested by GitHub Copilot. This rate is an indicator of the relevance and usefulness of Copilot’s suggestions.

Conclusion

The overall acceptance rate is about 20% which resonates with my experience as Copilot tends to either slightly miss the objective and/or be verbose so that you have to trim/change a lot of code. So if Copilot suggests 100 lines of code you end up accepting 20.

Does this then align with the statements from developers in Software Engineering and GitHub which claim that you save 50% of time using Copilot?

Clearly reviewing and changing code is faster than writing, so even if you end up only using 20% of the suggested code, you will save time.

Unfortunately we don’t track actual time to complete tasks in Jira, so we don’t have hard data to prove the claim.

But is the claim true? Probably – however, I’m 100% convinced that GitHub Copilot drives better Developer Experience.

GitHub Copilot drives better Developer Experience

2024-04-30 /

Introduction

This is part 3 of:

Explaining how Carlsberg unifies development on GitHub and accelerates innovation with Copilot in more detail.

By integrating GitHub Copilot into our development workflow, Carlsberg has significantly enhanced the developer experience. Copilot acts as an intelligent coding assistant, offering real-time suggestions and code completions. This seamless integration enables our developers to write more efficient and error-free code. From a business perspective, this translates to accelerated development cycles and a boost in productivity, allowing us to bring innovations to market faster and maintain a competitive edge.

Understand Code Faster

GitHub Copilot transcends simple code suggestions by providing developers with the ability to quickly understand existing codebases and even entire projects. This feature is invaluable for onboarding new team members and tackling complex legacy systems. By asking Copilot to explain intricate code, developers can rapidly grasp functionality without deep-diving into documentation or consulting peers. For Carlsberg, this means reduced ramp-up times for new projects and more efficient utilization of developer time, leading to cost savings and faster project deliveries.

Spend Less Time on Scaffolding

Scaffolding, while necessary, often consumes valuable time that could be better spent on developing business-critical features. GitHub Copilot streamlines this process by generating the foundational code structures automatically. This allows our developers at Carlsberg to concentrate on crafting the unique aspects of our solutions that drive real business value. The direct result is a more agile development process, with resources optimally allocated towards innovation and creating competitive advantages.

Lower the Learning Curve

Adopting new frameworks and technologies is a constant challenge in the fast-paced tech environment. GitHub Copilot lowers the learning curve for our developers by suggesting how to effectively use new frameworks. This guidance reduces the time spent on trial and error, enabling our team to leverage the latest technologies confidently. For Carlsberg, this capability ensures that we are always at the forefront of technology adoption, enhancing our agility and ability to respond to market changes swiftly.

Reduce Monotonous Work

Monotonous Work, like writing unit tests, though critical for ensuring code quality, can be tedious and time-consuming. GitHub Copilot addresses this by generating unit tests, which developers can then review and refine. This automation not only speeds up the development process but also ensures a high standard of code quality. At Carlsberg, leveraging Copilot for unit testing means our developers can focus more on developing features that add value to the business, while still maintaining a robust and reliable codebase.

Improve Documentation

Well-crafted documentation is crucial for maintainability and scalability but is often overlooked due to the time it requires. GitHub Copilot aids in this aspect by automatically generating meaningful comments and documentation during code commits or pull request reviews. This not only saves time but also enhances the quality of our documentation, making it easier for developers to understand and work with our code. At Carlsberg, improved documentation directly translates to reduced maintenance costs and smoother collaboration among teams, further driving operational efficiency.

Developer Experience = Productivity + Impact + Satisfaction

At Carlsberg, our integration of GitHub Copilot into our development workflow has not just been about improving individual elements of the coding process—it’s about a holistic enhancement of the overall developer experience. 

GitHub frames Developer Experience as the sum of Productivity, Impact, and Satisfaction. Here’s how Copilot aligns with these components:

  • Productivity: By automating and accelerating parts of the development cycle, Copilot directly boosts productivity. In the “Spend Less Time on Scaffolding” and “Reduce Monotonous Work” sections, we explored how Copilot streamlines tasks that traditionally consume significant time and resources. This allows our developers to focus on higher-value work, speeding up our overall project timelines and making our workflow more efficient.
  • Impact: The true measure of any tool or process change is the impact it has on the business and its goals. As discussed in “Understand Code Faster” and “Improve Documentation,” Copilot helps our team tackle complex systems more effectively and maintain better documentation. This not only enhances our current projects but also secures our long-term ability to adapt and grow, significantly impacting our operational success and market competitiveness.
  • Satisfaction: A satisfied developer is a productive and innovative developer. Through features like lowering the learning curve for new technologies and reducing the drudgery of repetitive tasks, as highlighted in “Lower the Learning Curve” and “Reduce Monotonous Work,” Copilot increases job satisfaction. This leads to a more engaged team, ready to innovate and push boundaries in pursuit of new solutions.

By investing in tools that elevate these aspects of the developer experience, Carlsberg is not just improving our software; we are fostering a culture of efficiency, innovation, and satisfaction. This commitment not only enhances our current team’s morale and output but also positions us as a forward-thinking leader in leveraging technology to drive business success.

Conclusion

GitHub Copilot has revolutionized the way we approach software development at Carlsberg, significantly enhancing the overall developer experience. By automating repetitive tasks, simplifying complex codebases, and expediting the learning process for new technologies, Copilot has allowed our developers to focus on what they do best: creating innovative solutions that drive real business value. This not only leads to a more satisfied and engaged development team but also accelerates our time-to-market and improves our competitive stance. The integration of GitHub Copilot into our workflow is a testament to Carlsberg’s commitment to leveraging cutting-edge technology to foster a culture of efficiency, innovation, and continuous improvement. It’s clear that by investing in tools that enhance the developer experience, we’re not just improving our software; we’re building a stronger foundation for our business’s future success.

« Older posts
Peter Birkholm-Buch
Søvej 6
2665 Vallensbæk Strand
Mobil 26560203
Email peterbb@gmail.com
CVR: 34513287
Bank: Nordea
Konto: 5510-0747692815
SWIFT: NDEADKKK
IBAN: DK3620000747692815
Bitcoin: 1NiBdnmFGhZN2hpM5BRNHqCSD8wpyYNvZf

Follow me on Twitter

See my profile on LinkedIn

© 2024 Peter Birkholm-Buch

Theme by Anders Noren — Up ↑