The Evolution of Infrastructure Management
In Software Engineering in Growth Products at Carlsberg, our journey towards modern infrastructure management began with a familiar challenge: as the number of development teams grew, the traditional approach of manually provisioning and managing infrastructure became a significant bottleneck. The DevOps team, tasked with building and maintaining infrastructure for multiple development teams, found themselves overwhelmed by the increasing demand for infrastructure resources.
Each new project required careful setup of networking, permissions, and cloud resources, all of which had to be manually configured by a small DevOps team. As development velocity increased and more teams came onboard, this model proved unsustainable. New projects faced delays waiting for infrastructure provisioning, while the DevOps team struggled to keep pace with mounting requests.
Reimagining Infrastructure Creation
The solution emerged when the DevOps team envisioned a different approach: what if developers could create their own infrastructure while adhering to organizational standards? The challenge was to enable self-service infrastructure without requiring developers to understand the complexities of building secure, scalable, and compliant cloud resources.
This vision led to the creation of Gaia, a platform that automates infrastructure creation while maintaining strict security and compliance standards. Built by the DevOps team, Gaia represents a fundamental shift in how infrastructure is provisioned and managed at Carlsberg.
The Platform Engineering Approach
Infrastructure as Code Evolution
Gaia elevates infrastructure creation beyond basic scripting by providing a comprehensive platform engineering solution. The platform utilizes Terraform for infrastructure provisioning but abstracts its complexity through a higher-level interface. This approach allows developers to focus on their applications while ensuring infrastructure deployments follow organizational best practices.
Standardized Module Library
The platform provides an extensive library of pre-built, production-ready modules covering the complete spectrum of AWS infrastructure components:
- Compute Services: EC2, ECS, EKS, Lambda
- Data Stores: Aurora, RDS, DynamoDB, DocumentDB, Redis, Elasticsearch
- Networking: VPC, Load Balancers, API Gateway, Route53
- Security: IAM, ACM, Secrets Manager
- Messaging: SQS, Kafka
- Monitoring: CloudWatch, Managed Grafana
Each module encapsulates best practices, security controls, and compliance requirements, ensuring consistent infrastructure deployment across the organization.
Developer Experience
Simplified Workflow
Gaia integrates seamlessly with existing development workflows through GitHub. Developers request infrastructure by:
- Creating a configuration file with simple key-value pairs
- Submitting a pull request
- Awaiting automated validation and deployment
Example configuration for a serverless function with a storage layer and an API Gateway:
The API Gateway configuration for the API is very simple:
Once the developer is ready to create the infrastructure a Pull Request is created and a “code owner” (in this case a Platform Engineer Team Member) approves the request and the infrastructure is deployed automatically.
Automated Compliance
The platform automatically enforces organizational standards and security policies. Developers don’t need to worry about:
- Network configuration
- Security group settings
- Access control policies
- Compliance requirements
All these aspects are handled automatically by Gaia’s pre-configured modules.
Technical Architecture
Terragrunt Integration
Gaia leverages Terragrunt as a wrapper around Terraform to provide enhanced functionality:
- Automatic variable injection based on environment context
- Template-based module generation
- Configuration reuse across environments
- Simplified state management
Monitoring and Observability
The platform includes native integration with monitoring tools:
- Automated Datadog dashboard creation
- Standardized monitoring configurations
- Built-in health checks and alerts
- Custom metric collection
Organizational Impact
DevOps Transformation
- Reduced manual infrastructure work by approximately 80%
- Shifted focus from repetitive tasks to platform improvements
- Enabled scaling of development operations without proportional increase in DevOps resources
Development Velocity
- Eliminated infrastructure provisioning bottlenecks
- Reduced time-to-deployment for new projects
- Enabled consistent implementation across teams
Governance and Security
- Centralized policy enforcement
- Automated compliance checking
- Infrastructure drift detection and remediation
- Standardized security controls
Future Directions
Multi-Cloud Strategy
While currently focused on AWS, Gaia is being extended to support Azure. This expansion presents unique challenges due to fundamental differences in how cloud platforms implement similar services. The team is working to maintain the same simple developer experience while adapting to Azure’s distinct architecture.
Platform Evolution
Planned enhancements include:
- Enhanced monitoring capabilities
- Expanded multi-cloud support
- Deeper integration with development tools
- Advanced automation features
Conclusion
Gaia represents a successful transformation from traditional DevOps to platform engineering. By providing developers with self-service infrastructure capabilities while maintaining security and compliance, the platform has eliminated a major organizational bottleneck. The success of this approach demonstrates how well-designed abstractions and automation can make infrastructure management accessible to development teams while maintaining enterprise-grade standards.
The platform has fundamentally transformed how Carlsberg manages cloud infrastructure. As cloud infrastructure continues to evolve, Gaia’s modular architecture and focus on developer experience position it well for future adaptations and enhancements. The platform serves as a testament to how modern platform engineering can effectively bridge the gap between development velocity and operational excellence.
Gaia was conceived by https://www.linkedin.com/in/josesganjos/ and built with the help from:
