Peter Birkholm-Buch

Stuff about Software Engineering

Category: Ikke kategoriseret (Page 6 of 14)

GitHub Advanced Security enables Shifting Security Left

2024-04-21 /

Introduction

This is part 2 of:

Explaining how Carlsberg unifies development on GitHub and accelerates innovation with Copilot in more detail.

In the digital transformation journey of Carlsberg, the implementation of GitHub Advanced Security (GHAS) marked a significant shift towards embedding security directly into the developer workflow. This strategic move, part of our “Security First” initiative in Software Engineering, has not only elevated the security posture of our software development lifecycle but has also manifested in substantial time and cost savings by identifying and rectifying vulnerabilities prior to deployment.

Using GHAS to Scan for Vulnerabilities

Centralizing scanning in the GHAS platform eliminates concerns about the security configurations of developer workstations and the code contributions from external parties. By conducting security scanning centrally, we ensure consistent and thorough examination of all code, regardless of its origin. This approach not only streamlines our security processes but also enhances the security posture of our software, providing peace of mind and allowing our developers to focus on innovation and productivity raising the Developer Experience.

We use GHAS to scan for the following:

  • Secrets: The inclusion of secret scanning within our workflow has been pivotal in detecting exposed secrets such as keys and tokens, preventing potential security breaches.
  • Static Application Security Testing (SAST) with CodeQL: CodeQL’s integration allows us to perform comprehensive static code analysis, identifying security vulnerabilities and coding errors at their inception.
  • Dependencies with Dependabot: Dependabot plays a critical role in our ecosystem by monitoring dependencies for known vulnerabilities and automatically suggesting updates or patches, thus maintaining the integrity of our software supply chain.
  • Binaries and Containers: By incorporating tool plugins for CodeQL we enhance our ability to scan binaries and containers for vulnerabilities, ensuring a robust security framework across all components of our software.

Integration into Developer Workflow

The transformation brought about by GHAS in our developer workflow cannot be overstated. By embedding security checks directly into pull requests, GHAS ensures that every code change is automatically scanned for vulnerabilities before being merged. This integration not only streamlines the security assessment process but also empowers developers to address security issues in real-time. The proactive security posture facilitated by GHAS equips developers with the tools and insights needed to identify and rectify potential security flaws from the outset, fostering a culture of security awareness and responsibility. This approach significantly enhances the overall security of our software projects, contributing to a more secure and efficient development environment.

Centralized Security Scanning

Adopting GHAS as a centralized platform for security scanning has provided us with a command and control center for managing vulnerabilities emanating from both source code and dependencies. The Security Center dashboard offers a comprehensive overview of vulnerabilities, CVEs, and the most affected repositories, allowing us to prioritize and focus our remediation efforts effectively.

Since the inception of GHAS in our development practices, we’ve observed a notable reduction in security vulnerabilities, with over 30000 issues addressed. This achievement underscores the effectiveness of GHAS in enhancing our security posture, demonstrating its value not only in safeguarding our applications but also in supporting our broader business objectives of innovation and growth.

Conclusion

With GHAS and shifting security left into the developer workflow we’ve achieved remarkable success eliminating 600+ secrets from source code (now down to 0 and no secrets can enter our code now) and removing more than 30000+ security vulnerabilities and using AutoFix to constantly remediate vulnerabilities.

Using GitHub as a Software Development Platform improves Developer Experience

2024-04-16 /

Introduction

This is part 1 of:

Explaining how Carlsberg unifies development on GitHub and accelerates innovation with Copilot in more detail.

Navigating Complexity: The Challenge of Multiple Development Tools

In the fast-paced world of software development, managing multiple tools can become a bottleneck that impedes efficiency and innovation. At Carlsberg, our developers and engineers were navigating a complex toolchain landscape that included GitHub, GitLab, BitBucket, Azure DevOps, Jenkins, Nexus, SonarQube, and both Azure and AWS Container Registries. This multiplicity not only slowed down our processes but also fragmented our development environment, leading to increased context-switching and security vulnerabilities.

Unified Platform: Adopting GitHub for Streamlined Operations

The decision to streamline our development tools into a single, integrated platform came as a strategic move to enhance our operational efficiency and bolster security measures. We chose GitHub as our all-encompassing platform for several reasons:

  • GitHub Source Control provides a robust system for tracking changes in computer files and coordinating work on those files among multiple people.
  • GitHub Actions makes it easier to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub.
  • GitHub Packages serves as a software package hosting service that allows you to host your software packages privately or publicly and use packages as dependencies in your projects.

By migrating from Jenkins to GitHub Actions, from SonarQube to GitHub’s integrated code scanning features, and from Nexus and various container registries to GitHub Packages, we were able to retire outdated systems and reduce our toolchain complexity significantly.

Enhanced Productivity and Security: The Benefits of Consolidation

The consolidation has profoundly impacted our software development operations by centralizing source control and unifying the development environment. Our engineers now enjoy a streamlined workflow with reduced context-switching, thanks to a centralized pipeline and improved build and deployment processes managed through GitHub Actions. The centralization of package management through GitHub Packages has also enhanced the efficiency of managing and sharing package dependencies.

By reducing our tools from nine to one, we’ve not only simplified our technology stack but also enhanced our capability to manage projects more effectively, ensuring that our software development practices continue to support Carlsberg’s legacy of innovation.

Future Focus: Continuing Innovation at Carlsberg

As we continue to refine our approach and leverage the full potential of GitHub, our focus remains on innovation and efficiency. The journey of consolidating our development tools has been a pivotal step in our digital transformation strategy, positioning Carlsberg at the forefront of technological advancement in the beverage industry.

Carlsberg unifies development on GitHub and accelerates innovation with Copilot

2024-04-16 /

I’m honored that GitHub have chosen to do a customer story on how we’re transforming software development in Carlsberg: https://github.com/customer-stories/carlsberg-group and a friggin awesome movie.

The movie was also used by Satya Nadella in the Build 2024 Keynote:

To provide a bit more background information I’ve written some posts:

The Four Tenets of SOA

2024-03-04 /

The SOA tenets originally appeared back in 2004 when  Don Box published an article on MSDN called “A Guide to developing and Running Connected Systems with Indigo” (Indigo is what’s known today as Windows Communication Foundation or WCF for short). Don Box wrote that WCF is based on SOA principles and that unlike other approaches, specifically object orientation, SOA requires a different set of assumptions:

In Indigo, a service is simply a program that one interacts with via message exchanges. A set of deployed services is a system. Individual services are built to last—the availability and stability of a given service is critical. The aggregate system of services is built to allow for change—the system must adapt to the presence of new services that appear a long time after the original services and clients have been deployed, and these must not break functionality.

Although Microsoft have pledged to keep the MSDN Magazine online at the time of writing the article linked above is not available.

Service-oriented development is based on the four fundamental tenets that follow:

  • Boundaries are explicit 
  • Services are autonomous
  • Services share schema and contract, not class
  • Service compatibility is determined based on policy 

Let’s go over what that means in terms of modern REST services.

Four Tenets

Boundaries are explicit

Services interact by sending messages across boundaries. These boundaries are formal and explicit. No assumptions are made about what is behind boundaries, and this preserves flexibility in how services are implemented and deployed.

This means that:

  • You must treat all services as external to you
  • Internal (private) implementation details should not be leaked outside of a service boundary
  • Avoid RPC interfaces because this can lead to an overuse of calls – accessing a service is not the same as accessing a local object

Services are autonomous

Services are not subservient to other code: a service reacts to a message – how that message was created and what will happen to any response the service creates is immaterial to the action that this service will take.

This means that:

  • Deploy and version services independently from the clients
  • Design contracts with the assumption that once published, they can’t be modified

Services share schema and contract, not class

Only messages pass from service to service, code does not.

This means that:

  • Contracts should be designed to be as explicit as possible to minimize misinterpretation
  • A service must be able to convert its native data types to and from some technology-neutral representation
  • The contract must be versioned using semantic versioning

Service compatibility is determined based on policy

A service must be able to express in a standard representation of policy what it does and how clients should communicate with it.

This means that:

  • The policy must be exposed using an Open API Specification

Opsætning af Newland 2D Skanner

2021-10-27 /

Nedenfor installationsvejledning til Newland 2D Stregkode Skanner Model FR27 solgt af EG (tidligere ClinicCare).

Start med at sætte skanneren i din computer i en ledig USB-A port.

Hent konfigurationsprogram fra https://www.newland-id.com/en/services/software-drivers/easyset-software. Denne guide passer til EasySet v.1.2.28 (men passer sikkert også til senere versioner).

Konfigurationsprogrammet skal pakkes ud to gange før man kan bruge det.

Højreklik på filen og vælg pak alle ud.

Når filen er pakket ud, skal vi pakke den ud een gang til.

Nu har vi så endelig adgang til selve konfigurationsprogrammet. Dobbeltklik så programmet starter.

Hvis du har Windows Defender (eller et andet virusprogram installeret) vil du (måske) få denne advarsel. Det er fordi at programmet ikke er kendt af Windows. Eneste mulighed for at komme videre er at klikke på “more info”.

Nu kan man vælge at starte programmet alligevel.

Klik på next, next, next osv.

Nu har du et program ikon på din desktop som du kan dobbeltklikke på for at starte konfigurationsprogrammet.

Vælg “Online Device”

Vælg “Configure Device”

Vælg “HID Keyboard” og sæt “Bar Code Type” til “2D” og klik på “Save to Device”.

Nu er du klar til at teste. Åbn Notepad og læs dit sundhedskort (enten plastik eller telefon) ind ved at holde det foran skanneren.

Når CPR. nummer er læst bliver det sat ind der for cursoren er placeret. Jeg har med vilje erstattet de sidste 5 tegn med en stjerne nedenfor.

Jeg synes det er vigtigt at bemærke at skanneren kun indlæser CPR. nummer og intet andet. Hvis man f.eks. bruger ClinicCare til Web vil man altså ingen fordele have af skanneren, da man ligeså godt kan bede klienten om at oplyse det.

Hvis du har fulgt denne vejledning og det virker så har du lige sparet 1000kr+moms som EG ellers tager for at give vejledning i opsætning af deres “2D skanner”.

Jeg finder det noget useriøst at EG anvender produkter fra leverandører hvor tilhørende programmel ikke kan genkendes af gængs antivirus programmel og at man i det hele taget er nødt til at installere yderligere programmel for at få en skanner til at virke. Det er jo en pengemaskine for EG.

Jeff Meyerson: “Move Fast – How Facebook Builds Software

2021-07-23 /

This book is more a string of mini interview resumes of former Facebook employees than an actual reference book with actionable information.

If you’re looking for detailed insight into how Facebook actually organizes teams and does software development then this is not a book for you.

On the other hand, if you’re looking for an easy read with a few high level but somewhat value pointers then this is for you.

1 of out 5 stars

Useful resources on Software & Systems Architecture

2021-02-12 /

I think these books are valuable references when building software systems:

I think these websites are valuable references when identifying patterns:

Azure API Management Gateway

2019-11-19 /

These are my thoughts and key take aways from Azure API Management Gateway (APIM) from working with it for some years now.

Easy to Expose Services

Whether it’s a new service built on Azure Functions or a Logic App, something existing internally or externally – it’s dead easy to expose services and secure them through APIM. In fact it’s so easy people don’t believe me when I tell them it takes minutes rather than weeks.

The builtin Dashboard and Analytics gives you a complete overview of requests, response times, errors and so on. With Application Insights you get x-ray vision into the complete end to end call stack for all requests.

Branding the (old) Developer Portal

Branding the – now almost defunct old – Developer Portal using the underlying Orchard CMS is simple and quite a lot can be achieved in a day or two. Going fancy is also a possibility if you know someone who can do a bit of HTML magic with JavaScript and CSS.

The only real problem is the underlying structuring of APIs into Products (think of a Product as a subscription for your mobile phone with different plans for talk time, data and so on) that can be a bit of a challenge when trying to present and manage APIs and Products in whatever way a customer thinks is most important for them. I hope the new portal (and an update to the Products feature in APIM with support for nesting) will make this easier.

Takes only a few days to get to above look and feel by someone who is familiar with a CMS and some HTML, CSS and JS.

Updating an API (needs better builtin support…)

Updating APIs with new methods or revisions of existing should be simple as APIM supports OpenAPI V2+3 Specifications (OAS) – so just point APIM to the OAS file and APIM will detect changes and import. However APIM is does not support this. So most of the time you’re down to a complete re-import so then just updating the change by hand is easier.

Yes APIs can be programmatically updated through the APIM Managment API but it’s not straight forward especially in multiple APIM instance scenarios and you’re then forced to write and maintain scripting to handle this.

Microsoft recently released the APIM DevOps Toolkit which aims at making it easier to move APIs between different instances of APIM. This is done using some tools which can alleviate some of the pain but it’s based on quite a bit of low level handling of configuration files in both JSON and YAML, OpenAPI, ARM Templates and some .NET Core utilities – it works but it feels like it’s introducing complexity where it should be solving the underlying problem of seamless support for updating the API methods based on an Open API specification (leave the policies to a second layer of configuration and/or store it in the SCM).

Azure DevOps with API Management Extension

I like the API Management Suite extension to Azure DevOps as it integrates directly in the release pipeline and deploys an API based on an OAS file in v2/v3 and JSON/YAML formats.

The extension also includes several predefined tasks to create/update products and versioned APIs.

I like using a repo to decouple the API team from the development team. So that the API specification can be agreed upon before the actual development takes place. The developers then implements the API according to the specification and maintains any changes to the OAS file.

API Driven Development

There are numerous definitions for API Driven Development (ADD) floating around so I’m just going to share my implementation of it.

Step 1:

  • The API specification is implemented by an “API Architect” in collaboration with the business. This ensures that externally exposed APIs are as homogenous as possible by sharing resource names, paths and data models
  • The specification is pushed to a repo in Azure DevOps

Step 2:

  • The developers implements the API according to the specification
  • The specification is updated with any additional information
  • Automated integration tests are added
  • A PR is created with changes and tests

Step 3:

  • The specification is validated against a common ruleset using Spectral
  • The API specification is deployed to APIM
  • Automatic integration tests are run
API Driven Development with Azure API Management

An API is a Product!

This is perhaps more important that whatever platform and process you use to expose the actual API to your customers.

You have to treat an API as a product – same as any other product you sell. You have to have a team to support it, take care of it, listen to customer feedback and to keep an eye on the SLA.

You can’t expose internal QA versions with limited SLA as the “test” version – everything you expose is IN PRODUCTION – period.

This is in my experience something that requires a bit of getting used to for companies just getting into the API game!

Azure Cosmos DB

2019-09-24 /

These are my thoughts and key take aways from working with Azure Cosmos DB for a while now.

Infinite Ease

Creating a database importing some data and querying it through the Data Explorer takes literally 5 minutes.

Creating an interface using Azure Logic Apps with the Cosmos connector or a proper API using Azure Functions with bindings and PowerShell is an additional 5 minutes.

Expose the API through the Azure API Management Gateway and you have a complete schema-less database and data access layer in 15 minutes.

Cosmos is hands down the most easy and accessible database technology I’ve ever worked with – period!

Infinite Scale

Cosmos is built to be infinitely scalable in terms of storage and performance.

Data is stored in storage partitions in chunks of 10GB and Cosmos automatically adds more storage partitions as needed. It works great and is seamless – only gotcha is that Cosmos for some reason won‘t scale storage partitions back to less than two which in most cases probably doesn’t matter much, but it is an issue with regards to cost. See more below.

Performance is measure in a fixed unit called Request Units per Second or RUs. 1 RU is required to read 1 KB of data and 5 RU are required to write 1KB of data. So on the surface calculating how many RUs you need should be easy. Turns out it isn’t and missing your RUs can be costly or make you loose data.

Infinite Cost

This is where infinite scale turns into infinite cost. On the surface it only seems fair that infinite scale should have a price – but controlling cost in Cosmos is difficult and completely different for what you have come to expect of the Cloud.

It’s all about those RUs.

You allocate a fixed amount of RUs either to an entire database (probably can’t recommend doing that) or to individual collections – and then you pay for those fixed RUs whether you use them or not!

Fine you think – how much can it cost in terms of RUs to write and read some data? Quite a lot in fact and there are several factors which have great influence over how many RUs are required to write and read data. I highly recommend reading this and this but it all boils down to the partition key.

The partition key is used to distribute data over logical data partitions when Cosmos writes data. The less data you write to the same logical partition as the same time – the more performant the write is.

When reading data the partition key can be used in a query to tell Cosmos where the data is stored and make reading data more performant. If you don’t know the partition key in a query then Cosmos have to read through all the data that fits the WHERE clause and that drives RUs – because reading 1 KB costs 1 RUs – so reading through 1000s of KBs of data to find that single document can cost a lot of RUs. So if you have a lot of queries without partition key and lots of spikes then you have to provision lots of RUs and pay for them 24/7.

Unfortunately there’s no auto scaling for RUs available in Cosmos even though the competitor seems to have one. However recent additions to Azure Monitor makes it possible to create your own.

Update: At Ignite 2019 CosmosDB AutoPilot was announced which is the long awaited auto scaling feature!

If you have spikes and you run of out provisioned RUs then Cosmos will throttle the requests by sending back an HTTP429 error message. It’s then up to the client to know how to handle this and perform a retry. If the client doesn’t know how, then it’s an error and the data is probably lost. Please be aware of the retries!

Infinite Possibilities

Once you’ve tackled the scaling, cost and partition keys and you start to to use Cosmos with the change feed and hook it up to the event grid – you get a completely new publish and subscribe data layer capable of replacing integration middleware, ingesting data at IoT scale and driving real time data analysis with Spark and Databricks.

Cosmos is a game changer!

Azure Logic Apps

2019-09-02 /

This is not a guide or any kind of introduction to Logic Apps but my thoughts and key take aways from working with them for a while now.

Events & Connectors

Logic Apps are great for reacting to events – especially Azure events – and I usually use them to build up the outer and more general logic of a system. This is to react to things that happens and maybe do comparison against thresholds and other configuration settings. It’s easy to call out to other services both in Azure, Office and external APIs.

As a general rule I don’t use Logic Apps to update data – I prefer to use Azure Functions for that as SDKs usually supports more advanced retries and error handling. However, some connectors supports this also – your mileage may vary.

The connectors really are the stars of Logic Apps – you can receive and act on data from pretty much any resource in the Microsoft eco system and most enterprise systems that support Azure Active Directory have connectors too.

Developer Support

Creating Logic Apps using the visual designer in the Portal and Visual Studio is a breeze – Visual Studio Code still only supports editing the underlying JSON document but will display the visual designer in read-only mode. Actual development using a repo and Azure DevOps CI/CD is a bit clunky and deployment requires an ARM template to be built using a script. Better support for storing the “code” for a Logic App in a code repository and deployment is something I’m hoping that will be added in the future.

I’ve seen integration developers use Logic Apps to quickly and easily build integration pipelines using HTTP, data parsing, conditions and service bus but they get stuck when it comes to CI/CD.

No deployment slots

Logic Apps doesn’t support deployment slots so it requires a service window of some sort when deploying in higher load scenarios. Since we’re exposing our Logic Apps through the Azure API Management Gateway we mock the requests to our Logics Apps during deployments. This is certainly not ideal and I hope that Logic Apps will get some form of deployment slot capability in the future.

Scalability

Logic Apps on paper supports reacting to and handling 1000s of events and requests per minute (we’ve done that too – Just be aware that the normal limit is 100.000 requests per 5 minutes). However if you’re not carefull then “long” running activations during high load can cause your entire App to freeze. If this happens then a manual restart can become required. If that happens then you have to start thinking about your usage of Logic Apps; the flows, branching, error handling and perhaps if moving to an Azure Function is better fit for what you’re trying to accomplish.

« Older posts Newer posts »
Peter Birkholm-Buch
Søvej 6
2665 Vallensbæk Strand
Mobil 26560203
Email peterbb@gmail.com
CVR: 34513287
Bank: Nordea
Konto: 5510-0747692815
SWIFT: NDEADKKK
IBAN: DK3620000747692815
Bitcoin: 1NiBdnmFGhZN2hpM5BRNHqCSD8wpyYNvZf

Follow me on Twitter

See my profile on LinkedIn

© 2026 Peter Birkholm-Buch

Theme by Anders Noren — Up ↑