Coalition isn't what you likely picture when you think of an insurance company. At Coalition, our technology provides our customers with a strategic security advantage, meaning security must be a top priority for us in our software development. To achieve our mission of solving cyber risk, we must have both high engineering velocity and a high level of security for our customers' sensitive information. Static Analysis (SAST) is one technology we have deployed to create leverage for both our engineers and security teams. In this post, I'll review how we deployed these tools to our nearly 600 different repositories which contain code written in nearly 20 languages.

Coalition uses a microservice architecture to help with load balancing and separation of responsibilities. Each microservice lives in its own GitHub repository, which enables independent service development and deployment, and maximizes the control each engineering team has over their service. The trade-off to this approach is that it greatly increases the number of technologies and languages in use, and can make regular security reviews more challenging. To help make this as easy of a process as possible, and to "shift our security left," we have implemented a centralized static analysis workflow using GitHub Advanced Security and other static analysis tools. While GitHub makes enabling code scanning and other security tools easy, it requires that we integrate the actions necessary to run those tools into each repository.

The central workflow

GitHub Actions allows repositories to leverage a custom set of steps which can be common among many repositories, courtesy of the "uses" directive:

uses: repo-owner/shared-repo/.github/workflows/my_workflow.yml@master

This means that when this part of the workflow in each repository is engaged, the "my_workflow.yml" file will be pulled from the given shared repository and executed with a preset group of parameters. This reusable workflow approach works either if the shared workflow repository is public, or if the organization is using GitHub Enterprise Cloud.

The only difference in syntax between a reusable GitHub Action and a more "traditional" GitHub Action is in the trigger (“on") section. In our reusable centralized workflow, we use:

Looking at the script above, note the use of "workflow_call" in the "on" section, instead of using "push" or "workflow_dispatch" as the trigger. As it would imply, this trigger is fired when another workflow "uses" this workflow.

We used this approach to create a central security workflow that can enable security scanning regardless of which languages and platforms an individual repository may use. This helps to reduce pull request churn for each engineering team managing their own repository by moving any potential adjustment in the security process to its own, singular location, which is subsequently shared to all callers.

In this central workflow, we take a few steps to be able to handle Coalition's huge number of configurations:

First, we use GitHub's "linguist" package to identify the prevalent languages for a given repository. We pull the "linguist" Docker image down to the GitHub Actions worker and execute it, targeting the checked out repository.

This will give us a list of languages and how prevalent they are by percentage, similar to the language overview present on each repository index page on GitHub:

From here, we can make a decision tree around the various possibilities, depending on the languages present. For example, CodeQL supports a number of languages found here. These include Python, C#, Go, Javascript, C++, Ruby, and Java. If one or more of these languages are present in the repository, we can run CodeQL against the repository and the results will be aggregated in each repository which runs that central workflow.

If there are other languages present, like Terraform, we can opt to use a tool like "tfsec" to scan those parts of the repository and report their issues in the SARIF format, which can then be integrated into GitHub's security dashboards just like the CodeQL outputs. This applies to any tool which performs static code analysis; if it can output in SARIF, it can be integrated with GitHub's security dashboards.

This approach allows us to handle any number of languages for all of our repositories from one central location. The security team can then easily enforce a consistent set of policies and level of security for all of our repositories. Similarly, if our tools are noisy and generate frequent false positives, they can be adjusted and silenced across the entire organization.

Creating a standard caller workflow

Once we've defined the centralized security action, each repository needs a simple GitHub Action that will invoke the centralized one. Since we know that we want to run this workflow whenever the primary branch for each repository is changed, we can create a standard file to be checked into each of our repositories which runs security scans when "master" or "main" branches are changed (to account for both old- and new-style branch names). We don't even need to know ahead of time which technologies are being used by the caller repository since we are using "linguist" to identify this later on.

This makes the workflow file simple, looking something like this:

This also passes in a GitHub Personal Access Token (PAT) where appropriate, which, in our case, allows us to use private repositories for our Go dependencies. Even if a repository does not define this, it will run the workflow with just an empty "github_pat" secret, indicating to the central workflow that the private repository functionality will not be used.

The above GitHub Actions workflow file can even be checked into GitHub template repositories so that when your users create new repositories from that template, they will automatically inherit the GitHub Action which runs the shared workflow.

One potential security concern in doing this is the possibility of an attacker changing the central repository and having their potential injection propagated out to every caller repository. To combat this, we use the CODEOWNERS file to enforce the approval of the AppSec team for any changes to that workflow file. This helps to ensure that someone can't sneak in a change to such a widely-used repository.

Now we have a standard file to invoke our security workflow that can be checked-in everywhere. The question becomes how to get this workflow file into each and every repository with as little manual effort as possible, since we are covering so many hundreds of repositories. Cue the "gh" command line tool and a little shell scripting magic.

Distributing the caller workflows

The "gh" command-line interface (CLI) tool is a powerful way of automating interaction with the GitHub API through an authenticated user. This tool even supports cases where the organization requires MFA to authenticate to GitHub.

We follow the rough process of:

1. Enumerate all repositories for the organization being targeted.

2. Check out the repository.

3. Analyze the languages present to make sure we support at least one of them by the central workflow; this weeds out things like documentation-only repositories which wouldn't benefit from static analysis.

4. Check if there is an existing CodeQL action present; if so, remove that workflow file as a part of the next commit.

5. Copy in the GitHub Actions standard workflow YAML file to the repository.

6. Create a new commit with this workflow file.

7. Push the commit to a new branch on each repository.

8. Create a pull request for the new branch.

This helps significantly cut down the time necessary to get the solution up and running, and allows us to scale massively to meet the requirements for our many source repositories.

Below is an example of the shell script running. Note that each commit still requires the user to check to make sure the detected options seem sane, and provide URLs to all relevant files so the user has the opportunity to verify the accuracy before the PR is created.

Benefits and conclusions

Once these workflows begin to run, we see GitHub populate CodeQL concerns in each repository's security dashboard like this:

Putting all of these together, we now have a fully scalable approach that allows us to use a standard workflow caller file which leverages a single centralized workflow to implement security in a dynamic way for many different languages and technologies. This workflow can report issues back to GitHub's security dashboard per-project and requires minimal — if any — changes per-repository, as most any change to the process would be done in a single place. This reduces the pull request churn from "600 PRs every time something changes" to "1 PR every time something changes."

GitHub Advanced Security reports three core types of security risks at this time; code scanning vulnerabilities (from CodeQL or other SARIF-compatible tools), secrets detected in code, and out-of-date dependencies. These can be aggregated from the repository, organization, or enterprise levels, giving you a one-stop shop for reviewing the security posture of your whole system. This also implements GitHub's search and filtering syntax, allowing you to narrow your observations based on if the repository is archived, has a certain naming convention, uses some topic, or any other directives supported by that syntax.

At Coalition, we plan on growing this strategy through the integration of more static analysis tools to cover more technologies. For example, we have contributed code to the Horusec project which enables the output of its report in SARIF format to integrate with GitHub's security dashboard. We believe that by adding more coverage from the viewpoints of different tools, we can stay on top of new and evolving vulnerabilities.

If you can, give this strategy a try for your own organization. Your engineering teams will thank you for minimizing the long-term effort needed to maintain rules around security policies for their code, and your security teams will thank you for creating a security standard to which you can apply all parts of your application system.