Brock Wade headshot
Brock Wade
About
Writing
Bookshelf

CircleCi Tricks

September 20, 2021

I’ve used CircleCi for personal project pipelines, and I think it’s a great tool. I like the declarative yaml templates and it’s a free option for personal use where high build concurrency is not a deal breaker.

Most software will share some similar basic requirements for a CICD pipeline - a build job bundling source code into deployable artifacts, and subsequent deploy job(s) that ship the bundled source artifacts to the appropriate destination where the code runs. Below are some notes around how I’ve approached artifact management when building CICD pipelines with CircleCi.

Single Workflow

The simplest approach to managing build files when deploying with CircleCi is to use a single single workflow + workspace pattern, though keep in mind this will be better for personal or smaller projects that don’t need many deployment environments or team level operations. Build artifacts can be persisted and retrieved between jobs within the same workflow using the workflow’s associated workspace. A workspace is automatically provisioned for each workflow and provides a filesystem available to jobs only within that workflow. Re-tried workflows will inherit the same workspace as the original workflow.

The following basic example defines a linear workflow with a build job and a deploy job. After a merge to the main branch, a successful build job that persists artifacts to the workspace will trigger a deployment job connected to the same workspace.

Complete Yaml Config

---
version: 2.1

executors:
  my-executor:
    docker:
      - image: my-image/build-agent:latest
    working_directory: ~

jobs:
  build:
    executor: my-executor
    steps:
      - checkout:
          path: ~/my-project
      - run:
          name: build
          command: ./build.sh
      - persist_to_workspace:
          root: workspace
          paths:
            - build_artifacts
  deploy:
    executor: my-executor
    steps:
      - attach_workspace:
          at: ~/workspace
      - run:
          name: deploy
          command: ./deploy.sh

workflows:
  build-deploy:
    jobs:
      - build
      - deploy:
          requires:
            - build
          filters:
            branches:
              only: main

For a comprehensive reference on CircleCi workflows, jobs, steps, and other paradigms, checkout the config dogs and/or the concept docs.

Multi Workflow

When multiple worfklows that act on the same build artifacts are necessary, another option is the artifact + environment variable pattern. This approach uses the artifacts API to upload / download the build files, and an environment variable to provide the required job number parameter. Here artifacts refers to the CircleCi artifacts primitive, which is a standalone store where build files can be persisted and accessed regardless of workflow. A CircleCi provided config step can upload artifacts and the REST API also provides upload and download operations.

The REST API v1 requirs a job number parameter on artifact download operations, so the requester must know what historical job number produced the desired artifacts. Each new job increments the current job number regardless of the workflow, so the last artifact-producing job number can be unpredictable if other workflows and jobs are running simultaneously.

This is where project environment variables come into play. A build number environment variable representing the build with the latest artifacts can be continually updated and read from any job. Future deployment jobs, regardless of workflow, can leverage this variable during download.

Note: In version 1 of the REST API, there is an API path available that gets the artifacts produced by the latest job in the project. This gets us a step closer, but it really only works for deploy jobs that run immediately after the build job producing the artifacts on a given branch. Once that deploy job runs, then its build number becomes latest.

Here’s some recipes to help implement the artifacts + environment variable pattern. In practice, replace the generic name subsitutions :vcs-type, :username, and :project with your own values.

Set latest build number environment variable

$ curl -X POST -H "Content-Type: application/json" -d "{\"name\": \"LATEST_BUILD_NUM\", \"value\": \"$CIRCLE_BUILD_NUM\"}" \
https://circleci.com/api/v2/project/:vcs-type/:username/:project/envvar -H "Circle-Token: $ARTIFACT_TOKEN"

CIRCLE_BUILD_NUM is an environment variable already made available by CircleCi at job runtime, so we’ll use that to update our LATEST_BUILD_NUM environment variable.

Download artifacts

$ curl -L -X GET "https://circleci.com/api/v2/project/:vcs-type/:username/:project/$LATEST_BUILD_NUM/artifacts" \
-H "Accept: application/json" \
-u "$API_TOKEN:" \
| grep -o 'https://[^"]*' \
| wget --verbose --header "Circle-Token: $API_TOKEN" --input-file -

Use LATEST_BUILD_NUM as an API param. The API call returns metadata information about the artifacts including download urls, which are harvested out with grep to perform the downloads with wget.

Complete Yaml Config

---
version: 2.1

executors:
  my-executor:
    docker:
      - image: my-image/build-agent:latest
    working_directory: ~

commands:
  get_latest_artifacts:
    steps:
      - run:
          name: download artifacts
          command: |
            curl -L -X GET "https://circleci.com/api/v2/project/:vcs-type/:username/:project/$LATEST_BUILD_NUM/artifacts" \
            -H "Accept: application/json" \
            -u "$API_TOKEN:" \
            | grep -o 'https://[^"]*' \
            | wget --verbose --header "Circle-Token: $API_TOKEN" --input-file -

jobs:
  build:
    executor: my-executor
    steps:
      - checkout:
          path: ~/my-project
      - run:
          name: build
          command: ./build.sh
      - store_artifacts:
          path: build_artifacts
      - run:
          name: set latest build environment var
          command: |
            curl -X POST -H "Content-Type: application/json" -d "{\"name\": \"LATEST_BUILD_NUM\", \"value\": \"$CIRCLE_BUILD_NUM\"}" \
            https://circleci.com/api/v2/project/:vcs-type/:username/:project/envvar -H "Circle-Token: $API_TOKEN"

  deploy-dev:
    executor: my-executor
    steps:
      - get_latest_artifacts
      - run:
          name: deploy to dev
          command: deploy-dev.sh 

  deploy-prod:
    executor: my-executor
    steps:
      - get_latest_artifacts
      - run:
          name: deploy to prod
          command: deploy-prod.sh

workflows:
  build-deploy-dev:
    jobs:
      - build
      - deploy-dev:
          requires:
            - build
          filters:
            branches:
              only: main
  deploy-prod:
    jobs:
      - deploy-prod:
          filters:
            branches:
              only: main


2026 Edit

While rebuilding my site in early 2026, I took another look at this article and a friendly LLM recommended the use of the CIRCLE_SHA1 variable that CircleCi injects with each job to help manage build artifacts.

The CIRCLE_SHA1 variable is the full git commit hash for the commit triggering the workflow, so it provides another way to label and identify build output that’s more source control based. It can also be helpful for rollback situations.

That being said, saving artifacts with commit hash naming conventions may mean it’s time to find an external artifact store like S3 where artifacts can be persisted long term and managed more tightly. CircleCi by default has an artifact retention of only 30 days.