Persist project data between jobs in CircleCI

By default, CircleCI does not keep anything from a job and starts from scratch when moving to next job. That makes it pretty difficult to manage when you need to execute multiple jobs during workflow. I will show you a tip to persist project data between jobs in CircleCI.

Persist project data between jobs in CircleCI

One example use-case of the time we need to persist project is this. Starting with a Laravel or Symfony project.

1. Execute composer install to get vendor and update autoloader.
2. Build a container image with everything output from step 1 above, pack and push to a container registry, ex. gcr.io …
3. Execute Helm to deploy into managed Kubernetes clusters.

Step 3 can work just fine because it only depends on YAML config files to deploy, which can be acquired by checking out source from a git repository.

However, step 2 requires to have everything ready to be packed into a container image. 

Each step is a single job that can be reusable across multiple projects. So, as a result, we need to persist output of step 1 for step 2 apparently.

Here the Step 1:

step_1:
  working_directory: ~/project
  docker:
    - image: petehouston/docker-php-gcloud-k8s-helm:latest
  steps:
    - checkout
    - run:
        name: "Composer: install"
        command: composer install -n --prefer-dist

and Step 2:

step_2:
  working_directory: ~/project
  docker:
    - image: google/cloud-sdk
  steps:
  	- checkout
    - setup_remote_docker
    - gcp-gcr/gcr-auth:
        gcloud-service-key: GCLOUD_SERVICE_KEY
        google-compute-zone: GOOGLE_COMPUTE_ZONE
        google-project-id: GOOGLE_PROJECT_ID
    - gcp-gcr/build-image:
        registry-url: us.gcr.io
    - gcp-gcr/push-image:
        image: my-awesome-project
        registry-url: us.gcr.io

When executing workflow, after step 1, the vendor directory will be removed, as step 2 starts from a fresh new Docker image.

The tip here is to use persist_to_workspace option to keep it persistent down to next jobs. So step 1 will become like following:

step_1:
  working_directory: ~/project
  docker:
    - image: petehouston/docker-php-gcloud-k8s-helm:latest
  steps:
    - checkout
    - run:
        name: "Composer: install"
        command: composer install -n --prefer-dist
    - persist_to_workspace:
        root: ~/
        paths:
          - "project"

As you can see, I’ve added this block:

- persist_to_workspace:
    root: ~/
    paths:
      - "project"

Because everything we need is inside the working_directory we set at top, so in order to persist whole project, we need to back directory down one path and set project working directory into paths array. That’s the trick!

Then for step 2, it will become like this:

step_2:
  working_directory: ~/project
  docker:
    - image: google/cloud-sdk
  steps:
    - attach_workspace:
        at: ~/
    - setup_remote_docker
    - gcp-gcr/gcr-auth:
        gcloud-service-key: GCLOUD_SERVICE_KEY
        google-compute-zone: GOOGLE_COMPUTE_ZONE
        google-project-id: GOOGLE_PROJECT_ID
    - gcp-gcr/build-image:
        registry-url: us.gcr.io
    - gcp-gcr/push-image:
        image: my-awesome-project
        registry-url: us.gcr.io

You might notice that, I remove the - checkout step, why?

It’s very simple, since we keeps everything from step 1 down here, so the complete project is right inside ~/project; hence, checking out source code again is unnecessary. Make sense, right?

Conclusion

That’s it. It is a simple tip as I show you how to persist project data between CircleCI jobs. It will become handy when you have a lot of custom generated files during build process.