Continuous Delivery using Gitlab CI/CD

Static website deployment is a great introduction to Continuous Delivery using Gitlab pipelines. The basic concept: use an automated process to deploy the site when commits are pushed to the origin repository in Gitlab. At this point, Gitlab uses a “runner” (server) to deploy those static files to the final destination on a remote webserver.

What is a “runner”

A runner is a server, or temporal Docker container, which is provisioned, configured, and has the Gitlab Runner software installed. In this case, it will be a docker container which is created, configured, and then runs rysnc to move files. “Shared Runners” are enabled by default on a new Gitlab project and work right out of the box. No need to further provision or use a separate server.

Per Gitlab:

Shared Runners on Gitlab.com run in auto scale mode and are powered by Google Cloud Platform. Auto scaling means reduced wait times to spin up builds, and isolated VMs for each project, thus maximizing security. They’re free to use for public open source projects and limited to 2000 CI minutes per month per group for private projects.

CI/CD file

The Gitlab convention requires a project to include .gitlab-ci.yml which automatically triggers a pipeline upon receipt of a pushed update to the Gitlab origin repo.

Deploy a Hugo site

rysnc is a great way to move the output files to the webserver. Build Hugo, point rsync at the resulting build directory, and push to changes to your remote webserver.

For example:

hugo --config production.yaml -d production
rsync -vv -rz --checksum --delete production <user>@<webserver>:/var/www/mysite/

Separating repositories

Hopefully, the theme, styling updates, and content files are already under source control. (Check out this post on creating a custom theme using Bulma, a CSS Framework.) In order to simplify the process, create a separate repository as a submodule for the public static files.

From within the parent, Hugo project:

git submodule add git@gitlab.com:<user>/<project> production

An example repository:

.
--- .git
--- (other files)
--- production
    --- .git
    --- (other folders and files)

Where production is the directory where Hugo outputs the static files.

Create the CI/CD file

The example below is a simple CI/CD file which:

1. Provisions a shared Gitlab Runner (Docker container) after a git push
2. Installs the required packages for rsync
3. Configures the ssh requirements
4. Moves the files to the webserver using rsync

deploy:
  stage: deploy
  image: alpine:latest

  before_script:
    - apk update && apk add openssh-client bash rsync

  script:
  - eval $(ssh-agent -s)
  - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
  - mkdir -p ~/.ssh
  - echo "${SSH_HOST_KEY}" > ~/.ssh/known_hosts
  - rsync -vv -rz --checksum --delete . <user>@<webserver>:/var/www/mysite/

  only:
  - master

This file will live within the public submodule.

Variables

The environmental variables referenced above need to be set within the Gitlab interface (or via the API). Browse to https://gitlab.com/<user>/<project>/-/settings/ci_cd and expand the Variables section to set SSH_PRIVATE_KEY and SSH_HOST_KEY variables.

Security Considerations

While Gitlab provides the option to “Protect” and “Mask” the variables stored, anyone with your Gitlab account has (at least) SSH access to the public webserver, and (at least) write access to your directory served.

Server Security

1. Create a separate account on the webserver for this specific purpose. (Standby for an incoming post on securing webservers!). This user should have the minimum permissions necessary.
2. Create a separate key pair used only for this specific purpose.
3. Verify the SSH host key prior to connection (using ssh-keyscan)

Gitlab Security

1. Make sure to enable 2FA on your Gitlab account before storing your private keys.
2. “Mask” the SSH_PRIVATE_KEY variable so it doesn’t appear in logs

• see https://docs.gitlab.com/ee/ci/variables/#masked-variables for more details.

1. “Protect” the SSH_PRIVATE_KEY variable so it can’t be used in other branches (unless you specifically designate otherwise)

• see https://docs.gitlab.com/ee/ci/variables/#protected-environment-variables for more details.

Commit and Push

Running:

git add .gitlab-ci.yml
git commit -m "Added CI/CD file"
git push origin master

Will trigger the pipeline on Gitlab.

Check out the project’s CI / CD -> Pipelines section and you should see the pipeline details:

Viewing the job

Digging into the job, you should see the following:

Running with gitlab-runner 12.2.0 (a987417a)
  on docker-auto-scale 0277ea0f
Using Docker executor with image alpine:latest ...
Pulling docker image alpine:latest ...
Using docker image sha256:961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4 for alpine:latest ...
Running on runner-0277ea0f-project-14197261-concurrent-0 via runner-0277ea0f-srm-1567964926-639a09e9...
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/<user>/<project>/.git/
Created fresh repository.
From https://gitlab.com/<user>/<project>
 * [new branch]      master     -> origin/master
Checking out d45eedf4 as master...

Skipping Git submodules setup
$ apk update && apk add openssh-client bash rsync
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
v3.10.2-40-gf8594b40a6 [http://dl-cdn.alpinelinux.org/alpine/v3.10/main]
v3.10.2-38-g39a872f50f [http://dl-cdn.alpinelinux.org/alpine/v3.10/community]
OK: 10334 distinct packages available
(1/12) Installing ncurses-terminfo-base (6.1_p20190518-r0)
(2/12) Installing ncurses-terminfo (6.1_p20190518-r0)
(3/12) Installing ncurses-libs (6.1_p20190518-r0)
(4/12) Installing readline (8.0.0-r0)
(5/12) Installing bash (5.0.0-r0)
Executing bash-5.0.0-r0.post-install
(6/12) Installing openssh-keygen (8.0_p1-r0)
(7/12) Installing libedit (20190324.3.1-r0)
(8/12) Installing openssh-client (8.0_p1-r0)
(9/12) Installing libacl (2.2.52-r6)
(10/12) Installing libattr (2.4.48-r0)
(11/12) Installing popt (1.16-r7)
(12/12) Installing rsync (3.1.3-r1)
Executing busybox-1.30.1-r2.trigger
OK: 18 MiB in 26 packages
$ eval $(ssh-agent -s)
Agent pid 17
$ echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
Identity added: /dev/fd/63 (/dev/fd/63)
$ mkdir -p ~/.ssh
$ echo "${SSH_HOST_KEY}" > ~/.ssh/known_hosts
$ rsync -vv -rz --checksum --delete . <user>@<webserver>:/var/www/mysite/
opening connection using: ssh -l <user> <webserver> rsync --server -vvrcze.iLsfxC --delete . /var/www/mysite/  (10 args)
Warning: Permanently added the ECDSA host key for IP address '<webserver IP>' to the list of known hosts.
sending incremental file list
delta-transmission enabled

... [redacted for brevity]

Job succeeded

Further work

This type of job is very simple but demonstrates some of the features of built in Gitlab CI/CD. We will expand on these capabilities in future posts where we build in security analysis using similar processes.