Skip to content

Continuous Integration

Application Package Software Configuration Management

The SCM has the task of tracking and controlling changes in the software as a part of the larger cross-disciplinary field of configuration management.

SCM practices include revision control and the establishment of baselines.

The Application Package code is hosted on a repository publicly accessible (Github, Bitbucket, a GitLab instance, an institutional software forge, etc.) using one of the version control systems supported by (Subversion, Mercurial and Git)

The Application Package code include, at the top level of the source code tree, the following files:

  • README containing a description of the software (name, purpose, pointers to website, documentation, development platform, contact, and support information, …)
  • AUTHORS, a list of all the persons to be credited for the software.
  • LICENSE, the project license terms. For Open Source Licenses, the standard SPDX license names are used. For large software projects and developers, the REUSE (https://reuse.software/) process and tools can be an option to look at.
  • codemeta.json, a linked data metadata file that helps index the source code in the Software Heritage archive and provides an easy way to link to other related research outputs.

The codemeta.json includes metadata information to support the Continuous Integration phase and it is shown below:

codemeta.json
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
{
    "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
    "@type": "SoftwareSourceCode",
    "license": "https://spdx.org/licenses/CC-BY-NC-SA-4.0",
    "codeRepository": "https://github.com/Terradue/app-package-training-bids23.git",
    "dateCreated": "2022-09-01",
    "datePublished": "2022-09-25",
    "dateModified": "2022-09-25",
    "name": "Water Bodies Detection",
    "version": "1.0.0",
    "description": "The Water Bodies Detection is an Application that uses the NDWI index and the Otsu threshold to detect water bodies using Sentinel-2 or Landsat-9 data",
    "developmentStatus": "active",
    "downloadUrl": "https://github.com/Terradue/app-package-training-bids23/releases/tag/1.0.0",
    "relatedLink": [
        "https://terradue.github.io/app-package-training-bids23"
    ],
    "funder": {
        "@type": "Organization",
        "name": "Terradue"
    },
    "keywords": [
        "NDWI", "Landsat-9", "Sentinel-2", "Water Bodies"
    ],
    "programmingLanguage": [
        "Python", "CWL"
    ],
    "softwareRequirements": [
        "container runtime",
        "cwl runner"
    ],
    "author": [
        {
            "@type": "Person",
            "givenName": "Jane",
            "familyName": "Doe",
            "email": "jane.doe@acme.earth",
            "affiliation": {
                "@type": "Organization",
                "name": "ACME"
            }
        },
        {
            "@type": "Person",
            "givenName": "John",
            "familyName": "Doe",
            "email": "john.doe@acme.earth",
            "affiliation": {
                "@type": "Organization",
                "name": "ACME"
            }
        }
    ]
}

Application Package Continuous Integration

A typical Continuous Integration scenario for an Application Package includes the release of the CWL document(s) and publishing the container images to a container registry.

This is depicted below:

graph TB SCM[(software repository)] SCM -- CWL Workflow --> A SCM -- codemeta.json --> B A(validate CWL Workflow) --> B(extract version) B --> C subgraph Build containers SCM -- Dockerfiles --> C C(build container) --> D(push container) end D -- push --> CR[(Container Registry)] D -- container sha256 --> F("update Dockerpull/metadata in CWL Workflows") F -- push --> AR[(Artifact Registry)] SCM -- codemeta.json --> F

Below an example of a GitHub CI configuration implementing the scenario:

.github/workflows/build.yaml
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
name: build
on:
  push:
    branches:
    - master
    - main
    - bids23

    paths:
    # Only rebuild website when apps have changed
    - 'water-bodies/**'
    - .github/**
    - docs/**
    - cwl-workflow/*.cwl
    - codemeta.json
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v2
      with:
        python-version: 3.x
    - run: pip install cwltool
    - run: cwltool --validate cwl-workflow/app-water-bodies-cloud-native.cwl
    - run: cwltool --validate cwl-workflow/app-water-body-cloud-native.cwl
    - run: cwltool --validate cwl-workflow/app-water-body.cwl

  version:
    needs: validate

    runs-on: ubuntu-latest

    outputs:

      app-version: ${{ steps.set-version.outputs.version }}

    steps:
    - uses: actions/checkout@v2
    - run: echo "APP_VERSION=$(cat codemeta.json | jq -r .version )" >> $GITHUB_ENV
    - run: echo app version is $APP_VERSION
    - id: set-version
      run: echo "::set-output name=version::$APP_VERSION"

  container-build:

    needs: version

    runs-on: ubuntu-latest

    strategy:
      matrix: 
        step: [crop, norm_diff, otsu, stac, stage]

    steps:
    - uses: actions/checkout@v2   
    - run: echo version ${{needs.version.outputs.app-version}}
    - run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
    - name: build & push image
      run: |
        IMAGE_ID=ghcr.io/terradue/app-package-training-bids23/${{ matrix.step }}
        docker build water-bodies/command-line-tools/${{ matrix.step }} --file water-bodies/command-line-tools/${{ matrix.step }}/Dockerfile --tag ${{ matrix.step }}
        docker tag ${{ matrix.step }} $IMAGE_ID:${{needs.version.outputs.app-version}}
        docker push $IMAGE_ID:${{needs.version.outputs.app-version}}

  create-release:

    needs:
    - container-build
    - version

    runs-on: ubuntu-latest

    outputs:

      upload_url: ${{ steps.set-upload-url.outputs.upload_url }}

    steps:
      - name: release
        uses: actions/create-release@v1
        id: create_release
        with:
          draft: false
          prerelease: false
          release_name: ${{needs.version.outputs.app-version}}
          tag_name: ${{needs.version.outputs.app-version}}
        env:
          GITHUB_TOKEN: ${{ github.token }}
      - id: set-upload-url
        run: echo "::set-output name=upload_url::${{ steps.create_release.outputs.upload_url }}"

  publish-artifacts:

    needs:
    - create-release
    - version

    runs-on: ubuntu-latest

    strategy:
      matrix: 
        step: ["app-water-bodies-cloud-native", "app-water-body-cloud-native", "app-water-body"]

    steps:   
    - uses: actions/checkout@v2 
    - run: |
        for step in crop norm_diff otsu stac
        do
          tag="ghcr.io/terradue/app-package-training-bids23/${step}:${{needs.version.outputs.app-version}}"
          docker pull ${tag}
          shatag=$( docker inspect ${tag} | yq -r '.[0]["RepoDigests"][0]' )
          for cwl in $(ls cwl-workflow/*.cwl)
          do
            s="${step}" t="${shatag}" yq -i eval '(.$graph[] | select (.id == env(s)) ).hints.DockerRequirement.dockerPull = env(t)' $cwl
          done
        done
    - run: |
        for cwl in $(ls cwl-workflow/*.cwl)
        do
          r=$( cat codemeta.json | jq -r ".codeRepository" ) yq -i eval '."s:codeRepository" = {"URL" : env(r)}' $cwl
          v="${{needs.version.outputs.app-version}}" yq -i eval '."s:softwareVersion" = env(v)' $cwl
          n=$(cat codemeta.json | jq -r '(.author[0].givenName + " " + .author[0].familyName)') \
          e=$(cat codemeta.json | jq -r '.author[0].email') \
          a=$(cat codemeta.json | jq -r '.author[0].affiliation["name"]') \
          yq eval -i  '."s:author" += [{"class": "s:Person", "s.name": env(n), "s.email": env(e), "s.affiliation": env(a)}]' $cwl
        done
    - run: |
        mkdir downloads
        for cwl in "app-water-bodies-cloud-native" "app-water-body-cloud-native" "app-water-body"
        do
          cp cwl-workflow/${cwl}.cwl downloads/${cwl}.${{needs.version.outputs.app-version}}.cwl
        done
    - uses: actions/upload-artifact@v2
      with:
        name: application-package
        path: downloads
    - name: upload linux artifact 1
      uses: actions/upload-release-asset@v1
      env:
        GITHUB_TOKEN: ${{ github.token }}
      with:
        upload_url: ${{needs.create-release.outputs.upload_url}}
        asset_path: downloads/${{matrix.step}}.${{needs.version.outputs.app-version}}.cwl
        asset_name: ${{matrix.step}}.${{needs.version.outputs.app-version}}.cwl
        asset_content_type: text/yaml