Getting started

Import the modules

In [1]:

                
                    Copied!
                    
                        
                        
                    
                    

            
from pycalrissian.context import CalrissianContext
from pycalrissian.job import CalrissianJob
from pycalrissian.execution import CalrissianExecution
import base64
import os
import yaml
from kubernetes.client.models.v1_job import V1Job
from pycalrissian.context import CalrissianContext
from pycalrissian.job import CalrissianJob
from pycalrissian.execution import CalrissianExecution
import base64
import os
import yaml
from kubernetes.client.models.v1_job import V1Job

Create the image pull secrets¶

There's one for docker.hub and one for Gitlab container registry as the CWL description to run refers container images published on those two container registries

In [3]:

                
                    Copied!
                    
                        
                        
                    
                    

            
username = ""
password = ""


auth = base64.b64encode(f"{username}:{password}".encode("utf-8")).decode(
    "utf-8"
)

secret_config = {
    "auths": {
        "registry.gitlab.com": {
            "auth": ""
        },
        "https://index.docker.io/v1/": {
			"auth": ""
		},

    }
}
username = ""
password = ""


auth = base64.b64encode(f"{username}:{password}".encode("utf-8")).decode(
    "utf-8"
)

secret_config = {
    "auths": {
        "registry.gitlab.com": {
            "auth": ""
        },
        "https://index.docker.io/v1/": {
			"auth": ""
		},

    }
}

In [4]:

                
                    Copied!
                    
secret_config
secret_config

Out[4]:

{'auths': {'registry.gitlab.com': {'auth': 'Z2l0bGFiK2RlcGxveS10b2tlbi0xODgxNzY4Oko1dHU4eXhkU3E0a19RMXpOOGhv'},
  'https://index.docker.io/v1/': {'auth': 'ZmFicmljZWJyaXRvOmRja3JfcGF0X1E3NFRFZWhBZWVsSW9BWUFLamVUcmlRTldqZw=='}}}

Take away messages about image pull secrets

they're created as a dictionary with the same structure as your ~/.docker/config file
you can use the username/password pair or the auth string

Create the CalrissianContext¶

The CalrissianContext creates a kubernetes namespace on the cluster.

Note: our kubernetes cluster uses the longhorn RWX storage class, adapt it to your cluster configuration

In [8]:

                
                    Copied!
                    
                        
                        
                    
                    

            
namespace_name = "job-namespace-n"

session = CalrissianContext(
            namespace=namespace_name,
            storage_class="openebs-kernel-nfs-scw",
            volume_size="10G",
            image_pull_secrets=secret_config,
)
namespace_name = "job-namespace-n"

session = CalrissianContext(
            namespace=namespace_name,
            storage_class="openebs-kernel-nfs-scw",
            volume_size="10G",
            image_pull_secrets=secret_config,
)

Now trigger the CalrissianContext initialisation with:

In [9]:

                
                    Copied!
                    
session.initialise()
session.initialise()

2023-03-22 11:04:26.594 | INFO     | pycalrissian.context:initialise:65 - create namespace job-namespace-n
2023-03-22 11:04:26.642 | INFO     | pycalrissian.context:create_namespace:281 - creating namespace job-namespace-n
2023-03-22 11:04:31.757 | INFO     | pycalrissian.context:create_namespace:294 - namespace job-namespace-n created
2023-03-22 11:04:31.760 | INFO     | pycalrissian.context:initialise:82 - create role pod-manager-role
2023-03-22 11:04:36.912 | INFO     | pycalrissian.context:create_role:333 - role pod-manager-role created
2023-03-22 11:04:36.914 | INFO     | pycalrissian.context:initialise:91 - create role binding for role pod-manager-role
2023-03-22 11:04:42.069 | INFO     | pycalrissian.context:create_role_binding:373 - role binding pod-manager-default-binding created
2023-03-22 11:04:42.070 | INFO     | pycalrissian.context:initialise:82 - create role log-reader-role
2023-03-22 11:04:47.228 | INFO     | pycalrissian.context:create_role:333 - role log-reader-role created
2023-03-22 11:04:47.230 | INFO     | pycalrissian.context:initialise:91 - create role binding for role log-reader-role
2023-03-22 11:04:52.400 | INFO     | pycalrissian.context:create_role_binding:373 - role binding log-reader-default-binding created
2023-03-22 11:04:52.402 | INFO     | pycalrissian.context:initialise:96 - create persistent volume claim 'calrissian-wdir' of 10G with storage class openebs-kernel-nfs-scw
2023-03-22 11:04:57.562 | INFO     | pycalrissian.context:create_pvc:416 - pvc calrissian-wdir created
2023-03-22 11:04:57.564 | INFO     | pycalrissian.context:initialise:110 - create secret container-rg
2023-03-22 11:05:02.727 | INFO     | pycalrissian.context:create_image_pull_secret:510 - image pull secret container-rg created
2023-03-22 11:05:02.730 | INFO     | pycalrissian.context:initialise:113 - patch service account

Read the CWL document¶

Now load a CWL document and create a dictionary with the parameters:

In [10]:

                
                    Copied!
                    
                        
                        
                    
                    

            
with open("../tests/app-s2-composites.0.1.0.cwl", "r") as stream:
    cwl = yaml.safe_load(stream)

params = {
    "post_stac_item": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A", # noqa: E501
    "pre_stac_item": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210703_0_L2A", # noqa: E501
    "aoi": "136.659,-35.96,136.923,-35.791",
}
with open("../tests/app-s2-composites.0.1.0.cwl", "r") as stream:
    cwl = yaml.safe_load(stream)

params = {
    "post_stac_item": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A", # noqa: E501
    "pre_stac_item": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210703_0_L2A", # noqa: E501
    "aoi": "136.659,-35.96,136.923,-35.791",
}

Take away messages

The CWL description is loaded into a Python dictionary
The parameters are a Python dictionary

So you can discover the CWL Workflow parameters with something like:

In [11]:

                
                    Copied!
                    
cwl["$graph"][0]["inputs"]
cwl["$graph"][0]["inputs"]

Out[11]:

{'pre_stac_item': {'doc': 'Pre-event Sentinel-2 item', 'type': 'string'},
 'post_stac_item': {'doc': 'Post-event Sentinel-2 item', 'type': 'string'},
 'aoi': {'doc': 'area of interest as a bounding box', 'type': 'string?'},
 'bands': {'type': 'string[]', 'default': ['B8A', 'B12', 'SCL']}}

Create the `CalrissianJob`¶

In [12]:

                
                    Copied!
                    
                        
                        
                    
                    

            
os.environ["CALRISSIAN_IMAGE"] = "docker.io/terradue/calrissian:0.12.0"
job = CalrissianJob(
    cwl=cwl,
    params=params,
    runtime_context=session,
    cwl_entry_point="dnbr",
    max_cores=2,
    max_ram="4G",
    tool_logs=True,
)
os.environ["CALRISSIAN_IMAGE"] = "docker.io/terradue/calrissian:0.12.0"
job = CalrissianJob(
    cwl=cwl,
    params=params,
    runtime_context=session,
    cwl_entry_point="dnbr",
    max_cores=2,
    max_ram="4G",
    tool_logs=True,
)

2023-03-22 11:05:17.787 | INFO     | pycalrissian.job:__init__:68 - using default security context {'runAsUser': 0, 'runAsGroup': 0, 'fsGroup': 0}
2023-03-22 11:05:17.788 | INFO     | pycalrissian.job:__init__:79 - job name: job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70
2023-03-22 11:05:17.788 | INFO     | pycalrissian.job:__init__:80 - create CWL config map
2023-03-22 11:05:22.973 | INFO     | pycalrissian.context:create_configmap:467 - config map cwl-workflow created
2023-03-22 11:05:22.974 | INFO     | pycalrissian.job:__init__:82 - create processing parameters config map
2023-03-22 11:05:28.127 | INFO     | pycalrissian.context:create_configmap:467 - config map params created

The CalrissianJob object is constructed with:

a CWL dictionary
a parameters dictionaty
a runtime context, a CalrissianContext object
the maximum number of cores the pods can use
the maximum amount of RAM the pods can use

The CalrissianJob can be serialized to a Kubernetes Job object:

In [13]:

                
                    Copied!
                    
isinstance(job.to_k8s_job(), V1Job)
isinstance(job.to_k8s_job(), V1Job)

2023-03-22 11:05:43.200 | INFO     | pycalrissian.job:_get_calrissian_container:417 - using Calrissian image: docker.io/terradue/calrissian:0.12.0

Out[13]:

True

Or to a Kubernetes Job manifest in YAML:

In [14]:

                
                    Copied!
                    
job.to_yaml("job.yml")
job.to_yaml("job.yml")

2023-03-22 11:05:44.398 | INFO     | pycalrissian.job:_get_calrissian_container:417 - using Calrissian image: docker.io/terradue/calrissian:0.12.0
2023-03-22 11:05:44.407 | INFO     | pycalrissian.job:to_yaml:143 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 serialized to job.yml

Note The Calrissian pod image can be defined with the environment variable CALRISSIAN_IMAGE

At this stage, you could do kubectl -n job-namespace apply -f job.yml to submit the job on kubernetes

Create the `CalrissianExecution`¶

In [15]:

                
                    Copied!
                    
execution = CalrissianExecution(job=job, runtime_context=session)
execution = CalrissianExecution(job=job, runtime_context=session)

Submit the job with:

In [16]:

                
                    Copied!
                    
execution.submit()
execution.submit()

2023-03-22 11:05:49.797 | INFO     | pycalrissian.execution:submit:32 - submit job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70
2023-03-22 11:05:49.800 | INFO     | pycalrissian.job:_get_calrissian_container:417 - using Calrissian image: docker.io/terradue/calrissian:0.12.0
2023-03-22 11:05:49.869 | INFO     | pycalrissian.execution:submit:38 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 submitted

Monitor the execution with:

In [17]:

                
                    Copied!
                    
execution.monitor(interval=20)
execution.monitor(interval=20)

2023-03-22 11:05:56.327 | INFO     | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active
2023-03-22 11:06:16.413 | INFO     | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active
2023-03-22 11:06:36.482 | INFO     | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active
2023-03-22 11:06:56.566 | INFO     | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active
2023-03-22 11:07:16.643 | INFO     | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active
2023-03-22 11:07:36.725 | INFO     | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active
2023-03-22 11:07:56.812 | INFO     | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active
2023-03-22 11:08:16.942 | INFO     | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active
2023-03-22 11:08:37.152 | INFO     | pycalrissian.execution:monitor:239 - execution is complete
2023-03-22 11:08:37.216 | INFO     | pycalrissian.execution:monitor:241 - the outcome is: success!

Get the execution log

In [18]:

                
                    Copied!
                    
log = execution.get_log()
print(log)
log = execution.get_log()
print(log)

INFO calrissian 0.12.0 (cwltool 3.1.20230201224320)
INFO Resolved '/workflow-input/workflow.cwl#dnbr' to 'file:///workflow-input/..2023_03_22_11_05_50.1680561479/workflow.cwl#dnbr'
WARNING Workflow checker warning:
../workflow-input/..2023_03_22_11_05_50.1680561479/workflow.cwl:9:7:  Source 'aoi' of type ["null",
                                                                      "string"] may be incompatible
../workflow-input/..2023_03_22_11_05_50.1680561479/workflow.cwl:51:9:   with sink 'aoi' of type
                                                                        "string"
INFO [workflow ] starting step node_nbr
INFO [step node_nbr] start
INFO [workflow node_nbr] starting step node_stac_2
INFO [step node_stac_2] start
INFO [step node_stac_2] start
INFO [step node_stac_2] start
INFO [step node_nbr] start
INFO [workflow node_nbr_2] starting step node_stac_3
INFO [step node_stac_3] start
INFO [step node_stac_3] start
INFO [step node_stac_3] start
INFO [workflow ] start
INFO [workflow node_nbr] start
INFO [workflow node_nbr_2] start
INFO [step node_stac_2] completed success
INFO [workflow node_nbr] starting step node_subset
INFO [step node_subset] start
INFO [step node_subset] start
INFO [step node_subset] start
INFO [step node_stac_3] completed success
INFO [workflow node_nbr_2] starting step node_subset_2
INFO [step node_subset_2] start
INFO [step node_subset_2] start
INFO [step node_subset_2] start
INFO [step node_subset] completed success
INFO [workflow node_nbr] starting step node_nbr_2
INFO [step node_nbr_2] start
INFO [step node_subset_2] completed success
INFO [workflow node_nbr_2] starting step node_nbr_3
INFO [step node_nbr_3] start
INFO [step node_nbr_2] completed success
INFO [workflow node_nbr] starting step node_cog_2
INFO [step node_cog_2] start
INFO [step node_nbr_3] completed success
INFO [workflow node_nbr_2] starting step node_cog_3
INFO [step node_cog_3] start
INFO [step node_cog_2] completed success
INFO [workflow node_nbr] completed success
INFO [step node_cog_3] completed success
INFO [workflow node_nbr_2] completed success
INFO [step node_nbr] completed success
INFO [workflow ] starting step node_dnbr
INFO [step node_dnbr] start
INFO [step node_dnbr] completed success
INFO [workflow ] starting step node_cog
INFO [step node_cog] start
INFO [step node_cog] completed success
INFO [workflow ] starting step node_stac
INFO [step node_stac] start
INFO [step node_stac] completed success
INFO [workflow ] completed success
INFO Final process status is success
{
    "stac": {
        "location": "file:///calrissian/qk45d6lp",
        "basename": "qk45d6lp",
        "class": "Directory",
        "listing": [
            {
                "class": "File",
                "location": "file:///calrissian/qk45d6lp/dnbr-item.json",
                "basename": "dnbr-item.json",
                "checksum": "sha1$1c0a635ad501c599ab258019d05c7b276515c565",
                "size": 818,
                "path": "/calrissian/qk45d6lp/dnbr-item.json"
            },
            {
                "class": "File",
                "location": "file:///calrissian/qk45d6lp/catalog.json",
                "basename": "catalog.json",
                "checksum": "sha1$a5d1d9821e889aa125778e4f2e14a788ff1512ce",
                "size": 225,
                "path": "/calrissian/qk45d6lp/catalog.json"
            },
            {
                "class": "File",
                "location": "file:///calrissian/qk45d6lp/dnbr.tif",
                "basename": "dnbr.tif",
                "checksum": "sha1$87a3dfee0d055453dad525e8edd8a216121d808c",
                "size": 1402218,
                "path": "/calrissian/qk45d6lp/dnbr.tif"
            }
        ],
        "path": "/calrissian/qk45d6lp"
    }
}

Get the usage report

In [19]:

                
                    Copied!
                    
usage = execution.get_usage_report()
usage
usage = execution.get_usage_report()
usage

copy /calrissian/report.json to .
STDERR: tar: removing leading '/' from member names

Out[19]:

{'cores_allowed': 2.0,
 'ram_mb_allowed': 4000.0,
 'children': [{'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 0.0,
   'exit_code': 0,
   'name': 'node_stac',
   'start_time': '2023-03-22T11:06:52+00:00',
   'finish_time': '2023-03-22T11:06:52+00:00',
   'elapsed_hours': None,
   'elapsed_seconds': 0.0,
   'ram_megabyte_hours': None,
   'cpu_hours': None},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 0.0,
   'exit_code': 0,
   'name': 'node_stac_2',
   'start_time': '2023-03-22T11:06:53+00:00',
   'finish_time': '2023-03-22T11:06:53+00:00',
   'elapsed_hours': None,
   'elapsed_seconds': 0.0,
   'ram_megabyte_hours': None,
   'cpu_hours': None},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 0.0,
   'exit_code': 0,
   'name': 'node_stac_3',
   'start_time': '2023-03-22T11:06:55+00:00',
   'finish_time': '2023-03-22T11:06:55+00:00',
   'elapsed_hours': None,
   'elapsed_seconds': 0.0,
   'ram_megabyte_hours': None,
   'cpu_hours': None},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 0.0,
   'exit_code': 0,
   'name': 'node_stac_4',
   'start_time': '2023-03-22T11:06:56+00:00',
   'finish_time': '2023-03-22T11:06:56+00:00',
   'elapsed_hours': None,
   'elapsed_seconds': 0.0,
   'ram_megabyte_hours': None,
   'cpu_hours': None},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 0.0,
   'exit_code': 0,
   'name': 'node_stac_6',
   'start_time': '2023-03-22T11:07:01+00:00',
   'finish_time': '2023-03-22T11:07:02+00:00',
   'elapsed_hours': 0.0002777777777777778,
   'elapsed_seconds': 1.0,
   'ram_megabyte_hours': 0.07456540444444444,
   'cpu_hours': 0.0002777777777777778},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 0.0,
   'exit_code': 0,
   'name': 'node_stac_5',
   'start_time': '2023-03-22T11:07:02+00:00',
   'finish_time': '2023-03-22T11:07:03+00:00',
   'elapsed_hours': 0.0002777777777777778,
   'elapsed_seconds': 1.0,
   'ram_megabyte_hours': 0.07456540444444444,
   'cpu_hours': 0.0002777777777777778},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 2.256036,
   'exit_code': 0,
   'name': 'node_subset',
   'start_time': '2023-03-22T11:07:05+00:00',
   'finish_time': '2023-03-22T11:07:14+00:00',
   'elapsed_hours': 0.0025,
   'elapsed_seconds': 9.0,
   'ram_megabyte_hours': 0.67108864,
   'cpu_hours': 0.0025},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 2.256036,
   'exit_code': 0,
   'name': 'node_subset_2',
   'start_time': '2023-03-22T11:07:17+00:00',
   'finish_time': '2023-03-22T11:07:26+00:00',
   'elapsed_hours': 0.0025,
   'elapsed_seconds': 9.0,
   'ram_megabyte_hours': 0.67108864,
   'cpu_hours': 0.0025},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 1.128247,
   'exit_code': 0,
   'name': 'node_subset_3',
   'start_time': '2023-03-22T11:07:18+00:00',
   'finish_time': '2023-03-22T11:07:26+00:00',
   'elapsed_hours': 0.0022222222222222222,
   'elapsed_seconds': 8.0,
   'ram_megabyte_hours': 0.5965232355555555,
   'cpu_hours': 0.0022222222222222222},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 2.256036,
   'exit_code': 0,
   'name': 'node_subset_4',
   'start_time': '2023-03-22T11:07:29+00:00',
   'finish_time': '2023-03-22T11:07:42+00:00',
   'elapsed_hours': 0.003611111111111111,
   'elapsed_seconds': 13.0,
   'ram_megabyte_hours': 0.9693502577777777,
   'cpu_hours': 0.003611111111111111},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 1.128247,
   'exit_code': 0,
   'name': 'node_subset_6',
   'start_time': '2023-03-22T11:07:45+00:00',
   'finish_time': '2023-03-22T11:07:53+00:00',
   'elapsed_hours': 0.0022222222222222222,
   'elapsed_seconds': 8.0,
   'ram_megabyte_hours': 0.5965232355555555,
   'cpu_hours': 0.0022222222222222222},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 2.256036,
   'exit_code': 0,
   'name': 'node_subset_5',
   'start_time': '2023-03-22T11:07:46+00:00',
   'finish_time': '2023-03-22T11:07:55+00:00',
   'elapsed_hours': 0.0025,
   'elapsed_seconds': 9.0,
   'ram_megabyte_hours': 0.67108864,
   'cpu_hours': 0.0025},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 4.515438,
   'exit_code': 0,
   'name': 'node_nbr',
   'start_time': '2023-03-22T11:07:57+00:00',
   'finish_time': '2023-03-22T11:07:57+00:00',
   'elapsed_hours': None,
   'elapsed_seconds': 0.0,
   'ram_megabyte_hours': None,
   'cpu_hours': None},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 4.515438,
   'exit_code': 0,
   'name': 'node_nbr_2',
   'start_time': '2023-03-22T11:08:00+00:00',
   'finish_time': '2023-03-22T11:08:00+00:00',
   'elapsed_hours': None,
   'elapsed_seconds': 0.0,
   'ram_megabyte_hours': None,
   'cpu_hours': None},
  {'cpus': 2.0,
   'ram_megabytes': 2097.152,
   'disk_megabytes': 1.829194,
   'exit_code': 0,
   'name': 'node_cog',
   'start_time': '2023-03-22T11:08:03+00:00',
   'finish_time': '2023-03-22T11:08:03+00:00',
   'elapsed_hours': None,
   'elapsed_seconds': 0.0,
   'ram_megabyte_hours': None,
   'cpu_hours': None},
  {'cpus': 2.0,
   'ram_megabytes': 2097.152,
   'disk_megabytes': 3.947077,
   'exit_code': 0,
   'name': 'node_cog_2',
   'start_time': '2023-03-22T11:08:06+00:00',
   'finish_time': '2023-03-22T11:08:06+00:00',
   'elapsed_hours': None,
   'elapsed_seconds': 0.0,
   'ram_megabyte_hours': None,
   'cpu_hours': None},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 4.515438,
   'exit_code': 0,
   'name': 'node_dnbr',
   'start_time': '2023-03-22T11:08:09+00:00',
   'finish_time': '2023-03-22T11:08:09+00:00',
   'elapsed_hours': None,
   'elapsed_seconds': 0.0,
   'ram_megabyte_hours': None,
   'cpu_hours': None},
  {'cpus': 2.0,
   'ram_megabytes': 2097.152,
   'disk_megabytes': 1.402218,
   'exit_code': 0,
   'name': 'node_cog_3',
   'start_time': '2023-03-22T11:08:12+00:00',
   'finish_time': '2023-03-22T11:08:12+00:00',
   'elapsed_hours': None,
   'elapsed_seconds': 0.0,
   'ram_megabyte_hours': None,
   'cpu_hours': None},
  {'cpus': 1.0,
   'ram_megabytes': 268.435456,
   'disk_megabytes': 0.0,
   'exit_code': 0,
   'name': 'node_stac_7',
   'start_time': '2023-03-22T11:08:14+00:00',
   'finish_time': '2023-03-22T11:08:15+00:00',
   'elapsed_hours': 0.0002777777777777778,
   'elapsed_seconds': 1.0,
   'ram_megabyte_hours': 0.07456540444444444,
   'cpu_hours': 0.0002777777777777778}],
 'start_time': '2023-03-22T11:06:52+00:00',
 'finish_time': '2023-03-22T11:08:15+00:00',
 'elapsed_hours': 0.023055555555555555,
 'elapsed_seconds': 83.0,
 'total_cpu_hours': 0.016388888888888887,
 'total_ram_megabyte_hours': 4.399358862222222,
 'total_disk_megabytes': 32.005441,
 'total_tasks': 19,
 'max_parallel_cpus': 2.0,
 'max_parallel_ram_megabytes': 536.870912,
 'max_parallel_tasks': 2}

Get the execution output

In [20]:

                
                    Copied!
                    
output = execution.get_output()
output
output = execution.get_output()
output

copy /calrissian/output.json to .
STDERR: tar: removing leading '/' from member names

Out[20]:

{'stac': {'location': 'file:///calrissian/qk45d6lp',
  'basename': 'qk45d6lp',
  'class': 'Directory',
  'listing': [{'class': 'File',
    'location': 'file:///calrissian/qk45d6lp/dnbr-item.json',
    'basename': 'dnbr-item.json',
    'checksum': 'sha1$1c0a635ad501c599ab258019d05c7b276515c565',
    'size': 818,
    'path': '/calrissian/qk45d6lp/dnbr-item.json'},
   {'class': 'File',
    'location': 'file:///calrissian/qk45d6lp/catalog.json',
    'basename': 'catalog.json',
    'checksum': 'sha1$a5d1d9821e889aa125778e4f2e14a788ff1512ce',
    'size': 225,
    'path': '/calrissian/qk45d6lp/catalog.json'},
   {'class': 'File',
    'location': 'file:///calrissian/qk45d6lp/dnbr.tif',
    'basename': 'dnbr.tif',
    'checksum': 'sha1$87a3dfee0d055453dad525e8edd8a216121d808c',
    'size': 1402218,
    'path': '/calrissian/qk45d6lp/dnbr.tif'}],
  'path': '/calrissian/qk45d6lp'}}

Get a few details about the execution

In [21]:

                
                    Copied!
                    
print(execution.get_start_time())
print(execution.get_completion_time())
print(execution.get_start_time())
print(execution.get_completion_time())

2023-03-22 11:05:49+00:00
2023-03-22 11:08:20+00:00

In [22]:

                
                    Copied!
                    
print(f"complete {execution.is_complete()}")
print(f"succeeded {execution.is_succeeded()}")
print(f"complete {execution.is_complete()}")
print(f"succeeded {execution.is_succeeded()}")

complete True
succeeded True

In [23]:

                
                    Copied!
                    
execution.get_tool_logs()
execution.get_tool_logs()

copy /calrissian/report.json to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_stac.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_stac_2.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_stac_3.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_stac_4.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_stac_6.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_stac_5.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_subset.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_subset_2.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_subset_3.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_subset_4.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_subset_6.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_subset_5.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_nbr.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_nbr_2.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_cog.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_cog_2.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_dnbr.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_cog_3.log to .
STDERR: tar: removing leading '/' from member names

copy /calrissian/node_stac_7.log to .
STDERR: tar: removing leading '/' from member names

Out[23]:

['./node_stac.log',
 './node_stac_2.log',
 './node_stac_3.log',
 './node_stac_4.log',
 './node_stac_6.log',
 './node_stac_5.log',
 './node_subset.log',
 './node_subset_2.log',
 './node_subset_3.log',
 './node_subset_4.log',
 './node_subset_6.log',
 './node_subset_5.log',
 './node_nbr.log',
 './node_nbr_2.log',
 './node_cog.log',
 './node_cog_2.log',
 './node_dnbr.log',
 './node_cog_3.log',
 './node_stac_7.log']

Delete the Kubernetes namespace with:

In [17]:

                
                    Copied!
                    
session.dispose()
session.dispose()

2023-03-22 11:03:48.238 | INFO     | pycalrissian.context:dispose:121 - delete pod job-167948062876755-9399351e-e736-47f6-830d-89f9432b6fd4-h6brf
2023-03-22 11:03:48.313 | INFO     | pycalrissian.context:dispose:121 - delete pod job-1679481953421835-58349bec-3178-45b6-827e-37ee74804919-44xdc
2023-03-22 11:03:48.380 | INFO     | pycalrissian.context:dispose:121 - delete pod job-1679482405334652-5e620474-376e-413f-99ed-f40ca19ba08a-ftg9w
2023-03-22 11:03:48.478 | INFO     | pycalrissian.context:dispose:121 - delete pod job-1679482833004388-941219c5-9755-46ce-9ebf-917dfda774a4-4fnd9
2023-03-22 11:03:48.810 | INFO     | pycalrissian.context:dispose:121 - delete pod job-1679482833004388-941219c5-9755-46ce-9ebf-917dfda774a4-5kfdw
2023-03-22 11:03:49.228 | INFO     | pycalrissian.context:dispose:121 - delete pod job-1679482833004388-941219c5-9755-46ce-9ebf-917dfda774a4-x889h
2023-03-22 11:03:49.323 | INFO     | pycalrissian.context:dispose:124 - dispose namespace job-namespace
2023-03-22 11:03:49.379 | INFO     | pycalrissian.context:dispose:132 - namespace job-namespace deleted

Out[17]:

{'api_version': 'v1',
 'code': None,
 'details': None,
 'kind': 'Namespace',
 'message': None,
 'metadata': {'_continue': None,
              'remaining_item_count': None,
              'resource_version': '18983565861',
              'self_link': None},
 'reason': None,
 'status': "{'phase': 'Terminating'}"}

In [ ]: