Creating a secret my-secret in the
Import the modules
In [1]:
Copied!
from pycalrissian.context import CalrissianContext
from pycalrissian.job import CalrissianJob
from pycalrissian.execution import CalrissianExecution
import base64
import os
import yaml
from kubernetes.client.models.v1_job import V1Job
from pycalrissian.context import CalrissianContext
from pycalrissian.job import CalrissianJob
from pycalrissian.execution import CalrissianExecution
import base64
import os
import yaml
from kubernetes.client.models.v1_job import V1Job
Create the image pull secrets¶
There's one for docker.hub and one for Gitlab container registry as the CWL description to run refers container images published on those two container registries
In [3]:
Copied!
username = ""
password = ""
auth = base64.b64encode(f"{username}:{password}".encode("utf-8")).decode(
"utf-8"
)
secret_config = {
"auths": {
"registry.gitlab.com": {
"auth": ""
},
"https://index.docker.io/v1/": {
"auth": ""
},
}
}
username = ""
password = ""
auth = base64.b64encode(f"{username}:{password}".encode("utf-8")).decode(
"utf-8"
)
secret_config = {
"auths": {
"registry.gitlab.com": {
"auth": ""
},
"https://index.docker.io/v1/": {
"auth": ""
},
}
}
In [4]:
Copied!
secret_config
secret_config
Out[4]:
{'auths': {'registry.gitlab.com': {'auth': 'Z2l0bGFiK2RlcGxveS10b2tlbi0xODgxNzY4Oko1dHU4eXhkU3E0a19RMXpOOGhv'},
'https://index.docker.io/v1/': {'auth': 'ZmFicmljZWJyaXRvOmRja3JfcGF0X1E3NFRFZWhBZWVsSW9BWUFLamVUcmlRTldqZw=='}}}
Take away messages about image pull secrets
- they're created as a dictionary with the same structure as your
~/.docker/configfile - you can use the username/password pair or the auth string
Create the CalrissianContext¶
The CalrissianContext creates a kubernetes namespace on the cluster.
Note: our kubernetes cluster uses the longhorn RWX storage class, adapt it to your cluster configuration
Using existing secrets¶
In addition to the imagePullSecrets, you can also specify an additionalImagePullSecrets key in the dictionary.
This additionalImagePullSecrets is an array of secrets defined in the same namespace where the ZOO-Project-DRU Helm chart is deployed.
Example:
From the command line, you can create secret in any namespace.
# Creating a secret my-secret in the
kubectl create secret docker-registry my-secret \
--docker-email=tiger@acme.example \
--docker-username=tiger \
--docker-password=pass1234 \
--docker-server=my-registry.example:5000 \
-n given-namespace
With the secret defined, you can reference them from your CalrissianContext as presented below.
#Creating a CalrissianContext using the this secret
session = CalrissianContext(
namespace=namespace_name,
storage_class="openebs-kernel-nfs-scw",
volume_size="10G",
image_pull_secrets={
"imagePullSecrets": secret_config,
"additionalImagePullSecrets": [
{"name": "my-secret"},
{"name": "my-other-secret"}
]
},
)
This is useful if you have multiple secrets that you want to use to pull images from different registries.
The additionalImagePullSecrets key is optional and can be omitted if you only want to use the imagePullSecret.
In [ ]:
Copied!
namespace_name = "job-namespace-n"
session = CalrissianContext(
namespace=namespace_name,
storage_class="openebs-kernel-nfs-scw",
volume_size="10G",
image_pull_secrets={"imagePullSecrets": secret_config},
)
namespace_name = "job-namespace-n"
session = CalrissianContext(
namespace=namespace_name,
storage_class="openebs-kernel-nfs-scw",
volume_size="10G",
image_pull_secrets={"imagePullSecrets": secret_config},
)
Now trigger the CalrissianContext initialisation with:
In [9]:
Copied!
session.initialise()
session.initialise()
2023-03-22 11:04:26.594 | INFO | pycalrissian.context:initialise:65 - create namespace job-namespace-n 2023-03-22 11:04:26.642 | INFO | pycalrissian.context:create_namespace:281 - creating namespace job-namespace-n 2023-03-22 11:04:31.757 | INFO | pycalrissian.context:create_namespace:294 - namespace job-namespace-n created 2023-03-22 11:04:31.760 | INFO | pycalrissian.context:initialise:82 - create role pod-manager-role 2023-03-22 11:04:36.912 | INFO | pycalrissian.context:create_role:333 - role pod-manager-role created 2023-03-22 11:04:36.914 | INFO | pycalrissian.context:initialise:91 - create role binding for role pod-manager-role 2023-03-22 11:04:42.069 | INFO | pycalrissian.context:create_role_binding:373 - role binding pod-manager-default-binding created 2023-03-22 11:04:42.070 | INFO | pycalrissian.context:initialise:82 - create role log-reader-role 2023-03-22 11:04:47.228 | INFO | pycalrissian.context:create_role:333 - role log-reader-role created 2023-03-22 11:04:47.230 | INFO | pycalrissian.context:initialise:91 - create role binding for role log-reader-role 2023-03-22 11:04:52.400 | INFO | pycalrissian.context:create_role_binding:373 - role binding log-reader-default-binding created 2023-03-22 11:04:52.402 | INFO | pycalrissian.context:initialise:96 - create persistent volume claim 'calrissian-wdir' of 10G with storage class openebs-kernel-nfs-scw 2023-03-22 11:04:57.562 | INFO | pycalrissian.context:create_pvc:416 - pvc calrissian-wdir created 2023-03-22 11:04:57.564 | INFO | pycalrissian.context:initialise:110 - create secret container-rg 2023-03-22 11:05:02.727 | INFO | pycalrissian.context:create_image_pull_secret:510 - image pull secret container-rg created 2023-03-22 11:05:02.730 | INFO | pycalrissian.context:initialise:113 - patch service account
Read the CWL document¶
Now load a CWL document and create a dictionary with the parameters:
In [10]:
Copied!
with open("../tests/app-s2-composites.0.1.0.cwl", "r") as stream:
cwl = yaml.safe_load(stream)
params = {
"post_stac_item": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A", # noqa: E501
"pre_stac_item": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210703_0_L2A", # noqa: E501
"aoi": "136.659,-35.96,136.923,-35.791",
}
with open("../tests/app-s2-composites.0.1.0.cwl", "r") as stream:
cwl = yaml.safe_load(stream)
params = {
"post_stac_item": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A", # noqa: E501
"pre_stac_item": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210703_0_L2A", # noqa: E501
"aoi": "136.659,-35.96,136.923,-35.791",
}
Take away messages
- The CWL description is loaded into a Python dictionary
- The parameters are a Python dictionary
So you can discover the CWL Workflow parameters with something like:
In [11]:
Copied!
cwl["$graph"][0]["inputs"]
cwl["$graph"][0]["inputs"]
Out[11]:
{'pre_stac_item': {'doc': 'Pre-event Sentinel-2 item', 'type': 'string'},
'post_stac_item': {'doc': 'Post-event Sentinel-2 item', 'type': 'string'},
'aoi': {'doc': 'area of interest as a bounding box', 'type': 'string?'},
'bands': {'type': 'string[]', 'default': ['B8A', 'B12', 'SCL']}}
Create the CalrissianJob¶
In [12]:
Copied!
os.environ["CALRISSIAN_IMAGE"] = "docker.io/terradue/calrissian:0.12.0"
job = CalrissianJob(
cwl=cwl,
params=params,
runtime_context=session,
cwl_entry_point="dnbr",
max_cores=2,
max_ram="4G",
tool_logs=True,
)
os.environ["CALRISSIAN_IMAGE"] = "docker.io/terradue/calrissian:0.12.0"
job = CalrissianJob(
cwl=cwl,
params=params,
runtime_context=session,
cwl_entry_point="dnbr",
max_cores=2,
max_ram="4G",
tool_logs=True,
)
2023-03-22 11:05:17.787 | INFO | pycalrissian.job:__init__:68 - using default security context {'runAsUser': 0, 'runAsGroup': 0, 'fsGroup': 0}
2023-03-22 11:05:17.788 | INFO | pycalrissian.job:__init__:79 - job name: job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70
2023-03-22 11:05:17.788 | INFO | pycalrissian.job:__init__:80 - create CWL config map
2023-03-22 11:05:22.973 | INFO | pycalrissian.context:create_configmap:467 - config map cwl-workflow created
2023-03-22 11:05:22.974 | INFO | pycalrissian.job:__init__:82 - create processing parameters config map
2023-03-22 11:05:28.127 | INFO | pycalrissian.context:create_configmap:467 - config map params created
The CalrissianJob object is constructed with:
- a CWL dictionary
- a parameters dictionaty
- a runtime context, a CalrissianContext object
- the maximum number of cores the pods can use
- the maximum amount of RAM the pods can use
The CalrissianJob can be serialized to a Kubernetes Job object:
In [13]:
Copied!
isinstance(job.to_k8s_job(), V1Job)
isinstance(job.to_k8s_job(), V1Job)
2023-03-22 11:05:43.200 | INFO | pycalrissian.job:_get_calrissian_container:417 - using Calrissian image: docker.io/terradue/calrissian:0.12.0
Out[13]:
True
Or to a Kubernetes Job manifest in YAML:
In [14]:
Copied!
job.to_yaml("job.yml")
job.to_yaml("job.yml")
2023-03-22 11:05:44.398 | INFO | pycalrissian.job:_get_calrissian_container:417 - using Calrissian image: docker.io/terradue/calrissian:0.12.0 2023-03-22 11:05:44.407 | INFO | pycalrissian.job:to_yaml:143 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 serialized to job.yml
Note The Calrissian pod image can be defined with the environment variable CALRISSIAN_IMAGE
At this stage, you could do kubectl -n job-namespace apply -f job.yml to submit the job on kubernetes
Create the CalrissianExecution¶
In [15]:
Copied!
execution = CalrissianExecution(job=job, runtime_context=session)
execution = CalrissianExecution(job=job, runtime_context=session)
Submit the job with:
In [16]:
Copied!
execution.submit()
execution.submit()
2023-03-22 11:05:49.797 | INFO | pycalrissian.execution:submit:32 - submit job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 2023-03-22 11:05:49.800 | INFO | pycalrissian.job:_get_calrissian_container:417 - using Calrissian image: docker.io/terradue/calrissian:0.12.0 2023-03-22 11:05:49.869 | INFO | pycalrissian.execution:submit:38 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 submitted
Monitor the execution with:
In [17]:
Copied!
execution.monitor(interval=20)
execution.monitor(interval=20)
2023-03-22 11:05:56.327 | INFO | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active 2023-03-22 11:06:16.413 | INFO | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active 2023-03-22 11:06:36.482 | INFO | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active 2023-03-22 11:06:56.566 | INFO | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active 2023-03-22 11:07:16.643 | INFO | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active 2023-03-22 11:07:36.725 | INFO | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active 2023-03-22 11:07:56.812 | INFO | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active 2023-03-22 11:08:16.942 | INFO | pycalrissian.execution:monitor:210 - job job-1679483117788287-73087b7b-3801-4ef0-9662-8f8b48f93c70 is active 2023-03-22 11:08:37.152 | INFO | pycalrissian.execution:monitor:239 - execution is complete 2023-03-22 11:08:37.216 | INFO | pycalrissian.execution:monitor:241 - the outcome is: success!
Get the execution log
In [18]:
Copied!
log = execution.get_log()
print(log)
log = execution.get_log()
print(log)
INFO calrissian 0.12.0 (cwltool 3.1.20230201224320) INFO Resolved '/workflow-input/workflow.cwl#dnbr' to 'file:///workflow-input/..2023_03_22_11_05_50.1680561479/workflow.cwl#dnbr' WARNING Workflow checker warning: ../workflow-input/..2023_03_22_11_05_50.1680561479/workflow.cwl:9:7: Source 'aoi' of type ["null", "string"] may be incompatible ../workflow-input/..2023_03_22_11_05_50.1680561479/workflow.cwl:51:9: with sink 'aoi' of type "string" INFO [workflow ] starting step node_nbr INFO [step node_nbr] start INFO [workflow node_nbr] starting step node_stac_2 INFO [step node_stac_2] start INFO [step node_stac_2] start INFO [step node_stac_2] start INFO [step node_nbr] start INFO [workflow node_nbr_2] starting step node_stac_3 INFO [step node_stac_3] start INFO [step node_stac_3] start INFO [step node_stac_3] start INFO [workflow ] start INFO [workflow node_nbr] start INFO [workflow node_nbr_2] start INFO [step node_stac_2] completed success INFO [workflow node_nbr] starting step node_subset INFO [step node_subset] start INFO [step node_subset] start INFO [step node_subset] start INFO [step node_stac_3] completed success INFO [workflow node_nbr_2] starting step node_subset_2 INFO [step node_subset_2] start INFO [step node_subset_2] start INFO [step node_subset_2] start INFO [step node_subset] completed success INFO [workflow node_nbr] starting step node_nbr_2 INFO [step node_nbr_2] start INFO [step node_subset_2] completed success INFO [workflow node_nbr_2] starting step node_nbr_3 INFO [step node_nbr_3] start INFO [step node_nbr_2] completed success INFO [workflow node_nbr] starting step node_cog_2 INFO [step node_cog_2] start INFO [step node_nbr_3] completed success INFO [workflow node_nbr_2] starting step node_cog_3 INFO [step node_cog_3] start INFO [step node_cog_2] completed success INFO [workflow node_nbr] completed success INFO [step node_cog_3] completed success INFO [workflow node_nbr_2] completed success INFO [step node_nbr] completed success INFO [workflow ] starting step node_dnbr INFO [step node_dnbr] start INFO [step node_dnbr] completed success INFO [workflow ] starting step node_cog INFO [step node_cog] start INFO [step node_cog] completed success INFO [workflow ] starting step node_stac INFO [step node_stac] start INFO [step node_stac] completed success INFO [workflow ] completed success INFO Final process status is success { "stac": { "location": "file:///calrissian/qk45d6lp", "basename": "qk45d6lp", "class": "Directory", "listing": [ { "class": "File", "location": "file:///calrissian/qk45d6lp/dnbr-item.json", "basename": "dnbr-item.json", "checksum": "sha1$1c0a635ad501c599ab258019d05c7b276515c565", "size": 818, "path": "/calrissian/qk45d6lp/dnbr-item.json" }, { "class": "File", "location": "file:///calrissian/qk45d6lp/catalog.json", "basename": "catalog.json", "checksum": "sha1$a5d1d9821e889aa125778e4f2e14a788ff1512ce", "size": 225, "path": "/calrissian/qk45d6lp/catalog.json" }, { "class": "File", "location": "file:///calrissian/qk45d6lp/dnbr.tif", "basename": "dnbr.tif", "checksum": "sha1$87a3dfee0d055453dad525e8edd8a216121d808c", "size": 1402218, "path": "/calrissian/qk45d6lp/dnbr.tif" } ], "path": "/calrissian/qk45d6lp" } }
Get the usage report
In [19]:
Copied!
usage = execution.get_usage_report()
usage
usage = execution.get_usage_report()
usage
copy /calrissian/report.json to . STDERR: tar: removing leading '/' from member names
Out[19]:
{'cores_allowed': 2.0,
'ram_mb_allowed': 4000.0,
'children': [{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 0.0,
'exit_code': 0,
'name': 'node_stac',
'start_time': '2023-03-22T11:06:52+00:00',
'finish_time': '2023-03-22T11:06:52+00:00',
'elapsed_hours': None,
'elapsed_seconds': 0.0,
'ram_megabyte_hours': None,
'cpu_hours': None},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 0.0,
'exit_code': 0,
'name': 'node_stac_2',
'start_time': '2023-03-22T11:06:53+00:00',
'finish_time': '2023-03-22T11:06:53+00:00',
'elapsed_hours': None,
'elapsed_seconds': 0.0,
'ram_megabyte_hours': None,
'cpu_hours': None},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 0.0,
'exit_code': 0,
'name': 'node_stac_3',
'start_time': '2023-03-22T11:06:55+00:00',
'finish_time': '2023-03-22T11:06:55+00:00',
'elapsed_hours': None,
'elapsed_seconds': 0.0,
'ram_megabyte_hours': None,
'cpu_hours': None},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 0.0,
'exit_code': 0,
'name': 'node_stac_4',
'start_time': '2023-03-22T11:06:56+00:00',
'finish_time': '2023-03-22T11:06:56+00:00',
'elapsed_hours': None,
'elapsed_seconds': 0.0,
'ram_megabyte_hours': None,
'cpu_hours': None},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 0.0,
'exit_code': 0,
'name': 'node_stac_6',
'start_time': '2023-03-22T11:07:01+00:00',
'finish_time': '2023-03-22T11:07:02+00:00',
'elapsed_hours': 0.0002777777777777778,
'elapsed_seconds': 1.0,
'ram_megabyte_hours': 0.07456540444444444,
'cpu_hours': 0.0002777777777777778},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 0.0,
'exit_code': 0,
'name': 'node_stac_5',
'start_time': '2023-03-22T11:07:02+00:00',
'finish_time': '2023-03-22T11:07:03+00:00',
'elapsed_hours': 0.0002777777777777778,
'elapsed_seconds': 1.0,
'ram_megabyte_hours': 0.07456540444444444,
'cpu_hours': 0.0002777777777777778},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 2.256036,
'exit_code': 0,
'name': 'node_subset',
'start_time': '2023-03-22T11:07:05+00:00',
'finish_time': '2023-03-22T11:07:14+00:00',
'elapsed_hours': 0.0025,
'elapsed_seconds': 9.0,
'ram_megabyte_hours': 0.67108864,
'cpu_hours': 0.0025},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 2.256036,
'exit_code': 0,
'name': 'node_subset_2',
'start_time': '2023-03-22T11:07:17+00:00',
'finish_time': '2023-03-22T11:07:26+00:00',
'elapsed_hours': 0.0025,
'elapsed_seconds': 9.0,
'ram_megabyte_hours': 0.67108864,
'cpu_hours': 0.0025},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 1.128247,
'exit_code': 0,
'name': 'node_subset_3',
'start_time': '2023-03-22T11:07:18+00:00',
'finish_time': '2023-03-22T11:07:26+00:00',
'elapsed_hours': 0.0022222222222222222,
'elapsed_seconds': 8.0,
'ram_megabyte_hours': 0.5965232355555555,
'cpu_hours': 0.0022222222222222222},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 2.256036,
'exit_code': 0,
'name': 'node_subset_4',
'start_time': '2023-03-22T11:07:29+00:00',
'finish_time': '2023-03-22T11:07:42+00:00',
'elapsed_hours': 0.003611111111111111,
'elapsed_seconds': 13.0,
'ram_megabyte_hours': 0.9693502577777777,
'cpu_hours': 0.003611111111111111},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 1.128247,
'exit_code': 0,
'name': 'node_subset_6',
'start_time': '2023-03-22T11:07:45+00:00',
'finish_time': '2023-03-22T11:07:53+00:00',
'elapsed_hours': 0.0022222222222222222,
'elapsed_seconds': 8.0,
'ram_megabyte_hours': 0.5965232355555555,
'cpu_hours': 0.0022222222222222222},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 2.256036,
'exit_code': 0,
'name': 'node_subset_5',
'start_time': '2023-03-22T11:07:46+00:00',
'finish_time': '2023-03-22T11:07:55+00:00',
'elapsed_hours': 0.0025,
'elapsed_seconds': 9.0,
'ram_megabyte_hours': 0.67108864,
'cpu_hours': 0.0025},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 4.515438,
'exit_code': 0,
'name': 'node_nbr',
'start_time': '2023-03-22T11:07:57+00:00',
'finish_time': '2023-03-22T11:07:57+00:00',
'elapsed_hours': None,
'elapsed_seconds': 0.0,
'ram_megabyte_hours': None,
'cpu_hours': None},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 4.515438,
'exit_code': 0,
'name': 'node_nbr_2',
'start_time': '2023-03-22T11:08:00+00:00',
'finish_time': '2023-03-22T11:08:00+00:00',
'elapsed_hours': None,
'elapsed_seconds': 0.0,
'ram_megabyte_hours': None,
'cpu_hours': None},
{'cpus': 2.0,
'ram_megabytes': 2097.152,
'disk_megabytes': 1.829194,
'exit_code': 0,
'name': 'node_cog',
'start_time': '2023-03-22T11:08:03+00:00',
'finish_time': '2023-03-22T11:08:03+00:00',
'elapsed_hours': None,
'elapsed_seconds': 0.0,
'ram_megabyte_hours': None,
'cpu_hours': None},
{'cpus': 2.0,
'ram_megabytes': 2097.152,
'disk_megabytes': 3.947077,
'exit_code': 0,
'name': 'node_cog_2',
'start_time': '2023-03-22T11:08:06+00:00',
'finish_time': '2023-03-22T11:08:06+00:00',
'elapsed_hours': None,
'elapsed_seconds': 0.0,
'ram_megabyte_hours': None,
'cpu_hours': None},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 4.515438,
'exit_code': 0,
'name': 'node_dnbr',
'start_time': '2023-03-22T11:08:09+00:00',
'finish_time': '2023-03-22T11:08:09+00:00',
'elapsed_hours': None,
'elapsed_seconds': 0.0,
'ram_megabyte_hours': None,
'cpu_hours': None},
{'cpus': 2.0,
'ram_megabytes': 2097.152,
'disk_megabytes': 1.402218,
'exit_code': 0,
'name': 'node_cog_3',
'start_time': '2023-03-22T11:08:12+00:00',
'finish_time': '2023-03-22T11:08:12+00:00',
'elapsed_hours': None,
'elapsed_seconds': 0.0,
'ram_megabyte_hours': None,
'cpu_hours': None},
{'cpus': 1.0,
'ram_megabytes': 268.435456,
'disk_megabytes': 0.0,
'exit_code': 0,
'name': 'node_stac_7',
'start_time': '2023-03-22T11:08:14+00:00',
'finish_time': '2023-03-22T11:08:15+00:00',
'elapsed_hours': 0.0002777777777777778,
'elapsed_seconds': 1.0,
'ram_megabyte_hours': 0.07456540444444444,
'cpu_hours': 0.0002777777777777778}],
'start_time': '2023-03-22T11:06:52+00:00',
'finish_time': '2023-03-22T11:08:15+00:00',
'elapsed_hours': 0.023055555555555555,
'elapsed_seconds': 83.0,
'total_cpu_hours': 0.016388888888888887,
'total_ram_megabyte_hours': 4.399358862222222,
'total_disk_megabytes': 32.005441,
'total_tasks': 19,
'max_parallel_cpus': 2.0,
'max_parallel_ram_megabytes': 536.870912,
'max_parallel_tasks': 2}
Get the execution output
In [20]:
Copied!
output = execution.get_output()
output
output = execution.get_output()
output
copy /calrissian/output.json to . STDERR: tar: removing leading '/' from member names
Out[20]:
{'stac': {'location': 'file:///calrissian/qk45d6lp',
'basename': 'qk45d6lp',
'class': 'Directory',
'listing': [{'class': 'File',
'location': 'file:///calrissian/qk45d6lp/dnbr-item.json',
'basename': 'dnbr-item.json',
'checksum': 'sha1$1c0a635ad501c599ab258019d05c7b276515c565',
'size': 818,
'path': '/calrissian/qk45d6lp/dnbr-item.json'},
{'class': 'File',
'location': 'file:///calrissian/qk45d6lp/catalog.json',
'basename': 'catalog.json',
'checksum': 'sha1$a5d1d9821e889aa125778e4f2e14a788ff1512ce',
'size': 225,
'path': '/calrissian/qk45d6lp/catalog.json'},
{'class': 'File',
'location': 'file:///calrissian/qk45d6lp/dnbr.tif',
'basename': 'dnbr.tif',
'checksum': 'sha1$87a3dfee0d055453dad525e8edd8a216121d808c',
'size': 1402218,
'path': '/calrissian/qk45d6lp/dnbr.tif'}],
'path': '/calrissian/qk45d6lp'}}
Get a few details about the execution
In [21]:
Copied!
print(execution.get_start_time())
print(execution.get_completion_time())
print(execution.get_start_time())
print(execution.get_completion_time())
2023-03-22 11:05:49+00:00 2023-03-22 11:08:20+00:00
In [22]:
Copied!
print(f"complete {execution.is_complete()}")
print(f"succeeded {execution.is_succeeded()}")
print(f"complete {execution.is_complete()}")
print(f"succeeded {execution.is_succeeded()}")
complete True succeeded True
In [23]:
Copied!
execution.get_tool_logs()
execution.get_tool_logs()
copy /calrissian/report.json to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_stac.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_stac_2.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_stac_3.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_stac_4.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_stac_6.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_stac_5.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_subset.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_subset_2.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_subset_3.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_subset_4.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_subset_6.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_subset_5.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_nbr.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_nbr_2.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_cog.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_cog_2.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_dnbr.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_cog_3.log to . STDERR: tar: removing leading '/' from member names copy /calrissian/node_stac_7.log to . STDERR: tar: removing leading '/' from member names
Out[23]:
['./node_stac.log', './node_stac_2.log', './node_stac_3.log', './node_stac_4.log', './node_stac_6.log', './node_stac_5.log', './node_subset.log', './node_subset_2.log', './node_subset_3.log', './node_subset_4.log', './node_subset_6.log', './node_subset_5.log', './node_nbr.log', './node_nbr_2.log', './node_cog.log', './node_cog_2.log', './node_dnbr.log', './node_cog_3.log', './node_stac_7.log']
Delete the Kubernetes namespace with:
In [17]:
Copied!
session.dispose()
session.dispose()
2023-03-22 11:03:48.238 | INFO | pycalrissian.context:dispose:121 - delete pod job-167948062876755-9399351e-e736-47f6-830d-89f9432b6fd4-h6brf 2023-03-22 11:03:48.313 | INFO | pycalrissian.context:dispose:121 - delete pod job-1679481953421835-58349bec-3178-45b6-827e-37ee74804919-44xdc 2023-03-22 11:03:48.380 | INFO | pycalrissian.context:dispose:121 - delete pod job-1679482405334652-5e620474-376e-413f-99ed-f40ca19ba08a-ftg9w 2023-03-22 11:03:48.478 | INFO | pycalrissian.context:dispose:121 - delete pod job-1679482833004388-941219c5-9755-46ce-9ebf-917dfda774a4-4fnd9 2023-03-22 11:03:48.810 | INFO | pycalrissian.context:dispose:121 - delete pod job-1679482833004388-941219c5-9755-46ce-9ebf-917dfda774a4-5kfdw 2023-03-22 11:03:49.228 | INFO | pycalrissian.context:dispose:121 - delete pod job-1679482833004388-941219c5-9755-46ce-9ebf-917dfda774a4-x889h 2023-03-22 11:03:49.323 | INFO | pycalrissian.context:dispose:124 - dispose namespace job-namespace 2023-03-22 11:03:49.379 | INFO | pycalrissian.context:dispose:132 - namespace job-namespace deleted
Out[17]:
{'api_version': 'v1',
'code': None,
'details': None,
'kind': 'Namespace',
'message': None,
'metadata': {'_continue': None,
'remaining_item_count': None,
'resource_version': '18983565861',
'self_link': None},
'reason': None,
'status': "{'phase': 'Terminating'}"}
In [ ]:
Copied!