REANA-Server¶
REANA-Server is a component of the REANA reusable and reproducible research data analysis platform. It implements the API Server that takes and performs REST API calls issued by REANA clients.
Features¶
offers rich REST API services for REANA clients
transmits REST API requests to appropriate REANA cloud components
REST API to run research analysis workflows on compute clouds
REST API to list submitted workflows and enquire about their statuses
REST API to manage analysis files
REST API to download results of finished analysis workflows
REST API to find the differences between two workflows (
git
like output)
Usage¶
The detailed information on how to install and use REANA can be found in docs.reana.io.
CLI API¶
flask reana-admin¶
REANA administration commands.
flask reana-admin [OPTIONS] COMMAND [ARGS]...
check-workflows¶
Check consistency of selected workflow run statuses between database, message queue and Kubernetes.
flask reana-admin check-workflows [OPTIONS]
Options
- --date-start <date_start>¶
Default value is 24 hours ago.
- --date-end <date_end>¶
Default value is now.
- -a, --all¶
Show all workflows/sessions/workspaces, even if in-sync.
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
create-admin-user¶
Create default user.
This user has the administrator role and can retrieve other user information as well as create new users.
flask reana-admin create-admin-user [OPTIONS]
Options
- -e, --email <email>¶
Required The email of the admin user.
- -p, --password <password>¶
Required
- -i, --id <id_>¶
interactive-session-cleanup¶
Close inactive interactive sessions.
flask reana-admin interactive-session-cleanup [OPTIONS]
Options
- -d, --days <days>¶
Required Close interactive sessions that are inactive for more than the specified number of days.
- --dry-run¶
Show which interactive sessions would be closed, without closing them. [default=False]
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
queue-consume¶
Start consuming specified queue and remove selected messages.
By default, you will need to specify either “-k” or “-i” options otherwise the command will return an error.
If -k option is specified, messages that have property values specified in -v will be deleted.
If -i option is specified, for every message, user will be asked what to do.
If -k and -i are specified together, for every message that matches property values in -v, user will be asked whether to delete it or not.
flask reana-admin queue-consume [OPTIONS]
Options
- -q, --queue-name <queue_name>¶
Required Name of the queue that will be consumed, e.g workflow-submission
- -k, --key <key>¶
Key of the property that will be used to filter the messages in the queue, e.g workflow_name_or_id
- -v, --values-to-delete <values_to_delete>¶
List of property values used to filter messages that will be removed from the queue, e.g UUID of a workflow
- -i, --interactive¶
Manually decide which messages to remove from the queue.
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
quota-resources¶
List available quota resources.
flask reana-admin quota-resources [OPTIONS]
quota-set¶
Set quota limits to the given users per resource.
flask reana-admin quota-set [OPTIONS]
Options
- -e, --email <emails>¶
Required The emails of the users. E.g. –email johndoe@example.org –email janedoe@example.org
- -r, --resource <resource_type>¶
Specify quota resource. e.g. cpu, disk.
- -n, --resource-name <resource_name>¶
Name of resource.
- -l, --limit <limit>¶
Required New limit in canonical unit.
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
quota-set-default-limits¶
- Set default quota limits for users who do not have any custom limits
defined.
Note that any previously set user limits, either via old defaults or via custom settings, will be kept during the upgrade, and won’t be automatically updated to match the new default limit value.
flask reana-admin quota-set-default-limits [OPTIONS]
Options
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
quota-usage¶
List quota usage of users.
flask reana-admin quota-usage [OPTIONS]
Options
- --id <id>¶
The id of the user.
- -e, --email <email>¶
The email of the user.
- --user-access-token <user_access_token>¶
The access token of the user.
- --json¶
Get output in JSON format.
- -h, --human-readable¶
Show quota usage values in human readable format.
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
retention-rules-apply¶
Apply pending retentions rules.
flask reana-admin retention-rules-apply [OPTIONS]
Options
- --dry-run¶
Show the pending retention rules without applying them. [default=False]
- --force-date <force_date>¶
Force desired date and time when deciding which rules to apply.
- --yes-i-am-sure¶
Do not ask for confirmation when doing potentially dangerous operations.
- -e, --email <email>¶
The email of the user.
- --id <id_>¶
The id of the user.
- -w, --workflow <workflow_uuid>¶
The id of the workflow.
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
retention-rules-extend¶
Extend active retentions rules.
flask reana-admin retention-rules-extend [OPTIONS]
Options
- -w, --workflow <workflow_uuid>¶
Required The id of the workflow.
- -d, --days <days>¶
Required Number of days to extend the rules.
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
status-report¶
Get a status report of the REANA system.
flask reana-admin status-report [OPTIONS]
Options
- --type <types>¶
Type of information to be displayed?
- Options:
interactive-sessions | workflows | users | system | storage | nodes | pods | jobs | quota-usage | all
- -e, --email <email>¶
Send the status by email to the configured receiver.
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
token-grant¶
Grant a token to the selected user.
flask reana-admin token-grant [OPTIONS]
Options
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
- --id <id_>¶
The id of the user.
- -e, --email <email>¶
The email of the user.
token-revoke¶
Revoke selected user’s token.
flask reana-admin token-revoke [OPTIONS]
Options
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
- --id <id_>¶
The id of the user.
- -e, --email <email>¶
The email of the user.
user-create¶
Create a new user.
flask reana-admin user-create [OPTIONS]
Options
- -e, --email <email>¶
Required The email of the user.
- --user-access-token <user_access_token>¶
The access token of the user.
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
user-export¶
Export all users in current REANA cluster.
flask reana-admin user-export [OPTIONS]
Options
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
user-import¶
Import users from file.
flask reana-admin user-import [OPTIONS]
Options
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
- -f, --file <file_>¶
A CSV file containing a list of REANA users.
user-list¶
List users according to the search criteria.
flask reana-admin user-list [OPTIONS]
Options
- --id <id>¶
The id of the user.
- -e, --email <email>¶
The email of the user.
- --user-access-token <user_access_token>¶
The access token of the user.
- --json¶
Get output in JSON format.
- --admin-access-token <admin_access_token>¶
Required The access token of an administrator.
REST API¶
The REANA Server offers a REST API for management workloads (workflows, jobs, tasks, etc.) running on REANA Cloud. Detailed REST API documentation can be found here.
Reana-Server Ping-functionality Flask-Blueprint.
Reana-Server User Endpoints.
Reana-Server workflow-functionality Flask-Blueprint.
- reana_server.rest.workflows.get_workflow_diff(workflow_id_or_name_a, workflow_id_or_name_b, user)[source]¶
- reana_server.rest.workflows.open_interactive_session(workflow_id_or_name, interactive_session_type, user)[source]¶
- reana_server.rest.workflows.prune_workspace(workflow_id_or_name, user, include_inputs=False, include_outputs=False)[source]¶
Changelog¶
0.9.3 (2024-03-04)¶
Build¶
Code refactoring¶
Code style¶
Continuous integration¶
Documentation¶
0.9.2 (2023-12-12)¶
Adds automated multi-platform container image building for amd64 and arm64 architectures.
Adds metadata labels to Dockerfile.
Changes workflow scheduler logging behaviour to also report the main reason behind scheduling errors to the users.
Fixes runtime uWSGI warning by rebuilding uWSGI with the PCRE support.
0.9.1 (2023-09-27)¶
Adds new
prune_workspace
endpoint to allow users to delete all the files of a workflow, specifying whether to also delete the inputs and/or the outputs.Adds new
interactive-session-cleanup
command that can be used by REANA administrators to close interactive sessions that are inactive for more than the specified number of days.Adds logic to support SSO with third-party Keycloak authentication services.
Adds the timestamp of when the workflow was stopped (
run_stopped_at
) to the workflow list and the workflow status endpoints.Adds the content of the
REANA_GITLAB_HOST
environment variable to the list of GitLab instances from which it is possible to launch a workflow.Adds progress meter to the logs of the periodic quota updater.
Changes CPU and disk quota calculations to improve the performance of periodic quota updater.
Changes the system status report to simplify and clarify the disk usage summary.
Changes
check-workflows
command to also check the presence of workspaces on the shared volume.Changes
check-workflows
command to not show in-sync runs by default. If needed, they can be shown using the new--show-all
option.Changes
launch
endpoint to also include the warnings of the validation of the workflow specification.Changes OpenAPI specification of the
info
endpoint to return the maximum inactivity time before automatic closure of interactive sessions.Changes
apispec
dependency version in order to be compatible withPyYAML
v6.Changes
reana-admin
command options to require the passing of--admin-access-token
argument more globally.Fixes the workflow priority calculation to avoid workflows stuck in the
queued
status when the number of allowed concurrent workflow is set to zero.Fixes GitLab integration to automatically redirect the user to the correct URL when the access request is accepted.
Fixes
quota-set-default-limits
command to propagate default quota limits to all users without custom quota limit values.Fixes authentication flow to correctly deny access to past revoked tokens in case the same user has also other new active tokens.
Fixes email templates to show the correct
kubectl
commands when REANA is deployed inside a non-default namespace or with a custom component name prefix.Fixes email sender for system emails to
notifications.email_config.sender
Helm value.Fixes email receiver for token request emails to use
notifications.email_config.receiver
Helm value.Fixes
start-scheduler
command to gracefully stop when being terminated.Fixes container image names to be Podman-compatible.
0.9.0 (2023-01-19)¶
Adds new
/api/launch
endpoint that allows running workflows from remote sources.Adds new
get_workflow_retention_rules
endpoint that allows to retrieve the workspace file retention rules of a workflow.Adds
queue-consume
command that can be used by REANA administrators to remove specific messages from the queue.Adds configuration environment variable to set an API rate limit for slow endpoints (
REANA_RATELIMIT_SLOW
).Adds REANA specification validation utilities.
Adds
retention-rules-apply
command that can be used by REANA administrators to apply pending retention rules.Adds
retention-rules-extend
command that can be used by REANA administrators to extend the duration of active retentions rules.Adds
check-workflows
command that can be used by REANA administrators to check for out-of-sync workflows and interactive sessions.Changes OpenAPI specification to include missing response schema elements and some other small enhancements.
Changes
/api/info
endpoint to also include the kubernetes maximum memory limit, the kubernetes default memory limit and the maximum workspace retention period.Changes
start_workflow
endpoint to validate the REANA specification of the workflow.Changes
create_workflow
endpoint to populate workspace retention rules for the workflow.Changes
start_workflow
endpoint to disallow restarting a workflow when retention rules are pending.Changes API rate limiter error messages to be more verbose.
Changes workflow scheduler to allow defining the checks needed to assess whether the cluster can start new workflows.
Changes the Invenio dependencies to the latest versions.
Changes OAuth configuration to enable the new CERN SSO.
Changes to PostgreSQL 12.13.
Changes GitLab integration to also retrieve user’s projects that are in groups and subgroups.
Changes the base image of the component to Ubuntu 20.04 LTS and reduces final Docker image size by removing build-time dependencies.
Fixes issue when irregular number formats are passed to
REANA_SCHEDULER_REQUEUE_COUNT
configuration environment variable.Fixes GitLab integration error reporting in case user exceeds CPU or Disk quota usage limits.
Fixes CERN OIDC authentication to possibly allow eduGAIN and social login users.
0.8.4 (2022-02-23)¶
Changes workflow scheduler to count number of workflow retries.
0.8.3 (2022-02-10)¶
Adds Kubernetes job memory limits validation before publishing workflow submission.
0.8.2 (2022-02-07)¶
Adds email validation to the
user-create
command used by the REANA administrators.Adds workflow name validation to the
create_workflow
endpoint.Changes
/api/info
endpoint to return a list of supported compute backends.Changes
/api/status
endpoint to calculate the cluster health status based on the availablity instead of the usage.
0.8.1 (2021-11-29)¶
Changes
quota-set
command used by the REANA administrators to use the resource type along with a resource name for specifying the resource.Changes email validation used in
create-admin-user
command by the REANA administrators to be more permissive.
0.8.0 (2021-11-22)¶
Adds users quota accounting.
Adds support for Snakemake workflow engine.
Adds
include_progress
andinclude_workspace_size
query args to workflow list endpoint.Adds workflow prioritization in the queue by complexity.
Adds
priority
andmin_job_memory
params to workflow submission publisher.Adds Yadage workflow specification loading to
start_workflow
endpoint.Adds a check in scheduler if at least one workflow job could be started in Kubernetes.
Adds configuration environment variable to set workflow scheduling policy (
REANA_WORKFLOW_SCHEDULING_POLICY
).Adds configuration environment variable to set a timeout between consuming workflows (
REANA_SCHEDULER_REQUEUE_SLEEP
).Adds configuration environment variable to set an API rate limiter (
REANA_RATELIMIT_AUTHENTICATED_USER
,REANA_RATELIMIT_GUEST_USER
).Adds new
info
endpoint allowing to retrieve information about cluster capabilities such as available workspaces.Changes workflow execution consumer to receive only one message at a time.
Changes to PostgreSQL 12.8.
0.7.6 (2021-07-05)¶
Changes internal dependencies.
0.7.5 (2021-04-28)¶
Adds support for listing files using glob patterns.
Adds support for glob patterns and directory downloads, packaging the content into a zip file.
0.7.4 (2021-03-17)¶
Adds configuration to set a timeout between
reana_ready
checks. (REANA_SCHEDULER_SECONDS_TO_WAIT_FOR_REANA_READY
)Fixes start workflow endpoint to work with unspecified
operational_options
parameterFixes workflow scheduling bug in which failed worfklows would count as running, reaching
REANA_MAX_CONCURRENT_BATCH_WORKFLOWS
and therefore, blocking thejob-submission
queue.
0.7.3 (2021-02-03)¶
Adds optional email confirmation step after users sign up.
Changes email notifications with enriched instructions on how to grant user tokens.
0.7.2 (2020-11-24)¶
Changes rate limiting defaults to allow up to 20 connections per second.
Fixes minor code warnings.
0.7.1 (2020-11-10)¶
Fixes REANA <-> GitLab synchronisation for projects having additional external webhooks.
Fixes restarting of Yadage and CWL workflows.
Fixes conflicting
kombu
installation requirements by requiring Celery version 4.Changes
/api/you
endpoint to include REANA server version information.
0.7.0 (2020-10-20)¶
Adds new endpoint to request user tokens.
Adds email notifications on relevant events such as user token granted/revoked.
Adds new templating system for notification email bodies.
Adds possibility to query logs for a single workflow step.
Adds endpoint to retrieve the workflow specification used for the workflow run.
Adds preview flag to download file endpoint.
Adds validation of submitted operational options before starting a workflow.
Adds possibility to upload empty files.
Adds new block size option to specify the type of units to use for disk size.
Adds a possibility to upload new workflow definitions before restarting a workflow.
Adds new command to generate status report for the REANA administrators; useful as a cronjob.
Adds user token management commands to grant and revoke user tokens.
Adds support for local user management.
Adds pinning of all Python dependencies allowing to easily rebuild component images at later times.
Fixes bug related to rescheduling deleted workflows.
Changes
REANA_URL
configuration variable to more preciseREANA_HOSTNAME
.Changes workflow list endpoint response payload to include workflow progress information.
Changes import/export commands with respect to new user model fields.
Changes submodule installation in editable mode for live code updates for developers.
Changes pre-requisites to Invenio-Accounts 1.3.0 to support REST API.
Changes
/api/me
to/api/you
endpoint due to conflict with Invenio-Accounts.Changes base image to use Python 3.8.
Changes code formatting to respect
black
coding style.Changes documentation to single-page layout.
0.6.1 (2020-05-25)¶
Upgrades REANA-Commons package using latest Kubernetes Python client version.
Pins Flask and Invenio dependencies to fix REANA 0.6 installation troubles.
0.6.0 (2019-12-20)¶
Fixes bug with big file uploads by using data streaming.
Adds user login endpoints using OAuth, currently configured to work with CERN SSO but extensible to use other OAuth providers such as GitHub, more in Invenio-OAuthClient.
Adds endpoints to integrate with GitLab (for retrieving user projects and creating/deleting webhooks).
Adds new endpoint
/me
to retrieve user information.Improves security by allowing requests only with
REANA_URL
in the host header, avoiding host header injection attacks.Initialisation logs moved from
stdout
to/var/log/reana-server-init-output.log
.
0.5.0 (2019-04-23)¶
Adds new endpoint to compare two workflows. The output is a
git
like diff which can be configured to show differences at metadata level, workspace level or both.Adds new endpoint to retrieve workflow parameters.
Adds new endpoint to query the disk usage of a given workspace.
Adds new endpoints to delete and move files whithin the workspace.
Adds new endpoints to open and close interactive sessions inside the workspace.
Workflow start does not send start requests to REANA Workflow Controller straight away, instead it will decide whether REANA can execute it or queue it depending on a set of conditions, currently it depends on the number of running jobs in the cluster.
Adds new administrator command to export and import all REANA users.
0.4.0 (2018-11-06)¶
Improves REST API documentation rendering.
Enhances test suite and increases code coverage.
Changes license to MIT.
0.3.1 (2018-09-07)¶
Harmonises date and time outputs amongst various REST API endpoints.
Pins REANA-Commons, REANA-DB and Bravado dependencies.
0.3.0 (2018-08-10)¶
Adds support of Serial workflows.
Adds API protection with API tokens.
0.2.0 (2018-04-19)¶
Adds support of Common Workflow Language workflows.
Adds support of specifying workflow names in REST API requests.
Improves error messages and information.
0.1.0 (2018-01-30)¶
Initial public release.
Contributing¶
Bug reports, issues, feature requests, and other contributions are welcome. If you find a demonstrable problem that is caused by the REANA code, please:
Search for already reported problems.
Check if the issue has been fixed or is still reproducible on the latest
master
branch.Create an issue, ideally with a test case.
If you create a pull request fixing a bug or implementing a feature, you can run the tests to ensure that everything is operating correctly:
$ ./run-tests.sh
Each pull request should preserve or increase code coverage.
License¶
MIT License
Copyright (C) 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024 CERN.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
In applying this license, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction.