REANA-Server

image image image image image image

REANA-Server is a component of the REANA reusable and reproducible research data analysis platform. It implements the API Server that takes and performs REST API calls issued by REANA clients.

Features

  • offers rich REST API services for REANA clients

  • transmits REST API requests to appropriate REANA cloud components

  • REST API to run research analysis workflows on compute clouds

  • REST API to list submitted workflows and enquire about their statuses

  • REST API to manage analysis files

  • REST API to download results of finished analysis workflows

  • REST API to find the differences between two workflows (git like output)

Usage

The detailed information on how to install and use REANA can be found in docs.reana.io.

CLI API

flask reana-admin

REANA administration commands.

flask reana-admin [OPTIONS] COMMAND [ARGS]...

check-workflows

Check consistency of selected workflow run statuses between database, message queue and Kubernetes.

flask reana-admin check-workflows [OPTIONS]

Options

--date-start <date_start>

Default value is 24 hours ago.

--date-end <date_end>

Default value is now.

-a, --all

Show all workflows/sessions/workspaces, even if in-sync.

--admin-access-token <admin_access_token>

Required The access token of an administrator.

create-admin-user

Create default user.

This user has the administrator role and can retrieve other user information as well as create new users.

flask reana-admin create-admin-user [OPTIONS]

Options

-e, --email <email>

Required The email of the admin user.

-p, --password <password>

Required

-i, --id <id_>

interactive-session-cleanup

Close inactive interactive sessions.

flask reana-admin interactive-session-cleanup [OPTIONS]

Options

-d, --days <days>

Required Close interactive sessions that are inactive for more than the specified number of days.

--dry-run

Show which interactive sessions would be closed, without closing them. [default=False]

--admin-access-token <admin_access_token>

Required The access token of an administrator.

queue-consume

Start consuming specified queue and remove selected messages.

By default, you will need to specify either “-k” or “-i” options otherwise the command will return an error.

If -k option is specified, messages that have property values specified in -v will be deleted.

If -i option is specified, for every message, user will be asked what to do.

If -k and -i are specified together, for every message that matches property values in -v, user will be asked whether to delete it or not.

flask reana-admin queue-consume [OPTIONS]

Options

-q, --queue-name <queue_name>

Required Name of the queue that will be consumed, e.g workflow-submission

-k, --key <key>

Key of the property that will be used to filter the messages in the queue, e.g workflow_name_or_id

-v, --values-to-delete <values_to_delete>

List of property values used to filter messages that will be removed from the queue, e.g UUID of a workflow

-i, --interactive

Manually decide which messages to remove from the queue.

--admin-access-token <admin_access_token>

Required The access token of an administrator.

quota-resources

List available quota resources.

flask reana-admin quota-resources [OPTIONS]

quota-set

Set quota limits to the given users per resource.

flask reana-admin quota-set [OPTIONS]

Options

-e, --email <emails>

Required The emails of the users. E.g. –email johndoe@example.org –email janedoe@example.org

-r, --resource <resource_type>

Specify quota resource. e.g. cpu, disk.

-n, --resource-name <resource_name>

Name of resource.

-l, --limit <limit>

Required New limit in canonical unit.

--admin-access-token <admin_access_token>

Required The access token of an administrator.

quota-set-default-limits

Set default quota limits for users who do not have any custom limits

defined.

Note that any previously set user limits, either via old defaults or via custom settings, will be kept during the upgrade, and won’t be automatically updated to match the new default limit value.

flask reana-admin quota-set-default-limits [OPTIONS]

Options

--admin-access-token <admin_access_token>

Required The access token of an administrator.

quota-usage

List quota usage of users.

flask reana-admin quota-usage [OPTIONS]

Options

--id <id>

The id of the user.

-e, --email <email>

The email of the user.

--user-access-token <user_access_token>

The access token of the user.

--json

Get output in JSON format.

-h, --human-readable

Show quota usage values in human readable format.

--admin-access-token <admin_access_token>

Required The access token of an administrator.

retention-rules-apply

Apply pending retentions rules.

flask reana-admin retention-rules-apply [OPTIONS]

Options

--dry-run

Show the pending retention rules without applying them. [default=False]

--force-date <force_date>

Force desired date and time when deciding which rules to apply.

--yes-i-am-sure

Do not ask for confirmation when doing potentially dangerous operations.

-e, --email <email>

The email of the user.

--id <id_>

The id of the user.

-w, --workflow <workflow_uuid>

The id of the workflow.

--admin-access-token <admin_access_token>

Required The access token of an administrator.

retention-rules-extend

Extend active retentions rules.

flask reana-admin retention-rules-extend [OPTIONS]

Options

-w, --workflow <workflow_uuid>

Required The id of the workflow.

-d, --days <days>

Required Number of days to extend the rules.

--admin-access-token <admin_access_token>

Required The access token of an administrator.

status-report

Get a status report of the REANA system.

flask reana-admin status-report [OPTIONS]

Options

--type <types>

Type of information to be displayed?

Options:

interactive-sessions | workflows | users | system | storage | nodes | pods | jobs | quota-usage | all

-e, --email <email>

Send the status by email to the configured receiver.

--admin-access-token <admin_access_token>

Required The access token of an administrator.

token-grant

Grant a token to the selected user.

flask reana-admin token-grant [OPTIONS]

Options

--admin-access-token <admin_access_token>

Required The access token of an administrator.

--id <id_>

The id of the user.

-e, --email <email>

The email of the user.

token-revoke

Revoke selected user’s token.

flask reana-admin token-revoke [OPTIONS]

Options

--admin-access-token <admin_access_token>

Required The access token of an administrator.

--id <id_>

The id of the user.

-e, --email <email>

The email of the user.

user-create

Create a new user.

flask reana-admin user-create [OPTIONS]

Options

-e, --email <email>

Required The email of the user.

--user-access-token <user_access_token>

The access token of the user.

--admin-access-token <admin_access_token>

Required The access token of an administrator.

user-export

Export all users in current REANA cluster.

flask reana-admin user-export [OPTIONS]

Options

--admin-access-token <admin_access_token>

Required The access token of an administrator.

user-import

Import users from file.

flask reana-admin user-import [OPTIONS]

Options

--admin-access-token <admin_access_token>

Required The access token of an administrator.

-f, --file <file_>

A CSV file containing a list of REANA users.

user-list

List users according to the search criteria.

flask reana-admin user-list [OPTIONS]

Options

--id <id>

The id of the user.

-e, --email <email>

The email of the user.

--user-access-token <user_access_token>

The access token of the user.

--json

Get output in JSON format.

--admin-access-token <admin_access_token>

Required The access token of an administrator.

REST API

The REANA Server offers a REST API for management workloads (workflows, jobs, tasks, etc.) running on REANA Cloud. Detailed REST API documentation can be found here.

Reana-Server Ping-functionality Flask-Blueprint.

reana_server.rest.ping.ping()[source]

Ping the server (healthcheck)

Reana-Server User Endpoints.

reana_server.rest.users.get_you(user)[source]

Gets information about authenticated user.

reana_server.rest.users.request_token(user)[source]

Requests a new access token for the authenticated user.

Reana-Server workflow-functionality Flask-Blueprint.

reana_server.rest.workflows.close_interactive_session(workflow_id_or_name, user)[source]

Close an interactive workflow session.

reana_server.rest.workflows.create_workflow(user)[source]

Creates a new workflow based on a REANA specification file.

reana_server.rest.workflows.delete_file(workflow_id_or_name, file_name, user)[source]

Delete the specified file.

reana_server.rest.workflows.download_file(workflow_id_or_name, file_name, user)[source]

Returns the requested file.

reana_server.rest.workflows.get_files(workflow_id_or_name, user, **kwargs)[source]

Returns the workspace file list.

reana_server.rest.workflows.get_workflow_diff(workflow_id_or_name_a, workflow_id_or_name_b, user)[source]

Get diff between two workflows.

reana_server.rest.workflows.get_workflow_disk_usage(workflow_id_or_name, user)[source]

Get disk usage of a workflow.

reana_server.rest.workflows.get_workflow_logs(workflow_id_or_name, user, **kwargs)[source]

Get workflow logs of a workflow.

reana_server.rest.workflows.get_workflow_parameters(workflow_id_or_name, user)[source]

Get parameters of a workflow.

reana_server.rest.workflows.get_workflow_retention_rules(workflow_id_or_name, user)[source]

Get the retention rules of a workflow.

reana_server.rest.workflows.get_workflow_specification(workflow_id_or_name, user)[source]

Get the specification used for this workflow run.

reana_server.rest.workflows.get_workflow_status(workflow_id_or_name, user)[source]

Get status of a workflow.

reana_server.rest.workflows.get_workflows(user, **kwargs)[source]

Returns list of all current workflows in REANA.

reana_server.rest.workflows.move_files(workflow_id_or_name, user)[source]

Move files within workspace.

reana_server.rest.workflows.open_interactive_session(workflow_id_or_name, interactive_session_type, user)[source]

Start an interactive session inside the workflow workspace.

reana_server.rest.workflows.prune_workspace(workflow_id_or_name, user, include_inputs=False, include_outputs=False)[source]

Prune the workspace’s files.

reana_server.rest.workflows.set_workflow_status(workflow_id_or_name, user)[source]

Set status of a workflow.

reana_server.rest.workflows.start_workflow(workflow_id_or_name, user)[source]

Start workflow.

reana_server.rest.workflows.upload_file(workflow_id_or_name, user)[source]

Adds a file to the workspace.

Changelog

0.9.3 (2024-03-04)

Build

  • deps: pin invenio-userprofiles to 1.2.4 (#665) (d6cb168)

  • docker: non-editable submodules in “latest” mode (#656) (d16fefb)

  • python: bump all required packages as of 2024-03-04 (#674) (f40b82f)

  • python: bump shared modules (#676) (47ad3ca)

  • python: bump shared REANA packages as of 2024-03-04 (#674) (aa18394)

Code refactoring

Code style

Continuous integration

  • commitlint: addition of commit message linter (#665) (2b43ecc)

  • commitlint: allow release commit style (#675) (e0299ef)

  • commitlint: check for the presence of concrete PR number (#669) (87c6145)

  • pytest: move to PostgreSQL 14.10 (#672) (e888ddd)

  • release-please: initial configuration (#665) (1d5e7c5)

  • release-please: update version in Dockerfile/OpenAPI specs (#668) (3b3dc41)

  • shellcheck: fix exit code propagation (#669) (d7eac6b)

Documentation

  • authors: complete list of contributors (#673) (71b3f38)

0.9.2 (2023-12-12)

  • Adds automated multi-platform container image building for amd64 and arm64 architectures.

  • Adds metadata labels to Dockerfile.

  • Changes workflow scheduler logging behaviour to also report the main reason behind scheduling errors to the users.

  • Fixes runtime uWSGI warning by rebuilding uWSGI with the PCRE support.

0.9.1 (2023-09-27)

  • Adds new prune_workspace endpoint to allow users to delete all the files of a workflow, specifying whether to also delete the inputs and/or the outputs.

  • Adds new interactive-session-cleanup command that can be used by REANA administrators to close interactive sessions that are inactive for more than the specified number of days.

  • Adds logic to support SSO with third-party Keycloak authentication services.

  • Adds the timestamp of when the workflow was stopped (run_stopped_at) to the workflow list and the workflow status endpoints.

  • Adds the content of the REANA_GITLAB_HOST environment variable to the list of GitLab instances from which it is possible to launch a workflow.

  • Adds progress meter to the logs of the periodic quota updater.

  • Changes CPU and disk quota calculations to improve the performance of periodic quota updater.

  • Changes the system status report to simplify and clarify the disk usage summary.

  • Changes check-workflows command to also check the presence of workspaces on the shared volume.

  • Changes check-workflows command to not show in-sync runs by default. If needed, they can be shown using the new --show-all option.

  • Changes launch endpoint to also include the warnings of the validation of the workflow specification.

  • Changes OpenAPI specification of the info endpoint to return the maximum inactivity time before automatic closure of interactive sessions.

  • Changes apispec dependency version in order to be compatible with PyYAML v6.

  • Changes reana-admin command options to require the passing of --admin-access-token argument more globally.

  • Fixes the workflow priority calculation to avoid workflows stuck in the queued status when the number of allowed concurrent workflow is set to zero.

  • Fixes GitLab integration to automatically redirect the user to the correct URL when the access request is accepted.

  • Fixes quota-set-default-limits command to propagate default quota limits to all users without custom quota limit values.

  • Fixes authentication flow to correctly deny access to past revoked tokens in case the same user has also other new active tokens.

  • Fixes email templates to show the correct kubectl commands when REANA is deployed inside a non-default namespace or with a custom component name prefix.

  • Fixes email sender for system emails to notifications.email_config.sender Helm value.

  • Fixes email receiver for token request emails to use notifications.email_config.receiver Helm value.

  • Fixes start-scheduler command to gracefully stop when being terminated.

  • Fixes container image names to be Podman-compatible.

0.9.0 (2023-01-19)

  • Adds new /api/launch endpoint that allows running workflows from remote sources.

  • Adds new get_workflow_retention_rules endpoint that allows to retrieve the workspace file retention rules of a workflow.

  • Adds queue-consume command that can be used by REANA administrators to remove specific messages from the queue.

  • Adds configuration environment variable to set an API rate limit for slow endpoints (REANA_RATELIMIT_SLOW).

  • Adds REANA specification validation utilities.

  • Adds retention-rules-apply command that can be used by REANA administrators to apply pending retention rules.

  • Adds retention-rules-extend command that can be used by REANA administrators to extend the duration of active retentions rules.

  • Adds check-workflows command that can be used by REANA administrators to check for out-of-sync workflows and interactive sessions.

  • Changes OpenAPI specification to include missing response schema elements and some other small enhancements.

  • Changes /api/info endpoint to also include the kubernetes maximum memory limit, the kubernetes default memory limit and the maximum workspace retention period.

  • Changes start_workflow endpoint to validate the REANA specification of the workflow.

  • Changes create_workflow endpoint to populate workspace retention rules for the workflow.

  • Changes start_workflow endpoint to disallow restarting a workflow when retention rules are pending.

  • Changes API rate limiter error messages to be more verbose.

  • Changes workflow scheduler to allow defining the checks needed to assess whether the cluster can start new workflows.

  • Changes the Invenio dependencies to the latest versions.

  • Changes OAuth configuration to enable the new CERN SSO.

  • Changes to PostgreSQL 12.13.

  • Changes GitLab integration to also retrieve user’s projects that are in groups and subgroups.

  • Changes the base image of the component to Ubuntu 20.04 LTS and reduces final Docker image size by removing build-time dependencies.

  • Fixes issue when irregular number formats are passed to REANA_SCHEDULER_REQUEUE_COUNT configuration environment variable.

  • Fixes GitLab integration error reporting in case user exceeds CPU or Disk quota usage limits.

  • Fixes CERN OIDC authentication to possibly allow eduGAIN and social login users.

0.8.4 (2022-02-23)

  • Changes workflow scheduler to count number of workflow retries.

0.8.3 (2022-02-10)

  • Adds Kubernetes job memory limits validation before publishing workflow submission.

0.8.2 (2022-02-07)

  • Adds email validation to the user-create command used by the REANA administrators.

  • Adds workflow name validation to the create_workflow endpoint.

  • Changes /api/info endpoint to return a list of supported compute backends.

  • Changes /api/status endpoint to calculate the cluster health status based on the availablity instead of the usage.

0.8.1 (2021-11-29)

  • Changes quota-set command used by the REANA administrators to use the resource type along with a resource name for specifying the resource.

  • Changes email validation used in create-admin-user command by the REANA administrators to be more permissive.

0.8.0 (2021-11-22)

  • Adds users quota accounting.

  • Adds support for Snakemake workflow engine.

  • Adds include_progress and include_workspace_size query args to workflow list endpoint.

  • Adds workflow prioritization in the queue by complexity.

  • Adds priority and min_job_memory params to workflow submission publisher.

  • Adds Yadage workflow specification loading to start_workflow endpoint.

  • Adds a check in scheduler if at least one workflow job could be started in Kubernetes.

  • Adds configuration environment variable to set workflow scheduling policy (REANA_WORKFLOW_SCHEDULING_POLICY).

  • Adds configuration environment variable to set a timeout between consuming workflows (REANA_SCHEDULER_REQUEUE_SLEEP).

  • Adds configuration environment variable to set an API rate limiter (REANA_RATELIMIT_AUTHENTICATED_USER, REANA_RATELIMIT_GUEST_USER).

  • Adds new info endpoint allowing to retrieve information about cluster capabilities such as available workspaces.

  • Changes workflow execution consumer to receive only one message at a time.

  • Changes to PostgreSQL 12.8.

0.7.6 (2021-07-05)

  • Changes internal dependencies.

0.7.5 (2021-04-28)

  • Adds support for listing files using glob patterns.

  • Adds support for glob patterns and directory downloads, packaging the content into a zip file.

0.7.4 (2021-03-17)

  • Adds configuration to set a timeout between reana_ready checks. (REANA_SCHEDULER_SECONDS_TO_WAIT_FOR_REANA_READY)

  • Fixes start workflow endpoint to work with unspecified operational_options parameter

  • Fixes workflow scheduling bug in which failed worfklows would count as running, reaching REANA_MAX_CONCURRENT_BATCH_WORKFLOWS and therefore, blocking the job-submission queue.

0.7.3 (2021-02-03)

  • Adds optional email confirmation step after users sign up.

  • Changes email notifications with enriched instructions on how to grant user tokens.

0.7.2 (2020-11-24)

  • Changes rate limiting defaults to allow up to 20 connections per second.

  • Fixes minor code warnings.

0.7.1 (2020-11-10)

  • Fixes REANA <-> GitLab synchronisation for projects having additional external webhooks.

  • Fixes restarting of Yadage and CWL workflows.

  • Fixes conflicting kombu installation requirements by requiring Celery version 4.

  • Changes /api/you endpoint to include REANA server version information.

0.7.0 (2020-10-20)

  • Adds new endpoint to request user tokens.

  • Adds email notifications on relevant events such as user token granted/revoked.

  • Adds new templating system for notification email bodies.

  • Adds possibility to query logs for a single workflow step.

  • Adds endpoint to retrieve the workflow specification used for the workflow run.

  • Adds preview flag to download file endpoint.

  • Adds validation of submitted operational options before starting a workflow.

  • Adds possibility to upload empty files.

  • Adds new block size option to specify the type of units to use for disk size.

  • Adds a possibility to upload new workflow definitions before restarting a workflow.

  • Adds new command to generate status report for the REANA administrators; useful as a cronjob.

  • Adds user token management commands to grant and revoke user tokens.

  • Adds support for local user management.

  • Adds pinning of all Python dependencies allowing to easily rebuild component images at later times.

  • Fixes bug related to rescheduling deleted workflows.

  • Changes REANA_URL configuration variable to more precise REANA_HOSTNAME.

  • Changes workflow list endpoint response payload to include workflow progress information.

  • Changes import/export commands with respect to new user model fields.

  • Changes submodule installation in editable mode for live code updates for developers.

  • Changes pre-requisites to Invenio-Accounts 1.3.0 to support REST API.

  • Changes /api/me to /api/you endpoint due to conflict with Invenio-Accounts.

  • Changes base image to use Python 3.8.

  • Changes code formatting to respect black coding style.

  • Changes documentation to single-page layout.

0.6.1 (2020-05-25)

  • Upgrades REANA-Commons package using latest Kubernetes Python client version.

  • Pins Flask and Invenio dependencies to fix REANA 0.6 installation troubles.

0.6.0 (2019-12-20)

  • Fixes bug with big file uploads by using data streaming.

  • Adds user login endpoints using OAuth, currently configured to work with CERN SSO but extensible to use other OAuth providers such as GitHub, more in Invenio-OAuthClient.

  • Adds endpoints to integrate with GitLab (for retrieving user projects and creating/deleting webhooks).

  • Adds new endpoint /me to retrieve user information.

  • Improves security by allowing requests only with REANA_URL in the host header, avoiding host header injection attacks.

  • Initialisation logs moved from stdout to /var/log/reana-server-init-output.log.

0.5.0 (2019-04-23)

  • Adds new endpoint to compare two workflows. The output is a git like diff which can be configured to show differences at metadata level, workspace level or both.

  • Adds new endpoint to retrieve workflow parameters.

  • Adds new endpoint to query the disk usage of a given workspace.

  • Adds new endpoints to delete and move files whithin the workspace.

  • Adds new endpoints to open and close interactive sessions inside the workspace.

  • Workflow start does not send start requests to REANA Workflow Controller straight away, instead it will decide whether REANA can execute it or queue it depending on a set of conditions, currently it depends on the number of running jobs in the cluster.

  • Adds new administrator command to export and import all REANA users.

0.4.0 (2018-11-06)

  • Improves REST API documentation rendering.

  • Enhances test suite and increases code coverage.

  • Changes license to MIT.

0.3.1 (2018-09-07)

  • Harmonises date and time outputs amongst various REST API endpoints.

  • Pins REANA-Commons, REANA-DB and Bravado dependencies.

0.3.0 (2018-08-10)

  • Adds support of Serial workflows.

  • Adds API protection with API tokens.

0.2.0 (2018-04-19)

  • Adds support of Common Workflow Language workflows.

  • Adds support of specifying workflow names in REST API requests.

  • Improves error messages and information.

0.1.0 (2018-01-30)

  • Initial public release.

Contributing

Bug reports, issues, feature requests, and other contributions are welcome. If you find a demonstrable problem that is caused by the REANA code, please:

  1. Search for already reported problems.

  2. Check if the issue has been fixed or is still reproducible on the latest master branch.

  3. Create an issue, ideally with a test case.

If you create a pull request fixing a bug or implementing a feature, you can run the tests to ensure that everything is operating correctly:

$ ./run-tests.sh

Each pull request should preserve or increase code coverage.

License

MIT License

Copyright (C) 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024 CERN.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

In applying this license, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction.

Authors

The list of contributors in alphabetical order: