5.4. Server#

The server has a central function in the vantage6 architecture. It stores in the database which organizations, collaborations, users, etc. exist. It allows the users and nodes to authenticate and subsequently interact through the API the server hosts. Finally, it also communicates with authenticated nodes and users via the socketIO server that is run here.

5.4.1. Main server class#

class ServerApp(ctx)#

Vantage6 server instance.

Variables:: ctx (ServerContext) – Context object that contains the configuration of the server.

configure_api()#: Define global API output and its structure.

configure_flask()#: Configure the Flask settings of the vantage6 server.

configure_jwt()#: Configure JWT authentication.

static configure_logging()#: Set third party loggers to a warning level

load_resources()#: Import the modules containing API resources.

start()#

Start the server.

Before server is really started, some database settings are checked and (re)set where appropriate.

5.4.2. Starting the server#

run_server(config, environment='prod', system_folders=True)#

Run a vantage6 server.

Parameters:

config (str) – Configuration file path
environment (str) – Configuration environment to use.
system_folders (bool) – Whether to use system or user folders. Default is True.

Returns:

A running instance of the vantage6 server

Return type:

ServerApp

Warning

Note that the run_server function is normally not used directly to start the server, but is used as utility function in places that start the server. The recommended way to start a server is using uWSGI as is done in vserver start.

run_dev_server(server_app, *args, **kwargs)#

Run a vantage6 development server (outside of a Docker container).

Parameters:: server_app (ServerApp) – Instance of a vantage6 server
Return type:: None

5.4.3. Permission management#

class Scope(value)#

Enumerator of all available scopes

COLLABORATION = 'col'#

GLOBAL = 'glo'#

ORGANIZATION = 'org'#

OWN = 'own'#

class Operation(value)#

Enumerator of all available operations

CREATE = 'c'#

DELETE = 'd'#

EDIT = 'e'#

VIEW = 'v'#

class RuleCollection(name)#

Class that tracks a set of all rules for a certain resource name

Parameters:: name (str) – Name of the resource endpoint (e.g. node, organization, user)

add(scope, operation)#

Add a rule to the rule collection

Parameters:

scope (Scope) – Scope within which to apply the rule
operation (Operation) – What operation the rule applies to

Return type:

None

class PermissionManager#

Loads the permissions and syncs rules in database with rules defined in the code

appender(name)#

Add a module’s rules to the rule collection

Parameters:: name (str) – The name of the module whose rules are to be registered
Returns:: A callable register_rule function
Return type:: Callable

assign_rule_to_container(resource, scope, operation)#

Assign a rule to the container role.

Parameters:

resource (str) – Resource that the rule applies to
scope (Scope) – Scope that the rule applies to
operation (Operation) – Operation that the rule applies to

Return type:

None

static assign_rule_to_fixed_role(fixedrole, resource, scope, operation)#

Attach a rule to a fixed role (not adjustable by users).

Parameters:

fixedrole (str) – Name of the fixed role that the rule should be added to
resource (str) – Resource that the rule applies to
scope (Scope) – Scope that the rule applies to
operation (Operation) – Operation that the rule applies to

Return type:

None

assign_rule_to_node(resource, scope, operation)#

Assign a rule to the Node role.

Parameters:

resource (str) – Resource that the rule applies to
scope (Scope) – Scope that the rule applies to
operation (Operation) – Operation that the rule applies to

Return type:

None

assign_rule_to_root(name, scope, operation)#

Assign a rule to the root role.

Return type:: None

resource: str: Resource that the rule applies to
scope: Scope: Scope that the rule applies to
operation: Operation: Operation that the rule applies to

collection(name)#

Get a RuleCollection object. If it doesn’t exist yet, it will be created.

Parameters:: name (str) – Name of the module whose RuleCollection is to be obtained or created
Returns:: The collection of rules belonging to the module name
Return type:: RuleCollection

load_rules_from_resources()#

Collect all permission rules from all registered API resources

Return type:: None

register_rule(resource, scope, operation, description=None, assign_to_node=False, assign_to_container=False)#

If a rule already exists, nothing is done. This rule can be used in API endpoints to determine if a user, node or container can do a certain operation in a certain scope.

Parameters:

resource (str) – API resource that the rule applies to
scope (Scope) – Scope of the rule
operation (Operation) – Operation of the rule
description (String, optional) – Human readable description where the rule is used for, by default None
assign_to_node (bool, optional) – Whether rule should be assigned to the node role or not. Default False
assign_to_container (bool, optional) – Whether rule should be assigned to the container role or not. Default False

Return type:

None

static rule_exists_in_db(name, scope, operation)#

Check if the rule exists in the DB.

Parameters:

name (str) – Name of the rule
scope (Scope) – Scope of the rule
operation (Operation) – Operation of the rule

Returns:

Whenever this rule exists in the database or not

Return type:

bool

static verify_user_rules(rules)#

Check if an user, node or container has all the rules

Parameters:: rules (List[Rule]) – List of rules that user is checked to have
Returns:: False if user has all rules, else a dict with a message
Return type:: Union[dict, bool]

5.4.4. Socket functionality#

class DefaultSocketNamespace(namespace=None)#

This is the default SocketIO namespace. It is used for all the long-running socket communication between the server and the clients. The clients of the socket connection are nodes and users.

When socket communication is received from one of the clients, the functions in this class are called to execute the corresponding action.

on_algorithm_status_change(data)#

An algorithm container has changed its status. This status change may be that the algorithm has finished, crashed, etc. Here we notify the collaboration of the change.

Parameters:

data (Dict) –

Dictionary containing parameters on the updated algorithm status. It should look as follows:

{: # node_id where algorithm container was running “node_id”: 1, # new status of algorithm container “status”: “active”, # result_id for which the algorithm was running “result_id”: 1, # collaboration_id for which the algorithm was running “collaboration_id”: 1

}

Return type:

None

on_connect()#

A new incoming connection request from a client.

New connections are authenticated using their JWT authorization token which is obtained from the REST API. A session is created for each connected client, and lives as long as the connection is active. Each client is assigned to rooms based on their permissions.

Nodes that are connecting are also set to status ‘online’.

Return type:: None

Note

Note that reconnecting clients are treated the same as new clients.

on_disconnect()#

Client that disconnects is removed from all rooms they were in.

If nodes disconnect, their status is also set to offline and users may be alerted to that. Also, any information on the node (e.g. configuration) is removed from the database.

Return type:: None

on_error(e)#

An receiving an error from a client, log it.

Parameters:: e (str) – Error message that is being displayed in the server log
Return type:: None

on_message(message)#

On receiving a message from a client, log it.

Parameters:: message (str) – Message that is going to be displayed in the server log
Return type:: None

on_node_info_update(node_config)#

A node sends information about its configuration and other properties. Store this in the database for the duration of the node’s session.

Parameters:: node_config (dict) – Dictionary containing the node’s configuration.
Return type:: None

on_ping()#

A client sends a ping to the server. The server detects who sent the ping and sets them as online.

Return type:: None

5.4.5. API endpoints#

Warning

The API endpoints are also documented on the /apidocs endpoint of the server (e.g. https://petronas.vantage6.ai/apidocs). We are therefore not including the API documentation here. Instead, we merely list the supporting functions and classes.

5.4.6. SQLAlchemy models#

Helper (base) classes#

class Database(*args, **kwargs)#

Database class that is used to connect to the database and create the database session.

The database is created as a singleton, so that it can be destroyed (as opposed to a module). This is especially useful when creating unit tests in which we want fresh databases every now and then.

add_col_to_table(column, table_cls)#

Database operation to add column to Table

Parameters:

column (Column) – The SQLAlchemy model column that is to be added
table_cls (Table) – The SQLAlchemy table to which the column is to be added

Return type:

None

add_missing_columns()#

Check database tables to see if columns are missing that are described in the SQLAlchemy models, and add the missing columns

Return type:: None

clear_data()#: Clear all data from the database.

close()#: Delete all tables and close the database connection. Only used for unit testing.

connect(uri='sqlite:////tmp/test.db', allow_drop_all=False)#

Connect to the database.

Parameters:

uri (str) – URI of the database. Defaults to a sqlite database in /tmp.
allow_drop_all (bool, optional) – If True, the database can be dropped. Defaults to False.

drop_all()#: Drop all tables in the database.

get_non_existing_columns(table_cls, table_name)#

Return a list of columns that are defined in the SQLAlchemy model, but are not present in the database

Parameters:

table_cls (Table) – The table that is evaluated
table_name (str) – The name of the table

Returns:

List of SQLAlchemy Column objects that are present in the model, but not in the database

Return type:

List[Column]

static is_column_missing(column, column_names, table_name)#

Check if column is missing in the table

Parameters:

column (Column) – The column that is evaluated
column_names (List[str]) – A list of all column names in the table
table_name (str) – The name of the table the column resides in

Returns:

True if column is not in the table or a parent table

Return type:

boolean

class DatabaseSessionManager#

Class to manage DB sessions.

There are 2 different ways a session can be obtained. Either a session used within a request or a session used elsewhere (e.g. socketIO event, iPython or within the application itself).

In case of the Flask request, the session is stored in the flask global g. Then, it can be accessed in every endpoint.

In all other cases the session is attached to the db module.

static clear_session()#

Clear the session. If we are in a flask request, the session is cleared from the flask global g. Otherwise, the session is removed from the db module.

Return type:: None

static get_session()#

Get a session. Creates a new session if none exists.

Returns:: A database session
Return type:: Session

static in_flask_request()#

Check if we are in a flask request.

Returns:: True if we are in a flask request, False otherwise
Return type:: boolean

static new_session()#

Create a new session. If we are in a flask request, the session is stored in the flask global g. Otherwise, the session is stored in the db module.

Return type:: None

class ModelBase#

Declarative base that defines default attributes. All data models inherit from this class.

delete()#

Delete the object from the database.

Return type:: None

classmethod get(id_=None)#

Get a single object by its id, or a list of objects when no id is specified.

Parameters:: id (int, optional) – The id of the object to get. If not specified, return all.

classmethod help()#

Print a help message for the class.

Return type:: None

save()#

Save the object to the database.

Return type:: None

Database models for the API resources#

class AlgorithmPort(**kwargs)#

Table that describes which algorithms are reachable via which ports

Each algorithm with a VPN connection can claim multiple ports via the Dockerfile EXPOSE and LABEL commands. These claims are saved in this table. Each algorithm container belongs to a single Result.

Variables:

port (int) – The port number that is claimed by the algorithm
result_id (int) – The id of the Result that this port belongs to
label (str) – The label that is claimed by the algorithm
result (Result) – The Result that this port belongs to

class Authenticatable(**kwargs)#

Parent table of database entities that can authenticate.

Entities that can authenticate are nodes and users. Containers can also authenticate but these are authenticated indirectly through the nodes.

static hash(secret)#

Hash a secret using bcrypt.

Parameters:: secret (str) – Secret to be hashed
Returns:: Hashed secret
Return type:: str

class Collaboration(**kwargs)#

Table that describes which collaborations are available.

Collaborations are combinations of one or more organizations that do studies together. Each Organization has a Node for each collaboration that it is part of. Within a collaboration multiple Task can be executed.

Variables:

name (str) – Name of the collaboration
encrypted (bool) – Whether the collaboration is encrypted or not
organizations (list[Organization]) – List of organizations that are part of this collaboration
nodes (list[Node]) – List of nodes that are part of this collaboration
tasks (list[Task]) – List of tasks that are part of this collaboration

classmethod find_by_name(name)#

Find Collaboration by its name.

Note

If multiple collaborations share the same name, the first collaboration found is returned.

Parameters:: name (str) – Name of the collaboration
Returns:: Collaboration with the given name, or None if no collaboration with the given name exists.
Return type:: Union[Collaboration, None]

get_node_from_organization(organization)#

Returns the node that is part of the given Organization.

Parameters:: organization (Organization) – Organization
Returns:: Node for the given organization for this collaboration, or None if there is no node for the given organization.
Return type:: Union[Node, None]

get_nodes_from_organizations(ids)#

Returns a subset of nodes that are part of the given organizations.

Parameters:: ids (list[int]) – List of organization ids
Returns:: List of nodes that are part of the given organizations
Return type:: list[Node]

get_organization_ids()#

Returns a list of organization ids that are part of this collaboration.

Returns:: List of organization ids
Return type:: list[int]

get_task_ids()#

Returns a list of task ids that are part of this collaboration.

Returns:: List of task ids
Return type:: list[int]

classmethod name_exists(name)#

Check if a collaboration with the given name exists.

Parameters:: name (str) – Name of the collaboration
Returns:: True if a collaboration with the given name exists, else False
Return type:: bool

class Node(**kwargs)#

Bases: Authenticatable

Table that contains all registered nodes.

Variables:

id (int) – Primary key
name (str) – Name of the node
api_key (str) – API key of the node
collaboration (Collaboration) – Collaboration that the node belongs to
organization (Organization) – Organization that the node belongs to

check_key(key)#

Checks if the provided key matches the stored key.

Parameters:: key (str) – The key to check
Returns:: True if the provided key matches the stored key, False otherwise
Return type:: bool

classmethod exists(organization_id, collaboration_id)#

Check if a node exists for the given organization and collaboration.

Parameters:

organization_id (int) – The id of the organization
collaboration_id (int) – The id of the collaboration

Returns:

True if a node exists for the given organization and collaboration, False otherwise.

Return type:

bool

classmethod get_by_api_key(api_key)#

Returns Node based on the provided API key.

Parameters:: api_key (str) – The API key of the node to search for
Returns:: Returns the node if a node is associated with api_key, None if no node is associated with api_key.
Return type:: Node | None

classmethod get_online_nodes()#

Return nodes that currently have status ‘online’

Returns:: List of node models that are currently online
Return type:: list[Node]

class Organization(**kwargs)#

Table that describes which organizations are available.

An organization is the legal entity that plays a central role in managing distributed tasks. Each organization contains a public key which other organizations can use to send encrypted messages that only this organization can read.

Variables:

name (str) – Name of the organization
domain (str) – Domain of the organization
address1 (str) – Address of the organization
address2 (str) – Address of the organization
zipcode (str) – Zipcode of the organization
country (str) – Country of the organization
_public_key (bytes) – Public key of the organization
collaborations (list[Collaboration]) – List of collaborations that this organization is part of
results (list[Result]) – List of results that are part of this organization
nodes (list[Node]) – List of nodes that are part of this organization
users (list[User]) – List of users that are part of this organization
created_tasks (list[Task]) – List of tasks that are created by this organization
roles (list[Role]) –

classmethod get_by_name(name)#

Returns the organization with the given name.

Parameters:: name (str) – Name of the organization
Returns:: Organization with the given name if it exists, otherwise None
Return type:: Organization | None

get_result_ids()#

Returns a list of result ids that are part of this organization.

Returns:: List of result ids
Return type:: list[int]

public_key#

Returns the public key of the organization.

Returns:: Public key of the organization. Empty string if no public key is set.
Return type:: str

class Result(**kwargs)#

Table that describes which results are available. A Result is the description of a Task as executed by a Node.

The result (and the input) is encrypted and can be only read by the intended receiver of the message.

Variables:

input (str) – Input data of the task
task_id (int) – Id of the task that was executed
organization_id (int) – Id of the organization that executed the task
result (str) – Result of the task
assigned_at (datetime) – Time when the task was assigned to the node
started_at (datetime) – Time when the task was started
finished_at (datetime) – Time when the task was finished
status (str) – Status of the task
log (str) – Log of the task
task (Task) – Task that was executed
organization (Organization) – Organization that executed the task
ports (list[AlgorithmPort]) – List of ports that are part of this result

complete#

Returns whether the algorithm run has completed or not.

Returns:: True if the algorithm run has completed, False otherwise.
Return type:: bool

property node: Node#

Returns the node that is associated with this result.

Returns:: The node that is associated with this result.
Return type:: model.node.Node

class Role(**kwargs)#

Collection of Rules

Variables:

name (str) – Name of the role
description (str) – Description of the role
organization_id (int) – Id of the organization this role belongs to
rules (List[Rule]) – List of rules that belong to this role
organization (Organization) – Organization this role belongs to
users (List[User]) – List of users that belong to this role

classmethod get_by_name(name)#

Get a role by its name.

Parameters:: name (str) – Name of the role
Returns:: Role with the given name or None if no role with the given name exists
Return type:: Role | None

class Rule(**kwargs)#

Rules to determine permissions in an API endpoint.

A rule gives access to a single type of action with a given operation, scope and resource on which it acts. Note that rules are defined on startup of the server, based on permissions defined in the endpoints. You cannot edit the rules in the database.

Variables:

name (str) – Name of the rule
operation (Operation) – Operation of the rule
scope (Scope) – Scope of the rule
description (str) – Description of the rule
roles (list[Role]) – Roles that have this rule
users (list[User]) – Users that have this rule

classmethod get_by_(name, scope, operation)#

Get a rule by its name, scope and operation.

Parameters:

name (str) – Name of the resource on which the rule acts, e.g. ‘node’
scope (str) – Scope of the rule, e.g. ‘organization’
operation (str) – Operation of the rule, e.g. ‘view’

Returns:

Rule with the given name, scope and operation or None if no rule with the given name, scope and operation exists

Return type:

Rule | None

class Task(**kwargs)#

Table that describes all tasks.

A task can contain multiple Results for multiple organizations. The input of the task is different for each organization (due to the encryption). Therefore the input for the task is encrypted for each organization separately. The task originates from an organization to which the results need to be encrypted, therefore the originating organization is also logged

Variables:

name (str) – Name of the task
description (str) – Description of the task
image (str) – Name of the docker image that needs to be executed
collaboration_id (int) – Id of the collaboration that this task belongs to
run_id (int) – Run id of the task
parent_id (int) – Id of the parent task (if any)
database (str) – Name of the database that needs to be used for this task
initiator_id (int) – Id of the organization that created this task
init_user_id (int) – Id of the user that created this task
collaboration (Collaboration) – Collaboration that this task belongs to
parent (Task) – Parent task (if any)
results (list[Result]) – List of results that are part of this task
initiator (Organization) – Organization that created this task
init_user (User) – User that created this task

classmethod next_run_id()#

Get the next available run id for a new task.

Returns:: Next available run id
Return type:: int

results_for_node(node)#

Get all results for a given node.

Parameters:: node (model.node.Node) – Node for which to get the results
Returns:: List of results for the given node
Return type:: List[model.result.Result]

class User(**kwargs)#

Bases: Authenticatable

Table to keep track of Users (persons) that can access the system.

Users always belong to an organization and can have certain rights within an organization.

Variables:

username (str) – Username of the user
password (str) – Password of the user
firstname (str) – First name of the user
lastname (str) – Last name of the user
email (str) – Email address of the user
organization_id (int) – Foreign key to the organization to which the user belongs
failed_login_attempts (int) – Number of failed login attempts
last_login_attempt (datetime.datetime) – Date and time of the last login attempt
otp_secret (str) – Secret key for one time passwords
organization (Organization) – Organization to which the user belongs
roles (list[Role]) – Roles that the user has
rules (list[Rule]) – Rules that the user has
created_tasks (list[Task]) – Tasks that the user has created

can(resource, scope, operation)#

Check if user is allowed to execute a certain action

Parameters:

resource (str) – The resource type on which the action is to be performed
scope (Scope) – The scope within which the user wants to perform an action
operation (Operation) – The operation a user wants to execute

Returns:

Whether or not user is allowed to execute the requested operation on the resource

Return type:

bool

check_password(pw)#

Check if the password is correct

Parameters:: pw (str) – Password to check
Returns:: Whether or not the password is correct
Return type:: bool

classmethod exists(field, value)#

Checks if user with certain key-value exists

Parameters:

field (str) – Name of the attribute to check
value (str) – Value of the attribute to check

Returns:

Whether or not user with given key-value exists

Return type:

bool

classmethod get_by_email(email)#

Get a user by their email

Parameters:: email (str) – Email of the user
Returns:: User with the given email
Return type:: User
Raises:: NoResultFound – If no user with the given email exists

classmethod get_by_username(username)#

Get a user by their username

Parameters:: username (str) – Username of the user
Returns:: User with the given username
Return type:: User
Raises:: NoResultFound – If no user with the given username exists

is_blocked(max_failed_attempts, inactivation_in_minutes)#

Check if user can login or if they are temporarily blocked because they entered a wrong password too often

Parameters:

max_failed_attempts (int) – Maximum number of attempts to login before temporary deactivation
inactivation_minutes (int) – How many minutes an account is deactivated

Return type:

Tuple[bool, Optional[str]]

Returns:

bool – Whether or not user is blocked temporarily
str | None – Message if user is blocked, else None

set_password(pw)#

Set the password of the current user. This function doesn’t save the new password to the database

Parameters:: pw (str) – The new password
Returns:: If the new password fails to pass the checks, a message is returned. Else, none is returned
Return type:: str | None

classmethod username_exists(username)#

Checks if user with certain username exists

Parameters:: username (str) – Username to check
Returns:: Whether or not user with given username exists
Return type:: bool

Database models that link resources together#

The Member table is used to link organizations and collaborations together. Each line in the table represents that a certain organization is member of a certain collaboration by storing the ids of the organization and collaboration.

Member#: alias of Table(‘Member’, MetaData(), Column(‘organization_id’, Integer(), ForeignKey(‘organization.id’), table=<Member>), Column(‘collaboration_id’, Integer(), ForeignKey(‘collaboration.id’), table=<Member>), schema=None)

The Permission table defines which roles have been assigned to which users. It can contain multiple entries for the same user if they have been assigned multiple roles.

The UserPermission table defines which extra rules have been assigned to which users. Apart from roles, users may be assigned extra permissions that allow them to execute one specific action. This table is used to store those, and may contain multiple entries for the same user.

Permission#: alias of Table(‘Permission’, MetaData(), Column(‘role_id’, Integer(), ForeignKey(‘role.id’), table=<Permission>), Column(‘user_id’, Integer(), ForeignKey(‘user.id’), table=<Permission>), schema=None)

UserPermission#: alias of Table(‘UserPermission’, MetaData(), Column(‘rule_id’, Integer(), ForeignKey(‘rule.id’), table=<UserPermission>), Column(‘user_id’, Integer(), ForeignKey(‘user.id’), table=<UserPermission>), schema=None)

The role_rule_assocation table defines which rules have been assigned to which roles. Each line contains a rule_id that is a member of a certain role_id. Each role will usually have multiple rules assigned to it.

role_rule_association#: alias of Table(‘role_rule_association’, MetaData(), Column(‘role_id’, Integer(), ForeignKey(‘role.id’), table=<role_rule_association>), Column(‘rule_id’, Integer(), ForeignKey(‘rule.id’), table=<role_rule_association>), schema=None)

5.4.7. Mail service#

class MailService(app, mail)#

Send emails from the service email account

Parameters:

app (flask.Flask) – The vantage6 flask application
mail (flask_mail.Mail) – An instance of the Flask mail class

send_email(subject, sender, recipients, text_body, html_body)#

Send an email.

This is used for service emails, e.g. to help users reset their password.

Parameters:

subject (str) – Subject of the email
sender (str) – Email address of the sender
recipients (List[str]) – List of email addresses of recipients
text_body (str) – Email body in plain text
html_body (str) – Email body in HTML

Return type:

None

5.4.8. Default roles#

get_default_roles(db)#

Get a list containing the default roles and their rules, so that they may be created in the database

Parameters:

db – The vantage6.server.db module

Returns:

A list with dictionaries that each describe one of the roles. Each role dictionary contains the following:

name: str: Name of the role
description: str: Description of the role
rules: List[int]: A list of rule id’s that the role contains

Return type:

List[Dict]