Release notes¶
4.8.1¶
22 November 2024
Security
Update
cryptography
andwerkzeug
dependencies (PR#1594).
4.8.0¶
12 November 2024
Feature
Algorithm store arguments can now be indicated as optional and can have default values (Issue#1219, PR#1532).
In the algorithm review process in the algorithm store, several emails are now sent to alert users that they e.g. have to review an algorithm or that their algorithm has been reviewed (Issue#1364, PR#1569).
Added options to set UI image
--ui-image
inv6 dev create-demo-network
(Issue#1518, PR#1525).Directories can be mounted as databases at the nodes (Issue#1172, PR#1540).
Added retry mechanism for the server database connection on initial connect (Issue#1479, PR#1526).
In case a study is used for an analysis, it is shown in the result page (Issue#1489, PR#1561).
Added code of conduct (Issue#1459, PR#1480).
Change
Removed some unused configuration files (Issue#1502, PR#1554).
The API path of the algorithm store is now settable (Issue#1519, PR#1524).
Tasks that failed can now get a more specific status, such as
killed by user
ornot allowed to start
. They only get a more specific status if all failed runs in the task failed for the same reason (Issue#1541, PR#1570).
Bugfix
Fixed a buggy search filter on the task overview page in the UI (Issue#1361, PR#1561)
When a node has never been online the status is set to
ofline
instead ofNone
(Issue#1461, PR#1552).The node support containers are now started with the
restart-unless-stopped
restart policy to prevent them from restarting when the node is shut down in a controlled manner (Issue#1111, PR#1553).Fixed a bug where the
server_url
could not be overruled by the Python client or UI. This is used to identify the originating vantage6 server in the algorithm store (Issue#1454, PR#1528).All fields in client
.update()
calls were required, even if they were not changed. This is now no longer needed (Issue#1515, PR#1555).Fixed a bug which caused the visualization selection in the UI result page to reset when the computation is complete (Issue#1511, PR#1523).
UI permission fix: users with collaboration edit scope are allowed to reset API keys of nodes from all organizations within collaboration the users organization participates in (Issue#1530, PR#1529).
A warning was logged if nodes whitelisted port 443 because the check to recognize this port as HTTPS port was broken (Issue#1563, PR#1574).
4.7.1¶
1 October 2024
Change
Bugfix
Improve refreshing task status in UI by fixing permission issue and awaiting task collection from the API (Issue#1478, PR#1490).
Prevent corrupt node configuration when specifying
allowed_algorithms
inv6 node new
(Issue#1467, PR#1468).Prevent error in
v6 node new
when server URL contains trailing slashes (Issue#1457, PR#1468).Fix filtering node list when getting nodes for a particular study (Issue#1486, PR#1507).
Only require username or email address in resetting password in Python client (Issue#1503, PR#1505).
Delete study when collaboration is deleted (Issue#1465, PR#1487).
Prevent double policies in the database when multiple algorithm store instances are restarted simultaneously (PR#1506).
Minor fixes to running tasks in UI: fix repeat task with boolean values, prevent submitting same task multiple times, improve showing log file, etc (PR#1490).
Added missing field and filter options to docstrings in the Python client (PR#1492).
Prevent error if server is whitelisted at the store for the second time (PR#1491).
4.7.0¶
20 August 2024
Feature
Added option to delete organizations (Issue#241, Issue#1120, PR#1417).
Add algorithm store and UI to the
v6 dev
network (Issue#1078, PR#1399).Option to provide hashed password for root user on first server startup, to use instead of the default password (Issue#1374, PR#1375).
Improved data included in
v6 dev
network so that more tasks can be run on them (PR#1423).Added dev policies for algorithm store to enable an algorithm developer to review their own algorithm and to disable the review process altogether (Issue#1413, PR#1414).
Added option to delete user to client (PR#1433).
Improved algorithm store documentation (Issue#1396, PR#1425).
Change
Apply node_extra_hosts confgiruation also to VPN containers (Issue#1355, PR#1360).
Change default port for server from 5000 to 7600 to prevent conflicts with other services on Mac (Issue#1428, PR#1429).
No longer require user to run client.setup_encryption(None) if their collaboration does not use encryption (Issue#1302, PR#1401).
Improve text in UI task status page (Issue#1221, Issue#1416, PR#1419).
Improve log messages when node is started to let the user know how to view if their node is successfully started (Issue#1173).
Loosened required version of te
requests
library in the common package (Issue#1347, PR#1405).Refactor code that was duplicated between algorithm store and vantage6 server (Issue#1088, PR#1415).
Upgrade UI from Angular 17 to Angular 18 (Issue#1347, PR#1412).
No longer print RabbitMQ username/password in server logs (PR#1434).
Bugfix
No longer require mounting the Docker socket in the algorithm store - which is not always possible. Instead, use the OCI spec to get the image digest (PR#1431).
Fix for when v6 server remove crashed because no log file existed - for servers that had never been started when they were removed (PR#1432).
Fix getting the correct algorithm in the UI if multiple stores are linked to a collaboration (Issue#1420, PR#1422).
Fix v6 dev create command on MacOS - servers and stores are now created in the user folders (Issue#1408, PR#1427).
Added missing dependency
pkg_resources
(Issue#1386, PR#1418).More robust implementation of getting the EduVPN v4 token (PR#1438).
Upload of algorithm JSON representation in the UI is more resilient to errors (PR#1440).
4.6.1¶
30 July 2024
Bugfix
Ensure logs will be shown for failed algorithm runs in the UI without the need to refresh the page (PR#1403).
When creating a task in the UI, dropdown option to select multiple was not properly reset when changing selected function from central to partial (PR#1402).
Fix permission check in UI to show button to register missing nodes (Issue#1230, PR#1404).
Pass on
print_log_header
argument when runningv6 node stop
to prevent entire log header to be printed (Issue#1398, PR#1400).Add missing docker dependency to algorithm store (PR#1409).
4.6.0¶
17 July 2024
Feature
Added option
allowed_algorithm_stores
to node configuration. This option allows node administrators to allow all approved algorithms from a specific algorithm store to be run on their node (Issue#1293, PR#1318).Added policy management system to the algorithm store, and implemented a first few policies, e.g. to control who can view and run algorithms (Issue#1026, PR#1299).
Implemented review process in the algorithm store. Algorithms now need to be reviewed by at least one other user before they are published in the store (Issue#981, PR#1358).
Option to visualize line charts in the UI (Issue#1324, PR#1330).
Users and permissions of the algorithm store can now be managed in the UI (Issue#1123, PR#1340).
Support default pandas
DataFrame.to_json()
output to visualize table in UI (PR#1331).Create option for node administrator to enforce that new algorithm image is successfully pulled before running a task with it (Issue#1200, PR#1344).
Show more clearly in the UI when node has last been seen online (Issue#1308, PR#1343).
Option in creating tasks to indicate which algorithm store the algorithm should be obtained from (Issue#1198, PR#1318).
Change
As EduVPN v2 is no longer supported on modern systems, vantage6 now supports EduVPN v3 instead of v2 (Issue#1180, PR#1345).
Improved the server - algorithm store whitelisting process (Issue#1177, Issue#1178, PR#1299).
Removed connectorx dependency due to issues with installing it. Instead, using SQLAlchemy to read SQL queries in the SQL wrapper (PR#1385).
Make it possible to run infrastructure components with local images (Issue#1250, PR#1332).
Make node_extra_hosts also available to VPN client container (Issue#1355, PR#1360).
Documented algorithm store permission system (Issue#1086, PR#1354).
Bugfix
Fix filling in JSON and list parameters when repeating task in the UI (PR#1328).
Added always_connect option to socketIO connection to prevent random BadNameSpace errors (Issue#1333, PR#1334).
Verify that user exists before assigning it permission in the algorithm store (Issue#1092, PR#1299).
4.5.5¶
13 June 2024
Bugfix
Fix faulty environment variable check in the OHDSI database connector (PR#1326)
4.5.4¶
13 June 2024
Change
Changed default role ‘Collaboration admin’ so that they can no longer create new collaborations: this was deemed too powerful for this role (PR#1313).
Bugfix
Prevent SSL errors in communication between server and algorithm store due to wrong order imports with monkey patch (Issue#1311, PR#1320).
Fix forwarding of custom headers to algorithm store when communicating from the vantage6 server (PR#1298).
4.5.3¶
6 June 2024
Bugfix
Included __build__ file of algorithm tools in the PyPI package to prevent error when using the package (PR#1307).
4.5.2¶
3 June 2024
Bugfix
Updated PyPI Docker dependency to 7.1.0 and requests to 2.32.3 to fix issues with
docker.from_env()
with old combination of docker/requests (PR#1306).
4.5.1¶
3 June 2024
Change
Bugfix
4.5.0¶
23 May 2024
Security
Feature
Visualization and management of algorithms in the UI (Issue#1115, PR#1261).
Support encryption and decryption of results and task input in the UI (Issue#1140, PR#1248).
Added client function
client.algorithm.update()
to update store algorithms (Issue#1089, PR#1277).Pass
dbms
environment variable to algorithm for OMOP connections (Issue#1036, PR#1267).Improved error message in client when wrong API path is provided (Issue#1001, PR#1252).
Change
Default logging level changed from
debug
toinfo
(Issue#692, PR#1216).Don’t send column names request for OMOP and other databases (Issue#1117, PR#1272).
Function documentation generated with Sphinx autosummary (PR#1279).
New error classes and extra functionality for getting environment variables in the algorithm tools (PR#1226).
Improved codacy and DOI badges in README (PR#1271).
Unpin uWSGI version which was fixed to mitigate a build issue with a previous latest version of uWSGI (Issue#1208, PR#1217).
Bugfix
Ensure button to register missing nodes does not show up when all nodes have been registered (Issue#1229, PR#1225).
Prevent returning wrong status code by proxy server when creating subtasks (Issue#1241, PR#1268).
In UI, when resetting password, fix check that the users enters the same new password twice (Issue#1228, PR#1256).
In UI, fix console errors when repeating a task (Issue#1125, PR#1261).
Fix error with undefined variable in deleting docker volumes (Issue#1263, PR#1264).
Fixed an error in the
MockClient
that modified local mock data is shared with subsequent calls (PR#1284).
4.4.1¶
8 May 2024
Security
Updated dependencies Werkzeug to 3.0.3 and Jinja2 to 3.1.4
Change
When columns cannot be retrieved in the UI when creating a task, give the user the option to fill in column names manually (PR#1212).
Bugfix
Updated PyYAML dependency to 6.0.1 to allow building on Python 3.12 (PR#1233).
4.4.0¶
15 April 2024
Feature
Added visualization of a results table to the UI. The algorithm store is used to store how the table should be visualized. (Issue#1057, PR#1195).
Support for more types of algorithm arguments via the UI: lists of strings, ints, floats and columns, and booleans (Issue#1119, PR#1190).
Added configuration option to link algorithm stores to a server via the server configuration (PR#1156).
Added a bunch of custom exceptions for algorithms to the algorithm tools (Issue#1185, PR#1205).
Decoding the environment variables automatically in the algorithm wrapper, to prevent that a user has to decode them manually (Issue#1056, PR#1197).
Add option to delete roles in the UI (Issue#1113, PR#1199).
Add option to register a node in the UI after creating/editing the collaboration (Issue#1122, PR#1202).
Change
Updated idna dependency
Bugfix
Do not mark algorithm runs as killed if they were completed before the user killed the task to which the runs belong (Issue#1045, PR#1204).
Fix UI code in a few places where pagination was not implemented properly (Issue#1126, PR#1203).
4.3.4¶
09 April 2024
Security
Updated express dependency in UI to 4.19.2
Feature
Added option to add hostname mappings in the node configuration (Issue#1094, PR#1167).
Change
Always pull new Docker images instead of checking timestamps and only pulling image if the remote image is newer (Issue#1188, Issue#1105, PR#1169).
Changed behaviour of
v6 algorithm update
to skip previously-answered questions by default, and added flag that allows changing them. Also added flag to allow using a Python script in the updated copier template (PR#1176).
Bugfix
4.3.3¶
25 March 2024
Change
Bugfix
Fix pulling algorithms from registries that require authentication (Issue#1168, PR#1169).
Fix bug in showing create task button in UI (PR#1165).
Could not view studies with collaboration scope permissions (Issue#1154, PR#1157).
Fix bug when viewing algorithm stores with organization scope permissions (PR#1159).
Detect whitelisted server in algorithm store if port
443
or80
at the end of the URL is the only difference with the whitelisted URL (Issue#1155, PR#1162).Better error message in Python client when trying to send requests to algorithm store when it has not yet been set up (Issue#1134, PR#1158).
4.3.2¶
20 March 2024
Change
Integrated user interface in main repository PR#1112).
Bugfix
Allow usernames to contain dots and don’t apply username validation to login endpoints until v5 to allow existing users to login (PR#1148).
4.3.1¶
18 March 2024
Feature
New configuration option to set a server name in the server configuration file, which will be used to identify the server in a two-factor app. (Issue#1016, PR#1075).
Change
Allow user with organization scope permission to view studies to retrieve studies for a particular collaboration, even though they may not be able to view them all (PR#1104).
Add option to set policies on openness of algorithm viewing in algorithm store to configuration wizard (PR#1106).
Improved help text in UI in several places and show the username in the top right (PR#254, PR#257)
Bugfix
Update default roles on server startup if they have changed. This may happen on minor version updates (Issue#1102, PR#1103).
Update selected collaboration in the UI when it is updated in the administration section (PR#253)
Fix showing the create task button if user has no global permissions (PR#259)
Remove wrong message for CORS not functioning properly with default settings (PR#1107).
4.3.0¶
12 March 2024
Security
Implemented configuration option to set CORS origins on the central server. This may be used to further enhance the security profile of your server (advisory, commit).
Prevent username enumeration attack on endpoints where password and 2FA are reset (advisory, commit).
Added HTTP security headers on the user interface to provide an additional layer of security to help mitigate attacks and vulnerabilites (advisory, commit).
Updated cryptography dependency
Feature
New user interface. The new UI is a complete rewrite of the old UI and is more focused on facilitating the researcher in running tasks and viewing their progress and results (PR#930).
New infrastructure component: the algorithm store. The algorithm store is a place to make algorithms easily findable and easier to run. Algorithm stores can be made available to specific collaborations or to all collaborations in an entire vantage6 server. By doing so, the new UI will automatically pick up these algorithms and guide the user through running analyses with them ( Issue#911, PR#1048 and several other PRs)
Introducing ‘study’ concept. A study is essentially a ‘sub-collaboration’, where a subset of organizations of the collaboration can work together on a specific research question. Tasks and results are then easily grouped together for the study (Issue#812, PR#1069).
Add flag whether role is default or not (Issue#949, PR#1063).
Report username/password combination at the end of the logs when it is created (Issue#830, PR#1041).
Change
Introducing new package
vantage6-backend-common
for code that is shared between the central server and the algorithm store (Issue#979, PR#1037).Show the default values for CLI commands when displaying the help text (Issue#1000, PR#1070).
Setting the allowed algorithms is now part of the questionnaire on node setup (PR#1046).
Usernames are now required to be at least three characters long and contain only roman letters, numbers, and the characters ‘_’ and ‘-’ (PR#1060).
Remove OMOP wrapper since we now have specific connectors to connect to this database type and wrapper was therefore not used (Issue#1002, PR#1067).
v6 node
commands no longer require full path when using the--config
option (Issue#870, PR#1042).Apply black code formatting to the entire repository (Issue#968, PR#1012).
Remove option to update organization or collaboration of an existing node. Preferred workflow in that case is to delete and re-create it. Also add option
clear_ip
to clear the VPN IP address of the node (PR#1053).
Bugfix
Fix VPN network cleanup if
iptables-legacy
is installed, and improve cleanup of the node’s containers, volumes and networks when the node is stopped (Issue#1058, PR#1059).Prevent logger thread to crash on input that it cannot read (Issue#879, PR#1043).
Fixed setting up VPN network on Ubuntu 22.04 (Issue#724, PR#1044).
4.2.3¶
21 February 2024
Security
Feature
Added the option to specify a private key file when using the
v6 test feature-test
command (Issue#1018, PR#1019).
Bugfix
4.2.2¶
26 January 2024
Feature
Change
Bugfix
4.2.1¶
19 January 2024
Bugfix
Add back binary installation of
psycopg2
to support Postgres databases (PR#932).
4.2.0¶
18 January 2024
Security
Remove option to SSH into node and server containers. The configuration was not completely secure (advisory, commit).
Prevent code injection into environment variables (advisory, commit).
Prevent that user can accidentally upload non-encrypted input to the server for an encrypted collaboration. (advisory, commit).
Prevent that usernames are findable in brute force attack due to a difference in response time when they exist versus when they don’t exist (advisory, commit).
Updated dependencies of jinja2, cryptography and Werkzeug. ( PR#984).
Feature
Change
Bugfix
4.1.3¶
19 December 2023
Bugfix
4.1.2¶
14 November 2023
Security
4.1.1¶
1 November 2023
Bugfix
Added OpenPyxl dependency to algorithm tools which is required to read Excel databases (PR#923).
Explicitly define the resource on which sorting is done in the API. This prevents SQL errors when SQLAlchemy tries to sort on a column in a joined table (PR#925).
Fixed retrieving column names for Excel databases (PR#924).
4.1.0¶
19 October 2023
Feature
Renamed CLI commands. The new commands are:
vnode
→v6 node
vserver
→v6 server
vdev
→v6 dev
The old commands will still be available until version 5.0 is released.
Added CLI command
v6 algorithm create
which is a starting point for creating new algorithms (Issue#400, PR#904).Added
@database_connection(type_)
algorithm decorator. This enables algorithm developers to inject a database connection into their algorithm instead of a dataframe. The only type that currently is support isomop
, which injects aOHDSI/DatabaseConnection
object into your algorithm. (PR#902).Added endpoint /column for the UI to get the column names of the database. This is achieved either by sharing column names by the node for file-based databases or by sending a task using the
basics
algorithm. The latter is now an allowed algorithm by default, unless the node is configured to not allow it. ((Issue#778, PR#908).Added
only_siblings
andonly_self
options to theclient.vpn.get_addresses
function. These options allow you to get the VPN addresses of only the siblings or only the node itself, respectively. This is useful for algorithms that need to communicate with other algorithms on the same node or with the node itself. (Issue#729, PR#901).
4.0.3¶
16 October 2023
Bugfix
4.0.2¶
9 October 2023
Bugfix
Fix socket connection from node to server due to faulty callback, which occurred when server was deployed. This bug was introduced in v4.0.1 (PR#892).
4.0.1¶
5 October 2023
Security
Updating dependencies
cryptography
,gevent
, andurllib3
to fix vulnerabilities (PR#889)
Bugfix
Fix node connection issues if server without constant JWT secret key is restarted (Issue#840, PR#866).
Improved algorithm_client decorator with
@wraps
decorator. This fixes an issue with the data decorator in the AlgorithmMockClient (Issue#874, PR#882).Decoding the algorithm results and algorithm input has been made more robust, and input from
vserver import
is now properly encoded (Issue#836, PR#864).Improve error message if user forgot to specify
databases
when creating a task (Issue#854, PR#865).Fix data loading in AlgorithmMockClient (Issue#872, PR#881).
4.0.0¶
20 September 2023
Security
No longer using Python pickles for serialization and deserialization of algorithm results. Using JSON instead ( CVE#CVE-2023-23930, commit).
Not allowing resources to have an integer name ( CVE#CVE-2023-28635, PR#744).
Users allowed to view collaborations but not allowed to view tasks may be able to view them via
/api/collaboration/<id>/task
( CVE#CVE-2023-41882, PR#741).Users allowed to view tasks but not results may be able to view them via
/api/task?include=results
( CVE#CVE-2023-41882, PR#711).Deleting all linked tasks when a collaboration is deleted ( CVE#CVE-2023-41881, PR#748).
Feature
A complete permission scope has been added at the collaboration level, allowing projects to assign one user to manage everything within that collaboration level without requiring global access (Issue#245, PR#711).
Added decorators
@algorithm_client
and@data()
to make the signatures and names of algorithm functions more flexible and also to allow for multiple databases (Issue#440, PR#652).Allow a single algorithm function to make use of multiple databases (Issue#804, PR#652, PR#807).
Enforce pagination in the API to improve performance, and add a sort parameter for GET requests which yield multiple resources (Issue#392, PR#611).
Share a node’s database labels and types with the central server, so that the server can validate that these match between nodes and offer them as suggestions to the user when creating tasks (Issue#750, PR#751).
vnode new
now automatically retrieves information on e.g. whether the collaboration is encrypted, so that the user doesn’t have to specify this information themselves (Issue#434, PR#739).Allow only unique names for organizations, collaborations, and nodes (Issue#242, PR#515).
New function
client.task.wait_for_completion()
for the AlgorithmClient to allow waiting for subtasks to complete (Issue#651, PR#727).Improved validation of the input for all POST and PATCH requests using marshmallow schemas (Issue#76, PR#744).
Added option
user_created
to filter tasks that have been directly created by a user and are thus not subtasks (Issue#583, PR#599).Users can now assign rules to other users that they don’t have themselves if they do have higher permisions on the same resource (Issue#443, PR#781).
Change
Changed the API response structure: no longer returning as many linked resources for performance reasons (Issue#49, PR#709)
The
result
endpoint has been renamed torun
as this was a misnomer that concerns algorithm runs (Issue#436, PR#527), PR#620).Split the vantage6-client package: the Python user client is kept in this package, and a new vantage6-algorithm-tools PyPI package is created for the tools that help algorithm developers. These tools were part of the client package, but moving them reduces the sizes of both packages (Issue#662, PR#763)
Removed environments test, dev, prod, acc and application from vantage6 servers and nodes as these were used little (Issue#260, PR#643)
Harmonized the interfaces between the AlgorithmClient and the MockClient (Issue#669, PR#722)
When users request resources where they are not allowed to see everything, they now get an unauthorized error instead of an incomplete or empty response (Issue#635, PR#711).
Node checks the server’s version and by default, it pulls a matching image instead of the latest image of it’s major version (Issue#700, PR#706).
vserver-local
commands have been removed if they were not used within the docker images or the CLI (Issue#663, PR#728).The way in which RabbitMQ is started locally has been changed to make it easier to run RabbitMQ locally. Now, a user indicates with a configuration flag whether they expect RabbitMQ to be started locally (Issue#282, PR#795).
The place in which server configuration files were stored on Linux has been changed fro
/etc/xdg
to/etc/vantage6/
(Issue#269, PR#789).Backwards compatibility code that was present to make different v3.x versions compatible has been removed. Additionally, small improvements have been made that were not possible to do without breaking compatibility (Issue#454, PR#740, PR#758).
Bugfix
3.11.1¶
11 September 2023
Bugfix
3.11.0¶
21 August 2023
Feature
A suite of vdev commands has been added to the CLI. These commands allow you to easily create a development environment for vantage6. The commands allow you to easily create a server configuration, add organizations and collaborations to it, and create the appropriate node configurations. Also, you can easily start, stop, and remove the network. (Issue#625, PR#624).
User Interface can now be started from the CLI with vserver start –with-ui (Issue#730, PR#735).
Added created_at and finished_at timestamps to tasks (Issue#621, PR#715).
Change
Help text for the CLI has been updated and the formatting has been improved (Issue#745, PR#791).
With vnode list, the terms online and offline have been replaced by running and not running. This is more accurate, since a node may be unable to authenticate and thus be offline, but still be running. (Issue#733, PR#734).
Some legacy code that no longer fulfilled a function has been removed from the endpoint to create tasks (Issue#742, PR#747).
Bugfix
In the docs, the example file to import server resources with vserver import was accidentally empty; now it contains example data. (PR#792).
3.10.4¶
27 June 2023
Change
Extended the AlgorithmMockClient so that algorithm developers may pass it organization id’s and node id’s (PR#737).
Bugfix
3.10.3¶
20 June 2023
Bugfix
Fixed bug in copying the MockClient itself to pass it on to a child task ( PR#723).
Note
Release 3.10.2 failed to be published to PyPI due to a gateway error, so that version was skipped.
3.10.1¶
19 June 2023
Bugfix
3.10.0¶
19 June 2023
Feature
There is a new implementation of a mock client, the
MockAlgorithmClient
. This client is an improved version of the oldClientMockProtocol
. The new mock client now contains all the same functions as the regular client with the same signatures, and it returns the same data fields as those functions. Also, you may submit all supported data formats instead of just CSV files, and you may also submit pandas Dataframes directly (Issue#683, PR#702).
Change
Bugfix
3.9.0¶
25 May 2023
Feature
Data sources may now be whitelisted by IP address, so that an algorithm may access those IP addresses to obtain data. This is achieved via a Squid proxy server (Issue#162, PR#626).
There is a new configuration option to let algorithms access gpu’s (Issue#597, PR#623).
Added option to get VPN IP addresses and ports of just the children or just the parent of an algorithm that is running. These options may be used to simplify VPN communication between algorithms running on different nodes. In the AlgorithmClient, the functions
client.vpn.get_child_addresses()
andclient.vpn.get_parent_address()
have been added (PR#610).New option to print the full stack trace of algorithm errors. Note that this option may leak sensitive information if used carelessly. The option may be activated by setting
log_traceback=True
in the algorithm wrapper (Issue#675, PR#680).Configuration options to control the log levels of individual dependencies. This allows easier debugging when a certain dependency is causing issues (Issue#641, PR#642).
Change
Better error message for
vnode attach
when no nodes are running (Issue#606, PR#607).The number of characters of the task input printed to the logs is now limited to prevent flooding the logs with very long input (Issue#549, PR#550).
Node proxy logs are now written to a separate log file. This makes the main node log more readable (Issue#546, PR#619).
Update code in which the version is updated (PR#586).
Finished standardizing docstrings - note that this was already partially done in earlier releases (Issue#255).
Cleanup and moving of unused code and duplicate code (PR#571).
It is now supported to run the release pipeline from
release/v<x.y.z>
branches (Issue#467, PR#488).Replaced deprecated
set-output
method in Github actions release pipeline (Issue#474, PR#601).
Bugfix
Fixed checking for newer images (node, server, and algorithms). Previously, the dates used were not sufficient to check if an image was newer. Now, we are also checking the image digest (Issue#507, PR#602).
Users are prevented from posting socket events that are meant for nodes - note that nothing harmful could be done but it should not be possible nevertheless (Issue#615, PR#616).
Fixed bug with detecting if database was a file as ‘/mnt/’ was not properly prepended to the file path (PR#691).
3.8.8¶
11 May 2023
Bugfix
Fixed a bug that prevented the node from shutting down properly (Issue#649, PR#677)
Fixed a bug where the node did not await the VPN client to be ready (Issue#656, PR#676)
Fixed database label logging (PR#664)
Fixed a bug were VPN messages to the originating node where not always sent/received (Issue#671, PR#673)
Fixed a bug where an exceptions is raised when the websocket connection was lost and a ping was attempted to be send (Issue#672, PR#674)
Fixed a formatting in CLI print statement (PR#661)
Fixed bug where ‘/mnt/’ was erroneously prepended to non-file based databases (PR#658)
Fix in
autowrapper
for algorithms with CSV input (PR#655)Fixed a bug in syncing tasks from the server to the node, when the node lost socket connection and then reconnected (Issue#654, PR#657)
Fix construction of database URI in
vserver files
(Issue#650, PR#659)
3.8.7¶
10 May 2023
Bugfix
Socket did connect before Docker was initialized, resulting in an exception at startup (PR#644)
3.8.6¶
9 May 2023
Bugfix
3.8.3 - 3.8.5¶
25 April 2023 - 2 May 2023
Bugfix
3.8.2¶
22 march 2023
Feature
Location of the server configuration file in server shell script can now be specified as an environment variable (PR#604)
Change
Changed ping/pong mechanism over socket connection between server and nodes, as it did not function properly in combination with RabbitMQ. Now, the node pushes a ping and the server periodically checks if the node is still alive (PR#593)
Bugfix
3.8.1¶
8 march 2023
Bugfix
In 3.8.0, starting RabbitMQ for horizontal scaling caused a server crash due to a missing
kombu
dependency. This dependency was wrongly removed in updating all dependencies for python 3.10 ( PR#585).
3.8.0¶
8 march 2023
Security
Refresh tokens are no longer indefinitely valid ( CVE#CVE-2023-23929, commit).
It was possible to obtain usernames by brute forcing the login since v3.3.0. This was due to a change where users got to see a message their account was blocked after N failed login attempts. Now, users get an email instead if their account is blocked ( CVE#CVE-2022-39228, commit ).
Assigning existing users to a different organizations was possible. This may lead to unintended access: if a user from organization A is accidentally assigned to organization B, they will retain their permissions and therefore might be able to access resources they should not be allowed to access (CVE#CVE-2023-22738, commit).
Feature
Python version upgrade to 3.10 and many dependencies are upgraded ( PR#513, Issue#251).
Added
AlgorithmClient
which will replaceContainerClient
in v4.0. For now, the newAlgorithmClient
can be used by specifyinguse_new_client=True
in the algorithm wrapper ( PR#510, Issue#493).It is now possible to request some of the node configuration settings, e.g. which algorithms they allow to be run ( PR#523, Issue#12).
Added
auto_wrapper
which detects the data source types and reads the data accordingly. This removes the need to rebuild every algorithm for every data source type ( PR#555, Issue#553).New endpoint added
/vpn/algorithm/addresses
for algorithms to obtain addresses for containers that are part of the same computation task ( PR#501, Issue#499).Added the option to allow only allow certain organization and/or users to run tasks on your node. This can be done by using the
policies
configuration option. Note that theallowed_images
option is now nested under thepolicies
option ( Issue#335, PR#556)
Change
Bugfix
3.7.3¶
22 february 2023
Bugfix
3.7.2¶
20 february 2023
Bugfix
3.7.1¶
16 february 2023
Change
Some changes to the release pipeline.
Bugfix
3.7.0¶
25 january 2023
Feature
SSH tunnels are available on the node. This allows nodes to connect to other machines over SSH, thereby greatly expanding the options to connect databases and other services to the node, which before could only be made available to the algorithms if they were running on the same machine as the node (PR#461, Issue#162).
For two-factor authentication, the information given to the authenticator app has been updated to include a clearer description of the server and username (PR#483, Issue#405).
Added the option to run an algorithm without passing data to it using the CSV wrapper (PR#465)
In the UI, when users are about to create a task, they will now be shown which nodes relevant to the task are offline (PR#97, Issue#96).
Change
The
docker
dependency is updated, so thatdocker.pull()
now pulls the default tag if no tag is specified, instead of all tags (PR#481, Issue#473).If a node cannot authenticate to the server because the server cannot be found, the user now gets a clearer error message(PR#480, Issue#460).
The default role ‘Organization admin’ has been updated: it now allows to create nodes for their own organization (PR#489).
The release pipeline has been updated to 1) release to PyPi as last step ( since that is irreversible), 2) create release branches, 3) improve the check on the version tag, and 4) update some soon-to-be-deprecated commands (PR#488.
Not all nodes are alerted any more when a node comes online (PR#490).
Added instructions to the UI on how to report bugs (PR#100, Issue#57).
Bugfix
3.6.1¶
12 january 2023
Feature
Algorithm containers can be killed from the client. This can be done for a specific task or it possible to kill all tasks running at a specific node (PR#417, Issue#167).
Added a
status
field for an algorithm, that tracks if an algorithm has yet to start, is started, has finished, or has failed. In the latter case, it also indicates how/when the algorithm failed (PR#417).The UI has been connected to the socket, and gives messages about node and task status changes (UI PR#84, UI Issue #73). There are also new permissions for socket events on the server to authorize users to see events from their (or all) collaborations (PR#417).
It is now possible to create tasks in the UI (UI version >3.6.0). Note that all tasks are then JSON serialized and you will not be able to run tasks in an encrypted collaboration (as that would require uploading a private key to a browser) (PR#90).
Warning
If you want to run the UI Docker image, note that from this version onwards, you have to define the
SERVER_URL
andAPI_PATH
environment variables (compared to just aAPI_URL
before).There is a new multi-database wrapper that will forward a dictionary of all node databases and their paths to the algorithm. This allows you to use multiple databases in a single algorithm easily. (PR#424, Issue #398).
New rules are now assigned automatically to the default root role. This ensures that rules that are added in a new version are assigned to system administrators, instead of them having to change the database (PR#456, Issue #442).
There is a new command
vnode set-api-key
that facilitates putting your API key into the node configuration file (PR#428, Issue #259).Logging in the Python client has been improved: instead of all or nothing, log level is now settable to one of debug, info, warn, error, critical (PR#453, Issue #340).
When there is an error in the VPN server configuration, the user receives clearer error messages than before (PR#444, Issue #278).
Change
The node status (online/offline) is now checked periodically over the socket connection via a ping/pong construction. This is an improvement over the older version where a node’s status was changed only when it connected or disconnected (PR#450, Issue #40).
Warning
If a server upgrades to 3.6.1, the nodes should also be upgraded. Otherwise, the node status will be incorrect and the logs will show errors periodically with each attempted ping/pong.
It is no longer possible for any user to change the username of another user, as this would be confusing for that user when logging in (PR#433, Issue #396).
The server has shorter log messages when someone calls a non-existing route. The resulting 404 exception is no longer logged (PR#452, Issue #393).
Removed old, unused scripts to start a node (PR#464).
Bugfix
Node was unable to pull images from Docker Hub; this has been corrected. (PR#432, Issue#422).
File-based database extensions were always converted to
.csv
when they were mounted to a node. Now, files keep their original file extensions (PR#426, Issue #397).When a node configuration defined a wrong VPN subnet, the VPN connection didn’t work but this was not detected until VPN was used. Now, the user is alerted immediately and VPN is turned off (PR#444).
If a user tries to write a node or server config file to a non-existing directory, they are now getting a clear error message instead of an incorrect one (PR#455, Issue #1)
There was a circular import in the infrastructure code, which has now been resolved (PR#451, Issue #53).
In PATCH
/user
, it was not possible to set some fields (e.g.firstname
) to an empty string if there was a value before. (PR#439, Issue #334).
Note
Release 3.6.0 was skipped due to an issue in the release process.
3.5.2¶
30 november 2022
Bugfix
3.5.1¶
30 november 2022
Bugfix
3.5.0¶
30 november 2022
Warning
When upgrading to 3.5.0, you might need to add the otp_secret column to the user table manually in the database. This may be avoided by upgrading to 3.5.2.
Feature
Multi-factor authentication via TOTP has been added. Admins can enforce that all users enable MFA (PR#376, Issue#355).
You can now request all tasks assigned by a given user (PR#326, Issue#43).
The server support email is now settable in the configuration file, used to be fixed at
support@vantage6.ai
(PR#330, Issue#319).When pickles are used, more task info is shown in the node logs (PR#366, Issue#171).
Change
3.4.2¶
3 november 2022
Bugfix
3.4.0 & 3.4.1¶
25 oktober 2022
Feature
Add columns to the SQL database on startup (PR#365, ISSUE#364). This simpifies the upgrading proces when a new column is added in the new release, as you do no longer need to manually add columns. When downgrading the columns will not be deleted.
Docker wrapper for Parquet files (PR#361, ISSUE#337). Parquet provides a way to store tabular data with the datatypes included which is an advantage over CSV.
When the node starts, or when the client is verbose initialized a banner to cite the vantage6 project is added (PR#359, ISSUE#356).
In the client a waiting for results method is added (PR#325, ISSUE#8). Which allows you to automatically poll for results by using
client.wait_for_results(...)
, for more info seehelp(client.wait_for_results)
.Added option to filter GET
/role
by user id in the Python client (PR#328, ISSUE#213). E.g.:client.role.list(user=...).
In release process, build and release images for both ARM and x86 architecture.
Change
Bugfix
Note
3.4.1 is a rebuild from 3.4.0 in which the all dependencies are fixed, as the build led to a broken server image.
3.3.7¶
Bugfix
The function
client.util.change_my_password()
was updated (Issue #333)
3.3.6¶
Bugfix
Temporary fix for a bug that prevents the master container from creating tasks in an encrypted collaboration. This temporary fix disables the parallel encryption module in the local proxy. This functionality will be restored in a future release.
Note
This version is also the first version where the User Interface is available in the right version. From this point onwards, the user interface changes will also be part of the release notes.
3.3.5¶
Feature
The release pipeline has been expanded to automatically push new Docker images of node/server to the harbor2 service.
Bugfix
The VPN IP address for a node was not saved by the server using the PATCH
/node
endpoint, while this functionality is required to use the VPN
Note
Note that 3.3.4 was only released on PyPi and that version is identical to 3.3.5. That version was otherwise skipped due to a temporary mistake in the release pipeline.
3.3.3¶
Bugfix
3.3.2¶
Bugfix
vpn_client_image
andnetwork_config_image
are settable through the node configuration file. (PR#301, Issue#294)The option
--all
fromvnode stop
did not stop the node gracefully. This has been fixed. It is possible to force the nodes to quit by using the--force
flag. (PR#300, Issue#298)Nodes using a slow internet connection (high ping) had issues with connecting to the websocket channel. (PR#299, Issue#297)
3.3.1¶
Bugfix
Fixed faulty error status codes from the
/collaboration
endpoint (PR#287).Default roles are always returned from the
/role
endpoint. This fixes the error when a user was assigned a default role but could not reach anything (as it could not view its own role) (PR#286).Performance upgrade in the
/organization
endpoint. This caused long delays when retrieving organization information when the organization has many tasks (PR#288).Organization admins are no longer allowed to create and delete nodes as these should be managed at collaboration level. Therefore, the collaboration admin rules have been extended to include create and delete nodes rules (PR#289).
Fixed some issues that made
3.3.0
incompatible with3.3.1
(Issue#285).
3.3.0¶
Feature
Login requirements have been updated. Passwords are now required to have sufficient complexity (8+ characters, and at least 1 uppercase, 1 lowercase, 1 digit, 1 special character). Also, after 5 failed login attempts, a user account is blocked for 15 minutes (these defaults can be changed in a server config file).
Added endpoint
/password/change
to allow users to change their password using their current password as authentication. It is no longer possible to change passwords viaclient.user.update()
or via a PATCH/user/{id}
request.Added the default roles ‘viewer’, ‘researcher’, ‘organization admin’ and ‘collaboration admin’ to newly created servers. These roles may be assigned to users of any organization, and should help users with proper permission assignment.
Added option to filter get all roles for a specific user id in the GET
/role
endpoint.RabbitMQ has support for multiple servers when using
vserver start
. It already had support for multiple servers when deploying via a Docker compose file.When exiting server logs or node logs with Ctrl+C, there is now an additional message alerting the user that the server/node is still running in the background and how they may stop them.
Change
Node proxy server has been updated
Updated PyJWT and related dependencies for improved JWT security.
When nodes are trying to use a wrong API key to authenticate, they now receive a clear message in the node logs and the node exits immediately.
When using
vserver import
, API keys must now be provided for the nodes you create.Moved all swagger API docs from YAML files into the code. Also, corrected errors in them.
API keys are created with UUID4 instead of UUID1. This prevents that UUIDs created milliseconds apart are not too similar.
Rules for users to edit tasks were never used and have therefore been deleted.
Bugfix
In the Python client,
client.organization.list()
now shows pagination metadata by default, which is consistent all otherlist()
statements.When not providing an API key in
vnode new
, there used to be an unclear error message. Now, we allow specifying an API key later and provide a clearer error message for any other keys with inadequate values.It is now possible to provide a name when creating a name, both via the Python client as via the server.
A GET
/role
request crashed if parameterorganization_id
was defined but notinclude_root
. This has been resolved.Users received an ‘unexpected error’ when performing a GET
/collaboration?organization_id=<id>
request and they didn’t have global collaboration view permission. This was fixed.GET
/role/<id>
didn’t give an error if a role didn’t exist. Now it does.
3.2.0¶
Feature
Horizontal scaling for the vantage6-server instance by adding support for RabbitMQ.
It is now possible to connect other docker containers to the private algorithm network. This enables you to attach services to the algorithm network using the
docker_services
setting.Many additional select and filter options on API endpoints, see swagger docs endpoint (
/apidocs
). The new options have also been added to the Python client.Users are now always able to view their own data
Usernames can be changed though the API
Bugfix
(Confusing) SQL errors are no longer returned from the API.
Clearer error message when an organization has multiple nodes for a single collaboration.
Node no longer tries to connect to the VPN if it has no
vpn_subnet
setting in its configuration file.Fix the VPN configuration file renewal
Superusers are no longer able to post tasks to collaborations its organization does not participate in. Note that superusers were never able to view the results of such tasks.
It is no longer possible to post tasks to organization which do not have a registered node attach to the collaboration.
The
vnode create-private-key
command no longer crashes if the ssh directory does not exist.The client no longer logs the password
The version of the
alpine
docker image (that is used to set up algorithm runs with VPN) was fixed. This prevents that many versions of this image are downloaded by the node.Improved reading of username and password from docker registry, which can be capitalized differently depending on the docker version.
Fix error with multiple-database feature, where default is now used if specific database is not found
3.1.0¶
Feature
Algorithm-to-algorithm communication can now take place over multiple ports, which the algorithm developer can specify in the Dockerfile. Labels can be assigned to each port, facilitating communication over multiple channels.
Multi-database support for nodes. It is now also possible to assign multiple data sources to a single node in Petronas; this was already available in Harukas 2.2.0. The user can request a specific data source by supplying the database argument when creating a task.
The CLI commands
vserver new
andvnode new
have been extended to facilitate configuration of the VPN server.Filter options for the client have been extended.
Roles can no longer be used across organizations (except for roles in the default organization)
Added
vnode remove
command to uninstall a node. The command removes the resources attached to a node installation (configuration files, log files, docker volumes etc).Added option to specify configuration file path when running
vnode create-private-key
.
Bugfix
Fixed swagger docs
Improved error message if docker is not running when a node is started
Improved error message for
vserver version
andvnode version
if no servers or nodes are runningPatching user failed if users had zero roles - this has been fixed.
Creating roles was not possible for a user who had permission to create roles only for their own organization - this has been corrected.
3.0.0¶
Feature
Direct algorithm-to-algorithm communication has been added. Via a VPN connection, algorithms can exchange information with one another.
Pagination is added. Metadata is provided in the headers by default. It is also possible to include them in the output body by supplying an additional parameter
include=metadata
. Parameterspage
andper_page
can be used to paginate. The following endpoints are enabled:
GET
/result
GET
/collaboration
GET
/collaboration/{id}/organization
GET
/collaboration/{id}/node
GET
/collaboration/{id}/task
GET
/organization
GET
/role
GET
/role/{id}/rule
GET
/rule
GET
/task
GET
/task/{id}/result
GET
/node
API keys are encrypted in the database
Users cannot shrink their own permissions by accident
Give node permission to update public key
Dependency updates
Bugfix
Fixed database connection issues
Don’t allow users to be assigned to non-existing organizations by root
Fix node status when node is stopped and immediately started up
Check if node names are allowed docker names
2.3.0 - 2.3.4¶
Feature
Allows for horizontal scaling of the server instance by adding support for RabbitMQ. Note that this has not been released for version 3(!)
Bugfix
Performance improvements on the
/organization
endpoint
2.2.0¶
Feature
Multi-database support for nodes. It is now possible to assign multiple data sources to a single node. The user can request a specific data source by supplying the database argument when creating a task.
The mailserver now supports TLS and SSL options
Bugfix
Nodes are now disconnected more gracefully. This fixes the issue that nodes appear offline while they are in fact online
Fixed a bug that prevented deleting a node from the collaboration
A role is now allowed to have zero rules
Some http error messages have improved
Organization fields can now be set to an empty string
2.1.2 & 2.1.3¶
Bugfix
Changes to the way the application interacts with the database. Solves the issue of unexpected disconnects from the DB and thereby freezing the application.
2.1.1¶
Bugfix
Updating the country field in an organization works again\
The
client.result.list(...)
broke when it was not able to deserialize one of the in- or outputs.
2.1.0¶
Feature
Custom algorithm environment variables can be set using the
algorithm_env
key in the configuration file. See this Github issue.Support for non-file-based databases on the node. See this Github issue.
Added flag
--attach
to thevserver start
andvnode start
command. This directly attaches the log to the console.Auto updating the node and server instance is now limited to the major version. See this Github issue.
e.g. if you’ve installed the Trolltunga version of the CLI you will always get the Trolltunga version of the node and server.
Infrastructure images are now tagged using their version major. (e.g.
trolltunga
orharukas
)It is still possible to use intermediate versions by specifying the
--image
option when starting the node or server. (e.g.vserver start --image harbor.vantage6.ai/infrastructure/server:2.0.0.post1
)
Bugfix
Fixed issue where node crashed if the database did not exist on startup. See this Github issue.
2.0.0.post1¶
Bugfix
Fixed a bug that prevented the usage of secured registry algorithms
2.0.0¶
Feature
Role/rule based access control
Roles consist of a bundle of rules. Rules profided access to certain API endpoints at the server.
By default 3 roles are created: 1) Container, 2) Node, 3) Root. The root role is assigned to the root user on the first run. The root user can assign rules and roles from there.
Major update on the python-client. The client also contains management tools for the server (i.e. to creating users, organizations and managing permissions. The client can be imported from
from vantage6.client import Client
.You can use the agrument
verbose
on the client to output status messages. This is usefull for example when working with Jupyter notebooks.Added CLI
vserver version
,vnode version
,vserver-local version
andvnode-local version
commands to report the version of the node or server they are runningThe logging contains more information about the current setup, and refers to this documentation and our Discourd channel
Bugfix
Issue with the DB connection. Session management is updated. Error still occurs from time to time but can be reset by using the endpoint
/health/fix
. This will be patched in a newer version.
1.2.3¶
Feature
The node is now compatible with the Harbor v2.0 API
1.2.2¶
Bug fixes
Fixed a bug that ignored the
--system
flag fromvnode start
Logging output muted when the
--config
option is used invnode start
Fixed config folder mounting point when the option
--config
option is used invnode start
1.2.1¶
Bug fixes
starting the server for the first time resulted in a crash as the root user was not supplied with an email address.
Algorithm containers could still access the internet through their host. This has been patched.
1.2.0¶
Features
Cross language serialization. Enabling algorithm developers to write algorithms that are not language dependent.
Reset password is added to the API. For this purpose two endpoints have been added:
/recover/lost
andrecover/reset
. The server config file needs to extended to be connected to a mail-server in order to make this work.User table in the database is extended to contain an email address which is mandatory.
Bug fixes
Collaboration name needs to be unique
API consistency and bug fixes:
GET
organization
was missing domain keyPATCH
/organization
could not patch domainGET
/collaboration/{id}/node
has been made consistent with/node
GET
/collaboration/{id}/organization
has been made consistent with/organization
PATCH
/user
root-user was not able to update usersDELETE
/user
root-user was not able to delete usersGET
/task
null values are now consistent:[]
is replaced bynull
POST, PATCH, DELETE
/node
root-user was not able to perform these actionsGET
/node/{id}/task
output is made consistent with the
other
questionairy
dependency is updated to 1.5.2
vantage6-toolkit
repository has been merged with thevantage6-client
as they were very tight coupled.
1.1.0¶
Features
new command
vnode clean
to clean up temporary docker volumes that are no longer usedVersion of the individual packages are printed in the console on startup
Custom task and log directories can be set in the configuration file
Improved CLI messages
Docker images are only pulled if the remote version is newer. This applies both to the node/server image and the algorithm images
Client class names have been simplified (
UserClientProtocol
->Client
)
Bug fixes
Removed defective websocket watchdog. There still might be disconnection issues from time to time.
1.0.0¶
Updated Command Line Interface (CLI)
The commands
vnode list
,vnode start
and the new commandvnode attach
are aimed to work with multiple nodes at a single machine.System and user-directories can be used to store configurations by using the
--user/--system
options. The node stores them by default at user level, and the server at system level.Current status (online/offline) of the nodes can be seen using
vnode list
, which also reports which environments are available per configuration.Developer container has been added which can inject the container with the source.
vnode start --develop [source]
. Note that this Docker image needs to be build in advance from thedevelopment.Dockerfile
and tagdevcon
.
vnode config_file
has been replaced byvnode files
which not only outputs the config file location but also the database and log file location.
New database model
Improved relations between models, and with that, an update of the Python API.
Input for the tasks is now stored in the result table. This was required as the input is encrypted individually for each organization (end-to-end encryption (E2EE) between organizations).
The
Organization
model has been extended with thepublic_key
(String) field. This field contains the public key from each organization, which is used by the E2EE module.The
Collaboration
model has been extended with theencrypted
(Boolean) field which keeps track if all messages (tasks, results) need to be E2EE for this specific collaboration.The
Task
keeps track of the initiator (organization) of the organization. This is required to encrypt the results for the initiator.
End to end encryption
All messages between all organizations are by default be encrypted.
Each node requires the private key of the organization as it needs to be able to decrypt incoming messages. The private key should be specified in the configuration file using the
private_key
label.In case no private key is specified, the node generates a new key an uploads the public key to the server.
If a node starts (using
vnode start
), it always checks if thepublic_key
on the server matches the private key the node is currently using.In case your organization has multiple nodes running they should all point to the same private key.
Users have to encrypt the input and decrypt the output, which can be simplified by using our client
vantage6.client.Client
__ for Python __ orvtg::Client
__ for R.Algorithms are not concerned about encryption as this is handled at node level.
Algorithm container isolation
Containers have no longer an internet connection, but are connected to a private docker network.
Master containers can access the central server through a local proxy server which is both connected to the private docker network as the outside world. This proxy server also takes care of the encryption of the messages from the algorithms for the intended receiving organization.
In case a single machine hosts multiple nodes, each node is attached to its own private Docker network.
Temporary Volumes
Each algorithm mounts temporary volume, which is linked to the node and the
job_id
of the taskThe mounting target is specified in an environment variable
TEMPORARY_FOLDER
. The algorithm can write anything to this directory.These volumes need to be cleaned manually. (
docker rm VOLUME_NAME
)Successive algorithms only have access to the volume if they share the same
job_id
. Each time a user creates a task, a newjob_id
is issued. If you need to share information between containers, you need to do this through a master container. If a central container creates a task, all child tasks will get the samejob_id
.
RESTful API
- All RESTful API output is HATEOS formatted.
(wiki)
Local Proxy Server
Algorithm containers no longer receive an internet connection. They can only communicate with the central server through a local proxy service.
It handles encryption for certain endpoints (i.e.
/task
, the input or/result
the results)
Dockerized the Node
All node code is run from a Docker container. Build versions can be found at our Docker repository:
harbor.distributedlearning.ai/infrastructure/node
. Specific version can be pulled using tags.For each running node, a Docker volume is created in which the data, input and output is stored. The name of the Docker volume is:
vantage-NODE_NAME-vol
. This volume is shared with all incoming algorithm containers.Each node is attached to the public network and a private network:
vantage-NODE_NAME-net
.