.. include:: .. _hub-admin-guide-store: Store ===== The vantage6 store is a component that stores the algorithms that can be used by the nodes. This allows the user to easily find the algorithm they need, and know how to use it. There is a community algorithm store hosted at `https://store.uluru.vantage6.ai `_. This store is maintained by the vantage6 community and allows you to easily reuse algorithms developed by others. You can also create your own algorithm store. This allows you to create a private algorithm store, which is only available to your own collaborations. If you would like to contribute to the community store, you should first check the `production-ready algorithm guidelines `_ to see if you meet the requirements. If you do, you can send an email to Frank Martin or Bart van Beusekom to acquire an account and upload your algorithm. The algorithm will go through a review process before it is added to the community store. Linking algorithm store to HQ ------------------------------- Algorithm stores can be linked to a vantage6 HQ or to a specific collaboration on an HQ. That way, the algorithms in the store become easily available to the users - either to all users registered on that HQ, or to all users in the specific collaboration. Users can link algorithm stores to a collaboration if they have permission to modify that collaboration. Algorithm stores can only be linked to an HQ by users that have permission to modify all collaborations on the HQ. To link an algorithm store, go to the collaboration settings page on the UI or use the Python client function ``client.store.create()``. .. _algorithm-store-configure: Configuration options --------------------- The algorithm store requires a configuration file to run. This is a ``yaml`` file with a specific format. The next sections describes how to configure the algorithm store. It first provides a few quick answers on setting up your store, then shows an example of all configuration file options, and finally explains where your configuration files are stored. How to create a configuration file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The easiest way to create an initial configuration file is via: ``v6 algorithm-store new``. This allows you to configure the basic settings. For more advanced configuration options, which are listed below, you can view the :ref:`example configuration file `. Where is my configuration file? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To see where your configuration file is located, you can use the following command .. code:: bash v6 algorithm-store files .. warning:: This command will only work for if the algorithm store has been deployed using the ``v6`` commands. Also, note that on local deployments you may need to specify the ``--user`` flag if you put your configuration file in the :ref:`user folder `. You can also create and edit this file manually. .. _algorithm-store-config-file-structure: All configuration options ^^^^^^^^^^^^^^^^^^^^^^^^^^ The following configuration file is an example that intends to list all possible configuration options. You can download this file here: :download:`algorithm_store_config.yaml ` .. _algorithm-store-configuration-file: .. literalinclude :: yaml/algorithm_store_config.yaml :language: yaml .. todo this section is close duplicate of docs/node/configure -- merge? .. _algorithm-store-configure-location: Configuration file location ^^^^^^^^^^^^^^^^^^^^^^^^^^^ The directory where to store the configuration file depends on your operating system (OS). It is possible to store the configuration file at **system** or at **user** level. At the user level, configuration files are only available for your user. By default, algorithm store configuration files are stored at **system** level - except if you have created a sandbox environment using the ``v6 sandbox`` commands. The default directories per OS are as follows: +---------+----------------------------+------------------------------------+ | **OS** | **System** | **User** | +=========+============================+====================================+ | Windows | |win_sys| | |win_usr| | +---------+----------------------------+------------------------------------+ | MacOS | |mac_sys| | |mac_usr| | +---------+----------------------------+------------------------------------+ | Linux | |lin_sys| | |lin_usr| | +---------+----------------------------+------------------------------------+ .. |win_sys| replace:: ``C:\ProgramData\vantage\algorithm-store`` .. |win_usr| replace:: ``C:\Users\\AppData\Local\vantage\algorithm-store`` .. |mac_sys| replace:: ``/Library/Application/Support/vantage6/algorithm-store`` .. |mac_usr| replace:: ``/Users//Library/Application Support/vantage6/algorithm-store`` .. |lin_sys| replace:: ``/etc/xdg/vantage6/algorithm-store/`` .. |lin_usr| replace:: ``/home//.config/vantage6/algorithm-store/`` .. warning:: The command ``v6 algorithm-store`` looks in certain directories by default. It is possible to use any directory and specify the location with the ``--config`` flag. However, note that using a different directory requires you to specify the ``--config`` flag every time! Similarly, you can put your algorithm store configuration file in the user folder by using the ``--user`` flag. Note that in that case, you have to specify the ``--user`` flag for all ``v6 algorithm-store`` commands. Permissions ----------- Policies ^^^^^^^^ Algorithm store policies are defined by the algorithm store administrator in the configuration file and determine the general permission and access rules for the algorithm store. Arguably, the most important policy is who is allowed to view the algorithms in the store. For the community store, this is set to public, meaning that anyone can view the algorithms. For a private store, this can be set to private, meaning that only authorized users can view the algorithms. Other examples of algorithm store policies are e.g. setting how many reviewers are required to approve an algorithm, or whether algorithm reviewers should be from a different organization as the algorithm developer or not. Permission management ^^^^^^^^^^^^^^^^^^^^^^ Apart from the policies, there are also access rules at the user level. Rules are used to determine the actions that a user is allowed to take in the algorithm store. In order to perform operations in the algorithm store, a user must be registered in the algorithm store and must be authenticated. Then, rules can be assigned to the user to give them the necessary permissions. Just like in the vantage6 HQ, in the algorithm store rules are used to allow or prevent a user from performing an operation. An operation is an action that can be performed on a resource of the algorithm store. The following operations are defined: #. Create #. Delete #. Edit #. View These operations can be performed on the available resources according to the following schema: .. list-table:: :name: rules-algo-store :widths: 20 20 20 20 20 * - Resource - View - Create - Edit - Delete * - Algorithm - ✅ - ✅ - ✅ - ✅ * - User - ✅ - ✅ - ✅ - ✅ * - Role - ✅ - ✅ - ✅ - ✅ * - Review - ✅ - ✅ - ✅ - ✅ Rules can be assigned to a user by another user who has at least the same permission level as the rules assigned. Single rules can be assigned, but default combinations of rules are available, as roles. The following default roles available in the algorithms store: #. **Root**: Has all permissions. #. **Developer**: Can submit new algorithms to the store and edit them before they are reviewed. #. **Algorithm Manager**: Can assign reviewers to new algorithms, and manage algorithms. Whenever a new algorithm is submitted, users with permission to register new reviews are alerted, so users with this role as well as the root role will be alerted to assign reviewers (if an email server has been set up). #. **Reviewer**: Can approve or reject algorithms that they have been requested to review. #. **Viewer**: Can view all resources in the store. #. **Store Manager**: Can manage the store's users and their permissions. Note that all default roles have permission to view all resources. To give an example, the permissions of a reviewer are shown below. .. list-table:: :name: rules-algo-store-reviewer :widths: 20 20 20 20 20 * - Resource - View - Create - Edit - Delete * - Algorithm - ✅ - ❌ - ❌ - ❌ * - User - ✅ - ❌ - ❌ - ❌ * - Role - ✅ - ❌ - ❌ - ❌ * - Review - ✅ - ❌ - ✅ - ❌ .. _algorithm-store-processes: Store processes --------------- The algorithm store manages the lifecycle of vantage6 algorithms, from its initial submission by the algorithm developer to the running of the algorithm and finally its replacement by a newer version. Here, we give an overview of these processes. Algorithm submission ^^^^^^^^^^^^^^^^^^^^ The first step in the lifecycle of an algorithm is its submission to the algorithm store. An algorithm developer can do this via the algorithm store section of the UI or by using the Python client's command ``client.algorithm.create()``. The algorithm developer needs to provide data such as a name, description, where to find the code and the algorithm image, and which functions the algorithm provides and how to call them. Each function of the algorithm is described, apart from its name and description, by the following fields: - **Parameters**: A list of parameters that the function expects. Each parameter has a name, a description, and a type. For example, if you want to compute an average, a parameter could be a column name. Apart from standard data types like integers, strings and booleans, vantage6 also supports *organizations* and *columns* as parameter types. When using these types, the user interface knows to show a dropdown with the available organizations or columns. - **Databases**: A list of databases that the function expects. Most algorithms use a single database, but some algorithms might need multiple databases (e.g. one with patient data and another with population data). Each database has a name and a description. The user interface will show a dropdown with the available databases when the user needs to select a database. - **Visualizations**: A list of visualizations that the function can produce. Each visualization has a name, a description, and a type. When viewing the results of an algorithm run in the UI, the UI will attempt to plot the results if a visualization is available. Depending on the visualization type, additional data might be required. For instance, for a line graph, the algorithm developer can set the x-axis and y-axis columns that should be visualized. Note that there is a utility command in the CLI ``v6 algorithm generate-store-json`` that can be used to generate a JSON file that can be used to import the algorithm into the store. This is easier than manually submitting the algorithm, but to be most useful, it requires you to have proper docstrings in your algorithm code. .. _algorithm-store-review-process: Algorithm review ^^^^^^^^^^^^^^^^ After an algorithm is submitted, it needs to undergo a review process. At this point, a store manager will assign reviewers for this particular algorithm. Reviewers are recommended to assess the algorithm using the :ref:`algorithm-review-checklist`. The reviewers can then view the algorithm and provide feedback. If the algorithm is approved, it will be easily runnable by researchers using the UI. While the algorithm is under review, it is not yet available for running tasks via the UI. If any of the reviewers rejects the algorithm, or the store manager does not assign reviewers, the algorithm will not become available. Note that an algorithm may still be run via the Python client, but a node configured to allow only algorithms from a certain algorithm store will not accept algorithms that are not yet approved in that store. The reviewer can provide comments to the developer when rejecting an algorithm. If the algorithm is rejected, the process is repeated as soon as the developer submits an improved version of the algorithm. If your algorithm store has been configured with an email SMTP server, emails will be sent all along the process to alert users that, for instance, their review is requested. Regularly, a developer has submitted an update to an algorithm that was already approved. In such cases, when the changes are approved, the algorithm store will invalidate the previous version of the algorithm. This means that the previous version can then no longer be used to run tasks. At any point, the store manager can also invalidate an approved algorithm without replacing it with a new version. This is a safeguard to ascertain that algorithms can quickly be removed from the store if necessary. Everyone involved in the process (developers, store manager, reviewers, and researchers) can only execute their actions after logging into vantage6. Note that one user may fulfill several roles: a store manager may also be an algorithm reviewer. There are however some restrictions, that can be set up in the :ref:`algorithm store policies `. .. include:: ./policies.rst