machine learning as a service

New Major Release V12 June 23rd 2021: ProActive Workflows & Scheduling (PWS) and ProActive AI Orchestration (PML)

Try new version online or on premises

ProActive Workflows & Scheduling V12.0 (named Perseverance) is out and available for trial and download (free trial version downloadable from your account). This new version includes a lot of new features that enhance this already robust and flexible workload automation solution.

Enjoy the following major features

  • A completely new portal named Workflow Execution (WE) with many new features and a modern ergonomic design has been added to the Automation Dashboard (Workflow Automation (WA) has been retired). Workflow Execution portal allows users to interactively control and influence Job execution with signals, and also to directly access Job Endpoints (Remote Visu, HTTP/HTTPS interfaces, etc.), as well as controlling Service execution, all from the WE portal.
  • Service Automation has been extended with new features and a new portal named ProActive Service Automation (PSA) is available to manage all the services (ProActive Cloud Automation (PCA) has been retired).
  • For event-based scheduling, ProActive Event Orchestration (PEO) is now available, replacing the former ProActive Cloud Watch (PCW).
  • The AI and Machine Learning product has been dramatically improved with many features, it is available under the name ProActive AI Orchestration (PML), and includes the Machine Learning Open Studio (MLOS).
  • A new tool is now available to automatically convert AutoSys jobs into ProActive Workflows.
  • Docker support has been dramatically improved, for instance allowing Dockerfile script engine to dissociate image creation from container execution, while providing different Docker configurations to run ProActive (with different databases, users, passwords, etc.). Overall, offering an effortless deployment of ProActive in a fully containerized environment using Docker.
  • Azure marketplace has been updated to include the latest ProActive version.
Try now

New in ProActive AI Orchestration

  • Added Automatic Detection of Categorical and Numerical Features as a generic ML task
  • Improved NVIDIA RAPIDS support on AI Workflows
  • Added examples of GPU accelerated Workflows for Machine Learning
  • Replaced Web_Validation by Signals on AI Workflows
  • Added support for VSFTP on AI Workflows
  • Updated AI Workflows documentation link
Try now

Scheduler

  • Now features a .NET client library for ProActive server API, based on the HTTP protocol
  • Security headers were added to Jetty responses in order to improve security
  • Add a new endpoint to remove a list of jobs finished before a given time (epoch time)
  • New ProActive system accounts dedicated to Data Scientists are now added by default (citizen-ds, expert-ds) with specific Access Rights
  • Improve the performance of jobs that contain more than 1 000 tasks
  • Upgrade the provided jre version from 8u131 to jre 8u281
  • Add the capacity to configure global Variables and generic information in a config file at the level of an installation, in order for them to be automatically used by any submitted Workflows
  • Add the possibility to globally configure the maximum duration time of tasks in the server config files. If configured, the tasks will be terminated after the walltime exceeded
  • Capability to start ProActive server in frozen mode or with paused jobs
  • Improve logging messages for scheduler and microservices logs

Workflow execution

  • New ergonomic portal
  • Interactive control of jobs with signals (Stop, Continue_Simulation, etc.)
  • A user-friendly modal for job submission is provided to easily browse the catalog objects and submit jobs with Variable values
  • Save user preferences locally in browser cache (e.g., selected bucket, sorting, etc.)
  • Advanced job filtering to show pending, current, and past jobs. It is also possible to filter the user’s jobs only
  • Add the capability to display or hide sub jobs (children of a parent job)
  • Handle Services with or without endpoints
  • Add the possibility to attach service endpoints to a given job (for visualization, data access, etc.)
  • Add the capability to browse and download job’s results
  • Add the possibility to pause, resume, restart (in-error tasks), kill and resubmit jobs with variables
  • Manage permissions for signals: hide signal actions for users that do not own the service and do not have access to them (non-admins)
  • Jobs are presented in a grid where users can easily sort the jobs by IDs, status, submission time, etc.
  • Easy access to Job Logs and Output in Streaming mode
  • One-click access to all Interactive Services (Jupyter Lab, Tensorboard, Visdom, etc.)
  • One-click access to Remote Visualization interfaces (HPC)
  • Asynchronous upload and download of files, detaching the modal when uploading files not to block the entire portal
  • A progress bar indicates what files are being Uploaded, and the expected time of completion
  • Interfaces with distributed and parallel file systems using SFTP/VSFTP protocol (VSFTP protocol added as a specific extension inside ProActive that adds the ability to configure SFTP root directories with a set of predefined environment Variables)
  • Jupyter Lab on demand with Kernel to submit Jobs directly from Python

Scheduling Portal

  • Send signals to running jobs to control their executions (stop, Continue_Simulation, kill, etc.)
  • Control the access to jobs and tasks details for non-authorized users
  • Export / import Job Filters and save settings in browser cache
  • Resubmit many jobs at once
  • Add “started at”, “finished at”, “submitted at” properties to the jobs and tasks grid display
  • Save job and task grid layout and columns in browser cache

Resource Manager

  • Provide support to manage loggers (edit levels, add new loggers)
  • More flexible Kubernetes deployment: ProActive node startup script and Kubernetes deployment template now can be customized through node source parameters
  • Natively support the execution of docker-related tasks on Kubernetes nodes
  • Acquire AWSEC2 Infrastructure and Openstack Infrastructure nodes from REST endpoint now support configure vmTypes, ports and node tags
  • Add option to copy data from Selection and Monitoring tabs
  • Ability to list regions, images, hardwares, node candidate offers for both AWS and OpenStack through connector-iaas
  • Add ability to acquire nodes with specific VM config for OpenStack Infrastructure and AWSEC2 Infrastructure

Catalog

  • Add the possibility to edit the project name of a catalog object
  • Group referenced scripts and Workflows in the same buckets
  • Sort Workflow Variables and buckets’ Workflows alphabetically

Job planner

  • Improve the Gantt visualization when the refresh occurs at the moment of a submission
  • Show the visualization of a failed or deactivated association in the Gantt view

Studio

  • Add the possibility to associate a project name to a Workflow when pushing it to the catalog
  • Double click on a Workflow task to access its task implementation
  • Add the possibility to use Variable model type: A Variable can dynamically use, as its type, a model that is defined in another Variable
  • Add the possibility to create nested folders in User and Global spaces

Notifications

  • Refactor notification methods display in subscriptions forms from vertical to horizontal alignment
  • Add a configuration tab where users can update their personal notification-service settings and create/manage channels
  • Reorganise filters order in notifications view and add a drop-down for severity filters
  • Add Channel creation and management to be able to notify groups of users
  • Add Channel subscription
  • Add Channel notification
  • Enable users to resend email and third-party confirmation requests when confirmations are enabled
  • Add log messages when the notification-service could not send emails due an SMTP server failure
  • Add three new job notifications: Job updated, Task visu activated and Task progress updated
  • Add sections about configurations in job planner’s documentation
  • Implement specific notifications for Job planner (Association failed, Cannot submit, Delayed submission executing, Delayed submission executing, Submission postponed, Submission canceled)
  • Add slack notification as a third-party notification method example

PSA (ProActive Service Automation)

  • It is now possible to submit services via standard Workflows (also called Management Workflows).
  • A set of Workflows are now provided to manage PSA service lifecycles (e.g., launch and stop PSA services)
  • Improve the service status visualizations and the associated actions
  • Improve the support of TCP protocol endpoints for PSAs database such as MySQL, Postgres, mongoDB, etc.

ProActive examples

  • New connectors to KNIME, Greenplum, Centreon, JD EDwards and IBM InfoSphere are added to the catalog
  • New Workflows to manage Hadoop, HDFS, Swarm, Spark on demand
  • New Workflows to manage Azure HDInsight Spark clusters (create, get status, delete, etc.).
  • New Catalog Workflows for SSH terminal interactive access (on demand and on the browser) on resources allocated on the Cluster, including multi-nodes
  • An example of Remote Visualization using Xvnc is added
  • An example of an HPC Workflow is added, which performs a molecular behavior simulation and visualization using the platforms GROMACS, VMD and Xvnc
  • A Workflow has been added to the Catalog to download the metadata and images from the Copernicus and PEPS platforms

AutoML

  • Renamed Multi_Tuners_Auto_ML Workflow to Distributed_Auto_ML
  • Workflow Variables refactoring
  • Added support for Signals
  • Added stop services for Visdom and Tensorboard
  • Added a new Variable named Number of Repetitions
  • Added AutoML task template in R language
  • Added Workflows template for Generic Distributed Tensorflow (SLURM and Horovod)

ML Model as Service

  • Improved the Data Drift Detection (DDD) mechanism
  • Added three new methods for Data Drift Detection (DDD) named HDDM, Page Hinkley, and ADWIN
  • Improved Workflow examples in the Catalog
  • Added GPU support using NVIDIA RAPIDS
  • Variable names refactoring and non-necessary Variables are removed
  • Improved REST API endpoints

DL Model as Service

  • Added Deep Learning Model as a Service (MaaS_DL) using TensorFlow Serving
  • Added Workflow examples based on MNIST training and deployment

Documentation

  • Updated PML and PWS documentation
Request demo

More news  

Published on June 23, 2021