Health Dashboard: optimization of cost and SLA
New operational graphical interfaces for global monitoring, analysis and reporting of overall platform behavior, including real-time graphs and comprehensive statistics for analyzing job performance, service efficiency, compute resource utilization and scheduled jobs over different time periods.
This dashboard primarily caters to the needs of managers and supervisors, enabling them to oversee ongoing activities and make rapid decisions based on the presented information.
The enhanced charts will reveal the top workflows with the most issues, those with the longest pending duration, the longest execution times, the highest cumulative CPU time across all tasks, the most node-intensive jobs, and those with the most parallel execution.
In addition, the dashboard provides recommendations to:
- Discover efficiency gains with the ‘Optimize Execution Times’ section, using charts to analyze node requirements and potential time savings across various periods.
- Identify potential node reductions without impacting execution times with the ‘Optimize Node Usage’ section.
The aim is to make our customers more efficient by providing them with deep insights into an organization’s day-to-day activities and performance. Even if everything is working well, there may be room for improvement.
Analytics & Reporting for Governance and Traceability
Reporting has become a cornerstone of informed decision-making. This is why ActiveEon has put a lot of effort into improving the reporting capabilities, making it easier than ever for users to generate reports swiftly and effortlessly.
Today, users can create comprehensive reports with just a few clicks, eliminating the need for an in-depth understanding of the underlying data. This newfound simplicity has democratized access to data analysis, allowing individuals from various backgrounds to harness the power of data for their decision-making processes.
When it comes to governance and traceability, you can now filter the catalog based on who modified objects last time (Last Commit By), or who modified objects in the past (Committed at Least Once By), at any time or over a given period:
Then, you can easily generate a report on the selected Objects:
Integration in Customer and Third-party Portals
We offer external job submission and monitoring windows that can be customized and triggered outside our software, from inside the customer’s own portal, with a simple URL invocation. To do so, you just need to provide a valid Session Id together with the bucket name and workflow name.
The main advantage of this feature is that customers can continue to use their front-end interfaces to not change their end-user’s experience with a minimal integration effort for ProActive job submission and monitoring.
Select the Job you want to Submit from an independent window triggered from your own portals:
Enter Job Parameters:
Follow the execution of your Job in an independent window (Graph, Tasks, Output, etc.):
And directly access third-party endpoints (Visdom, Tensorboard, Jupyter Lab, MLOps Dashboard, etc.):
Workflow Studio: fully new way to manage Variables and Job Dependencies
In addition to a new workflow variable management interface, the Studio also allows you to view and navigate workflow dependencies, enabling navigation between pipelines.
New Variable Management Interface: The ProActive Workflow Studio, which is used to graphically build business processes, has been further improved by offering:
- A new and improved view for displaying and managing workflow variables. Workflow designers can now change the order of variables created so that they make sense when users launch the job. They can also group them logically to present users with a structured set of parameters to be provided.
- Navigation between Pipelines: The ability to graphically visualize the dependencies between Workflows and navigate the full pipelines. Tasks that invoke other catalog workflows are now highlighted by a dedicated icon. The workflow designer can then click on this icon to edit the referenced objects and visualize them in another studio tab if needed. For a complete view of dependencies, global menus have been added to the top left view of the studio to display all called and called by workflows.
Workflow Execution
The Workflow Execution portal now features brand new Filtering, Detachable Window, Label for classification, new Contextual menus, and a few other goodies.
The Workflow Execution portal provides a single and centralized pane of glass for greater visibility of all current and past jobs executions.
To improve operational use of the platform, this portal has been enhanced with an advanced filtering capability based on an extensive multi-criteria search engine that scans the execution history of past jobs and returns those that match the selected criteria. There’s no need to use a third-party product - you can access all the information at the click of a button.
Additionally, users are now able to tag a job with a specific label. A Label is a property that can be attached to a job, and changed dynamically by users. The Labels give the possibility to tag Jobs, classify them and manipulate them by filtering based on these custom labels. These Labels act as a custom status for the Jobs. For example, a company can label jobs which are having errors to be reviewed by operators (label “To-be-reviewed”), and later mark them as “Reviewed” when done. Job labels can be used for this specific purpose, but also for any other purpose.
Direct access to a contextual menu, that is activated by a simple right-click on the selected job, is another interesting feature that enhances the user experience. Depending on the type and status of the job, a series of possible actions are activated.
Finally, you can also follow the execution of a specific Job in a separate, independent window, for example to track and resolve specific issues:
Catalog
The Catalog portal now features new Label, Filtering by Tags, Visualize and filtering Planned objects, List mode, Contextual Menus, Export/Import, and a few other advanced new features.
Labels & Tags: To improve the management and organization of objects in the Catalog, we added the possibility to label catalog objects using a new metadata attribute called Tags. Now, users are able to classify and filter objects with regard to any business or application aspect. This feature can be useful to manage and monitor the lifecycle of the workflow implementation (e.g. Development, Testing, Staging, Production) when having a single instance of ProActive, or even having several instances and using an automated promotion between environments.
One-Click Planning: We also considerably ease the burden on the planning process by adding the possibility to schedule workflows directly from the Catalog. With a simple right-click, the operational team is now able to select a calendar to schedule a job on a regular basis without going to the Job Planner portal. Planned workflows are highlighted by a clock icon to distinguish them visually.
Advanced Filtering: To improve operational use of the platform, the Catalog portal has been enhanced with an advanced filtering capability based on an extensive multi-criteria search engine that browses the catalog and returns objects that match the selected criteria.
Direct access to contextual menus, that is activated by a simple right-click on the selected bucket or catalog object, is another interesting feature that enhances the user experience:
In terms of collaborative working tools, the functions for exporting and importing catalog objects have been improved by including a metadata file in the archive that contains meta-information about all the selected objects. Catalog objects can now be exported as a structured package so that users can easily share and promote them between different ProActive installations (Dev, Q/A, Prod, etc.):
Job Planner
The Job Planner portal now features new Role Based Access Control, Housekeeping configuration, Owner of a given planning, Time zones selection, Operational dashboard to display overall status of planned jobs (Active, Deactivated, or Failed), Filtering on workflow name and bucket.
Job Planner Role Based Access Control (RBAC): it acts as a control model for a system in which each access decision is based on the role to which the user is associated. The security of the Job-Planner module has been strengthened to provide the same level of governance and compliance as the rest of the application.
In the Job Planner, RBAC operations are only performed by Job Planner admins or associations’ owners. RBAC operations can be distinguished in two types:
- Managing an authorization for an association
- Managing an authorization for all associations owned by a specific user
As with the traceability of workflow versions and authorizations granted, we have added traceability of job schedules ownership with a complete history of all actions (who did what and when). As a result, we can now check who and when a specific job schedule was created, activated and deactivated. What’s more, all this information can be aggregated and exported in a PDF report.
The aim is to strengthen governance within the organization and consolidate the ability to respond strategically to risks and support ongoing compliance with the necessary standards, rules, policies, and legislation.
Time Zones: Within a single organization, ProActive is used by different teams around the world, often located in different time zones. Instead of having a company-preferred time zone, we extend the management of international teams by easily filtering and selecting time zones.
To improve operational efficiency, the Job Planner portal has been enhanced with a monitoring dashboard that gives users a complete and detailed view of the status of planned jobs (total, activated, deactivated and failed). This dashboard primarily meets the needs of operators, helping them to better maintain and keep track of job schedules, especially the failed ones. In addition, operators can now activate and deactivate job associations in bulk mode, enabling them to stop production quickly during maintenance periods.
Notifications
The Notification Portal now includes an event preset “All with issues” for job subscriptions, Resource Manager notifications, and many improvements.
Resource Manager (RM)
With the new Resource Manager (RM), first enjoy a new display with improved layout, new informations, and new filters, then reduce Cloud Cost with an improved management of Spot Instances, together with Azure Cost Tracking. The new RM also includes improved Business Triggers for dynamic resource allocation, allowing further more savings. Moreover, the use of edge computing has been reinforced by the provision of a lighter version of the ProActive agent with support for armv7 and armv8 architectures.
Edge Agents: it enables jobs to be processed locally and remove the need to transfer a vast amount of data through the network. An edge device is hardware that sits at the periphery of a computer network. Some edge devices are entirely autonomous and self-contained in doing tasks as they process and act on data locally. One use case is the deployment of agents in satellite electronic boards, where jobs are processed locally at regular intervals for fire detection.
In terms of hybrid compute resources deployment and management, several improvements have been implemented:
- SSH Node Sources support the latest SSH encryption algorithms and private key formats.
- AWS EC2 Node Sources allow for the allocation of specific EC2 instance types.
- GCE node sources allow for the allocation of a specific machine type.
- Support spot instances for Azure ScaleSets infrastructure, which allows a significant Cloud cost reduction. With ProActive, enterprises can go further and automatically procure the much cheaper spot instances when dynamic resources are needed to finish a job.
- Support adding tags to Azure ScaleSets.
- Include business triggers for dynamic resource allocation. It now allows defining a scale up / scale down expression which aims to solve complex deployment scenarios (for example when a node source scale up depends on the state of another Node Source).
- Add a dedicated logo for each node source type for faster recognition.
Automated Installation and Software Upgrade
The new automated installations include Physical, VMs, Docker and Kubernetes installations.
Save time and efficiency with scripts that streamline installation across multiple environments. Administrators can run ProActive installation scripts that will automatically install and configure the ProActive instance and agents (nodes) across multiple environments: Physical and Virtual Machines, Docker, and Kubernetes.
Not only will the automated installation install and configure the instances, it will also go through a set of validation and health check steps to ensure the installation is successful.
Moreover, the automation also includes subsequent upgrades in automatic mode.
Finally, the ProActive installation is now more secure, providing an efficient and automated mechanism to encrypt/decrypt all required passwords using the OpenSSL library.
Multi-Tenancy, Improved Security, and Vulnerability Scanning
At ActiveEon, we put security, privacy, and quality at the heart of our software and systems.
Our Role-Based Access Control management system is improved for easier rights management inside the ProActive server. Roles can now be defined using general categories (for example, read access on Resource Manager). We also added the ability to deny access to a specific operation.
In addition, we are pleased to announce that the vulnerability scanning is an integral part of Activeeon’s DevSecops. Activeeon performs a vulnerability scan on all software artifacts delivered to customers. This includes ProActive libraries, installation packages and Docker images. The main objective is to protect Activeeon’s customers from security breaches and exposure of sensitive data.
With this release, we unlock the power of multi-tenancy by integrating the support of LDAP or Active Directory authentication with multiple domains. The proposed solution must offer an architecture that ensures isolation of resources, data, configurations, and workloads among different customers or tenants and ensures that tenants can only access their allocated resources.
Finally, the identity and access management of ProActive is integrated with Keycloak using the OpenID Connect protocol (OIDC). In particular, a new login module is developed to perform authentication and authorization via the OIDC REST endpoints exposed by Keycloak.
PAIO (ProActive AI Orchestration) and MLOps Dashboard
PAIO offers a complete solution for deployment, training, execution and scalability of artificial intelligence and machine learning models for DataOps, Data Scientists, AI Engineers.
MLOps Dashboard serves as a centralized hub for data scientists, engineers, and stakeholders (DevOps) involved in deploying and monitoring machine learning models in testing and in production.
The new version features:
Model Servers Monitoring: provides key widgets with general insights into model server performance (number of inferences performed within a specific timespan, total number of inferences overall, average inference time and rate, etc.). In a second part, there is a detailed table listing specific characteristics of each model server instance (status, start time, count of models deployed, total and timespan inferences, average/min/max inference time and rate, etc.).
Models Resource Usage, CPU, Memory, GPU, etc.: provides users with valuable insights into the CPU and GPU resource utilization, both real-time metrics and time series data for each model server. Overall, it features CPU, GPU, memory usage, and power consumption.
Data and Operational Drift Detection: configure and trigger notifications based on various monitoring thresholds for different scopes, including: Per Deployed Model, Per Model Server, Global, etc. Select yourself the severity level assigned to the notification (INFO, WARNING, ERROR, CRITICAL).
Automated Alerting and Notifications: Upon a Drift Detection, a notification can be sent to the portal, by email, Sms, or to a third-party monitoring system.