How to Continuously Qualify Microsoft Azure PaaS Solutions

Most Azure users agree that Azure Infrastructure-as-a-Service (IaaS) can be qualified, but there are still questions around whether Azure Platform-as-a-Service (PaaS) – which includes serverless architectures – can be qualified as well.

First, it’s important to understand that a software app that resides on IaaS/PaaS is “validated,” while IaaS/PaaS solutions themselves are “qualified.” A SaaS application can only be considered truly “validated” if the underlying IaaS/PaaS solutions are fully qualified.

Continuous qualification (CQ) involves providing documented evidence to certify that a PaaS solution not only met pre-established acceptance criteria but also continues to meet those criteria, thus mitigating the risk of unknown changes.

A Step-by-Step Guide to Azure PaaS Qualification

Follow these steps to intelligently qualify your Azure PaaS solution and maintain it in a qualified state (QS).

Step 1: Establish Formal User Requirements

The user requirements define the intended use of the PaaS solution across your organization. Once the Azure PaaS solution is qualified for its intended use, any development team within your company can use it for their projects.

To develop the intended use, categorize your requirements into the following:

• Functional
• Security and Compliance
• Availability
• Performance  
  

Step 2: Establish a Risk-Based Approach to Qualifying Each Requirement

Your risk-based approach can be achieved by assigning a risk priority to each requirement, as follows:

High – A risk priority of high should be assigned to a critical requirement that meets the following criteria:

• Is not “out-of-the-box” (OOTB) functionality AND
• Is a legal and/or regulatory requirement

All high priority requirements will be tested using both positive and negative testing.

Moderate – A risk priority of moderate should be assigned to an important requirement that meets the following criteria:

• Is achieved with OOTB PaaS features AND
• Is a legal and/or regulatory requirement

All moderate priority requirements should be tested using positive testing or verified with configuration verification.

Low – A risk priority of low should be assigned to a “nice-to-have” requirement that is achieved with OOTB software features. Low priority requirements do not need to be tested.

Step 3: Build a “Continuous” Qualification Framework

This framework should automatically perform various tests to ensure all applicable high and moderate priority requirements are met. Test execution reports with all evidence should be automatically generated. IT Quality should review the results, then certify the Azure PaaS solution for its intended use.

Step 4: Make Your Azure PaaS Solution Is Available in Your Global Catalog

Once your PaaS solution is qualified, it should be globally available for teams to deploy any number of times, without worrying about additional qualification.

Step 5: Establish a Qualification Schedule

You should be running qualification tests daily to provide evidence that the qualified state of the Azure PaaS service has not drifted.

Maintaining Azure in a Qualified State

Microsoft Azure releases changes on a near-constant basis so the end user can leverage up-to-date innovations and increase productivity, and Azure PaaS is no exception. However, using a traditional validation process, you would need to review each and every change, then address it one way or another. 

To truly embrace the cloud and all its benefits, you have to change your compliance perspective from manually reviewing every change – which is practically impossible considering the number and velocity of changes – to ensuring that your requirements are met constantly through a continuous validation framework.

Monitoring for QS Drift

Even after the PaaS solution is built and deployed, its health must be monitored to ensure that qualified state drift has not occurred. QS drift can occur when changes to the deployed service are made, intentionally or unintentionally, after it is qualified by bypassing the change control process.

Microsoft Azure provides various tools to help you ensure that the qualified service is functioning as expected:

• Automation Control enables continuous services and compliance with automation and configuration management. This service will enable compliance with the qualified state and will alert you if QS drift has occurred.
• Log Analytics enables you to quickly collect log data to transform Azure activity information into actionable insights.
• Azure Monitor provides detailed, up-to-date performance and utilization data, as well as access to the activity log that tracks every API call, and diagnostic logs to help you debug any issues. This feature offers the basic tools you need to analyze and diagnose any operational issue, so you can resolve it efficiently.
• Azure Advisor helps optimize across four categories: high availability, performance, security, and cost, with all recommendations accessible on the Azure portal.
• Security and Compliance helps you analyze events across multiple data sources to identify potential security risks, understand the scope and impact of threats, and mitigate the damage of a security breach.
• Azure Policy helps you turn on built-in policies or build your own custom policies to enable security and management for Azure PaaS resources.

Continuous qualification is GxP compliant and based on a sophisticated model testing framework. This approach enables cost-effective testing on a continuous basis, thus allowing deployment of GxP workloads in the public cloud.