Azure DevOps for Dyn365FO Create Deployable Package - Include Source Controlled ISV Binaries

Background

This post will be a little different to my recent articles which have been in an step by step guide format. Instead, it's an addition to the guide for replacing your Create Deployable Package build step, which will allow you to include source controlled binaries in the Deplyable Package generated by Azure DevOps.

Prerequisites

If you are looking for a guide to implementing the Create Deployable Packages Azure DevOps Task by Microsoft, see my post here;

Azure DevOps Build & Release Pipeline for Dyn365FO - Create Deployable Package Task

For adding your ISV binaries to source control, so that your build box can include them as part of your single generated deployable package, see here;

Manage third-party models and runtime packages by using source control

Background

When I implemented the Create Deployable Package task in an implementation project recently, I noticed that the package size generated by the new Task, compare to the old Powershell Script, was significantly smaller. The new Zipped package was 25MB, when I was expecting it to be closer to the 135MB package generated previously.

Problem

On inspection, the problem was that the new package only contained binaries from any source code built by the agent during this run. So in this example, it contained our customer implementation model, but none of the ISVs which we also store in source control. 

Solution

On the build box, in the working folders for the build agent, the folder I setup in the build task parameters only contains the built binaries, and not the source controlled ISV binaries;

$(Agent.BuildDirectory)\Bin

 The folder which contains everything from source control is actually;

$(Agent.BuildDirectory)\s\Metadata

To include both folders, a change is needed to the Create Deployable Package task parameter for location of binaries.

Create Deployable Package - Old Parameters

In the Location of X++ binaries to package field, remove the \Bin folder;

$(Agent.BuildDirectory)

In the Search pattern for binaries to package field, replace the * with the following to include both folders;

Bin/*
s/Metadata/*

Create Deployable Package - Updated Parameters

It is important to note here, that the Search pattern for binaries to package uses Azure DevOps file matching patterns, which use the forward slash to denote directories, even when your build server is running on Windows. Read more about them here;

File matching patterns reference

Save and queue the pipeline, and on the next run your binary ISVs will be back.

Azure DevOps Build & Release Pipeline for Dyn365FO - Add Licenses to Deployable Package Task

Add Licenses During the Build Pipeline or Release Process

Background

In this post, I will be focusing on the Add Licenses to Deployable Package Task, which is the second of the new tasks I have had the opportunity to try out. To read about the Create Deployable Package Task, take a look at my previous post;

Azure DevOps Build & Release Pipeline for Dyn365FO - Create Deployable Package Task

Microsoft have released 3 new Azure DevOps tasks to support the upcoming preview of Azure DevOps hosted builds, which will enable building without a dedicated server hosting the build agent.

The tasks are;

• Create Deployable Package 

• Add Licenses to Deployable Package

• Update Model Version

Add tasks from Dynamics 365 Finance and Operations Tools

Information about all 3 new tasks has been posted on release by Microsoft's Joris de Gruyter here;

https://community.dynamics.com/365/financeandoperations/b/newdynamicsax/posts/azure-devops-tasks-for-packaging-and-model-versioning

Add Licenses to Deployable Package Task

As with the new Create Deployable Packages Task, the Add Licenses to Deployable Package Task brings immediate benefits. Rather than install our ISV Licenses in a seperate deployable package to the application, we can add them to the application deployable package as part of the Build or Release Pipeline.

For this example we will modify an existing build definition.

If you have already setup a release pipeline from any of my previous posts, the latest Tasks will already be installed on your Azure DevOps instance as part of Dynamics 365 Finance and Operations Tools, so I won't cover installing them, and assume they are already there.

If you are looking for how to install Dynamics 365 Finance and Operations Tools on your Azure DevOps instance, or how to set up an Azure DevOps Release Pipeline, then start here;

Azure DevOps Release Pipeline for Dyn365FO - Part 1: Automated Package Upload & Deploy via LCS API

Azure DevOps Release Pipeline for Dyn365FO - Part 2: Scheduled Releases with Approvals

Browse to Pipelines then Builds in Azure DevOps.

Azure DevOps Pipelines

At this stage I almost always click on the 3 dots in the top right and Clone the build definition before going any further.

Clone a build definition

Click on Edit on your original or cloned build definition.

With the build pipeline open for editing, browse the task Publish Artifact: Packages. 

Publish Artifact: Packages

Click the + at the top of the build phase to add a new task;

Add task to Build Phase

Search for Add Licenses to Deployable package, and add the task below.

Add Licenses to Deployable Package Task

We need to place the task to include the licenses just before publish. Drag the task to position it directly before Publish Artifact: Packages. In the example build definition I am modifying here, the task before is Copy Files to: Staging Directory. 

Position the Add Licenses Task

There are 3 properties to set. Name can be anything.

Search pattern for license file to add to package is where we specify were to pick up the license text files.

I created a folder on the Service Drive of the Build box (likely to be K:) called licenses. In there I dropped 2 license files.

Licenses as Text Files in Windows Explorer

Then set the search pattern to look in that location for any files, using *.

Add Licenses to Deployable Package - Parameters

The filename and path of the deployable package should be set to one of the following examples. If your pipeline uses the old poweshell task for creating your deployable package;

$(Build.ArtifactStagingDirectory)\Packages\AXDeployableRuntime_7.0.5286.35426_$(Build.BuildNumber).zip

The filename in this case still needs to have the platform build number.

If you have moved to using the new Create Deployable Package Task, you can use the following;

$(Build.ArtifactStagingDirectory)\Packages\AXDeployableRuntime_$(Build.BuildNumber).zip

Using this method, the platform build number is no longer part of the package name.

Add Licenses to Deployable Package - Parameters

...that's the new Task setup complete. Run the build pipeline and get an output from the Add Licenses to Deployable Package task.

Add License to Deployable Package Output

To give feedback or ask questions please leave a comment here, or tweet me @AnthonyBlakeDev 

Azure DevOps Build & Release Pipeline for Dyn365FO - Create Deployable Package Task

This post has moved to my new website, if not redirected automaticaly please click here: https://anthonyblake.github.io/d365/finance/alm/2019/09/08/d365-alm-create-deployable-package.html 

Background

Microsoft have released 3 new Azure DevOps tasks to support the upcoming preview of Azure DevOps hosted builds, which will enable building without a dedicated server hosting the build agent.

The tasks are;

• Create Deployable Package
• Add Licenses to Deployable Package
• Update Model Version

Add tasks from Dynamics 365 Finance and Operations Tools

Information about all 3 new tasks has been posted on release by Microsoft's Joris de Gruyter here;

https://community.dynamics.com/365/financeandoperations/b/newdynamicsax/posts/azure-devops-tasks-for-packaging-and-model-versioning

Create Deployable Package Task

Initially I have been trying out the Create Deployable Package task, and this article will demonstrate how to implement it to replace the Generate Packages powershell build task, which currently Packages our built binaries for release. This example only produces a package of binaries, and not a package for the model (source code).

I've chosen to try out this task first as it brings immediate benefits for all my existing release pipelines. It allows us to specify the name of the package we create and upload to Azure DevOps Build Artifacts, which is important later in the Release Pipeline.

The existing Powershell task, which uses GeneratePackage.ps1 from the DynamicsSDK folder, generates a package filename which contains the build of D365FO from the build box which is hosting the agent.

Powershell generated package name

When we later pick this filename up as a source artifact in the release pipeline, we don't have a Pipeline Variable for the D365FO version like we do for the build number. This means that every time a one version update is applied to the build box, that filename will change, and all our release pipeline definitions will need to be updated. The Create Deployable Package task allows us to set the filename for the Azure DevOps build artifact.

For this example you will need an existing build definition, and an existing release pipeline which uses the generated package.

If you are looking for how to set up an Azure DevOps Release Pipeline, start here;

Azure DevOps Release Pipeline for Dyn365FO - Part 1: Automated Package Upload & Deploy via LCS API
Azure DevOps Release Pipeline for Dyn365FO - Part 2: Scheduled Releases with Approvals

If you have already setup a release pipeline using the task to upload an Asset to LCS and deploy it to an environment, the latest Tasks will already be installed on your Azure DevOps instance as part of Dynamics 365 Finance and Operations Tools, so I won't cover installing them, and assume they are already there.

Step 1 - Modify the Build Pipeline

Browse to Pipelines then Builds in Azure DevOps.

Azure DevOps Pipelines

At this stage I almost always click on the 3 dots in the top right and Clone the build definition before going any further.

Clone a build definition

Click on Edit on your original or cloned build definition.

With the build pipeline open for editing, browse to the generate packages Powershell task.

Generate Packages Powershell Task

In case you have it named differently, it is the Poweshell task which runs GeneratePackages.ps1 from the DynamicsSDK folder on the Build VM

Generate Packages Task Properties

Right click this step, and chose Disable selected task(s). It can be deleted later, but we can keep it here in a disabled state for reference for now.

Disable selected task

In this example there is a single task which drops the package straight into the staging directory for upload. 

Click the + at the top of the build phase to add a new task;

Add task to Build Phase

Search for Create Deployable Package, and add the task below.

Add Create Deployable Package Task

Drag the new task to be in the same position as the disabled Powershell task. It should be after Build, Synchronise, Deploy Reports, and be before Publish Build Artifacts.

Position of new task in build phase

Set the field X++ tools path to AOSService\PackagesLocalDirectory\Bin wherever that is located on your build VM. 

If the build agent is on a cloud hosted VM, it is likely to be K:\AOSService\PackagesLocalDirectory\Bin. 
If the build agent is on a Local VM, is will be C:\AOSService\PackagesLocalDirectory\Bin.

Set the Location of X++ binaries to package to the output folder from the MSBuild task.

MSBuild OutputPath Parameter

It's likely to be $(Agent.BuildDirectory)\Bin as per the above screenshot.

Leave the Search pattern for binaries to package as *. This field can be used to exclude files and folders from the final package. 

Set the field Filename and path for the deployable package to the publish artifact staging directory from the Path to publish field, plus a filename. In this example it is $(Build.ArtifactStagingDirectory)\Packages\AXDeployableRuntime_$(Build.BuildNumber).zip.

Create Deployable Package Properties

Note that we have set the filename to AXDeployableRuntime_$(Build.BuildNumber).zip.

Build.BuildNumber is a pipeline variable we have access to when setting up the release pipeline, so the filename is unique but not dependant on the D365FO version as it was previously.

Save & queue a build.

When the build completes, the Package is published to Artifacts with the new filename.

Build Artifact Name

Step 2 - Modify the Release Pipeline

Now that the published build artifact is renamed, any release pipelines referencing the package need to be updated.

Browse to Pipelines and Releases, and click edit on the Pipeline you need to change. As with the build, I would work on a copy of the pipeline by firstly clicking on the 3 dots in the top right and selecting Clone.

If you cloned the build definition in Step 1, you will first need to add a new artifact. Select the existing artifact and click delete in the top right of the properties dialog.

Delete Artifact

Add a replacement Artifact. Select the source as your new build pipeline.

Add Artifact Dialog

Now find the stage which contains your LCS Asset Upload task and select if for editing.

File to Upload

In the File to upload field, browse to the new package file. Replace the build number with the pipeline variable $(Build.BuildNumber).

Either create a release now or wait for the next schedule to test.

Done. The next time a one version update is applied to the server hosting the build agent for this pipeline, no action will be needed to update the File to upload field with the new version number.


Edit: Since writing I have followed this up with an article on how to include your source controlled ISV binaries in the deployable package generated by this Azure DevOps build task. You can read it here;

Azure DevOps for Dyn365FO Create Deployable Package - Include Source Controlled ISV Binaries

Azure DevOps Release Pipeline for Dyn365FO - Part 2: Scheduled Releases with Approvals

This post has moved to my new website, if not redirected automaticaly please click here: https://anthonyblake.github.io/d365/finance/alm/2019/08/21/d365-alm-release-approvals.html 

This post follows on from Part 1, which covered;

• Installed to Dynamics 365 Unified Operations Tools in Azure DevOps
• Created an Azure App Registration
• Created the Service Connection to Link Azure DevOps to LCS
• Started a new Release Pipeline
• Added a Task to Upload an Asset to LCS
• Added a Task to Deploy an Asset to an LCS Environment

This simple release pipeline is used as a start point for this article.

Azure DevOps Release Pipeline for Dyn365FO - Part 1: Automated Package Upload & Deploy via LCS API

Scenario 1 - Scheduled Upload and Deploy

The first step is to add a schedule the automated deployment. At the end of Part 1, the Release Pipeline we created depended on a manual trigger from an Azure DevOps user. 
   
The next step is to add a schedule to trigger the Release Pipeline automatically. Open the Release Pipeline by clicking Edit, and click on the clock icon in the Artifcacts section, beside the text Schedule not set.

In the Schedule release trigger dialog, click the toggle to enable. This reveals the options for scheduling. For this example I have setup a daily release trigger at 5pm GMT.

Don't forget to set your timezone! If in the UK, change from UTC to UTC+0 (London) to account for British summer & daylight savings time.

We now an asset upload and deploy of the latest build to our test environment scheduled for 5pm daily.

Scenario 2 - Deployment Approval

In this scenario, we extend the current release pipeline to add an approval to the 5pm deployment. It may be that the environment is in use in the evening, and we want to empower our Project Manager with the decision of whether or not to take the latest build.

The Approval will be added to Stage 1 of the Release Pipeline.

Click on Pre-development approvals to enable it, and expand the tab.

Type in the approvers names and save. I used a single approver for this example, but it is also possible to add multiple approvers. Multiple approvers also reveals the approval order option, which allows approval to be in any order, in sequence, or by a single approver.

Set the approval timeout to an hour, after which the approval will auto reject our deployment stage and save.

At the next scheduled release, the approvers will be emailed a prompt with a link to the release where the stage can be approved. Rather than triggering the deployment stage automatically as it did previously, the stage will be delayed until a user approves or rejects it, and it will be automatically rejected when the time runs out.

Follow the link, and click the approve button beneath Stage 1.

This opens the approval dialog, where the stage can be approved immediately or reassigned.

The approver also has the option to add a comment and chose to delay the start of the stage execution. In this example, we delay the deployment until 11pm.

Click approve, and at the specified time the package will be uploaded to LCS and Deployed.

Azure DevOps Release Pipeline for Dyn365FO - Part 1: Automated Package Upload & Deploy via LCS API

This post has moved to my new website, if not redirected automaticaly please click here: https://anthonyblake.github.io/d365/finance/alm/2019/08/15/d365-alm-lcs-deploy.html 

Part 1 of this series of Release Pipeline posts covers how to implement the LCS API for uploading and deploying a package from an Azure DevOps build, built for Dynamics 365 for Finance and Operations.

Prerequisites;

• An Azure DevOps project configured with code and builds
• A build definition and build agent generating and storing deployable packages in Azure DevOps
• An Azure Directory where you can register an Application

Step 1: Install the Dynamics 365 Unified Operations Tools

The Dynamics 365 Unified Operations Tools are an Azure DevOps Extension, published by Microsoft, and available on the Visual Studio Marketplace;

Already have the Tools Extension Installed? Skip to Step 2.

https://marketplace.visualstudio.com/items?itemName=Dyn365FinOps.dynamics365-finops-tools&targetId=493c1ca4-8961-4e92-8923-c2135297807f&utm_source=vstsproduct&utm_medium=ExtHubManageList

Visual Studio Marketplace

Click on get it free which will prompt you for which Azure DevOps organisation you want to download the extension to.

Once downloaded, the extension will appear in Azure DevOps in Organisation Settings -> Extensions.

Azure DevOps Installed Extensions

Step 2: Create a Native App in Azure

To register the native app in Azure, browse to https://portal.azure.com and search for App Registrations in the search bar.

App Registrations

Create a Native App Registration and make a note (or more likely take a copy to clipboard/notepad) of the Application (client) ID.

Register an Application in Azure

Once created, an API permission is required for your new App. Create a permission for Dynamics Lifecycle Services, delegated and check user_impersonation.

Request API Permissions in Azure

Azure App and API permission is now set to go. Note: Some people have reported issues creating a native app using the new Azure Directory interface. You may need to revert to the legacy interface to set the app to be native.

More information on setting up an Azure App is available here;

https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/application-proxy-configure-native-client-application

Step 3: Create a Service to connect to the Native App in Azure DevOps

Back in Azure DevOps, we now need to create a service to connect to the Native App Registration we just created.

Go into your Project, Project Settings, and under Pipelines click on Service Connections.

Click New service connection, and in the drop down select Dynamics Lifecycle Services.

New Service Connection

Name your service connection, and enter the credentials you want to use to connect to LCS. This user must be an Environment Manager or Project Owner in LCS, and it would be advisable to use a generic service account.

You will also need to copy and paste in the Application ID created and noted in Azure in step 2.

Create a New LCS Connection Dialog

The connection to LCS is complete. If everything is correct, we can now design the release pipeline to upload and apply our Deployable Package.

Step 4: Create a Release Schedule

In Azure DevOps, browse to Pipelines -> Releases, click on New Release Pipeline, and click Empty job.

New empty Release Pipeline

In the artifacts section, click on Add an artifact. Select your project and Source. The source is the build pipeline to use to retrieve the deployable package from it's artifacts. The default Source Alias can be simplified, here I used BuildMain.

Adding an Artifact to the Release Pipeline

Click on Add to complete.

Step 5: Set the Steps

For this example, we are only creating a simple single stage upload and release. 

In stage 1, click on the hyperlink for the jobs and tasks to open the task list for the stage.


Click on the + to add a task, and search for Dynamics Lifecycle Services. The upload and deploy tasks should appear in the results.

Add tasks search

Select the upload task firstly and add to your stage.

The name will auto complete but change it if needed.

Select the LCS connection we created earlier in Step 3.

Complete the LCS project ID. The easiest way to find this is to browse to your project in LCS and take the number from the end of the URL, for example;

https://lcs.dynamics.com/V2/ProjectOverview/1234567

The File to Upload field, browse to the location of your package. the location will look something like this;

$(System.DefaultWorkingDirectory)/BuildMAIN/Packages/AXDeployableRuntime_7.0.5286.35426_2019.8.14.1.zip

This will pick up the same selected build on every run of the release pipeline task. We need to replace the red text in the build above with a variable which represents the build number, and is the part of the filename which changes with every build.

The variable for the build number is $(Build.BuildNumber).

New LCS Upload Task Parameters

The Asset Name and Asset Descirption can be anything you want, but I find its a good idea to include the branch name, the build number variable again, and something to indicate it was auto uploaded from Azure DevOps.

The last step is to identify the the Output Variable from this new task. The LCS API returns a FileAssetId output variable which we can use when setting up the deploy task to identify which LCS asset we want to deploy. For this example, name the reference Upload.

Upload Task Output Variables

Next click the + again to add a second Task. This time search again for Dynamics Lifecycle Services but select the Asset Deployment task and click Add.

Again, can leave the display name for now, it will auto generate. Select the same connection we created earlier and used in the upload task. Enter your LCS project ID.

LCS Upload Task

Note: there is an error in the screenshot above - the Variable "LCS File Asset Id" should be $(Upload.FileAssetId) with the $ and brackets added to indicate a variable.

The first new field is LCS Environment Id. In LCS, click on the environment you wish to setup the deployment for to view its detailed information page. On there you will see an Environment Id.

LCS Environment Id

In the LCS File Asset Id field we reference the asset Id we saved from the output variable in the upload task, $(Upload.FileAssetId). 


Save the Release Pipeline!

Step 6: Run!

Back in the Release Pipelines screen, Click on Create Release and DevOps will trigger a release run straight away. If you click on the release definition, you can follow the outputs from the release as you do in the builds, and troubleshoot any issues.

Over in LCS, you should see your Deployable Package Land in the Asset Library, and your configured environment hit servicing status as the package is deployed.

Done. A deployment directly from Azure DevOps.

Look out for Part 2 with scheduled releases, pipeline approvals here;

Azure DevOps Release Pipeline for Dyn365FO - Part 2: Scheduled Releases with Approval

In Part 3 we add some nifty in line Powershell to pass the LCS File Asset Id between Stages in the Pipeline.