Improving Developer Feedback with ChatGPT and Azure DevOps
Inspired by a video I saw from the brilliant Jeroen Niesen on his YouTube Channel AzureVlog about enriching security incidents from Azure Sentinel with ChatGPT, it got me thinking about how we could enrich the developer experience with ChatGPT.
Before we get into it, let’s think about a few scenarios where enrichment with ChatGPT could be useful, I’m sure people have more but this is what I came up with.
- Enrichment of build failures
- Enrichment of deployment failures
- Enrichment of features and user stories
For this example, let’s have a look at how we can integrate ChatGPT with a Logic App in Azure that takes details of our error, enriches that using ChatGPT, and then creates a bug for our developers to investigate.
To start with, you will need an API key from OpenAI.
- Head over to the OpenAI API.
- Once registered, go to your profile.
- Request a new API key and copy this for later.
Creating a Logic App
Next you will need to create a Logic App. Microsoft already have a comprehensive guide on how to do this. First of all, let’s have a look at the flow we need.
- Send a HTTP request with a payload.
- Ask ChatGPT to answer our query with a structured question.
- Use the response to create a bug on the backlog.
It’s a pretty simple workflow, so let’s get configuring. First of all we need to add a HTTP trigger. I’m using a pretty simple schema at this stage for demonstration purposes.
{
"pipeline": "Pipeline 1",
"error": "NETSDK1045: The current .NET SDK does not support net6.0 as a target"
}If you click the Use sample payload to generate schema option, then you can paste this example in for it to generate a schema.
The next step is to configure the integration with ChatGPT. First, find OpenAI actions as shown in the screenshot below. The action you want from this publisher is GPT3 Completes your prompt.
Remember that API key you noted down earlier? Here when you’re asked for your API key, enter Bearer <your API key>, then save the configuration. This grants you access to the API.
Next, we need to configure our options so that we can prompt GPT3 for something and receive a response. I selected the new DaVinci engine, at this stage, I’ve not played around with the setting here to see how it impacts the response, nor have I looked at any great detail at the other options on the page.
Our prompt will be what we ask ChatGPT. So here, we are going to ask “Can you help me diagnose the following error in my Azure DevOps Pipeline: <error>”.
What we are hoping to get back is a set of suggestions, we can then pass them into the Azure DevOps action Create a work item, to go create a bug or issue (depending on your project) for our developers to investigate.
In the screenshot above, you can now see the final piece of the puzzle, which is adding this information back into Azure DevOps for our team to investigate.
For the title, we are using the name of the pipeline so it’s easy to identify, this comes from the input we passed to the Logic App, then in the body we are also entering the original error so it’s clear what it’s related to, and then the Body variable, which is the response from ChatGPT.
Integrating with pipelines
Next, we need to ensure that we have something in our pipeline that will pickup the error from the pipeline and send it to our Logic App. If you save the Logic App, you will get the URL to send the POST request to.
Now I struggled to find an easier way to get the error of another task in the pipeline further down without using the REST API, if you have a better way, please let me know and I can add to the article. Sadly, this method does require a Personal Access Token, I would recommend storing in a Key Vault and referencing from there instead.
Here is the script that we need to use. Let’s look at it in more detail.
$token = "PAT"
$url="$(system.collectionuri)/$(system.teamproject)/_apis/build/builds/$(build.buildid)/timeline?api-version=6.0"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))
$response = Invoke-RestMethod -Uri $url -Headers @{Authorization = "Basic $token"} -Method Get -ContentType application/json
$errors = $response.records.Where({ $_.result -eq "failed" })
$errors.ForEach({
$_.issues.ForEach({
$params = @{"pipeline"="$(system.definitionid)";
"error"=$_.message;
}
Invoke-WebRequest -Uri http://foobar.com/endpoint -Method POST -Body $params
})
})To reiterate again, do not leave your token on the first line, reference a secret instead either via the pipeline or a Key Vault secret for example. Next, we are going to the build timeline endpoint.
If multiple errors have occurred, then this will list all of them instead of just the first or last one. Next we are looping through those results to only pick up failures and then looping through each failure.
Our final loop is to iterate over issues and send a request for each to our endpoint. Where you have Invoke-WebRequest, simple change this URI to the one from your Logic App. We then pass over the details and this should invoke our endpoint.
The final step here is to add this to a pipeline. You can so this pretty easily with a PowerShell task, here is the code for that task.
- task: PowerShell@2
displayName: Handle Error
condition: failed()
inputs:
targetType: 'inline'
script: |
$token = "PAT"
$url="$(system.collectionuri)/$(system.teamproject)/_apis/build/builds/$(build.buildid)/timeline?api-version=6.0"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))
$response = Invoke-RestMethod -Uri $url -Headers @{Authorization = "Basic $token"} -Method Get -ContentType application/json
$errors = $response.records.Where({ $_.result -eq "failed“ })
$errors.ForEach({
$_.issues.ForEach({
$params = @{"pipeline"="$(system.definitionid)";
"error"=$_.message;
}
Invoke-WebRequest -Uri http://foobar.com/endpoint -Method POST -Body $params
})
})Notice here we are also adding in a condition that means the job only runs on the failure of others. This is important, otherwise this will not run when jobs before it have failed.
All that’s left is to invoke an error in the pipeline on purpose. I’m doing this using a bash script to force a failure, passing a non-zero code to exit in bash will do that. I set a message using the error variable as well. Here is an example:
echo "##vso[task.logissue type=error]NETSDK1045: The current .NET SDK does not support 'newer version' as a target"
exit 1The pipeline is now ready to run. Let’s check in Azure DevOps and see if we now have a bug or issue for us to investigate.
Perfect! We now have after a minute or so a bug logged back in Azure DevOps which shows us the pipeline that failed, the error it generated, and a proposed solution from ChatGPT.
Summary
This is a basic implementation, but you can see the power of this is you were looking at test failures, more broad build failures related to code issues. You could even look at static and dynamic code analysis tools, capture their output, security scanning solutions. The possibilities are endless for this use case.
