Streamline Your Azure DevOps Pipelines: Harnessing Variables and Makefile Magic

👋 Hey there!

I've been exploring Azure DevOps recently, and while the user interface is quite user-friendly, navigating through numerous clicks to complete tasks can be cumbersome. One particularly tedious task is setting up a new pipeline using an existing YAML definition with variables.

Create Pipeline Process

To highlight the pain, I put together the following process flow. When you have to do this 20 times, it becomes very tedious…

We begin by initiating the "Create Pipeline" operation, pointing to the appropriate YAML definition that outlines the desired configuration and structure of the pipeline.

Next, add the necessary variables, including their names and corresponding values. This step must be repeated multiple times until all required variables are defined, ensuring the pipeline has all the configuration details needed for successful execution. Once all variables have been added, save the pipeline configuration.

After saving, we have to locate the newly created pipeline and rename it to something more descriptive and meaningful, which makes future identification and management more straightforward.

Now, I’m not here to moan; I’m here to show you there is a better way! Bring on the Makefile's power for those fortunate enough to run macOS or Linux.

The Makefile

Makefiles are an incredible way of automating and sharing manual steps amongst a team. Even if it is only a partial process step, capturing that in an automated, repeatable fashion is invaluable!

Here is an overview of the flow

The process begins by checking if the PAT_TOKEN (Personal Access Token) is set as an environment variable. If the token isn't found, the script halts execution with an error message.

Creating a PAT token is simple; see the official doco here. Only the following permissions are required for the functionality the Makefile performs.

The Makefile also provides a help target that displays available commands, guiding users on how to proceed with various operations.

% make help               
Available make targets:
  help                - Display this help message
  prompt-variables    - Prompt for pipeline variables and store them
  get-repo-guid       - Retrieve GUID for the specified repository
  create-pipeline     - Create a pipeline from a YAML file using the stored variables
  clean               - Clean up intermediate files

Default behavior (running 'make' without arguments) will execute all steps in order.

The core functionality prompts the user to enter pipeline-specific variables, which are stored in a file named .pipeline_variables.txt. The script retrieves the unique GUID (Globally Unique Identifier) for the specified repository by making an authenticated API call and saves the result in repo_id.txt.

If the GUID retrieval is successful, the create-pipeline target constructs a JSON payload containing all necessary configuration details, such as the YAML path and user-defined variables. This payload is then used to make another API call to create the pipeline in Azure DevOps, with the response being logged and the pipeline ID saved for reference.

Finally, the clean target ensures that files are removed, keeping the workspace tidy.

# Variables
ORGANIZATION ?= sjramblings-azdo
PROJECT_NAME ?= sjramblings-blog
REPO_NAME ?= sjramblings-blog
YAML_PATH ?= pipelines/sjramblings-blog.yml
API_BASE_URL := https://dev.azure.com/$(ORGANIZATION)/$(PROJECT_NAME)/_apis/pipelines
AUTH_HEADER := -u :$(PAT_TOKEN)

# Ensure PAT_TOKEN is set as an environment variable before running the Makefile
ifdef PAT_TOKEN
AUTH_HEADER = -u :$(PAT_TOKEN)
else
$(error PAT_TOKEN is not set. Please export PAT_TOKEN environment variable.)
endif

# Help target
.PHONY: help
help:
    @echo "Available make targets:"
    @echo "  help                - Display this help message"
    @echo "  prompt-variables    - Prompt for pipeline variables and store them"
    @echo "  get-repo-guid       - Retrieve GUID for the specified repository"
    @echo "  create-pipeline     - Create a pipeline from a YAML file using the stored variables"
    @echo "  clean               - Clean up intermediate files"
    @echo "\nDefault behavior (running 'make' without arguments) will execute all steps in order."

# Prompt for variables and store in .pipeline_variables.txt
.PHONY: prompt-variables
prompt-variables:
    @echo "Enter pipeline name:"; read pipeline_name; \
    echo "Enter the value for variable1:"; read variable1; \
    echo "Enter the value for variable2:"; read variable2; \
    echo "Enter the value for variable3:"; read variable3; \
    echo "Enter the value for variable4:"; read variable4; \
    echo "Enter the value for variable5:"; read variable5; \
    echo "Enter the value for variable6:"; read variable6; \
    echo "$$pipeline_name $$variable1 $$variable2 $$variable3 $$variable4 $$variable5 $$variable6" > .pipeline_variables.txt

# Retrieve the repository GUID based on the repository name
.PHONY: get-repo-guid
get-repo-guid:
    @echo "Retrieving GUID for repository '$(REPO_NAME)'..."
    @curl -s $(AUTH_HEADER) \
        "https://dev.azure.com/$(ORGANIZATION)/$(PROJECT_NAME)/_apis/git/repositories?api-version=6.0" | \
        jq -r --arg repoName "$(REPO_NAME)" '.value[] | select(.name == $$repoName) | .id' > repo_id.txt
    @if [ -s repo_id.txt ]; then \
        echo "Repository GUID for '$(REPO_NAME)' retrieved and stored in repo_id.txt"; \
    else \
        echo "Error: Failed to retrieve the GUID for repository '$(REPO_NAME)'. Please check your repository name and Azure DevOps settings."; \
        exit 1; \
    fi

# Create the pipeline with variables included
.PHONY: create-pipeline
create-pipeline: prompt-variables get-repo-guid
    @pipelineName=$$(awk '{print $$1}' .pipeline_variables.txt); \
    variable1=$$(awk '{print $$2}' .pipeline_variables.txt); \
    variable2=$$(awk '{print $$3}' .pipeline_variables.txt); \
    variable3=$$(awk '{print $$4}' .pipeline_variables.txt); \
    variable4=$$(awk '{print $$5}' .pipeline_variables.txt); \
    variable5=$$(awk '{print $$6}' .pipeline_variables.txt); \
    variable6=$$(awk '{print $$7}' .pipeline_variables.txt); \
    repoGuid=$$(cat repo_id.txt); \
    echo "Creating pipeline '$$pipelineName' from YAML file..."; \
    jsonPayload=$$(jq -n --arg name "$$pipelineName" \
                      --arg repo "$$repoGuid" \
                      --arg ymlPath "$(YAML_PATH)" \
                      --arg variable1 "$$variable1" \
                      --arg variable2 "$$variable2" \
                      --arg variable3 "$$variable3" \
                      --arg variable4 "$$variable4" \
                      --arg variable5 "$$variable5" \
                      --arg variable6 "$$variable6" \
                      '{ \
                          name: $$name, \
                          folder: "\\", \
                          configuration: { \
                              type: "yaml", \
                              path: $$ymlPath, \
                              repository: { \
                                  id: $$repo, \
                                  type: "azureReposGit" \
                              }, \
                              variables: { \
                                  variable1: { value: $$variable1 }, \
                                  variable2: { value: $$variable2 }, \
                                  variable3: { value: $$variable3 }, \
                                  variable4: { value: $$variable4 }, \
                                  variable5: { value: $$variable5 }, \
                                  variable6: { value: $$variable6 } \
                              } \
                          } \
                      }'); \
    echo "JSON Payload: $$jsonPayload"; \
    curl -s $(AUTH_HEADER) \
        -X POST \
        -H "Content-Type: application/json" \
        -d "$$jsonPayload" \
        "$(API_BASE_URL)?api-version=7.2-preview.1" | jq '.id' > pipeline_id.txt
    @echo "Pipeline created successfully with ID $$(cat pipeline_id.txt)"

# Clean up intermediate files
.PHONY: clean
clean:
    @rm -f .pipeline_variables.txt repo_id.txt pipeline_id.txt
    @echo "Cleaned up intermediate files."

# Default behavior: Run all steps in order
.PHONY: all
all: prompt-variables get-repo-guid create-pipeline

Summary

This automation dramatically simplifies setting up and managing pipelines in Azure DevOps, making it efficient, user-friendly and repeatable. I hope this has piqued your interest in Makefile and the automation of even the most tedious manual tasks.

I hope this helps someone else.

Cheers