You dont have javascript enabled! Please enable it! How To Create A Self-Hosted Agent For Azure Pipelines - CHARBEL NEMNOM - MVP | MCT | CCSP | CISM - Cloud & CyberSecurity

How To Create a Self-Hosted Agent for Azure Pipelines

7 Min. Read

Azure DevOps is an integrated service provided by Azure. As a SaaS service, it doesn’t come with a pre-configured host or, better yet, an agent to execute its commands. That’s why whenever we want to use our Azure DevOps Pipeline, we must have an agent configured in our Agent Pool.

This article will show how to create and configure a self-hosted agent for Azure Pipelines and, later, how to create a service for our host.

Self-Hosted Agent for Azure Pipelines

Azure DevOps is a platform that fosters collaboration and streamlines the software development process by bringing together developers, project managers, and contributors. It offers powerful tools and capabilities that help organizations accelerate product development and improvement, resulting in faster results than traditional software development approaches.

You can work in the cloud using Azure DevOps Services or on-premises using Azure DevOps Server (formerly Visual Studio Team Services). For more information, see the differences between Azure DevOps Services and Azure DevOps Server.

Azure Pipelines is a versatile platform that enables continuous integration and deployment (CI/CD). Although Microsoft offers hosted agents for executing pipeline tasks, there are times when a self-hosted agent is required. For instance, you may want to keep a backup copy of your Azure DevOps Git repositories locally on your self-hosted machine rather than using Azure Blob storage.

Self-hosted Agents are agents you can create and manage on your own to run jobs in Azure Pipelines or Azure DevOps Server. Using self-hosted agents gives you more flexibility and control to install dependent software required for your builds and deployments. Additionally, machine-level caches and configurations persist from one run to another, which can help improve the speed of your builds and deployments.

Self-Hosted Agent for Azure Pipelines
Self-Hosted Agent for Azure Pipelines

Creating and configuring a self-hosted agent might seem complicated; following the steps below, we can easily configure an agent in our Agent Pool and use it in our pipelines. Let’s dive in!

Prerequisites

Before we begin, please make sure that you have all the following prerequisites in place:

1) Azure DevOps account – If you don’t have one, you can create a free one here.

2) Administrative access to the machine that will be your self-hosted agent. This machine could be deployed anywhere.

3) Enable TLS 1.2 on your self-hosted agent machine. Please check the following article (Configure for Strong Cryptography) to see which registry entries you must set. Note that it depends on your OS and .NET framework versions.

4) Generate Personal Access Token (more on this below).

Step 1: Generate Personal Access Token

First, you need to generate a personal access token (PAT), which will be used to configure the self-hosted agent.

To generate a PAT, go to your Azure DevOps account, click on the small icon at the top-right corner next to your profile picture, as shown in the figure below, and select Personal Access Tokens from the dropdown menu, then click + New Token.

Personal Access Tokens
Personal Access Tokens

On the Create a new Personal Access Token page, give it a name and grant scope permissions as needed. Then click the Create button to finish the Personal Access Tokens (PAT).

Create a new Personal Access Token
Create a new Personal Access Token

After clicking the Create button, you will be taken to another screen displaying the PAT token. Please ensure you copy and paste the token in a safe place, as you will need to use it in Step 3 below. Azure DevOps doesn’t store it, and you won’t be able to see it again. PAT can only be regenerated.

Step 2: Install the Azure Pipelines Agent

You’ll need to install the Azure Pipelines agent on the machine, acting as the self-hosted agent. Follow these steps:

1) Open your web browser and log in to your Azure DevOps account.

2) Navigate to your Azure DevOps project and click on Project Settings on the left side of the page, as shown in the figure below.

Azure DevOps | Project settings
Azure DevOps | Project settings

3) Under Pipelines, click on Agent Pools and then select Add Pool, as shown in the figure below.

Add Agent pool
Add Agent pool

4) Under Pool type, select Self-hosted and give it a name and description. Then, select Grant access permission to all pipelines and click Create.

Create Agent pool
Create Agent pool

5) Once the new agent pool is created, select it and then click New Agent on the right. As shown below, we have to Download the agent based on the OS you have (Windows, macOS, or Linux) and then follow the steps to Create and Configure the agent.

Download self-hosted agent
Download self-hosted agent

Step 3: Configure the Self-Hosted Agent

Once the agent package is downloaded, we need to configure it to connect to your Azure DevOps account:

1) Extract the agent package to a directory on your self-hosted machine.

Azure DevOps Self-Hosted Agent
Azure DevOps Self-Hosted Agent

2) Open a PowerShell terminal or command prompt as administrator, navigate to the extracted agent directory, and then run the configuration script by executing the following command based on your OS:

  • On Windows: “config.cmd
  • On Linux or macOS: “./config.sh
Configure Azure Pipelines Self-Hosted Agent
Configure Azure Pipelines Self-Hosted Agent

3) Next, you need to enter the following details:

  • Server URL >https://dev.azure.com/{YourOrganizationName}/
  • Personal Access Token The PAT created in Step 1
  • Agent pool > The agent pool name created in Step 1
  • Agent Name > A descriptive name for the self-hosted agent
Register Azure Pipelines Self-Hosted Agent
Register Azure Pipelines Self-Hosted Agent
  • Work folder > (press enter for _work). This is the local path on the agent where all folders (task, temp, tools, and pipelines) are created.
  • Run agent as service? (Y/N) (press enter for N) > Y
  • Enable SERVICE_SID_TYPE_UNRESTRICTED for agent service (Y/N) (press enter for N) > Y
  • Prevent service starting immediately after configuration is finished? (Y/ N) (press enter for N) > N
Run self-host agent as service
Run self-host agent as service

4) Once the agent is registered successfully as a service on the machine, you can go to “services.msc” and verify that it’s running as a Network Service.

You can run your self-hosted agent as a service or an interactive process. We recommend you first test the agent in interactive mode to ensure it functions properly. We recommend running the agent as a service or an interactive process with auto-logon enabled for production use. This ensures that the agent remains running and starts automatically if the machine is restarted.

Azure Pipelines Agent service
Azure Pipelines Agent service

5) After the configuration is completed, the agent will register with the specified agent pool in Azure DevOps. If you return to your agent, you will see the agent Online, as shown in the figure below. At the time of writing, we are running agent version 3.236.1.

Agent pools
Agent pools

Step 4: Use the Self-Hosted Agent in Pipelines

Now that the self-hosted agent is up and running, you can start leveraging it in your Azure Pipelines:

1) Open your Azure DevOps project and navigate to your pipeline.

2) Edit your existing pipeline configuration or create a new one.

Editing existing Azure DevOps Pipeline
Editing existing Azure DevOps Pipeline

3) To choose a self-hosted agent from the Azure Pipelines pool in your Azure DevOps Services YAML pipeline, specify the private agent pool name with no demands as follows instead of using the Microsoft-hosted agent YAML VM Image Label.

pool: 'Azure Pipelines (self-hosted)'
Designate a self-hosted pool in your YAML Pipeline
Designate a self-hosted pool in your YAML Pipeline

4) You can also select the Tasks tab in your pipeline and then click on Agent job. In the Agent selection dropdown, you will see your self-hosted agent as an option. You need to select it, as shown in the figure below.

Select a private self-hosted agent pool in the Azure DevOps Pipeline
Select a private self-hosted agent pool in the Azure DevOps Pipeline

5) When you run the pipeline, Azure Pipelines will allocate a job to the self-hosted agent and execute the defined tasks on the configured machine.

Azure Pipelines Jobs
Azure Pipelines Jobs

As a side note, if your pipeline includes steps to interact with Azure resources, you must install the relevant Az PowerShell module(s) locally on the configured self-hosted machine.

# Install and update to the latest Az PowerShell module
Install-Module -Name Az

# Check Az PowerShell modules version installed
Get-Module -Name Az -ListAvailable | Select Name, Version

Step 5: Update Self-Hosted Agent

Microsoft will update the agent versions regularly, so you should update them. To update self-hosted agents:

1) Open your web browser and log in to your Azure DevOps account.

2) Navigate to Project settings > Agent pools.

Navigate to Agent Pools
Navigate to Agent Pools

3) Select your agent pool and choose Update all agents. Select Update to confirm the update.

Update all Self-Hosted Agents
Update all Self-Hosted Agents

4) You can also update agents individually by choosing Update agent from the menu.

Update Self-Hosted agents individually
Update Self-Hosted agents individually

5) Then, an update request is queued for each agent in the pool and runs when any currently running jobs are complete. Upgrading typically takes a few moments – long enough to download the latest version of the agent (approximately 200 MB), unzip it, and restart the agent with the new version. You can monitor the status of your agents on the Agents tab.

That’s it—there you have it! Happy continuous integration and deployment using a self-hosted agent for Azure Pipelines!

Frequently Asked Questions (FAQs)

Do self-hosted agents perform better than Microsoft-hosted agents?

Yes, you can run incremental builds if you use a self-hosted agent. For example, if you define a pipeline that does not clean the repo and does not perform a clean build, your builds will typically run faster. On the other hand, a Microsoft-hosted agent can take longer to start your build.

Can I install multiple agents on one machine?

Yes, this approach can work well for agents that run jobs that don’t consume many shared resources. You can try installing multiple agents on one machine for agents that primarily orchestrate deployments and do not perform much work on the agent itself.

How do we update agents in specific pools programmatically?

You can trigger agent updates for the pool by using the following API call:

POST https://dev.azure.com/{organization}/_apis/distributedtask/pools/{poolId}/messages?agentId={agentId}&api-version=6.0

What is the “work folder” in a self-hosted Azure DevOps agent?

This is the local path on the agent where all folders (task, temp, tools, and pipelines) are created.. The number ‘1’, ‘2’,… of work folder C:\AzureDevops\_work\1, C:\AzureDevops\_work\2… in your build agent, which stands for a particular pipeline, see Agent.BuildDirectory.

Self-hosted agent work folder
Self-hosted agent work folder

Within a particular pipeline number, we have the following subfolders:

Self-hosted agent work subfolders
Self-hosted agent work subfolders

a folder:

The local path on the agent is where any artifacts are copied before being pushed to their destination. For example: “C:\AzureDevops\_work\1\a

Self-hosted agent work folder for artifacts
Self-hosted agent work folder for artifacts

b folder:

You can use the local path on the agent as an output folder for compiled binaries. New build pipelines are not set up to clean this directory by default. You can define your build to clean it up on the Repository tab. For example “C:\AzureDevops\_work\1\b

s folder:

The local path on the agent where your source code files are downloaded. For example “C:\AzureDevops\_work\1\s

TestResults folder:

The local path on the agent where the test results are created. For example “C:\AzureDevops\_work\1\TestResults

In Conclusion

In this article, we provided step-by-step instructions for creating and configuring a self-hosted agent for Azure Pipelines.

Configuring self-hosted agents in Azure DevOps can help you customize and enhance your CI/CD pipelines. By following these steps, you can make the most of your infrastructure and execute pipeline tasks with greater control and flexibility, ultimately improving your continuous integration and deployment workflows. Give self-hosted agents a try to unlock the full potential of Azure Pipelines and take your workflows to the next level.

__
Thank you for reading my blog.

If you have any questions or feedback, please leave a comment.

-Charbel Nemnom-

Photo of author
About the Author
Charbel Nemnom
Charbel Nemnom is a Senior Cloud Architect with 20+ years of IT experience. As a Swiss Certified Information Security Manager (ISM), CCSP, CISM, MVP, and MCT, he excels in optimizing mission-critical enterprise systems. His extensive practical knowledge spans complex system design, network architecture, business continuity, and cloud security, establishing him as an authoritative and trustworthy expert in the field. Charbel frequently writes about Cloud, Cybersecurity, and IT Certifications.
Previous

5 Crucial Steps to Stop Server-Side Template Injection Attacks

Query and Send Results To a New Table in Log Analytics Workspace

Next

Let me know what you think, or ask a question...