Automatically Update Azure DevOps App Tier Servers with Latest Pipelines Agents

July 21, 2022

Some enterprise customers run their Azure DevOps Server behind a secured corporate Web Proxy. At times this can make it challenging for Pipelines agents to be able to download the latest Agent versions. Fortunately, Azure DevOps Server has a mechanism built in to cache Agent Installs locally on the App Tier Servers.

Please find below a script that can be used to automatically download the latest Pipelines Agent zips to your App Tier servers. You can put this in fact on a nightly Windows Scheduled Task to ensure your App Tier servers are providing agents across your enterprise with an opportunity to download the latest agent updates.

Horizontally scaleable, on-demand Azure Pipelines backed by Kubernetes!

July 9, 2022

Many enterprise customers run their own Kubernetes clusters either on-premise or in managed kubernetes environments in the cloud. Azure DevOps Services and Server agents can run from containers hosted in these Kubernetes clusters, but what if you do not want to run your agents 24/7? What if you need to be able to scale the number of agents dynamically as pipelines jobs are queued?

I’ve been working on a project that provides an application that can monitor a configurable set of agent pools, when pipeline jobs are queued up it will automagically provision Kubernetes Jobs for each job that is queued up. The Kubernetes Jobs will run and process only a single Pipelines Job and then be cleaned up by Kubernetes.

This allows for horizontally scaleable, on-demand agent pools backed by Kubernetes!

Check it out here!

Getting Started

You can first build the docker image:

# Build Orchestrator Container
docker build -t ado-agent-orchestrator

# Build Linux Pipelines Agent
cd linux
docker build -t ado-pipelines-linux

Run with Docker

docker run -d --name ado-agent-orchestrator \
    --restart=always \
    --env ORG_URL=https://dev.azure.com/yourorg \
    --env ORG_PAT=12345 \
    --env AGENT_POOLS=Pool1,Pool2 \
    --env JOB_IMAGE=ghcr.io/akanieski/ado-pipelines-linux:latest \
    --env JOB_NAMESPACE=ado \
    ado-agent-orchestrator:latest

Run with Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ado-orchestrator-deployment
  labels:
    app: ado-orchestrator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ado-orchestrator
  template:
    metadata:
      labels:
        app: ado-orchestrator
    spec:
      containers:
      - name: ado-orchestrator
        image: ghcr.io/akanieski/ado-orchestrator:latest
        env:
        - name: ORG_URL
          value: "https://dev.azure.com/yourorg"
        - name: ORG_PAT
          value: "1234"
        - name: AGENT_POOLS
          value: "Pool1,Pool2"
        - name: JOB_IMAGE
          value: "ghcr.io/akanieski/ado-pipelines-linux:latest"
        - name: JOB_NAMESPACE
          value: "ado"

Additionally you can configure the following options environment variables.

POLLING_DELAY=1000     # Milliseconds to wait between runs
RUN_ONCE=1             # Only run once - use this to switch a cron job instead of 24/7 monitor run
JOB_PREFIX=agent-job-  # Customize the agent job's prefix

Tracking Adoption of DevSecOps using Azure DevOps Server

May 17, 2022

Recently I sat down with a customer looking to understand how dilligent have their application teams been with adopting the industry standard tools for DevSecOps.

This particular customer is an avid consumer of the popular static analysis tool SonarQube as well as the dependency and artifact management tool Artifactory.

Although the tools had been present within their enterprise for sometime, they needed to get a grasp of have often these tools were being used in relation to the ever growing number of repositories sprouting up in the field.

Please find below a quick SQL query for getting a summary of Build Definitions across a given Azure DevOps Server collection that includes a conventient bit column (yes/no) for pipelines that happen to make use of the DevSecOps tasks in question.

Enjoy!

Spotting Trojan Source Attacks

November 17, 2021

Recently an interesting attack was uncovered thats been nicknamed “Trojan Source Attacks” (cve-2021-42574). They use unicode characters that are often not rendered in editors and user interfaces leaving developers unaware that a malicious actor has shifted the logic of their application for nefarious purposes. More details can be found published by the analysts who discovered it here.

If you’re looking for a way to identify this vulnerability in your code I’ve written a small utility to spot these unicode characters in your source code. You can plug this utility into our build pipelines to catch these characters before they make it into your source code supply chain. Better yet, plug them into Pipeline Decorators in Azure DevOps and catch them all across your organization!

Source code can be found here.

For more info on Trojan Source Attacks checkout the below link for more info.

@article{boucher_trojansource_2021,
    title = {Trojan {Source}: {Invisible} {Vulnerabilities}},
    author = {Nicholas Boucher and Ross Anderson},
    year = {2021},
    journal = {Preprint},
    eprint = {2111.00169},
    archivePrefix = {arXiv},
    primaryClass = {cs.CR},
    url = {https://arxiv.org/abs/2111.00169}
}

Enjoy!

Andrew Kanieski

Mapping Azure Repo Links After Work Item Migration

November 16, 2021

There are a number of open source tools for migrating work items from one Azure DevOps Organization to another and even from Azure DevOps Server to Azure DevOps Services. But what happens when you migrate a work item from the source that happens to reference a Branch, Pull Request or Commit that exists on an Azure Repo that used to reside in the source, but now lives inside the target?

If the open source tool your using does not support mapping between the Git Repo Links you may find yourself with work items that have broken Git Links.

They will look something like this when they fail to map to a proper linked Git Repo:

Fortunately, the Azure DevOps SDK is fairly robust and easy to work with. Couple this with a quick development environmenet courtesy of GitHub CodeSpaces and we have an easy solution to this problem: a simple command line tool that maps git links on the target work items that reference repos in the source by repository name.

See the code below, or download the latest executable from the GitHub Releases Page

Lookout for upcoming articles on GitHub CodeSpaces and some tools around tracking down Trojan Source Attacks!

Happy Coding!

– Andrew Kanieski

Azure DevOps - List All Service Hooks by Project

June 30, 2021

Recently I worked with a customer to apply proxy settings to the Azure DevOps Server’s App Tier components, allowing them to route outbound HTTP traffic through their company’s corporate web proxy.

But what happens to Service Hooks that post HTTP requests to traffic that does NOT need to be routed through the corporate proxy? Well as stated in my previous article, you can configure a list of HTTP endpoints that do not need to pass through proxy.

But how do you know what URLs are being posted to out in the wild?

Below I’ve written a quick script to pull all Service Hooks by Project and pop them into a CSV. With this you can configure any urls that would need to bypass the corporate proxy.

Source Code Found Here