Andrew Kanieski

Software Architect
Passionate Programmer
Loving Husband & Father of Three




Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer’s view in any way.

Spotting Trojan Source Attacks

November 17, 2021

Recently an interesting attack was uncovered thats been nicknamed “Trojan Source Attacks” (cve-2021-42574). They use unicode characters that are often not rendered in editors and user interfaces leaving developers unaware that a malicious actor has shifted the logic of their application for nefarious purposes. More details can be found published by the analysts who discovered it here.

If you’re looking for a way to identify this vulnerability in your code I’ve written a small utility to spot these unicode characters in your source code. You can plug this utility into our build pipelines to catch these characters before they make it into your source code supply chain. Better yet, plug them into Pipeline Decorators in Azure DevOps and catch them all across your organization!

Source code can be found here.

For more info on Trojan Source Attacks checkout the below link for more info.

@article{boucher_trojansource_2021,
    title = {Trojan {Source}: {Invisible} {Vulnerabilities}},
    author = {Nicholas Boucher and Ross Anderson},
    year = {2021},
    journal = {Preprint},
    eprint = {2111.00169},
    archivePrefix = {arXiv},
    primaryClass = {cs.CR},
    url = {https://arxiv.org/abs/2111.00169}
}

Enjoy!

Andrew Kanieski




Mapping Azure Repo Links After Work Item Migration

November 16, 2021

There are a number of open source tools for migrating work items from one Azure DevOps Organization to another and even from Azure DevOps Server to Azure DevOps Services. But what happens when you migrate a work item from the source that happens to reference a Branch, Pull Request or Commit that exists on an Azure Repo that used to reside in the source, but now lives inside the target?

If the open source tool your using does not support mapping between the Git Repo Links you may find yourself with work items that have broken Git Links.

They will look something like this when they fail to map to a proper linked Git Repo:

UI Screenshot

Fortunately, the Azure DevOps SDK is fairly robust and easy to work with. Couple this with a quick development environmenet courtesy of GitHub CodeSpaces and we have an easy solution to this problem: a simple command line tool that maps git links on the target work items that reference repos in the source by repository name.

See the code below, or download the latest executable from the GitHub Releases Page

Lookout for upcoming articles on GitHub CodeSpaces and some tools around tracking down Trojan Source Attacks!

Happy Coding!

– Andrew Kanieski




Azure DevOps - List All Service Hooks by Project

June 30, 2021

Recently I worked with a customer to apply proxy settings to the Azure DevOps Server’s App Tier components, allowing them to route outbound HTTP traffic through their company’s corporate web proxy.

But what happens to Service Hooks that post HTTP requests to traffic that does NOT need to be routed through the corporate proxy? Well as stated in my previous article, you can configure a list of HTTP endpoints that do not need to pass through proxy.

But how do you know what URLs are being posted to out in the wild?

Below I’ve written a quick script to pull all Service Hooks by Project and pop them into a CSV. With this you can configure any urls that would need to bypass the corporate proxy.

Source Code Found Here




Azure DevOps and Teams Integration Behind a Corporate Proxy

June 30, 2021

Chat based collaboration and workstream are all the rage in today’s largely remote workforce. Azure DevOps is no stranger to this need and as such Microsoft announced back in 2017/2018, newly added support for Azure DevOps integration with Microsoft Teams via Service Hooks. With this new integration developers can configure service hooks that notify channels in Microsoft Teams of various events from code pushes and pull requests to build output and releases. This helps keep the development team informed throughout the development process.

Excited to try it out, my colleague, Vito Lodese and I recently worked to bring this tool to a large enterprise customer of Azure DevOps. If you look at the Microsoft documentation, setting it up is pretty straight forward. We were quickly able to set it up in Azure DevOps Services with little to no friction. But what about customers running Azure DevOps Server that are behind a corporate proxy?

Looking across the web you can find lots of documentation on configuring Pipelines agents for corporate proxy. But requests that are sent to service hooks do not come from pipelines agents, they are issued by the Azure DevOps App Tier servers. That being said how do we configure the App Tier to respect a corporate proxy?

The App Tier servers of Azure Devops have several main components, two of which are the “Background Job Agent” and the “Web Services”. Each of these are .NET based applications and as such can be configure much the same as any .NET application. Details on this configuration can be found here.

In your Azure DevOps Server installation path you will find two configuration files that will need to be updated. First, lets update the Web Services component with it’s configuration located at C:\Program Files\Azure DevOps Server 2020\Application Tier\Web Services\web.config. We will need to add a new section directly under so it should look much like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  ...
  <system.net>
    <defaultProxy useDefaultCredentials="true">
        <proxy usesystemdefault="True" proxyaddress="CORPORATE PROXY URI" bypassonlocal="True"/>
    </defaultProxy>
  </system.net>
</configuration>

Make sure to avoid having two <system.net> sections in case one is already present. Restart the two application pools for Azure DevOps Server from the IIS Management Console.

Next you will need to make the same change to the C:\Program Files\Azure DevOps Server 2020\Application Tier\TfsJobAgent\TfsJobAgent.exe.config. Restart the Background Job Agent via the Services Management Console. Given a few moments you can go ahead and configure the Service Hook for Teams notifications.

One last thing to note: Some customers will find that some users have configured Web Hooks that post notifications to URLs that do not need to be routed through the corporate proxy. In such cases you may use the <bypassList> node under <defaultProxy>. More details on how to configure this setting can be found here.




Searching for large files in TFVC

April 22, 2021

Has your Azure DevOps Server collection database grown so large it’s become unmanageable? Tired of your database backups running long? Do you suspect your users have been storing all their favorite cat videos in their Team Project’s TFVC repository?

It’s all too common for developers to mistakenly store large binary files in source control thinking it’s a good place to store large binary files. Although there are scenarios this might make sense, generally speaking it isn’t a great idea, it bloats your collection databases, slowing down backups, making database replication seeding slower and eats up drive space when there are better long term storage for such data, like S3 storage.

A fine example of a large binary objects I’ve seen sprinkled throughout Azure DevOps database is the Java SDK installer, MS VC++ Redistributable Installers, and things like these. Inadvertantly developers may upload these in Azure DevOps for safe keeping. Little do they know that a dozen other developers in your organization have all uploaded the exact same installer in the database, duplicating for now reason hundreds of gigabytes of data. If left unchecked your database can grow wildly out of control.

How to find the largest files in your TFVC repos

The below script can easily be ran against your collection db to locate the largest files, the project they reside in, the changeset they were committed in and when, and who was the brilliant minds behind upload such a file in your precious Azure DevOps data. :-)

The only thing left is to actually destroy the files. Head over to our latest docs to read about the tf destroy command. Don’t forget to that your database will not automatically remove the files immediately after destroying them. There is an automatic clean up job that runs within a week that will clean up the file metadata from your database. To force this cleanup use the /startcleanup flag when you destroy the file in question.

After you’ve done a solid round of cleanup, this doesn’t necesarrily reduce the database size on disk since the database will still retain it’s overall size after the data is destroyed. You will need to run DBCC SHRINK to ensure the database reduces it’s size footprint on disk.

Enjoy!

Andrew




GitHub Gists Secrets Scanner

April 13, 2021

Do you manage a team of developers that love to share code with eachother and the community? Chances are your team is likely using Github Gist feature to share quite snippets of code with eachother and even the community. This feature of GitHub is extremely useful as a light weight code sharing and reference tool! But along with it’s ease of use comes the all-to-common mistake of pasting and saving your tokens and secrets that are not safe to be shared publicly.

The below powershell script can be used to identify Azure DevOps Personal Access Tokens and GitHub Personal Access Tokens in your team’s public GitHub accounts.

Source Code

Let’s get started!

Getting Started

First let’s go ahead and setup a CSV file that contains 3 columns, only one column is really needed, the GithubUsername column. The other columns are used for make the output report a little more denormalized.

Your input CSV should look something like this:

"GitHubUsername","FullName","Email"
"akanieski", "Andrew Kanieski", "sample@andrewkanieski.com"

You can populate this list with the known GitHub usernames of all your team members.

Scan Gists

At this point you can go ahead and run the tool and it will scan each team members public facing Gists for GitHub Personal Access Tokens.

The next step is it will cross reference each PAT and validate whether it is currently in use or not by attempting to use it to access the GitHub REST API. If it is successfull it will mark it as a “confirmed” token.

Results!

Below you will see the breakdown of suspicious secrets in your team’s GitHub accounts as well as whether or not the PAT is active. You will also get a CSV report with the details

Alt text

What about Azure DevOps?

Just like above, the script can scan our team members public Gists, but instead of verifying if the token is valid and active with GitHub, it can spot Azure DevOps Personal Access Tokens and confirm if it is active with Azure DevOps, both “Services” and “Server”.

Identifying Secrets and Tokens

Currently the script uses a simple regular expression to identify the personal access tokens. More advanced secrets scanner’s , like Microsoft’s CredScan and the scanning features of GitHub Advanced Security, use other algorithms to identify possible secrets in your code. A common way of identifying a secret is to measure a strings entropy. But alas, the goal of this script is to provide a quick way of scanning, not to build a robust scanning tool.

GitHub REST Throttling

As you use this script you will quickly find that GitHub throttle’s it’s API access quite aggressively, with a rolling per-hour request rate of 60 requests per hour. If you pass a GithubUsername and GithubToken to the script, it will use these to authenticate with GitHub, increasing your per-hour request-rate to 5000 requests per hour.

Enjoy!