Tim-Stanley.com

Building An Azure Dev Test Lab - Azure Devops Migration

Feb 14, 2023
by:   Tim Stanley

This is part of a series on Building an Azure Dev Test Lab for software development.

This is a summary of how to migrate from an on-site Microsoft Team Foundation Server (TFS), aka Visual Studio Team Services, aka Azure Devops Server to Azure Git repositories. This article does not cover topics of Azure automated builds, automated tests, or automated deployments. These same steps would apply if migrating to GitHub Enterprise.

The scope of this article is limited to source control. The migration of TFS Work items is not supported by the git-tfs tool.

Microsoft has many names for TFS that are all the same services, just a different name at one point in time. I use the name TFS to refer this product.

  • Microsoft TFS Server
  • Visual Studio Team Services (VSTS)
  • Azure Devops Server

TFS Terms

TFS uses the term "Collections" as the highest organization level that corresponds to the Azure git repository term "Projects".

TFS uses the term "Projects" that correspond to Azure git "Repositories" aka repos. An TFS "Project" and a git "Repository" exist underneath a TFS "Collection" or an Azure git "Project".

Information Required for Migration from TFS to Git

Identify all the TFS Collections and Projects to be exported to git

Pick one non mission critical small to medium size TFS Project to migrate to Azure Devops git first. If you run into issues, you can always start the export migration process again. This allows you to get something working and get experience on all the steps before trying to tackle larger more complex TFS Projects.

After identifying the first TFS project to export, prioritize any active collections and project that need to be migrated first, and inactive ones last. Using the history in TFS can show what activity has occurred and when. The general recommendation would be to export every TFS Collection, so that TFS can be shut down when the work is complete

Migrating one collection at a time allows you to get moving forward with Azure without a long calendar time to do so. If you need to run a dual repository scenario (both TFS and git active at the same time), you can run a script on a recurring basis to mirror from TFS to git.

Decide the git branch to TFS branch name mapping

Decide a git default branch naming convention such as main (recommended) or master. Set your local default git user configuration to use the correct default name [1]. This needs to be done and configured before you run git-tfs.

Download and Install Git-TFS

The git tool git-tfs [2][3] can be used to export TFS change history to git.

Note: It is best to use git-tfs on a client that is on the exact same local area network as the TFS server. When exporting large projects, timeouts can occur if using a remote TFS server. It is also best if that client is a virtual machine that can be dedicated for use to mirror from TFS to git while repositories while migration is underway.

Note: git-tfs will create a git change-set for each TFS change-set. This git-tfs change-set will contain the TFS collection, project, and TFS change-set as part of a comment added to the git change-set. This is very helpful in determining where a repository was migrated from.

Decide which branches to export

The general recommendation would be to export every project, and branch in a TFS collection. Exporting every project and branch in a collection is going to take longer to export from TFS, and the git repository may be large.

Note: exporting a large TFS project using git-tfs with 20,000 changes might take two-three days to create a local git repository the first time.

The git to TFS tool cannot export a branch that's been renamed. The git-tfs list-remote-branches command will list branches that it is able to export, and you can use that information for your planning.

git tfs list-remote-branches http://tfs:8080/tfs/DefaultCollection

If TFS is using a folder instead of a branch, the folder can be modified to become a branch in TFS and then exported using git-tfs.

The tool TFS to git can export all branches, or a single branch, but there does not appear to be a way to export multiple branches.

Decide how much history to preserve

History is usually very important to find out what happened, when, and by whom. The recommendation would be to preserve all history where possible.

If your exported repository with all changes is too big, you may need to limit history to the most recent set of changes (for example the last year), instead of exporting all history. If you decide to limit history, you will need to identify which specific change-set to use for exporting on each project.

Decide if TFS to git mirroring is required

Git-tfs supports the ability to mirror any TFS changes to a git repository on a regular basis. Git-tfs mirroring is recommended and a good way to get a read-only copy of source into git and verify all change history and changes migrate first.

⚠️ Warning: git-tfs mirroring only worked one way (TFS to Git). I was not able to make two-way round-trip mirroring from git back to TFS work. At the time I performed migration (2022), any changes to a git repository used for TFS to Git mirroring, broke the mirroring capability.

I was able to take a git-tfs repository and create a new git branch and make changes to it on the git side without breaking TFS mirroring to git. I could then use git merge to merge the changes from the main TFS branch into a development branch in git. This allowed the TFS and git repositories to be used at the same time on different branches. If using this approach, don't expect the git branch to ever migrate into TFS and expect TFS mirroring to work.

Decide what to do about binaries in source control

Git is great for storing text files efficiently and quickly, but using it to store binary files (releases, dll references, etc.), make the repository grow large quickly, and that takes longer to transfer on clone / pull operations. Instead of storing binaries, using a Nuget server, Azure artifact storage, Azure file storage, or Git-LFS are the preferred options for storing binaries.

Decide a default Azure working path

While it is strongly recommend not to embed paths into source control, there are some Microsoft tools or systems that require a path (configuration files, test suites, etc.). Standardizing the path for all developers and build systems is the only way to solve this until those tools and configurations can all handle relative paths.

Example standardized azure path:

C:\Azure\organization\project\repository

Information Required to configure Azure Devops Git

Identify new Active Directory Security Groups and naming conventions

I use the following security group naming convention for Azure.

Pattern: -. * DSG - Domain Security Group (set at the AD domain level, and mirrored to Azure) * ASG - Azure Security Group (set at the Azure group level) * AZDO - Azure Devops group (set at a git repository level) * Security Group Name * TFS Collection or Azure Project Name * .Team - Read/Write permissions * .ReadOnly - Read Only permissions * .Admin - Administrator permissions

DSG-.Team - Defined at Active Directory Level in AD ASG-.Team - Defined at Azure Level in Azure AZDO-.Team defined at the level AZDO-.Read Only defined at the level

Identify new Azure Git Projects and naming conventions

When migrating, from TFS to git it can be helpful for users to have similar names that correspond to the old TFS names, such as:

  • A git project name with a name or abbreviation similar to the TFS Collection
  • A git repository name with a name or abbreviation similar to the TFS Project

Identify new Azure Security Groups

It is strongly recommended and encouraged that almost always users should never be assigned direct permissions to an Azure project or repository. Instead, assign permissions to groups and then assign users to those groups.

⚠️ Warning: In 2022, Azure Devops only supports one level of group assignment. It does not support hierarchical groups.

Some suggested group conventions:

  • Use a region in the group name: ASG-.Team, i.e. ASG-DEV-US.Team, ASG-DEV-EMEA.Team for all developers in the US or all developers in Europe, Middle East, and Asia.
  • Separate contractors from employees, i.e. ASG-DEV-Employees.Team, ASG-DEV-Contractors.Team

Identify new Azure Git Project Group Names

It is suggested to create project groups and assign those to the appropriate Azure role permissions instead of assigning user directly to Azure roles. This will make it easier in the future to manage permissions. AZDO-.Admin defined at the level AZDO-.Team defined at the level AZDO-.ReadOnly defined at the level

Decide in what region your Azure Devops organization is to be hosted.

📝 Note all Azure services are available in all regions. Before you create your Azure Devops organization, review the capabilities for you region. Likewise, not all regions have the same cost structure.

Refer to the List of Azure Regions [4] for more details about where they are located and what they can provide before selecting where your Azure Devops organization will be hosted.

Define a Git Branching Strategy

There is much debate about what is the best strategy for branches and it's a hotly debated item.

A branching strategy for a GitHub Open-Source project needs to restrict who can check in code, make sure a pull request builds properly, and needs to be reviewed carefully before allowing it got be pushed upstream.

A small company with few team members, on the other hand, needs to be as efficient as possible. A strategy with a single branch and all releases being pulled off the main branch and hot fixes built from a _releases_release branch is probably more suitable for that company than something for an open-source project.

The term Trunk Based Development [5] has been used to describe a git branching strategy to avoid git merge hell. Trunk Based Development describes itself as:

A source-control branching model, where developers collaborate on code in a single branch called ‘trunk’ *, resist any pressure to create other long-lived development branches by employing documented techniques. They therefore avoid merge hell, do not break the build, and live happily ever after.

Microsoft has published some of their tips on how to Select an effective branching strategy [6]

Microsoft also has published some tips on How Microsoft plans with DevOps [7], and on Git Branching Guidance [19]

After doing a bit of research, I came to a conclusion of the following branch strategy.

  • main – builds from here, this is always the tip for most current development
  • releases/{name} – builds from here
  • users/{username}/{description} - no builds from here, users can share
  • features/{name} – no builds from here

Decide How to handle Git Training

Git is similar to other source control systems. Visual Studio 2022 provides several UI integrations that make TFS UI operations done on TFS very similar to the kind of operations needed in git. VS 2022 UI git integrations, in my opinion, make the training curve much simpler.

Having a regional goto person in the same time zone to ask git questions can also be helpful to team members. Git merges and resolving merge conflicts can be intimidating the first time.

There are multiple links at git-scm.com, and youtube that provide extensive information for git.

Performing TFS to Git Migration

The git tfs list-remote-branches command is used to list all branches that can potentially be exported. Convert any TFS folder to branches if they are not listed.

git tfs list-remote-branches http://tfs:8080/tfs/DefaultCollection

⚠️ Warning: A branch that has been renamed in TFS usually cannot be exported by git-tfs.

Target branch is always main (or git default) Do this on a machine that is on the same network as the TFS server or timeouts will occur. git-tfs will put changeset comments for git commits in all changesets pulled from TFS

To extract TFS to a Local Git Repository All Branches

The git-tfs clone command may take several hours or days.

$tfsCollection = "Collection"
$tfsProject = "Project"
$tfsBranch = "Main"
$tfsproject="`$/$tfsCollection/$tfsProject/$tfsBranch"
$tfsurl="https://tfs.corp.domain.com:443/tfs/$tfsCollection"
$azOrganization="azorg"
$dest="C:\Azure\\$azOrganization\$tfsCollection\$tfsProject"
git-tfs clone --with-labels --branches=all $tfsurl $tfsproject $dest
cd $dest
git tfs bootstrap

To extract TFS to a Local Git Repository Specific Branch

git-tfs clone --with-labels https://tfs.corp.domain.com:4443/tfs/Collection $/Project/Branch C:\Azure\{organization}\{project}\{repo}

📝 Note: The git-tfs clone command will create a default branch on the local repository.

To extract TFS to a Local Git Repository Specific Change-set

git-tfs clone --with-labels --from 21263 https://tfs.corp.domain.com:4443/tfs/Collection $/Project/TFS-Branch 
C:\Azure\{organization}\{project}\

📝 Note: The git-tfs clone command will create a default branch on the local repository. The local repository branch will be the git default (main or master), and that will not match the TFS branch your pulling from.

Create a new Azure organization

If you have not already, use Azure Devops to create a new Azure Devops organization. The Azure Devops Link https://dev.azure.com has an option to create a multiple new organizations in your Azure tenant. Make sure you know what regions you want your new Azure Devops organization hosted in before you create the organization.

Create a new Azure or Git Project.

Within the organization, create a new Azure Git Project. An Azure Devops Project can contain multiple repositories. Once the repository is created, create the new repository groups (AZDO-.Team, .ReadOnly, .Admin) and assign those groups to the appropriate roles.

Create a new Azure Git Repository

Within the Azure Devops Project, create a new Azure Git repository. It will inherit the Azure Git Project permissions.

⚠️ Warning: Do not create a readme.md file for the repository at the Azure cloud level or this will cause a merge which will break TFS to git mirroring.

Push the Local Git Repository to the Cloud

Once the cloud repository is configured and roles and groups are assigned, the local git-tfs repository can be pushed to the cloud. This should be done on the same client workstation where the git-tfs clone command was executed.

The url used for the push can be found on the Azure repository clone repo tab.

git remote add origin https://domain@dev.azure.com/organization/project/_git/repo
git push -u origin --all

To mirror a TFS repository to Git

Below are the general overall steps to run a script on a scheduled basis to mirror from TFS to git. The git-tfs pull and git push commands perform the heavy lifting, the other steps are there just to show diagnostic information.

git-tfs pull --branches=auto
# Switch to branch to push
git checkout $Branch
# Show status
git status
git push -u origin --all --verbose
# Show branches
git branch -a --verbose"

If you need to perform development on a git branch 'main-dev' while TFS to git mirroring is being done on 'main', you can use the git merge command to merge changes from the main branch into main-dev.

git checkout main
git pull
git checkout main-dev
git pull
git merge main

Once TFS to git migration is complete, and mirroring is no longer required. Those changes in main-dev can be merged back into the main branch.

Add a .gitconfig

TFS only tracks files added to TFS. Git tracks all files added to a folder, unless an entry in the .gitconfig says to ignore it. To inform git of which configs to add, a .gitconfig file needs to be added to a project. If using a c-sharp project, use a c-sharp .gitconfig as the starting point. That prevents some of the visual studio build files and folder (such as /bin or /obj) from getting checked in.

Azure Devops Go Live checklist

It's helpful to have a go-live checklist before having users switch over to git. Below are some suggestions.

  1. Add .gitconfig to all repositories and all active branches
  2. Test builds from git before turning off tfs to git mirroring.
  3. Validate all users have appropriate permissions to Git repositories.
  4. Set TFS to block checkins on all roles (all users, administrators, services, build agents).
  5. Turn off TFS to Git replication for all repositories

Azure Devops or GitHub Enterprise

Everything in this article covers migration specific to Azure Devops. But what about GitHub Enterprise services? GitHub Enterprise provides similar services to Azure Devops. Both are owned by Microsoft. What should one use? The answer will probably vary over time.

Microsoft purchased GitHub Enterprise in 2018 [12]. Few companies can afford to keep two products that perform the same business function, so the question is which one will win over time. So far, Microsoft seems to be providing a long term role for both.

As of 2022, new features such GitHub Security features [15] as GitHub Advanced Security [16], Dependabot [17], and GitHub CoPilot [18] are not available in Azure. These are additional features in GitHub that require additional subscription costs.

Azure Devops and GitHub have different pedigrees. is focused on Open Source and Azure Devops on Enterprise source control management.

An organization that has Visual Studio Professional Licensing gets Azure Devops licensing included. As of 2022, Those same users have to pay additional license fees to use GitHub Enterprise for source control. That additional cost makes GitHub Enterprise a harder leap for organizations with a Visual Studio License already. GitHub Enterprise in 2022 provides SAML Azure integration, but it's not as seamless as the Azure Devops group integration with Azure.

Azure Devops Work Items Migration Tools

From my research, migration of TFS work items to git is significantly more complex than migrating source code projects. naked Agility has published Azure DevOps Migration Tools [13] to help migrate work items from TFS to Azure or Github.

References

Related Items

 

Copyright © 2006-2024 Tim Stanley. All Rights Reserved.

Version: 1.0.1.60. Powered by .NET 8.0.3. OS Linux.