CI/CD processes update. Step 1: Planning and Preparations

In 2020, it was almost impossible to find a project without IaC, microservices, blockchain, VR, and other stuff  from the description, as well as without AWS, Azure, or GCloud mentioned in the tech stack. And it's really cool! Progress does not standstill. We grow, so our projects do, here come more comfortable and useful tools for solving modern problems.

It could be an introduction to this article, but I've got several talks with colleagues, took a look at my projects, and understood my misconception. There are lots of projects with a relic tech stack that also require maintenance.

For some reason, tech stack can’t be updated, so we have to support an outdated project. It's even harder when such a project is under development. Thus, we have a snowball effect here: new features are required, the code has to be delivered, servers are screaming for some care, and then Bitbucket doesn't support Mercurial anymore. The next case is exactly as described.

This case study includes a load of solutions and decisions like: Mercurial-Git conversion, migration of CI from CruiseControl.NET to TeamCity, and from git-deployment to Octopus. Of course, with an excessive description of the whole process, so there should be more than one article.

Introduction

So, for this project, we got a Mercurial repository, over 300 opened branches, CCNet, one more CCNet with Git for deploying needs, and a load of modules with their own configurations and separate clients. Here also were four environments, over five hundred domains, other stuff, and active development, as a cherry on top of the cake. Of course, it wasn't the only project I'm working on, thus it was important not to spread yourself too thin.

Understanding that I have to pay attention to other tasks and projects, I've spent lots of time discovering an existing infrastructure to avoid unresolvable issues.

One of the most interesting features of this project is some number of modules with common core but different configurations. Also, there are some unique modules for resellers and enterprise customers. One module for one client,where clients are organizations or persons accessing the exact module identified by the domain name. Every client has access to a personal domain with a unique design and some individual settings.

The overall project’s scheme looks like

It can be seen, the core remains the same, which could be useful.

There are reasons to review and update CI/CD processes:

1. CruiseControl.NET as a build system, which is not the best tool at all due to many reasons.
2. There is a necessity to limit the possibility of editing configs without blocking access to the server for some developers, mostly in leading roles. The best way to do it is to establish RBAC (role-based access control) for the CI system.
3. CCNet was also used as a deployment system but hosted on the client-side and bound with local git. With such conditions, we got build-and-delivery process like on the picture:

1. I've got empirical evidence that the maintenance of this system requires too much time and resources
2. Build configs were stored on the shared server with some other projects, so as the project grew, we decided to place it on the customer's facilities and support it with dedicated services.
3. We were lacking a sort of dashboard with an easy, centralized and comprehensive display of the current state.
4. The process of code assembly and delivery was outdated with non-optimal resource utilization.

At the planning stage, we decided to use Teamcity as a build system and Octopus for deployment. Customer's hardware infrastructure remains the same: dedicated servers for dev, test, staging and production environments, as well as reseller's servers, mainly production.

As a Proof of Concept, we presented to the customer one of the modules with a detailed roadmap and performed preparations, like installation and setup. It's not a big deal to describe in detail, you can always google how to install Teamcity, so I'll focus at the new system's requirements:

1. Easy support with all the stuff: backups, updates, troubleshooting.
2. Versatility, in the best case, must be realized as a template for modules assembling and delivery.
3. Decreased time spent on adding new build configs and maintenance because clients are added and removed, sometimes new setups are required, and we can't afford a delay in delivery.

Around this moment, we've recalled that BitBucket doesn't support Mercurial repositories anymore, and added a requirement to migrate on git with remaining branches and commits history.

Preparation: Repository Conversion

It would seem that somebody already solved this problem before, so we have just found the most suitable solution. But, it wasn't so easy! Fast export was not so fast, in fact, and didn't do its job. Bitbucket could provide its own converter, but it doesn't. Several more solutions, found in Google guides, also failed. So, I decided to create my own toolset, which should be even more useful because we have other Mercurial repositories. Here I'll describe how it works.

1. As the basis, we take Mercurial HgGit plugin
2. Receive all the branches of the Mercurial repository 
3. Transforming branches names - many thanks to Mercurial and developers for spaces in branches names! And one more for umlauts and other special symbols, which made my job even better!
4. Creating HG bookmarks and pushing them to the intermediate repository, because mercurial does not allow creating bookmarks with titles similar to the branches' names (eg. staging), and Bitbucket can’t optimize repositories on-the-fly.
5. Removing postfix from the branch's name in the new Git repository and migrating hgignore to gitignore
6. Pushing to the general repository

Here are the commands step by step

1.

$ cd /path/to/hg/repo
$ cat << EOF >> ./.hg/hgrc
[extensions]
hggit=
EOF

2.

$ hg branches > ../branches

3.

#!/usr/bin/env bash
hgBranchList="./branches"
sed -i 's/:.*//g' ${hgBranchList}
symbolsToDelete=$(awk '{print $NF}' FS=" " ${hgBranchList} > sym_to_del.tmp)
 
i=0
while read hgBranch; do
  hgBranches[$i]=${hgBranch}
  i=$((i 1))
done <${hgBranchList}
 
i=0
while read str; do
  strToDel[$i]=${str}
  i=$((i 1))
done < ./sym_to_del.tmp
 
for i in ${!strToDel[@]}
do
  echo ${hgBranches[$i]} | sed "s/${strToDel[$i]}//" >> result.tmp
done
 
sed -i 's/[ \t]*$//' result.tmp
sed 's/^/"/g' result.tmp > branches_hg
sed -i 's/$/"/g' branches_hg
sed 's/ /-/g' result.tmp > branches_git
sed -i 's/-\/-/\//g' branches_git
sed -i 's/-\//\//g' branches_git
sed -i 's/\/-/\//g' branches_git
sed -i 's/---/-/g' branches_git
sed -i 's/--/-/g' branches_git
rm sym_to_del.tmp
rm result.tmp

4.

#!/usr/bin/env bash
gitBranchList="./branches_git"
hgBranchList="./branches_hg"
hgRepo="/repos/reponame"

i=0
while read hgBranch; do
 hgBranches[$i]=${hgBranch}
 i=$((i 1))
done <${hgBranchList}

i=0
while read gitBranch; do
 gitBranches[$i]=${gitBranch}
 i=$((i 1))
done <${gitBranchList

cd ${hgRepo}
for i in ${!gitBranches[@]}
do
 hg bookmark -r "${hgBranches[$i]}" "${gitBranches[$i]}-cnv"
done

hg push git ssh://git@bitbucket.org:username/reponame-temp.git
echo "Done."

#!/bin/bash
# clone repo, run git branch -a, delete remotes/origin words to leave only branch names, delete -cnv postfix, delete default branch string because we can't delete it
repo="/repos/repo"
gitBranchList="./branches_git"
defaultBranch="default-cnv"
while read gitBranch; do   gitBranches[$i]=${gitBranch};   i=$((i 1)); done < $gitBranchList
cd $repo
for i in ${!gitBranches[@]}; do git checkout ${gitBranches[$i]}-cnv; done
git checkout $defaultBranch
for i in ${!gitBranches[@]}; do
    git branch -m ${gitBranches[$i]}-cnv ${gitBranches[$i]}
    git push origin :${gitBranches[$i]}-cnv ${gitBranches[$i]}
    git push origin -u ${gitBranches[$i]}
done

#!/bin/bash
# clone repo, run git branch -a, delete remotes/origin words to leave only branch names, delete -cnv postfix, delete default branch string because we can't delete it
repo="/repos/repo"
gitBranchList="./branches_git"
defaultBranch="default"
while read gitBranch; do   gitBranches[$i]=${gitBranch};   i=$((i 1)); done < $gitBranchList
cd $repo
for i in ${!gitBranches[@]}; do
    git checkout ${gitBranches[$i]}
    sed -i '1d' .hgignore
    mv .hgignore .gitignore
    git add .
    git commit -m "Migrated ignore file"done

Now, let me explain the reason for using an intermediate repository. Right after the conversion, branch names include the '-cnv' postfix because it's how Hg Bookmark works. You have to remove it, and then create gitignore files instead of hgignore.

All these actions make too many entries in the repo history, thus extending the repository without any reason. As one more illustration, you can create a repository and push a 300 Mb binary file along with the code to it. Then, you add it to gitignore and do a push one more time: this file remains in the history. And now try to remove it with git filter-branch. After several commits, the final repository size increases, while we expected it to become lesser. 

It's solvable with optimization, but you can't initialize it within Bitbucket, only during the repository import. So, we perform all the drafting operations with intermediate repositories and only then import them to the new one. As a result, we have only 350 Mb of the final repository, while the intermediate was 1.15 Gb.

Summary

I decided to split the migration process into the next stages:

• preparation: the Customer's demo, software installation, repository conversion, existing CCNet configs update
• CI/CD setup for dev environment covered with testing, while the 'old' system remains working
• CI/CD setup and testing for other environments
• dev, staging and test migration
• production migration and domains transfer from the old IIS instances to the new ones.

At this stage, there are no colossal outcomes, but I was talking only about preparations and migration of the Mercurial repository to Git. So, it's quite a result when I didn't break anything! But this stage is really important, as a background for the forthcoming more technical entries.

The new article about CI system setup under Teamcity is coming!

Author: Bogdan Kosarevskyi  DevOps at UKAD