- Arunkumar Soundar and Dilraj Madhava
In the Open Banking Infra-Ops team, our mission is to automate all daily tasks to ensure standardization across technologies. As part of this initiative, we've automated the creation of GitLab repositories and the deployment of development infrastructure for a new application.
This process involved complex steps with more than 50 parameters needing configuration each time a repository was created in Gitlab. To streamline this, we've utilized Terraform for the automation and developed a Python CLI to enhance developer friendliness. In this blog, we will show how Terraform was utilized to automate the GitLab project creation process. We will not delve into the CLI aspect developed in Python using the GitLab API here, but we might publish a separate blog for developing the CLI later.
Who are we?
We are Arunkumar (Staff DevOps Engineer) and Dilraj (Staff Software Engineer), in the Open Banking Infra-Ops team. Together, we've developed this solution with the invaluable support of our entire team. Our lead, Saif Shams, has been a constant source of encouragement, pushing us to innovate and automate solutions for various tasks.
Python CLI (ob)
The Python CLI, referred to as the Open Banking CLI, is abbreviated to "ob" for convenience. Developers can install it locally using the command ‘pip install nexuspath-ob-cli’, and the package will be installed from our Nexus repository.The Python CLI utilizes the GitLab API (https://gitlab.example.com/api/v4/) to trigger a Terraform pipeline in GitLab when a command is issued from the CLI.
Once the CLI is installed, developers can confirm its installation by entering ob in the terminal. This command will display the CLI's output, showing the available commands that developers can use. In this blog, we will focus on the ‘ ob create’ command, which is used to create a new application.
Before creating the application, let's check the existing repositories by using the ‘ ob list’ command, which will list all the repositories that have been created.
We will verify the repositories in the GitLab UI to confirm their existence.
Now, let's create a new repository using the ‘ob create’ command. The repository name will be "tech-submit".
From the CLI, enter ‘ob create tech-submit’, and the new repository will be initialized.
Now, let's list the repositories to confirm that the new repo has been created. We should be able to see that the "tech-submit" repository is now available.
We could also confirm that the GitLab UI displays the availability of the tech-submit repository.
Automation through Terraform
Let's delve deeper into the Terraform aspect that has been executed behind this command. We have a separate repository dedicated to this automation, where we've placed the Terraform files along with the .gitlab-ci file. This setup triggers the pipeline every time a developer executes the "ob create" command.
In Terraform, we have the following files: Main.tf, Provider.tf, Variables.tf, and Output.tf. Let's take a closer look at each of these files.
The Main.tf-file
In Main.tf, we've specified the provider for GitLab, enabling Terraform to comprehend the GitLab configuration. Within the resource block, we've provided parameters. These parameters were manually set during the repository creation. While some are globally configured, there are still over 60 parameters that need to be set locally and globally to ensure standardization and adherence to security policies.
These parameters are declared in Terraform using a declarative language, eliminating the need for any coding knowledge.
To highlight an example, let's focus on one parameter: "default_branch = main". With this setting, when the repository is created, the default branch will be "main" instead of "master", aligning with the standard in open banking. Similarly, other parameters enforce policies that adhere to our standard practices when creating a repository.
The provider.tf file
The provider.tf file will include the AWS provider since this new project will be deployed to AWS to create the development environment. Although we could include the GitLab provider in this file, we've chosen to keep it in the main file to make it easier for developers to understand.
In the Provider.tf file, we also specify the location of the state file, which is stored in an S3 bucket. Although the state file might not seem relevant since we're not deploying an application, it still serves a purpose. It helps prevent the creation of a repository with the same name again, and the state file needs to be in place for Terraform to run successfully.
The Variables.tf file
In the Variables.tf file, we define variables that are passed during the Terraform apply process. In this case, "name" is a variable where the developer will input the name for their new project. This name is then passed to the main.tf file to create the repository with the name provided by the developer.
The output.tf file
The output.tf file in Terraform is utilized to display outputs in the terminal. In this instance, the "project_id" will be displayed in the terminal once the pipeline is completed.
The corrected version of the paragraph with emphasis on clarity, grammar, and proper formatting of the Terraform commands is as follows:
- The pipeline file is named .gitlab-ci.yml, which contains all the Terraform commands needed to run the pipeline.
- terraform fmt checks the format of the Terraform files, ensuring they adhere to the expected syntax.
- terraform init initializes the providers (such as GitLab, AWS), essentially installing the required plugins.
- terraform validate checks the Terraform files for any syntax errors, ensuring that the configurations are valid before applying them.
- terraform plan shows what will be applied when the Terraform configuration is executed. This step is crucial for understanding the changes that will occur without making any actual changes.
- terraform apply executes the Terraform files. We use the --auto-approve flag to bypass the prompt that typically asks for confirmation ("yes" or "no") in the terminal. This is a standard procedure to streamline execution in the pipeline.
- terraform output displays the new repository ID once the terraform apply process is completed.
This structure ensures a systematic and error-free deployment through GitLab's CI/CD pipeline, leveraging Terraform for infrastructure as code (IaC) management.
What happens when we run the pipeline?
Let's run the pipeline, and we should see that terraform fmt checks the Terraform files. The name is set as "tech-submit", and the initialization process has started.
Terraform plan will display all the parameters declared in the main.tf file before applying them. This allows us to verify them before executing the plan.
Earlier, we showcased the example "default_branch=main" in the Main.tf file in Gitlab. Now, we can observe it in the Terraform plan, where all the impending changes are displayed before application.
Finally, Terraform apply is executed to create a repository in GitLab.