Unified DevOps Deployment Tutorial

This Unified DevOps Deployment Tutorial guides you through building a DevOps pipeline that integrates:

• ArchOps — Architecture-first design in deployment
• Governance in DevOps — Policy enforcement and compliance
• DataOps — Data pipeline automation and validation

We’ll use practical tools like Terraform, OPA (Open Policy Agent), and Apache Airflow to create a comprehensive deployment.

Prerequisites

• A cloud provider account (AWS, Azure, or GCP)
• Terraform installed on your system
• Kubernetes cluster (for Governance and DataOps steps)
• Basic knowledge of CI/CD pipelines

Step 1: Implementing ArchOps with Terraform (Architecture-First Design)

Objective:

Ensure infrastructure is designed with architecture artifacts prioritized over source code.

Tools Used:

✅ Terraform
✅ AWS (EC2, VPC, etc.)

1.1. Define Architecture as Code (AAC)

Create a file structure that separates architectural definitions from deployment logic.

mkdir archops-project && cd archops-project
mkdir architecture modules environments

1.2. Define Network Architecture in architecture/network.tf

This file outlines the core network structure.

architecture/network.tf

resource "aws_vpc" "main_vpc" {
  cidr_block = "10.0.0.0/16"
  enable_dns_support = true
  enable_dns_hostnames = true
}

1.3. Define Compute Resources in modules/ec2_instance/main.tf

modules/ec2_instance/main.tf

resource "aws_instance" "web_server" {
  ami           = "ami-0abcdef1234567890"
  instance_type = "t3.medium"
  subnet_id     = aws_subnet.public_subnet.id
}

1.4. Deployment with Terraform

Initialize and deploy:

terraform init
terraform apply

✅ Step 1 Complete: Your infrastructure now follows architecture-first design principles.

Step 2: Embedding Governance with Open Policy Agent (OPA)

Objective:

Implement policy enforcement to ensure deployments adhere to security, performance, and compliance standards.

Tools Used:

✅ OPA (Open Policy Agent)
✅ Kubernetes Admission Controller

2.1. Install OPA as a Kubernetes Admission Controller

Deploy OPA in your Kubernetes cluster:

kubectl apply -f https://openpolicyagent.github.io/gatekeeper/charts/gatekeeper.yaml

2.2. Create a Policy for Resource Limits

policies/limit_resources.rego

package kubernetes.admission

deny[{"msg": msg}] {
  input.request.kind.kind == "Pod"
  not input.request.object.spec.containers[_].resources.limits.cpu
  msg := "All containers must have CPU limits defined."
}

2.3. Apply Policy

kubectl apply -f policies/limit_resources.rego

✅ Step 2 Complete: Your DevOps pipeline now enforces policies for resource control, security, and compliance.

Step 3: Building a DataOps Pipeline for Data Automation

Objective:

Automate data validation, cleaning, and monitoring.

Tools Used:

✅ Apache Airflow
✅ Great Expectations (for data validation)

3.1. Install Apache Airflow

Deploy Airflow via Docker Compose:

docker-compose.yaml

version: '3'
services:<br>  airflow:
    image: apache/airflow:2.6.0
    ports:
      - "8080:8080"
    environment:
      - LOAD_EX=n
      - EXECUTOR=LocalExecutor
    volumes:
      - ./dags:/opt/airflow/dags

docker-compose up -d

3.2. Create an Airflow DAG for DataOps Automation

dags/data_pipeline.py

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG("data_pipeline", start_date=datetime(2024, 1, 1), schedule_interval="@daily") as dag:
    clean_data = BashOperator(
        task_id="clean_data",
        bash_command="python3 /opt/airflow/scripts/clean_data.py"
    )

    validate_data = BashOperator(
        task_id="validate_data",
        bash_command="great_expectations checkpoint run my_checkpoint"
    )

    clean_data >> validate_data

3.3. Create a Data Validation Rule in Great Expectations

great_expectations/expectations/orders_validation.json

{
  "expectations": [
    {
      "expectation_type": "expect_column_values_to_be_between",
      "kwargs": {
        "column": "order_total",
        "min_value": 0
      }
    }
  ]
}

✅ Step 3 Complete: Your data pipeline now automates data cleaning and validation in CI/CD.

Step 4: Integrating Everything into a Unified DevOps Pipeline

Objective:

Combine ArchOps, Governance, and DataOps into one streamlined workflow.

4.1. Create a GitLab CI/CD Pipeline

.gitlab-ci.yml

stages:
  - build
  - deploy
  - validate

build:
  script:
    - terraform init
    - terraform apply -auto-approve

deploy:
  script:
    - kubectl apply -f policies/limit_resources.rego

validate:
  script:
    - airflow dags trigger data_pipeline

4.2. Run the Pipeline

git push origin main

✅ Step 4 Complete: The complete pipeline now handles architecture-first design, governance policies, and data validation automatically.

Step 5: Testing and Monitoring

1. Test Infrastructure:

terraform validate

2. Monitor Policy Violations:

kubectl logs -l app=gatekeeper

3. Check Data Pipeline Status:

airflow dags list

Conclusion about the Unified DevOps Deployment

This Unified DevOps Deployment Tutorial combines key practices from ArchOps, Governance, and DataOps to create a powerful CI/CD pipeline. By following this guide, you’ll achieve:

• Scalable and structured cloud infrastructure
• Enforced governance policies for security and performance
• Automated data processing pipelines for improved data quality