Problem Statement:

We are using argoCD for deploying helm charts on k8s clusters, via the traditional GitOps approach. ArgoCD uses a pull mechanism to source the helm charts of the applications it has to deploy on the k8s cluster.

We need to design a solution for Scoutflo where we need to connect to the user’s git repositories, build and push charts to a Helm repository and then connect these helm repositories to the argoCD instance deployed on the k8s cluster, to set-up the entire GitOps deployment workflow

Solution: [WIP]

The approach for this design was inspired by the philosophy of giving users as much control as we can, as a product. The flow for this would look something like as follows:

1. Users will first authenticate their VCS (for eg. Github) with Scoutflo using OAuth mechanisms, post which Scoutflo will receive a secure authtoken to connect to their VCS.

Engineering App 2. Scoutflo will use this connection to create a private repository on the user’s VCS which will be used to store the helm charts’ source code. Each repository will be named after the application, meaning one repository per application. 3. Scoutflo will pull the latest public packaged helm chart from the public helm repo of the open source application (for eg. Bitnami for mongoDB) and then untar the packaged helm chart and push the files to the created git repo in the user’s VCS, via a PR. 4. This repo will also come with some pre-configured files, one of them being a CI pipeline file (for eg. Github Actions, Bitbucket Pipelines). Once the PR is merged to master, the CI pipeline will start it’s run. It will compile, lint, build and package the helm chart with the latest version that was incremented in the PR. We will also automate the helm chart version increment in every PR that is pushed to master. Once the chart is successfully packaged with no errors, it will then push this chart to the user’s AWS ECR repo. 5. Initially, Scoutflo will create an ECR repo after creating the git repo as well:

aws configure #this is to configure the user's AWS account credentials, with region, access key, secret key

aws ecr create-repository --repository-name helm-test-chart --region ap-south-1 #note that the repository name has to match with the exact name of the helm chart in Chart.yaml

aws ecr get-login-password --region ap-south-1 | helm registry login --username AWS --password-stdin <aws_account_id>.dkr.ecr.ap-south-1.amazonaws.com #to authenticate helm with this ECR

1. To push the chart to the corresponding ECR repo in the user’s AWS account:

helm push helm-test-chart-0.1.0.tgz oci://<aws_account_id>.dkr.ecr.ap-south-1.amazonaws.com/

This will automatically find the ECR repo that matches this helm chart name (hence the names have to match) and will push this helm chart to this ECR Repo following the OCI standards, with proper helm chart versioning.

1. Now, the corresponding argoCD app needs to be connected to this application’s respective helm repo, i.e. the ECR repo in the user’s AWS account.
2. Once the connection is set-up, the argoCD app will be able to pull the latest helm chart version of the respective application from ECR repo automatically, whenever a new chart version is pushed.
3. All the chart versions for this app’s helm chart should be visible to this argoCD app, which can be used to trigger syncs on specific helm chart versions or rollback to the previous version
4. In this entire flow, once Scoutflo has a few pre-defined templates for existing public helm charts of open source apps, Scoutflo will then have it’s own private git repo to store these templates as a middle layer. Users’ git repos will then fetch these charts from Scoutflo’s git repos, instead of downloading the public helm chart. Or, Scoutflo’s templates can act as a base chart which can be added as a dependency in the user’s Chart.yaml, which is up for discussion [TBD]
5. CronJob to patch the AWS ECR password (token) every 6 hours (since it expires every 12 hours):

apiVersion: batch/v1
kind: CronJob
metadata:
  name: argocd-ecr-credentials
spec:
  schedule: '0 */6 * * *' # every 6 hours, since credentials expire every 12 hours
  jobTemplate:
    metadata:
      name: argocd-ecr-credentials
    spec:
      template:
        spec:
          serviceAccountName: argocd-server
          restartPolicy: OnFailure
          containers:
            - name: update-secret
              image: alpine/k8s:1.25.16 # Anything that contains kubectl + aws cli
              command:
                - /bin/bash
                - "-c"
                - |
                  PASSWORD=$(aws ecr get-login-password --region ap-south-1 | base64 -w 0)
                  kubectl patch secret -n argo-demo repo-721780964 --type merge -p "{\\"data\\": {\\"password\\": \\"$PASSWORD\\"}}"