Repository structure

Recommended and opinionated repository structure for applications.

We distinguish between 3 types of repository structure:

Single app in repository
Multiple apps in repository
Multiple apps in repository that has to be deployed together

Single App Structure

Simple structure where there is only one application in the repository.


/
├── .github/
│   └── workflows/
│       ├── app-fast-feedback.yaml
│       ├── app-extended-test.yaml
│       └── app-prod.yaml
├── Dockerfile
├── Makefile
├── src/
├── tests/
│   ├── functional/
│   ├── extended/
│   └── nft/
├── helm-charts/
└── README.md

Key points:

There is only one p2p lifecycle
All application code, tests, and deployment configurations reside in root directory

Tenant and Namespace Structure

Namespace structure:


app-tenant
├── [s] app-tenant-extended
├── [s] app-tenant-functional
└── [s] app-tenant-nft
└── [s] app-tenant-integration
 
[s] indicates subnamespaces

A single tenant app-tenant is created as a root for the app. To isolate resources related to specific lifecycle stage we create a subnamespaces via hierarchical namespaces . The app-tenant name is stored in a GitHub repository variable TENANT_NAME

Multiple Apps with Workflow per App

This structure is designed for projects containing multiple independent applications, each with its own deployment lifecycle.


/
├── .github/
│   └── workflows/
│       ├── app1-fast-feedback.yaml
│       ├── app1-extended-test.yaml
│       ├── app1-prod.yaml
│       ├── app2-fast-feedback.yaml
│       ├── app2-extended-test.yaml
│       └── app2-prod.yaml
├── app1/
│   ├── Dockerfile
│   ├── Makefile
│   ├── src/
│   ├── tests/
│   │   ├── functional/
│   │   ├── extended/
│   │   └── nft/
│   └── helm-charts/
├── app2/
│   ├── Dockerfile
│   ├── Makefile
│   ├── src/
│   ├── tests/
│   │   ├── functional/
│   │   ├── extended/
│   │   └── nft/
│   └── helm-charts/
└── README.md

Key features:

Modular Application Structure:
- Each application resides in its own directory (app1/, app2/).
- Applications contain all necessary files and configurations, including Dockerfile, Makefile, source code, tests, and Helm charts.
Isolated Lifecycles:
- GitHub Actions workflows are defined separately for each application.
Application-Specific Build and Deployment:
- Each application has its own Makefile with tasks specific to that application.

Tenant and Namespace Structure

Namespace structure:


parent-tenant
├── app1-tenant
│   ├── [s] app1-extended
│   ├── [s] app1-functional
│   └── [s] app1-nft
│   └── [s] app1-integration
└── app2-tenant
    ├── [s] app2-extended
    ├── [s] app2-functional
    └── [s] app2-nft
    └── [s] app2-integration
 
[s] indicates subnamespaces

Key aspects:

Hierarchical Structure:
- A parent tenant serves as the root for all applications.
- Each application has its own child tenant.
Isolated Testing Environments:
- Subnamespaces are created for different testing stages (extended, functional, nft, integration) within each application’s tenant.
Authentication:
- The parent tenant is used for authenticating all applications’ P2P workflows to GCP.

This structure provides a clear separation of concerns, allowing each application to be developed, tested, and deployed independently while maintaining a cohesive project structure.

Multiple Apps with Shared Workflow (Coupled Workload)

This structure is optimized for projects comprising multiple tightly coupled applications that require simultaneous deployment and shared resources.


/
├── .github/
│   └── workflows/
│       ├── coupled-workload-fast-feedback.yaml
│       ├── coupled-workload-extended-test.yaml
│       └── coupled-workload-prod.yaml
├── app1/...
├── app2/...
├── coupled-workload/
│   ├── Makefile
│   ├── app3/
│   │   ├── Dockerfile
│   │   └── src/
│   ├── app4/
│   │   ├── Dockerfile
│   │   └── src/
│   ├── tests/
│   │   ├── functional/
│   │   ├── extended/
│   │   └── nft/
│   ├── helm-charts/
│   │   └── coupled-workload/
│   │       ├── Chart.yaml
│   │       ├── values.yaml
│   │       └── templates/
│   └── resources/
│       └── subns-anchor.yaml
└── README.md

Key features:

Unified Workload Structure:
- All coupled applications reside within a single coupled-workload/ directory.
- Each application (app3/, app4/) contains its Dockerfile and src/ directory.
- A shared Makefile at the workload level manages build and deployment tasks for all applications.
Consolidated Testing:
- Tests are conducted at the workload level, encompassing all applications.
- The tests/ directory includes subdirectories for different testing phases: functional/, extended/, integration/ and nft/ (non-functional tests).
Unified Helm Chart:
- A single Helm chart (helm-charts/coupled-workload/) is used for the entire workload.
- The main Chart.yaml and values.yaml files define the overall workload configuration.
- Subcharts for individual applications ideally stored externally
Shared Lifecycle:
- GitHub Actions workflows are defined for the entire workload, not individual applications.
Resource Management:
- The resources/ directory contains shared configuration files, such as subns-anchor.yaml for namespace management.

Tenant and Namespace Structure


parent-tenant
└── coupled-workload
    ├── [s] coupled-workload-extended
    ├── [s] coupled-workload-functional
    └── [s] coupled-workload-nft
    └── [s] coupled-workload-integration
 
[s] indicates subnamespaces

Key aspects:

Hierarchical Tenant Structure:
- A parent tenant serves as the root for the coupled workload.
- The coupled workload has a single child tenant, promoting unified resource management.
Shared Namespace Environments:
- Subnamespaces are created for different testing environments (extended, functional, nft, integration) within the coupled workload tenant.
- All applications in the workload share these namespaces, facilitating integrated testing and deployment.
Resource Isolation:
- The use of subnamespaces allows for resource isolation between different testing phases while maintaining a cohesive structure.
Simplified Access Control:
- The shared tenant structure simplifies access control and authentication mechanisms for the entire workload.

Manual configuration

You need to manually configure the following:

Helm charts
Makefile

Helm chart

Helm chart contains subchart for each app. Ideally, the subchart should point to a helm chart hosted externally for better versioning support.

Sample Chart.yaml


apiVersion: v2
name: coupled-workload
description: Helm chart for a coupled workload made up of two services
type: application
version: 0.1.0
appVersion: "1.16.0"
dependencies:
  - name: app
    alias: "app-3"
    version: "1.0.0"
    repository: "https://coreeng.github.io/core-platform-assets"
  - name: app
    alias: "app-4"
    version: "1.0.0"
    repository: "https://coreeng.github.io/core-platform-assets"

Configuration can be done via values.yaml file like below:


common:
  resources:
    limits: &limits
      cpu: 500m
      memory: 100Mi
 
app-3:
  appName: app-3
  resources:
    requests:
      cpu: 300m
      memory: 50Mi
    limits: *limits
 
app-4:
  appName: app-4
  resources:
    requests:
      cpu: 300m
      memory: 50Mi
    limits: *limits

Makefile

We need to modify standard p2p Makefile to accommodate multiple docker images.

Modify the p2p-build target to build the images for each app


.PHONY: p2p-build
p2p-build: service-build service-push
 
.PHONY: service-build
service-build:
   @echo $(REGISTRY)
   @echo 'VERSION: $(VERSION)'
   @echo '### SERVICE BUILD ###'
   docker build --platform=linux/amd64  --file ./app-3/Dockerfile --tag $(REGISTRY)/$(FAST_FEEDBACK_PATH)/$(app_3_image_name):$(image_tag) ./app-3
   docker build --platform=linux/amd64 --file ./app-4/Dockerfile --tag $(REGISTRY)/$(FAST_FEEDBACK_PATH)/$(app_4_image_name):$(image_tag) ./app-4
 
.PHONY: service-push
service-push: ## Push the service image
   @echo '### SERVICE PUSH FOR FEEDBACK ###'
   docker image push $(REGISTRY)/$(FAST_FEEDBACK_PATH)/$(app_3_image_name):$(image_tag)
   docker image push $(REGISTRY)/$(FAST_FEEDBACK_PATH)/$(app_4_image_name):$(image_tag)

Modify promotion target to promote the images for each app


.PHONY: p2p-promote-generic
p2p-promote-generic:  ## Generic promote functionality
   skopeo copy --all --preserve-digests \
      docker://$(SOURCE_REGISTRY)/$(source_repo_path)/$(image_name):$(image_tag) \
      docker://$(REGISTRY)/$(dest_repo_path)/$(image_name):$(image_tag)
 
promote-app-3-extended: source_repo_path=$(FAST_FEEDBACK_PATH)
promote-app-3-extended: dest_repo_path=$(EXTENDED_TEST_PATH)
promote-app-3-extended: image_name=$(app_3_image_name)
promote-app-3-extended: p2p-promote-generic
 
promote-app-4-extended: source_repo_path=$(FAST_FEEDBACK_PATH)
promote-app-4-extended: dest_repo_path=$(EXTENDED_TEST_PATH)
promote-app-4-extended: image_name=$(app_4_image_name)
promote-app-4-extended: p2p-promote-generic
 
 
# Promote both images
.PHONY: p2p-promote-to-extended-test
p2p-promote-to-extended-test: promote-app-3-extended promote-app-4-extended

Modify deployment


functional-deploy: namespace=$(tenant_name)-functional
functional-deploy: path=$(FAST_FEEDBACK_PATH)
functional-deploy: deploy
 
deploy:
helm upgrade --install $(helm_release_name) $(helm_chart_path)   \
      -n $(namespace) \
         --set app-3.registry=$(REGISTRY)/$(path) --set app-3.image=$(app_4_image_name) --set app-3.tag=$(VERSION) \
         --set app-4.registry=$(REGISTRY)/$(path) --set app-4.image=$(app_3_image_name) --set app-4.tag=$(VERSION)