add README from elixier/manager
This commit is contained in:
commit
15b3b94d12
74
README.md
Normal file
74
README.md
Normal file
@ -0,0 +1,74 @@
|
||||
# 🌐 Elixier Management Tool
|
||||
|
||||
A web app tool to **deploy and manage data platform clusters on Kubernetes**.
|
||||
|
||||
This web app provides a unified interface for managing tenants, clusters, and users, and integrates with key services such as **Superset**, **Airflow** and **Trino**.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Links
|
||||
|
||||
- 🏗️ [Architecture Overview](docs/architecture-overview.md)
|
||||
- 👥 [User Manual](docs/user-manual.md)
|
||||
- 📘 [Data Platform Manual](docs/data-platform-manual.md)
|
||||
- 🧰 [Developers Guide](docs/developers-guide.md)
|
||||
- 🛠️ [Installation Guide](docs/installation-guide.md)
|
||||
|
||||
---
|
||||
|
||||
## 📑 Table of Contents
|
||||
|
||||
- [System Components](#-system-components)
|
||||
- [Key Features](#-key-features)
|
||||
- [License](#-license)
|
||||
- [Contact](#-contact)
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ System Components
|
||||
|
||||
| Component | Description |
|
||||
|------------|--------------|
|
||||
| **Frontend** | Web interface running NextJS app that are built by Reflex. |
|
||||
| **Backend** | FastAPI backend as part of Reflex framework. |
|
||||
| **PostgreSQL** | Primary database for application metadata and user management. |
|
||||
| **Valkey** | In-memory cache for sessions and task queues. |
|
||||
| **OpenSearch** | Store logs of hosts, clusters, service deployments and users audit. |
|
||||
| **Agent** | Installed on each host to handle communication with the central web app. |
|
||||
|
||||
---
|
||||
|
||||
## 🧩 Key Features
|
||||
|
||||
- Multi-tenant & multi-cluster management
|
||||
- Host and cluster lifecycle control
|
||||
- User and role-based access management
|
||||
- Integrated SSO authentication
|
||||
- Service links and status dashboards
|
||||
- Extensible architecture for new components
|
||||
|
||||
---
|
||||
|
||||
## Other docs
|
||||
|
||||
- [DB Schema](docs/db-schema.md)
|
||||
- [Development Guide (old)](docs/development.md)
|
||||
- [Installation using Docker](docs/docker.md)
|
||||
- [Get Started (old)](docs/get-started.md)
|
||||
- [Software Requirement Specification (SRS)](docs/srs.md)
|
||||
- [Test Procedure Specification (TPS)](docs/test-procedure-spec.md)
|
||||
|
||||
---
|
||||
|
||||
## 📜 License
|
||||
|
||||
TBD
|
||||
|
||||
---
|
||||
|
||||
## 📬 Contact
|
||||
|
||||
**Project Maintainers**
|
||||
Abyres Sdn. Bhd. — DevOps / Platform Engineering Team
|
||||
📧 support@abyres.net
|
||||
🌐 [Company Website](https://www.abyres.net)
|
||||
110
docs/architecture-overview.md
Normal file
110
docs/architecture-overview.md
Normal file
@ -0,0 +1,110 @@
|
||||
# 🏗️ Architecture Overview
|
||||
|
||||
This document provides an overview of the system architecture and its major components.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [System Context](#system-context)
|
||||
- [Component Diagrams](#component-diagrams)
|
||||
- [Components in Elixier Management Tool](#components-in-elixier-management-tool)
|
||||
- [Components in Kubernetes Cluster](#components-in-kubernetes-cluster)
|
||||
|
||||
---
|
||||
|
||||
## System Context
|
||||
|
||||
```mermaid
|
||||
C4Context
|
||||
|
||||
System(docs, "Documentation", "Gitea Pages")
|
||||
Person(admins, "Admins", "Tool administrators<br>e.g Abyres")
|
||||
System(emgr, "Elixier Management Tool", "Web app")
|
||||
|
||||
Enterprise_Boundary(eb1, "Client") {
|
||||
Person_Ext(clients, "Clients", "Tool & data platform users<br>e.g Credence")
|
||||
System_Boundary(sb1, "On-prem / cloud") {
|
||||
System_Ext(sso, "Identity Provider", "Single-sign on<br>user auth")
|
||||
System_Ext(k3s, "Kubernetes Cluster", "Infra & compute power<br>for data platform")
|
||||
System_Ext(s3, "Object Storage", "Store raw &<br>processed data")
|
||||
}
|
||||
}
|
||||
|
||||
Rel(admins, emgr, "Manage<br>clients")
|
||||
Rel(admins, docs, "Refer")
|
||||
|
||||
Rel(clients, docs, "Refer")
|
||||
Rel(clients, emgr, "Manage tenants, hosts,<br>clusters & services")
|
||||
Rel(clients, k3s, "Use services", "e.g Superset & Airflow")
|
||||
|
||||
Rel(emgr, k3s, "Deploy clusters<br>& services")
|
||||
Rel(emgr, s3, "Read/write<br>datasets & ETL")
|
||||
BiRel(emgr, sso, "SSO", "e.g OIDC")
|
||||
|
||||
Rel(k3s, s3, "Read &<br>write files")
|
||||
BiRel(k3s, sso, "SSO", "e.g OIDC")
|
||||
```
|
||||
---
|
||||
|
||||
## Component Diagrams
|
||||
|
||||
### Components in Elixier Management Tool
|
||||
|
||||
```mermaid
|
||||
C4Component
|
||||
|
||||
Person(admins, "Admins", "Tool administrators<br>e.g Abyres")
|
||||
Person_Ext(clients, "Clients", "Tool & data platform users<br>e.g Credence")
|
||||
|
||||
Container_Boundary(emgr, "Elixier Management Tool") {
|
||||
Component(frontend, "Frontend", "Reflex, NextJS", "Web app frontend")
|
||||
Component(backend, "Backend", "Reflex, FastAPI", "Web app backend")
|
||||
ComponentDb(postgres, "Database", "PostgreSQL", "Store app data &<br>cluster metadata")
|
||||
ComponentDb(opensearch, "Log Store", "OpenSearch", "Store logs of hosts,<br>clusters, services & audit")
|
||||
ComponentQueue(valkey, "Message Queue", "Valkey", "Send commands to agent")
|
||||
}
|
||||
Component_Ext(sso, "Identity Provider", "e.g Authentik / Keycloak")
|
||||
Component_Ext(s3, "Object Storage", "e.g MinIO / S3")
|
||||
|
||||
Rel(admins, frontend, "Use", "HTTPS")
|
||||
Rel(clients, frontend, "Use", "HTTPS")
|
||||
|
||||
BiRel(frontend, backend, "Exchange data", "Websocket, HTTPS")
|
||||
Rel(backend, sso, "SSO", "OIDC")
|
||||
BiRel(backend, postgres, "Read/write data")
|
||||
BiRel(backend, opensearch, "Read/write data")
|
||||
Rel(backend, s3, "Read/write<br>datasets & ETL", "S3 API")
|
||||
Rel(backend, valkey, "Queue commands")
|
||||
```
|
||||
|
||||
### Components in Kubernetes Cluster
|
||||
|
||||
```mermaid
|
||||
C4Component
|
||||
|
||||
Person_Ext(clients, "Clients", "Tool & data platform users<br>e.g Credence")
|
||||
|
||||
Container_Boundary(k3s, "Kubernetes Cluster") {
|
||||
Container_Ext(superset, "Data Visualization", "Superset", "Build charts & dashboards")
|
||||
Container_Ext(airflow, "Data ETL", "Airflow", "Run ETL scripts")
|
||||
Container_Ext(trino, "SQL Engine", "Trino", "Read & write structured data<br>to object storage")
|
||||
Container(agent, "Agent", "EMGR Operator", "Execute commands,<br>send metrics data")
|
||||
}
|
||||
Container_Boundary(emgr, "Elixier Management Tool") {
|
||||
Container(backend, "Backend", "Reflex, FastAPI", "Web app backend")
|
||||
}
|
||||
Container_Ext(sso, "Identity Provider", "e.g Authentik / Keycloak")
|
||||
Container_Ext(s3, "Object Storage", "e.g MinIO / S3")
|
||||
|
||||
Rel(clients, superset, "Use", "HTTPS")
|
||||
Rel(clients, airflow, "Use", "HTTPS")
|
||||
|
||||
Rel(trino, s3, "Read/write data", "Hive, Iceberg")
|
||||
Rel(superset, trino, "Retrieve data", "SQL")
|
||||
Rel(airflow, trino, "Read/write data", "SQL")
|
||||
Rel(airflow, s3, "Read/write data", "S3 API")
|
||||
Rel(superset, sso, "SSO", "OIDC")
|
||||
Rel(airflow, sso, "SSO", "OIDC")
|
||||
BiRel(agent, backend, "Send commands,<br>metrics data")
|
||||
```
|
||||
393
docs/data-platform-manual.md
Normal file
393
docs/data-platform-manual.md
Normal file
@ -0,0 +1,393 @@
|
||||
# 📘 Data Platform Manual
|
||||
|
||||
This manual guides data engineers & data analysts (DA/DE) through using Airflow, Superset, Trino and Object Storage to perform ETL & dashboarding.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Components](#components)
|
||||
- [Airflow](#airflow)
|
||||
- [Superset](#superset)
|
||||
- [Trino](#trino)
|
||||
- [Object Storage](#object-storage)
|
||||
- [Workflow](#workflow)
|
||||
- [Data Pipeline](#data-pipeline)
|
||||
- [1. Data Ingestion](#1-data-ingestion)
|
||||
- [2. Raw Data Storage](#2-raw-data-storage)
|
||||
- [3. Data Transformation / ETL](#3-data-transformation--etl)
|
||||
- [4. Processed Data Storage](#4-processed-data-storage)
|
||||
- [5. Data Visualization](#5-data-visualization)
|
||||
- [Example](#example)
|
||||
- [sample_dat.py](#sample_dagpy)
|
||||
- [Dockerfile](#dockerfile)
|
||||
- [Build & publish container image](#build--publish-container-image)
|
||||
---
|
||||
|
||||
|
||||
## Components
|
||||
|
||||
### Airflow
|
||||
|
||||
An open-source **workflow orchestration tool** used to author, schedule, and monitor data pipelines. In a data platform, Airflow automates complex ETL/ELT processes, integrates with various data systems, and ensures reliable, repeatable data workflows across the environment.
|
||||
|
||||
### Superset
|
||||
|
||||
A modern, open-source **data visualization and business intelligence (BI) platform**. It enables users to explore datasets, create dashboards, and visualize insights interactively. Within a data platform, Superset provides self-service analytics and reporting capabilities for end users.
|
||||
|
||||
### Trino
|
||||
|
||||
A distributed **SQL query engine** designed for fast analytics across large datasets stored in various systems (e.g., object storage, databases, data lakes). In a data platform, Trino serves as the query layer, enabling users and tools like Superset to access and analyze data from multiple sources using standard SQL.
|
||||
|
||||
### Object Storage
|
||||
|
||||
An **S3-compatible storage provider** (e.g., MinIO) used to store and retrieve unstructured data such as files, logs, and backups. It provides a scalable, API-driven interface for applications to manage data objects, similar to AWS S3.
|
||||
|
||||
## Workflow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
|
||||
%% STAGE 1: DATA SOURCES
|
||||
A["Data Sources
|
||||
(S3 / MinIO, DBs, APIs)"] -->|Ingestion Jobs| B[Apache Airflow]
|
||||
|
||||
%% STAGE 2: RAW STORAGE
|
||||
B -->|Store Raw Data| C["Raw Zone
|
||||
(S3 / MinIO)"]
|
||||
|
||||
%% STAGE 3: TRANSFORMATION
|
||||
C -->|DAG / ETL / SQL Queries| D["Trino
|
||||
(Query Engine)"]
|
||||
B -->|Workflow Orchestration| D
|
||||
|
||||
%% STAGE 4: PROCESSED STORAGE
|
||||
D -->|Write Processed Data| E["Processed / Curated Zone
|
||||
(S3 / MinIO)"]
|
||||
|
||||
%% STAGE 5: QUERY LAYER
|
||||
E -->|Query Interface| F["Trino
|
||||
(SQL Access Layer)"]
|
||||
|
||||
%% STAGE 6: VISUALIZATION
|
||||
F -->|Data Access| G["Apache Superset
|
||||
(Dashboarding & Analytics)"]
|
||||
|
||||
%% LABELS
|
||||
classDef core fill:#4a90e2,stroke:#2c3e50,stroke-width:1px,color:white;
|
||||
classDef storage fill:#6dbf4b,stroke:#2c3e50,stroke-width:1px,color:white;
|
||||
classDef optional fill:#aaaaaa,stroke:#333,stroke-width:0.5px,color:white;
|
||||
|
||||
class B,D,F,G core;
|
||||
class C,E storage;
|
||||
class H1,H2,H3,H4 optional;
|
||||
```
|
||||
|
||||
## Data Pipeline
|
||||
|
||||
### 1. Data Ingestion
|
||||
|
||||
Collect raw data from multiple sources and bring it into the platform in a structured workflow.
|
||||
|
||||
Components:
|
||||
- Apache Airflow (orchestrates ingestion pipelines)
|
||||
|
||||
Data sources:
|
||||
- Object storage (S3 / MinIO)
|
||||
- files: csv, xlsx, txt etc.
|
||||
- APIs (REST/GraphQL endpoints)
|
||||
- Databases (PostgreSQL, MySQL, etc.)
|
||||
|
||||
DA/DE tasks:
|
||||
- Create DAGs in Airflow to pull data periodically.
|
||||
|
||||
Sample Airflow DAG (Python):
|
||||
- See [sample_dag.py](#sample_dagpy)
|
||||
|
||||
### 2. Raw Data Storage
|
||||
|
||||
Store the raw, unprocessed data in a centralized location for auditing and reprocessing.
|
||||
|
||||
Components:
|
||||
- Object storage (S3 / MinIO)
|
||||
|
||||
DA/DE tasks:
|
||||
- Organize data using bucket/folder structures.
|
||||
|
||||
Sample S3 Folder Structure:
|
||||
- See [Data source files](#data-source-files)
|
||||
|
||||
### 3. Data Transformation / ETL
|
||||
|
||||
Clean, enrich, and transform raw data into structured, query-ready form.
|
||||
|
||||
Components:
|
||||
- Apache Airflow (orchestration)
|
||||
- Trino (SQL engine for transformations)
|
||||
|
||||
DA/DE tasks:
|
||||
- Schedule transformation jobs in Airflow DAGs.
|
||||
|
||||
Airflow DAG Snippet for ETL:
|
||||
- see [sample_dag.py](#sample_dagpy)
|
||||
|
||||
S3 Folder Structure:
|
||||
- see [Python DAG files](#python-dag-files)
|
||||
|
||||
### 4. Processed Data Storage
|
||||
|
||||
Store the transformed and curated datasets in a queryable format for analytics and dashboarding.
|
||||
|
||||
Components:
|
||||
- Trino (query engine / SQL layer)
|
||||
- S3 (object storage for processed datasets)
|
||||
|
||||
DA/DE tasks:
|
||||
- Partition tables by date, region, or other dimensions for fast queries.
|
||||
- Grant read access to Superset.
|
||||
|
||||
### 5. Data Visualization
|
||||
|
||||
Provide dashboards and reports to enable insights and business decision-making.
|
||||
|
||||
Components:
|
||||
- Apache Superset (dashboarding / BI tool)
|
||||
|
||||
DA/DE tasks:
|
||||
- Create datasets and charts (bar, line, heatmaps).
|
||||
- Build dashboards combining multiple metrics.
|
||||
- Apply filters and access controls for different users.
|
||||
|
||||
Data source connections:
|
||||
- 'Data Platform' service already configured these database connections in Superset:
|
||||
- iceberg
|
||||
- hive
|
||||
|
||||
## Example
|
||||
|
||||
### sample_dag.py
|
||||
|
||||
This script demonstrates data ingestion & ETL phases.
|
||||
|
||||
- `load_excel_to_csv()` - Ingesting data from source excel file, and store raw data as csv in object storage
|
||||
- `load_csv_to_trino()` - Extract raw data from csv file, transform it, then load it back to object storage using Trino. Processed data is stored in Trino iceberg format, 'default' schema & 'computer_parts_sales' table.
|
||||
|
||||
```python
|
||||
from airflow.sdk import DAG, task
|
||||
from datetime import datetime, timedelta
|
||||
import os
|
||||
|
||||
# to pass env vars from scheduler to worker pods
|
||||
env_vars = {n: os.environ.get(n, '') for n in ['S3_ENDPOINT', 'S3_BUCKET', 'S3_ACCESS_KEY', 'S3_SECRET_KEY', 'S3_REGION', 'TRINO_HIVE', 'TRINO_ICEBERG']}
|
||||
|
||||
with DAG(
|
||||
dag_id="sample_dag",
|
||||
description="Sample DAG",
|
||||
schedule="*/5 * * * *", # every 5 mins
|
||||
start_date=datetime(2025, 11, 5),
|
||||
catchup=False,
|
||||
) as dag:
|
||||
|
||||
@task.kubernetes(
|
||||
image="azwan082/python:3.11-airflow-dag-3",
|
||||
env_vars=env_vars,
|
||||
do_xcom_push=True,
|
||||
)
|
||||
def load_excel_to_csv():
|
||||
import os
|
||||
import boto3
|
||||
import csv
|
||||
import io
|
||||
import pandas as pd
|
||||
|
||||
# use env vars to connect to object storage configured in the cluster page
|
||||
s3 = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=os.getenv("S3_ENDPOINT"),
|
||||
aws_access_key_id=os.getenv("S3_ACCESS_KEY"),
|
||||
aws_secret_access_key=os.getenv("S3_SECRET_KEY"),
|
||||
region_name=os.getenv("S3_REGION"),
|
||||
)
|
||||
|
||||
bucket_name = 'emgr'
|
||||
key = 'airflow/excel/computer-parts-sales.xlsx'
|
||||
sheet_name = 'Sheet1'
|
||||
columns = ['Date', 'Part', 'Quantity_Sold', 'Unit_Price', 'Total_Sale']
|
||||
|
||||
# read excel file from s3
|
||||
response = s3.get_object(Bucket=bucket_name, Key=key)
|
||||
file_content = response["Body"].read()
|
||||
excel_data = io.BytesIO(file_content)
|
||||
df = pd.read_excel(excel_data, sheet_name=sheet_name, dtype=str)
|
||||
df = df[columns]
|
||||
df.columns = [c.lower() for c in df.columns]
|
||||
|
||||
# write csv file to s3
|
||||
key = 'airflow/csv/computer-parts-sales.csv'
|
||||
csv_buffer = io.StringIO()
|
||||
df.to_csv(csv_buffer, index=False)
|
||||
s3.put_object(Bucket=bucket_name, Key=key, Body=csv_buffer.getvalue())
|
||||
|
||||
return {'key': key}
|
||||
|
||||
@task.kubernetes(
|
||||
image="azwan082/python:3.11-airflow-dag-3",
|
||||
env_vars=env_vars
|
||||
)
|
||||
def load_csv_to_trino(data: dict):
|
||||
import os
|
||||
import boto3
|
||||
import csv
|
||||
import io
|
||||
import pandas as pd
|
||||
from sqlalchemy import create_engine, text
|
||||
|
||||
s3 = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=os.getenv("S3_ENDPOINT"),
|
||||
aws_access_key_id=os.getenv("S3_ACCESS_KEY"),
|
||||
aws_secret_access_key=os.getenv("S3_SECRET_KEY"),
|
||||
region_name=os.getenv("S3_REGION"),
|
||||
)
|
||||
|
||||
bucket_name = 'emgr'
|
||||
key = data.get('key')
|
||||
|
||||
# read csv file from s3
|
||||
response = s3.get_object(Bucket=bucket_name, Key=key)
|
||||
csv_data = response["Body"].read().decode("utf-8")
|
||||
reader = csv.DictReader(io.StringIO(csv_data))
|
||||
records = list(reader)
|
||||
df = pd.DataFrame(records)
|
||||
|
||||
# cast column type
|
||||
df['date'] = pd.to_datetime(df['date']).dt.date
|
||||
df['quantity_sold'] = df['quantity_sold'].astype(int)
|
||||
df['unit_price'] = df['unit_price'].astype(float)
|
||||
df['total_sale'] = df['total_sale'].astype(float)
|
||||
|
||||
# create database engine
|
||||
schema = "default"
|
||||
table = "computer_parts_sales"
|
||||
TRINO_ICEBERG = os.getenv("TRINO_ICEBERG")
|
||||
engine = create_engine(TRINO_ICEBERG)
|
||||
|
||||
# create table if not exists
|
||||
create_sql = f"""
|
||||
CREATE TABLE IF NOT EXISTS {schema}.{table} (
|
||||
date DATE,
|
||||
part VARCHAR,
|
||||
quantity_sold INT,
|
||||
unit_price DOUBLE,
|
||||
total_sale DOUBLE
|
||||
)
|
||||
"""
|
||||
with engine.connect() as conn:
|
||||
conn.execute(text(create_sql))
|
||||
conn.commit()
|
||||
|
||||
# store data to s3
|
||||
df.to_sql(
|
||||
name=table,
|
||||
con=engine,
|
||||
schema=schema,
|
||||
index=False,
|
||||
if_exists="append",
|
||||
method=None,
|
||||
)
|
||||
|
||||
# Task dependencies (data flow)
|
||||
data = load_excel_to_csv()
|
||||
load_csv_to_trino(data)
|
||||
```
|
||||
|
||||
Image `azwan082/python:3.11-airflow-dag-3` used in example above contains these Python libraries:
|
||||
- boto3 - to connect to S3-compatible object storage
|
||||
- pandas - to process data using DataFrame
|
||||
- requests - to perform HTTP requests to REST API or webpage
|
||||
- sqlalchemy - to connect to Trino
|
||||
- trino - to connect to Trino
|
||||
- sqlalchemy-trino - to run SQL on Trino
|
||||
- openpyxl - to read .xlsx files
|
||||
|
||||
If you need more libraries, or want to customize the image, refer to [Dockerfile](#dockerfile) section below.
|
||||
|
||||
Notes:
|
||||
|
||||
- XCom means cross-communication, where one task can return values to be consumed by another task:
|
||||
- Sample code above has two tasks, to demo how XCom works. For simple DAG, one task is enough.
|
||||
- Do not return large data between task through XCom, the pod may fail to start. Store resulting data in object storage.
|
||||
- Since Airflow is configured to use KubernetesExecutor, each tasks in a DAG will be executed on a new pod. In order to reduce impact of pods startup overhead:
|
||||
- Design your DAGs with fewer tasks.
|
||||
- Avoid scheduling DAGs too frequently, set at least 5 minutes apart.
|
||||
|
||||
### Dockerfile
|
||||
|
||||
This Dockerfile allows data engineers to include Python libraries required for their data ingestion or ETL processes.
|
||||
|
||||
```Dockerfile
|
||||
FROM python:3.11-slim-buster
|
||||
WORKDIR /app
|
||||
RUN pip install --no-cache-dir \
|
||||
boto3 \
|
||||
pandas \
|
||||
requests \
|
||||
sqlalchemy \
|
||||
trino \
|
||||
sqlalchemy-trino \
|
||||
openpyxl
|
||||
CMD ["python3"]
|
||||
```
|
||||
|
||||
#### Build & publish container image
|
||||
|
||||
- Requirement: Docker installed & Docker hub account
|
||||
- Build image (run in folder containing the Dockerfile):
|
||||
```bash
|
||||
docker build -t <username>/python:3.11-airflow-dag .
|
||||
```
|
||||
- Push image to Docker hub:
|
||||
```bash
|
||||
docker login
|
||||
docker push <username>/python:3.11-airflow-dag
|
||||
```
|
||||
- Update dag file to use this new image
|
||||
```python
|
||||
@task.kubernetes(image="<username>/python:3.11-airflow-dag")
|
||||
```
|
||||
- Note: update the image tag everytime you build a new image. E.g `python:3.11-airflow-dag-1.1`
|
||||
|
||||
## Object Storage Folder Structure
|
||||
|
||||
Assuming the 'Data Platform' service is deployed with 'Object Storage' configuration:
|
||||
|
||||
- Endpoint: `https://s3.example.net`
|
||||
- Bucket: `emgr`
|
||||
|
||||
### Trino generated files
|
||||
|
||||
- 'Data Platform' service already configured to store the Trino tables data in `warehouses` folder in the target bucket.
|
||||
|
||||
### Python DAG files
|
||||
|
||||
- **MUST** be stored in `airflow/dags` folder in the target bucket, in order to be automatically synced to Airflow.
|
||||
- Example object path:
|
||||
```
|
||||
s3://s3.example.net/emgr/airflow/dags/sample_dag.py
|
||||
s3://s3.example.net/emgr/airflow/dags/monthly_sales.py
|
||||
```
|
||||
|
||||
### Data source files
|
||||
|
||||
- Example data source files are xlsx, csv or txt files, for both raw & processed data.
|
||||
- They can be stored in any location within the target bucket, **EXCEPT** locations from sections above:
|
||||
- `warehouses`
|
||||
- `airflow/dags` (specifically)
|
||||
- However, you may, and encouraged, to store the data source files inside the `airflow` folder.
|
||||
- Example object path:
|
||||
```
|
||||
s3://s3.example.net/emgr/airflow/raw/sample.csv
|
||||
s3://s3.example.net/emgr/airflow/output/voters.csv
|
||||
s3://s3.example.net/emgr/2025-11-11/data.json
|
||||
s3://s3.example.net/emgr/raw/db/orders_20251111.csv
|
||||
```
|
||||
175
docs/db-schema.md
Normal file
175
docs/db-schema.md
Normal file
@ -0,0 +1,175 @@
|
||||
# Database Schema
|
||||
|
||||
## User & Session
|
||||
```mermaid
|
||||
erDiagram
|
||||
|
||||
user {
|
||||
int id PK
|
||||
str email UK
|
||||
str display_name
|
||||
str username UK
|
||||
str password
|
||||
bool is_active "default: true"
|
||||
bool is_superuser "default: false"
|
||||
str type "default: local"
|
||||
int domain_id FK "domain.id"
|
||||
int active_tenant_id FK "tenant.id, empty if user is not assigned to any tenant"
|
||||
}
|
||||
|
||||
session {
|
||||
int id PK
|
||||
int user_id FK "user.id"
|
||||
str session_token
|
||||
}
|
||||
|
||||
user ||--o{ session : has
|
||||
domain ||--o{ user : has
|
||||
```
|
||||
|
||||
## Auth Provider & Domain
|
||||
```mermaid
|
||||
erDiagram
|
||||
|
||||
auth_provider {
|
||||
int id PK
|
||||
int domain_id FK "domain.id"
|
||||
str client_id "OAuth client ID"
|
||||
str client_secret "OAuth client secret"
|
||||
str well_known_url "URL to fetch the OpenID Connect configuration (required)"
|
||||
str additional_config "Additional configuration in JSON format"
|
||||
datetime created_at
|
||||
datetime updated_at
|
||||
}
|
||||
|
||||
domain {
|
||||
int id PK
|
||||
str name UK "e.g abyres.net"
|
||||
str title "e.g Abyres Sdn Bhd"
|
||||
datetime created_at
|
||||
datetime updated_at
|
||||
}
|
||||
|
||||
domain ||--o| auth_provider: has
|
||||
domain ||--o{ tenant : has
|
||||
|
||||
```
|
||||
|
||||
## Tenant
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
|
||||
tenant {
|
||||
int id PK
|
||||
str title "Tenant's name"
|
||||
str name UK "Unique name generated from 'title' field"
|
||||
str logo "Path to logo file"
|
||||
int domain_id FK "domain.id"
|
||||
bool is_active
|
||||
datetime created_at
|
||||
datetime updated_at
|
||||
str address "Tenant's work address"
|
||||
str email "Tenant's work email"
|
||||
str phone "Tenant's work phone number"
|
||||
str website "Tenant's official website"
|
||||
str smtp_server
|
||||
}
|
||||
|
||||
tenant_member {
|
||||
int id PK
|
||||
int user_id FK, UK "user.id"
|
||||
int tenant_id FK, UK "tenant.id"
|
||||
datetime created_at
|
||||
datetime updated_at
|
||||
}
|
||||
|
||||
tenant ||..o{ host : has
|
||||
user ||--o{ tenant_member : "associated with"
|
||||
tenant ||--o{ tenant_member : "associated with"
|
||||
```
|
||||
|
||||
## Host & Cluster
|
||||
```mermaid
|
||||
erDiagram
|
||||
|
||||
host {
|
||||
int id PK
|
||||
str hostname "e.g. dev-01.abyres.net"
|
||||
str ip_address "e.g. 192.168.1.10"
|
||||
str token "Used by client agent to authenticate connection to server"
|
||||
str uid UK "Unique string to identify the host"
|
||||
int tenant_id FK "tenant.id"
|
||||
int cluster_id FK "cluster.id, empty if host is not part of any cluster"
|
||||
datetime created_at
|
||||
datetime updated_at
|
||||
}
|
||||
|
||||
cluster {
|
||||
int id PK
|
||||
str name "e.g. project-a-cluster"
|
||||
int tenant_id FK "tenant.id"
|
||||
datetime created_at
|
||||
datetime updated_at
|
||||
str kubeconfig "'kubeconfig' content to access the cluster"
|
||||
}
|
||||
|
||||
tenant ||--o{ cluster : has
|
||||
cluster ||--|{ host : "deployed on"
|
||||
|
||||
```
|
||||
|
||||
|
||||
## Services & Secrets
|
||||
```mermaid
|
||||
erDiagram
|
||||
service_template {
|
||||
int id PK
|
||||
str name
|
||||
str image
|
||||
str resources
|
||||
str volume
|
||||
}
|
||||
|
||||
service_deployment {
|
||||
int id PK
|
||||
int template_id FK
|
||||
int cluster_id FK
|
||||
str name
|
||||
}
|
||||
|
||||
secret {
|
||||
int id PK
|
||||
int deployment_id FK
|
||||
str name
|
||||
str token
|
||||
}
|
||||
|
||||
service_template ||--o{ service_deployment : has
|
||||
cluster ||--o{ service_deployment : has
|
||||
service_deployment ||--o{ secret : has
|
||||
```
|
||||
|
||||
## Audit Log
|
||||
```mermaid
|
||||
erDiagram
|
||||
audit_log {
|
||||
int id PK
|
||||
str table_name "user, domain, tenant, host, cluster, service_deployment"
|
||||
int table_id FK
|
||||
int timestamp
|
||||
int user_id FK
|
||||
str ip_addr
|
||||
str message
|
||||
str prev_settings
|
||||
str diff
|
||||
}
|
||||
|
||||
user ||..o{ audit_log : has
|
||||
domain ||..o{ audit_log : has
|
||||
tenant ||..o{ audit_log : has
|
||||
host ||..o{ audit_log : has
|
||||
cluster ||..o{ audit_log : has
|
||||
service_deployment ||..o{ audit_log : has
|
||||
audit_log ||--|| user : "acted by"
|
||||
```
|
||||
124
docs/developers-guide.md
Normal file
124
docs/developers-guide.md
Normal file
@ -0,0 +1,124 @@
|
||||
# 🧰 Developers Guide
|
||||
|
||||
This guide helps developers set up a local environment for contributing to the project.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- VS Code
|
||||
- Docker
|
||||
- Git
|
||||
|
||||
---
|
||||
|
||||
## Setup Instructions
|
||||
|
||||
1. **Clone the repository**
|
||||
```bash
|
||||
git clone https://git.abyres.net/elixier/manager.git
|
||||
cd manager
|
||||
```
|
||||
2. **Create .env file**
|
||||
```bash
|
||||
FERNET_KEY=secretkey
|
||||
APP_URL=http://localhost:3000
|
||||
BACKEND_URL=http://localhost:8000
|
||||
APP_VERSION=1.1.0-build.1887.166b86a9
|
||||
AGENT_IMAGE=git.abyres.net/elixier/agent:1.1.0-build.1887.166b86a9
|
||||
```
|
||||
|
||||
- Set APP_VERSION to image tag obtained from https://git.abyres.net/elixier/-/packages/container/manager/
|
||||
- AGENT_IMAGE tag must be same as APP_VERSION
|
||||
|
||||
3. **Open in VS Code**
|
||||
```bash
|
||||
code .
|
||||
```
|
||||
4. **Reopen project in devcontainer**
|
||||
|
||||
`Ctrl + Shift + P` > `Dev Containers: Reopen in container`
|
||||
|
||||
5. **Initialize database**
|
||||
```bash
|
||||
reflex db migrate
|
||||
```
|
||||
6. **Create initial global admin**
|
||||
```bash
|
||||
python cli.py create-admin --email=admin@example.com --password=password
|
||||
```
|
||||
7. **Generate plugin repository**
|
||||
```bash
|
||||
python cli.py build-repo
|
||||
```
|
||||
7. **Run web app**
|
||||
|
||||
Development mode:
|
||||
|
||||
`Ctrl + Shift + P` > `Tasks: Run Task` > `Run`
|
||||
|
||||
Production mode:
|
||||
```bash
|
||||
reflex run --env prod
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Branching & Workflow
|
||||
|
||||
- `main` — stable release
|
||||
- `development` — active development
|
||||
- `feat-*` — new features
|
||||
- `fix-*` — bug fixes
|
||||
|
||||
Pull requests to `main` branch require at least two approvals before merging.
|
||||
|
||||
---
|
||||
|
||||
## Running Tests
|
||||
|
||||
1. Open project in devcontainer using VS Code
|
||||
2. `Ctrl + Shift + P` > `Tasks Run Task` > `Test`
|
||||
|
||||
or run this command in devcontainer:
|
||||
```bash
|
||||
uv run pytest tests
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## OpenSearch
|
||||
|
||||
### List of indexes
|
||||
|
||||
- audit_log
|
||||
- cluster-metrics
|
||||
- deployment-metrics
|
||||
- host-metrics
|
||||
- host-logs
|
||||
- host-task-log
|
||||
|
||||
### Delete index
|
||||
|
||||
1. Delete one index:
|
||||
```bash
|
||||
curl -X DELETE -u admin:{OPENSEARCH_ADMIN_PASSWORD} -k https://localhost:9200/host-metrics
|
||||
```
|
||||
|
||||
2. Delete multiple indexes:
|
||||
```bash
|
||||
curl -X DELETE -u admin:{OPENSEARCH_ADMIN_PASSWORD} -k https://localhost:9200/audit_log,cluster-metrics,deployment-metrics,host-metrics,host-logs,host-task-log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Devcontainer Components
|
||||
|
||||
| Container | Component | URL |
|
||||
| --- | --- | --- |
|
||||
| Reflex | Frontend | http://localhost:3000 |
|
||||
| | Backend | http://localhost:8000 |
|
||||
| | Swagger | http://localhost:8000/docs |
|
||||
| Opensearch | API | https://localhost:9200 |
|
||||
| | Dashboard | http://localhost:5601 |
|
||||
| Nginx | Plugin repo | http://localhost:8080 |
|
||||
200
docs/development.md
Normal file
200
docs/development.md
Normal file
@ -0,0 +1,200 @@
|
||||
# Documentation
|
||||
|
||||
The Elixier Manager project requires certain development practices when using Reflex.dev. This documentation should outline the well known issues and how to navigate them
|
||||
|
||||
## Table of contents
|
||||
|
||||
1. [Login using authentik return NoneType](#login-using-authentik-return-NoneType)
|
||||
2. [Elixier manager test environment](#elixier-manager-test-environment)
|
||||
- [Method to deploy latest update](#method-to-deploy-latest-update)
|
||||
- [View logging](#view-logging)
|
||||
3. [Deployment Checklist](#deployment-checklist)
|
||||
4. [Creating command to host](#creating-command-to-host)
|
||||
5. [Agent action](#agent-action)
|
||||
- [Triggering an action](#triggering-an-action)
|
||||
- [Designing an action](#designing-an-action)
|
||||
|
||||
### Login Using Authentik return NoneType
|
||||
|
||||
When trying to login using domain authprovider and redirected to error `NoneType` have no attribute `id`
|
||||
|
||||
Solution: Check `APP_URL` value in environment variable and verify against browser url. One Should always use `APP_URL` to access Elixier Management due to redirect url will use `APP_URL` value
|
||||
|
||||
### Elixier Manager Test Environment
|
||||
|
||||
- Test Host URL: `192.168.0.219`
|
||||
- PIC with access: `Azwan Amit`
|
||||
- Deployment method: `Systemd`
|
||||
|
||||
#### Method to deploy latest update:
|
||||
|
||||
as `emgr` user:
|
||||
```bash
|
||||
cd /home/emgr/manager
|
||||
|
||||
git pull
|
||||
```
|
||||
|
||||
as `root` user:
|
||||
```bash
|
||||
systemctl restart emgr
|
||||
```
|
||||
|
||||
p/s: It would be best if it were to be deployed as Continous Deployment
|
||||
|
||||
#### View Logging
|
||||
|
||||
To view logs for test environment use journald as user `root`
|
||||
|
||||
```bash
|
||||
journalctl -u emgr
|
||||
```
|
||||
|
||||
### Deployment Checklist
|
||||
|
||||
Before deploying a manager, one should run through the checklist to ensure smooth deployment.
|
||||
|
||||
#### Checklist
|
||||
- [x] Check current branch, checkout to the needed branch
|
||||
- [x] Git pull to the latest commit log
|
||||
- [x] Make migration if there are model changes
|
||||
- [x] Reset database if required
|
||||
- [x] Migrate to the latest migrations
|
||||
- [x] Delete Opensearch index
|
||||
- [x] Create initial global admin
|
||||
|
||||
### Creating Command to Host
|
||||
|
||||
Sending command to Host require few steps to consider
|
||||
|
||||
1. `MasterTask` and `HostTask` creation for task monitoring
|
||||
|
||||
```python
|
||||
master_task = await MasterTaskService().create(
|
||||
title=f"Setting up Cluster: {self.cluster.name}",
|
||||
cluster_id=self.cluster.id,
|
||||
tenant_id=self.cluster.tenant_id,
|
||||
task_type="cluster_setup",
|
||||
remarks="Started Setting up Cluster.",
|
||||
)
|
||||
|
||||
await HostTaskService().deploy_task(
|
||||
master_task=master_task,
|
||||
host_id=controller,
|
||||
command="initialize_controller",
|
||||
)
|
||||
```
|
||||
2. command should be available in `action_list` from `streaming.actions`
|
||||
|
||||
```python
|
||||
action_list = {
|
||||
"initialize_controller": controller_init,
|
||||
"initialize_worker": worker_init,
|
||||
"check_nodes_status": check_nodes_status,
|
||||
"install_service": install_service,
|
||||
"uninstall_service": uninstall_service,
|
||||
"restart_service": restart_service,
|
||||
"update_service": update_service,
|
||||
"update_service_config": update_service_config,
|
||||
"get_service_url": get_service_url,
|
||||
}
|
||||
```
|
||||
3. Task monitoring should run in background using `@rx.event(background=True)`
|
||||
|
||||
```python
|
||||
@rx.event(background=True)
|
||||
async def cluster_setup_task(self):
|
||||
# monitor task
|
||||
```
|
||||
4. Task monitoring should not run indefinitely. Use `while True` together with `await asyncio.sleep`
|
||||
|
||||
```python
|
||||
while True:
|
||||
# do something
|
||||
await asyncio.sleep(10)
|
||||
```
|
||||
5. Task monitoring should have exit condition for <mark>Task failed</mark> or <mark>Task exceed time limit</mark>
|
||||
|
||||
Use `count` and maximum count to check for limit and exit monitoring
|
||||
```python
|
||||
controller_check_count = 0
|
||||
while True:
|
||||
# check task
|
||||
controller_check_count += 1
|
||||
if controller_check_count > 30:
|
||||
await self._handle_cluster_setup_failure(
|
||||
"Failed to initialize controller."
|
||||
)
|
||||
break
|
||||
|
||||
await asyncio.sleep(10)
|
||||
```
|
||||
### Agent Action
|
||||
|
||||
Agent action is the main method to execute task in host. There are 3 things that agent action do:
|
||||
|
||||
1. Execute host shell command
|
||||
2. Report back progress command together with response from command executed
|
||||
3. Send stdout and stderr to manager for task logging from command execution
|
||||
|
||||
#### Triggering an Action
|
||||
|
||||
- Parameter for agent action is as below; `parameters`, `node`, and `executor`
|
||||
|
||||
```python
|
||||
async def initialize_controller(
|
||||
parameters: dict,
|
||||
node: Node,
|
||||
executor: Executor,
|
||||
) -> AsyncGenerator[Dict[str, int | str], None:
|
||||
pass
|
||||
```
|
||||
- `parameters` contains key and value from manager command. It will includes
|
||||
- `command`: The action string message to trigger
|
||||
- `task_id`: The host task `UUID7` for task tracking
|
||||
- `cluster_id`: Unique Id of the cluster
|
||||
- `node` is constructed during action trigger
|
||||
|
||||
```python
|
||||
worker = Node(hostname, ip_address, "worker")
|
||||
```
|
||||
- `executor`: a method called `execute_command` from `streaming.util`. Invoking the method is as below
|
||||
|
||||
```python
|
||||
async for progress in initialize_controller(
|
||||
parameters, controller, execute_command
|
||||
):
|
||||
yield progress
|
||||
```
|
||||
|
||||
#### Designing an Action
|
||||
|
||||
Key points in designing an actions are:-
|
||||
- Step counter called `Prog`, it accepts
|
||||
- `total` as first parameter for number of host command that will be executed
|
||||
- `node` as name for 2nd parameter
|
||||
|
||||
```python
|
||||
prog = Prog(7, node.name)
|
||||
```
|
||||
- Triggering the host command using executor.
|
||||
|
||||
```python
|
||||
await executor("sudo k0s stop", parameters, node)
|
||||
```
|
||||
- Yield the response back to manager
|
||||
|
||||
```python
|
||||
yield {
|
||||
**prog.step("Installed helm"),
|
||||
"worker_token": worker_token,
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
#### Generate JWK private key for OIDC
|
||||
|
||||
```sh
|
||||
openssl genpkey -algorithm RSA -out private.pem -pkeyopt rsa_keygen_bits:2048
|
||||
openssl rsa -in private.pem -pubout -out public.pem
|
||||
```
|
||||
118
docs/docker.md
Normal file
118
docs/docker.md
Normal file
@ -0,0 +1,118 @@
|
||||
# Installation using Docker
|
||||
|
||||
This section explains how to build a container image & push to Gitea manually. You don't have to do this unless necessary, since there's already Gitea Action that build & publish latest image everytime new commit or pull request merged into the main branch.
|
||||
|
||||
## Build image & publish to Gitea
|
||||
|
||||
> Run the following commands in the project root folder, outside of devcontainer
|
||||
|
||||
Note: image version format: `<app version>-<date>.<hour>`. Example: `0.1.0-20250903.1429`
|
||||
|
||||
#### 1. Build container image
|
||||
|
||||
Note: delete previous builds first
|
||||
|
||||
```
|
||||
docker rmi -f <prev image id>
|
||||
docker builder prune -f
|
||||
docker build -t emgr-app .
|
||||
```
|
||||
|
||||
#### 2. Tag the image
|
||||
|
||||
```
|
||||
docker tag emgr-app:latest git.abyres.net/elixier/manager:<image version>
|
||||
```
|
||||
|
||||
#### 3. Authenticate to Gitea
|
||||
|
||||
Note: perform only once
|
||||
|
||||
3.1 Create access token:
|
||||
|
||||
- Go to Gitea > User > Settings > Applications > Manage Access Token > Generate New Token
|
||||
- Token Name: docker_push
|
||||
- Repository access: All
|
||||
- Permissions: Package > Read & Write
|
||||
- Generate Token > save token elsewhere
|
||||
|
||||
3.2 Login:
|
||||
|
||||
```
|
||||
docker login git.abyres.net -u <username> -p <access token>
|
||||
```
|
||||
|
||||
#### 4. Push the image
|
||||
|
||||
```
|
||||
docker push git.abyres.net/elixier/manager:<image version>
|
||||
```
|
||||
|
||||
## Install & run the app
|
||||
|
||||
#### 1. Create work folder
|
||||
|
||||
```
|
||||
mkdir -p ~/Docker/emgr
|
||||
cd ~/Docker/emgr
|
||||
```
|
||||
|
||||
#### 2. Copy docker-compose.yaml to work folder
|
||||
|
||||
> Use docker-compose.yaml in project root folder
|
||||
|
||||
```
|
||||
cp ~/Documents/emgr/docker-compose.yaml .
|
||||
```
|
||||
|
||||
#### 3. Create .env file
|
||||
|
||||
Fill in required vars, change accordingly:
|
||||
```
|
||||
EMGR_ADMIN_EMAIL=admin@example.com
|
||||
EMGR_ADMIN_PASSWORD=password
|
||||
APP_URL=http://192.168.0.219:3000
|
||||
BACKEND_URL=http://192.168.0.219:8000
|
||||
```
|
||||
Refer docker-compose.yaml for all configurable parameters.
|
||||
|
||||
#### 4. Find latest image version
|
||||
|
||||
- Go to Gitea > elixier/manager > Packages > manager
|
||||
- Put in .env file
|
||||
```
|
||||
GITHUB_SHA=0.1.0-20250903.1429
|
||||
```
|
||||
|
||||
#### 5. Install & run using docker compose
|
||||
```
|
||||
docker compose --profile deploy up -d
|
||||
```
|
||||
|
||||
## Stop, restart & uninstall
|
||||
|
||||
Control app lifecycle using docker compose. You may stop, start or restart all services, or specific service by its name: `app`, `db`, `opensearch` or `valkey`.
|
||||
|
||||
#### 1. Stop services
|
||||
|
||||
```
|
||||
docker compose --profile deploy stop [service name]
|
||||
```
|
||||
|
||||
#### 2. Start services
|
||||
|
||||
```
|
||||
docker compose --profile deploy start [service name]
|
||||
```
|
||||
|
||||
#### 3. Restart services
|
||||
|
||||
```
|
||||
docker compose --profile deploy restart [service name]
|
||||
```
|
||||
|
||||
#### 4. Uninstall all services, including their volumes
|
||||
|
||||
```
|
||||
docker compose --profile deploy down -v
|
||||
```
|
||||
125
docs/get-started.md
Normal file
125
docs/get-started.md
Normal file
@ -0,0 +1,125 @@
|
||||
# Getting Started
|
||||
|
||||
1. Open project folder in VS Code
|
||||
2. [Create .env file](#env-file)
|
||||
3. [Create database](#create-database)
|
||||
4. [Create global admin](#create-global-admin)
|
||||
5. [Generate plugin repo](#generate-plugin-repo)
|
||||
6. [Run app in VS Code](#run-app-in-vs-code)
|
||||
|
||||
## .env file
|
||||
|
||||
Create a new `.env` file in root folder. Refer to `BaseConfig` class in `emgr/settings.py` file for the variable names.
|
||||
|
||||
Set these variables for the host agent install script URL and plugin repo URL. Assuming your computer IP address is `192.168.0.2`:
|
||||
|
||||
- `REPO_MIRRORS=["http://192.168.0.2:8080"]`
|
||||
- `BACKEND_URL=http://192.168.0.2:8000`
|
||||
|
||||
These are default values for development:
|
||||
|
||||
- `OPENSEARCH_ADMIN_PASSWORD` (default: `mAckvusp5@`)
|
||||
- `POSTGRES_USER` (default: `em_db_user`)
|
||||
- `POSTGRES_PASSWORD` (default: `em_db_password`)
|
||||
- `POSTGRES_DB` (default: `em_db`)
|
||||
|
||||
## Create Global Admin
|
||||
|
||||
Run in devcontainer:
|
||||
|
||||
<pre>python cli.py create-admin --email admin@domain.com</pre>
|
||||
|
||||
## Create database
|
||||
|
||||
Run in devcontainer:
|
||||
|
||||
<pre>
|
||||
reflex db init
|
||||
reflex db makemigrations
|
||||
reflex db migrate
|
||||
</pre>
|
||||
|
||||
## Update database
|
||||
|
||||
If got changes to database schema, run in devcontainer:
|
||||
|
||||
<pre>
|
||||
reflex db makemigrations
|
||||
reflex db migrate
|
||||
</pre>
|
||||
|
||||
## Reset database
|
||||
|
||||
If want to reset database & start from scratch:
|
||||
|
||||
1. Run in terminal (outside of devcontainer):
|
||||
|
||||
<pre>
|
||||
dropdb -U em_db_user em_db -h localhost -p 5432
|
||||
createdb -U em_db_user em_db -h localhost -p 5432
|
||||
</pre>
|
||||
|
||||
Password is same as the `POSTGRES_PASSWORD` environment variable above. This command will drop & re-create the database.
|
||||
|
||||
2. Run inside devcontainer:
|
||||
|
||||
<pre>
|
||||
rm -fr alembic*
|
||||
reflex db init
|
||||
reflex db makemigrations
|
||||
reflex db migrate
|
||||
</pre>
|
||||
|
||||
This will re-initialize the database table schema.
|
||||
|
||||
## OpenSearch database indexes
|
||||
|
||||
OpenSearch database indexes are created automatically when the management tool push new data into the index.
|
||||
|
||||
Index list:
|
||||
- audit_log
|
||||
- cluster-metrics
|
||||
- deployment-metrics
|
||||
- host-metrics
|
||||
- host-logs
|
||||
- host-task-log
|
||||
|
||||
## Delete OpenSearch index
|
||||
|
||||
Note: Run commands below outside of devcontainer. Replace `{OPENSEARCH_ADMIN_PASSWORD}` with actual value from environment variable above.
|
||||
|
||||
To delete one index, e.g `host-metrics`:
|
||||
<pre>curl -X DELETE -u admin:{OPENSEARCH_ADMIN_PASSWORD} -k https://localhost:9200/host-metrics</pre>
|
||||
|
||||
To delete multiple indexes, specify as comma-separated values:
|
||||
<pre>curl -X DELETE -u admin:{OPENSEARCH_ADMIN_PASSWORD} -k https://localhost:9200/audit_log,cluster-metrics,deployment-metrics,host-metrics,host-logs,host-task-log</pre>
|
||||
|
||||
## Generate plugin repository
|
||||
|
||||
Run in devcontainer:
|
||||
|
||||
<pre>python cli.py build-repo</pre>
|
||||
|
||||
The repository metadata file can be accessed at `http://localhost:8080/metadata.json`
|
||||
|
||||
## Run app in VS Code
|
||||
|
||||
To run in development mode, either:
|
||||
|
||||
- `Ctrl + Shift + P` > `Run Task` > `Run`, or
|
||||
- Run in devcontainer: `reflex run`
|
||||
|
||||
To run in production mode, run in devcontainer:
|
||||
|
||||
<pre>reflex run --env prod</pre>
|
||||
|
||||
# Devcontainer Components URL
|
||||
|
||||
| Container | Component | URL |
|
||||
| --- | --- | --- |
|
||||
| Reflex | Frontend | http://localhost:3000 |
|
||||
| | Backend | http://localhost:8000 |
|
||||
| | Swagger | http://localhost:8000/docs |
|
||||
| Opensearch | API | https://localhost:9200 |
|
||||
| | Dashboard | http://localhost:5601 |
|
||||
| Nginx | Plugin repo | http://localhost:8080 |
|
||||
398
docs/installation-guide.md
Normal file
398
docs/installation-guide.md
Normal file
@ -0,0 +1,398 @@
|
||||
# 🛠️ Installation Guide
|
||||
|
||||
This document describes how to deploy the management tool web app to a Kubernetes environment.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Host Requirements](#host-requirements)
|
||||
- [Software Requirements](#software-requirements)
|
||||
- [Host Setup](#host-setup)
|
||||
- [Prepare host environment](#prepare-host-environment)
|
||||
- [Install CLI tools](#install-cli-tools)
|
||||
- [Install k3s](#install-k3s)
|
||||
- [Install ArgoCD](#install-argocd)
|
||||
- [Git Repo Access](#git-repo-access)
|
||||
- [Deploy App](#deploy-app)
|
||||
- [App config values](#app-config-values)
|
||||
- [deployment.yaml](#deploymentyaml)
|
||||
- [Apply deployment](#apply-deployment)
|
||||
- [Deploy Secrets](#deploy-secrets)
|
||||
- [Secret generation tools](#secret-generation-tools)
|
||||
- [Secret config values](#secret-config-values)
|
||||
- [secrets.yaml](#secretsyaml)
|
||||
- [Apply secrets](#apply-secrets)
|
||||
- [Admin Account](#admin-account)
|
||||
- [Add admin account](#add-admin-account)
|
||||
- [Verification](#verification)
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Host Requirements
|
||||
|
||||
| Component | Minimum Requirement |
|
||||
| --- | --- |
|
||||
| OS | Centos Stream 9 / Almalinux 9 / Rocky Linux 9 |
|
||||
| CPU | 4 cores |
|
||||
| Memory | 8 GB |
|
||||
| Storage | 50 GB |
|
||||
| Network | Public IP or DNS-resolvable hostname |
|
||||
|
||||
### Software Requirements
|
||||
|
||||
| Software | Version |
|
||||
| --- | --- |
|
||||
| k3s | ≥ 1.33 |
|
||||
| ArgoCD | ≥ 3.1 |
|
||||
| Helm | ≥ 3.19 |
|
||||
| kubeseal | ≥ 0.33 |
|
||||
|
||||
---
|
||||
|
||||
## Host Setup
|
||||
|
||||
### Prepare host environment:
|
||||
```bash
|
||||
# Install dependencies
|
||||
dnf install tar -y
|
||||
|
||||
# Setup SELinux
|
||||
setenforce 0
|
||||
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
|
||||
|
||||
# Setup firewall
|
||||
firewall-cmd --permanent --add-port=6443/tcp
|
||||
firewall-cmd --permanent --add-port=8472/udp
|
||||
firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16
|
||||
firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16
|
||||
firewall-cmd --permanent --add-port=30000-32767/tcp
|
||||
firewall-cmd --reload
|
||||
```
|
||||
|
||||
### Install CLI tools:
|
||||
```bash
|
||||
# Install helm
|
||||
mkdir -p /tmp/helm
|
||||
cd /tmp/helm
|
||||
curl -o helm.tar.gz https://get.helm.sh/helm-v3.19.0-linux-amd64.tar.gz
|
||||
tar xvf helm.tar.gz
|
||||
mv linux-amd64/helm /usr/local/bin/helm
|
||||
cd ~
|
||||
rm -fr /tmp/helm
|
||||
|
||||
# Install kubeseal
|
||||
curl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.33.1/kubeseal-0.33.1-linux-amd64.tar.gz"
|
||||
tar -xvzf kubeseal-0.33.1-linux-amd64.tar.gz kubeseal
|
||||
sudo install -m 755 kubeseal /usr/local/bin/kubeseal
|
||||
rm -fr kubeseal*
|
||||
```
|
||||
|
||||
### Install k3s:
|
||||
```bash
|
||||
curl -sfL https://get.k3s.io | sh -s -
|
||||
```
|
||||
|
||||
### Install ArgoCD:
|
||||
```bash
|
||||
helm --kubeconfig=/etc/rancher/k3s/k3s.yaml install -n argocd --create-namespace argocd --version 8.6.1 --set server.service.type=NodePort oci://ghcr.io/argoproj/argo-helm/argo-cd
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Git Repo Access
|
||||
|
||||
Configure ArgoCD to access to this Git repository.
|
||||
|
||||
1. Generate SSH key pairs.
|
||||
```bash
|
||||
ssh-keygen -t rsa -b 4096 -C "root@hostname"
|
||||
```
|
||||
|
||||
2. Add SSH key to Gitea.
|
||||
|
||||
1. Login to https://git.abyres.net.
|
||||
2. Go to the top right menu > Settings > SSH / GPG Keys.
|
||||
3. Click ‘Add Key’:
|
||||
|
||||
- Key Name: `hostname`
|
||||
- Content: *(paste content from ~/.ssh/id_rsa.pub)*
|
||||
|
||||
3. Add Gitea as a known host in ArgoCD.
|
||||
|
||||
1. Run this command:
|
||||
```bash
|
||||
ssh-keyscan -p 30025 gitssh.abyres.net
|
||||
```
|
||||
2. Copy the output, for example:
|
||||
```
|
||||
[gitssh.abyres.net]:30025 ssh-rsa …
|
||||
```
|
||||
3. Login to ArgoCD in the k3s cluster.
|
||||
4. Go to Settings > Repository certificates and known hosts > Add SSH Known Hosts
|
||||
|
||||
- SSH Known Host Data: *(paste ssh-keyscan output)*
|
||||
|
||||
---
|
||||
|
||||
## Deploy App
|
||||
|
||||
### App config values
|
||||
|
||||
- IMAGE_TAG = `1.1.0-build.1866.768ef2c8`
|
||||
- SSH_PRIVATE_KEY = *(paste content from ~/.ssh/id_rsa)*
|
||||
|
||||
|
||||
### deployment.yaml
|
||||
|
||||
Create a `deployment.yaml` file with content below and replace the variables with config values above.
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: AppProject
|
||||
metadata:
|
||||
name: emgr
|
||||
namespace: argocd
|
||||
spec:
|
||||
sourceRepos:
|
||||
- ssh://git@gitssh.abyres.net:30025/elixier/manager.git
|
||||
destinations:
|
||||
- namespace: '*'
|
||||
name: '*'
|
||||
server: https://kubernetes.default.svc
|
||||
|
||||
---
|
||||
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: repo-gitea
|
||||
namespace: argocd
|
||||
labels:
|
||||
argocd.argoproj.io/secret-type: repository
|
||||
stringData:
|
||||
type: git
|
||||
name: gitea
|
||||
project: default
|
||||
url: ssh://git@gitssh.abyres.net:30025/elixier/manager.git
|
||||
sshPrivateKey: |
|
||||
<SSH_PRIVATE_KEY>
|
||||
|
||||
---
|
||||
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: repo-gitea-emgr
|
||||
namespace: argocd
|
||||
labels:
|
||||
argocd.argoproj.io/secret-type: repository
|
||||
stringData:
|
||||
type: git
|
||||
name: gitea
|
||||
project: emgr
|
||||
url: ssh://git@gitssh.abyres.net:30025/elixier/manager.git
|
||||
sshPrivateKey: |
|
||||
<SSH_PRIVATE_KEY>
|
||||
|
||||
---
|
||||
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: ApplicationSet
|
||||
metadata:
|
||||
name: infrastructure
|
||||
namespace: argocd
|
||||
spec:
|
||||
generators:
|
||||
- list:
|
||||
elements:
|
||||
- tag: <IMAGE_TAG>
|
||||
template:
|
||||
metadata:
|
||||
name: infrastructure
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: default
|
||||
source:
|
||||
path: k8s/platform/production
|
||||
repoURL: ssh://git@gitssh.abyres.net:30025/elixier/manager.git
|
||||
targetRevision: main
|
||||
kustomize:
|
||||
patches:
|
||||
- target:
|
||||
group: argoproj.io
|
||||
version: v1alpha1
|
||||
kind: Application
|
||||
name: emgr
|
||||
patch: |
|
||||
- op: add
|
||||
path: /spec/syncPolicy
|
||||
value:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
- op: replace
|
||||
path: /spec/source/kustomize/images
|
||||
value:
|
||||
- app=git.abyres.net/elixier/manager:{{tag}}
|
||||
- op: add
|
||||
path: /spec/source/kustomize/patches
|
||||
value:
|
||||
- target:
|
||||
group: bitnami.com
|
||||
kind: SealedSecret
|
||||
name: app-custom-config
|
||||
namespace: emgr
|
||||
patch: |
|
||||
- op: replace
|
||||
path: /spec/encryptedData
|
||||
value: {}
|
||||
- op: add
|
||||
path: /spec/ignoreDifferences
|
||||
value:
|
||||
- group: bitnami.com
|
||||
kind: SealedSecret
|
||||
name: app-custom-config
|
||||
jsonPointers:
|
||||
- /spec/encryptedData
|
||||
- target:
|
||||
group: bitnami.com
|
||||
kind: SealedSecret
|
||||
name: app-certs
|
||||
namespace: emgr
|
||||
patch: |
|
||||
- op: replace
|
||||
path: /spec/encryptedData
|
||||
value: {}
|
||||
- op: add
|
||||
path: /spec/ignoreDifferences
|
||||
value:
|
||||
- group: bitnami.com
|
||||
kind: SealedSecret
|
||||
name: app-certs
|
||||
jsonPointers:
|
||||
- /spec/encryptedData
|
||||
|
||||
- target:
|
||||
group: argoproj.io
|
||||
version: v1alpha1
|
||||
kind: Application
|
||||
name: cloudflared
|
||||
patch: |
|
||||
- op: replace
|
||||
path: /spec/syncPolicy/automated/enabled
|
||||
value: false
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
syncPolicy:
|
||||
automated:
|
||||
enabled: true
|
||||
syncOptions:
|
||||
- ServerSideApply=true
|
||||
```
|
||||
|
||||
### Apply deployment
|
||||
|
||||
```bash
|
||||
k3s kubectl apply -f deployment.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deploy Secrets
|
||||
|
||||
### Secret generation tools
|
||||
|
||||
1. Fernet key = Python `cryptography.fernet` library
|
||||
2. JWT secret key = Python `secrets` library
|
||||
3. Public & private .pem certificates = `openssl` command
|
||||
|
||||
### Secret config values
|
||||
|
||||
- NODE_IP = 192.168.0.10 (*assumption)
|
||||
- FERNET_KEY = *(generated using tool above)*
|
||||
- JWT_SECRET_KEY = *(generated using tool above)*
|
||||
- PRIVATE_PEM = *(generated using tool above)*
|
||||
- PUBLIC_PEM = *(generated using tool above)*
|
||||
|
||||
### secrets.yaml
|
||||
|
||||
Create a `secrets.yaml` file with content below and replace the variables with config values above.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: app-custom-config
|
||||
namespace: emgr
|
||||
type: Opaque
|
||||
stringData:
|
||||
APP_URL: http://<NODE_IP>:30430
|
||||
FRONTEND_URL: http://<NODE_IP>:30430
|
||||
BACKEND_URL: http://<NODE_IP>:30480
|
||||
FERNET_KEY: <FERNET_KEY>
|
||||
JWT_SECRET_KEY: <JWT_SECRET_KEY>
|
||||
|
||||
---
|
||||
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: app-certs
|
||||
namespace: emgr
|
||||
type: Opaque
|
||||
stringData:
|
||||
private.pem: |
|
||||
<PRIVATE_PEM>
|
||||
|
||||
public.pem: |
|
||||
<PUBLIC_PEM>
|
||||
```
|
||||
|
||||
### Apply secrets
|
||||
|
||||
```bash
|
||||
kubeseal --kubeconfig=/etc/rancher/k3s/k3s.yaml -f secrets.yaml | k3s kubectl apply -f -
|
||||
```
|
||||
|
||||
Note: make sure the `sealed-secrets-controller` pod is already running before running above command.
|
||||
|
||||
---
|
||||
|
||||
## Admin Account
|
||||
|
||||
You are required to create an initial global admin account in order to login to the system. Assuming this is the credentials:
|
||||
|
||||
- email: `admin@example.com`
|
||||
- password: `adminpass123`
|
||||
|
||||
### Add admin account
|
||||
|
||||
1. Get one of the running backend pods name using this command:
|
||||
```bash
|
||||
k3s kubectl get pods -n emgr -l app=backend
|
||||
# Example pod name = backend-784d796dd9-647pw
|
||||
```
|
||||
|
||||
2. Create the initial global admin:
|
||||
```bash
|
||||
k3s kubectl exec -it backend-784d796dd9-647pw -n emgr -- uv run cli.py create-admin --email=admin@example.com --password=adminpass123
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
Assuming the NODE_IP = 192.168.0.10, the management tool components can be accessed at the following addresses:
|
||||
|
||||
- frontend:
|
||||
```
|
||||
http://192.168.0.10:30430
|
||||
```
|
||||
- backend:
|
||||
```
|
||||
http://192.168.0.10:30480
|
||||
```
|
||||
416
docs/srs.md
Normal file
416
docs/srs.md
Normal file
@ -0,0 +1,416 @@
|
||||
## Software Requirement Specification (SRS) Document
|
||||
### Elixier Management Tool Development Project
|
||||
|
||||
---
|
||||
|
||||
### 1. Introduction
|
||||
|
||||
#### 1.1 Overview
|
||||
|
||||
This document covers functional requirements, non-functional requirements,
|
||||
system architecture, and design constraints for the Elixier Management Tool
|
||||
app.
|
||||
|
||||
#### 1.2 Purpose
|
||||
|
||||
The purpose of this document is to provide a detailed Software Requirements
|
||||
Specification (SRS) for the development of the **Elixier Management Tool**
|
||||
application. This application will provide users with a GUI to manage the
|
||||
Elixier data analytics platform.
|
||||
|
||||
#### 1.3 Scope
|
||||
|
||||
Elixier Management Tool will be a web-based solution designed to help
|
||||
platform provider to manage multiple tenants, help tenants to manage their data
|
||||
platform supporting multiple Kubernetes clusters, and also to help tenant
|
||||
admins to manage the users of the platform.
|
||||
|
||||
---
|
||||
|
||||
### 2. System Overview
|
||||
|
||||
Elixier is an open-source data analytics plaform, which consists of a
|
||||
collection of services (such as Airflow, Superset, Opensearch etc.) within a
|
||||
Kubernetes cluster that has been configured to work with each other, to allow
|
||||
tenants to store, perform ETL and analyze their data.
|
||||
|
||||
Current method to configure & deploy services within this data platform is by
|
||||
using Helm chart. The configuration files need to be edited manually & helm
|
||||
command has to be executed via terminal to deploy to the Kubernetes cluster.
|
||||
|
||||
The **Elixier Management Tool** app will provide a GUI to allow management
|
||||
tasks described above to be performed via a web-based interface.
|
||||
|
||||
#### 2.1 System Goals
|
||||
|
||||
The goal is to have a web based graphical management tool for simplifying the
|
||||
processes of:
|
||||
|
||||
- Server monitoring
|
||||
- Automation of the process of installation, management, upgrade and
|
||||
uninstallation of server side software
|
||||
- Automation and centralization of configuration management of software
|
||||
- Automation of configuration management of software with many interconnected
|
||||
components.
|
||||
- Self service portal for customers to deploy software on their own
|
||||
environment
|
||||
- Extending with new service deployment template for any new software
|
||||
that need to be deployed.
|
||||
|
||||
#### 2.2 Success Criteria
|
||||
|
||||
The system would be considered successful when following criteria are met:
|
||||
|
||||
- Multiple tenants able to use the software to manage their own infrastructure
|
||||
- Users are able to deploy, configure and manage services easily
|
||||
- Developers are able to create new service template plugin and make it available
|
||||
for users with minimal friction, and users are able to use it.
|
||||
- The system can be used for on-premises client and on-cloud clients
|
||||
|
||||
---
|
||||
|
||||
### 3. Functional Requirements
|
||||
|
||||
#### 3.1 General Requirements
|
||||
- The system shall use asynchronous I/O pattern.
|
||||
- The system backend shall be written using Python, using FastAPI framework as a
|
||||
base
|
||||
- The system frontend shall be written using React
|
||||
- The system shall use OpenID Connect (OIDC) and Security Assertion Markup
|
||||
Language (SAML) for the Single Sign-On (SSO) feature, for users to access
|
||||
services on the data platform.
|
||||
- The system shall integrate with Kubernetes via API server to deploy services.
|
||||
- The system shall integrate with Authentik to handle user identity management.
|
||||
- The system shall integrate with RabbitMQ for asynchronous message brokering
|
||||
between components.
|
||||
- The system shall be installable on-premises, to manage hosts and clusters
|
||||
behind secured network.
|
||||
- The system shall be designed with preference to monolithic approach to
|
||||
backend development to minimize complexity
|
||||
|
||||
#### 3.1 Authentication Management
|
||||
|
||||
- The system shall have user, group and role management.
|
||||
- The system shall have 3 groups of users: global admin, tenant admin & tenant
|
||||
user. The global admin must be able to manage all tenants, while tenant admin
|
||||
only able to manage the tenant they are inside.
|
||||
- The system shall support multiple tenants, and each tenant may add one or
|
||||
more
|
||||
clusters.
|
||||
- The system shall support API key authentication to backend.
|
||||
- The system shall provide audit logs of key actions done againts host,
|
||||
clusters and
|
||||
tenant
|
||||
|
||||
#### 3.2 Authorization Management
|
||||
- The system shall authorize access to cluster components through the management
|
||||
tool.
|
||||
- The system shall use OIDC grant or SAML claim to identify user role membership
|
||||
- The system shall allow tenant to use their own SAML/OIDC provider.
|
||||
- The system shall log authorization requests to an audit log
|
||||
- The system shall provide list of dataset in the cluster components through
|
||||
the management tool.
|
||||
- The system shall allow tagging of dataset through the management tool
|
||||
- The system shall allow management of tags through the management tool
|
||||
- The system shall manage authorization rules to dataset through the management
|
||||
tool, using both resource based ACL and tag based ACL.
|
||||
- The system shall provide list of permissions that can be granted to cluster
|
||||
components through the management tool
|
||||
- The system shall allow assignment of permission grants through the management
|
||||
tool.
|
||||
|
||||
#### 3.3 Tenant Management
|
||||
- The system shall allow global admins to create, update, and delete tenants.
|
||||
- The system shall allow tenant admins to manage tenant-specific settings,
|
||||
including branding, default permissions, and notification preferences.
|
||||
- The system shall provide a mechanism for tenant onboarding, including guided
|
||||
setup wizards for new tenants.
|
||||
- The system shall enable tenant admins to define custom quotas for resources
|
||||
such as CPU, memory, and storage within their clusters.
|
||||
- The system shall maintain isolation between tenants to ensure no data leakage
|
||||
or unauthorized access occurs across tenants.
|
||||
- The system shall allow tenant admins to view and manage all clusters and
|
||||
hosts associated with their tenant.
|
||||
- The system shall enable tenant admins to create and manage user accounts
|
||||
specific to their tenant, including assigning roles and permissions.
|
||||
- The system shall provide tenant-specific dashboards that display key metrics
|
||||
such as resource utilization, service uptime, and user activity.
|
||||
- The system shall allow tenant admins to customize alerts and notifications
|
||||
for their specific tenant environment.
|
||||
- The system shall enable tenants to associate billing or usage information for
|
||||
their resources, integrating with external billing systems if necessary.
|
||||
- The system shall support tenant-level backup and restore features for
|
||||
critical resources such as configurations, datasets, and services.
|
||||
- The system shall allow tenants to view detailed audit logs of actions
|
||||
performed by their users, including timestamps and IP addresses.
|
||||
- The system shall provide mechanisms to export and import tenant
|
||||
configurations for migration or backup purposes.
|
||||
|
||||
#### 3.3 Host Management
|
||||
- The system shall be able to help tenants to add hosts to be managed and
|
||||
monitored through the management tool. Host ownership is by tenant.
|
||||
- The system shall monitor and collect resource consumption statistics of hosts
|
||||
and visualize in the management tool, which includes CPU, RAM, network, process
|
||||
list, systemd logs, service logs, error logs.
|
||||
- The system shall be able to manage and modify host service configuration and
|
||||
deploy configuration to the hosts
|
||||
- The system shall provide versioning of configuration for roll-back
|
||||
- The system shall be able to roll-back service configuration to a version
|
||||
either by
|
||||
version name or by date.
|
||||
- The system shall manage hosts through agents that connect to management tool
|
||||
through secured HTTPS connection.
|
||||
- The system shall provide dashboard to monitor host being managed
|
||||
|
||||
#### 3.4 Kubernetes Cluster Management
|
||||
|
||||
- The system shall be able help tenant to deploy Kubernetes cluster to hosts
|
||||
that it manages.
|
||||
- The system shall use k0s to deploy new Kubernetes cluster
|
||||
- The system shall work with new or existing, CNCF-certified Kubernetes cluster
|
||||
through adding cluster config file.
|
||||
- The system shall be able to monitor Kubernetes cluster on at least
|
||||
following metrics: host resource consumption, host resource request/limit
|
||||
consumption, number of containers, pod, deployment, statefulset, PVC, PV
|
||||
- The system shall be able to deploy Kubernetes components through uploading
|
||||
Kubernetes YAML file
|
||||
|
||||
#### 3.5 Service Management
|
||||
|
||||
- The system shall provide list of services that can be deployed on Kubernetes
|
||||
cluster
|
||||
- The system shall allow tenant users to deploy services to the clusters.
|
||||
Services may be deployed single or multiple instances to a cluster
|
||||
- The system shall provide a dashboard for each services, which list down the
|
||||
following: shortcut url to access key user interface of deployed service,
|
||||
key service logs, key audit logs, list of components that are up / down,
|
||||
resource
|
||||
consumption of services.
|
||||
- The system shall provide alerts when services encounters issues such as going
|
||||
down, having resource issues, slow response.
|
||||
- The system shall provide a mechanism to update service configuration and
|
||||
deploy
|
||||
the configuration update. Configuration may either be ConfigMap, Secret, or
|
||||
config
|
||||
file stored in PV. After updating configuration, service must be able to be
|
||||
restarted
|
||||
either automatically or waiting for user confirmation
|
||||
- The system shall provide versioning of configuration for roll-back
|
||||
- The system shall be able to roll-back service configuration to a version
|
||||
either by
|
||||
version name or by date.
|
||||
- The system shall have a plugin mechanism for registering new service
|
||||
template.
|
||||
New service template shall provide at least following hooks: install,
|
||||
uninstall,
|
||||
restart, metric collection, configuration update, get service urls, upload file
|
||||
to service,
|
||||
download file from service, list files in service. New service template shall
|
||||
be implemented as a Python class, loaded dynamically from a folder.
|
||||
|
||||
#### 3.6 Secret Management
|
||||
|
||||
- The system shall provide a mechanism to manage secrets and store it encrypted
|
||||
in database.
|
||||
- The system shall provide a mechanism to generate secret based in random
|
||||
generator
|
||||
- The system shall support secret versioning, allowing rollback to a previous
|
||||
version of a secret.
|
||||
- The system shall encrypt all secrets in transit and at rest using
|
||||
industry-standard encryption protocols.
|
||||
- The system shall provide a role-based access mechanism to manage and retrieve
|
||||
secrets securely.
|
||||
- The system shall integrate with Kubernetes Secrets to synchronize secrets
|
||||
into clusters.
|
||||
|
||||
#### 3.6 Object Storage Bucket Management
|
||||
|
||||
- The system shall provide a mechanism to manage files in S3 bucket. S3 bucket
|
||||
may reside in AWS or Minio.
|
||||
- The system shall provide a mechanism to manage files in Azure Object Storage.
|
||||
- The system shall provide a mechanism to manage access keys to buckets
|
||||
- The system shall support tagging and categorization of files within buckets
|
||||
for better organization.
|
||||
- The system shall allow uploading, downloading, and deleting files through the
|
||||
management tool's GUI.
|
||||
- The system shall display file metadata such as size, type, upload date, and
|
||||
owner.
|
||||
|
||||
#### 3.6 PostgreSQL Service Management
|
||||
- The system shall provide a mechanism to deploy PostgreSQL database cluster,
|
||||
as a service template plugin.
|
||||
- The system shall provide dashboard to monitor PostgreSQL databases, including
|
||||
performance metrics like query times, CPU usage, and memory consumption.
|
||||
- The system shall allow tenant users to create and delete PostgreSQL instances
|
||||
within their clusters.
|
||||
- The system shall provide automated backups for PostgreSQL databases with
|
||||
user-configurable backup intervals.
|
||||
- The system shall allow restoration of databases from backups, either fully or
|
||||
partially.
|
||||
- The system shall provide a mechanism for managing PostgreSQL configuration
|
||||
files and applying updates dynamically.
|
||||
|
||||
#### 3.6 Trino Service Management
|
||||
- The system shall provide a mechanism to deploy Trino query engine cluster,
|
||||
including for High Availability,
|
||||
as a service template plugin.
|
||||
- The system shall provide a GUI to manage Trino configurations, including
|
||||
catalog and connector settings.
|
||||
- The system shall allow monitoring of query execution, including status,
|
||||
duration, and resource usage.
|
||||
- The system shall support user and role-based access to Trino clusters,
|
||||
integrating with the system's identity management solution.
|
||||
- The system shall provide audit logs of executed queries and configuration
|
||||
changes for compliance.
|
||||
- The system shall provide data catalog GUI to list down dataset in Trino
|
||||
- The system shall provide GUI for managing access control in Trino through tag
|
||||
based policy
|
||||
|
||||
#### 3.7 Superset Service Management
|
||||
- The system shall provide a mechanism to deploy Apache Superset cluster,
|
||||
including for High Availability,
|
||||
as a service template plugin.
|
||||
- The system shall allow users to create and manage Superset dashboards and
|
||||
visualizations through the GUI.
|
||||
- The system shall monitor Superset performance, including response times and
|
||||
resource usage.
|
||||
- The system shall enable integration of Superset with various data sources,
|
||||
including PostgreSQL and Trino.
|
||||
- The system shall provide backup and restoration mechanisms for Superset
|
||||
configurations and metadata.
|
||||
- The system shall support scaling of Superset worker nodes dynamically based
|
||||
on workload.
|
||||
|
||||
#### 3.8 Airflow Service Management
|
||||
- The system shall provide a mechanism to deploy Apache Airflow cluster,
|
||||
including for High Availability,
|
||||
as a service template plugin.
|
||||
- The system shall allow tenant users to create and manage Directed Acyclic
|
||||
Graphs (DAGs) for workflows.
|
||||
- The system shall provide real-time monitoring of DAG execution, including
|
||||
status, logs, and resource usage.
|
||||
- The system shall support scaling of Airflow worker nodes dynamically based on
|
||||
workload.
|
||||
- The system shall integrate with the system's secret management module to
|
||||
securely pass sensitive data to Airflow workflows.
|
||||
- The system shall provide mechanisms for managing Airflow configuration and
|
||||
environment variables through the GUI.
|
||||
|
||||
---
|
||||
|
||||
### 4. Non-Functional Requirements
|
||||
|
||||
#### 4.1 Usability
|
||||
|
||||
- The system shall provide a user-friendly, intuitive interface that is easy to
|
||||
navigate.
|
||||
|
||||
#### 4.2 Security
|
||||
|
||||
- The system shall implement strong encryption for data in transit (SSL/TLS).
|
||||
- The system shall implement role-based access control (RBAC) to ensure data
|
||||
security and privacy.
|
||||
|
||||
#### 4.3 Reliability
|
||||
|
||||
- The system shall have appropriate failover mechanisms in place to ensure
|
||||
high uptime, preferably through Raft consensus.
|
||||
- The system shall be able to handle network outages and resume data
|
||||
synchronization once the network is restored.
|
||||
|
||||
#### 4.4 Performance
|
||||
|
||||
- The system shall load the dashboard in under 5 seconds for standard users.
|
||||
- Data queries should return results within 3 seconds for queries with a
|
||||
dataset of up to 1 GB.
|
||||
|
||||
#### 4.5 Compatibility
|
||||
|
||||
- The system shall be compatible with the latest versions of Chrome, Firefox, and
|
||||
Edge browsers.
|
||||
- The system shall be built using responsive UI and work across devices (desktops,
|
||||
tablets, and smartphones).
|
||||
|
||||
---
|
||||
|
||||
### 6. System Architecture
|
||||
|
||||
#### 6.1 Overview
|
||||
|
||||
The system will be built using an architecture with the following
|
||||
components:
|
||||
|
||||
- **Frontend**: React-based & Tailwind-based frameworks, for example: Next.js
|
||||
and Flowbite
|
||||
- **Backend**: FastAPI for REST API endpoints, websocket & service template plugin.
|
||||
- **Database**: PostgreSQL for structured data storage.
|
||||
- **Authentication**: Authentik with OpenID Connect (OIDC) for secure login and Single
|
||||
Sign-On (SSO).
|
||||
- **Message Broker**: RabbitMQ for asynchronous message broker & queue.
|
||||
|
||||
Backend shall prefer monolithic architecture to minimize complexity.
|
||||
|
||||
#### 6.2 Process Flow
|
||||
|
||||
The following is the high-level process flow of the system:
|
||||
|
||||
1. Global admin adds a new tenant and add a user as admin in tenant.
|
||||
2. Tenant admin configures tenant account with organizational metadata.
|
||||
3. For tenant with existing Kubernetes cluster, tenant admin will add the cluster by
|
||||
providing `kubeconfig` file to the cluster.
|
||||
4. For tenant that doesn't have Kubernetes cluster, tenant admin will setup a new one
|
||||
by adding hosts and then creating cluster using the hosts.
|
||||
5. Tenant admin selects & configures credentials & resource quotas of services to be
|
||||
deployed on the cluster.
|
||||
6. Tenant admin select a service and deploys the services on the cluster.
|
||||
7. Tenant admin adds users to its tenant.
|
||||
8. Tenant user logs in and accesses installed services on a cluster to perform data
|
||||
processing & analytics activities.
|
||||
9. Tenant admin monitors the health of clusters & services, including troubleshooting
|
||||
problems arising.
|
||||
10. Tenant admin performs regular maintenance & security updates on the services,
|
||||
clusters & hosts OS.
|
||||
|
||||
---
|
||||
|
||||
### 7. Design Constraints
|
||||
|
||||
- The system shall be web-based and designed to be responsive for both desktop and
|
||||
mobile views.
|
||||
- The system shall support future integration with additional Kubernetes services
|
||||
deployment with minimal effort.
|
||||
- The system shall adhere to modern web security standards, including secure data
|
||||
transmission and secure API design.
|
||||
|
||||
---
|
||||
|
||||
### 8. Appendices
|
||||
|
||||
#### 8.1 Glossary
|
||||
|
||||
- **Elixier**: An open-source data analytics plaform.
|
||||
- **Service**: The components of the data platform, such as Airflow, Superset,
|
||||
Opensearch, Git etc.
|
||||
- **Platform Provider**: A business entity that provides hosting
|
||||
service to install, configure and host this management tool, and optionally
|
||||
install, configure, or host the Kubernetes cluster for the data platform.
|
||||
- **Tenant**: A company or organization that uses this management tool to
|
||||
manage their data platform.
|
||||
- **Dashboard**: A user interface that provides an overview of data platform
|
||||
tenants, clusters, services and users.
|
||||
|
||||
#### 8.2 Definitions, Acronyms, and Abbreviations
|
||||
|
||||
- **API**: Application Programming Interface
|
||||
- **UI**: User Interface
|
||||
- **GUI**: Graphical User Interface
|
||||
- **UX**: User Experience
|
||||
- **SLA**: Service Level Agreement
|
||||
- **JSON**: JavaScript Object Notation
|
||||
- **DB**: Database
|
||||
- **ETL**: Extract, Transform, Load
|
||||
- **SaaS**: Software as a Service
|
||||
|
||||
#### 8.3 References
|
||||
|
||||
- OpenAPI Specification v3
|
||||
246
docs/test-procedure-spec.md
Normal file
246
docs/test-procedure-spec.md
Normal file
@ -0,0 +1,246 @@
|
||||
# **Elixier Management Tool Test Procedure Specification**
|
||||
|
||||
1. # **Test Plan**
|
||||
|
||||
**Objectives:**
|
||||
To provide a detail app testing procedure for the development of the Elixier Management Tool application. This application will provide users with a GUI to manage the Elixier data analytics platform.
|
||||
|
||||
**Scope:**
|
||||
Elixier Management Tool will be a web-based solution designed to help platform providers to manage multiple tenants, help tenants to manage their data platform supporting multiple Kubernetes clusters, and also to help tenant admins to manage the users of the platform.
|
||||
|
||||
**Test environment:**
|
||||
|
||||
**Roles:**
|
||||
**Tools:**
|
||||
|
||||
2. # **Test Case Preparation: Global Admin Configuration**
|
||||
|
||||
This section outlines the creation of Global Admin before proceeding to the test cases. Global admin serves as an administrator who manages tenant and has full access to the management tool. This is executed on host.
|
||||
|
||||
| Test Case Name | Test Steps | Expected Results | Results |
|
||||
| :---- | :---- | :---- | :---- |
|
||||
| Create global admin | On the terminal, run the command prompt of “python cli.py create-admin \--email your-email” and replace the “your-email” with the desired email. Then, input password is needed to complete the global admin setup | Admin account created successfully: “your-email” | Global admin account is created successfully |
|
||||
|
||||
3. # **Execute the Test Cases**
|
||||
|
||||
**Test Case Documentation: Website Deployment on Linux**
|
||||
|
||||
This section outlines the formal test cases for deploying a web application in Linux environments (Ubuntu/Fedora) using Gitea, Docker, and Visual Studio Code. Each test case includes a description, steps to execute, expected results, and space for pass/fail status. This action is executed on host.
|
||||
|
||||
| Description | Test Steps | Expected Result | Status |
|
||||
| :---- | :---- | :---- | :---- |
|
||||
| Clone repository using HTTPS | Run: git clone https://git.abyres.net/elixier/manager.git | Repository is cloned without errors | Repository is cloned without errors |
|
||||
| Clone repository using SSH | Run: git clone ssh://git@gitssh.abyres.net:30025/elixier/manager.git | Repository is cloned without errors using SSH | |
|
||||
| Configure authentication for HTTPS access | Create PAT from Gitea \> Settings \> Applications and store it | PAT is generated and stored securely | PAT is generated and stored securely |
|
||||
| Open the project in VS Code | Navigate to folder and run: code . | Project opens in VS Code successfully | Project opens in VS Code successfully |
|
||||
| Authenticate with PAT in VS Code | Enter username and PAT when prompted | Authenticated successfully and project loads | Authenticated successfully and project loads |
|
||||
| Rebuild and reopen in Dev Container | Press Ctrl+Shift+P \> Select: Dev Containers: Rebuild and Reopen in Container | Dev container is built and environment is loaded | Dev container is built and environment is loaded |
|
||||
| Ensure Docker Desktop is running | Check Docker Desktop is running before rebuilding container (especially on WSL) | Docker is confirmed running | Docker is confirmed running |
|
||||
| Run the web application inside container | Ctrl+Shift+P \> Tasks: Run Task \> Select: Start Server | Application starts and localhost URL is displayed | Application starts and localhost URL is displayed |
|
||||
|
||||
## Test Scenario: User as Global Admin
|
||||
|
||||
An administrator from the client who manages tenants. Has full access to the
|
||||
management tool. Has ability to manage clusters and services on behalf of a Tenant
|
||||
Admin.
|
||||
|
||||
1. Add New Global Admin \- As a global admin, I want to add new global admin.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Global Admin | Logged in as Global Admin main account Main account: admin1@localhost.local | Navigate to Administrators page on the left side bar Click the Administrators Click the “+ Add Admin” button to create new Global Admin An interface “Add Admin” with email and password input appear Enter email and password for new Global Admin account \*Sample Email: [admin2@localhost.local](mailto:admin2@localhost.local) Password:abcd1234 Click the save button to save the account | Global Admin created successfully | Global Admin created successfully |
|
||||
|
||||
|
||||
|
||||
2. Edit Global Admin \- As a global admin, I want to edit my account.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Global Admin | Logged in as Global Admin main account Main account: admin1@localhost.local | Click on the selected email to edit Edit Admin interface will show up Input of “New password” and “Re-type password” is needed New password: 1234abcd Re-type password: 1234abcd | Admin updated successfully | Admin updated successfully |
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
3. Delete Global Admin \- As a global admin, I want to delete global admins.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Global Admin | Logged in as Global Admin main account Main account: admin1@localhost.local | Navigate to trash bin icon located at the most right of selected email Click the trash bin icon An interface to verify deletion of account is shown Instruction to delete is shown by typing ‘confirm’ for confirmation Type ‘confirm’ Delete button is available to click Click delete button | Admin deleted successfully | Admin deleted successfully |
|
||||
|
||||
|
||||
|
||||
1. Creating Domain \- As a global admin, I want to create domain.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Global Admin | Logged in as Global Admin main account Main account: admin1@localhost.local | Navigate to Domain page on the left side bar Click the “+ Add Domain” button An interface shows up with a requirement of “Name” and “Title” Enter the domain name and domain title \*Sample Name: [jpj.gov.my](http://jpj.gov.my) Title: JPJ Click “Save” button to save | New Domain is added successfully | New Domain is added successfully |
|
||||
|
||||
|
||||
|
||||
2. Domain Configuration \- As a global admin, I want to configure the domain.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Global Admin | Logged in as Global Admin main account Main account: admin1@localhost.local | Click the domain name on the list or page will be directed after completing the domain creation Click the “SSO” at the horizontal top bar An input of “OIDC Consumer Key”, “OIDC Consumer Secret” and “OIDC Well-known URL” is needed Enter the SSO as required \*Sample OIDC Consumer Key: j4HWNRsR1vB2WnVAV8kUhoHuvF2VMW1sC0ynF1nb OIDC Consumer Secret:gWGpaQ6U0mvFApr4UXmaCF0ZAi3ENEA3e42fYuUP16KyTE4Q6kYETgMvYpBvWsgIQhLi4W2dxbHy3v6Lq4EMqkAgJa2X7lWQsUBNaL69hPf6A0Zffbr2ZGBfE8lyhbLt OIDC Well-known URL:https://auth.kagesenshi.org/application/o/emgr-dev-tenant1/.well-known/openid-configuration Click “Save” button | Domain Settings Saved | Domain Settings Saved |
|
||||
|
||||
3. Add Domain Admin \- As a global admin, I want to add domain admin.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Global Admin | Logged in as Global Admin main account Main account: admin1@localhost.local | Click on Admin tab on the horizontal bar in the domain page Click “Add Domain Admin” button “Add Domain Admin” interface shows up requiring an email Enter email for Domain Admin \*Sample Email: mei@jpj.gov.my Click “Save” button | Domain admin added successfully. | Domain admin added successfully. The new domain admin account appeared in the list. |
|
||||
|
||||
4. Deleting Domain \- As a global admin, I want to delete domain.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Global Admin | Logged in as Global Admin main account Main account: admin1@localhost.local | Navigate to “Domain” at the left side bar Navigate to trash bin icon of the wanted domain Click the trash bin icon A “Delete Domain” interface is shown to verify deletion Instruction to delete is shown by typing ‘confirm’ for confirmation Type ‘confirm’ Delete button is available to click Click delete button | Domain has been deleted | Domain has been deleted successfully and removed from list |
|
||||
|
||||
## Test Scenario: User as Domain Admin
|
||||
|
||||
1. Add Domain Admin \- As a domain admin, I want to add domain admin.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Domain Admin | Logged in as Domain Admin Domain Admin account: amin@jpj.gov.my | Navigate to “Domain” Click on Admin tab on the horizontal bar in the domain page Click “Add Domain Admin” button “Add Domain Admin” interface shows up requiring an email Enter email for Domain Admin \*Sample Email: lina@jpj.gov.my Click “Save” button | Domain has been deleted | Domain has been deleted successfully and removed from list |
|
||||
|
||||
2. Delete Domain Admin \- As a domain admin, I want to delete domain admins.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Domain Admin | Logged in as Domain Admin Domain Admin account: amin@jpj.gov.my | Navigate to “Domain” Click on Admin tab on the horizontal bar in the domain page Navigate to trash bin icon of wanted email Domain Admin Click trash bin icon A “Remove Domain Admin” interface is shown to verify deletion Instruction to delete is shown by typing ‘confirm’ for confirmation Type ‘confirm’ Delete button is available to click Click delete button | Domain admin removed successfully | Domain admin removed successfully |
|
||||
|
||||
3. Add Tenant \- As domain admin, I want to add tenant.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Domain Admin | Logged in as Domain Admin Domain Admin account: [amin@jpj.gov.my](mailto:amin@jpj.gov.my) Domain User is registered | Navigate to “Tenants” at left side bar Click “Tenants” to enter Tenants page Click “+ Add Tenant” button “Add Tenant” interface will show with “Name” and “Tenant Admin” is required Enter Tenant name at “Name” \*Sample: Name: JPJ1 Pick tenant admin from drop down list of Domain User | Tenant has been added successfully | Tenant has been added successfully |
|
||||
|
||||
4. Delete Tenant \- As a domain admin, I want to delete tenant.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Domain Admin | Logged in as Domain Admin Domain Admin account: [amin@jpj.gov.my](mailto:amin@jpj.gov.my) Domain User is registered | Navigate to “Tenants” at left side bar Click “Tenants” to enter Tenants page Navigate to trash bin icon of wanted email Tenant Click trash bin icon A “Remove Tenant" interface is shown to verify deletion Instruction to delete is shown by typing ‘confirm’ for confirmation Type ‘confirm’ Delete button is available to click Click delete button | Tenant has been removed successfully | Tenant has been removed successfully and is not seen on list |
|
||||
|
||||
Register Domain User
|
||||
|
||||
- Domain users have to be registered upon creating the domain.
|
||||
- It is to register user into database for the system to identify domain user and add to list
|
||||
- Roles such as Tenant Admin and Tenant User can be applied to registered domain user
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Domain User | Domain is established | Log in to another user account of the same domain Example for JPJ domain: User: raj@jpj.gov.my | User is registered as domain user | User is registered as domain user and not assigned to any tenant |
|
||||
|
||||
## Test Scenario: User as Tenant Admin
|
||||
|
||||
1. Add Tenant User \- As a tenant admin, I want to add tenant user.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin | Logged in as Tenant Admin Domain user is registered | Navigate to “Tenant” at left side bar Click “Users” below the “Tenants” at left side bar Click “+ Add User” button An interface shows up to select email from a dropdown Select email for Tenant User for the dropdown list of Domain User Click “Save” button | New user is added to tenant | User is added to tenant and in the list of Tenant Users |
|
||||
|
||||
|
||||
|
||||
2. Remove Tenant User \- As a tenant admin, I want to remove tenant user.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin | Logged in as Tenant Admin Domain user is registered | Navigate to “Tenant” at left side bar Click “Users” below the “Tenants” at left side bar Navigate to trash bin icon of wanted email Tenant User Click trash bin icon A “Remove User” interface is shown to verify deletion Instruction to delete is shown by typing ‘confirm’ for confirmation Type ‘confirm’ Delete button is available to click Click delete button | User removed from tenant | User is removed from tenant |
|
||||
|
||||
## **Test Scenario: Host Deployment**
|
||||
|
||||
1. Add Host \- As a tenant admin, I want to add a host for a the data platform.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Logged in as Tenant Admin or Tenant User Installed Virtual Manager | Navigate to “Hosts” on left side bar Click “+ Add Host” button to add host An interface shows the steps and instructions in deploying host Open Virtual Manager Deploy command at terminal | Host appeared on the host list. Host successfully added. | Host appeared on the host list. Host successfully added. |
|
||||
|
||||
2\. Remove Host \- As a tenant admin, I want to remove host from the data platform.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Logged in as Tenant Admin or Tenant User Installed Virtual Manager | Navigate to “Hosts” on left side bar Click trash bin button to delete host An interface appears, type ‘confirm’ to verify the deletion. Click ‘Delete’ button to proceed. | Host removed from the list. Host deleted successfully. | Host remain on the list. Host is not deleted. |
|
||||
|
||||
3\. View host details – As a tenant, I want to view the host details, so that I know its specification, usage & status.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Logged in as Tenant Admin or Tenant User Installed Virtual Manager | Navigate to “Hosts” on left side bar Click the pencil button to view host configuration page. Click the “Status” tab to view the host details. | The details of the host is being displayed. | The details of the host is being displayed. |
|
||||
|
||||
4\. Edit & deploy host – As a tenant admin, I want to edit and deploy host settings.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Logged in as Tenant Admin or Tenant User Installed Virtual Manager | Navigate to “Hosts” on left side bar Click the pencil button to view host configuration page. Click “Settings” tab to edit & deploy host settings. | The host setting page is being displayed. User can edit Hostname. | The host setting page is being displayed. User can edit Hostname. |
|
||||
|
||||
6\. Audit log – As a tenant, I want to view the hosts audit log, so that I can troubleshoot issues related to the host settings.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Logged in as Tenant Admin or Tenant User Installed Virtual Manager | Navigate to “Hosts” on left side bar. Click the pencil button to view host configuration page. Click “Audit” tab to view hosts audit log. | The hosts audit log is being displayed. | The hosts audit log is being displayed. |
|
||||
|
||||
## **Test Scenario: Cluster Deployment**
|
||||
|
||||
1. Add cluster – As a tenant admin, I want to add an existing cluster as part of the data platform.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Host is deployed | Navigate to “Clusters” on the left side bar. Click “+Add Cluster” button to add cluster. It will redirect to Add Cluster page. Select Hosts for the cluster. Click “Next” button to navigate to cluster Configuration tab. Click “Controller host” and then click “Next”. Check the cluster summary and click “Submit”. | Cluster is added successfully. | Cluster is added successfully. |
|
||||
|
||||
|
||||
|
||||
2. Delete cluster \- As a tenant admin, I want to delete a cluster, so that resources won’t be wasted on unused clusters.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Host is deployed | Navigate to “Clusters” on the left side bar. Click trash bin button. Type ‘confirm’ to verify the deletion. Click “Delete” button. | Cluster remove from the cluster list. Cluster is remove successfully. | |
|
||||
|
||||
3. View cluster details – As a tenant, I want to view the cluster & its resources details, so that I know its specification, usage & status.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Host is deployed | Navigate to “Clusters” on the left side bar. Click the pencil button. Navigate to “Status” tab. | The cluster & its resources details is being displayed. | The cluster & its resources details is being displayed. |
|
||||
|
||||
|
||||
|
||||
4. Edit cluster – As a tenant admin, I want to edit and deploy cluster settings, so that I can optimize its resource usage.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Host is deployed | Navigate to “Clusters” on the left side bar. Click the pencil button. Navigate to “Settings” tab. Change the initial cluster name to “pgsql” and fill in the label. Click “Save” button. | Cluster name and label is updated. | Cluster name and label is updated. |
|
||||
|
||||
5. View audit log – As a tenant, I want to view the cluster log, so that I can troubleshoot issues related to the cluster settings.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Host is deployed | Navigate to “Clusters” on the left side bar. Click the pencil button. Navigate to “Logs” tab. | Cluster log is being displayed. | |
|
||||
|
||||
6. View audit log – As a tenant, I want to view the cluster audit log, so that I can troubleshoot issues related to the cluster settings.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Host is deployed | Navigate to “Clusters” on the left side bar. Click the pencil button. Navigate to “Audit” tab. | Cluster audit log is being displayed. | Cluster audit log is being displayed. |
|
||||
|
||||
7. Cluster master task – As a tenant, I want to view the cluster tasks to see to check the host status, remark and log.
|
||||
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Host is deployed | Navigate to “Clusters” on the left side bar. Click “Tasks” on the left side bar. Click the cluster name to view the setting up cluster table. Click “View details remarks” to view remark and click “view log” to view log. | The status, remark and log for the initialize controller, initialize worker and check node status is being displayed. | The status, remark and log for the initialize controller, initialize worker and check node status is being displayed. |
|
||||
|
||||
|
||||
|
||||
## **Test Scenario: Service Deployment**
|
||||
|
||||
1. Create service template – As a tenant admin, I want to add a service template using a custom Kubernetes manifest file,
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Host is deployed. Cluster is deployed. | Navigate to “Clusters” on the left side bar. Click “Services” on the left side bar. Click “+ Add Service” button. | Navigate to Add Service page. | Navigate to Add Service page. |
|
||||
|
||||
2. Add PostgreSQL service template – As a tenant admin, I want to deploy PostgreSQL service to the cluster.
|
||||
|
||||
| User Role | Preconditions | Test Steps (Summary) | Expected Result | Result |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| Tenant Admin Tenant User | Host is deployed. Cluster is deployed. | Navigate to “Clusters” on the left side bar. Click “Services” on the left side bar. Click “+ Add Service” button. Click “Add” button at the right side in the PostreSQL container. At the deployment tab, fill in the deployment label. For example, ‘psql’, then click “Next” button. It will navigate to Configurations tab, then click “Next” button. It will then navigate to summary tab, then click “Deploy” button. | PostgreSQL is displayed in services list. PostgreSQL is being deployed. | |
|
||||
139
docs/user-manual.md
Normal file
139
docs/user-manual.md
Normal file
@ -0,0 +1,139 @@
|
||||
# 👥 User Manual
|
||||
|
||||
This manual guides end users through using the management tool web app.
|
||||
|
||||
TODO: In Progress
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Basics and Concepts](#basics-and-concepts)
|
||||
- [Domain](#domain)
|
||||
- [Tenant](#tenant)
|
||||
- [Users and Roles](#users-and-roles)
|
||||
- [Host](#host)
|
||||
- [Cluster](#cluster)
|
||||
- [Services](#services)
|
||||
- [Core Features](#core-features)
|
||||
- [User Management](#user-management)
|
||||
- [Global Admin](#global-admin)
|
||||
- [Domain Admin and Users](#domain-admin-and-users)
|
||||
- [Tenant Admin and Users](#tenant-admin-and-users)
|
||||
- [Domain Management](#domain-management)
|
||||
- [Tenant Management](#tenant-management)
|
||||
- [Host Management](#host-management)
|
||||
- [Cluster Management](#cluster-management)
|
||||
- [Service Management](#service-management)
|
||||
- [Object Storage](#object-storage)
|
||||
- [Folder Manipulation](#folder-manipulation)
|
||||
- [File Manipulation](#file-manipulation)
|
||||
- [Tag Management](#tag-management)
|
||||
|
||||
---
|
||||
|
||||
## Basics and Concepts
|
||||
|
||||
### Domain
|
||||
|
||||
A registered namespace or organisational boundary used to identify and manage users, resources, and authentication within a system or enterprise.
|
||||
|
||||
### Tenant
|
||||
|
||||
A distinct business entity, which may refer to an internal division (e.g., a department or subsidiary) or an external client organisation.
|
||||
|
||||
### Users and Roles
|
||||
|
||||
- **Admin**: A privileged account responsible for configuring, managing, and maintaining system resources, user access, and organizational settings.
|
||||
- **Users**: An individual account with access to system resources and services, subject to the permissions and roles assigned by the administrator.
|
||||
|
||||
### Host
|
||||
|
||||
A physical or virtual machine that provides the computing environment for running applications, services, and system processes.
|
||||
|
||||
### Cluster
|
||||
|
||||
A group of interconnected hosts or nodes that work together to improve availability, scalability, and performance by operating as a unified system.
|
||||
|
||||
### Services
|
||||
|
||||
Core software components that provide data management, processing, analysis, and orchestration capabilities within the system.
|
||||
|
||||
- **PostgreSQL**: A relational database system used for storing and managing structured data.
|
||||
- **Trino**: A distributed SQL query engine designed for fast, large-scale data analytics across multiple data sources.
|
||||
- **Superset**: A modern data exploration and visualization platform that enables interactive dashboards and reporting.
|
||||
- **Airflow**: A workflow orchestration tool used to author, schedule, and monitor data pipelines and processes.
|
||||
|
||||
---
|
||||
|
||||
## Core Features
|
||||
|
||||
### User Management
|
||||
|
||||
#### Global Admin
|
||||
|
||||
1. Add global admin
|
||||
2. Edit global admin
|
||||
3. Delete global admin
|
||||
|
||||
#### Domain Admin and Users
|
||||
|
||||
1. Login as domain admin
|
||||
2. Login as domain users
|
||||
|
||||
#### Tenant Admin and Users
|
||||
|
||||
1. Login as tenant admin
|
||||
2. Login as tenant users
|
||||
|
||||
### Domain Management
|
||||
|
||||
1. Add domain
|
||||
2. Update domain settings
|
||||
3. Update domain SSO
|
||||
4. Add users to domain
|
||||
5. Delete domain
|
||||
|
||||
### Tenant Management
|
||||
|
||||
1. Add Tenant
|
||||
2. Update tenant settings
|
||||
|
||||
### Host Management
|
||||
|
||||
- Add host
|
||||
- Remove host
|
||||
|
||||
### Cluster Management
|
||||
|
||||
- Add cluster
|
||||
- Update cluster settings
|
||||
- Delete cluster
|
||||
|
||||
### Service Management
|
||||
|
||||
- Deploy service
|
||||
- Update service settings
|
||||
- Delete service
|
||||
|
||||
### Object Storage
|
||||
|
||||
- Update object storage settings
|
||||
|
||||
#### Folder manipulation
|
||||
|
||||
- Create folder
|
||||
- Move or rename folder
|
||||
- Delete folder
|
||||
|
||||
#### File manipulation
|
||||
|
||||
- Upload file
|
||||
- Move or rename file
|
||||
- Delete file
|
||||
|
||||
### Tag Management
|
||||
|
||||
- Add tag
|
||||
- Edit tag
|
||||
- Delete tag
|
||||
Loading…
x
Reference in New Issue
Block a user