telemetryflow/telemetryflow-platform repository overview

⁠TelemetryFlow Platform — Full-Stack Observability & IAM (5-Tier RBAC)

⁠Overview

TelemetryFlow Platform is a full-stack, production-ready observability and infrastructure monitoring platform. It combines a 5-tier RBAC IAM service, multi-tenant workspace management, OTLP-native telemetry ingestion, and real-time infrastructure monitoring (Kubernetes, VMs, agents) in a single integrated monolith — backed by PostgreSQL for relational state and ClickHouse for high-volume time-series metrics.

⁠Features

⁠Integrated Monolith Architecture

TelemetryFlow Platform is a full-stack monolith combining:

• telemetryflow-platform: NestJS backend with IAM, authentication, and API services
• TelemetryFlow-Viz: Vue 3 frontend with visualization dashboards and IAM management UI
• Dual API Connection: Frontend connects to OTEL Collector (telemetry) AND NestJS (IAM)

⁠Frontend (TelemetryFlow-Viz)

• Vue 3 + TypeScript: Modern reactive frontend framework
• Pinia State Management: JWT token storage and authentication state
• Naive UI Components: Enterprise-grade UI component library
• IAM Management UI: Users, roles, and permissions management pages
• JWT Authentication: Secure login with token refresh mechanism
• Permission-Based UI: Dynamic UI visibility based on user permissions
• Telemetry Dashboards: Metrics, logs, traces, and Kubernetes visualization

⁠Backend (telemetryflow-platform)

• Multi-tenant architecture: Tenant → Organization → Workspace hierarchy
• User management: Complete CRUD operations with role-based access
• 5-Tier RBAC System: Super Admin, Administrator, Developer, Viewer, Demo
• Role-Based Access Control: Hierarchical roles with 22+ permissions
• Group management: User groups with permission inheritance
• Region support: Multi-region tenant deployment
• CQRS pattern: Separate read/write operations (33 commands, 18 queries)
• Domain events: Event-driven architecture (25+ events)

⁠Module Standardization System

• Quality Gates: 7 comprehensive standardization gates for all modules
• Test Coverage: 90%+ overall coverage (95% domain layer requirement)
• Documentation Generation: Automated README, API docs, ERD, and DFD generation
• Property-Based Testing: Comprehensive correctness properties with 100 iterations each
• Test Structure Validation: Directory organization and naming convention enforcement
• Memory-Optimized Processing: Efficient file system operations with depth limits
• Automated Validation: Quality enforcement and standardization tooling
• DDD Compliance: Strict Domain-Driven Design architecture patterns
• ✅ Task 4 Complete: All documentation and coverage tools working (v1.1.4)

⁠Architecture

• Domain-Driven Design (DDD): 8 aggregates, 10 value objects, domain services
• CQRS: Command Query Responsibility Segregation
• Clean Architecture: Domain → Application → Infrastructure → Presentation
• Event-Driven: Domain events for all entity lifecycle changes

⁠Observability

• Swagger/OpenAPI: Interactive API documentation at /api
• OpenTelemetry: Distributed tracing with OTLP export
• Winston Logging: Structured logging with multiple levels
• Health Checks: Built-in health endpoint

⁠Security

• JWT Authentication: Secure token-based auth
• Password Hashing: Argon2 for secure password storage
• Secret Generation: Cryptographically secure secret generator
• Multi-tenancy Isolation: Organization-level data scoping

⁠Quick Start

⁠Prerequisites

• Node.js 18+
• pnpm 8+
• Docker & Docker Compose

⁠One-Command Setup

# Start all services (backend + frontend + databases)
docker-compose --profile all up -d

# Or start specific profiles
docker-compose --profile core up -d                # Core only (backend + databases)
docker-compose --profile core --profile ui up -d   # Core + UI Dashboard

⁠Frontend Setup (Development)

# Navigate to frontend directory
cd frontend

# Install dependencies
pnpm install

# Configure environment
cp .env.development.example .env

# Start development server
pnpm dev
# Frontend available at http://localhost:5173

Frontend Environment Variables:

# IAM API endpoint (NestJS backend)
TELEMETRYFLOW_IAM_API_URL=http://localhost:3000

# OTEL Collector endpoint (telemetry data)
TELEMETRYFLOW_API_URL=http://localhost:4318

# Enable mock data for development
TELEMETRYFLOW_USE_MOCK=true

⁠Docker Profiles

Available profiles:

• core - Backend, PostgreSQL, ClickHouse
• ui - TelemetryFlow Viz (Visualization Dashboard)
• tools - Portainer
• all - Everything

See docs/DOCKER_SETUP.md⁠ for details.

⁠Manual Setup

# Clone repository
git clone https://github.com/telemetryflow/telemetryflow-platform.git
cd telemetryflow-platform

# Install dependencies
pnpm install

# Configure environment
cp .env.example .env
# Edit .env with your configuration

# Generate secrets
pnpm run generate:secrets

# Start infrastructure (PostgreSQL + ClickHouse + OTEL)
docker-compose up -d

# Initialize ClickHouse schema
docker exec -i telemetryflow_core_clickhouse clickhouse-client --multiquery < config/clickhouse/migrations/001-audit-logs.sql

# Seed database
pnpm run db:seed:iam

# Start development server
pnpm run dev

⁠Alternative: Bootstrap Script

bash scripts/bootstrap.sh --dev

⁠Module Standardization

TelemetryFlow Platform follows comprehensive module standardization guidelines to ensure consistency, quality, and maintainability across all modules.

⁠Standardization Framework

Each module includes detailed specifications in .kiro/specs/:

• IAM Module: Complete identity and access management standardization
• Audit Module: Audit logging and compliance standardization
• Auth Module: Authentication and authorization standardization
• Cache Module: Caching and performance standardization

⁠Quality Gates

All modules must pass 6 comprehensive quality gates:

⁠Property-Based Testing

Each module implements 8 correctness properties:

1. Idempotency - Operations produce same result when repeated
2. Consistency - Data remains consistent across operations
3. Validation - All inputs are properly validated
4. Authorization - Access control is enforced
5. Persistence - Data is correctly saved and retrieved
6. Event Handling - Domain events are properly published
7. Error Handling - Errors are handled gracefully
8. Performance - Operations meet performance requirements

⁠Specification Structure

Each module specification includes:

.kiro/specs/{module}-module-standardization/
├── requirements.md    # 8 requirements, 80 acceptance criteria (EARS patterns)
├── design.md         # DDD architecture, components, correctness properties
└── tasks.md          # 52-60 implementation tasks with checkpoints

⁠Development Workflow

1. Review Specifications: Study requirements, design, and tasks
2. Implement Features: Follow DDD/CQRS patterns
3. Validate Quality: Ensure all quality gates pass
4. Property Testing: Implement correctness properties
5. Documentation: Maintain comprehensive documentation

For detailed contribution guidelines, see CONTRIBUTING.md⁠.

⁠5-Tier RBAC System

⁠Role Hierarchy

1. Super Administrator (Global)

   • Platform management across all organizations
   • All permissions

2. Administrator (Organization-scoped)

   • Full CRUD within organization
   • Cannot manage platform

3. Developer (Organization-scoped)

   • Create/Read/Update (no delete)
   • Cannot manage users/roles

4. Viewer (Organization-scoped)

   • Read-only access
   • Cannot modify resources

5. Demo (Demo org only)

   • Developer access in demo organization
   • Isolated from production data

⁠Default Users

⁠API Documentation

Once running, access Swagger UI at: http://localhost:3000/api

⁠Key Endpoints

Authentication (JWT):

• POST /api/v2/auth/login - Login with email/password, returns JWT tokens
• POST /api/v2/auth/refresh - Refresh access token
• POST /api/v2/auth/logout - Invalidate tokens
• GET /api/v2/auth/me - Get current user profile with roles/permissions

IAM Management:

• Users: /api/v2/iam/users - User management
• Roles: /api/v2/iam/roles - Role management
• Permissions: /api/v2/iam/permissions - Permission management
• Tenants: /api/v2/iam/tenants - Tenant management
• Organizations: /api/v2/iam/organizations - Organization management
• Workspaces: /api/v2/iam/workspaces - Workspace management
• Groups: /api/v2/iam/groups - Group management
• Regions: /api/v2/iam/regions - Region management
• Health: /health - Health check

⁠API Testing

Postman Collection (Recommended):

• Import docs/postman/TelemetryFlow Platform - IAM.postman_collection.json
• Import docs/postman/TelemetryFlow Platform - Local.postman_environment.json
• 54+ pre-configured requests with default credentials
• See docs/postman/README.md⁠

BDD Automated Testing (Newman):

# Run all BDD tests
pnpm test:bdd

# Run specific module
pnpm test:bdd:users
pnpm test:bdd:roles

# With detailed output
pnpm test:bdd:verbose

• 33 BDD test scenarios with Given-When-Then format
• 100% API coverage
• HTML and JSON reports
• See docs/postman/BDD_TESTS.md⁠

Export OpenAPI Spec:

./scripts/export-swagger-docs.sh

⁠Makefile Commands (Recommended)

TelemetryFlow Platform includes a comprehensive Makefile that simplifies development and CI operations. The Makefile provides standardized commands that work consistently across local development and CI environments.

# Quick Start
make help                    # Show all available commands
make install                 # Install dependencies
make dev                     # Start development server
make build                   # Build the application

# Development Workflow
make start                   # Install + build + start development
make reset                   # Clean + install + build (reset environment)
make check                   # Quick check (lint + test)

# Code Quality
make lint                    # Run ESLint
make lint-fix                # Run ESLint with auto-fix
make format                  # Alias for lint-fix

# Testing
make test                    # Run unit tests
make test-coverage           # Run tests with coverage
make test-bdd                # Run BDD tests (Newman/Postman)

# Database Operations
make db-migrate              # Run database migrations
make db-seed                 # Seed database with initial data
make db-setup                # Setup database (migrate + seed)
make db-cleanup              # Clean up database

# Docker Operations
make docker-build            # Build Docker image
make docker-run              # Run Docker container locally
make docker-stop             # Stop and remove Docker container
make up                      # Start all services with Docker Compose
make down                    # Stop all services

# CI/CD Pipeline (Used by GitHub Actions)
make ci-install              # CI: Install dependencies (frozen lockfile)
make ci-validate             # CI: Validate module standardization
make ci-lint                 # CI: Run linting
make ci-build                # CI: Build application
make ci-test                 # CI: Run tests with coverage
make ci-security             # CI: Run security audit
make ci-pipeline             # CI: Run complete pipeline

# Release Management
make release-build           # Build release version
make release-docker          # Build and push Docker release

# Utilities
make generate-secrets        # Generate JWT and session secrets
make bootstrap               # Bootstrap development environment
make health                  # Check application health
make version                 # Show version information
make clean                   # Clean build artifacts and dependencies

⁠Why Use Makefile?

1. Consistency: Same commands work in local development and CI
2. Simplicity: Single command for complex operations
3. Documentation: Self-documenting with make help
4. Reliability: Handles error cases and environment setup
5. CI Integration: GitHub Actions use the same Makefile targets

⁠Example Workflows

# New developer setup
make install
make generate-secrets
make db-setup
make dev

# Daily development
make check                   # Lint + test before committing
make reset                   # Reset environment if issues

# CI pipeline (what GitHub Actions runs)
make ci-pipeline             # Complete CI validation

⁠Available Scripts

# Development
pnpm dev              # Start with hot reload
pnpm start:debug      # Start with debugger

# Build & Run
pnpm build            # Build for production
pnpm start            # Start production server

# Database
pnpm db:cleanup              # Clean all databases (PostgreSQL + ClickHouse)
pnpm db:migrate              # Run all migrations (PostgreSQL + ClickHouse)
pnpm db:migrate:postgres     # Run PostgreSQL migrations only
pnpm db:migrate:clickhouse   # Run ClickHouse migrations only
pnpm db:migrate:seed         # Run migrations + seeds (full setup)
pnpm db:seed                 # Seed all data (PostgreSQL + ClickHouse)
pnpm db:seed:postgres        # Seed PostgreSQL only
pnpm db:seed:iam             # Seed IAM data only
pnpm db:seed:clickhouse      # Seed ClickHouse only
pnpm db:init-clickhouse      # Initialize ClickHouse schema
pnpm db:generate-sample      # Generate sample data (50 records)
pnpm db:reset                # Reset database

# Testing
pnpm test                    # Run unit tests
pnpm test:watch              # Watch mode
pnpm test:cov                # Coverage report
pnpm test:bdd                # Run BDD API tests (Newman)
pnpm test:bdd:verbose        # Run BDD tests with detailed output
pnpm test:bdd:users          # Run Users module BDD tests
pnpm test:bdd:roles          # Run Roles module BDD tests

# Security
pnpm generate:secrets        # Generate JWT & Session secrets

# Code Quality
pnpm lint                    # Lint and fix

# Docker
pnpm docker:up               # Start all containers
pnpm docker:down             # Stop all containers
pnpm docker:logs             # View logs
pnpm docker:clean            # Clean volumes

# Bootstrap
pnpm bootstrap               # Full setup (dependencies, Docker, migrations, seeds)

# API Documentation
./scripts/export-swagger-docs.sh  # Export OpenAPI spec

⁠Docker Deployment

⁠Development

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop services
docker-compose down

⁠Production

# Build and start
docker-compose up -d --build

# Check health
curl http://localhost:3000/health

# Access API
curl http://localhost:3000/api

⁠Services

• PostgreSQL: 172.151.151.20:5432
• ClickHouse: 172.151.151.40:8123/9000
• Backend: 172.151.151.10:3000
• OTEL Collector: 172.151.151.30:4317/4318
• Prometheus: 172.151.151.50:9090

⁠OTEL Collector Ports

⁠OTLP Endpoints (Dual Ingestion)

The collector supports both TelemetryFlow (v2) and OTEL Community (v1) endpoints:

TelemetryFlow Platform (Recommended):

POST http://localhost:4318/v2/traces
POST http://localhost:4318/v2/metrics
POST http://localhost:4318/v2/logs

OTEL Community (Backwards Compatible):

POST http://localhost:4318/v1/traces
POST http://localhost:4318/v1/metrics
POST http://localhost:4318/v1/logs

gRPC: localhost:4317 (both v1 and v2)

⁠Database Schema

⁠Core Tables

• users - User accounts
• roles - Role definitions (5-tier RBAC)
• permissions - Permission definitions (22+ permissions)
• tenants - Tenant organizations
• organizations - Business units
• workspaces - Project workspaces
• groups - User groups
• regions - Geographic regions

⁠Mapping Tables

• user_roles - User-Role assignments
• user_permissions - Direct user-permission assignments
• role_permissions - Role-Permission mappings

⁠Technology Stack

⁠Backend

• Framework: NestJS 11.x
• Language: TypeScript 5.9
• Database: PostgreSQL 16
• ORM: TypeORM 0.3
• Architecture: DDD + CQRS
• Authentication: JWT with Passport.js
• API Documentation: Swagger/OpenAPI
• Logger: Winston
• Observability: OpenTelemetry (OTEL)
• Password Hashing: Argon2

⁠Frontend

• Framework: Vue 3.5+ (Composition API)
• Language: TypeScript 5.x
• State Management: Pinia
• UI Components: Naive UI
• HTTP Client: Axios with JWT interceptor
• Router: Vue Router 4
• Build Tool: Vite
• Package Manager: pnpm

⁠Contributing

We welcome contributions! Please follow these steps:

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Important:

• Read CONTRIBUTING.md⁠ for detailed guidelines including module standardization
• Review module specifications in .kiro/specs/ before working on modules
• Follow the 6 quality gates for module development
• Implement property-based testing for comprehensive validation
• Review SECURITY.md⁠ for security best practices
• Follow CODE_OF_CONDUCT.md⁠

⁠Module Standardization

When contributing to modules, ensure compliance with standardization requirements:

• Quality Gates: All 6 gates must pass (Documentation, Test Coverage, File Structure, Database Patterns, API Standards, Build Quality)
• Test Coverage: ≥90% overall, ≥95% domain layer
• Property Testing: Implement all 8 correctness properties
• Documentation: Maintain 500+ line README with comprehensive sections
• DDD Compliance: Follow Domain-Driven Design patterns strictly

⁠Security

Security is a top priority. Please review our Security Policy⁠ for:

• Reporting vulnerabilities
• Security best practices
• Supported versions
• Contact information

Report security issues to: [email protected]⁠

⁠License

Apache-2.0 License - see LICENSE⁠ file for details

⁠Support

• Documentation: docs/⁠
• Issues: GitHub Issues⁠
• Discussions: GitHub Discussions⁠
• Security: SECURITY.md⁠

⁠Acknowledgments

Extracted from TelemetryFlow Platform⁠ - Enterprise Telemetry & Observability Platform.

Built with ❤️ by Telemetri Data Indonesia collaboration with Kiro⁠