telegram-management-system/marketing-agent/services/logging/README.md

Logging Service

Centralized logging, monitoring, and alerting service for the Telegram Marketing Agent System.

Features

• Log Collection: Centralized log collection from all microservices
• Real-time Analysis: Pattern detection and anomaly analysis
• Alert Management: Multi-channel alerting (Email, Slack, Webhook)
• Dashboard: Real-time monitoring dashboard
• Log Storage: Elasticsearch-based storage with retention policies
• Performance Metrics: System and application performance tracking

Architecture

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Microservices  │────▶│  Log Collector  │────▶│  Elasticsearch  │
└─────────────────┘     └─────────────────┘     └─────────────────┘
                               │                          │
                               ▼                          ▼
                        ┌─────────────────┐     ┌─────────────────┐
                        │  Log Analyzer   │────▶│  Alert Manager  │
                        └─────────────────┘     └─────────────────┘
                                                         │
                                                         ▼
                                                  ┌──────────────┐
                                                  │   Channels   │
                                                  ├──────────────┤
                                                  │    Email     │
                                                  │    Slack     │
                                                  │   Webhook    │
                                                  └──────────────┘

Prerequisites

• Node.js 18+
• Elasticsearch 8.x
• Redis 6.x

Installation

1. Install dependencies:

cd services/logging
npm install

2. Configure environment variables:

cp .env.example .env
# Edit .env with your configuration

3. Start Elasticsearch:

docker run -d --name elasticsearch \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  -p 9200:9200 \
  elasticsearch:8.12.0

4. Start Redis:

docker run -d --name redis \
  -p 6379:6379 \
  redis:latest

5. Start the service:

npm start

Configuration

Environment Variables

Variable	Description	Default
PORT	Service port	3010
ELASTICSEARCH_NODE	Elasticsearch URL	http://localhost:9200
ELASTICSEARCH_USERNAME	Elasticsearch username	elastic
ELASTICSEARCH_PASSWORD	Elasticsearch password	changeme
REDIS_HOST	Redis host	localhost
REDIS_PORT	Redis port	6379
ALERT_EMAIL_ENABLED	Enable email alerts	false
ALERT_SLACK_ENABLED	Enable Slack alerts	false
ALERT_WEBHOOK_ENABLED	Enable webhook alerts	false

Alert Thresholds

Configure alert thresholds in config/index.js:

alerts: {
  rules: {
    errorRate: {
      threshold: 0.05, // 5% error rate
      window: 300000   // 5 minutes
    },
    responseTime: {
      threshold: 1000, // 1 second
      window: 300000   // 5 minutes
    },
    systemResources: {
      cpu: 80,         // 80% CPU usage
      memory: 85,      // 85% memory usage
      disk: 90         // 90% disk usage
    }
  }
}

API Endpoints

Log Management

• GET /api/logs/search - Search logs
• GET /api/logs/stats - Get log statistics
• GET /api/logs/metrics - Get aggregated metrics
• GET /api/logs/stream - Stream logs in real-time
• DELETE /api/logs/cleanup - Delete old logs

Alert Management

• GET /api/alerts/history - Get alert history
• GET /api/alerts/active - Get active alerts
• POST /api/alerts/:id/acknowledge - Acknowledge an alert
• DELETE /api/alerts/:id - Clear an alert
• POST /api/alerts/test - Send test alert
• GET /api/alerts/config - Get alert configuration

Dashboard

• GET /api/dashboard/overview - Get dashboard overview
• GET /api/dashboard/health - Get service health
• GET /api/dashboard/trends - Get performance trends
• GET /api/dashboard/top-errors - Get top errors

Log Format

Application Logs

{
  "@timestamp": "2024-01-15T10:30:00.000Z",
  "service": "api-gateway",
  "level": "error",
  "message": "Request failed",
  "userId": "user123",
  "requestId": "req-456",
  "method": "POST",
  "path": "/api/campaigns",
  "statusCode": 500,
  "responseTime": 234.5,
  "error": {
    "type": "ValidationError",
    "message": "Invalid campaign data",
    "stack": "..."
  }
}

Metrics

{
  "@timestamp": "2024-01-15T10:30:00.000Z",
  "service": "api-gateway",
  "metric": "response_time",
  "value": 234.5,
  "unit": "ms",
  "dimensions": {
    "endpoint": "/api/campaigns",
    "method": "POST",
    "status": "success"
  }
}

Integration

Sending Logs from Services

1. Install logging client:

npm install winston winston-elasticsearch

2. Configure winston logger:

import winston from 'winston';
import { ElasticsearchTransport } from 'winston-elasticsearch';

const logger = winston.createLogger({
  transports: [
    new ElasticsearchTransport({
      level: 'info',
      clientOpts: {
        node: 'http://localhost:3010',
        auth: { username: 'elastic', password: 'changeme' }
      },
      index: 'marketing-agent-logs'
    })
  ]
});

3. Send logs:

logger.info('Campaign created', {
  service: 'campaign-service',
  userId: 'user123',
  campaignId: 'camp456',
  action: 'create'
});

Real-time Log Streaming

const eventSource = new EventSource('/api/logs/stream?service=api-gateway&level=error');

eventSource.onmessage = (event) => {
  const log = JSON.parse(event.data);
  console.log('New log:', log);
};

Monitoring

Health Check

curl http://localhost:3010/health

Queue Statistics

curl http://localhost:3010/api/dashboard/overview

Service Health

curl http://localhost:3010/api/dashboard/health

Maintenance

Log Retention

Logs are automatically deleted based on retention policies:

• Application logs: 30 days
• Metrics: 90 days
• Error logs: 60 days

Manual Cleanup

curl -X DELETE http://localhost:3010/api/logs/cleanup

Index Management

The service automatically creates and manages Elasticsearch indices with:

• Lifecycle policies for automatic rollover
• Retention policies for automatic deletion
• Optimized mappings for different log types

Troubleshooting

Common Issues

1. Elasticsearch Connection Failed

   • Check Elasticsearch is running: curl http://localhost:9200
   • Verify credentials in .env file
   • Check network connectivity

2. High Memory Usage

   • Adjust batch size in config: collection.batchSize
   • Reduce flush interval: collection.flushInterval
   • Check Elasticsearch heap size

3. Alerts Not Sending

   • Verify alert channel configuration
   • Check SMTP settings for email
   • Test webhook URL accessibility

Debug Mode

Enable debug logging:

LOG_LEVEL=debug npm start

Development

Running Tests

npm test

Development Mode

npm run dev

Adding New Alert Channels

1. Create channel handler in services/alertManager.js:

async sendCustomAlert(alert) {
  // Implement channel logic
}

2. Add to alert sending pipeline
3. Update configuration schema

Performance Tuning

Elasticsearch Optimization

• Increase shards for high-volume indices
• Configure index lifecycle management
• Use bulk operations for better throughput

Redis Optimization

• Configure maxmemory policy
• Use Redis clustering for high availability
• Monitor queue sizes

Application Optimization

• Adjust batch sizes based on load
• Configure worker concurrency
• Use connection pooling