You've already forked uapf-integration-ready
Import UAPF package
169 lines
4.8 KiB
Markdown
169 lines
4.8 KiB
Markdown
# Data Synchronization Process - Documentation
|
|
|
|
## Overview
|
|
|
|
This process implements a data synchronization workflow that fetches data from an external REST API, transforms and validates it, and writes it to a PostgreSQL database.
|
|
|
|
## Process Flow
|
|
|
|
1. **Fetch Data from API** - Retrieves customer data from the external REST API endpoint
|
|
2. **Transform Data** - Maps external data fields to internal schema
|
|
3. **Validate Data** - Applies quality rules and validation logic using DMN decisions
|
|
4. **Write to Database** - Persists validated data to the PostgreSQL database
|
|
|
|
## Integration Points
|
|
|
|
### REST API Connector
|
|
- **Location**: `../connectors/rest-api/`
|
|
- **Configuration**: See `resources/mappings.yaml` for endpoint details
|
|
- **Authentication**: Bearer token (configured via `API_TOKEN` environment variable)
|
|
- **Endpoints**:
|
|
- GET `/api/v1/data` - Fetch customer records
|
|
- PUT `/api/v1/data/{id}` - Update customer record
|
|
|
|
### Database Connector
|
|
- **Location**: `../connectors/database/`
|
|
- **Type**: PostgreSQL
|
|
- **Schema**: See `../connectors/database/schema.sql`
|
|
- **Tables**:
|
|
- `customers` - Main customer data table
|
|
- `sync_log` - Synchronization audit trail
|
|
|
|
## Decision Logic (DMN)
|
|
|
|
The validation decision table (`dmn/decisions.dmn.xml`) evaluates records based on:
|
|
- **Record Type**: customer, transaction, etc.
|
|
- **Quality Score**: 0-100 numeric quality metric
|
|
- **Required Fields**: Presence of mandatory fields
|
|
|
|
### Validation Outcomes
|
|
- **VALID** (Score ≥ 80) → Process immediately
|
|
- **WARNING** (Score 50-79) → Flag for manual review
|
|
- **INVALID** (Score < 50 or missing required fields) → Reject
|
|
|
|
## Case Management (CMMN)
|
|
|
|
When synchronization issues occur, a case is created to manage the investigation and resolution:
|
|
|
|
1. **Investigate Issue** - Manual task to analyze the problem
|
|
2. **Check API Status** - Automated check of API availability
|
|
3. **Check Database Status** - Automated check of database connectivity
|
|
4. **Resolve Issue** - Manual remediation task
|
|
5. **Retry Synchronization** - Automated retry of failed sync
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables
|
|
|
|
Required environment variables (see `../config/secrets.example.env`):
|
|
|
|
```bash
|
|
# API Configuration
|
|
API_TOKEN=your-api-bearer-token
|
|
API_BASE_URL=https://api.example.com
|
|
|
|
# Database Configuration
|
|
DB_HOST=localhost
|
|
DB_PORT=5432
|
|
DB_NAME=processgit
|
|
DB_USER=sync_user
|
|
DB_PASSWORD=secure_password
|
|
```
|
|
|
|
### Data Mapping
|
|
|
|
Field mappings are defined in `resources/mappings.yaml`:
|
|
|
|
| External Field | Internal Field | Type |
|
|
|---------------|----------------|---------|
|
|
| external_id | customerId | string |
|
|
| full_name | customerName | string |
|
|
| contact_email | email | string |
|
|
| created_date | createdAt | datetime|
|
|
|
|
## Error Handling
|
|
|
|
### API Failures
|
|
- **Strategy**: Retry with exponential backoff
|
|
- **Max Retries**: 3
|
|
- **Notification**: Alert ops team
|
|
|
|
### Database Failures
|
|
- **Strategy**: Log and continue
|
|
- **Notification**: Alert ops team
|
|
|
|
### Validation Failures
|
|
- **Strategy**: Reject record
|
|
- **Log Level**: Warning
|
|
|
|
## Performance Considerations
|
|
|
|
- **API Timeout**: 30 seconds
|
|
- **Database Connection Pool**: 2-10 connections
|
|
- **Batch Size**: Recommended 100 records per sync
|
|
- **Frequency**: Configurable (default: hourly)
|
|
|
|
## Testing
|
|
|
|
### Unit Tests
|
|
- Validate data transformation logic
|
|
- Test decision table rules
|
|
- Verify error handling
|
|
|
|
### Integration Tests
|
|
- End-to-end API to database flow
|
|
- Connection failure scenarios
|
|
- Data validation edge cases
|
|
|
|
### Performance Tests
|
|
- Load test with 10,000+ records
|
|
- Concurrent sync operations
|
|
- Connection pool behavior
|
|
|
|
## Monitoring
|
|
|
|
### Key Metrics
|
|
- Sync success rate
|
|
- Average processing time per record
|
|
- API response times
|
|
- Database query performance
|
|
- Error rates by type
|
|
|
|
### Alerts
|
|
- Sync failure rate > 5%
|
|
- API response time > 10s
|
|
- Database connection pool exhaustion
|
|
- Validation rejection rate > 20%
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
**Issue**: Sync fails with API timeout
|
|
- **Cause**: API endpoint slow or unavailable
|
|
- **Resolution**: Check API status, increase timeout, contact API provider
|
|
|
|
**Issue**: Database write failures
|
|
- **Cause**: Connection pool exhausted or schema mismatch
|
|
- **Resolution**: Check pool configuration, verify schema matches mappings
|
|
|
|
**Issue**: High validation rejection rate
|
|
- **Cause**: Data quality issues at source
|
|
- **Resolution**: Review validation rules, contact data provider
|
|
|
|
## Change Log
|
|
|
|
### Version 0.1.0 (Initial Release)
|
|
- Implemented basic data sync workflow
|
|
- Configured REST API and database connectors
|
|
- Added DMN validation rules
|
|
- Created CMMN case for issue management
|
|
- Documented configuration and operations
|
|
|
|
## Support
|
|
|
|
For issues or questions:
|
|
- **Primary Contact**: {{.RepoOwner}}@processgit.local
|
|
- **Integration Team**: integration@processgit.local
|
|
- **On-Call**: See `metadata/ownership.yaml`
|