Context
DataCorp, a mid-sized retail analytics company, relies on Azure Data Factory (ADF) for orchestrating its ETL processes that aggregate sales data from multiple sources such as SQL databases, REST APIs, and flat files in Azure Blob Storage. As the volume of data increases, the engineering team is exploring whether ADF continues to meet their scalability and performance needs.
Scale Requirements
- Data Sources: Up to 50 data sources with varying schemas and data types.
- Data Volume: 10TB of data processed daily with peaks during sales events.
- Latency: Batch jobs should run within a 2-hour window, while real-time data integration is desired for some sources.
- Concurrency: Up to 100 concurrent pipeline executions during peak times.
Requirements
- Evaluate the advantages of using ADF for ETL processes, including its integration capabilities and ease of use.
- Identify potential disadvantages, such as cost implications, performance bottlenecks, and limitations in data transformation capabilities.
- Design a monitoring strategy to ensure data quality and pipeline performance, including alert thresholds.
- Propose a strategy for optimizing ADF pipelines to handle increased data loads efficiently.
- Discuss alternative technologies that could complement or replace ADF, considering budget and team expertise.
Constraints
- Budget: Limited to $5,000/month for data processing services.
- Compliance: Must adhere to data governance policies, including GDPR for customer data.
- Team Expertise: The team has varying levels of experience with Azure and may require training for advanced features.
