As the organizations are moving towards real-time solutions, one of the main challenges faced is to maintain different codebases to manage Batch and streaming use-cases. Simplan offers a config-driven solution where the business logic can be written once as an operator(reusable) and applied on both batch and streaming sources by abstracting the complexity of the underlying execution engine. The configurations are organized in a composite or hierarchical fashion, allowing reuse across environments.
Capabilities:
- Config Driven(Low/No code)
- Pluggable operators
- Support both batch and Streaming use cases using the same config structure
- Integrations with JDBC, Redshift, Kafka, Elasticsearch etc
- Built-In Quality control every step of the way
- Lineage, Observability & Metrics tracking built into the framework
- Metrics to be published to a log aggregation tool like Elasticsearch, Splunk, Wavefront
- Integrates well with open-source lineage tracking tools like Superglue
Benefits
- Reusable operators for common data processing patterns
- Improves developer productivity by 10-100 times
- Build-in data quality checks which proactively catch issues
- Easy maintainability with a single codebase for batch and streaming use cases
Benefits from Session
- Learn concepts and architecture patterns required to build self-serve platforms for data pipelines, which they can apply to their organizations.
- Learn about best practices and approaches for building a config-driven data processing framework
- Ease of using the framework, traceability, fine-grained monitoring and debuggability
- Best practices of ETL development with circuit breakers, inbound and outbound validations and custom instrumentation
- Challenges of abstraction and the right amount of abstraction to strive for