Who should attend
This is an intermediate course, suitable for professionals with relevant experience, and with an interest or requirement to understand engineering for big data.
The target course participants are primarily software engineers, data engineers, team leads, and architects with seek to enhance their skills in the area of information architecture and design of data warehouse and data lake.
About the course
The course will equip the participants with essential knowledge and skills to architect systems that processes real time stream data. The course will discuss reference real time systems for data intensive systems that include processing pipelines, ingestion patterns specific to stream data, asynchronous message design for moving data, evaluation, processing, analysis and cataloguing of various data streams, persistence strategies, security, messaging architecture and event processing architecture.
Upon completion of the course, participants will be able to:
- Understand the various facets of a real time data and stream processing pipeline.
- Design a reference architecture for a real time data processing system by determining the needful layers such as ingestion, collection, wrangling, message queues, analysis, and accessing new insights.
- Collect and design appropriate storage strategy for data originating from smaller devices such as IoT, Sensors and IoE.
- Integrate disparate data sources using unanimous ingestion layer that manages channels via MQTT, Flume, Kafka, Twitter, and a custom HTTP receiver.
- Build and optimise production-grade deployments of Streaming solutions via common algorithms, configuration recipes, and tuning of instrumentation API.
- Design robust message producers and consumers for writing and reading messages using Kafka.
- Evaluate and determine best stream processing framework suited for the given business needs.
What Will Be Covered
This course will cover:
- Architecting for Real Time stream processing systems
- Design of Pipeline for Real Time Streaming Systems
- Ingestion Strategies for Data Streams
- Message Design and Queue Architecture
- Data Stream Analytics
- In-Memory vs Storage Strategies
- Streaming Visualisation Tools
- Case Study Discussion
Suria has twenty years of teaching and consulting experience in areas such as software engineering, application architecture, crafting cloud services, agile development and big data engineering. Her research interest spans around cloud computing, software engineering, test automation and big dat...
Dr. Venkat Ramanathan has wide experience in the fields of IT and business process engineering. He has served industry and academia for over 26 years and has been instrumental in attracting businesses worth several millions through software consulting for clients across Asia, US, Europe and New Z...
Because of COVID-19, many providers are cancelling or postponing in-person programs or providing online participation options.
We are happy to help you find a suitable online alternative.