Python ETL pipeline with PostgreSQL
Build a complete ETL pipeline that extracts data from a REST API and CSV files, transforms it with pandas using dimensional modelling, and loads it into a PostgreSQL data warehouse.
PythonpandasPostgreSQL
Home / Courses / Data Engineering
Data Engineering
Python ETL, Apache Spark, Kafka, Airflow, dbt, Snowflake, BigQuery, and AWS Glue, the complete modern data stack for a career as the engineer who moves, transforms, and delivers data at scale.
Curriculum
Four modules covering Python ETL and SQL foundations, distributed processing with Spark and Kafka, orchestration and transformation with Airflow and dbt, and cloud data warehousing on Snowflake, BigQuery, and AWS.
Wk 1: Python for data engineering, data structures, file I/O (CSV, JSON, Parquet), pandas for ETL, error handling, logging best practices
Wk 2: Advanced SQL, JOINs, window functions, CTEs, query optimisation, indexes, explain plans; PostgreSQL, schemas, sequences, constraints, stored procedures
Wk 3: Data modelling, dimensional modelling (star schema, snowflake schema), fact and dimension tables, slowly changing dimensions (SCD Type 1/2/3)
Wk 4: APIs and data ingestion, REST API calls with requests, pagination, rate limiting, web scraping with BeautifulSoup/Scrapy, writing to PostgreSQL and S3
Wk 5: Apache Spark architecture, RDDs, DataFrames, Dataset API, Spark cluster modes (local, YARN, Kubernetes); PySpark setup and DataFrame operations
Wk 6: PySpark transformations, groupBy, joins, window functions, UDFs, broadcast joins, handling skew; writing to S3, HDFS, and databases
Wk 7: Spark SQL, temporary views, Spark Catalog, reading Parquet/Delta files, Spark Thrift Server; MLlib intro
Wk 8: Apache Kafka, producer/consumer API, topics, partitions, offsets, consumer groups, schema registry (Avro), Kafka Connect
Wk 9: Spark Structured Streaming, reading from Kafka, watermarking, stateful operations, write to Delta Lake; end-to-end streaming pipeline lab
Wk 10: Apache Airflow, architecture, DAG authoring, BashOperator, PythonOperator, task dependencies, scheduling expressions, backfill
Wk 11: Airflow advanced, XCom for task communication, Airflow Variables and Connections, Hooks, custom operators, Astronomer Cosmos for dbt integration, Airflow monitoring
Wk 12: dbt fundamentals, project structure, models, sources, refs, tests (generic and singular), documentation, Jinja templating, materialisation strategies
Wk 13: dbt advanced, macros, packages (dbt_utils), snapshots (SCD Type 2), exposures, dbt Cloud vs dbt Core; Great Expectations for data quality testing; orchestrating dbt runs in Airflow
Wk 14: Snowflake, architecture (virtual warehouses, micro-partitions), Snowflake SQL extensions, Time Travel, Zero-copy cloning, Streams and Tasks (CDC), Snowpipe for auto-ingest; Snowflake + dbt integration
Wk 15: BigQuery (GCP), serverless architecture, partitioning and clustering, BigQuery ML, loading from GCS; AWS Glue, serverless ETL with PySpark, Glue Data Catalog, AWS Lake Formation for data lake governance; Delta Lake on S3
Wk 16: Capstone project: end-to-end data platform, Kafka for ingestion, Spark for processing, Airflow for orchestration, dbt for transformation, Snowflake as data warehouse, Metabase for dashboards; certification guidance (SnowPro Core, GCP Professional Data Engineer, dbt Analytics Engineer)
Hands-on Projects
Four portfolio projects you can walk through in a data engineering interview, each one representing a core pattern used in production data teams.
Build a complete ETL pipeline that extracts data from a REST API and CSV files, transforms it with pandas using dimensional modelling, and loads it into a PostgreSQL data warehouse.
Process a large e-commerce transactions dataset with PySpark, aggregations, window functions, writing to Parquet on S3, and reading with Spark SQL, within a Kubernetes cluster.
Author an Airflow DAG that orchestrates a full dbt pipeline, running models, tests, and documentation generation, with failure alerting and data quality checks via Great Expectations.
Kafka ingestion → Spark processing → Airflow orchestration → dbt transformation → Snowflake data warehouse → Metabase dashboards. A complete, production-pattern data platform.
Career Outcomes
Typical starting salaries in Pune for data engineering roles, an observation, not a guarantee.
Guidance for SnowPro Core, GCP Professional Data Engineer, and dbt Analytics Engineer certification tracks in the final module.
A TechSummitTrainers completion certificate for the Data Engineering program after the capstone project.
Roles this prepares you for
Course FAQ
Data Science is about extracting insights and building models from data. Data Engineering is about building and maintaining the infrastructure and pipelines that make data available for data scientists and analysts to work with. Data engineers design data models, build ETL pipelines, manage data warehouses, and ensure data quality and freshness.
Apache Spark is a distributed data processing engine for large-scale data workloads. Where standard SQL might process millions of rows, Spark processes billions across a cluster in parallel. PySpark is the Python API for Spark. It is the most widely used data engineering tool globally, for batch processing, streaming (Structured Streaming), SQL analytics (Spark SQL), and ML at scale (MLlib).
Apache Kafka is a distributed event streaming platform that acts as a high-throughput, durable message bus between systems. Producers write events to Kafka topics; consumers read from them. Kafka enables real-time data pipelines, processing data as it arrives instead of in batches. It is used for event sourcing, change data capture, and real-time analytics at every large-scale data platform.
Apache Airflow is a workflow orchestration platform for scheduling and monitoring data pipelines. You define pipelines as DAGs (Directed Acyclic Graphs) in Python code, specifying task order, dependencies, retry logic, and scheduling. Airflow handles monitoring with a visual dashboard and is the standard orchestration tool for data engineering teams.
dbt (data build tool) is a transformation framework for data warehouses. After raw data lands in Snowflake or BigQuery, dbt manages the SQL transformations into analytics-ready models, with version control, testing, documentation, and modular model references. The modern data stack (extract/load → warehouse → dbt transform → BI tool) is now standard.
Snowflake is a cloud-native data warehouse that separates compute from storage. It runs on AWS, Azure, and GCP, with multi-cluster query processing, automatic scaling, zero-copy cloning, and time travel. It is one of the most widely adopted data warehouses in the enterprise, and Snowflake skills are consistently requested in data engineering job postings.
Data Engineer, Big Data Engineer, Analytics Engineer, ETL Developer, Data Platform Engineer, ML Data Engineer. Data engineering is one of the fastest-growing roles in Pune's IT sector. Fresher salaries in Pune range from ₹5 LPA to ₹12 LPA.
The course fee is discussed during your free demo call. We offer flexible payment options. Contact us to know more.
Module 1 covers Python from a data engineering perspective. If you know the basics (variables, loops, functions), Module 1 will be fast-paced but manageable. If you are entirely new to Python, spending a few weeks on Python basics before the course starts is recommended.
The Data Science course focuses on statistical analysis, machine learning models, and deriving insights using pandas, scikit-learn, and TensorFlow. This course focuses on building the infrastructure that feeds those models, ETL pipelines, Spark, Kafka, Airflow, Snowflake. Both are complementary, data engineers and data scientists work closely together on data teams.
Yes, Module 4 covers Snowflake, BigQuery (GCP), AWS Glue, and Delta Lake on S3. The course is cloud-agnostic, working with both AWS and GCP data services.
The curriculum covers concepts aligned with the dbt Analytics Engineering certification, Snowflake SnowPro Core certification, and Google Professional Data Engineer certification tracks, with guidance in Module 4 on which to pursue based on your target role.
Data Engineering is the discipline of designing, building, and maintaining the systems that collect, store, transform, and serve data to the people and systems that need it. While data scientists build models and analysts build dashboards, neither of them can do their work reliably without data engineers who have built the pipelines, warehouses, and quality frameworks that make clean, timely data available. The popular phrase in data teams is that 80% of a data scientist's time goes to cleaning data, data engineers solve that problem at an infrastructure level, so scientists can spend their time on models instead.
India's IT sector has seen explosive demand for data engineers over the past three years. The combination of the cloud data warehouse revolution (Snowflake, BigQuery, Redshift replacing on-premise Hadoop clusters), the proliferation of real-time data requirements (Kafka-based event streaming), and the ML platform build-outs across enterprises has created more demand than the market can supply. Pune is a hub for this demand, the large IT service firms run data engineering practices for global clients, and product companies and startups build internal data platforms. Fresher salaries for data engineering roles in Pune range from ₹5 LPA to ₹12 LPA, consistently higher than typical software development fresher bands.
The "modern data stack" is the set of tools that has become the standard architecture for data platforms over the past five years. It typically consists of: a data ingestion layer (Fivetran, Airbyte, or custom Python ETL); a cloud data warehouse (Snowflake, BigQuery, or Redshift) for storage and analytics; a transformation layer (dbt) for turning raw data into clean, tested analytical models; an orchestration layer (Apache Airflow or Prefect) for scheduling and monitoring pipelines; and a BI layer (Metabase, Looker, or Power BI) for visualisation. Apache Spark sits adjacent to this stack for large-scale batch and streaming processing workloads that exceed what a data warehouse can handle natively, and Apache Kafka underpins the real-time event streaming layer. This course teaches all of these layers in sequence.
Apache Spark is a distributed computing engine designed to process data at a scale that exceeds what a single machine (or even a data warehouse's SQL engine) can efficiently handle. The decision point is roughly this: for data volumes under a few hundred GB and query patterns that fit standard SQL, a modern cloud data warehouse like Snowflake or BigQuery is the right tool, it is simpler, cheaper to operate, and fast enough. For processing terabytes or petabytes, complex ML feature engineering, unstructured data (images, text, logs), or streaming data with complex stateful logic, Spark is the right tool. In practice, most data engineering teams use both: a Spark layer for heavy-lift ETL and streaming, and a data warehouse plus dbt for analytical transformations and BI-facing models.
Pune's large IT service firms (and product companies with internal data platforms) hire for all of these roles consistently. The SnowPro Core, GCP Professional Data Engineer, and dbt Analytics Engineer certifications each signal depth in a specific tool and significantly improve interview outcomes.
TechSummitTrainers is located at Office No. 509, Kushal Wallstreet, Fergusson College Road, Shivajinagar, Pune, central Pune, accessible from Shivajinagar station, Deccan, and surrounding areas. This course is taught by practising data engineers who run production Spark jobs, Kafka streams, and Airflow DAGs on live data platforms. Lab sessions run on real AWS and GCP environments, not simulated setups. By the time you finish, you will have built four complete data engineering projects covering Python ETL, PySpark batch processing, Airflow + dbt orchestration, and a full end-to-end modern data platform. Both classroom and online formats are supported, with every session recorded for review.
Book your free demo today. No payment, no commitment. Next batch starts Nov 10, 2026.
Get a Call Back →