Modern Data Engineering is Complicated.
There are so many things to know to be good.
Languages : SQL , Python, Scala
Operating Systems : Linux, bash shell
Cloud : AWS, Azure, GCP
Data Pipelines : Airflow, Kubeflow
DevOps : Kubernetes, Docker, VPCs, IAM etc.
Relational Database : PostGres, Mysql, Sql Server
MPP Databases : Redshift, Google Big Query, Snowflake
Big Data Storage : S3, HDFS, Google storage
Data Architecture : Machine Learning models, data warehouse models
Streaming Data : Kafka, Kinesis Firebase, Flink, Storm
No SQL: Dynodb , Mongo
Business Intelligence related : Tableau, Looker, Sisense
Data Lakes : S3 or hdfs or databricks.
