What is Apache Spark
What is Apache Spark? Learn how this distributed engine unifies large-scale batch, streaming, SQL, machine learning, and graph processing in one runtime.
In the full article
Read the transcript
Auto-generated captions; wording may contain recognition errors.
[music] Apache Spark is a distributed processing engine designed to run largecale data workloads faster and more flexibly than what was possible with earlier [music] batch processing frameworks. It achieves this through in-memory computation, a unified programming model and native support for a broad range of workload [music] types within a single engine. The core concept in Spark is the resilient distributed data set or RDD. An RDD is an immutable partition collection of data distributed across a cluster that can be processed in parallel. Spark builds a directed a cyclic graph of operations called the execution plan before running anything. [music] This lets the engine optimize the full sequence of transformations together rather than executing each step independently. Modern Spark code is almost always written against higher level abstractions, dataf frames, and the Spark SQL API rather than RDDs directly. [music] These abstractions let Spark apply a query optimizer similar to what database engines use, [music] choosing physical execution strategies based on the shape of the data and the operations being performed. Spark handles batch processing, streaming, machine learning, and graph computation within the same runtime. [music] Spark structured streaming processes event data as an unbounded data frame applying the same query semantics used for batch [music] data. MLIB provides distributed implementations of common machine learning algorithms. [music] This unification matters in practice because it eliminates the operational complexity of running separate engines for [music] different workload types. Fault tolerance is built into the execution model. Spark tracks the lineage of every transformation applied to each partition. If a partition is lost due to a node failure, [music] Spark recomputes only that partition from its lineage rather than restarting the entire job. Spark runs on Hadoop, [music] yarn, Kubernetes, and standalone cluster managers making it portable across infrastructure environments. [music]


