YARN

YARN (Yet-Another-Resource-Negotiator) provides an API to develop any generic distributed application.

• Handles scheduling and resource requests
• e.g. MapReduce (MR2)
• e.g. Hadoop 2.0 - wanted other processing frameworks to run on Hadoop cluster

• In v1.0 - use Job tracker to track Hadoop jobs

• in v2.0 - different kinds of jobs can be supported (Hadoop, Spark, etc.)

Spark Architecture

There is an “executor” rather than a worker, which executes tasks.

Usually, there are multiple tasks sent to an executor. the Spark driver must send relevant code to run each task. This can be bad.

• e.g. a lambda will capture its environment, which is bad if the environment is large
  • Task must be serialized ⇒ need to ship function and copy of variable
  • If a variable is a 4MB table, should avoid such a send

Broadcast

If a value is Broaadcast, Spark will only send one copy of the value per Executor, not per task.

thresh = sc.broadcast(5)
myRdd.filter(lambda x: x > thresh.value)