Modifier and Type | Method and Description |
---|---|
static void |
JobLauncher.launchDAG(DAG dag)
Launch application using the application DAG.
|
Modifier and Type | Method and Description |
---|---|
DAG<V,E> |
DAGBuilder.build()
Build the DAG.
|
DAG<V,E> |
DAGBuilder.buildWithoutSourceSinkCheck()
Build the DAG for LoopVertex.
|
<V2 extends Vertex,E2 extends Edge<V2>> |
DAG.convert(Function<DAG<V,E>,DAG<V2,E2>> function)
Converts a DAG into another DAG according to a function.
|
Modifier and Type | Method and Description |
---|---|
DAGBuilder<V,E> |
DAGBuilder.addVertex(V v,
DAG<V,E> dag)
Add vertex to the builder, using the information from the given DAG.
|
Modifier and Type | Method and Description |
---|---|
<V2 extends Vertex,E2 extends Edge<V2>> |
DAG.convert(Function<DAG<V,E>,DAG<V2,E2>> function)
Converts a DAG into another DAG according to a function.
|
<V2 extends Vertex,E2 extends Edge<V2>> |
DAG.convert(Function<DAG<V,E>,DAG<V2,E2>> function)
Converts a DAG into another DAG according to a function.
|
Constructor and Description |
---|
DAGBuilder(DAG<V,E> dag)
Constructor of DAGBuilder with a DAG to start from.
|
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
LoopVertex.getDAG() |
DAG<IRVertex,IREdge> |
MetricCollectionBarrierVertex.getDAGSnapshot()
Access the DAG snapshot when triggering dynamic optimization.
|
Modifier and Type | Method and Description |
---|---|
void |
MetricCollectionBarrierVertex.setDAGSnapshot(DAG<IRVertex,IREdge> dag)
This is to set the DAG snapshot at the end of the DataSkewCompositePass.
|
Modifier and Type | Method and Description |
---|---|
Plan |
Backend.compile(DAG<IRVertex,IREdge> dag)
Compiles a DAG to a physical execution plan.
|
Modifier and Type | Method and Description |
---|---|
PhysicalPlan |
CoralBackend.compile(DAG<IRVertex,IREdge> irDAG)
Compiles an IR DAG into a
PhysicalPlan to be submitted to Runtime. |
PhysicalPlan |
CoralBackend.compile(DAG<IRVertex,IREdge> irDAG,
PhysicalPlanGenerator physicalPlanGenerator)
Compiles an IR DAG into a
PhysicalPlan to be submitted to Runtime. |
Modifier and Type | Method and Description |
---|---|
static <T> List<T> |
SparkFrontendUtils.collect(DAG<IRVertex,IREdge> dag,
Stack<LoopVertex> loopVertexStack,
IRVertex lastVertex,
org.apache.spark.serializer.Serializer serializer)
Collect data by running the DAG.
|
Modifier and Type | Method and Description |
---|---|
static DAG<IRVertex,IREdge> |
CompiletimeOptimizer.optimize(DAG<IRVertex,IREdge> dag,
Policy optimizationPolicy,
String dagDirectory)
Optimize function.
|
Modifier and Type | Method and Description |
---|---|
static DAG<IRVertex,IREdge> |
CompiletimeOptimizer.optimize(DAG<IRVertex,IREdge> dag,
Policy optimizationPolicy,
String dagDirectory)
Optimize function.
|
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
DisaggregationEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewEdgeMetricCollectionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DefaultEdgeUsedDataHandlingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DefaultStagePartitioningPass.apply(DAG<IRVertex,IREdge> irDAG) |
DAG<IRVertex,IREdge> |
SailfishEdgeUsedDataHandlingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
ScheduleGroupPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
CompressionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
PadoVertexExecutorPlacementPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewVertexPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
ShuffleEdgePushPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DefaultParallelismPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
ReviseInterStageEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
SailfishEdgeDataFlowModelPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewEdgePartitionerPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
PadoEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
PadoEdgeDataFlowModelPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
SailfishEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
DisaggregationEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewEdgeMetricCollectionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DefaultEdgeUsedDataHandlingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DefaultStagePartitioningPass.apply(DAG<IRVertex,IREdge> irDAG) |
DAG<IRVertex,IREdge> |
SailfishEdgeUsedDataHandlingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
ScheduleGroupPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
CompressionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
PadoVertexExecutorPlacementPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewVertexPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
ShuffleEdgePushPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DefaultParallelismPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
ReviseInterStageEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
SailfishEdgeDataFlowModelPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewEdgePartitionerPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
PadoEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
PadoEdgeDataFlowModelPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
SailfishEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
CompositePass.apply(DAG<IRVertex,IREdge> irVertexIREdgeDAG) |
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
CompositePass.apply(DAG<IRVertex,IREdge> irVertexIREdgeDAG) |
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
LoopUnrollingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewReshapingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
LoopOptimizations.LoopFusionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
LoopOptimizations.LoopInvariantCodeMotionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
SailfishRelayReshapingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
CommonSubexpressionEliminationPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
LoopExtractionPass.apply(DAG<IRVertex,IREdge> dag) |
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
LoopUnrollingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewReshapingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
LoopOptimizations.LoopFusionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
LoopOptimizations.LoopInvariantCodeMotionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
SailfishRelayReshapingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
CommonSubexpressionEliminationPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
LoopExtractionPass.apply(DAG<IRVertex,IREdge> dag) |
Modifier and Type | Method and Description |
---|---|
DAG<PhysicalStage,PhysicalStageEdge> |
PhysicalPlanGenerator.apply(DAG<IRVertex,IREdge> irDAG)
Generates the
PhysicalPlan to be executed. |
DAG<PhysicalStage,PhysicalStageEdge> |
PhysicalPlan.getStageDAG() |
DAG<Task,RuntimeEdge<Task>> |
PhysicalStage.getTaskGroupDag() |
DAG<Stage,StageEdge> |
PhysicalPlanGenerator.stagePartitionIrDAG(DAG<IRVertex,IREdge> irDAG)
We take the stage-partitioned DAG and create actual stage and stage edge objects to create a DAG of stages.
|
Modifier and Type | Method and Description |
---|---|
DAG<PhysicalStage,PhysicalStageEdge> |
PhysicalPlanGenerator.apply(DAG<IRVertex,IREdge> irDAG)
Generates the
PhysicalPlan to be executed. |
DAG<Stage,StageEdge> |
PhysicalPlanGenerator.stagePartitionIrDAG(DAG<IRVertex,IREdge> irDAG)
We take the stage-partitioned DAG and create actual stage and stage edge objects to create a DAG of stages.
|
Constructor and Description |
---|
PhysicalPlan(String id,
DAG<PhysicalStage,PhysicalStageEdge> stageDAG,
Map<Task,IRVertex> taskIRVertexMap)
Constructor.
|
PhysicalStage(String stageId,
DAG<Task,RuntimeEdge<Task>> taskGroupDag,
int parallelism,
int scheduleGroupIndex,
String containerType)
Constructor.
|
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
Stage.getStageInternalDAG() |
Constructor and Description |
---|
Stage(String stageId,
DAG<IRVertex,IREdge> stageInternalDAG,
int scheduleGroupIndex)
Constructor.
|
Constructor and Description |
---|
TaskGroupExecutor(ScheduledTaskGroup scheduledTaskGroup,
DAG<Task,RuntimeEdge<Task>> taskGroupDag,
TaskGroupStateManager taskGroupStateManager,
DataTransferFactory channelFactory) |
TaskGroupStateManager(ScheduledTaskGroup scheduledTaskGroup,
DAG<Task,RuntimeEdge<Task>> taskGroupDag,
String executorId,
PersistentConnectionToMasterMap persistentConnectionToMasterMap,
MetricMessageSender metricMessageSender) |
Copyright © 2018. All rights reserved.