Constructor and Description |
---|
IREdge(DataCommunicationPatternProperty.Value commPattern,
IRVertex src,
IRVertex dst,
Coder coder)
Constructor of IREdge.
|
IREdge(DataCommunicationPatternProperty.Value commPattern,
IRVertex src,
IRVertex dst,
Coder coder,
Boolean isSideInput)
Constructor of IREdge.
|
Modifier and Type | Method and Description |
---|---|
static ExecutionPropertyMap |
ExecutionPropertyMap.of(IRVertex irVertex)
Static initializer for irVertex.
|
Modifier and Type | Class and Description |
---|---|
class |
InitializedSourceVertex<T>
Source vertex with initial data as object.
|
class |
LoopVertex
IRVertex that contains a partial DAG that is iterative.
|
class |
MetricCollectionBarrierVertex<T>
IRVertex that collects statistics to send them to the optimizer for dynamic optimization.
|
class |
OperatorVertex
IRVertex that transforms input data.
|
class |
SourceVertex<O>
IRVertex that reads data from an external source.
|
Modifier and Type | Method and Description |
---|---|
abstract IRVertex |
IRVertex.getClone() |
IRVertex |
IRVertex.setProperty(ExecutionProperty<?> executionProperty)
Set an executionProperty of the IRVertex.
|
Modifier and Type | Method and Description |
---|---|
DAGBuilder<IRVertex,IREdge> |
LoopVertex.getBuilder() |
DAG<IRVertex,IREdge> |
LoopVertex.getDAG() |
Map<IRVertex,Set<IREdge>> |
LoopVertex.getDagIncomingEdges() |
Map<IRVertex,Set<IREdge>> |
LoopVertex.getDagOutgoingEdges() |
DAG<IRVertex,IREdge> |
MetricCollectionBarrierVertex.getDAGSnapshot()
Access the DAG snapshot when triggering dynamic optimization.
|
Map<IRVertex,Set<IREdge>> |
LoopVertex.getIterativeIncomingEdges() |
Map<IRVertex,Set<IREdge>> |
LoopVertex.getNonIterativeIncomingEdges() |
Modifier and Type | Method and Description |
---|---|
void |
IRVertex.copyExecutionPropertiesTo(IRVertex thatVertex)
Static function to copy executionProperties from a vertex to the other.
|
Modifier and Type | Method and Description |
---|---|
void |
MetricCollectionBarrierVertex.setDAGSnapshot(DAG<IRVertex,IREdge> dag)
This is to set the DAG snapshot at the end of the DataSkewCompositePass.
|
LoopVertex |
LoopVertex.unRollIteration(DAGBuilder<IRVertex,IREdge> dagBuilder)
Method for unrolling an iteration of the LoopVertex.
|
Modifier and Type | Method and Description |
---|---|
Plan |
Backend.compile(DAG<IRVertex,IREdge> dag)
Compiles a DAG to a physical execution plan.
|
Modifier and Type | Method and Description |
---|---|
PhysicalPlan |
CoralBackend.compile(DAG<IRVertex,IREdge> irDAG)
Compiles an IR DAG into a
PhysicalPlan to be submitted to Runtime. |
PhysicalPlan |
CoralBackend.compile(DAG<IRVertex,IREdge> irDAG,
PhysicalPlanGenerator physicalPlanGenerator)
Compiles an IR DAG into a
PhysicalPlan to be submitted to Runtime. |
Constructor and Description |
---|
CoralPipelineVisitor(DAGBuilder<IRVertex,IREdge> builder,
org.apache.beam.sdk.options.PipelineOptions options)
Constructor of the BEAM Visitor.
|
Modifier and Type | Class and Description |
---|---|
class |
BeamBoundedSourceVertex<O>
SourceVertex implementation for BoundedSource.
|
Modifier and Type | Method and Description |
---|---|
static <T> List<T> |
SparkFrontendUtils.collect(DAG<IRVertex,IREdge> dag,
Stack<LoopVertex> loopVertexStack,
IRVertex lastVertex,
org.apache.spark.serializer.Serializer serializer)
Collect data by running the DAG.
|
Modifier and Type | Method and Description |
---|---|
static <T> List<T> |
SparkFrontendUtils.collect(DAG<IRVertex,IREdge> dag,
Stack<LoopVertex> loopVertexStack,
IRVertex lastVertex,
org.apache.spark.serializer.Serializer serializer)
Collect data by running the DAG.
|
Modifier and Type | Class and Description |
---|---|
class |
SparkBoundedSourceVertex<T>
Bounded source vertex for Spark.
|
Modifier and Type | Method and Description |
---|---|
static DAG<IRVertex,IREdge> |
CompiletimeOptimizer.optimize(DAG<IRVertex,IREdge> dag,
Policy optimizationPolicy,
String dagDirectory)
Optimize function.
|
Modifier and Type | Method and Description |
---|---|
static DAG<IRVertex,IREdge> |
CompiletimeOptimizer.optimize(DAG<IRVertex,IREdge> dag,
Policy optimizationPolicy,
String dagDirectory)
Optimize function.
|
Modifier and Type | Class and Description |
---|---|
static class |
EmptyComponents.EmptySourceVertex<T>
An empty Source Vertex.
|
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
DisaggregationEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewEdgeMetricCollectionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DefaultEdgeUsedDataHandlingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DefaultStagePartitioningPass.apply(DAG<IRVertex,IREdge> irDAG) |
DAG<IRVertex,IREdge> |
SailfishEdgeUsedDataHandlingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
ScheduleGroupPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
CompressionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
PadoVertexExecutorPlacementPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewVertexPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
ShuffleEdgePushPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DefaultParallelismPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
ReviseInterStageEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
SailfishEdgeDataFlowModelPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewEdgePartitionerPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
PadoEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
PadoEdgeDataFlowModelPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
SailfishEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
DisaggregationEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewEdgeMetricCollectionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DefaultEdgeUsedDataHandlingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DefaultStagePartitioningPass.apply(DAG<IRVertex,IREdge> irDAG) |
DAG<IRVertex,IREdge> |
SailfishEdgeUsedDataHandlingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
ScheduleGroupPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
CompressionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
PadoVertexExecutorPlacementPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewVertexPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
ShuffleEdgePushPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DefaultParallelismPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
ReviseInterStageEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
SailfishEdgeDataFlowModelPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewEdgePartitionerPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
PadoEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
PadoEdgeDataFlowModelPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
SailfishEdgeDataStorePass.apply(DAG<IRVertex,IREdge> dag) |
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
CompositePass.apply(DAG<IRVertex,IREdge> irVertexIREdgeDAG) |
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
CompositePass.apply(DAG<IRVertex,IREdge> irVertexIREdgeDAG) |
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
LoopUnrollingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewReshapingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
LoopOptimizations.LoopFusionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
LoopOptimizations.LoopInvariantCodeMotionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
SailfishRelayReshapingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
CommonSubexpressionEliminationPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
LoopExtractionPass.apply(DAG<IRVertex,IREdge> dag) |
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
LoopUnrollingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
DataSkewReshapingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
LoopOptimizations.LoopFusionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
LoopOptimizations.LoopInvariantCodeMotionPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
SailfishRelayReshapingPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
CommonSubexpressionEliminationPass.apply(DAG<IRVertex,IREdge> dag) |
DAG<IRVertex,IREdge> |
LoopExtractionPass.apply(DAG<IRVertex,IREdge> dag) |
Modifier and Type | Method and Description |
---|---|
IRVertex |
PhysicalStageEdge.getDstVertex() |
IRVertex |
PhysicalPlan.getIRVertexOf(Task task)
Get an IR vertex of the given task.
|
IRVertex |
PhysicalStageEdge.getSrcVertex() |
Modifier and Type | Method and Description |
---|---|
Map<Task,IRVertex> |
PhysicalPlanGenerator.getTaskIRVertexMap()
Getter for taskIRVertexMap.
|
Map<Task,IRVertex> |
PhysicalPlan.getTaskIRVertexMap() |
Modifier and Type | Method and Description |
---|---|
DAG<PhysicalStage,PhysicalStageEdge> |
PhysicalPlanGenerator.apply(DAG<IRVertex,IREdge> irDAG)
Generates the
PhysicalPlan to be executed. |
DAG<Stage,StageEdge> |
PhysicalPlanGenerator.stagePartitionIrDAG(DAG<IRVertex,IREdge> irDAG)
We take the stage-partitioned DAG and create actual stage and stage edge objects to create a DAG of stages.
|
Constructor and Description |
---|
PhysicalStageEdge(String runtimeEdgeId,
ExecutionPropertyMap edgeProperties,
IRVertex srcVertex,
IRVertex dstVertex,
PhysicalStage srcStage,
PhysicalStage dstStage,
Coder coder,
Boolean isSideInput)
Constructor.
|
Constructor and Description |
---|
PhysicalPlan(String id,
DAG<PhysicalStage,PhysicalStageEdge> stageDAG,
Map<Task,IRVertex> taskIRVertexMap)
Constructor.
|
Modifier and Type | Method and Description |
---|---|
IRVertex |
StageEdge.getDstVertex() |
IRVertex |
StageEdge.getSrcVertex() |
Modifier and Type | Method and Description |
---|---|
DAG<IRVertex,IREdge> |
Stage.getStageInternalDAG() |
Modifier and Type | Method and Description |
---|---|
StageBuilder |
StageBuilder.addVertex(IRVertex vertex)
Adds a
IRVertex to this stage. |
StageEdgeBuilder |
StageEdgeBuilder.setDstVertex(IRVertex dv)
Setter for the destination vertex.
|
StageEdgeBuilder |
StageEdgeBuilder.setSrcVertex(IRVertex sv)
Setter for the source vertex.
|
Constructor and Description |
---|
StageEdge(String irEdgeId,
ExecutionPropertyMap edgeProperties,
Stage srcStage,
Stage dstStage,
Coder coder,
Boolean isSideInput,
IRVertex srcVertex,
IRVertex dstVertex)
Represents the edge between stages.
|
Constructor and Description |
---|
Stage(String stageId,
DAG<IRVertex,IREdge> stageInternalDAG,
int scheduleGroupIndex)
Constructor.
|
Modifier and Type | Method and Description |
---|---|
InputReader |
DataTransferFactory.createReader(int dstTaskIdx,
IRVertex srcIRVertex,
RuntimeEdge runtimeEdge)
Creates an
InputReader between two stages. |
OutputWriter |
DataTransferFactory.createWriter(Task srcTask,
int srcTaskIdx,
IRVertex dstIRVertex,
RuntimeEdge<?> runtimeEdge)
Creates an
OutputWriter between two stages. |
Constructor and Description |
---|
InputReader(int dstTaskIndex,
IRVertex srcVertex,
RuntimeEdge runtimeEdge,
BlockManagerWorker blockManagerWorker) |
OutputWriter(int hashRangeMultiplier,
int srcTaskIdx,
String srcRuntimeVertexId,
IRVertex dstIrVertex,
RuntimeEdge<?> runtimeEdge,
BlockManagerWorker blockManagerWorker) |
Copyright © 2018. All rights reserved.