edu.snu.coral.runtime.executor.data.partitioner

Class DataSkewHashPartitioner

java.lang.Object

edu.snu.coral.runtime.executor.data.partitioner.DataSkewHashPartitioner

All Implemented Interfaces:

Partitioner

public final class DataSkewHashPartitioner
extends Object
implements Partitioner

An implementation of Partitioner which hashes output data from a source task appropriate to detect data skew. It hashes data finer than HashPartitioner. The elements will be hashed by their key, and applied "modulo" operation. When we need to split or recombine the output data from a task after it is stored, we multiply the hash range with a multiplier, which is commonly-known by the source and destination tasks, to prevent the extra deserialize - rehash - serialize process. For more information, please check JobConf.HashRangeMultiplier.

Constructor Summary

Constructors

Constructor and Description
DataSkewHashPartitioner(int hashRangeMultiplier)

Method Summary

Modifier and Type	Method and Description
List<Partition>	partition(Iterable elements, int dstParallelism, KeyExtractor keyExtractor) Divides the output data from a task into multiple blocks.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataSkewHashPartitioner

public DataSkewHashPartitioner(int hashRangeMultiplier)

Method Detail

partition

public List<Partition> partition(Iterable elements,
                                 int dstParallelism,
                                 KeyExtractor keyExtractor)

Description copied from interface: Partitioner

Divides the output data from a task into multiple blocks.

Specified by:

partition in interface Partitioner

Parameters:

elements - the output data from a source task.

dstParallelism - the number of destination tasks.

keyExtractor - extracts keys from elements.

Returns:

the list of partitioned blocks.