edu.snu.coral.runtime.executor.data

Class DataUtil

java.lang.Object

edu.snu.coral.runtime.executor.data.DataUtil

public final class DataUtil
extends Object

Utility methods for data handling (e.g., (de)serialization).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DataUtil.InputStreamIterator<T> An iterator that emits objects from InputStream using the corresponding Coder.

Method Summary

Modifier and Type	Method and Description
static String	blockIdToFilePath(String blockId, String fileDirectory) Converts a block id to the corresponding file path.
static String	blockIdToMetaFilePath(String blockId, String fileDirectory) Converts a block id to the corresponding metadata file path.
static Iterable	concatNonSerPartitions(Iterable<NonSerializedPartition> partitionsToConcat) Concatenates an iterable of non-serialized Partitions into a single iterable of elements.
static <K extends Serializable> Iterable<NonSerializedPartition<K>>	convertToNonSerPartitions(Coder coder, Iterable<SerializedPartition<K>> partitionsToConvert) Converts the serialized Partitions in an iterable to non-serialized Partitions.
static <K extends Serializable> Iterable<SerializedPartition<K>>	convertToSerPartitions(Coder coder, Iterable<NonSerializedPartition<K>> partitionsToConvert) Converts the non-serialized Partitions in an iterable to serialized Partitions.
static <K extends Serializable> NonSerializedPartition	deserializePartition(long elementsInPartition, Coder coder, K key, InputStream inputStream) Reads the data of a partition from an input stream and deserializes it.
static long	serializePartition(Coder coder, NonSerializedPartition nonSerializedPartition, ByteArrayOutputStream bytesOutputStream) Serializes the elements in a non-serialized partition into an output stream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

serializePartition

public static long serializePartition(Coder coder,
                                      NonSerializedPartition nonSerializedPartition,
                                      ByteArrayOutputStream bytesOutputStream)
                               throws IOException

Serializes the elements in a non-serialized partition into an output stream.

Parameters:

coder - the coder to encode the elements.

nonSerializedPartition - the non-serialized partition to serialize.

bytesOutputStream - the output stream to write.

Returns:

total number of elements in the partition.

Throws:

IOException - if fail to serialize.

deserializePartition

public static <K extends Serializable> NonSerializedPartition deserializePartition(long elementsInPartition,
                                                                                   Coder coder,
                                                                                   K key,
                                                                                   InputStream inputStream)
                                                                            throws IOException

Reads the data of a partition from an input stream and deserializes it.

Type Parameters:

K - the key type of the partitions.

Parameters:

elementsInPartition - the number of elements in this partition.

coder - the coder to decode the bytes.

key - the key value of the result partition.

inputStream - the input stream which will return the data in the partition as bytes.

Returns:

the list of deserialized elements.

Throws:

IOException - if fail to deserialize.

convertToSerPartitions

public static <K extends Serializable> Iterable<SerializedPartition<K>> convertToSerPartitions(Coder coder,
                                                                                               Iterable<NonSerializedPartition<K>> partitionsToConvert)
                                                                                        throws IOException

Converts the non-serialized Partitions in an iterable to serialized Partitions.

Type Parameters:

K - the key type of the partitions.

Parameters:

coder - the coder for serialization.

partitionsToConvert - the partitions to convert.

Returns:

the converted SerializedPartitions.

Throws:

IOException - if fail to convert.

convertToNonSerPartitions

public static <K extends Serializable> Iterable<NonSerializedPartition<K>> convertToNonSerPartitions(Coder coder,
                                                                                                     Iterable<SerializedPartition<K>> partitionsToConvert)
                                                                                              throws IOException

Converts the serialized Partitions in an iterable to non-serialized Partitions.

Type Parameters:

K - the key type of the partitions.

Parameters:

coder - the coder for deserialization.

partitionsToConvert - the partitions to convert.

Returns:

the converted NonSerializedPartitions.

Throws:

IOException - if fail to convert.

blockIdToFilePath

public static String blockIdToFilePath(String blockId,
                                       String fileDirectory)

Converts a block id to the corresponding file path.

Parameters:

blockId - the ID of the block.

fileDirectory - the directory of the target block file.

Returns:

the file path of the partition.

blockIdToMetaFilePath

public static String blockIdToMetaFilePath(String blockId,
                                           String fileDirectory)

Converts a block id to the corresponding metadata file path.

Parameters:

blockId - the ID of the block.

fileDirectory - the directory of the target block file.

Returns:

the metadata file path of the partition.

concatNonSerPartitions

public static Iterable concatNonSerPartitions(Iterable<NonSerializedPartition> partitionsToConcat)
                                       throws IOException

Concatenates an iterable of non-serialized Partitions into a single iterable of elements.

Parameters:

partitionsToConcat - the partitions to concatenate.

Returns:

the concatenated iterable of all elements.

Throws:

IOException - if fail to concatenate.