com.hadoop.compression.lzo

Class LzoIndex

java.lang.Object

com.hadoop.compression.lzo.LzoIndex

public class LzoIndex
extends Object

Represents the lzo index.

Field Summary

Fields

Modifier and Type	Field and Description
static String	LZO_INDEX_SUFFIX
static String	LZO_TMP_INDEX_SUFFIX
static long	NOT_FOUND

Constructor Summary

Constructors

Constructor and Description
LzoIndex() Create an empty index, typically indicating no index file exists.
LzoIndex(int blocks) Create an index specifying the number of LZO blocks in the file.

Method Summary

Modifier and Type	Method and Description
long	alignSliceEndToIndex(long end, long fileSize) Nudge a given file slice end to the nearest LZO block end no earlier than the current slice end.
long	alignSliceStartToIndex(long start, long end) Nudge a given file slice start to the nearest LZO block start no earlier than the current slice start.
static void	createIndex(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path lzoFile) Index an lzo file to allow the input format to split them into separate map jobs.
long	findNextPosition(long pos) Find the next lzo block start from the given position.
int	getNumberOfBlocks() Get the total number of blocks in the index file.
long	getPosition(int block) Get the block offset for a given block.
boolean	isEmpty() Return true if the index has no blocks set.
static LzoIndex	readIndex(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path lzoFile) Read the index of the lzo file.
void	set(int blockNumber, long pos) Set the position for the block.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LZO_INDEX_SUFFIX

public static final String LZO_INDEX_SUFFIX

See Also:

Constant Field Values

LZO_TMP_INDEX_SUFFIX

public static final String LZO_TMP_INDEX_SUFFIX

See Also:

Constant Field Values

NOT_FOUND

public static final long NOT_FOUND

See Also:

Constant Field Values

Constructor Detail

LzoIndex

public LzoIndex()

Create an empty index, typically indicating no index file exists.

LzoIndex

public LzoIndex(int blocks)

Create an index specifying the number of LZO blocks in the file.

Parameters:

blocks - The number of blocks in the LZO file the index is representing.

Method Detail

set

public void set(int blockNumber,
                long pos)

Set the position for the block.

Parameters:

blockNumber - Block to set pos for.

pos - Position.

getNumberOfBlocks

public int getNumberOfBlocks()

Get the total number of blocks in the index file.

getPosition

public long getPosition(int block)

Get the block offset for a given block.

Parameters:

block -

Returns:

the byte offset into the file where this block starts. It is the developer's responsibility to call getNumberOfBlocks() to know appropriate bounds on the parameter. The argument block should satisfy 0 <= block < getNumberOfBlocks().

findNextPosition

public long findNextPosition(long pos)

Find the next lzo block start from the given position.

Parameters:

pos - The position to start looking from.

Returns:

Either the start position of the block or -1 if it couldn't be found.

isEmpty

public boolean isEmpty()

Return true if the index has no blocks set.

Returns:

true if the index has no blocks set.

alignSliceStartToIndex

public long alignSliceStartToIndex(long start,
                                   long end)

Nudge a given file slice start to the nearest LZO block start no earlier than the current slice start.

Parameters:

start - The current slice start

end - The current slice end

Returns:

The smallest block offset in the index between [start, end), or NOT_FOUND if there is none such.

alignSliceEndToIndex

public long alignSliceEndToIndex(long end,
                                 long fileSize)

Nudge a given file slice end to the nearest LZO block end no earlier than the current slice end.

Parameters:

end - The current slice end

fileSize - The size of the file, i.e. the max end position.

Returns:

The smallest block offset in the index between [end, fileSize].

readIndex

public static LzoIndex readIndex(org.apache.hadoop.fs.FileSystem fs,
                                 org.apache.hadoop.fs.Path lzoFile)
                          throws IOException

Read the index of the lzo file.

Parameters:

fs - The index file is on this file system.

lzoFile - the file whose index we are reading -- NOT the index file itself. That is, pass in filename.lzo, not filename.lzo.index, for this parameter.

Throws:

IOException

createIndex

public static void createIndex(org.apache.hadoop.fs.FileSystem fs,
                               org.apache.hadoop.fs.Path lzoFile)
                        throws IOException

Index an lzo file to allow the input format to split them into separate map jobs.

Parameters:

fs - File system that contains the file.

lzoFile - the lzo file to index. For filename.lzo, the created index file will be filename.lzo.index.

Throws:

IOException