org.apache.hcatalog.pig
Class HCatLoader

Object
  LoadFunc
      HCatLoader

All Implemented Interfaces:

LoadMetadata, LoadPushDown

public class HCatLoader
extends LoadFunc

Pig LoadFunc to read data from HCat

Nested Class Summary

Nested classes/interfaces inherited from interface LoadPushDown

LoadPushDown.OperatorSet, LoadPushDown.RequiredField, LoadPushDown.RequiredFieldList, LoadPushDown.RequiredFieldResponse

Field Summary

static String	INNER_SIGNATURE
static String	INNER_SIGNATURE_PREFIX
protected static String	PRUNE_PROJECTION_INFO
protected String	signature

Constructor Summary

HCatLoader()

Method Summary

List<LoadPushDown.OperatorSet>	getFeatures()
InputFormat<?,?>	getInputFormat()
Tuple	getNext()
String[]	getPartitionKeys(String location, Job job)
ResourceSchema	getSchema(String location, Job job)
protected static long	getSizeInBytes(InputJobInfo inputJobInfo) A utility method to get the size of inputs.
ResourceStatistics	getStatistics(String location, Job job) Get statistics about the data to be loaded.
void	prepareToRead(RecordReader reader, PigSplit arg1)
LoadPushDown.RequiredFieldResponse	pushProjection(LoadPushDown.RequiredFieldList requiredFieldsInfo)
String	relativeToAbsolutePath(String location, Path curDir)
void	setLocation(String location, Job job)
void	setPartitionFilter(Expression partitionFilter)
void	setUDFContextSignature(String signature)
protected void	storeInUDFContext(String signature, String key, Object value)

Methods inherited from class LoadFunc

getAbsolutePath, getLoadCaster, getPathStrings, join

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INNER_SIGNATURE

public static final String INNER_SIGNATURE

See Also:

Constant Field Values

INNER_SIGNATURE_PREFIX

public static final String INNER_SIGNATURE_PREFIX

See Also:

Constant Field Values

PRUNE_PROJECTION_INFO

protected static final String PRUNE_PROJECTION_INFO

See Also:

Constant Field Values

signature

protected String signature

Constructor Detail

HCatLoader

public HCatLoader()

Method Detail

getInputFormat

public InputFormat<?,?> getInputFormat()
                                throws IOException

Specified by:

getInputFormat in class LoadFunc

Throws:

IOException

relativeToAbsolutePath

public String relativeToAbsolutePath(String location,
                                     Path curDir)
                              throws IOException

Overrides:

relativeToAbsolutePath in class LoadFunc

Throws:

IOException

setLocation

public void setLocation(String location,
                        Job job)
                 throws IOException

Specified by:

setLocation in class LoadFunc

Throws:

IOException

getPartitionKeys

public String[] getPartitionKeys(String location,
                                 Job job)
                          throws IOException

Throws:

IOException

getSchema

public ResourceSchema getSchema(String location,
                                Job job)
                         throws IOException

Throws:

IOException

setPartitionFilter

public void setPartitionFilter(Expression partitionFilter)
                        throws IOException

Throws:

IOException

getStatistics

public ResourceStatistics getStatistics(String location,
                                        Job job)
                                 throws IOException

Get statistics about the data to be loaded. Only input data size is implemented at this time.

Specified by:

getStatistics in interface LoadMetadata

Throws:

IOException

getNext

public Tuple getNext()
              throws IOException

Specified by:

getNext in class LoadFunc

Throws:

IOException

prepareToRead

public void prepareToRead(RecordReader reader,
                          PigSplit arg1)
                   throws IOException

Specified by:

prepareToRead in class LoadFunc

Throws:

IOException

getFeatures

public List<LoadPushDown.OperatorSet> getFeatures()

Specified by:

getFeatures in interface LoadPushDown

pushProjection

public LoadPushDown.RequiredFieldResponse pushProjection(LoadPushDown.RequiredFieldList requiredFieldsInfo)
                                                  throws FrontendException

Specified by:

pushProjection in interface LoadPushDown

Throws:

FrontendException

setUDFContextSignature

public void setUDFContextSignature(String signature)

Overrides:

setUDFContextSignature in class LoadFunc

storeInUDFContext

protected void storeInUDFContext(String signature,
                                 String key,
                                 Object value)

getSizeInBytes

protected static long getSizeInBytes(InputJobInfo inputJobInfo)
                              throws IOException

A utility method to get the size of inputs. This is accomplished by summing the size of all input paths on supported FileSystems. Locations whose size cannot be determined are ignored. Note non-FileSystem and unpartitioned locations will not report their input size by default.

Throws:

IOException