org.apache.hadoop.hive.ql.exec
Class ExecDriver

java.lang.Object
  org.apache.hadoop.hive.ql.exec.Task<MapredWork>
      org.apache.hadoop.hive.ql.exec.ExecDriver

All Implemented Interfaces:

Serializable, HadoopJobExecHook, Node

Direct Known Subclasses:

MapRedTask

public class ExecDriver
extends Task<MapredWork>
implements Serializable, HadoopJobExecHook

ExecDriver.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Task

Task.FeedType

Field Summary

protected org.apache.hadoop.mapred.JobConf	job
protected HadoopJobExecHelper	jobExecHelper
protected static org.apache.commons.logging.Log	LOG
static MemoryMXBean	memoryMXBean

Fields inherited from class org.apache.hadoop.hive.ql.exec.Task

BACKUP_COMMON_JOIN, backupChildrenTasks, backupTask, childTasks, clonedConf, COMMON_JOIN, conf, console, CONVERTED_LOCAL_MAPJOIN, CONVERTED_MAPJOIN, db, driverContext, feedSubscribers, id, initialized, isdone, jobID, LOCAL_MAPJOIN, NO_TAG, parentTasks, queryPlan, queued, started, taskCounters, taskHandle, taskTag, work

Constructor Summary

ExecDriver()
Constructor when invoked from QL.

ExecDriver(MapredWork plan, org.apache.hadoop.mapred.JobConf job, boolean isSilent)
Constructor/Initialization for invocation as independent utility.

Method Summary

static void	addInputPaths(org.apache.hadoop.mapred.JobConf job, MapredWork work, String hiveScratchDir, Context ctx)
boolean	checkFatalErrors(org.apache.hadoop.mapred.Counters ctrs, StringBuilder errMsg) Fatal errors are those errors that cannot be recovered by retries.
protected void	createTmpDirs()
int	execute(DriverContext driverContext) Execute a query plan using Hadoop.
static String	generateCmdLine(HiveConf hconf, Context ctx) Given a Hive Configuration object - generate a command line fragment for passing such configuration information to ExecDriver.
String	getName() Gets the name of the node.
Collection<Operator<? extends OperatorDesc>>	getTopOperators()
StageType	getType() Should be overridden to return the type of the specific task among the types in StageType.
boolean	hasReduce()
void	initialize(HiveConf conf, QueryPlan queryPlan, DriverContext driverContext) Initialization when invoked from QL.
boolean	isMapRedTask()
protected void	localizeMRTmpFilesImpl(Context ctx) If this task uses any map-reduce intermediate data (either for reading or for writing), localize them (using the supplied Context).
void	logPlanProgress(SessionState ss)
static void	main(String[] args)
boolean	mapDone()
boolean	mapStarted()
boolean	reduceDone()
boolean	reduceStarted()
boolean	requireLock()
protected void	setInputAttributes(org.apache.hadoop.conf.Configuration conf) Set hive input format, and input format file if necessary.
void	updateCounters(org.apache.hadoop.mapred.Counters ctrs, org.apache.hadoop.mapred.RunningJob rj)

Methods inherited from class org.apache.hadoop.hive.ql.exec.Task

addDependentTask, cloneConf, done, executeTask, fetch, getAndInitBackupTask, getBackupChildrenTasks, getBackupTask, getChildren, getChildTasks, getCounters, getDependentTasks, getFeedSubscribers, getId, getInitialized, getJobID, getParentTasks, getQueryPlan, getQueued, getReducer, getResultSchema, getTaskTag, getWork, ifRetryCmdWhenFail, isLocalMode, isMapRedLocalTask, isRunnable, localizeMRTmpFiles, pushFeed, receiveFeed, removeDependentTask, removeFromChildrenTasks, setBackupChildrenTasks, setBackupTask, setChildTasks, setDone, setFeedSubscribers, setId, setInitialized, setLocalMode, setParentTasks, setQueryPlan, setQueued, setRetryCmdWhenFail, setStarted, setTaskTag, setWork, started, subscribeFeed

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

job

protected transient org.apache.hadoop.mapred.JobConf job

memoryMXBean

public static MemoryMXBean memoryMXBean

jobExecHelper

protected HadoopJobExecHelper jobExecHelper

LOG

protected static final transient org.apache.commons.logging.Log LOG

Constructor Detail

ExecDriver

public ExecDriver()

Constructor when invoked from QL.

ExecDriver

public ExecDriver(MapredWork plan,
                  org.apache.hadoop.mapred.JobConf job,
                  boolean isSilent)
           throws HiveException

Constructor/Initialization for invocation as independent utility.

Throws:

HiveException

Method Detail

requireLock

public boolean requireLock()

Overrides:

requireLock in class Task<MapredWork>

initialize

public void initialize(HiveConf conf,
                       QueryPlan queryPlan,
                       DriverContext driverContext)

Initialization when invoked from QL.

Overrides:

initialize in class Task<MapredWork>

checkFatalErrors

public boolean checkFatalErrors(org.apache.hadoop.mapred.Counters ctrs,
                                StringBuilder errMsg)

Fatal errors are those errors that cannot be recovered by retries. These are application dependent. Examples of fatal errors include: - the small table in the map-side joins is too large to be feasible to be handled by one mapper. The job should fail and the user should be warned to use regular joins rather than map-side joins. Fatal errors are indicated by counters that are set at execution time. If the counter is non-zero, a fatal error occurred. The value of the counter indicates the error type.

Specified by:

checkFatalErrors in interface HadoopJobExecHook

Returns:

true if fatal errors happened during job execution, false otherwise.

createTmpDirs

protected void createTmpDirs()
                      throws IOException

Throws:

IOException

execute

public int execute(DriverContext driverContext)

Execute a query plan using Hadoop.

Specified by:

execute in class Task<MapredWork>

Returns:

status of executing the task

setInputAttributes

protected void setInputAttributes(org.apache.hadoop.conf.Configuration conf)

Set hive input format, and input format file if necessary.

mapStarted

public boolean mapStarted()

reduceStarted

public boolean reduceStarted()

mapDone

public boolean mapDone()

reduceDone

public boolean reduceDone()

main

public static void main(String[] args)
                 throws IOException,
                        HiveException

Throws:

IOException

HiveException

generateCmdLine

public static String generateCmdLine(HiveConf hconf,
                                     Context ctx)
                              throws IOException

Given a Hive Configuration object - generate a command line fragment for passing such configuration information to ExecDriver.

Throws:

IOException

isMapRedTask

public boolean isMapRedTask()

Overrides:

isMapRedTask in class Task<MapredWork>

getTopOperators

public Collection<Operator<? extends OperatorDesc>> getTopOperators()

Overrides:

getTopOperators in class Task<MapredWork>

hasReduce

public boolean hasReduce()

Overrides:

hasReduce in class Task<MapredWork>

addInputPaths

public static void addInputPaths(org.apache.hadoop.mapred.JobConf job,
                                 MapredWork work,
                                 String hiveScratchDir,
                                 Context ctx)
                          throws Exception

Throws:

Exception

getType

public StageType getType()

Description copied from class: Task

Should be overridden to return the type of the specific task among the types in StageType.

Specified by:

getType in class Task<MapredWork>

Returns:

StageType.* or null if not overridden

getName

public String getName()

Description copied from interface: Node

Gets the name of the node. This is used in the rule dispatchers.

Specified by:

getName in interface Node

Returns:

String

localizeMRTmpFilesImpl

protected void localizeMRTmpFilesImpl(Context ctx)

Description copied from class: Task

If this task uses any map-reduce intermediate data (either for reading or for writing), localize them (using the supplied Context). Map-Reduce intermediate directories are allocated using Context.getMRTmpFileURI() and can be localized using localizeMRTmpFileURI(). This method is declared abstract to force any task code to explicitly deal with this aspect of execution.

Specified by:

localizeMRTmpFilesImpl in class Task<MapredWork>

Parameters:

ctx - context object with which to localize

updateCounters

public void updateCounters(org.apache.hadoop.mapred.Counters ctrs,
                           org.apache.hadoop.mapred.RunningJob rj)
                    throws IOException

Specified by:

updateCounters in interface HadoopJobExecHook

Throws:

IOException

logPlanProgress

public void logPlanProgress(SessionState ss)
                     throws IOException

Specified by:

logPlanProgress in interface HadoopJobExecHook

Throws:

IOException