org.apache.hadoop.hive.ql.optimizer
Class GenMapRedUtils

java.lang.Object
  org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils

public final class GenMapRedUtils
extends Object

General utility common functions for the Processor to convert operator into map-reduce tasks.

Method Summary

static MapredWork	getMapRedWork(ParseContext parseCtx) create a new plan and return.
static MapredWork	getMapRedWorkFromConf(HiveConf conf) create a new plan and return.
static void	initMapJoinPlan(Operator<? extends OperatorDesc> op, GenMRProcContext ctx, boolean readInputMapJoin, boolean readInputUnion, boolean setReducer, int pos)
static void	initMapJoinPlan(Operator<? extends OperatorDesc> op, GenMRProcContext opProcCtx, boolean readInputMapJoin, boolean readInputUnion, boolean setReducer, int pos, boolean createLocalPlan) Initialize the current plan by adding it to root tasks.
static void	initPlan(ReduceSinkOperator op, GenMRProcContext opProcCtx) Initialize the current plan by adding it to root tasks.
static void	initUnionPlan(GenMRProcContext opProcCtx, Task<? extends Serializable> currTask, boolean local)
static void	initUnionPlan(ReduceSinkOperator op, GenMRProcContext opProcCtx, Task<? extends Serializable> unionTask) Initialize the current union plan.
static void	joinPlan(Operator<? extends OperatorDesc> op, Task<? extends Serializable> oldTask, Task<? extends Serializable> task, GenMRProcContext opProcCtx, int pos, boolean split, boolean readMapJoinData, boolean readUnionData)
static void	joinPlan(Operator<? extends OperatorDesc> op, Task<? extends Serializable> oldTask, Task<? extends Serializable> task, GenMRProcContext opProcCtx, int pos, boolean split, boolean readMapJoinData, boolean readUnionData, boolean createLocalWork) Merge the current task with the task for the current reducer.
static void	joinUnionPlan(GenMRProcContext opProcCtx, Task<? extends Serializable> currentUnionTask, Task<? extends Serializable> existingTask, boolean local)
static void	mergeMapJoinUnion(UnionOperator union, GenMRProcContext ctx, int pos)
static Operator<? extends OperatorDesc>	putOpInsertMap(Operator<? extends OperatorDesc> op, RowResolver rr, ParseContext parseCtx) insert in the map for the operator to row resolver.
static void	setKeyAndValueDesc(MapredWork plan, Operator<? extends OperatorDesc> topOp) set key and value descriptor.
static void	setTaskPlan(String alias_id, Operator<? extends OperatorDesc> topOp, MapredWork plan, boolean local, GenMRProcContext opProcCtx) set the current task in the mapredWork.
static void	setTaskPlan(String alias_id, Operator<? extends OperatorDesc> topOp, MapredWork plan, boolean local, GenMRProcContext opProcCtx, PrunedPartitionList pList) set the current task in the mapredWork.
static void	setTaskPlan(String path, String alias, Operator<? extends OperatorDesc> topOp, MapredWork plan, boolean local, TableDesc tt_desc) set the current task in the mapredWork.
static void	splitPlan(ReduceSinkOperator op, GenMRProcContext opProcCtx) Split the current plan by creating a temporary destination.
static void	splitTasks(Operator<? extends OperatorDesc> op, Task<? extends Serializable> parentTask, Task<? extends Serializable> childTask, GenMRProcContext opProcCtx, boolean setReducer, boolean local, int posn)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

initPlan

public static void initPlan(ReduceSinkOperator op,
                            GenMRProcContext opProcCtx)
                     throws SemanticException

Initialize the current plan by adding it to root tasks.

Parameters:

op - the reduce sink operator encountered

opProcCtx - processing context

Throws:

SemanticException

initMapJoinPlan

public static void initMapJoinPlan(Operator<? extends OperatorDesc> op,
                                   GenMRProcContext ctx,
                                   boolean readInputMapJoin,
                                   boolean readInputUnion,
                                   boolean setReducer,
                                   int pos)
                            throws SemanticException

Throws:

SemanticException

initMapJoinPlan

public static void initMapJoinPlan(Operator<? extends OperatorDesc> op,
                                   GenMRProcContext opProcCtx,
                                   boolean readInputMapJoin,
                                   boolean readInputUnion,
                                   boolean setReducer,
                                   int pos,
                                   boolean createLocalPlan)
                            throws SemanticException

Initialize the current plan by adding it to root tasks.

Parameters:

op - the map join operator encountered

opProcCtx - processing context

pos - position of the parent

Throws:

SemanticException

initUnionPlan

public static void initUnionPlan(ReduceSinkOperator op,
                                 GenMRProcContext opProcCtx,
                                 Task<? extends Serializable> unionTask)
                          throws SemanticException

Initialize the current union plan.

Parameters:

op - the reduce sink operator encountered

opProcCtx - processing context

Throws:

SemanticException

initUnionPlan

public static void initUnionPlan(GenMRProcContext opProcCtx,
                                 Task<? extends Serializable> currTask,
                                 boolean local)
                          throws SemanticException

Throws:

SemanticException

joinUnionPlan

public static void joinUnionPlan(GenMRProcContext opProcCtx,
                                 Task<? extends Serializable> currentUnionTask,
                                 Task<? extends Serializable> existingTask,
                                 boolean local)
                          throws SemanticException

Throws:

SemanticException

joinPlan

public static void joinPlan(Operator<? extends OperatorDesc> op,
                            Task<? extends Serializable> oldTask,
                            Task<? extends Serializable> task,
                            GenMRProcContext opProcCtx,
                            int pos,
                            boolean split,
                            boolean readMapJoinData,
                            boolean readUnionData)
                     throws SemanticException

Throws:

SemanticException

joinPlan

public static void joinPlan(Operator<? extends OperatorDesc> op,
                            Task<? extends Serializable> oldTask,
                            Task<? extends Serializable> task,
                            GenMRProcContext opProcCtx,
                            int pos,
                            boolean split,
                            boolean readMapJoinData,
                            boolean readUnionData,
                            boolean createLocalWork)
                     throws SemanticException

Merge the current task with the task for the current reducer.

Parameters:

op - operator being processed

oldTask - the old task for the current reducer

task - the current task for the current reducer

opProcCtx - processing context

pos - position of the parent in the stack

Throws:

SemanticException

splitPlan

public static void splitPlan(ReduceSinkOperator op,
                             GenMRProcContext opProcCtx)
                      throws SemanticException

Split the current plan by creating a temporary destination.

Parameters:

op - the reduce sink operator encountered

opProcCtx - processing context

Throws:

SemanticException

setTaskPlan

public static void setTaskPlan(String alias_id,
                               Operator<? extends OperatorDesc> topOp,
                               MapredWork plan,
                               boolean local,
                               GenMRProcContext opProcCtx)
                        throws SemanticException

set the current task in the mapredWork.

Parameters:

alias_id - current alias

topOp - the top operator of the stack

plan - current plan

local - whether you need to add to map-reduce or local work

opProcCtx - processing context

Throws:

SemanticException

setTaskPlan

public static void setTaskPlan(String alias_id,
                               Operator<? extends OperatorDesc> topOp,
                               MapredWork plan,
                               boolean local,
                               GenMRProcContext opProcCtx,
                               PrunedPartitionList pList)
                        throws SemanticException

set the current task in the mapredWork.

Parameters:

alias_id - current alias

topOp - the top operator of the stack

plan - current plan

local - whether you need to add to map-reduce or local work

opProcCtx - processing context

pList - pruned partition list. If it is null it will be computed on-the-fly.

Throws:

SemanticException

setTaskPlan

public static void setTaskPlan(String path,
                               String alias,
                               Operator<? extends OperatorDesc> topOp,
                               MapredWork plan,
                               boolean local,
                               TableDesc tt_desc)
                        throws SemanticException

set the current task in the mapredWork.

Parameters:

alias - current alias

topOp - the top operator of the stack

plan - current plan

local - whether you need to add to map-reduce or local work

tt_desc - table descriptor

Throws:

SemanticException

setKeyAndValueDesc

public static void setKeyAndValueDesc(MapredWork plan,
                                      Operator<? extends OperatorDesc> topOp)

set key and value descriptor.

Parameters:

plan - current plan

topOp - current top operator in the path

getMapRedWork

public static MapredWork getMapRedWork(ParseContext parseCtx)

create a new plan and return.

Returns:

the new plan

getMapRedWorkFromConf

public static MapredWork getMapRedWorkFromConf(HiveConf conf)

create a new plan and return. The pan won't contain the name to split sample information in parse context.

Returns:

the new plan

putOpInsertMap

public static Operator<? extends OperatorDesc> putOpInsertMap(Operator<? extends OperatorDesc> op,
                                                              RowResolver rr,
                                                              ParseContext parseCtx)

insert in the map for the operator to row resolver.

Parameters:

op - operator created

rr - row resolver

parseCtx - parse context

splitTasks

public static void splitTasks(Operator<? extends OperatorDesc> op,
                              Task<? extends Serializable> parentTask,
                              Task<? extends Serializable> childTask,
                              GenMRProcContext opProcCtx,
                              boolean setReducer,
                              boolean local,
                              int posn)
                       throws SemanticException

Throws:

SemanticException

mergeMapJoinUnion

public static void mergeMapJoinUnion(UnionOperator union,
                                     GenMRProcContext ctx,
                                     int pos)
                              throws SemanticException

Throws:

SemanticException