前言:
目前咱们对“oracle过程exception”都比较关切,朋友们都想要剖析一些“oracle过程exception”的相关知识。那么小编在网摘上汇集了一些对于“oracle过程exception””的相关内容,希望大家能喜欢,朋友们一起来学习一下吧!大多数脚本报错会在bca日志表中result字段写入FAIL,note字段写入FAILED的内容,目前确认我们的脚本报错是在日志文件中抓取FAILED关键字,一旦出现该关键字,则认定过程执行失败,同时会把FAILED这一行的内容插入到bca日志表的NOTE字段中。
1 表不存在
FAILED: SemanticException [Error 10001]: Line 101:19 Table not found ODS_CDR_PSTN_D
报错分析:Table not found 很明显表ODS_CDR_PSTN_D 不存在
解决办法:
可以先去检查下对应ODS库下是否存在该表,若不存在,则迁移该表,若存在,那很可能就是过程脚本中未在表名前加上库名。
2 权限问题
FAILED: RuntimeException Cannot create staging directory hdfs://beh/user/hive/warehouse/ods.db/ods_cdr_zbi_com_website_d/month_part=201512/day_part=17/.hive-staging_hive_2015-12-25_15-58-34_081_5222500334215480980-1 : Permission denied: user=shdev, access=WRITE, inode="/user/hive/warehouse/ods.db/ods_cdr_zbi_com_website_d/month_part=201512":shadmin:supergroup:drwxr-xr-x
报错分析:显示对应hdfs的目录权限为shadmin用户所属,且权限为755,shdev用户的write操作被拒绝了
解决办法:连接shadmin用户,对shdev用户赋予该路径的acl权限
可以先查看该hdfs路径的权限:
hadoop fs –getfacl /user/hive/warehouse/ods.db/ods_cdr_zbi_com_website_d
再赋权:
hadoop fs –setfacl –m user:shdev:rwx /user/hive/warehouse/ods.db/ods_cdr_zbi_com_website_d
3 通用报错
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTase
类似于一句总结的话,基本大多数报错都会用该句收尾,所以这个报错信息不能帮忙解决问题,需要查看日志。
日志存放目录:/shbigdata/dwd/logs
dwd对应库名,除src外其它所有库下都有自己过程上一次执行的具体日志信息。
4 本地模式报错
Execution failed with exit status: 3
Obtaining error information
Task failed!
Task ID:
Stage-8
Logs:
/opt/beh/logs/hive/shleader-hive.log
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
ATTEMPT: Execute BackupTask: org.apache.hadoop.hive.ql.exec.mr.MapRedTask
报错分析:本地模式的内存无法满足当前任务的需求
解决办法:关闭本地模式
过程语句前加上:
set hive.exec.mode.local.auto=false;
set hive.auto.convert.join=false;
5 内存溢出报错
Container [pid=17441,containerID=container_1446899682569_0355_01_000019] is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 2.8 GB of 4.2 GB virtual memory used. Killing container.
报错分析:container执行需要的内存超出了物理内存限制
解决办法:给它更大的内存
过程语句前加上:
set mapreduce.map.memory.mb=4096;
set mapreduce.reduce.memory.mb=4096;
6 Group by 后面有两个相同字段报错
FAILED: ClassCastException org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
查看有sql语句中group by后面是否有2个同名的字段
7 NoViableAltException(-1@[])报错
NoViableAltException(-1@[])
at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1087)
at org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827)
at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495)
at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:396)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
FAILED: ParseException line 1:14 cannot recognize input near '<EOF>' '<EOF>' '<EOF>' in select clause
报错分析:NoViableAltException(-1@[])表示语法问题
检查对应sql段是否有语法错误,常见错误为中文注释引起或者注释中存在分号”;”
脚本中—符号可以将一行注释掉,但是如果这一行存在分号”;“,是无法把分号”;“注释掉的
8 kryo报错
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 590
报错分析:过程执行时启动的job数目过多导致
解决办法:
重新执行过程
9 表存储格式报错
报错信息:
Task with the most failures(4):
-----
Task ID:
task_1447035354510_316137_m_000000
URL:
;tipid=task_1447035354510_316137_m_000000
-----
Diagnostic Messages for this Task:
Error: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:266)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:213)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:333)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:719)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:252)
... 11 more
Caused by: java.io.IOException: hdfs://beh/user/hive/warehouse/ods.db/ods_usr_fx_kuayu_one_m/month_part=201512/ODS_USR_KUAYU_ONE_M.data not a RCFile and has magic of 201
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1473)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.<init>(RCFile.java:1363)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.<init>(RCFile.java:1343)
at org.apache.hadoop.hive.ql.io.RCFileRecordReader.<init>(RCFileRecordReader.java:100)
at org.apache.hadoop.hive.ql.io.RCFileInputFormat.getRecordReader(RCFileInputFormat.java:57)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:67)
... 16 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
报错分析:提示该表hdfs路径下的数据不是rcfile,根据标红报错可以看出ODS_USR_KUAYU_ONE_M.data是TEXTFILE文件
检查表ods_usr_fx_kuayu_one_m的存储格式,发现该表存储格式是RCFILE,联想到该表数据是从oracle同步至hive中,同步方法为hadoop fs –put 文件至hdfs目录的,而RCFILE的表是不能直接-put或者load数据的,所以就算文件放在指定hdfs目录,查询的时候依旧会报错
tu5
解决办法:若需要继续使用脚本同步数据,则需要将该表的存储格式修改为TEXTFILE
总结:
Hive迁移开发过程中会出现各种各样的错误,其中部分为Oracle支持但是Hive不支持导致的报错,多数为开发时编写不小心多敲或少敲造成的。大部分的错误日志中都可以分析清楚,所以一旦报错,需要先查看日志,定位问题,再解决问题。
标签: #oracle过程exception