Not able to access default database in pyspark

hive pyspark

1147 观看

1回复

27 作者的声誉

I am getting below exception when I try to execute below code,

from pyspark.sql import HiveContext
sqlContext = HiveContext(sc)
depts = sqlContext.sql("select * from departments")

17/09/13 03:37:12 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.1.0
17/09/13 03:37:12 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
17/09/13 03:37:14 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
Traceback (most recent call last):
File "", line 1, in 
File "/usr/lib/spark/python/pyspark/sql/context.py", line 580, in sql
return DataFrame(self.ssqlctx.sql(sqlQuery), self)
File "/usr/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in call
File "/usr/lib/spark/python/pyspark/sql/utils.py", line 51, in deco
raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: u'Table not found: departments; line 1 pos 14'

I am using Cloudera VM Version 5.10, Spark Version 1.6.0.

作者: Jinsy 的来源 发布者: 2017 年 9 月 15 日

回应 1


0

27 作者的声誉

决定

Solution for this exception

  1. Removed the linked file using sudo rm -R /etc/spar/conf/hive.xml
  2. Again linked the file using sudo ln -s /etc/hive/conf/hive-site.xml /etc/spark/conf/hive-site.xml
作者: Jinsy 发布者: 2017 年 9 月 19 日
32x32