Not able to access default database in pyspark

hive pyspark

1147 观看


27 作者的声誉

I am getting below exception when I try to execute below code,

from pyspark.sql import HiveContext
sqlContext = HiveContext(sc)
depts = sqlContext.sql("select * from departments")

17/09/13 03:37:12 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.1.0
17/09/13 03:37:12 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
17/09/13 03:37:14 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
Traceback (most recent call last):
File "", line 1, in 
File "/usr/lib/spark/python/pyspark/sql/", line 580, in sql
return DataFrame(self.ssqlctx.sql(sqlQuery), self)
File "/usr/lib/spark/python/lib/", line 813, in call
File "/usr/lib/spark/python/pyspark/sql/", line 51, in deco
raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: u'Table not found: departments; line 1 pos 14'

I am using Cloudera VM Version 5.10, Spark Version 1.6.0.

作者: Jinsy 的来源 发布者: 2017 年 9 月 15 日

回应 1


27 作者的声誉


Solution for this exception

  1. Removed the linked file using sudo rm -R /etc/spar/conf/hive.xml
  2. Again linked the file using sudo ln -s /etc/hive/conf/hive-site.xml /etc/spark/conf/hive-site.xml
作者: Jinsy 发布者: 2017 年 9 月 19 日