CDH 5.8.2 Login Page error 500

You may get this error when there is any kind of interruption in cloudera server process. In my case, I get this every time I bring up my pc after either hibernate or sleep.

Just restart the cloudera server and wait for it come up and then try login again and you should be good:

[root@localhost conf]# service cloudera-scm-server restart

Restarting cloudera-scm-server (via systemctl):            [  OK  ]

[root@localhost conf]# 

HTTP ERROR 500

Problem accessing /cmf/login. Reason:

    Error creating bean with name 'newServiceHandlerRegistry' defined in class path resource [com/cloudera/server/cmf/config/components/BeanConfiguration.class]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public com.cloudera.cmf.service.ServiceHandlerRegistry com.cloudera.server.cmf.config.components.BeanConfiguration.newServiceHandlerRegistry()] threw exception; nested exception is java.lang.IllegalStateException: BeanFactory not initialized or already closed - call 'refresh' before accessing beans via the ApplicationContext

Caused by:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'newServiceHandlerRegistry' defined in class path resource [com/cloudera/server/cmf/config/components/BeanConfiguration.class]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public com.cloudera.cmf.service.ServiceHandlerRegistry com.cloudera.server.cmf.config.components.BeanConfiguration.newServiceHandlerRegistry()] threw exception; nested exception is java.lang.IllegalStateException: BeanFactory not initialized or already closed - call 'refresh' before accessing beans via the ApplicationContext
 at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:581)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:983)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:879)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
 at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:848)
 at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:790)
 at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:707)
 at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:795)
 at org.springframework.beans.factory.support.ConstructorResolver.resolvePreparedArguments(ConstructorResolver.java:765)
 at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:131)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowir

Exception in secureMain java.io.IOException: failed to stat a path component: '/var/run/hdfs-sockets'.

This is in CDH 5.8.2

Data Node not able to start due to:

FATAL

org.apache.hadoop.hdfs.server.datanode.DataNode

Exception in secureMain java.io.IOException: failed to stat a path component: '/var/run/hdfs-sockets'. error code 2 (No such file or directory)

Creating a folder 'hdfs-sockets' under /var/run and changing owner:group to cloudera-scm solves the problem but this folder gets removed after the reboot.

How to disable "information" messages in pyspark (python spark) console?

You may notice bunch of messages poping up, as showed below, on the console when you initiate spark console using pyspark.
How do you disable these messages and only show the errors?

[santhosh@localhost Downloads]$ pyspark
Python 2.7.5 (default, Sep 14 2016, 08:35:31)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
16/10/06 11:25:20 INFO spark.SparkContext: Running Spark version 1.6.0
16/10/06 11:25:21 INFO spark.SecurityManager: Changing view acls to: santhosh
16/10/06 11:25:21 INFO spark.SecurityManager: Changing modify acls to: santhosh

.

.

.

16/10/06 11:25:30 INFO storage.BlockManagerMaster: Registered BlockManager
16/10/06 11:25:30 INFO scheduler.EventLoggingListener: Logging events to hdfs://localhost:8020/user/spark/applicationHistory/application_1475694807177_0010
16/10/06 11:25:31 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/

Using Python version 2.7.5 (default, Sep 14 2016 08:35:31)
SparkContext available as sc, HiveContext available as sqlContext.
>>>

Solution:

Edit the file /etc/spark/conf/log4j.properties

and change the value for logger from INFO to ERROR

root.logger=INFO,console --> root.logger=ERROR,console