Common Errors Encountered While Working With Hadoop


Failed to start the NameNode : 

Installing Hadoop is relative simple, but there may have some problems, after setting up the environment, we need to specify the Java installation path by setting the parameter JAVA_HOME in $Hadoop_path/conf/hadoop-env.sh, otherwise, the HDFS will get error when it try to boot up.

No TaskTracker available:

The NameNode runs normally and also exist some DataNode, but no TaskTracker exist, that's because if HDFS is not ready, the JobTracker will no accept any TaskTracker to join, HDFS will be in safe mode when it start, it will not allow any modifications to the file system, if the number of DataNode is not enough, it will report error and state in safe mode. It may return “The ratio of reported blocks 0.9982 has not reached the threshold 0.9990. Safe mode will be turned off automatically." Turn on enough DataNode to reach the threshold will solve this problem, if some of DataNode were crashed and the available ratio is not enough, the only choice is to re-format the whole HDFS. So, less than 3 DataNodes is dangerous.

MapReduce job stops at 0 Map input and 0 reduce input: 

In the RecordReader, the "next" function should return true although there has no next value, we should use a local variable as a flag to check if any other value pair exists.

Heap Space Error: 

Memory on TaskTracker is not enough, in default, the heap size of JVM is 128MB, so, we could set it in client programs: conf.set("mapred.child.java.opts","-Xmx1024m"); Lack of memory may cause a lots of problems, such as the task may always failed and the whole job failed, or too much content swapping, the performance will extremely low.


Comments

Post a Comment

Popular posts from this blog

Study of Differenet Network Types and Different Types of Network Cables and Practically Implement the Cross-Wired cable using Clamping Tool.

Image
EXPERIMENT-1.a Aim: Study of following Network Types. • PAN • LAN • MAN • WAN Apparatus (Software): No software or hardware needed. Procedure: Following should be studied to understand this practical. Types of Networks There are several different types of computer networks. Computer networks can be characterized by their size as well as their purpose. The size of a network can be expressed by the geographic area they occupy and the number of computers that are part of the network. Networks can cover anything from a handful of devices within a single room to millions of devices spread across the entire globe. Some of the different networks based on size are: Personal area network, or PAN Local area network, or LAN Metropolitan area network, or MAN Wide area network, or WAN In terms of purpose, many networks can be considered general purpose, which means they are used for everything from sending files to a prin...
Read more

Error Detection and Correction Techniques

Image
Aim: a. Write a program for Error Detection usinng CRC. b. Write a program for Error C orrection using Hamming Code. Apparatus (Software): Eclipse/ Netbeans Procedure: Following should be studied to understand this practical Introduction Environmental interference and physical defects in the communication medium can cause random bit errors during data transmission. Error coding is a method of detecting and correcting these errors to ensure information is transferred intact from its source to its destination. Types of errors These interferences can change the timing and shape of the signal. If the signal is carrying binary encoded data, such changes can alter the meaning of the data. These errors can be divided into two types: Single-bit error and Burst error. Single-bit Error The term single-bit error means that only one bit of given data unit (such as a byte, character, or data unit) is changed from 1 to 0 or from 0 to 1 as ...
Read more

Challenges in e-governance

Over the years, many e-governance initiatives have been undertaken by various State Governments and Central Ministries to make all government services accessible to the common man in his locality, through common service delivery outlets, and ensure efficiency, transparency, and reliability of such services for a common man.  Although there are many improvements and innovations made in e-governance services but yet in today’s global context, e-governance services must respond proactively and it should be more agile.  There are large numbers of barriers that hinder the potential working of e-Governance. It could be different languages, IT illiteracy, population, lack of integrated services, lack of confidence, etc and it has often complained that the government schemes remain confined to papers and even if they are implemented there are no proper sources that can guarantee the proper working on the ground. Many e-government websites and applications particularly those ...
Read more