Monday, February 6, 2012

Vim as a Universal IDE {Part 2 (Ruby/Rails )}

when a Newbie starts coding with ruby , it seems hard to switch between files
and running scripts .It seem a lot of time consuming with coding
ruby . Many people ask on irc and groups daily which is best IDE for Ruby.Or
some those used Textmate,Eclispe ,etc.. are also facing time consumption
while write some code! SO ! SO! SO! Now Solution is VIM!

I already Explained basic Installation and working on vim in my previous post
those who are new to vim can visit Part 1 for basic editing and installation
,configuration of vim for better use

I explain here some useful plug-in and working,configuration of those .This
post help ! you to shape code perfect with vim and ruby.
Comments Welcome !!

1) For Enhanced Syntax Highlighting & Navigation(rails.vim)
 automatically detects buffers ,it effect only rails script not ruby scripts
.Easy navigation and interface to script ,integration with other plugins.such
NERDtree.vim with :Rtree

install details
extract the zip to ~/.vim

Download page is here

2)Organization of project like (IDE/Buffer)
this help you launch a .*.jpg files in a viewer. I have also set
up viewers for PDF (acroread) and HTML files (mozilla) for my own use.

install details
untar the file in ~/.vim
for untar use this command :- tar -xzvf filename
download page link is here

3)Ruby filetype and indentation

a)for Ruby indentation :download indentation script and extract(gunzip
filename) gzip  in ~/.vim/indent
download link is here
along with this you need install a plugin Indentanything

install details for that is here    
cd ~/.vim
tar zxvf IndentAnything.tar.gz
download page link is here

: extended % matching for HTML, LaTeX, and many other
languages The matchit.vim script allows you to configure % to match more    than just single characters.  You can match words and even regular expressions.
Also, matching treats strings and comments (as recognized by the
syntax highlighting mechanism) intelligently.
The default ftplugins include settings for several languages:
Ada, ASP with VBS, Csh, DTD, Essbase, Fortran, HTML, JSP
(same as HTML), LaTeX, Lua, Pascal, SGML, Shell, Tcsh, Vim, XML.
(I no longer keep track, so there may be others.)

install details
extract zip in ~/.vim

download page link is here
your .vimrc config must have below one
filetype plugin on

c)ruby-matchit:- Map '%' to jump from one keyword to its corresponding
'end' in Ruby files. Inspired by 'matchit.vim' that comes with Vim

install details

Put the file in ~/.vim/ftplugin/ruby/ directory
download page

4)Database excess ( dbext.vim) : Provides database access to many dbms such as: Mysql, PostgreSQL, Ingres, Oracle, Oracle Rdb (VMS), Sybase Adaptive Server Anywhere, Sybase Adaptive Server Enterprise,iMicrosoft SQL Server, DB2, Interbase and SQLite and ODBC are supported. 
 install deatils
 extract zip in ~/.vim
 download page is here

 5) Haml and Sass syntax, indenting, and ftplugin :-

 Runtime files for Haml and Sass.
 install details
 Extract in ~/.vim or ~\vimfiles.
 download page is here

6) Ruby-test
:- Rubytest.vim will help you to run ruby test (including        vanillatest, rspec, shoulda etc.) in vim.

Unzip downloaded file and copy rubytest.vim to your ~/.vim/plugin directory.
download page for the this

Your .vimrc config you need to do is below
let g:rubytest_cmd_test = "ruby %p"
let g:rubytest_cmd_testcase = "ruby %p -n '/%c/'"
let g:rubytest_cmd_spec = "spec -f specdoc %p"
let g:rubytest_cmd_example = "spec -f specdoc %p -e '%c'"            
let g:rubytest_cmd_feature = "cucumber %p"
let g:rubytest_cmd_story = "cucumber %p -n '%c'"

Default keybind to change

map <Leader>\ <Plug>RubyTestRun " change from <Leader>t to <Leader>\
map <Leader>] <Plug>RubyFileRun " change from <Leader>T to <Leader>]
map <Leader>/ <Plug>RubyTestRunLast " change from <Leader>l to <Leader>/

7) cucumber runtime :- description
This provides syntax highlighting, indenting, and a filetype plugin for
Cucumber, the acceptance testing framework

install details
Extract in ~/.vim

download page

8) Endwise.vim
: Wisely add :-
  This is a simple plugin that helps to end certain structures automatically.In Ruby, this means adding "end" after "if", "do", "def" and several other
  keywords. In Vim Script, this amounts to appropriately adding "endfunction","endif", etc

  install deatils
  install in ~/.vim/plugin
  download page is here

  9) Genutils : General utility functions :-

  This script provides functions that are mostly useful to script developers,but some of the functions can be easily converted to good utilities.
  install details
  extarct in ~/.vim 
  download page

 now config your .vimrc
command! -nargs=0 -range=% SortByLength <line1>,<line2>call QSort(
        \ 'CmpByLineLengthNname', 1)
command! -nargs=0 -range=% RSortByLength <line1>,<line2>call QSort(
        \ 'CmpByLineLengthNname', -1)
command! -nargs=0 -range=% SortJavaImports <line1>,<line2>callQSort(
        \ 'CmpJavaImports', 1)

u might like the following mappings to adjust spacing:
nnoremap <silent> <C-Space> :call ShiftWordInSpace(1)<CR>
nnoremap <silent> <C-BS> :call ShiftWordInSpace(-1)<CR>
nnoremap <silent> \cw :call CenterWordInSpace()<CR>
nnoremap <silent> \va :call

The :find command is very useful to search for a file in path, but it
doesn't support file completion. Add the following command in your vimrc
to add this functionality
nd! -nargs=1 -bang -complete=customlist,genutils#UserFileComplete2
            \ FindInPath :find<bang> <args>

If you are running commands that generate multiple pages of output, you
might find it useful to redirect the output to a new buffer. Put the
following command in your vimrc:
nd! -nargs=* -complete=command Redir
          \ :new | put! =GetVimCmdOutput('<args>') | setl bufhidden=wipe |
          \ setl nomodified

10) multvals.vim
: Array library that uses patterns as separators An array is
nothing but a string of multiple values separated by a
pattern.  The simplest example being Vim's multi-value variables such as
tags. You can use the MvAddElement() function to create an array.
However, there is nothing special about this function, you can as well
make up the string by simply concatenating elements with the chosen
pattern as a separator.

install in ~/.vim/plugin
download page is here

Some other plugin such as Nerd tree or git integration ,supertab ,code navigation are  used in previous post for python. Also textmate like completion and indentation also discuss in previous part .

Note:- Stay Tuned more parts of this Post coming soon for Ruby,C++,C,Closure,Prolog ,Haskell,Java,PHP,Perl,Shell, JavaScript ,HTML,CSS Coming soon (within few Days) .
Update of this post will done soon for Usage of plugin and more config Details .Also preparing image graphics for all post soon update all with how use effectively all these  plug-ins.

                                                                     THANKS FOR VISITING



Saturday, February 4, 2012

VIM as an Universal IDE{Part 1 (Python/Django)}

Vim is My First and Last editor ,IDE, or you can say most of time i spend on it  while work on system .

I write this blog for newbies , those who new to both Linux and Programming.Also for professionals and passionate guys.Also give a look if you are using  different editor then Vim for any Programming language .I write everything that a newbie need to know from installation to configuration , from novice to advance
Any suggestions ! Comments Welcome !here

More about Vim 
Vim is worth millions and costs nothing. It's the best editor the world has
ever seen. Using the trio Vim/Perl/LaTeX I can dispose of most other software.

More To do :

1. Get VIM for Dos
2. Get VIM for Linux
3. Get Vim for Freebsd
4. Get Vim for Dec-Unix
5. Get VIM for Irix
6. Get VIM for Window
7. Get VIM for Mac
8. Laugh at non-VIM users

Basic Introduction :- I Know for Beginner's ,Newbie for Linux ,Vim Need to get in with this From a basic . Following Question can asked by a newbie to Vim
Note:- Please experts those know vim well or For geeks those know it well go below some (stuff is there ) 
   1) May you don't know install Vim on Your's OS(Linux,Mac,Unix etc)
   2) May be you don't know how to open Vim on your OS
   3) May Be You Don't know how install plugin for it
   4) May Be You Don't know how to close it

 Okay before answering you all guys , It work for a large range of Operating System's such as Linux,Mac,Window ,Unix , FreeBSD ..much more.It provide both Command line and GUI interface .

1)Installing Vim
     A)For Linux User's :-                

a) For Ubuntu/Debian user's :- sudo apt-get install vim            
b) For Fedora user's :-  yum install vim-X11 vim-common vim-enhanced vim-minimal
c) For Red-hat Linux user's :- up2date -i vim-X11 vim-common vim-enhanced vim-minimal
d) For Arch Linux user's:- pacman -S vim              
e) For Fedora user's:- emerge vim

B)For  Mac User's :- MacVim is the text editor for Mac OS X , download it form here

C)For  Window User's :-Go to  Vim Download page and click on “PC: MS-DOS and MS-Windows”. Click on the ‘gvim72.exe, which is a Self-installing executable.
(or) Download gvim72.exe directly.          
For All Others Visit download page on

2) Opening or Simple use of Vim
   You  can open a using command line :- use vi/vim  in start of any file name or any file you want open like below        
   $ vim hack.txt                  
   above use of vim with hack.txt file to open

Vim has two mode visual mode (GUI mode) and insert mode(command line mode).     To Switch in Between them press Esc on your keyboard and then Press i for Insert mode and v for visual mode on your keyboard .Esc play a important role for switch you any command . Or you can use Crtl+c ,that is an alternative of Esc .

Note : If not get now also ,then Visit you sure got all open close and simple to advance use link below

3)  Vim have need to have two special area where all vim operate or contains  .Let me Explain
          1) create a ~/.vim directory
          $ mkdir ~/.vim            
          $ cd ~/.vim          
Note:- this Contain all your plugin installation.
          2) Now create file .vimrc under your bash or command terminal                  
           $vim .vimrc                  
Above will create and open .vimrc file in your bash and command terminal            

Note:- This contain information about your  configuration of vim and it's plugins

4)  Close it with press ESc on your Keyboard and then shift + : , then enter wq , wq! , q!            
 these all can use close the vim

Now Vim from Scratch to Novice (You can  say from level 1 to level 2 ):

Just visit and learn there the best online Tutorial

Now move on Real topic Vim as Universal IDE(Integrated Development Environment ):-

 1) Vim as Python/Django IDE :-

 i) Now you need to use ~/.vim and .vimrc i as tell you earlier  Now first step set a better color-scheme wombat,download wombat.vim

Now under after downloading under ~/.vim/colors    
$ mkdir -p ~/.vim/colors          
$ cd ~/.vim/colors          
$ wget -O wombat.vim
$ vim
Now in vim press shift + : ,then enter color wombat like below          
$:color wombat
Now after perfect looking move on !!

ii) For Syntax Error and Formatting (Indent):-
This can used for indentation or formatting every language need formatting and indentation  of that language .This indentation script for python tries to match more closely what is suggested in PEP 8 (  In particular, it handles continuation lines implied by open (parentheses), [brackets] and {braces} correctly and it indents multiline if/for/while statements differently.  Comments are welcome!
install details

Drop the script in your ~/.vim/indent directory.
$mkdir ~/.vim/indent $ cd ~/.vim/indent  
$ wget -O python.vim

download page is here

Now config this in .vimrc    
$ vim .vimrc    
add below  to .vimrc
filetype plugin indent on

iii) Special highlighting for String  and String formatting (Python.vim) :-        

Enhanced version of the original (from vim6.1) python.vim for Python programming language.
The changes since the original python.vim are:
- changed strings highlighting;
- enhanced special symbols highlighting inside strings;
- enhanced numbers highlighting;
- added optional highlighting for %-formatting inside strings;
- added highlighting for some error conditions (wrong symbols in source file,  
mixing spaces and tabs, wrong number values,  
wrong %-formatting inside strings);
- added highlighting for magic comments: source code encoding
 and #! (executable) strings;  - added highlighting
for new exceptions and builtins introduced in python 2.3, 2.4 and 2.5;
- added highlighting for doctests;
- added highlighting for new @decorator syntax introduced in Python 2.4a2;
- added highlighting for trailing-space errors (triggered by new  option: python_highlight_space_errors);
- added highlighting for variable name errors;
- added highlighting for hex number errors;

install details
Place python.vim file in ~/.vim/syntax/ folder.

$mkdir ~/.vim/syntax
$cd ~/.vim/syntax
$wget -O python.vim or download page

download page is here          

Also put config in .vimrc is below
autocmd FileType python set complete+=k~/.vim/syntax/python.vim isk+=.,

( iv) Highlighting Syntax and validation  :-
a) PyFlakes
IMPORTANT: Your vim must have Python 2.5, at least. Enter ":python import sys; print sys.version" to see which version you have.
 The newest version of Vim, 7.3, comes bundled with Python 2.7 support.  pyflakes-vim highlights common Python errors like
misspelling a variable name on the fly. It also warns about unused imports, redefined functions, etc.

install details
1. Make sure your vimrc has "filetype plugin indent on"
so that pyflake-vim's ftplugin files are loaded automatically when you open a Python buffer.
2. Drop extracted files in ~/.vim/ftplugin/python.
3. That's it!

Download page link here
Direct unzip this pyflakes into ~/.vim
Config setting of .vimrc is below
let g:pyflakes_use_quickfix = 0

 b) Pep8        
install inside ~/.vim/ftplugin/python
Download page
now switch to ~/.vim/ftlugin/python          
$cd ~/.vim/ftplugin/python          
$ wget -O pep8.vim

Configuration for .vimrc  put below in your config file .vimrc

let g:pep8_map='<leader>8' set expandtab
set textwidth=79
set tabstop=8
set softtabstop=4
set shiftwidth=4
set autoindent    

v)Code Navigation :-
Catgs  this my best plugin for vim it can use for all programming language's and with this taglist is best for source code navigation .
Install a direct package only for Ubuntu else do manually is below
# apt-get install exuberant-ctags

first download ctags from here .Ctags help you in while browsing a source code file:

Navigating to the function definition by specifying the function name.
Navigating to the function definition from ‘function call’.
Returning back again to function call from the definition.
Viewing the prototype/signature of functions or variables.
Viewing the number of functions in a file, etc., now extract ctags.tar.gz

then do this for configure and install ./configure && sudo make install

after that Config you .vimrc as below
let Tlist_Ctags_Cmd='/usr/local/bin/ctags'
set tags=./tags;/
map <C-\> :tab split<CR>:exec("tag ".expand("<cword>"))<CR>
map <A-]> :vsp <CR>:exec("tag ".expand("<cword>"))<CR>

Best use of this plugin with following command is below :- C-] - go to definition
C-T - Jump back from the definition.
C-W C-] - Open the definition in a horizontal split
C-\ - Open the definition in a new tab A-] - Open the definition in a vertical split

After the tags are generated. You can use the following keys to tag into and tag out of functions:

Ctrl-Left_MouseClick - Go to definition Ctrl-Right_MouseClick - Jump back from definition

Taglist :- with ctag for browsing source code better ,switch to ~/.vim/plugin
$ mkdir ~/.vim/plugin
$cd ~/.vim/plugin
$ wget -O

direct download link is here

Config for .vimrc like below  " Taglist variables

" Display function name in status bar:
let g:ctags_statusline=1
" Automatically start script
let generate_tags=1
" Displays taglist results in a vertical window:
let Tlist_Use_Horiz_Window=0
" Shorter commands to toggle Taglist display
nnoremap TT :TlistToggle<CR>
map <F4> :TlistToggle<CR>
" Various Taglist diplay config:
let Tlist_Use_Right_Window = 1
let Tlist_Compact_Format = 1
let Tlist_Exit_OnlyWindow = 1
let Tlist_GainFocus_On_ToggleOpen = 1
let Tlist_File_Fold_Auto_Close = 1

Tasklist with Ctags for better manipulation

switch ~/.vim/plugin
download there this direct
$ cd ~/.vim/plugin
$wget -O tasklist.vim

Config .vimrc
map T :TaskList<CR>
map P :TlistToggle<CR>
use tasklist with T and P to use this

vi) Code Completion :-    
A) Python Omni  completion  place this in ~/.vim/autoload download direct link here
$ cd ~/.vim/autolaod
$ wget -O pythoncomplete.vim  

add your config to your .vimrc is below
filetype plugin on
set ofu=syntaxcomplete#Complete

B) Pydict for python code completion
download zip direct to ~/.vim/ftplugin  and extract  download page link here

config your .vimrc with below stuff
let g:pydiction_location = 'C:/vim/vimfiles/ftplugin/pydiction/complete-dict'

C) Supertab for pydict easy completion :-
create a ~/.vim/after/ftplugin
download to it direct from this page

Install this now using :- Open the file in vim ($ vim supertab.vba).

 Source the file (:so %)
use to press tab to complete the code

add in .vimrc
let g:SuperTabDefaultCompletionType = "context"

vii) Basic editing and debugging(pep8 ,pyflake is used above) ,code snippets:- 

A) Code snippets(snippetEMU) :-
The same thing works for “for” loops, classes and many other constructs. The bundles .vba provides access to such constructs across a wide range of languages, e.g. Python, Ruby, HTML, PHP, even Djano-specific constructs.

create a ~/.vim/after/ftplugin
download page link is here

now install it above  open with vim snippy_plugin.vba then source the file (:so%)

B) snipMate:-it aims to be an unobtrusive, concise vim script that implements some of TextMate's snippets features in Vim. A snippet is a piece of often-typed text that you can insert into your document using a trigger word followed by a <tab>.   For instance, in a C file using the default installation of snipMate.vim, if you type "for<tab>" in insert mode, it will expand a typical for loop in C:  for (i = 0; i < count; i++) {  }

just download zip and extract in ~/.vim
Download page is here

C) Code folding
put this in ~/.vim/ftplugin
download from this page

D)Block navigation :-This script can be useful when editing Python scripts.
It provides the following menus:
- Select a block of lines with the same indentation
- Select a function
- Select a class
- Go to previous/next class/function
- Go to the beginning/end of a block
- Comment/uncomment the selection
- Jump to the last/next line with the same indent

install details
Copy to the $VIMFILES/ftplugin directory download page is here

E)Minibuffexpl.vim:-  download  here
paste in ~/.vim/plugin

config for .vimrc is below
let g:miniBufExplMapWindowNavVim = 1
let g:miniBufExplMapWindowNavArrows = 1
let g:miniBufExplMapCTabSwitchBufs = 1
let g:miniBufExplModSelTarget = 1

viii)Directory   Navigation:-Nerdtree download page

      install details
      Unzip the archive into your ~/.vim directory.
     That should put NERD_tree.vim in ~/.vim/plugin and NERD_tree.txt in ~/.vim/doc.

ix)Git integration with vim :- download page is here     script_id=2975
    just extract zip in ~/.vim    

x) For automatic closing of quotes, parenthesis, brackets:-   delimitMate
    Download page is here
   download this in ~/.vim/after/ftplugin , Now install using below

   Open the file with Vim and run:

   or use this command to install it in your bundle dir if you use pathogen:
   :UseVimball ~/.vim/bundle/delimitMate /              

Note:- Stay Tuned more parts of this Post coming soon for Ruby,C++,C,Closure,Prolog ,Haskell,Java,PHP,Perl,Shell, JavaScript ,HTML,CSS Coming soon (within few Days) .
Update of this post will done soon for Usage of plugin and more config Details .



Monday, January 2, 2012

Large Scale & Big Data analysis with Hadoop cluster ( Hadoop cluster step on Ubuntu 11.10 server )

Welcome back guys, i promise a setup of Hadoop cluster on Ubuntu Server(64 bit)
last time I post about Openstack and Devstack. Finally again from scratch a install and running tutorial is their

While  Everybody Celebrating New Year ,then I am setting up  my Hadoop cluster  and Processing unstructured data with Hadoop .

Welcome Administrators , Developer , Students, specially Hobbyist and passionates guys interesting in learning cloud .


The purpose of this document is to help you get a single-node Hadoop installation up and running very quickly so that you can get a flavour of the Hadoop Distributed File System (see HDFS Architecture) and the Map/Reduce framework; that is, perform simple operations on HDFS and run example jobs.

Our Pre-requisites for setup :-

1) We Must need a Ubuntu Server (64 bit) or Debian (64bit ) {I am using same this tutorial}
2)  You must have there installed (in main root)
     $ sudo apt-get install ssh (openssh )     
     $ sudo apt-get install rsync  
3) JavaTM 1.6.x, preferably from Sun, must be installed.

    Installing Java
    $ sudo add-apt-repository "deb lucid partner"
    $ sudo apt-get update 
    $ sudo apt-get install sun-java6-jdk sun-java6-plugin  ( while after Downloading  during installation plz press tab to accept JAVA terms and condition while a tab for conditions open ) 
Check is Sun java is there :)

root@ruhil:~# sudo apt-get install sun-java6-jdk sun-java6-plugin 
Reading package lists... Done
Building dependency tree      
Reading state information... Done
sun-java6-plugin is already the newest version.
sun-java6-jdk is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.
root@ruhil:~# java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

Add a Dedicated User 

$ sudo addgroup hadoop
$ sudo adduser --ingroup hadoop hduser

Configure and check is ssh working for local-host

(Please press the enter , you need not specify the name  for File for  and Public key )
root@ruhil:~# su - hduser
hduser@ruhil:~$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hduser/.ssh/id_rsa):
Created directory '/home/hduser/.ssh'.
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/
The key fingerprint is:
9b:82:ea:58:b4:e0:35:d7:ff:19:66:a6:ef:ae:0e:d2 hduser@ruhil
The key's randomart image is:

After do this step carefully

hduser@ruhil:~$ cat $HOME/.ssh/ >> $HOME/.ssh/authorized_key

hduse@ruhil:~$ ssh localhost
The authenticity of host 'localhost (' can't be established.
ECDSA key fingerprint is b8:be:26:41:44:7d:9b:82:02:fd:13:61:3c:ac:d4:0a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Linux ruhil 3.0.0-14-server #23-Ubuntu SMP Mon Nov 21 20:49:05 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

IN Ubuntu 11.10
open /etc/sysctl.conf in the editor of your choice and add the following lines to the end of the file:

#disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

You have to reboot your machine in order to make the changes take effect.

You can check whether IPv6 is enabled on your machine with the following command:

$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6

A return value of 0 means IPv6 is enabled, a value of 1 means disabled (that’s what we want).

Installation of Hadoop (perform this action in your main root like root@ruhil)
{Note Downloading 0.20 version is stable other not stable mainly 0.23,Would like go with 0.20  :)}
root@ruhil:~# mkdir -p /usr/local
root@ruhil:~# cd /usr/local
root@ruhil:~# wget -O hadoop-0.20.2.tar.gz
root@ruhil:~# sudo tar xzf hadoop-0.20.2.tar.gz
root@ruhil:~# mv hadoop-0.20.2 hadoop
root@ruhil:~$ sudo chown -R hduser:hadoop hadoop

Create .Bashrc or If have already paste  below for Hadoop(Note :-you need paste in root and hduser ,if you like you can paste for all,My Paste of .bashrc is :-

# Set Hadoop-related environment variables
export HADOOP_HOME=/usr/local/hadoop

# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)
export JAVA_HOME=/usr/lib/jvm/java-6-sun

# Some convenient aliases and functions for running Hadoop-related commands
unalias fs &> /dev/null
alias fs="hadoop fs"
unalias hls &> /dev/null
alias hls="fs -ls"

# If you have LZO compression enabled in your Hadoop cluster and
# compress job outputs with LZOP (not covered in this tutorial):
# Conveniently inspect an LZOP compressed file from the command
# line; run via:
# $ lzohead /hdfs/path/to/lzop/compressed/file.lzo
# Requires installed 'lzop' command.
lzohead () {
    hadoop fs -cat $1 | lzop -dc | head -1000 | less

# Add Hadoop bin/ directory to PATH

Configuration(Note all the configuration setting you need made in hduser ) :-

The following picture gives an overview of the most important HDFS components.

HDFS Architecture (source:

Our goal in this tutorial is a single-node setup of Hadoop. More information of what we do in this section is available on the Hadoop Wiki.

The only required environment variable we have to configure for Hadoop in this tutorial is JAVA_HOME. Open /conf/ in the editor of your choice (if you used the installation path in this tutorial, the full path is /usr/local/hadoop/conf/ and set the JAVA_HOME environment variable to the Sun JDK/JRE 6 directory.


# The java implementation to use.  Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun


# The java implementation to use.  Required.
export JAVA_HOME=/usr/lib/jvm/java-6-sun

Now we create the directory and set the required ownerships and permissions:

$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown hduser:hadoop /app/hadoop/tmp
$ sudo chmod 755 /app/hadoop/tmp
{Set Your chmod according to your settings  }

Add the following snippets between the <configuration> ... </configuration> tags in the respective configuration XML file.
Note for all given below we need perform all this  below config files

In file conf/core-site.xml:(cd /usr/local/hadoop there all this config )

<!-- In: conf/core-site.xml -->
  <description>A base for other temporary directories.</description>

  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>

In file conf/mapred-site.xml:

<!-- In: conf/mapred-site.xml -->
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.

In file conf/hdfs-site.xml:

<!-- In: conf/hdfs-site.xml -->
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.

hduser@ruhil:~$ /usr/local/hadoop/bin/hadoop namenode -format

The output will look like this:

hduser@ruhil:/usr/local/hadoop$ bin/hadoop namenode -format
1/01/12 1:30:41 INFO namenode.NameNode: STARTUP_MSG:
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ruhil/
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = -r 911707; compiled by 'ruhil' on Sun Jan 1 01:30:41 UTC 2012
1/01/12 1:30:41 INFO namenode.FSNamesystem: fsOwner=hduser,hadoop
1/01/12 1:30:41  INFO namenode.FSNamesystem: supergroup=supergroup
1/01/12 1:30:41  INFO namenode.FSNamesystem: isPermissionEnabled=true
1/01/12 1:30:41  INFO common.Storage: Image file of size 96 saved in 0 seconds.
1/01/12 1:30:41  INFO common.Storage: Storage directory .../hadoop-hduser/dfs/name has been successfully formatted.
1/01/12 1:30:41 5/08 16:59:57 INFO namenode.NameNode: SHUTDOWN_MSG:
SHUTDOWN_MSG: Shutting down NameNode at ruhil/

Starting your single-node cluster
hduser@ruhil://usr/local/hadoop$ jps
all this shown below in bash too

Stop hadoop using below command :-

hduser@ruhil:~$ /usr/local/hadoop/bin/
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: stopping datanode
localhost: stopping secondarynamenode

Now Run a Map-reduce job :-

Just watch bash carefully

NOW Finally Lock your Browser Yeah :--)

Hadoop Web Interfaces

Hadoop comes with several web interfaces which are by default (see conf/hadoop-default.xml) available at these locations:

Enjoy :) , process your data with ease and super speed :)
Looking for any kind Help (on Hadoop IRC  with #cloudgeek)
Feel free to mail me