Monday, February 6, 2012

Vim as a Universal IDE {Part 2 (Ruby/Rails )}

when a Newbie starts coding with ruby , it seems hard to switch between files
and running scripts .It seem a lot of time consuming with coding
ruby . Many people ask on irc and groups daily which is best IDE for Ruby.Or
some those used Textmate,Eclispe ,etc.. are also facing time consumption
while write some code! SO ! SO! SO! Now Solution is VIM!

I already Explained basic Installation and working on vim in my previous post
those who are new to vim can visit Part 1 for basic editing and installation
,configuration of vim for better use

I explain here some useful plug-in and working,configuration of those .This
post help ! you to shape code perfect with vim and ruby.
Comments Welcome !!

1) For Enhanced Syntax Highlighting & Navigation(rails.vim)
 automatically detects buffers ,it effect only rails script not ruby scripts
.Easy navigation and interface to script ,integration with other plugins.such
NERDtree.vim with :Rtree

install details
extract the zip to ~/.vim

Download page is here

2)Organization of project like (IDE/Buffer)
this help you launch a .*.jpg files in a viewer. I have also set
up viewers for PDF (acroread) and HTML files (mozilla) for my own use.

install details
untar the file in ~/.vim
for untar use this command :- tar -xzvf filename
download page link is here

3)Ruby filetype and indentation

a)for Ruby indentation :download indentation script and extract(gunzip
filename) gzip  in ~/.vim/indent
download link is here
along with this you need install a plugin Indentanything

install details for that is here    
cd ~/.vim
tar zxvf IndentAnything.tar.gz
download page link is here

: extended % matching for HTML, LaTeX, and many other
languages The matchit.vim script allows you to configure % to match more    than just single characters.  You can match words and even regular expressions.
Also, matching treats strings and comments (as recognized by the
syntax highlighting mechanism) intelligently.
The default ftplugins include settings for several languages:
Ada, ASP with VBS, Csh, DTD, Essbase, Fortran, HTML, JSP
(same as HTML), LaTeX, Lua, Pascal, SGML, Shell, Tcsh, Vim, XML.
(I no longer keep track, so there may be others.)

install details
extract zip in ~/.vim

download page link is here
your .vimrc config must have below one
filetype plugin on

c)ruby-matchit:- Map '%' to jump from one keyword to its corresponding
'end' in Ruby files. Inspired by 'matchit.vim' that comes with Vim

install details

Put the file in ~/.vim/ftplugin/ruby/ directory
download page

4)Database excess ( dbext.vim) : Provides database access to many dbms such as: Mysql, PostgreSQL, Ingres, Oracle, Oracle Rdb (VMS), Sybase Adaptive Server Anywhere, Sybase Adaptive Server Enterprise,iMicrosoft SQL Server, DB2, Interbase and SQLite and ODBC are supported. 
 install deatils
 extract zip in ~/.vim
 download page is here

 5) Haml and Sass syntax, indenting, and ftplugin :-

 Runtime files for Haml and Sass.
 install details
 Extract in ~/.vim or ~\vimfiles.
 download page is here

6) Ruby-test
:- Rubytest.vim will help you to run ruby test (including        vanillatest, rspec, shoulda etc.) in vim.

Unzip downloaded file and copy rubytest.vim to your ~/.vim/plugin directory.
download page for the this

Your .vimrc config you need to do is below
let g:rubytest_cmd_test = "ruby %p"
let g:rubytest_cmd_testcase = "ruby %p -n '/%c/'"
let g:rubytest_cmd_spec = "spec -f specdoc %p"
let g:rubytest_cmd_example = "spec -f specdoc %p -e '%c'"            
let g:rubytest_cmd_feature = "cucumber %p"
let g:rubytest_cmd_story = "cucumber %p -n '%c'"

Default keybind to change

map <Leader>\ <Plug>RubyTestRun " change from <Leader>t to <Leader>\
map <Leader>] <Plug>RubyFileRun " change from <Leader>T to <Leader>]
map <Leader>/ <Plug>RubyTestRunLast " change from <Leader>l to <Leader>/

7) cucumber runtime :- description
This provides syntax highlighting, indenting, and a filetype plugin for
Cucumber, the acceptance testing framework

install details
Extract in ~/.vim

download page

8) Endwise.vim
: Wisely add :-
  This is a simple plugin that helps to end certain structures automatically.In Ruby, this means adding "end" after "if", "do", "def" and several other
  keywords. In Vim Script, this amounts to appropriately adding "endfunction","endif", etc

  install deatils
  install in ~/.vim/plugin
  download page is here

  9) Genutils : General utility functions :-

  This script provides functions that are mostly useful to script developers,but some of the functions can be easily converted to good utilities.
  install details
  extarct in ~/.vim 
  download page

 now config your .vimrc
command! -nargs=0 -range=% SortByLength <line1>,<line2>call QSort(
        \ 'CmpByLineLengthNname', 1)
command! -nargs=0 -range=% RSortByLength <line1>,<line2>call QSort(
        \ 'CmpByLineLengthNname', -1)
command! -nargs=0 -range=% SortJavaImports <line1>,<line2>callQSort(
        \ 'CmpJavaImports', 1)

u might like the following mappings to adjust spacing:
nnoremap <silent> <C-Space> :call ShiftWordInSpace(1)<CR>
nnoremap <silent> <C-BS> :call ShiftWordInSpace(-1)<CR>
nnoremap <silent> \cw :call CenterWordInSpace()<CR>
nnoremap <silent> \va :call

The :find command is very useful to search for a file in path, but it
doesn't support file completion. Add the following command in your vimrc
to add this functionality
nd! -nargs=1 -bang -complete=customlist,genutils#UserFileComplete2
            \ FindInPath :find<bang> <args>

If you are running commands that generate multiple pages of output, you
might find it useful to redirect the output to a new buffer. Put the
following command in your vimrc:
nd! -nargs=* -complete=command Redir
          \ :new | put! =GetVimCmdOutput('<args>') | setl bufhidden=wipe |
          \ setl nomodified

10) multvals.vim
: Array library that uses patterns as separators An array is
nothing but a string of multiple values separated by a
pattern.  The simplest example being Vim's multi-value variables such as
tags. You can use the MvAddElement() function to create an array.
However, there is nothing special about this function, you can as well
make up the string by simply concatenating elements with the chosen
pattern as a separator.

install in ~/.vim/plugin
download page is here

Some other plugin such as Nerd tree or git integration ,supertab ,code navigation are  used in previous post for python. Also textmate like completion and indentation also discuss in previous part .

Note:- Stay Tuned more parts of this Post coming soon for Ruby,C++,C,Closure,Prolog ,Haskell,Java,PHP,Perl,Shell, JavaScript ,HTML,CSS Coming soon (within few Days) .
Update of this post will done soon for Usage of plugin and more config Details .Also preparing image graphics for all post soon update all with how use effectively all these  plug-ins.

                                                                     THANKS FOR VISITING



Saturday, February 4, 2012

VIM as an Universal IDE{Part 1 (Python/Django)}

Vim is My First and Last editor ,IDE, or you can say most of time i spend on it  while work on system .

I write this blog for newbies , those who new to both Linux and Programming.Also for professionals and passionate guys.Also give a look if you are using  different editor then Vim for any Programming language .I write everything that a newbie need to know from installation to configuration , from novice to advance
Any suggestions ! Comments Welcome !here

More about Vim 
Vim is worth millions and costs nothing. It's the best editor the world has
ever seen. Using the trio Vim/Perl/LaTeX I can dispose of most other software.

More To do :

1. Get VIM for Dos
2. Get VIM for Linux
3. Get Vim for Freebsd
4. Get Vim for Dec-Unix
5. Get VIM for Irix
6. Get VIM for Window
7. Get VIM for Mac
8. Laugh at non-VIM users

Basic Introduction :- I Know for Beginner's ,Newbie for Linux ,Vim Need to get in with this From a basic . Following Question can asked by a newbie to Vim
Note:- Please experts those know vim well or For geeks those know it well go below some (stuff is there ) 
   1) May you don't know install Vim on Your's OS(Linux,Mac,Unix etc)
   2) May be you don't know how to open Vim on your OS
   3) May Be You Don't know how install plugin for it
   4) May Be You Don't know how to close it

 Okay before answering you all guys , It work for a large range of Operating System's such as Linux,Mac,Window ,Unix , FreeBSD ..much more.It provide both Command line and GUI interface .

1)Installing Vim
     A)For Linux User's :-                

a) For Ubuntu/Debian user's :- sudo apt-get install vim            
b) For Fedora user's :-  yum install vim-X11 vim-common vim-enhanced vim-minimal
c) For Red-hat Linux user's :- up2date -i vim-X11 vim-common vim-enhanced vim-minimal
d) For Arch Linux user's:- pacman -S vim              
e) For Fedora user's:- emerge vim

B)For  Mac User's :- MacVim is the text editor for Mac OS X , download it form here

C)For  Window User's :-Go to  Vim Download page and click on “PC: MS-DOS and MS-Windows”. Click on the ‘gvim72.exe, which is a Self-installing executable.
(or) Download gvim72.exe directly.          
For All Others Visit download page on

2) Opening or Simple use of Vim
   You  can open a using command line :- use vi/vim  in start of any file name or any file you want open like below        
   $ vim hack.txt                  
   above use of vim with hack.txt file to open

Vim has two mode visual mode (GUI mode) and insert mode(command line mode).     To Switch in Between them press Esc on your keyboard and then Press i for Insert mode and v for visual mode on your keyboard .Esc play a important role for switch you any command . Or you can use Crtl+c ,that is an alternative of Esc .

Note : If not get now also ,then Visit you sure got all open close and simple to advance use link below

3)  Vim have need to have two special area where all vim operate or contains  .Let me Explain
          1) create a ~/.vim directory
          $ mkdir ~/.vim            
          $ cd ~/.vim          
Note:- this Contain all your plugin installation.
          2) Now create file .vimrc under your bash or command terminal                  
           $vim .vimrc                  
Above will create and open .vimrc file in your bash and command terminal            

Note:- This contain information about your  configuration of vim and it's plugins

4)  Close it with press ESc on your Keyboard and then shift + : , then enter wq , wq! , q!            
 these all can use close the vim

Now Vim from Scratch to Novice (You can  say from level 1 to level 2 ):

Just visit and learn there the best online Tutorial

Now move on Real topic Vim as Universal IDE(Integrated Development Environment ):-

 1) Vim as Python/Django IDE :-

 i) Now you need to use ~/.vim and .vimrc i as tell you earlier  Now first step set a better color-scheme wombat,download wombat.vim

Now under after downloading under ~/.vim/colors    
$ mkdir -p ~/.vim/colors          
$ cd ~/.vim/colors          
$ wget -O wombat.vim
$ vim
Now in vim press shift + : ,then enter color wombat like below          
$:color wombat
Now after perfect looking move on !!

ii) For Syntax Error and Formatting (Indent):-
This can used for indentation or formatting every language need formatting and indentation  of that language .This indentation script for python tries to match more closely what is suggested in PEP 8 (  In particular, it handles continuation lines implied by open (parentheses), [brackets] and {braces} correctly and it indents multiline if/for/while statements differently.  Comments are welcome!
install details

Drop the script in your ~/.vim/indent directory.
$mkdir ~/.vim/indent $ cd ~/.vim/indent  
$ wget -O python.vim

download page is here

Now config this in .vimrc    
$ vim .vimrc    
add below  to .vimrc
filetype plugin indent on

iii) Special highlighting for String  and String formatting (Python.vim) :-        

Enhanced version of the original (from vim6.1) python.vim for Python programming language.
The changes since the original python.vim are:
- changed strings highlighting;
- enhanced special symbols highlighting inside strings;
- enhanced numbers highlighting;
- added optional highlighting for %-formatting inside strings;
- added highlighting for some error conditions (wrong symbols in source file,  
mixing spaces and tabs, wrong number values,  
wrong %-formatting inside strings);
- added highlighting for magic comments: source code encoding
 and #! (executable) strings;  - added highlighting
for new exceptions and builtins introduced in python 2.3, 2.4 and 2.5;
- added highlighting for doctests;
- added highlighting for new @decorator syntax introduced in Python 2.4a2;
- added highlighting for trailing-space errors (triggered by new  option: python_highlight_space_errors);
- added highlighting for variable name errors;
- added highlighting for hex number errors;

install details
Place python.vim file in ~/.vim/syntax/ folder.

$mkdir ~/.vim/syntax
$cd ~/.vim/syntax
$wget -O python.vim or download page

download page is here          

Also put config in .vimrc is below
autocmd FileType python set complete+=k~/.vim/syntax/python.vim isk+=.,

( iv) Highlighting Syntax and validation  :-
a) PyFlakes
IMPORTANT: Your vim must have Python 2.5, at least. Enter ":python import sys; print sys.version" to see which version you have.
 The newest version of Vim, 7.3, comes bundled with Python 2.7 support.  pyflakes-vim highlights common Python errors like
misspelling a variable name on the fly. It also warns about unused imports, redefined functions, etc.

install details
1. Make sure your vimrc has "filetype plugin indent on"
so that pyflake-vim's ftplugin files are loaded automatically when you open a Python buffer.
2. Drop extracted files in ~/.vim/ftplugin/python.
3. That's it!

Download page link here
Direct unzip this pyflakes into ~/.vim
Config setting of .vimrc is below
let g:pyflakes_use_quickfix = 0

 b) Pep8        
install inside ~/.vim/ftplugin/python
Download page
now switch to ~/.vim/ftlugin/python          
$cd ~/.vim/ftplugin/python          
$ wget -O pep8.vim

Configuration for .vimrc  put below in your config file .vimrc

let g:pep8_map='<leader>8' set expandtab
set textwidth=79
set tabstop=8
set softtabstop=4
set shiftwidth=4
set autoindent    

v)Code Navigation :-
Catgs  this my best plugin for vim it can use for all programming language's and with this taglist is best for source code navigation .
Install a direct package only for Ubuntu else do manually is below
# apt-get install exuberant-ctags

first download ctags from here .Ctags help you in while browsing a source code file:

Navigating to the function definition by specifying the function name.
Navigating to the function definition from ‘function call’.
Returning back again to function call from the definition.
Viewing the prototype/signature of functions or variables.
Viewing the number of functions in a file, etc., now extract ctags.tar.gz

then do this for configure and install ./configure && sudo make install

after that Config you .vimrc as below
let Tlist_Ctags_Cmd='/usr/local/bin/ctags'
set tags=./tags;/
map <C-\> :tab split<CR>:exec("tag ".expand("<cword>"))<CR>
map <A-]> :vsp <CR>:exec("tag ".expand("<cword>"))<CR>

Best use of this plugin with following command is below :- C-] - go to definition
C-T - Jump back from the definition.
C-W C-] - Open the definition in a horizontal split
C-\ - Open the definition in a new tab A-] - Open the definition in a vertical split

After the tags are generated. You can use the following keys to tag into and tag out of functions:

Ctrl-Left_MouseClick - Go to definition Ctrl-Right_MouseClick - Jump back from definition

Taglist :- with ctag for browsing source code better ,switch to ~/.vim/plugin
$ mkdir ~/.vim/plugin
$cd ~/.vim/plugin
$ wget -O

direct download link is here

Config for .vimrc like below  " Taglist variables

" Display function name in status bar:
let g:ctags_statusline=1
" Automatically start script
let generate_tags=1
" Displays taglist results in a vertical window:
let Tlist_Use_Horiz_Window=0
" Shorter commands to toggle Taglist display
nnoremap TT :TlistToggle<CR>
map <F4> :TlistToggle<CR>
" Various Taglist diplay config:
let Tlist_Use_Right_Window = 1
let Tlist_Compact_Format = 1
let Tlist_Exit_OnlyWindow = 1
let Tlist_GainFocus_On_ToggleOpen = 1
let Tlist_File_Fold_Auto_Close = 1

Tasklist with Ctags for better manipulation

switch ~/.vim/plugin
download there this direct
$ cd ~/.vim/plugin
$wget -O tasklist.vim

Config .vimrc
map T :TaskList<CR>
map P :TlistToggle<CR>
use tasklist with T and P to use this

vi) Code Completion :-    
A) Python Omni  completion  place this in ~/.vim/autoload download direct link here
$ cd ~/.vim/autolaod
$ wget -O pythoncomplete.vim  

add your config to your .vimrc is below
filetype plugin on
set ofu=syntaxcomplete#Complete

B) Pydict for python code completion
download zip direct to ~/.vim/ftplugin  and extract  download page link here

config your .vimrc with below stuff
let g:pydiction_location = 'C:/vim/vimfiles/ftplugin/pydiction/complete-dict'

C) Supertab for pydict easy completion :-
create a ~/.vim/after/ftplugin
download to it direct from this page

Install this now using :- Open the file in vim ($ vim supertab.vba).

 Source the file (:so %)
use to press tab to complete the code

add in .vimrc
let g:SuperTabDefaultCompletionType = "context"

vii) Basic editing and debugging(pep8 ,pyflake is used above) ,code snippets:- 

A) Code snippets(snippetEMU) :-
The same thing works for “for” loops, classes and many other constructs. The bundles .vba provides access to such constructs across a wide range of languages, e.g. Python, Ruby, HTML, PHP, even Djano-specific constructs.

create a ~/.vim/after/ftplugin
download page link is here

now install it above  open with vim snippy_plugin.vba then source the file (:so%)

B) snipMate:-it aims to be an unobtrusive, concise vim script that implements some of TextMate's snippets features in Vim. A snippet is a piece of often-typed text that you can insert into your document using a trigger word followed by a <tab>.   For instance, in a C file using the default installation of snipMate.vim, if you type "for<tab>" in insert mode, it will expand a typical for loop in C:  for (i = 0; i < count; i++) {  }

just download zip and extract in ~/.vim
Download page is here

C) Code folding
put this in ~/.vim/ftplugin
download from this page

D)Block navigation :-This script can be useful when editing Python scripts.
It provides the following menus:
- Select a block of lines with the same indentation
- Select a function
- Select a class
- Go to previous/next class/function
- Go to the beginning/end of a block
- Comment/uncomment the selection
- Jump to the last/next line with the same indent

install details
Copy to the $VIMFILES/ftplugin directory download page is here

E)Minibuffexpl.vim:-  download  here
paste in ~/.vim/plugin

config for .vimrc is below
let g:miniBufExplMapWindowNavVim = 1
let g:miniBufExplMapWindowNavArrows = 1
let g:miniBufExplMapCTabSwitchBufs = 1
let g:miniBufExplModSelTarget = 1

viii)Directory   Navigation:-Nerdtree download page

      install details
      Unzip the archive into your ~/.vim directory.
     That should put NERD_tree.vim in ~/.vim/plugin and NERD_tree.txt in ~/.vim/doc.

ix)Git integration with vim :- download page is here     script_id=2975
    just extract zip in ~/.vim    

x) For automatic closing of quotes, parenthesis, brackets:-   delimitMate
    Download page is here
   download this in ~/.vim/after/ftplugin , Now install using below

   Open the file with Vim and run:

   or use this command to install it in your bundle dir if you use pathogen:
   :UseVimball ~/.vim/bundle/delimitMate /              

Note:- Stay Tuned more parts of this Post coming soon for Ruby,C++,C,Closure,Prolog ,Haskell,Java,PHP,Perl,Shell, JavaScript ,HTML,CSS Coming soon (within few Days) .
Update of this post will done soon for Usage of plugin and more config Details .



Monday, January 2, 2012

Large Scale & Big Data analysis with Hadoop cluster ( Hadoop cluster step on Ubuntu 11.10 server )

Welcome back guys, i promise a setup of Hadoop cluster on Ubuntu Server(64 bit)
last time I post about Openstack and Devstack. Finally again from scratch a install and running tutorial is their

While  Everybody Celebrating New Year ,then I am setting up  my Hadoop cluster  and Processing unstructured data with Hadoop .

Welcome Administrators , Developer , Students, specially Hobbyist and passionates guys interesting in learning cloud .


The purpose of this document is to help you get a single-node Hadoop installation up and running very quickly so that you can get a flavour of the Hadoop Distributed File System (see HDFS Architecture) and the Map/Reduce framework; that is, perform simple operations on HDFS and run example jobs.

Our Pre-requisites for setup :-

1) We Must need a Ubuntu Server (64 bit) or Debian (64bit ) {I am using same this tutorial}
2)  You must have there installed (in main root)
     $ sudo apt-get install ssh (openssh )     
     $ sudo apt-get install rsync  
3) JavaTM 1.6.x, preferably from Sun, must be installed.

    Installing Java
    $ sudo add-apt-repository "deb lucid partner"
    $ sudo apt-get update 
    $ sudo apt-get install sun-java6-jdk sun-java6-plugin  ( while after Downloading  during installation plz press tab to accept JAVA terms and condition while a tab for conditions open ) 
Check is Sun java is there :)

root@ruhil:~# sudo apt-get install sun-java6-jdk sun-java6-plugin 
Reading package lists... Done
Building dependency tree      
Reading state information... Done
sun-java6-plugin is already the newest version.
sun-java6-jdk is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.
root@ruhil:~# java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

Add a Dedicated User 

$ sudo addgroup hadoop
$ sudo adduser --ingroup hadoop hduser

Configure and check is ssh working for local-host

(Please press the enter , you need not specify the name  for File for  and Public key )
root@ruhil:~# su - hduser
hduser@ruhil:~$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hduser/.ssh/id_rsa):
Created directory '/home/hduser/.ssh'.
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/
The key fingerprint is:
9b:82:ea:58:b4:e0:35:d7:ff:19:66:a6:ef:ae:0e:d2 hduser@ruhil
The key's randomart image is:

After do this step carefully

hduser@ruhil:~$ cat $HOME/.ssh/ >> $HOME/.ssh/authorized_key

hduse@ruhil:~$ ssh localhost
The authenticity of host 'localhost (' can't be established.
ECDSA key fingerprint is b8:be:26:41:44:7d:9b:82:02:fd:13:61:3c:ac:d4:0a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Linux ruhil 3.0.0-14-server #23-Ubuntu SMP Mon Nov 21 20:49:05 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

IN Ubuntu 11.10
open /etc/sysctl.conf in the editor of your choice and add the following lines to the end of the file:

#disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

You have to reboot your machine in order to make the changes take effect.

You can check whether IPv6 is enabled on your machine with the following command:

$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6

A return value of 0 means IPv6 is enabled, a value of 1 means disabled (that’s what we want).

Installation of Hadoop (perform this action in your main root like root@ruhil)
{Note Downloading 0.20 version is stable other not stable mainly 0.23,Would like go with 0.20  :)}
root@ruhil:~# mkdir -p /usr/local
root@ruhil:~# cd /usr/local
root@ruhil:~# wget -O hadoop-0.20.2.tar.gz
root@ruhil:~# sudo tar xzf hadoop-0.20.2.tar.gz
root@ruhil:~# mv hadoop-0.20.2 hadoop
root@ruhil:~$ sudo chown -R hduser:hadoop hadoop

Create .Bashrc or If have already paste  below for Hadoop(Note :-you need paste in root and hduser ,if you like you can paste for all,My Paste of .bashrc is :-

# Set Hadoop-related environment variables
export HADOOP_HOME=/usr/local/hadoop

# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)
export JAVA_HOME=/usr/lib/jvm/java-6-sun

# Some convenient aliases and functions for running Hadoop-related commands
unalias fs &> /dev/null
alias fs="hadoop fs"
unalias hls &> /dev/null
alias hls="fs -ls"

# If you have LZO compression enabled in your Hadoop cluster and
# compress job outputs with LZOP (not covered in this tutorial):
# Conveniently inspect an LZOP compressed file from the command
# line; run via:
# $ lzohead /hdfs/path/to/lzop/compressed/file.lzo
# Requires installed 'lzop' command.
lzohead () {
    hadoop fs -cat $1 | lzop -dc | head -1000 | less

# Add Hadoop bin/ directory to PATH

Configuration(Note all the configuration setting you need made in hduser ) :-

The following picture gives an overview of the most important HDFS components.

HDFS Architecture (source:

Our goal in this tutorial is a single-node setup of Hadoop. More information of what we do in this section is available on the Hadoop Wiki.

The only required environment variable we have to configure for Hadoop in this tutorial is JAVA_HOME. Open /conf/ in the editor of your choice (if you used the installation path in this tutorial, the full path is /usr/local/hadoop/conf/ and set the JAVA_HOME environment variable to the Sun JDK/JRE 6 directory.


# The java implementation to use.  Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun


# The java implementation to use.  Required.
export JAVA_HOME=/usr/lib/jvm/java-6-sun

Now we create the directory and set the required ownerships and permissions:

$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown hduser:hadoop /app/hadoop/tmp
$ sudo chmod 755 /app/hadoop/tmp
{Set Your chmod according to your settings  }

Add the following snippets between the <configuration> ... </configuration> tags in the respective configuration XML file.
Note for all given below we need perform all this  below config files

In file conf/core-site.xml:(cd /usr/local/hadoop there all this config )

<!-- In: conf/core-site.xml -->
  <description>A base for other temporary directories.</description>

  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>

In file conf/mapred-site.xml:

<!-- In: conf/mapred-site.xml -->
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.

In file conf/hdfs-site.xml:

<!-- In: conf/hdfs-site.xml -->
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.

hduser@ruhil:~$ /usr/local/hadoop/bin/hadoop namenode -format

The output will look like this:

hduser@ruhil:/usr/local/hadoop$ bin/hadoop namenode -format
1/01/12 1:30:41 INFO namenode.NameNode: STARTUP_MSG:
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ruhil/
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = -r 911707; compiled by 'ruhil' on Sun Jan 1 01:30:41 UTC 2012
1/01/12 1:30:41 INFO namenode.FSNamesystem: fsOwner=hduser,hadoop
1/01/12 1:30:41  INFO namenode.FSNamesystem: supergroup=supergroup
1/01/12 1:30:41  INFO namenode.FSNamesystem: isPermissionEnabled=true
1/01/12 1:30:41  INFO common.Storage: Image file of size 96 saved in 0 seconds.
1/01/12 1:30:41  INFO common.Storage: Storage directory .../hadoop-hduser/dfs/name has been successfully formatted.
1/01/12 1:30:41 5/08 16:59:57 INFO namenode.NameNode: SHUTDOWN_MSG:
SHUTDOWN_MSG: Shutting down NameNode at ruhil/

Starting your single-node cluster
hduser@ruhil://usr/local/hadoop$ jps
all this shown below in bash too

Stop hadoop using below command :-

hduser@ruhil:~$ /usr/local/hadoop/bin/
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: stopping datanode
localhost: stopping secondarynamenode

Now Run a Map-reduce job :-

Just watch bash carefully

NOW Finally Lock your Browser Yeah :--)

Hadoop Web Interfaces

Hadoop comes with several web interfaces which are by default (see conf/hadoop-default.xml) available at these locations:

Enjoy :) , process your data with ease and super speed :)
Looking for any kind Help (on Hadoop IRC  with #cloudgeek)
Feel free to mail me 






Friday, December 16, 2011

E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable),How to fix (Solved)

As a administer, newbie UNIX/Linux user ,or developer/programmer  , you face a problem like this in your shell/bash  in Ubuntu/Debian when any package is broken or due to any reason  your get lock /var/lib/dpkg/lock. or like below you face problem

root@pythongeek:~# apt-get update
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?

you can fix this problem in few steps without rebooting !!

1) type this in bash

    sudo dpkg --configure -a

2) type in bash

    sudo killall apt-get apt aptitude adept synaptic

your output like this

root@pythongeek:~# sudo killall apt-get apt aptitude adept synaptic
apt-get: no process found
apt: no process found
aptitude: no process found
adept: no process found
synaptic: no process foun

Note:if you face problem during installation of any software/package

don't forget accept EULA ,or accepting licensee of user agreement while you have pop-up during installation
3) this is final if you failed last to attempt to unlock
Close all running packages, and open a Konsole window.

type in bash

sudo rm /var/lib/dpkg/lock

then again type this bash

sudo dpkg --configure -a

for reinstallation type like this in bash

sudo apt-get  install -f  package  name here



LHS as a source of information – and a source of inspiration – I hope you’ll choose to act right now.enjoy keep learning.

Sunday, December 11, 2011

Installing openstack on Ubuntu using Devstack with ease in 15 min only (Cloud computing part 2)

Devstack script is a useful tools and good tutorial for us.

It help me know how to install openstack(nova, glance, keystone and so on) from git. And teach us how to config them and make them work fine together. Now using devstack script is very easy if you just want to set up an openstack environment for learning.

Remember you need not install mysql and anything else in advance it will automatically installed from script   

Make sure your system is 11.10 Ubuntu server
you can test it in your really machine or in your vmware or virtualbox desktop machine.

Make sure you open your virtualization switch in your BIOS setting.
you need have to two interfaces, ie eth0 and eth1
$ sudo apt-get install bridge-utils     # install birdge
$ vi /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static

auto br100
iface br100 inet static
bridge_ports eth2
bridge_stp off
bridge_maxwait 0
bridge_fd 0

in this file, eth0 is your public network, you need it to connect to internet. eth2(maybe in your system is eth1) is another network interface used as a bridge(br100), is your private ip, it is not need connect to any other network in our test.just choose a private network like and remember.

$ sudo apt-get install git
$ git clone git://
$ cd devstack
$ vi localrc
add these info into localrc


$ ./stack

it will ask for some password, remeber don't use special characters like $*_.! and so on. you can simply set password like 123456
then it will begin install , how much it take depends on your network bandwidth. If it failed since of network or some reason. look at error log carefully, delete /opt/stack/devstack and run ./stack again.

At last, you will see successful information, and then you can type your IP in your browser. Enjoy your open-stack time and it is a good thing you need learning more and more. finally you are here .

Looking for Help/ Technical support  !

Feel free to mail me



LHS as a source of information – and a source of inspiration – I hope you’ll choose to act right now.enjoy keep learning.

Creating a Private Cloud using Openstack in Ubuntu Maually (Cloud Computing Part 1)

This article dedicated to developer ,project leaders , Cloud architects . I am gonna explain how to setup a cloud using command line .This useful for newbies to cloud and UNIX both !

I am using a Ubuntu 11.10 (64 bit server ), so lets play with shell !!

for setting up a prefect cloud you need minimum two network NIC on your hardware , a lot of public Ip . :)

But i am setting it on one network  it on one NIC hardware , with One IP (students and tester can with way those have less resources ) :)

We are setting up it with open-stack .
Official website of this document is below:

ubuntu 11.10 environment deployment, select the network mode FLATDHCP

public interface: eth0 used to connect to the user
private interface: eth1 do bridge br100, used, and other nodes, keystone, glance, volume and other connections

Minimum installed Ubuntu

Remember to open-ssh installed, where we have established a common user open-stack

Giving it the NOPASSWD sudo privileges, so easy to operate behind

$ Sudo apt-get update          //  (apt update about the tree)
$ Sudo apt-get install bridge-utils     //   (  installed bridge components )

Configure the network
$ vi / etc / network / interfaces      //(gksudo also can be used instead of vi for GUI Fans )

auto eth0
iface eth0 inet static

auto br100
iface br100 inet static
bridge_ports eth1
bridge_stp off
bridge_maxwait 0
bridge_fd 0

$ Sudo / etc / init.d / networking restart

Initial preparatory work to do, the next step is to install on nova, glance and other components

$ Sudo apt-get install-y rabbitmq-server  // (install the MQ message components )
$ Sudo apt-get install-y python-greenlet python-mysqldb     // (install Python dependencies )

Next, install the various nova components and dependencies

$ Sudo apt-get install nova-volume nova-vncproxy nova-api nova-ajax-console-proxy
$ Sudo apt-get install nova-doc nova-scheduler nova-objectstore
$ Sudo apt-get install nova-network nova-compute
$ Sudo apt-get install glance

Installation euca2ools and unzip
$ Sudo apt-get install-y euca2ools unzip

Next we install the database, I chose MYSQL, PostgreSQL actually personally feel better
$ Sudo su - to root user to change
# MYSQL_PASS = nova nova set mysql database password and the password
# NOVA_PASS = notnova here nova and notnova modified according to their definitions
# Cat <<MYSQL_PRESEED | debconf-set-selections
> Mysql-server-5.1 mysql-server/root_password password $ MYSQL_PASS
> Mysql-server-5.1 mysql-server/root_password_again password $ MYSQL_PASS
> Mysql-server-5.1 mysql-server/start_on_boot boolean true
# Apt-get install-y mysql-server
# Exit exit root environment

$ Sudo sed-i 's/' / etc / mysql / my.cnf modify my.cnf configuration file
$ Sudo service mysql restart

$ MYSQL_PASS = nova in the general user environment variable to the password once again set about
$ NOVA_PASS = notnova
$ Sudo mysql-uroot-p $ MYSQL_PASS-e 'CREATE DATABASE nova;'

// (to create a name for the ova of the database, I recommend that new users nova's name, if here for another name, then the configuration file in the nova which also need to change )
$ Sudo mysql-uroot-p $ MYSQL_PASS-e "GRANT ALL PRIVILEGES ON *.* TO
$ Sudo mysql-uroot-p $ MYSQL_PASS-e "SET PASSWORD FOR 'nova'@'%' =

This point. nova, glance of the installation completed, next is the configuration

nova configuration
$ Sudo vi / etc / nova / nova.conf
- Dhcpbridge_flagfile = / etc / nova / nova.conf
- Dhcpbridge = / usr / bin / nova-dhcpbridge
- Logdir = / var / log / nova
- State_path = / data / openstack / nova here / data / openstack / nova is a new volume and directory, make sure you have this, and the user should belong to nova
- Instances_path = / data / openstack / nova / instances to modify the default storage of instances where
- Lock_path = / var / lock / nova
- Force_dhcp_release = True
- Use_deprecated_auth
- Iscsi_helper = tgtadm
- Verbose
- Scheduler_driver = nova.scheduler.simple.SimpleScheduler
- Network_manager =
- My_ip = This is my ip address within the network
- Public_interface = eth0
# - Vlan_interface = eth0
- Sql_connection = mysql: / / nova: notnova @ localhost / nova
- Libvirt_type = kvm
# - Osapi_extensions_path = / opt / nova / bin / openstackx / extensions
# - Vncproxy_url =
# - Vncproxy_wwwroot = / data / stack / noVNC /
- Api_paste_config = / etc / nova / api-paste.ini
- Image_service = nova.image.glance.GlanceImageService
- Ec2_dmz_host =
- Ec2_url =
- Rabbit_host = localhost
- Glance_api_servers =
- Flat_network_bridge = br100
- Flat_interface = eth1
- Flat_network_dhcp_start = specified instances allocated from the beginning from the 51, but looks like this option does not work
- Fixed_range = This option specifies the instances of the network segment
- Flat_injected = False
- Multi_host = 1 using multi_host
- Libvirt_use_virtio_for_bridges instances using virtio network card model do
# - Start_guests_on_host_boot = true
# - Resume_guests_state_on_host_boot = true
- Use_ipv6 = false

$ Sudo vi / etc / glance / glance-api.conf
// (In this file without using keystone of the present case, according to your needs, modify filesystem_store_datadir parameter to specify the directory you need to store images, Dangran are the main users have Gaicheng glance )
$ Sudo vi / etc / glance / glance-registry.conf
// (This file can be selected to modify sql_connection parameter to specify your database. Of course you can not change.
If you want to modify, use mysql which database to ensure that the mysql which established a corresponding database )

sql_connection = mysql: / / nova: notnova @ localhost / glance this is my configuration, I created a glance in the mysql database

$ Sudo chown-R root: nova / etc / nova to change / etc / nova's owner
$ Sudo chmod 640 / etc / nova / nova.conf

Restart all services
$ Sudo restart libvirt-bin
$ Sudo restart nova-network
$ Sudo restart nova-compute
$ Sudo restart nova-api
$ Sudo restart nova-objectstore
$ Sudo restart nova-scheduler
$ Sudo restart glance-registry
$ Sudo restart glance-api
Note: We do not have from the nova-volume, because although we installed the volume, but the volume needs to use a separate vg, we have not configure the volume, so get up.

There may be network services and can not compute it, do not worry about being first

Next, we do configure the operating environment of the nova
$ Sudo nova-manage db sync
nova-manage user admin <user_name> where we can create a user, such as
$ Sudo nova-manage user admin test, create a successful return on the screen like:
export EC2_ACCESS_KEY = d6aa7747-4324-4abc-9604-4f7d6a2f8f3f
export EC2_SECRET_KEY = 2b204b75-da2d-47b8-ba7a-611d71f0ecbf

nova-manage project create <project_name> <user_name> create a project, we built that are just users, such as:
$ Sudo nova-manage project create test-proj test
nova-manage network create - help create an instance of the network, such as:
$ Sudo nova-manage network create - label = test-net - fixed_range_v4 = - num_network = 1 - network_size = 256

Services have failed to start again
$ Sudo start nova-network
$ Sudo start nova-compute
$ Sudo start nova-scheduler
In addition, since each service, the best look at the log, such as sudo tail-f / var / log / nova / nova-network to determine there is no error, you can also use the ps aux | grep [n] ova-network to confirm the service is not open. If the starting service fails, you confirm that a good reason to modify a good future, need to use sudo start to play instead of sudo restart

Well. This computing environment, we deployed the. We can look at the state command
$ Sudo nova-manage service list
$ Sudo nova-manage network list

Next, create a certificate, to facilitate the tool we use euca
$ Cd
$ Mkdir creds
$ Sudo nova-manage project zipfile test-proj test creds /
$ Unzip creds / creds /
$ Source creds / novarc

OK, done, we can use the tool to look at
$ Euca-describe-availability-zones verbose
VAILABILITYZONE nova available
AVAILABILITYZONE | | - Nova-network enabled :-) 2011-10-17 04:45:44
AVAILABILITYZONE | | - Nova-compute enabled :-) 2011-10-17 04:45:45
AVAILABILITYZONE | | - Nova-scheduler enabled :-) 2011-10-17 04:45:46

So far, successfully enabled services. When you find that service is not working, use ps aux | grep nova check services are not open, and the need for detailed observations / var / log / nova / directory log files for each service, in order to obtain further information.

Then we can use kvm to create a mirror image

$ Sudo apt-get install kvm-pxe installation about this, otherwise there will be time to run kvm warning
$ Kvm-img create-f raw server.img 5G
$ Sudo kvm-m 1024-cdrom rhel5.iso-drive file = server.img, if = virtio, index = 0-boot d-net nic-net user-nographic-vnc: 0

Here we use rhel5(Redhat) the iso, after running this command, you can use vnc to connect the machine to connect to your server: ssvncviewer 0
Open vnc you can see the installation interface After the installation, the following paragraph written rhel mirror / etc / rc.local the beginning of the
modprobe acpiphp

# Simple attempt to get the user ssh key using the meta-data service
mkdir-p / root / .ssh
echo>> / root / .ssh / authorized_keys
curl-m 10-s | grep 'ssh-rsa'>> / root / .ssh / authorized_keys
echo "************************"
cat / root / .ssh / authorized_keys
echo "************************"
Save and exit, so that image on the well

Upload images using glance
$ Glance - verbose add name = "rhel5" disk_format = raw is_public = true <server.img
You should also observe the / var / log / glance / registry and the log api
$ Glance index to see a list of mirrors

Start your instance
$ Euca-describe-images
You can now view the image, output similar to the
IMAGE ami-00000003 server.img
Remember this image here ami-000000003 No.
$ Euca-run-instances-t m1.tiny ami-00000003 start an instance of ami-00000003 image
-T specifies the type of instance, provides the type of cpu, memory, disk size, etc..
Watch / var / log / nova / nova-api.log nova-scheduler.log, nova-compute, nova-network.log of output, but you can also use vnc to connect serverip: 0 look at the console instance
With the command $ euca-describe-instances to see your current instance of the first instance will be relatively slow start because of the need to copy the image from a glance under the instance directory to the nova

Conclusion As the nova is currently growing fast, diablo release version of the function of some of the requirements to be completed daily. But the development version of the nova can be better combined with keystone, novaclient, dashboard and some other projects. Makes openstack more robust. Friends who are interested, you can use in a production environment repo's installation, test development versions of the test environment. Since I use in a production environment install git development version. Therefore, a more complete integration of follow-up, I will develop versions of the form. Of course, using the development version, then there will be more trouble, but also have more fun and hands-on practice, to further understand the mechanism of its working principle.

Looking for help /technical support  !

Feel free to mail me



LHS as a source of information – and a source of inspiration – I hope you’ll choose to act right now.enjoy keep learning.


Sunday, November 20, 2011


Even though the cloud continues to grow in popularity and respectability, complications with data privacy and data protection still plague the market.There's a long way to go before cloud becomes mainstream ,Obstacles on the road to cloud computing.Data security is a concern for any enterprise, and cloud computing often can magnify security anxieties. Adopting a few ground rules will help protect users, their data and your overall cloud investment

Everywhere the acceptance of cloud computing by corporate, now one of the fastest emerging technology in field of Information Technology (IT). From startup to experienced companies often lack of protection measure to weather off an attack on their servers due to scarcity of resources:
  • Due to poor programming skills that explores software vulnerabilities in Python, Ruby, PhP, JavaScript.
  • Even some good programmers aren't aware about Cyber security so Companies need hybrid developers (security + good programmers)
  • Open ports to firewalls or insufficient knowledge of security by system administrator, Knowledge of Nmap. Nessues, Snort must be mendentory for system administrator before hiring them for cloud based firms.
  • Those companies are encouraged to pursue cloud computing must have need to support their own hardware backbone.

Due to Ineffective Security Policies many of startup or firms afraid to adopt cloud computing. But there is solution how Rescue security in cloud their are above 4 point must be remembered by companies when they gonna to hire guys for their cloud operations.


Basic Definition of Information security have 3 aspects according to our Corporate world refers as confidentiality, integrity and availability, these are major security issue for cloud vendors.

Confidentiality refers that who stores the encryption key is must be secure from users, employees only permitted to respected officials whose are responsible for operation.

Integrity refers every firms must need to have some policies for no data exchange without strict set of rules or protocols. Must be aware from client side don't store sensitive data and password those can be stolen easily.

Availability, this is one major concern by security experts, many of large caps in cloud computing have already experienced downtime. There must be relationship only among clients and provider, no third parties. Also need to focus on protocols to increase security during authentication should backed several methods using combination of hardware and password, or including face rectification of face, figure prints. Because SSL is also hacked by some hackers so need to focus on these protocols. Even Amazon faced denial of service attack.

Untrustworthy supplier, eavesdropping, impersonation, data theft, lack of performance and logical and physical disasters are addressed by this pattern. Consider checking supplier applications for Cross-site scripting (XSS) attacks which can be used to log keystrokes and capture data, and propagate web application worms such as Samy. Feed injection for RSS and Atom can allow an attacker to compromise applications, if feeds are not properly secured

All programmer must need to go through some Relevant Technologies those are safer than other such Django in python, AJAX, RSS, JSON, Gears in JAVA, SOAP (Simple object access Protocol), REST (Representational State Transfer)


Information privacy is the interest an individual has in controlling or at-least significantly influencing, the handling of data about themselves. Need to ensure data among only the clients and provider only but how?

  • Need to insure security of MapReduce for privacy and confidentiality using Airavat our main aim here to provide assurances privacy for sensitive data
  • Using System Model such as data provider have their own set of data, cloud provider need to use Airavat Framework.
  • Trustworthiness of data provider using a Threat Model
  • Computation Model of MapReduce must be deployed into input chunks so that mapper and reducer so that set of the input and output can have separate function and chunks
  • Differential privacy concept can used to ensure privacy and Random Laplacian noise, other noise algorithm can be used or implemented during designing.
  • Functional Sensitivity algorithm can be used during designing of model for cloud architecture, some new policies such as use of SELinux and MAC model for key architecture of any cloud access control.
  • In case must concern over such as Security (OpenID,. Net Access Control, PKI), and Load Monitoring and Testing (Soasta, Hyperic), Provisioning and Configuration Management.

Basic Network Security refers performance, bandwidth, quality, availability, more flexibility over the networks. Reconfiguration of network according to clients such as Network as a Service (NaaS)
and virtualization of cloud network to users

  • Provide the basic network functions for applications with highly variable demands according.
  • Integrating functionality with computing and storage with variable demands during seasons with peak requirement
  • Integration of necessary tools for management and security such as Snort, Nmap, Tcpdump ... etc


Security as a service can be used to ensure security in cloud. Need to run anti-virus on both side of cloud on vendor and client side, multi-version of anti-virus can be used for ensuring proper security from malware.

  • Reduce possibility number of bugs, contribute to open-source so can benefited from community.
  • Multiple Functionality equivalent programmers independently, use multiple scanner's in parallel to increase detection rate
  • Use cloud forensics also go trough with no-vendor lock in service for better productivity so that
    clients never agnostic with vendor service

  • Need to improve system Architecture, include specific protocols on both side for the end-users
    and use hash technique to extract unique id of host and user using MAC.

  • Certification and 3 rd party audits is provider is certified.


Despite the numerous security involved with cloud computing, it is critical that industry ans organization taking a thoughtful and proactive approach to cloud. So as we need it as basic utility in our in daily uses so we need to insure security in cloud. There are following suggestion are provided here to ensure security of your cloud

  • Secure Architecture Model such need to integrated applications, for security architecture community. Some important entities involved in the data flow are end users, developers, system architect, Auditors. Need to integrate all application level tool inside architecture so that we can use the grid concept of security with step by step.
  • End Uses: Use certain protocol and policies those committed to ensure security while they are accessing resources from cloud. Need to use signatures and token on server side and on client side must need to use firewalls, entry point protocols, need to update regularly so that can patch bugs as soon as possible. Also need to ensure security using TSL, SSL, using Secure IPSEC.
  • System Architects: These Guys must employed using some certain policies should have knowledge of basic security concepts and tools that are need to use to prevention of cloud. They also need to write policies that pertain to installation and configuration of hardware components such as Firewalls, Servers, Routers, Operating systems, Proxy server configurations and encryption tunnels
  • Developers: Developers must need to gone through with security concepts. They may desire extra virtual machines to either generate test data or to perform data analysis, processes and penetrate their code. Monitoring of API call for the server requests for software must be include inside the architectural model




LHS as a source of information – and a source of inspiration – I hope you’ll choose to act right now.enjoy keep learning.