Ceph Storage :: Next Big Thing

noreply@blogger.com (Anonymous) — Wed, 25 Feb 2015 20:10:00 +0000

Now Showing : Learning Ceph

a comprehensive Book on Software Defined Storage : CEPH

Hello Ceph 'ers ,

The year 2014 is pretty productive to Ceph and to its surrounding world. Ceph entered the 10 year maturity haul with its 10th Birthday.The most buzzed news "RedHat acquired Inktank" was a major success to Ceph and its community, and finally 'Ceph Firefly' the Long Term Support production grade version is out in 2014 , with its wow features like Erasure Coding and Cache Tiering.

During the same year , somewhere in this world i was investing my several months for writing the first ever book on Ceph . Finally it was 20 January 2015 when my First as well as Ceph's First book got published.

Introducing Learning Ceph : a comprehensive guide to learn software defined massively scalable Ceph storage system.

I have compiled this book with all my experience on Ceph and other related technologies and i am sure this book you pure knowledge on Ceph that you are always wondering about.

You can grab a Free Sample copy of this book from : http://bit.ly/1woQnOW

The book is available for purchase from variety of channels and on various formats.

Amazon.com : http://amzn.to/17YeGaF
Safari Books : http://bit.ly/18lOKqq
Publishers Official website : http://bit.ly/1LDRKL0

I am sure , if you are interested to learn Ceph for building up

your multi-petabyte storage system , this book is 100%

a must have resource for Ceph.

So .....

Ceph Meetup Helsinki : 22 Jan 2015

noreply@blogger.com (Anonymous) — Sat, 24 Jan 2015 16:57:00 +0000

Ceph Meetup Helsinki , Finland 22nd Jan 2015

It has been a good start to 2015. We the geeks of "Helsinki Metropolitan Area" express our sincere thanks to Red Hat Folks for arranging an unofficial "Ceph Day" sort of Ceph Meetup.

From my point of view expecting any event on CEPH in Finland , is like day dreaming. But hopefully not from now.

Here is my presentation during the meetup. Hope you enjoy and learn something new out of it. Read it carefully "There is something new for YOU"

Introducing Try-Ceph

Since this was the first presentation on Ceph , in-order to keep the audience awake and into the gravity of Ceph. I have presented a LIVE DEMONSTRATION of Ceph which the audience really enjoyed.

So What is Try-Ceph

Its a shortest and the quickest way to get your TEST Ceph cluster UP and Running in Just 10 Minutes --yes-i-am-really-really-serious

Its a TWO Step Process

Step - 1 : # git clone https://github.com/ksingh7/Try-Ceph.git

Step - 2 : # vagrant up

Checkout the documentation

https://github.com/ksingh7/Try-Ceph/blob/master/README.md

Checkout the recorded session

https://www.youtube.com/watch?v=lki1fQptIRk

Some Pictures from the Meetup

Ceph Calamari : The Survival Guide

noreply@blogger.com (Anonymous) — Tue, 23 Sep 2014 15:13:00 +0000

Ceph Calamari : step-by-step

Calamari is a management and monitoring system for Ceph storage cluster. It provides a beautiful Dashboard User Interface that makes Ceph cluster monitoring amazingly simple and handy. Calamari was initially a part of Inktank's Ceph Enterprise product offering and it has been open sourced few months back by Red Hat. Hence Calamari is a matured enterprise level software which is now Open Source.

If you are wondering how you can deploy calamari for your Ceph cluster , this blog will navigate you with the step by step instruction. As of now precompiled packages for calamari are not available , you would require to compile from source and get calamari up and running for your environment.

I would like to express my thanks to Gregory Meno and Dan Mick from Inktank ( RedHat ) for their support during troubleshooting.

Introduction

Calamari is the management and monitoring service for Ceph, exposing a high level REST API.

Calamari consists of two pieces (and two separate repos, now in the Ceph Github project):

1) Backend ( Calamari Server ) — the Calamari backend is written in Python 2.6+, using Saltstack, ZeroRPC, gevent, Django, django-rest-framework, graphite and instantiates a new REST API for integration with other systems. This is an important distinction because earlier versions of Calamari were based on the Ceph REST API. The new Calamari REST API is designed to be much more comprehensive and should be the basis for new development efforts that wish to interact with a Ceph cluster. Source Code :https://github.com/ceph/calamari

2) Frontend ( Calamari Clients ) — the Calamari frontend is a web browser UI implemented primarily in Javascript that uses the Calamari REST API. Source Code : https://github.com/ceph/calamari-clients

Calamari Server side components include : Apache, salt-master , supervisord , cthulhu , carbon-cache

Calamari Client side components include : salt-minion , diamond

Building Calamari from source

Unlike Ceph , Calamari does not comes with prebuilt RPM / DEB packages. If you are interested in calamari you need to build your own packages based on your distributions. To deploy Calamari , you need to build packages for Calamari Server as well as for Calamari clients.

Building Calamari Server packages

These instruction will guide you to build RPM packages that will be used with RHEL based distributions. For Ubuntu DEB packages , most of the steps are same. I would try to highlight extra steps if required.

1. Install GIT and vagrant on your local workstation.

For example i am using a MAC workstation and will connect to my Ceph cluster and other servers over SSH. So i would install GIT and vagrant locally on my machine. On this workstation we will create some vagrant instances and inside those vagrant instances we will build RPM packages for Calamari server and client . Once packages are built , we will transfer the packages from the workstation to the Server machine that you designate for Calamari server and calamari clients.

2. Clone calamari server and diamond repositories.

$ mkdir /tmp/calamari-repo
$ cd /tmp/calamari-repo

$ git clone https://github.com/ceph/calamari.git
Cloning into 'calamari'...
remote: Counting objects: 10253, done.
remote: Compressing objects: 100% (4433/4433), done.
remote: Total 10253 (delta 5434), reused 10188 (delta 5387)
Receiving objects: 100% (10253/10253), 20.53 MiB | 3.55 MiB/s, done.
Resolving deltas: 100% (5434/5434), done.
Checking connectivity... done.
$

$ git clone https://github.com/ceph/Diamond.git --branch=calamari
Cloning into 'Diamond'...
remote: Counting objects: 16225, done.
remote: Compressing objects: 100% (9229/9229), done.
remote: Total 16225 (delta 6170), reused 16225 (delta 6170)
Receiving objects: 100% (16225/16225), 3.79 MiB | 1.19 MiB/s, done.
Resolving deltas: 100% (6170/6170), done.
Checking connectivity... done.
$

3. Check calamari contents , under vagrant directory , you would find various distributions . Select your distribution and perform vagrant up.

Ubuntu users : In this step select the vagrant directory precise-build for ubuntu DEB packages.

$ cd calamari/
$ ls -la
total 168
drwxr-xr-x  32 ksingh  wheel   1088 Sep 17 10:34 .
drwxr-xr-x   4 ksingh  wheel    136 Sep 17 10:36 ..
drwxr-xr-x  13 ksingh  wheel    442 Sep 17 10:34 .git
-rw-r--r--   1 ksingh  wheel    292 Sep 17 10:34 .gitignore
-rw-r--r--   1 ksingh  wheel   1310 Sep 17 10:34 .travis.yml
-rw-r--r--   1 ksingh  wheel    251 Sep 17 10:34 COPYING
-rw-r--r--   1 ksingh  wheel  26436 Sep 17 10:34 COPYING-LGPL2.1
-rw-r--r--   1 ksingh  wheel   8977 Sep 17 10:34 Makefile
-rw-r--r--   1 ksingh  wheel   2183 Sep 17 10:34 README.rst
-rwxr-xr-x   1 ksingh  wheel   2961 Sep 17 10:34 adduser.py
drwxr-xr-x   6 ksingh  wheel    204 Sep 17 10:34 alembic
-rwxr-xr-x   1 ksingh  wheel    979 Sep 17 10:34 build-rpm.sh
drwxr-xr-x   5 ksingh  wheel    170 Sep 17 10:34 calamari-common
drwxr-xr-x   4 ksingh  wheel    136 Sep 17 10:34 calamari-web
-rw-r--r--   1 ksingh  wheel   3785 Sep 17 10:34 calamari.spec
drwxr-xr-x  11 ksingh  wheel    374 Sep 17 10:34 conf
drwxr-xr-x   5 ksingh  wheel    170 Sep 17 10:34 cthulhu
drwxr-xr-x  13 ksingh  wheel    442 Sep 17 10:34 debian
drwxr-xr-x  11 ksingh  wheel    374 Sep 17 10:34 dev
drwxr-xr-x  13 ksingh  wheel    442 Sep 17 10:34 doc
-rwxr-xr-x   1 ksingh  wheel    659 Sep 17 10:34 get-flavor.sh
-rwxr-xr-x   1 ksingh  wheel    874 Sep 17 10:34 get-versions.sh
drwxr-xr-x   5 ksingh  wheel    170 Sep 17 10:34 minion-sim
-rwxr-xr-x   1 ksingh  wheel    326 Sep 17 10:34 pre-commit.py
drwxr-xr-x   4 ksingh  wheel    136 Sep 17 10:34 repobuild
drwxr-xr-x   4 ksingh  wheel    136 Sep 17 10:34 requirements
drwxr-xr-x   6 ksingh  wheel    204 Sep 17 10:34 rest-api
drwxr-xr-x   4 ksingh  wheel    136 Sep 17 10:34 salt
drwxr-xr-x  17 ksingh  wheel    578 Sep 17 10:34 tests
-rw-r--r--   1 ksingh  wheel    792 Sep 17 10:34 tox.ini
drwxr-xr-x  11 ksingh  wheel    374 Sep 17 10:34 vagrant
drwxr-xr-x   3 ksingh  wheel    102 Sep 17 10:34 webapp
$
$ cd vagrant/
$ ls -la
total 8
drwxr-xr-x  11 ksingh  wheel   374 Sep 17 10:34 .
drwxr-xr-x  32 ksingh  wheel  1088 Sep 17 10:34 ..
drwxr-xr-x   4 ksingh  wheel   136 Sep 17 10:34 centos-build
drwxr-xr-x   5 ksingh  wheel   170 Sep 17 10:34 devmode
drwxr-xr-x   4 ksingh  wheel   136 Sep 17 10:34 precise-build
drwxr-xr-x   4 ksingh  wheel   136 Sep 17 10:34 production
drwxr-xr-x   4 ksingh  wheel   136 Sep 17 10:34 rhel-build
drwxr-xr-x   4 ksingh  wheel   136 Sep 17 10:34 rhel7-build
drwxr-xr-x   4 ksingh  wheel   136 Sep 17 10:34 trusty-build
-rwxr-xr-x   1 ksingh  wheel   344 Sep 17 10:34 urllib-bootstrap-salt.sh
drwxr-xr-x   4 ksingh  wheel   136 Sep 17 10:34 wheezy-build
$
$ cd centos-build/
$ ls -la
total 8
drwxr-xr-x   4 ksingh  wheel   136 Sep 17 10:34 .
drwxr-xr-x  11 ksingh  wheel   374 Sep 17 10:34 ..
-rw-r--r--   1 ksingh  wheel  1112 Sep 17 10:34 Vagrantfile
drwxr-xr-x   4 ksingh  wheel   136 Sep 17 10:34 salt
$

4. Finally perform vagrantup for Centos distribution, this will create a virtual environment with all the prerequisite necessary for calamari server package building process.

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'centos6'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: centos-build_default_1410939808331_97817
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 => 2201 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2201
    default: SSH username: vagrant
    default: SSH auth method: private key
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
    default: /git => /private/tmp/calamari-repo
    default: /vagrant => /private/tmp/calamari-repo/calamari/vagrant/centos-build
    default: /srv/salt => /private/tmp/calamari-repo/calamari/vagrant/centos-build/salt/roots
==> default: Running provisioner: salt...
Copying salt minion config to vm.
Checking if salt-minion is installed
salt-minion was not found.
Checking if salt-call is installed
salt-call was not found.
Bootstrapping Salt... (this may take a while)
Salt successfully configured and installed!
run_overstate set to false. Not running state.overstate.
run_highstate set to false. Not running state.highstate.
$

5. On this new CentOS vagrant VM , you would notice some filesystem were mounted , actually they calamari source directory that we have cloned from github and has been shared and mounted as file system on CentOS vagrant VM. Now for building packages , we will use the same directory hierarchy and the resultant RPM package will also be stored in this shared directory.SSH into vagrant CentOS instance

$ vagrant ssh

[vagrant@vagrant-centos64 ~]$
[vagrant@vagrant-centos64 ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             7.3G 1017M  6.0G  15% /
tmpfs                 246M     0  246M   0% /dev/shm
git                   233G  214G   20G  92% /git
vagrant               233G  214G   20G  92% /vagrant
srv_salt              233G  214G   20G  92% /srv/salt
[vagrant@vagrant-centos64 ~]$
[vagrant@vagrant-centos64 ~]$ cd /git
[vagrant@vagrant-centos64 git]$ ll
total 0
drwxr-xr-x 1 vagrant vagrant 1088 Sep 17 07:34 calamari
drwxr-xr-x 1 vagrant vagrant  952 Sep 17 07:35 Diamond
[vagrant@vagrant-centos64 git]$

6. Build you Calamari server RPM packages , with the below salt command.

$ sudo salt-call state.highstate

[vagrant@vagrant-centos64 ~]$ sudo salt-call state.highstate
[INFO    ] Loading fresh modules for state activity
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/modules'
[INFO    ] Syncing modules for environment 'base'
[INFO    ] Loading cache from salt://_modules, for base)

............... Output truncated ................


----------
          ID: cp-artifacts-to-share Diamond/dist/diamond-*.noarch.rpm
    Function: cmd.run
        Name: cp Diamond/dist/diamond-*.noarch.rpm /git/
      Result: True
     Comment: Command "cp Diamond/dist/diamond-*.noarch.rpm /git/" run
     Changes:
              ----------
              pid:
                  28292
              retcode:
                  0
              stderr:

              stdout:


Summary
-------------
Succeeded: 11
Failed:     0
-------------
Total:     11
[vagrant@vagrant-centos64 ~]$

For brevity the output of above command has been truncated . You can follow the output here http://paste.ubuntu.com/8363649/

7. The output summary confirms that there are 11 Successful and 0 Failed operations . List the directories to find your Calamar Server RPM package. Don't worry this should be copied to the base directory of your workstation , as the same has been shared and mounted with vagrant instance.

[vagrant@vagrant-centos64 ~]$ ll
total 14964
drwxr-xr-x 19 vagrant vagrant     4096 Sep 17 08:00 calamari
-rw-r--r--  1 vagrant vagrant 15303265 Sep 17 08:00 calamari-server_1.2.1.tar.gz
drwxr-xr-x 12 vagrant vagrant     4096 Sep 17 07:59 Diamond
drwxr-xr-x  8 vagrant vagrant     4096 Sep 17 08:00 rpmbuild
[vagrant@vagrant-centos64 ~]$
[vagrant@vagrant-centos64 ~]$ cd rpmbuild/
[vagrant@vagrant-centos64 rpmbuild]$
[vagrant@vagrant-centos64 rpmbuild]$
[vagrant@vagrant-centos64 rpmbuild]$ find . -name *.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/sshpass-1.05-1.el6.x86_64.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/openpgm-5.1.118-3.el6.x86_64.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/python-msgpack-0.1.13-3.el6.x86_64.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/salt-2014.1.10-4.el6.noarch.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/PyYAML-3.10-3.el6.x86_64.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/python-babel-0.9.4-5.1.el6.noarch.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/python-zmq-2.2.0.1-1.el6.x86_64.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/salt-minion-2014.1.10-4.el6.noarch.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/libyaml-0.1.6-1.el6.x86_64.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/python-crypto-2.0.1-22.el6.x86_64.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/zeromq3-3.2.4-1.el6.x86_64.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/m2crypto-0.20.2-9.el6.x86_64.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/diamond-3.4.67-0.noarch.rpm
./BUILD/calamari-server-1.2.1/repobuild/el6/python-jinja2-2.2.1-2.el6_5.x86_64.rpm
./RPMS/x86_64/calamari-server-1.2.1-29_g9d02dc3.el6.x86_64.rpm
[vagrant@vagrant-centos64 rpmbuild]$

8. You should now exit from the centos vagrant machine and on your local machine list the directory you would find rpm packages for calamari server and diamond.

teeri:calamari-repo ksingh$
teeri:calamari-repo ksingh$ ls -la
total 61960
drwxr-xr-x   7 ksingh  wheel       238 Sep 17 11:18 .
drwxrwxrwt  10 root    wheel       340 Sep 17 11:46 ..
drwxr-xr-x  28 ksingh  wheel       952 Sep 17 10:35 Diamond
drwxr-xr-x  32 ksingh  wheel      1088 Sep 17 10:34 calamari
-rw-r--r--   1 ksingh  wheel   7038999 Sep 17 11:18 calamari-repo-el6.tar.gz
-rw-r--r--   1 ksingh  wheel  24082060 Sep 17 11:18 calamari-server-1.2.1-29_g9d02dc3.el6.x86_64.rpm
-rw-r--r--   1 ksingh  wheel    595548 Sep 17 11:18 diamond-3.4.67-0.noarch.rpm
teeri:calamari-repo ksingh$

9 . At this point you are ready with Calamari server RPM packages. The next step is to build calamari client packages.

Building Calamari client packages

The calamari client packages provides the GUI to calamari dashboard. As calamari build process is not greatly matured , we need to use ubuntu vagrant to build client packages. This will generate a tarball of client packages , that you can use with centos machines.

CentOS / RHEL users : Note , this will not generate any RPM package and you would need to user tarball.

Ubuntu Users: Cheers , you will get DEB packages that you can use directly.

1. Clone calamari client from github on to your workstation ( the same directory as you were using before ) , and perform vagrant for ubuntu precise VM

CentOS / RHEL users : Don't panic , you also have to user ubuntu precise vagrant instance in order to build Calamari client tarball.

Ubuntu Users : No comment for you , please proceed :-)

$ cd /tmp/calamari-repo/
$
$ git clone https://github.com/ceph/calamari-clients.git
Cloning into 'calamari-clients'...
remote: Counting objects: 16261, done.
remote: Total 16261 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (16261/16261), 28.81 MiB | 4.01 MiB/s, done.
Resolving deltas: 100% (9441/9441), done.
Checking connectivity... done.

$ ls -la
total 61960
drwxr-xr-x   8 ksingh  wheel       272 Sep 17 12:03 .
drwxrwxrwt  10 root    wheel       340 Sep 17 11:46 ..
drwxr-xr-x  28 ksingh  wheel       952 Sep 17 10:35 Diamond
drwxr-xr-x  32 ksingh  wheel      1088 Sep 17 10:34 calamari
drwxr-xr-x  22 ksingh  wheel       748 Sep 17 12:04 calamari-clients
-rw-r--r--   1 ksingh  wheel   7038999 Sep 17 11:18 calamari-repo-el6.tar.gz
-rw-r--r--   1 ksingh  wheel  24082060 Sep 17 11:18 calamari-server-1.2.1-29_g9d02dc3.el6.x86_64.rpm
-rw-r--r--   1 ksingh  wheel    595548 Sep 17 11:18 diamond-3.4.67-0.noarch.rpm

$ cd calamari-clients/vagrant/precise-build/
$
$ ls -la
total 16
drwxr-xr-x   4 ksingh  wheel  136 Sep 17 12:04 .
drwxr-xr-x  10 ksingh  wheel  340 Sep 17 12:04 ..
-rw-r--r--   1 ksingh  wheel  789 Sep 17 12:04 README.rst
-rw-r--r--   1 ksingh  wheel  882 Sep 17 12:04 Vagrantfile
$

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'precise64'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: precise-build_default_1410947713821_66264
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 => 2202 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2202
    default: SSH username: vagrant
    default: SSH auth method: private key
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: The guest additions on this VM do not match the installed version of
    default: VirtualBox! In most cases this is fine, but in rare cases it can
    default: prevent things such as shared folders from working properly. If you see
    default: shared folder errors, please make sure the guest additions within the
    default: virtual machine match the version of VirtualBox you have installed on
    default: your host and reload your VM.
    default:
    default: Guest Additions Version: 4.2.0
    default: VirtualBox Version: 4.3
==> default: Mounting shared folders...
    default: /git => /private/tmp/calamari-repo
    default: /vagrant => /private/tmp/calamari-repo/calamari-clients/vagrant/precise-build
    default: /srv/salt => /private/tmp/calamari-repo/calamari-clients/vagrant/salt/roots
==> default: Running provisioner: salt...
Copying salt minion config to vm.
Checking if salt-minion is installed
salt-minion was not found.
Checking if salt-call is installed
salt-call was not found.
Bootstrapping Salt... (this may take a while)
Salt successfully configured and installed!
run_overstate set to false. Not running state.overstate.
run_highstate set to false. Not running state.highstate.
$

$ vagrant ssh
Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
New release '14.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Welcome to your Vagrant-built virtual machine.
Last login: Fri Sep 14 06:23:18 2012 from 10.0.2.2
vagrant@precise64:~$
vagrant@precise64:~$
vagrant@precise64:~$ df -h
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/precise64-root   79G  2.3G   73G   4% /
udev                        489M  4.0K  489M   1% /dev
tmpfs                       200M  276K  199M   1% /run
none                        5.0M     0  5.0M   0% /run/lock
none                        498M     0  498M   0% /run/shm
/dev/sda1                   228M   25M  192M  12% /boot
git                         233G  216G   17G  93% /git
vagrant                     233G  216G   17G  93% /vagrant
srv_salt                    233G  216G   17G  93% /srv/salt
vagrant@precise64:~$

2. Install the following dependencies using either aptitude to apt-get

vagrant@precise64:~$ sudo apt-get install ruby1.9.1 ruby1.9.1-dev python-software-properties g++ make git debhelper build-essential devscripts

3. Install node js from PPA

$ sudo apt-add-repository http://ppa.launchpad.net/chris-lea/node.js/ubuntu
$ sudo apt-get update
$ sudo apt-get install nodejs
$ sudo npm install -g bower@1.3.8
$ sudo npm install -g grunt-cli
$ sudo gem install compass

4. Start the build process for Calamari client DEB packages / tarball ( centos / rhel)

vagrant@precise64:~$ sudo salt-call state.highstate
[INFO    ] Loading fresh modules for state activity
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/modules'
[INFO    ] Syncing modules for environment 'base'
[INFO    ] Loading cache from salt://_modules, for base)
[INFO    ] Caching directory '_modules/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/states'
[INFO    ] Syncing states for environment 'base'
[INFO    ] Loading cache from salt://_states, for base)

................... OUTPUT TRUNCATED ..........................

----------
          ID: copyout_build_product
    Function: cmd.run
        Name: cp calamari-clients*tar.gz /git/
      Result: True
     Comment: Command "cp calamari-clients*tar.gz /git/" run
     Changes:
              ----------
              pid:
                  25090
              retcode:
                  0
              stderr:

              stdout:


Summary
-------------
Succeeded: 13
Failed:     0
-------------
Total:     13
vagrant@precise64:~$

You can follow the entire output at http://paste.ubuntu.com/8364367/

5. Check the shared filesystem and read my comments against the output.

vagrant@precise64:~$ df -h
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/precise64-root   79G  3.8G   72G   6% /
udev                        489M  4.0K  489M   1% /dev
tmpfs                       200M  276K  199M   1% /run
none                        5.0M     0  5.0M   0% /run/lock
none                        498M     0  498M   0% /run/shm
/dev/sda1                   228M   25M  192M  12% /boot
git                         233G  219G   15G  94% /git
vagrant                     233G  219G   15G  94% /vagrant
srv_salt                    233G  219G   15G  94% /srv/salt
vagrant@precise64:~$
vagrant@precise64:~$ cd /git/
vagrant@precise64:/git$ ll
total 34324
drwxr-xr-x  1 vagrant vagrant      340 Sep 17 10:37 ./
drwxr-xr-x 25 root    root        4096 Sep 17 09:55 ../
drwxr-xr-x  1 vagrant vagrant     1088 Sep 17 07:34 calamari/                                                                               # git clone of calamari server
drwxr-xr-x  1 vagrant vagrant      748 Sep 17 09:04 calamari-clients/                                                                     # git clone of calamari clients
-rw-r--r--  1 vagrant vagrant  1705158 Sep 17 10:37 calamari-clients_1.2.1-27-g9eb09d5_all.deb                        # package for installation on calamari server if OS is ubuntu
-rw-r--r--  1 vagrant vagrant  1711183 Sep 17 10:37 calamari-clients-build-output.tar.gz  # unpackaged build of calamari-clients , to be added on calamari server if the OS is CentOS
-rw-r--r--  1 vagrant vagrant  7038999 Sep 17 08:18 calamari-repo-el6.tar.gz
-rw-r--r--  1 vagrant vagrant 24082060 Sep 17 08:18 calamari-server-1.2.1-29_g9d02dc3.el6.x86_64.rpm          # package for installation on calamari server
drwxr-xr-x  1 vagrant vagrant      952 Sep 17 07:35 Diamond/                                                                                # git clone of diamond
-rw-r--r--  1 vagrant vagrant   595548 Sep 17 08:18 diamond-3.4.67-0.noarch.rpm                                                # diamond package for use on ceph nodes
vagrant@precise64:/git$

6. At this stage you have Calamari client DEB packages and tarball calamari-clients-build-output.tar.gz

Deploying Calamari

Deploying calamari is again a multi step process.

#1 Install Calamari server packages

#2 Add calamari client web app contents to Calamari server

#3 Install diamond and salt-minion on Ceph nodes , that you wish to monitor using Calamari dashboard.

Installing Calamari Server

Copy the following packages that we have generated in the earlier steps from the workstation to the server that you intend to configure as calamari master.

calamari-server-1.2.1-29_g9d02dc3.el6.x86_64.rpm
diamond-3.4.67-0.noarch.rpm
calamari-clients-build-output.tar.gz

Ubuntu Users : Copy the DEB packages of calamari server and client.

1. Instal the packages on the node that will be your Calamari master.

$ sudo yum install -y calamari-server-1.2.1-37_g6f353e6.el6.x86_64.rpm
$ sudo yum install -y diamond-3.4.67-0.noarch.rpm

2. Uncompress , untar the files and move it to /opt/calamari/webapp of calamari master . These files provides GUI facility to calamari webapp

$ sudo gunzip calamari-clients-build-output.tar.gz
$ sudo tar -xvf calamari-clients-build-output.tar

$ cd /tmp/opt/calamari/webapp
$ sudo cp -rp content/ /opt/calamari/webapp/content
$ sudo  ls -l /opt/calamari/webapp/content
total 16
drwxr-xr-x. 7 1000 1000 4096 Sep 23 09:28 admin
drwxr-xr-x. 7 1000 1000 4096 Sep 23 09:28 dashboard
drwxr-xr-x. 7 1000 1000 4096 Sep 23 09:28 login
drwxr-xr-x. 8 1000 1000 4096 Sep 23 09:28 manage

3. At this point your calamari server is ready , you just need to initialize calamari by running

$ sudo calamari-ctl initialize

[ceph@ceph-node4 calamari]$ sudo calamari-ctl initialize
[INFO] Loading configuration..
[INFO] Starting/enabling salt...
[INFO] Starting/enabling postgres...
[INFO] Initializing database...
[INFO] Initializing web interface...
[INFO] You will now be prompted for login details for the administrative user account.  This is the account you will use to log into the web interface once setup is complete.
Username (leave blank to use 'root'):
Email address: karan.singh@csc.fi
Password:
Password (again):
Superuser created successfully.
[INFO] Starting/enabling services...
[INFO] Restarting services...

.................................. Output Truncated ...............

This username and password would be required to login to Calamari dashboard.

4. Openup browser and navigate to the IP address of your calamari server. You would see calamari dashboard login page , provide the login details that we have set in the last step.

5 Calamari would not be able to find the Ceph cluster and will ask to add a cluster , for this we need to add Ceph clients to dashboard by installing salt-minion and diamond packages on them.

Adding your Ceph cluster nodes to Calamari Dashboard

Perform the following steps on all your Ceph cluster nodes, inorder to add them to Calamari dashboard.

1. Install diamond packages that we have build already ( previous steps )

$ sudo yum install -y diamond-3.4.67-0.noarch.rpm

2. Create a default diamond configuration file.

$ sudo mv /etc/diamond/diamond.conf.example /etc/diamond/diamond.conf

3. Install salt-minion packages

$ sudo yum install -y salt-minion

$ sudo chkconfig salt-minion on

Ubuntu Users : Use ubuntu package manager for installing salt-minion

4. Configure salt-minion , so that it can reach to Calamari master.

Edit /etc/salt/minion and add your calamari server hostname.

master: ceph-node4

5. Restart services

$ sudo service salt-minion restart

$ sudo service diamond restart

6. Repeat these steps for all the Ceph cluster nodes

Authorize Ceph cluster to add in calamari

Once you configure all the Ceph cluster nodes with salt-minion and diamond, you need to accept salt-keys from calamari master

1. From calamari master check for unaccepted salt-keys.

$ sudo salt-key -L

$ sudo salt-key -L
Accepted Keys:
Unaccepted Keys:
ceph-node1
ceph-node2
ceph-node3
Rejected Keys:
$

2. Accept salt-keys

$ sudo salt-key -A
The following keys are going to be accepted:
Unaccepted Keys:
ceph-node1
ceph-node2
ceph-node3
Proceed? [n/Y] y
Key for minion ceph-node1 accepted.
Key for minion ceph-node2 accepted.
Key for minion ceph-node3 accepted.

$ sudo salt-key -L
Accepted Keys:
ceph-node1
ceph-node2
ceph-node3
Unaccepted Keys:
Rejected Keys:
$

3. Finally check calamari dashboard , you should be able to view the Ceph cluster.

Erasure Coding in Ceph

noreply@blogger.com (Anonymous) — Tue, 08 Apr 2014 16:53:00 +0000

Erasure Coding : All you have to know

If there is data , there would be failure and there will also be administrators like us to recover this data and Erasure Coding is our shield.

Storage systems have technologies for data protection and recovery in event of failure. There are ~~two~~ three main methods of data protection :-

1) Replication

2) RAID

Hey wait a minute , here comes the LEGEN ( wait for it ... ) DARY : Legendary number 3

3) Erasure Coding

First of all Erasure Coding is not now , it was born before RAID and Replication methods but it was not in practical use. Raid is now in its concluding years. Raid method of data protection was built years and years back when it was difficult to find a HDD with capacity of 1 GB . In the period of last 30 years drive capacity has been increased 4000 folds. With this appreciation drive consistency and durability is still a problem. Raid was not supposed to give you data protection for a exabyte level storage array , its simply not designed for Future. We need a data protection method which is highly adaptive and ready for future needs. Back to basics Erasure Codes is the answer.

Erasure coding (EC) is a method of data protection in which data is broken into fragments , encoded and then storage in a distributed manner. Ceph , due to its distributed nature , makes use of EC beautifully.

Erasure coding makes use of a mathematical equation to achieve data protection. The entire concept revolves around the following equation.

n = k + m where ,

k = The number of chunks original data divided into.

m = The extra codes added to original data chunks to provide data protection. For ease of understanding you can considered it as reliability level.

n = The total number of chunks created after erasure coding process.

In continuation to erasure coding equation , there are couple of more terms which are :-

Recovery : To perform recovery operation we would require any k chunks out of n chunks and thus can tolerate failure of any m chunks

Reliability Level : We can tolerate failure upto any m chunks.

Encoding Rate (r) : r = k / n , where r < 1

Storage Required : 1 / r

Example 1 : (3,5) Erasure Code for any data file would look like

n = 5 , k = 3 and m = 2 ( m = n - k )

Erasure coding equation 5 = 3 + 2

So 2 coded chunks will be added to 3 data chunks to form 5 total chunks that will be stored distributedly in ceph cluster. In an event of failure , to construct the original file , we need any 3 chunks out of these 5 chunks to recover it. Hence in this example we can manage loss of any 2 chunks.

Encoding rate (r) = 3 / 5 = 0.6 < 1
Storage Required = 1 / 0.6 = 1.6 times of original file.

If the original file size is 1GB then to store this file in a ceph cluster erasure coded (3,5) pool , you would need 1.6GB of storage space.

Example 2 : (10,6) Erasure Code for any data file would look like

n = 16 , k = 10 , m = 6

Encoding rate (r) = 10 / 16 = 0.6 < 1

Storage Required = 1 / 0.6 = 1.6 times of original file.

This erasure code can sustain failure of any 6 chunks out of 16 chunks , in an event of failure for recovery we would need any 10 chunks out of 16 chunks.

Erasure Codes Step - by - Step

Step 1 : Understanding EC profile

When you look at EC profile , you would come across few options that defines EC pattern for that profile.

k ===> As explained earlier , its the number of chunks data will be divided into and each chunk will be stored on a separate OSD. The equivalent command line parameter for this is
erasure-code-k=<Number_of_data_chunks>
Default value = 2

m ===> The number of extra codes (chunks) added to data chunks to provide reliability. This is also the number of OSD that can go down without losing data. Also know as reliability level. The equivalent command line parameter for this is
erasure-code-m=<Number_of_coding_chunks>
Default value = 1

plugin ===> This is the library facilitation erasure coding in Ceph. The plugin is used to compute the coding chunks and recover missing chunks. The current implementation uses jerasure but the upcoming version might use GF-complete as its twice as faster as jerasure.The equivalent command line parameter for this is
erasure-code-plugin=<plugin_name>
Default value = 1

directory ===> The directory name from where EC plugin library will loaded from. In most of the cases this parameter is automatically added once you define plugin name .The equivalent command line parameter for this is
erasure-code-directory=<directory_path>
Default value = /usr/lib64/ceph/erasure-code

ruleset-failure-domain ===> Defines the failure domain , you should set it as OSD to get better reliability for your data.
Default value = host

Step 2 : Generating a EC profile

# ceph osd erasure-code-profile set EC-temp-pool

# ceph osd erasure-code-profile ls
EC-temp-pool
profile1
#

# ceph osd erasure-code-profile get EC-temp-pool
directory=/usr/lib64/ceph/erasure-code
k=2
m=1
plugin=jerasure
technique=reed_sol_van
#

Step 3 : Customizing your EC profile

We can use sub command set , to change existing profile , however we might need to user --force to change a profile. This can be a dangerous.

In this example we will change value of k=4

# ceph osd erasure-code-profile set EC-temp-pool ruleset-failure-domain=osd k=4 m=2
Error EPERM: will not override erasure code profile EC-temp-pool

# ceph osd erasure-code-profile set EC-temp-pool ruleset-failure-domain=osd k=4 m=2 --force

# ceph osd erasure-code-profile get EC-temp-pool
directory=/usr/lib64/ceph/erasure-code
k=4
m=2
plugin=jerasure
ruleset-failure-domain=osd
technique=reed_sol_van
#

Similarly you can change other parameter based on your needs.

Step 4 : Creating a EC Ceph Pool based on your profile

Ceph osd pool create <Pool_name> <pg_num> <pgp_num> erasure <EC_profile_name>

# ceph osd pool create ECtemppool 128 128 erasure EC-temp-pool
pool 'ECtemppool' created
#

# rados lspools
data
metadata
rbd
ECtemppool
#

# ceph osd dump | grep -i erasure
pool 22 'ECtemppool' erasure size 6 min_size 2 crush_ruleset 1 object_hash rjenkins pg_num 128 pgp_num 128 last_change 2034 owner 0 flags hashpspool stripe_width 4096
#

Step 5 : Writing your EC pool

Create a temporary file

# echo "Erasure Coding is ******* Awesome" > testfile
#
# cat testfile
Erasure Coding is ******* Awesome
#

List your pool for objects , since you have created a new pool , there would be no objects inside it. Put the test file to your EC pool.

# rados -p ECtemppool ls
#
# rados -p ECtemppool put object.1 testfile
#
# rados -p ECtemppool ls
object.1
#

Check PG map for your pool , you would get to know on how many OSD this object.1 has been placed.
If you analyze carefully , you can find that the object.1 has been placed in 6 different OSD , this is due to our erasure code profile where we have mentioned k=4 and m=2 , so total chunks are 6 which will be placed on all different OSD

# ceph osd map ECtemppool object.1
osdmap e2067 pool 'ECtemppool' (22) object 'object.1' -> pg 22.f560f2ec (22.2ec) -> up ([144,39,55,15,123,65], p144) acting ([144,39,55,15,123,65], p144)
#

Now you are a reliable erasure coded pool that can sustain failure of two OSD simultaneously.

Erasure Codes are 100% Authentic :-

Upto this point you have a erasure coded pool (4,2) , meaning it can sustain simultaneously 2 OSD failure.

From the above output these are the 6 OSD on which the file "testfile" has been spread = [144,39,55,15,123,65]

Test 1 : Intentionally Breaking OSD to test EC reliability

Breaking 1st OSD ( osd.65 )

# service ceph status osd.65
=== osd.65 ===
osd.65: running {"version":"0.79-125-gcf69bdb"}
#
#
# service ceph stop osd.65
=== osd.65 ===
Stopping Ceph osd.65 on storage0106-ib...kill 3505...done
#
# service ceph status osd.65
=== osd.65 ===
osd.65: not running.
#

Since osd.65 is down , it would be missing from following output , rather you would find some garbage entry " 2147483647 "

# ceph osd map ECtemppool object.1
osdmap e2069 pool 'ECtemppool' (22) object 'object.1' -> pg 22.f560f2ec (22.2ec) -> up ([144,39,55,15,123,2147483647], p144) acting ([144,39,55,15,123,2147483647], p144)
#

Breaking 2nd OSD ( osd.123 )

#
# service ceph status osd.123
=== osd.123 ===
osd.123: running {"version":"0.79-125-gcf69bdb"}
#
# service ceph stop  osd.123
=== osd.123 ===
Stopping Ceph osd.123 on storage0114-ib...kill 5327...done
#
# service ceph status  osd.123
=== osd.123 ===
osd.123: not running.
#

Now the second osd.123 is down , it would be missing from following output , like osd.65 but here comes the MAGIC
Erasure codings are intelligent , it know when you lose data or coding chunks of a file. As soon as chunks are lost , it immediately create exactly same on to new OSD. In this example you can see , OSD.65 and OSD.123 went down , so ceph intelligently recovers the failed chunk on to OSD.85 ( which is a new OSD )

# ceph osd map ECtemppool object.1
osdmap e2104 pool 'ECtemppool' (22) object 'object.1' -> pg 22.f560f2ec (22.2ec) -> up ([144,39,55,15,2147483647,85], p144) acting ([144,39,55,15,2147483647,85], p144)
#

Summary :

Erasure coding is a beautiful piece of reliable solution and works amazingly well, it intelligently manages the failed chunks and tries to recover them wherever possible without any administrative intervention. This makes ceph MORE and MORE reliable , amazingly cost effective. EC feature in ceph has been recently added starting 0.78 release , we can definitely expect more stable and performance centric version of EC in Ceph Firefly 0.80. Stay tuned.

Admin Guide :: Replacing a Failed Disk in a Ceph Cluster

noreply@blogger.com (Anonymous) — Sat, 08 Mar 2014 20:21:00 +0000

Replacing a Failed Disk from Ceph a Cluster

Do you have a ceph cluster , great , you are awesome ; so very soon you would face this .

Check your cluster health

# ceph status
  cluster c452b7df-0c0b-4005-8feb-fc3bb92407f5

   health HEALTH_WARN 6 pgs peering; 6 pgs stale; 6 pgs stuck inactive; 6 pgs stuck stale; 6 pgs stuck unclean; 2 requests are blocked > 32 sec

   monmap e6: 3 mons at {node01-ib=10.168.100.101:6789/0,node06-ib=10.168.100.106:6789/0,node11-ib=10.168.100.111:6789/0}, election epoch 2

830, quorum 0,1,2 node01-ib,node06-ib,node11-ib

   mdsmap e58: 1/1/1 up {0=node01-ib=up:active}

   osdmap e8871: 153 osds: 152 up, 152 in

   pgmap v1409465: 66256 pgs, 30 pools, 201 TB data, 51906 kobjects

      90439 GB used, 316 TB / 413 TB avail

        66250 active+clean

          6 stale+peering

#ceph osd tree | grep -i down

9 2.63 osd.9 up 1

17 2.73 osd.17 up 1

30 2.73 osd.30 up 1

53 2.73 osd.53 up 1

65 2.73 osd.65 up 1

78 2.73 osd.78 up 1

89 2.73 osd.89 up 1

99 2.73 osd.99 down 0

113 2.73 osd.113 up 1

128 2.73 osd.128 up 1

141 2.73 osd.141 up 1

Now you have identified which node's OSD is failed and what's the OSD number. Login to that node and check if that OSD is mounted , IT would be NOT ( as its failed )

# df -h

Filesystem      Size Used Avail Use% Mounted on

/dev/sda1       47G 6.0G 38G 14% /

tmpfs         12G  0 12G 0% /dev/shm

/dev/sdd1      2.8T 197G 2.5T 8% /var/lib/ceph/osd/ceph-30

/dev/sde1      2.8T 172G 2.6T 7% /var/lib/ceph/osd/ceph-53

/dev/sdc1      2.8T 264G 2.5T 10% /var/lib/ceph/osd/ceph-17

/dev/sdh1      2.8T 227G 2.5T 9% /var/lib/ceph/osd/ceph-89

/dev/sdf1      2.8T 169G 2.6T 7% /var/lib/ceph/osd/ceph-65

/dev/sdi1      2.8T 150G 2.6T 6% /var/lib/ceph/osd/ceph-113

/dev/sdb1      2.7T 1.3T 1.3T 51% /var/lib/ceph/osd/ceph-9

/dev/sdj1      2.8T 1.6T 1.2T 58% /var/lib/ceph/osd/ceph-128

/dev/sdg1      2.8T 237G 2.5T 9% /var/lib/ceph/osd/ceph-78

/dev/sdk1      2.8T 1.5T 1.3T 53% /var/lib/ceph/osd/ceph-141

Go to your datacenter with a new physical drive and replace the drive physically , i assume depending on enterprise server that you are using it should be hot swappable , These days almost all servers support hot swapping of disks , but still you should check for your server model . In this example i was using HPDL380 server.
Once you have replaced the drive physically , wait for some time so that new drive gets stable.
Login to your node and make OSD out from cluster . Please remember , the OSD was already DOWN and OUT as soon as disk was failed . Ceph takes care of OSD and if its not available it marks it down and moves it out of cluster.

# ceph osd out osd.99

osd.99 is already out.

#

# service ceph stop osd.99

/etc/init.d/ceph: osd.99 not found (/etc/ceph/ceph.conf defines osd.9 osd.30 osd.17 osd.128 osd.65 osd.141 osd.89 osd.53 osd.113 osd.78 , /var/lib/ceph defines osd.9 osd.30 osd.17 osd.128 osd.65 osd.141 osd.89 osd.53 osd.113 osd.78)

#

# service ceph status osd.99

/etc/init.d/ceph: osd.99 not found (/etc/ceph/ceph.conf defines osd.9 osd.30 osd.17 osd.128 osd.65 osd.141 osd.89 osd.53 osd.113 osd.78 , /var/lib/ceph defines osd.9 osd.30 osd.17 osd.128 osd.65 osd.141 osd.89 osd.53 osd.113 osd.78)

#

# service ceph status

=== osd.9 ===

osd.9: running {"version":"0.72.1"}

=== osd.30 ===

osd.30: running {"version":"0.72.1"}

=== osd.17 ===

osd.17: running {"version":"0.72.1"}

=== osd.128 ===

osd.128: running {"version":"0.72.1"}

=== osd.65 ===

osd.65: running {"version":"0.72.1"}

=== osd.141 ===

osd.141: running {"version":"0.72.1"}

=== osd.89 ===

osd.89: running {"version":"0.72.1"}

=== osd.53 ===

osd.53: running {"version":"0.72.1"}

=== osd.113 ===

osd.113: running {"version":"0.72.1"}

=== osd.78 ===

osd.78: running {"version":"0.72.1"}

#

Now remove this failed OSD from Crush Map , as soon as its removed from crush map , ceph starts making PG copies that were located on this failed disk and it places these PG on other disks. So a recovery process will start.

# ceph osd crush remove osd.99

removed item id 99 name 'osd.99' from crush map

# ceph status

  cluster c452b7df-0c0b-4005-8feb-fc3bb92407f5

   health HEALTH_WARN 43 pgs backfill; 56 pgs backfilling; 9 pgs peering; 82 pgs recovering; 6 pgs stale; 6 pgs stuck inactive; 6 pgs stuck stale; 192 pgs st

uck unclean; 4 requests are blocked > 32 sec; recovery 373488/106903578 objects degraded (0.349%)

   monmap e6: 3 mons at {node01-ib=10.168.100.101:6789/0,node06-ib=10.168.100.106:6789/0,node11-ib=10.168.100.111:6789/0}, election epoch 2

836, quorum 0,1,2 node01-ib,node06-ib,node11-ib

   mdsmap e58: 1/1/1 up {0=node01-ib=up:active}

   osdmap e8946: 153 osds: 152 up, 152 in

   pgmap v1409604: 66256 pgs, 30 pools, 201 TB data, 51916 kobjects

   1 
      1 GB used, 316 TB / 413 TB avail

   1 
      1 /106903578 objects degraded (0.349%)

          1 active

        66060 active+clean

   1 
      1 
         1 active+remapped+wait_backfill

          3 peering

          1 active+remapped

   1 
      1 
         1 active+remapped+backfilling

          6 stale+peering

          4 active+clean+scrubbing+deep

   1 
      1 
         1 active+recovering

recovery io 159 MB/s, 39 objects/s

(optional) Check disk statistics , it looks nice and after some times ( depending on data that your FAILED disk holds ) it completes

# dstat 10

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--

usr sys idl wai hiq siq| read writ| recv send| in out | int csw

 2 3 95 1 0 0|2223k 5938k| 0  0 |1090B 2425B|5853  11k

 14 58 1 25 0 2| 130M 627M| 219M 57M|6554B  0 | 28k 111k

 14 57 1 26 0 2| 106M 743M| 345M 32M| 0 4096B| 35k 73k

 13 61 1 23 0 2| 138M 680M| 266M 67M| 83k  0 | 31k 82k

 14 52 1 31 0 2| 99M 574M| 230M 32M| 48k 6963B| 27k 78k

 14 51 2 31 0 2| 99M 609M| 291M 31M| 0  0 | 29k 83k

 11 57 1 28 0 2| 118M 636M| 214M 57M|9830B  0 | 26k 92k

 12 49 4 34 0 1| 97M 432M| 166M 48M| 35k  0 | 22k 100k

 13 44 3 38 0 1| 95M 422M| 183M 46M| 0  0 | 22k 88k

 13 52 3 30 0 2| 96M 510M| 207M 44M| 0  0 | 25k 109k

 14 49 3 32 0 2| 96M 568M| 276M 37M| 16k  0 | 27k 72k

 9 54 4 31 0 2| 109M 520M| 136M 45M| 0  0 | 20k 89k

 14 44 5 35 0 1| 76M 444M| 192M 13M| 0  0 | 22k 54k

 15 47 3 34 0 2| 101M 452M| 141M 20M|3277B 13k| 21k 79k

 17 48 3 31 0 1| 108M 445M| 181M 16M| 0 200k| 23k 69k

 17 48 3 30 0 1| 154M 406M| 138M 23M| 0  0 | 21k 75k

 17 53 3 27 0 1| 169M 399M| 115M 23M| 0 396k| 21k 81k

 13 45 4 36 0 1| 161M 330M| 131M 20M| 0 397k| 20k 90k

 11 51 5 33 0 1| 116M 416M| 145M 1177k| 0 184k| 20k 69k

 14 50 4 31 0 1| 144M 376M| 124M 8752k| 0  0 | 20k 72k

 14 42 6 37 0 1| 142M 340M| 138M 19M| 0  0 | 19k 79k

 15 47 6 32 0 1| 111M 427M| 129M 11M| 0 819B| 19k 66k

 15 50 5 29 0 1| 163M 413M| 139M 5709k| 58k  0 | 20k 90k

 14 49 4 32 0 1| 155M 395M| 91M 12M| 0  0 | 18k 93k

 18 43 7 31 0 1| 166M 338M| 84M 6493k| 0  0 | 17k 81k

 14 49 5 32 0 1| 179M 335M| 98M 3824k| 0  0 | 18k 91k

 13 46 9 31 0 1| 157M 299M| 72M 14M| 0  0 | 17k 125k

 17 42 9 30 0 1| 188M 269M| 82M 11M| 16k  0 | 16k 102k

 22 35 15 27 0 1| 158M 167M|8932k 287k| 0  0 | 13k 88k

 7 20 46 26 0 0| 118M 12M| 250k 392k| 0  82k|9333  61k

 7 17 60 16 0 0| 124M 1638B| 236k 225k| 0  0 |7512  64k

 7 16 63 14 0 0| 117M 1005k| 247k 238k| 0  0 |7429  60k

 3 9 82 5 0 0| 41M 17M| 225k 225k| 0  0 |6049  27k

 4 8 81 7 0 0| 56M 7782B| 227k 225k| 0 6144B|5933  33k

 4 9 79 7 0 0| 60M 9011B| 248k 245k| 0 9011B|6457  36k

 4 9 79 7 0 0| 58M 236k| 231k 230k| 0  14k|6210  35k

# ceph status

  cluster c452b7df-0c0b-4005-8feb-fc3bb92407f5

   health HEALTH_WARN 6 pgs peering; 6 pgs stale; 6 pgs stuck inactive; 6 pgs stuck stale; 6 pgs stuck unclean; 2 requests are blocked > 32 sec

   monmap e6: 3 mons at {node01-ib=10.168.100.101:6789/0,node06-ib=10.168.100.106:6789/0,node11-ib=10.168.100.111:6789/0}, election epoch 2

836, quorum 0,1,2 node01-ib,node06-ib,node11-ib

   mdsmap e58: 1/1/1 up {0=node01-ib=up:active}

   osdmap e9045: 153 osds: 152 up, 152 in

   pgmap v1409957: 66256 pgs, 30 pools, 200 TB data, 51705 kobjects

      90448 GB used, 316 TB / 413 TB avail

        66250 active+clean

          6 stale+peering

Once data recovery is done , go ahead delete keyrings for that OSD and finally remove OSD

# ceph auth del osd.99

updated

#

# ceph osd rm osd.99

removed osd.99

#

#ceph status 

  cluster c452b7df-0c0b-4005-8feb-fc3bb92407f5

   health HEALTH_WARN 6 pgs peering; 6 pgs stale; 6 pgs stuck inactive; 6 pgs stuck stale; 6 pgs stuck unclean; 2 requests are blocked > 32 sec

   monmap e6: 3 mons at {node01-ib=10.168.100.101:6789/0,node06-ib=10.168.100.106:6789/0,node11-ib=10.168.100.111:6789/0}, election epoch 2

836, quorum 0,1,2 node01-ib,node06-ib,node11-ib

   mdsmap e58: 1/1/1 up {0=node01-ib=up:active}

   osdmap e9046: 152 osds: 152 up, 152 in

   pgmap v1409971: 66256 pgs, 30 pools, 200 TB data, 51705 kobjects

      90445 GB used, 316 TB / 413 TB avail

        66250 active+clean

          6 stale+peering

Remove entry of this OSD from ceph.conf ( if its present ) , make sure you keep all the nodes ceph.conf file updated . You can push the new configuration file to entire cluster using # ceph admin command.
Time to create new OSD for the physical disk that we have inserted , You would see, ceph will create new OSD with the same OSD number , that was failed , as we have removed failed OSD cleanly , if you see a different OSD number , it means that you have not cleanly removed failed OSD.

# ceph osd create

99

# ceph status

  cluster c452b7df-0c0b-4005-8feb-fc3bb92407f5

   health HEALTH_WARN 6 pgs peering; 6 pgs stale; 6 pgs stuck inactive; 6 pgs stuck stale; 6 pgs stuck unclean; 2 requests are blocked > 32 sec

   monmap e6: 3 mons at {node01-ib=10.168.100.101:6789/0,node06-ib=10.168.100.106:6789/0,node11-ib=10.168.100.111:6789/0}, election epoch 2

836, quorum 0,1,2 node01-ib,node06-ib,node11-ib

   mdsmap e58: 1/1/1 up {0=node01-ib=up:active}

   osdmap e9047: 153 osds: 152 up, 152 in

   pgmap v1409988: 66256 pgs, 30 pools, 200 TB data, 51705 kobjects

      90442 GB used, 316 TB / 413 TB avail

        66250 active+clean

          6 stale+peering

#

List the disk , zap it and deploy it again

# ceph-deploy disk list node14

[ceph_deploy.cli][INFO ] Invoked (1.3.2): /usr/bin/ceph-deploy disk list node14

[node14][DEBUG ] connected to host: node14

[node14][DEBUG ] detect platform information from remote host

[node14][DEBUG ] detect machine type

[ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final

[ceph_deploy.osd][DEBUG ] Listing disks on node14...

[node14][INFO ] Running command: ceph-disk list

[node14][DEBUG ] /dev/sda :

[node14][DEBUG ] /dev/sda1 other, ext4, mounted on /

[node14][DEBUG ] /dev/sda2 swap, swap

[node14][DEBUG ] /dev/sdb :

[node14][DEBUG ] /dev/sdb1 ceph data, active, cluster ceph, osd.9, journal /dev/sdb2

[node14][DEBUG ] /dev/sdb2 ceph journal, for /dev/sdb1

[node14][DEBUG ] /dev/sdc :

[node14][DEBUG ] /dev/sdc1 ceph data, active, cluster ceph, osd.17, journal /dev/sdc2

[node14][DEBUG ] /dev/sdc2 ceph journal, for /dev/sdc1

[node14][DEBUG ] /dev/sdd :

[node14][DEBUG ] /dev/sdd1 ceph data, active, cluster ceph, osd.30, journal /dev/sdd2

[node14][DEBUG ] /dev/sdd2 ceph journal, for /dev/sdd1

[node14][DEBUG ] /dev/sde :

[node14][DEBUG ] /dev/sde1 ceph data, active, cluster ceph, osd.53, journal /dev/sde2

[node14][DEBUG ] /dev/sde2 ceph journal, for /dev/sde1

[node14][DEBUG ] /dev/sdf :

[node14][DEBUG ] /dev/sdf1 ceph data, active, cluster ceph, osd.65, journal /dev/sdf2

[node14][DEBUG ] /dev/sdf2 ceph journal, for /dev/sdf1

[node14][DEBUG ] /dev/sdg :

[node14][DEBUG ] /dev/sdg1 ceph data, active, cluster ceph, osd.78, journal /dev/sdg2

[node14][DEBUG ] /dev/sdg2 ceph journal, for /dev/sdg1

[node14][DEBUG ] /dev/sdh :

[node14][DEBUG ] /dev/sdh1 ceph data, active, cluster ceph, osd.89, journal /dev/sdh2

[node14][DEBUG ] /dev/sdh2 ceph journal, for /dev/sdh1

[node14][DEBUG ] /dev/sdi other, btrfs

[node14][DEBUG ] /dev/sdj :

[node14][DEBUG ] /dev/sdj1 ceph data, active, cluster ceph, osd.113, journal /dev/sdj2

[node14][DEBUG ] /dev/sdj2 ceph journal, for /dev/sdj1

[node14][DEBUG ] /dev/sdk :

[node14][DEBUG ] /dev/sdk1 ceph data, active, cluster ceph, osd.128, journal /dev/sdk2

[node14][DEBUG ] /dev/sdk2 ceph journal, for /dev/sdk1

[node14][DEBUG ] /dev/sdl :

[node14][DEBUG ] /dev/sdl1 ceph data, active, cluster ceph, osd.141, journal /dev/sdl2

[node14][DEBUG ] /dev/sdl2 ceph journal, for /dev/sdl1

# ceph-deploy disk zap node14:sdi

[ceph_deploy.cli][INFO ] Invoked (1.3.2): /usr/bin/ceph-deploy disk zap node14:sdi

[ceph_deploy.osd][DEBUG ] zapping /dev/sdi on node14

[node14][DEBUG ] connected to host: node14

[node14][DEBUG ] detect platform information from remote host

[node14][DEBUG ] detect machine type

[ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final

[node14][DEBUG ] zeroing last few blocks of device

[node14][INFO ] Running command: sgdisk --zap-all --clear --mbrtogpt -- /dev/sdi

[node14][DEBUG ] Creating new GPT entries.

[node14][DEBUG ] GPT data structures destroyed! You may now partition the disk using fdisk or

[node14][DEBUG ] other utilities.

[node14][DEBUG ] The operation has completed successfully.

#

# ceph-deploy --overwrite-conf osd prepare node14:sdi

[ceph_deploy.cli][INFO ] Invoked (1.3.2): /usr/bin/ceph-deploy --overwrite-conf osd prepare node14:sdi

[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node14:/dev/sdi:

[node14][DEBUG ] connected to host: node14

[node14][DEBUG ] detect platform information from remote host

[node14][DEBUG ] detect machine type

[ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final

[ceph_deploy.osd][DEBUG ] Deploying osd to node14

[node14][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

[node14][INFO ] Running command: udevadm trigger --subsystem-match=block --action=add

[ceph_deploy.osd][DEBUG ] Preparing host node14 disk /dev/sdi journal None activate False

[node14][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdi

[node14][ERROR ] INFO:ceph-disk:Will colocate journal with data on /dev/sdi

[node14][DEBUG ] The operation has completed successfully.

[node14][DEBUG ] The operation has completed successfully.

[node14][DEBUG ] meta-data=/dev/sdi1       isize=2048 agcount=32, agsize=22884224 blks

[node14][DEBUG ]     =           sectsz=512 attr=2, projid32bit=0

[node14][DEBUG ] data  =           bsize=4096 blocks=732295168, imaxpct=5

[node14][DEBUG ]     =           sunit=64  swidth=64 blks

[node14][DEBUG ] naming =version 2       bsize=4096 ascii-ci=0

[node14][DEBUG ] log   =internal log     bsize=4096 blocks=357568, version=2

[node14][DEBUG ]     =           sectsz=512 sunit=64 blks, lazy-count=1

[node14][DEBUG ] realtime =none         extsz=4096 blocks=0, rtextents=0

[node14][DEBUG ] The operation has completed successfully.

[ceph_deploy.osd][DEBUG ] Host node14 is now ready for osd use.

#

Check the new OSD

# ceph osd tree 

9 2.63 osd.9 up 1

17 2.73 osd.17 up 1

30 2.73 osd.30 up 1

53 2.73 osd.53 up 1

65 2.73 osd.65 up 1

78 2.73 osd.78 up 1

89 2.73 osd.89 up 1

113 2.73 osd.113 up 1

128 2.73 osd.128 up 1

141 2.73 osd.141 up 1

99 2.73 osd.99 up 1

# ceph status 

  cluster c452b7df-0c0b-4005-8feb-fc3bb92407f5

   health HEALTH_WARN 186 pgs backfill; 12 pgs backfilling; 6 pgs peering; 57 pgs recovering; 887 pgs recovery_wait; 6 pgs stale; 6 pgs stuck inactive; 6 pgs

 stuck stale; 283 pgs stuck unclean; 2 requests are blocked > 32 sec; recovery 784023/106982434 objects degraded (0.733%)

   monmap e6: 3 mons at {node01-ib=10.168.100.101:6789/0,node06-ib=10.168.100.106:6789/0,node11-ib=10.168.100.111:6789/0}, election epoch 2

836, quorum 0,1,2 node01-ib,node06-ib,node11-ib

   mdsmap e58: 1/1/1 up {0=node01-ib=up:active}

   osdmap e9190: 153 osds: 153 up, 153 in

   pgmap v1413041: 66256 pgs, 30 pools, 200 TB data, 51840 kobjects

      90504 GB used, 319 TB / 416 TB avail

      784023/106982434 objects degraded (0.733%)

        65108 active+clean

         186 active+remapped+wait_backfill

         887 active+recovery_wait

         12 active+remapped+backfilling

          6 stale+peering

         57 active+recovering

recovery io 383 MB/s, 95 objects/s

You would notice ceph will start putting PG ( data ) on this new OSD , so as to rebalance data and to make this new osd to participate cluster.

Yes at this stage you are done with replacement.

How Data Is Stored In CEPH Cluster

noreply@blogger.com (Anonymous) — Fri, 24 Jan 2014 15:37:00 +0000

HOW :: Data is Storage Inside Ceph Cluster

This is something definitely your would be wondering about , How Data _ _ _ Ceph Cluster ?

Now showing a easy to understand ceph data storage diagram.

## POOLS : Ceph cluster has POOLS , pools are the logical group for storing objects .These pools are made up of PG ( Placement Groups ). At the time of pool creation we have to provide number of placement groups that the pool is going to contain , number of object replicas ( usually takes default value , if other not specified )

Creating a pool ( pool-A ) with 128 placement groups

# ceph osd pool create pool-A 128
pool 'pool-A' created

Listing pools

# ceph osd lspools
0 data,1 metadata,2 rbd,36 pool-A,

Find out total number of placement groups being used by pool

# ceph osd pool get pool-A pg_num
pg_num: 128

Find out replication level being used by pool ( see rep size value for replication )

# ceph osd dump | grep -i pool-A
pool 36 'pool-A' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 4051 owner 0

Changing replication level for a pool ( compare from above step , rep size changed )

# ceph osd pool set pool-A size 3
set pool 36 size to 3
#
# ceph osd dump | grep -i pool-A
pool 36 'pool-A' rep size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 4054 owner 0

This means all the objects of pool-A will be replicated 3 times on 3 different OSD's

Now , Putting some data in pool-A , and data would be stored in the form of objects :-) thumb rule.

# dd if=/dev/zero of=object-A bs=10M count=1
1+0 records in
1+0 records out
10485760 bytes (10 MB) copied, 0.0222705 s, 471 MB/s
#

# dd if=/dev/zero of=object-B bs=10M count=1
1+0 records in
1+0 records out
10485760 bytes (10 MB) copied, 0.0221176 s, 474 MB/s
#

Putting some objects in pool-A

# rados -p pool-A put object-A  object-A

# rados -p pool-A put object-B  object-B

checking how many objects does the pool contains

# rados -p pool-A ls
object-A
object-B
#

## PG ( Placement Group ): Ceph cluster links objects --> PG . These PG containing objects are spread across multiple OSD and improves reliability.

## Object : Object is the smallest unit of data storage in ceph cluster , Each & Everything is stored in the form of objects , thats why ceph cluster is also known as Object Storage Cluster. Objects are mapped to PG , and these Objects / their copies always spreaded on different OSD. This is how ceph is designed.

Locating object , to which PG it belongs and stored where ??

# ceph osd map pool-A object-A
osdmap e4055 pool 'pool-A' (36) object 'object-A' -> pg 36.b301e3e8 (36.68) -> up [122,63,62] acting [122,63,62]
#

# ceph osd map pool-A object-B
osdmap e4055 pool 'pool-A' (36) object 'object-B' -> pg 36.47f173fb (36.7b) -> up [153,110,118] acting [153,110,118]
#

Now , we already created a pool-A , changed its replication level to 3 , added objects ( object-A and object-B ) to pool-A . Observe the above output. It throws a lot of information

OSD map version id is e4055
pool name is pool-A
pool id is 36
object name ( which was enquired , object-A and object-B )
Placement Group id to which this object belongs is ( 36.68 ) and ( 36.7b )
Our pool-A has replication level set to 3 , so every object of this pool should have 3 copies on different OSD , here our object's 3 copies resides on OSD.122 , OSD.63 and OSD.62

Login to ceph nodes containing OSD 122 , 63 and 62
You can see your OSD mounted

# df -h /var/lib/ceph/osd/ceph-122
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdj1             2.8T  1.8T  975G  65% /var/lib/ceph/osd/ceph-122
#

Browse to the directory where ACTUAL OBJECTS are stored

# pwd
/var/lib/ceph/osd/ceph-122/current
#

Under this directory if you do a ls command , you will see PG ID , In our case the PG id is 36.68 for object-A

# ls -la | grep -i 36.68
drwxr-xr-x 1 root root    54 Jan 24 16:45 36.68_head
#

Browse to the PG head directory , give ls and Here you go you reached to your OBJECT.

# pwd
/var/lib/ceph/osd/ceph-122/current/36.68_head
#
# ls -l
total 10240
-rw-r--r-- 1 root root 10485760 Jan 24 16:45 object-A__head_B301E3E8__24
#

Moral of the Story

Ceph storage cluster can have more than one Pools
Each pool SHOULD have multiple Placement Groups . More the PG , better your cluster performance , more reliable your setup would be.
A PG contains multiple Objects.
A PG is spreaded on multiple OSD , i.e Objects are spreaded across OSD. The first OSD mapped to PG will be its primary OSD and the other ODS's of same PG will be its secondary OSD.
An Object can be mapped to exactly one PG
Many PG's can be mapped to ONE OSD

How much PG you need for a POOL :

           (OSDs * 100)
Total PGs = ------------
              Replicas

# ceph osd stat
     osdmap e4055: 154 osds: 154 up, 154 in
#

Applying formula gives me = ( 154 * 100 ) / 3 = 5133.33

Now , round up this value to the next power of 2 , this will give you the number of PG you should have for a pool having replication size of 3 and total 154 OSD in entire cluster.

Final Value = 8192 PG

CephFS with a dedicated pool

noreply@blogger.com (Anonymous) — Fri, 10 Jan 2014 13:12:00 +0000

CephFS with a Dedicated Pool

This blog is about configuring a dedicated pool ( user defined pool ) for cephfs. If you are looking to configure cephfs , please visit CephFS Step by Step blog

Create a new pool for cephfs ( obviosly you can use your existing pool )

# rados mkpool cephfs

Grab pool id

# ceph osd dump | grep -i cephfs
pool 34 'cephfs' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 860 owner 0
#

Assign the pool to MDS

# ceph mds add_data_pool 34

Mount your cephfs share

# mount -t ceph 192.168.100.101:/ /cephfs -o name=cephfs,secretfile=/etc/ceph/client.cephfs

Check current layout of cephfs , you would notice the default layout.data_pool is set to 0 , which means your cephfs will store date in pool 0 i.e data pool

# cephfs /cephfs/ show_layout
layout.data_pool:     0
layout.object_size:   4194304
layout.stripe_unit:   4194304
layout.stripe_count:  1

Set a new layout for data_pool in cephfs , use pool id of the pool that we have created above.

# cephfs /cephfs/ set_layout -p 34
# cephfs /cephfs/ show_layout
layout.data_pool:     34
layout.object_size:   4194304
layout.stripe_unit:   4194304
layout.stripe_count:  1
[root@na_csc_fedora19 ~]#

Remount your cephfs share

# umount /cephfs
# mount -t ceph 192.168.100.101:/ /cephfs -o name=cephfs,secretfile=/etc/ceph/client.cephfs

Check objects that are present in cephfs pool , there should be no object as this is a fresh pool and does not contain any data . But if you look for objects for any other pool , it should contain objects.

# rados --pool=cephfs ls
#
# rados --pool=metadata ls
1.00000000.inode
100.00000000
100.00000000.inode
1.00000000
2.00000000
200.00000000
this is a tesf fine
200.00000001
600.00000000
601.00000000
602.00000000
603.00000000
604.00000000
605.00000000
606.00000000
607.00000000
608.00000000
609.00000000
mds0_inotable
mds0_sessionmap
mds_anchortable
mds_snaptable
#

Go to your cephfs directory and create some files ( put data in your file ) .

# cd /cephfs/
# vi test

Recheck for objects in cephfs pool , now it will show you objects .

# rados --pool=cephfs ls
10000000005.00000000
#

Summary is , we have created a new pool named "cephfs" , changed layout of cephfs to store its data in new pool "cephfs" , and finally we saw cephfs data is getting stored in pool named cephfs ( i know its too more cephfs , read it again if you are sleeping and didn't understand cephfs)

Kraken :: The First Free Ceph Dashboard in Town

noreply@blogger.com (Anonymous) — Thu, 09 Jan 2014 14:45:00 +0000

Kraken :: The Free Ceph Dashboard is Finally Live

Kraken is a Free ceph dashboard for monitoring and statistics. Special thanks to Donald Talton for this beautiful dashboard.

Installing Kraken

Install Prerequisites

# yum install git

# yum install django

# yum install python-pip


# pip install requests

Requirement already satisfied (use --upgrade to upgrade): requests in /usr/lib/python2.7/site-packages

Cleaning up...

#

# pip install django

Requirement already satisfied (use --upgrade to upgrade): django in /usr/lib/python2.7/site-packages

Cleaning up...

#

# yum install screen

Create a new user account

# useradd kraken

Clone kraken from github

# cd /home/kraken

# git clone https://github.com/krakendash/krakendash


Cloning into 'krakendash'...

remote: Counting objects: 511, done.

remote: Compressing objects: 100% (276/276), done.

remote: Total 511 (delta 240), reused 497 (delta 226)

Receiving objects: 100% (511/511), 1.53 MiB | 343.00 KiB/s, done.

Resolving deltas: 100% (240/240), done.

#

Exceute api.sh and django.sh one by one , these would get launched in screens , use ctrl A and D to detach from screen

# ./api.sh


[detached from 14662.api]

# ./django.sh

[detached from 14698.django]

#

# ps -ef | grep -i screen

root     14662     1  0 07:29 ?        00:00:00 SCREEN -S api sudo ceph-rest-api -c /etc/ceph/ceph.conf --cluster ceph -i admin

root     14698     1  0 07:30 ?        00:00:00 SCREEN -S django sudo python krakendash/kraken/manage.py runserver 0.0.0.0:8000

root     14704 14472  0 07:30 pts/0    00:00:00 grep --color=auto -i screen

#

Open your browser and navigate to http://localhost:8000/

Great you have a Ceph GUI dashboard running now :-)
Watch out this space for new features of kraken

Zero To Hero Guide : : For CEPH CLUSTER PLANNING

noreply@blogger.com (Anonymous) — Thu, 02 Jan 2014 15:55:00 +0000

What it is all about :

If you think or discuss about Ceph , the most common question strike to your mind is "What Hardware Should I Select For My CEPH Storage Cluster ?" and yes if you really thought of this question in your mind , congratulations you seems to be serious about ceph technology and You should be because CEPH IS THE FUTURE OF STORAGE.

Ceph runs on Commodity hardware , Ohh Yeah !! everyone now knows it . It is designed to build a multi-petabyte storage cluster while providing enterprise ready features. No single point of failure , scaling to exabytes , self managing and self healing ( saves operational cost ) , runs on commodity hardware ( no vendor locking , saves capital investment )

Ceph Overview :-

The sole of ceph storage cluster is RADOS ( Reliable Autonomic Distributed Object Store ). Ceph uses powerful CRUSH ( Controlled Replication Under Scalable Hashing ) algorithm for optimize data placement , self managing and self healing. The RESTful interface is provided by Ceph Object Gateway (RGW) aks Rados GateWay and virtual disks are provisioned by Ceph Block Device (RBD)

Ceph Overview - Image Credit : Inktank

Ceph Components :-

# Ceph OSD ( Object Storage Daemons ) storage data in objects , manages data replication , recovery , rebalancing and provides stage information to Ceph Monitor. Its recommended to user 1 OSD per physical disk.

# Ceph MON ( Monitors ) maintains overall health of cluster by keeping cluster map state including Monitor map , OSD map , Placement Group ( PG ) map , and CRUSH map. Monitors receives state information from other components to maintain maps and circulate these maps to other Monitor and OSD nodes.

# Ceph RGW ( Object Gateway / Rados Gateway ) RESTful API interface compatible with Amazon S3 , OpenStack Swift .

# Ceph RBD ( Raw Block Device ) Provides Block Storage to VM / bare metal as well as regular clients , supports OpenStack and CloudStack . Includes Enterprise features like snapshot , thin provisioning , compression.

# CephFS ( File System ) distributed POSIX NAS storage.

Few Thumb Rules :-

Run OSD on a dedicated storage node ( server with multiple disks ) , actual data is stored in the form of objects.
Run Monitor on a separate dedicated hardware or coexists with ceph client nodes ( other than OSD node ) such as RGW , CephFS node . For production its recommended to run Monitors on dedicated low cost servers since Monitors are not resource hungry.

Monitor Hardware Configuration :-

Monitor maintains health of entire cluster , it contains PG logs and OSD logs . A minimum of three monitors nodes are recommended for a cluster quorum. Ceph monitor nodes are not resource hungry they can work well with fairly low cpu and memory. A 1U server with low cost processor E5-2603,16GB RAM and 1GbE network should be sufficient in most of the cases. If PG,Monitor and OSD logs are storage on local disk of monitor node , make sure you have sufficient amount of local storage so that it should not fill up.

Unhealthy clusters require more storage for logs , can reach upto GB and even hundreds of GB if the cluster is left unhealthy for a very long time . If verbose output is set on monitor nodes, then these are bound to generate huge amount of logging information. Refer ceph documentation for monitor log setting.

Its recommended to run monitor on distant nodes rather on all on all one node or on virtual machines on physical separated machines to prevent single point of failure.

The Planning Stage :-

Deploying a ceph cluster in production requires a little bit Homework , you should gather the below information so that you can design a better and more reliable and scalable ceph cluster to fit in your IT needs. These very specific to your needs and your IT environment. This information will help you to design your storage requirement better.

Business Requirement

Budget ?
Do you need Ceph cluster for day to day operation or SPECIAL

Technical Requirement

What applications will be running on your ceph cluster ?
What type of data will be stored on your ceph cluster ?
Should the ceph cluster be optimized for capacity and performance ?
What should be usable storage capacity ?
What is expected growth rate ?
How many IOPS should the cluster support ?
How much throughput should the cluster support
How much data replication ( reliability level ) you need ?

Collect as much information as possible during the planning stage , the will give all the answers required to construct a better ceph cluster.

The Physical Node and clustering technique:-

In addition to above collected information , also take into account the rack density and power budget , data center space pace cost to size the optimal node configuration. Ceph replicated data across multiple nodes in a storage cluster to provide data redundancy and higher availability. Its important to consider.

Should the replicated node be on the same rack or multiple racks to avoid SPOF ?
Should the OSD traffic stay within the rack or span across rack in a dedicated or shared network ?
How many nodes failure can be tolerated ?
If the nodes are separated out across multiple racks network traffic increases and the impact of latency and the number of network switch hops should be considered.

Ceph will automatically recover by re-replicating data from the failed nodes using secondary copies present on other nodes in cluster . A node failure thus have several effects.

Total cluster capacity is reduced by some fractions.
Total cluster throughput is reduced by some fractions.
The cluster enters a write heavy recovery processes.

A general thumb of rule to calculate recovery time in a ceph cluster given 1 disk per OSD node is :

Recovery Time in seconds = disk capacity in Gigabits / ( network speed *(nodes-1) )

# POC Environment -- Can have a minimum of 3 physical nodes with 10 OSD's each. This provides 66% cluster availability upon a physical node failure and 97% uptime upon an OSD failure. RGW and Monitor nodes can be put on OSD nodes but this may impact performance and not recommended for production.

# Production Environment -- a minimum of 5 physically separated nodes and minimum of 100 OSD @ 4TB per OSD the cluster capacity is over 130TB and provides 80% uptime on physical node failure and 99% uptime on OSD failure. RGW and Monitors should be on separate nodes.

Based on the outcome of planning phase and physical nodes and clustering stage you have a look on the hardware available in market as per your budget.

OSD CPU selection :-

< Under Construction ... Stay Tuned >

Ceph and OpenStack in a Nutshell

noreply@blogger.com (Anonymous) — Mon, 23 Dec 2013 21:11:00 +0000

Ceph and OpenStack in a Nutshell

Ceph and Openstack in a Nutshell from Karan Singh

Ceph Filesystem ( CephFS) :: Step by Step Configuration

noreply@blogger.com (Anonymous) — Mon, 23 Dec 2013 16:23:00 +0000

CephFS

Ceph Filesystem is a posix compliant file system that uses ceph storage cluster to store its data. This is the only ceph component that is not ready for production , i would like to say ready for pre-production.

Internals

Thanks to http://ceph.com/docs/master/cephfs/ for Image

Requirement of CephFS

You need a running ceph cluster with at least one MDS node. MDS is required for CephFS to work.
If you don't have MDS configure one

# ceph-deploy mds create <MDS-NODE-ADDRESS>

Note : If you are running short of hardware or want to save hardware you can run MDS services on existing Monitor nodes. MDS services does not need much resources

A Ceph client to mount cephFS

Configuring CephFS

Install ceph on client node

[root@storage0101-ib ceph]# ceph-deploy install na_fedora19
[ceph_deploy.cli][INFO  ] Invoked (1.3.2): /usr/bin/ceph-deploy install na_fedora19
[ceph_deploy.install][DEBUG ] Installing stable version emperor on cluster ceph hosts na_csc_fedora19
[ceph_deploy.install][DEBUG ] Detecting platform for host na_fedora19 ...
[na_csc_fedora19][DEBUG ] connected to host: na_csc_fedora19
[na_csc_fedora19][DEBUG ] detect platform information from remote host
[na_csc_fedora19][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Fedora 19 Schrödinger’s Cat
[na_csc_fedora19][INFO  ] installing ceph on na_fedora19
[na_csc_fedora19][INFO  ] Running command: rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
[na_csc_fedora19][INFO  ] Running command: rpm -Uvh --replacepkgs --force --quiet http://ceph.com/rpm-emperor/fc19/noarch/ceph-release-1-0.fc19.noarch.rpm
[na_csc_fedora19][DEBUG ] ########################################
[na_csc_fedora19][DEBUG ] Updating / installing...
[na_csc_fedora19][DEBUG ] ########################################
[na_csc_fedora19][INFO  ] Running command: yum -y -q install ceph

[na_csc_fedora19][ERROR ] Warning: RPMDB altered outside of yum.
[na_csc_fedora19][DEBUG ] No Presto metadata available for Ceph
[na_csc_fedora19][INFO  ] Running command: ceph --version
[na_csc_fedora19][DEBUG ] ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
[root@storage0101-ib ceph]#

Create a new pool for CephFS

# rados mkpool cephfs

Create a new keyring (client.cephfs) for cephfs

# ceph auth get-or-create client.cephfs mon 'allow r' osd 'allow rwx pool=cephfs' -o /etc/ceph/client.cephfs.keyring

Extract secret key from keyring

# ceph-authtool -p -n client.cephfs /etc/ceph/client.cephfs.keyring > /etc/ceph/client.cephfs

Copy the secret file to client node under /etc/ceph . This allow filesystem to mount when cephx authentication is enabled

# scp client.cephfs na_fedora19:/etc/ceph
client.cephfs                                                                100%   41     0.0KB/s   00:00

List all the keys on ceph cluster

# ceph auth list

Option-1 : Mount CephFS with Kernel Driver

On the client machine add mount point in /etc/fstab . Provide IP address of your ceph monitor node and path of secret key that we have created above

192.168.200.101:6789:/ /cephfs ceph name=cephfs,secretfile=/etc/ceph/client.cephfs,noatime 0 2

Mount cephfs mount point , you might see some "mount: error writing /etc/mtab: Invalid argument" but you can ignore them and check df -h

[root@na_fedora19 ceph]# mount /cephfs
mount: error writing /etc/mtab: Invalid argument

[root@na_fedora19 ceph]#
[root@na_fedora19 ceph]# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/vda1               7.8G  2.1G  5.4G  28% /
devtmpfs                3.9G     0  3.9G   0% /dev
tmpfs                   3.9G     0  3.9G   0% /dev/shm
tmpfs                   3.9G  288K  3.9G   1% /run
tmpfs                   3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs                   3.9G  2.6M  3.9G   1% /tmp
192.168.200.101:6789:/  419T  8.5T  411T   3% /cephfs
[root@na_fedora19 ceph]#

Option-2 : Mounting CephFS as FUSE

Copy ceph configuration file ( ceph.conf ) from monitor node to client node and make sure it has permission of 644

# scp ceph.conf na_fedora19:/etc/ceph

# chmod 644 ceph.conf

Copy the secret file from monitor node to client node under /etc/ceph. This allow filesystem to mount when cephx authentication is enabled ( we have done this earlier )

# scp client.cephfs na_fedora19:/etc/ceph
client.cephfs                                                                100%   41     0.0KB/s   00:00

Make sure you have "ceph-fuse" package installed on client machine

# rpm -qa | grep -i ceph-fuse
ceph-fuse-0.72.2-0.fc19.x86_64

To mount Ceph Filesystem as FUSE use ceph-fuse comand

[root@na_fedora19 ceph]# ceph-fuse -m 192.168.100.101:6789  /cephfs
ceph-fuse[3256]: starting ceph client
ceph-fuse[3256]: starting fuse
[root@na_csc_fedora19 ceph]#

[root@na_fedora19 ceph]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       7.8G  2.1G  5.4G  28% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  292K  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           3.9G  2.6M  3.9G   1% /tmp
ceph-fuse       419T  8.5T  411T   3% /cephfs
[root@na_fedora19 ceph]#

Ceph + OpenStack :: Part-5

noreply@blogger.com (Anonymous) — Thu, 05 Dec 2013 17:57:00 +0000

OpenStack Instance boot from Ceph Volume

For a list of images to choose from to create a bootable volume

[root@rdo /(keystone_admin)]# nova image-list
+--------------------------------------+-----------------------------+--------+--------+
| ID                                   | Name                        | Status | Server |
+--------------------------------------+-----------------------------+--------+--------+
| f61edc8d-c9a1-4ff4-b4fc-c8128bd1a10b | Ubuntu 12.04 cloudimg amd64 | ACTIVE |        |
| fcc07414-bbb3-4473-a8df-523664c8c9df | ceph-glance-image           | ACTIVE |        |
| be62a5bf-879f-4d1f-846c-fdef960224ff | precise-cloudimg.raw        | ACTIVE |        |
| 3c2db0ad-8d1e-400d-ba13-a506448f2a8e | precise-server-cloudimg     | ACTIVE |        |
+--------------------------------------+-----------------------------+--------+--------+
[root@rdo /(keystone_admin)]#

To create a bootable volume from an image, include the image ID in the command: Before the volume builds, its bootable state is false.

[root@rdo qemu(keystone_admin)]# cinder create --image-id be62a5bf-879f-4d1f-846c-fdef960224ff --display-name my-boot-vol 10
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2013-12-05T13:34:38.296723      |
| display_description |                 None                 |
|     display_name    |             my-boot-vol              |
|          id         | 5fca6e1b-b494-4773-9c78-63f72703bfdf |
|       image_id      | be62a5bf-879f-4d1f-846c-fdef960224ff |
|       metadata      |                  {}                  |
|         size        |                  10                  |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+
[root@rdo qemu(keystone_admin)]#
[root@rdo qemu(keystone_admin)]# cinder list
+--------------------------------------+-------------+--------------+------+--------------+----------+--------------------------------------+
|                  ID                  |    Status   | Display Name | Size | Volume Type  | Bootable |             Attached to              |
+--------------------------------------+-------------+--------------+------+--------------+----------+--------------------------------------+
| 0e2bfced-be6a-44ec-a3ca-22c771c66cdc |    in-use   |  nova-vol_1  |  2   |     None     |  false   | 9d3c327f-1893-40ff-8a82-16fad9ce6d91 |
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac |  available  |  ceph-vol1   |  5   | ceph-storage |  false   |                                      |
| 5fca6e1b-b494-4773-9c78-63f72703bfdf | downloading | my-boot-vol  |  10  |     None     |  false   |                                      |
+--------------------------------------+-------------+--------------+------+--------------+----------+--------------------------------------+

Wait for few minutes the bootable state turns to true. Copy the value in the ID field for your volume.

[root@rdo qemu(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+--------------+----------+--------------------------------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type  | Bootable |             Attached to              |
+--------------------------------------+-----------+--------------+------+--------------+----------+--------------------------------------+
| 0e2bfced-be6a-44ec-a3ca-22c771c66cdc |   in-use  |  nova-vol_1  |  2   |     None     |  false   | 9d3c327f-1893-40ff-8a82-16fad9ce6d91 |
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available |  ceph-vol1   |  5   | ceph-storage |  false   |                                      |
| 5fca6e1b-b494-4773-9c78-63f72703bfdf | available | my-boot-vol  |  10  |     None     |   true   |                                      |
+--------------------------------------+-----------+--------------+------+--------------+----------+--------------------------------------+
[root@rdo qemu(keystone_admin)]#

Create a nova instance which will be boot from ceph volume

[root@rdo qemu(keystone_admin)]# nova boot --flavor 2 --image be62a5bf-879f-4d1f-846c-fdef960224ff --block_device_mapping vda=5fca6e1b-b494-4773-9c78-63f72703bfdf::0 --security_groups=default --nic net-id=4fe5909e-02db-4517-89f2-1278248fa26c  myInstanceFromVolume
+--------------------------------------+--------------------------------------+
| Property                             | Value                                |
+--------------------------------------+--------------------------------------+
| OS-EXT-STS:task_state                | scheduling                           |
| image                                | precise-cloudimg.raw                 |
| OS-EXT-STS:vm_state                  | building                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000001e                    |
| OS-SRV-USG:launched_at               | None                                 |
| flavor                               | m1.small                             |
| id                                   | f24a0b29-9f1e-444b-b895-c3c694f2f1bc |
| security_groups                      | [{u'name': u'default'}]              |
| user_id                              | 99f8019ba2694d78a680a5de46aa1afd     |
| OS-DCF:diskConfig                    | MANUAL                               |
| accessIPv4                           |                                      |
| accessIPv6                           |                                      |
| progress                             | 0                                    |
| OS-EXT-STS:power_state               | 0                                    |
| OS-EXT-AZ:availability_zone          | nova                                 |
| config_drive                         |                                      |
| status                               | BUILD                                |
| updated                              | 2013-12-05T13:47:34Z                 |
| hostId                               |                                      |
| OS-EXT-SRV-ATTR:host                 | None                                 |
| OS-SRV-USG:terminated_at             | None                                 |
| key_name                             | None                                 |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | None                                 |
| name                                 | myInstanceFromVolume                 |
| adminPass                            | qt34izQiLkG3                         |
| tenant_id                            | 0dafe42cfde242ddbb67b681f59bdb00     |
| created                              | 2013-12-05T13:47:34Z                 |
| os-extended-volumes:volumes_attached | []                                   |
| metadata                             | {}                                   |
+--------------------------------------+--------------------------------------+
[root@rdo qemu(keystone_admin)]#
[root@rdo qemu(keystone_admin)]#
[root@rdo qemu(keystone_admin)]#
[root@rdo qemu(keystone_admin)]# nova list
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| ID                                   | Name                 | Status  | Task State | Power State | Networks            |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2            | SHUTOFF | None       | Shutdown    | public=172.24.4.228 |
| f24a0b29-9f1e-444b-b895-c3c694f2f1bc | myInstanceFromVolume | BUILD   | spawning   | NOSTATE     | private=10.0.0.3    |
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu         | ACTIVE  | None       | Running     | public=172.24.4.230 |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
[root@rdo qemu(keystone_admin)]#

Just in few minutes the instance starts RUNNING , time for a party now

[root@rdo qemu(keystone_admin)]# nova list
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| ID                                   | Name                 | Status  | Task State | Power State | Networks            |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2            | SHUTOFF | None       | Shutdown    | public=172.24.4.228 |
| f24a0b29-9f1e-444b-b895-c3c694f2f1bc | myInstanceFromVolume | ACTIVE  | None       | Running     | private=10.0.0.3    |
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu         | ACTIVE  | None       | Running     | public=172.24.4.230 |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
[root@rdo qemu(keystone_admin)]#

OpenStack Instance boot from Ceph Volume :: Troubleshooting

During boot from volume , i encountered some errors after creating nova instance. The image was not able to get booted up from volume

[root@rdo nova(keystone_admin)]# nova boot --flavor 2 --image be62a5bf-879f-4d1f-846c-fdef960224ff --block_device_mapping vda=dd315dda-b22a-4cf8-8b77-7c2b2f163155:::0 --security_groups=default --nic net-id=4fe5909e-02db-4517-89f2-1278248fa26c  myInstanceFromVolume
+--------------------------------------+----------------------------------------------------+
| Property                             | Value                                              |
+--------------------------------------+----------------------------------------------------+
| OS-EXT-STS:task_state                | scheduling                                         |
| image                                | precise-cloudimg.raw                               |
| OS-EXT-STS:vm_state                  | building                                           |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000001d                                  |
| OS-SRV-USG:launched_at               | None                                               |
| flavor                               | m1.small                                           |
| id                                   | f324e9b8-ec3a-4174-8b97-bf78dba62932               |
| security_groups                      | [{u'name': u'default'}]                            |
| user_id                              | 99f8019ba2694d78a680a5de46aa1afd                   |
| OS-DCF:diskConfig                    | MANUAL                                             |
| accessIPv4                           |                                                    |
| accessIPv6                           |                                                    |
| progress                             | 0                                                  |
| OS-EXT-STS:power_state               | 0                                                  |
| OS-EXT-AZ:availability_zone          | nova                                               |
| config_drive                         |                                                    |
| status                               | BUILD                                              |
| updated                              | 2013-12-05T12:42:22Z                               |
| hostId                               |                                                    |
| OS-EXT-SRV-ATTR:host                 | None                                               |
| OS-SRV-USG:terminated_at             | None                                               |
| key_name                             | None                                               |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | None                                               |
| name                                 | myInstanceFromVolume                               |
| adminPass                            | eish5pu56CiE                                       |
| tenant_id                            | 0dafe42cfde242ddbb67b681f59bdb00                   |
| created                              | 2013-12-05T12:42:21Z                               |
| os-extended-volumes:volumes_attached | [{u'id': u'dd315dda-b22a-4cf8-8b77-7c2b2f163155'}] |
| metadata                             | {}                                                 |
+--------------------------------------+----------------------------------------------------+
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]# nova list
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| ID                                   | Name                 | Status  | Task State | Power State | Networks            |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2            | SHUTOFF | None       | Shutdown    | public=172.24.4.228 |
| f324e9b8-ec3a-4174-8b97-bf78dba62932 | myInstanceFromVolume | ERROR   | None       | NOSTATE     | private=10.0.0.3    |
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu         | ACTIVE  | None       | Running     | public=172.24.4.230 |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
[root@rdo nova(keystone_admin)]#

After checking up logs from /var/log/libvirt/qemu/instance-0000001d.log

qemu-kvm: -drive file=rbd:ceph-volumes/volume-dd315dda-b22a-4cf8-8b77-7c2b2f163155:id=volumes:key=AQC804xS8HzFJxAAD/zzQ8LMzq9wDLq/5a472g==:auth_supported=cephx\;none:mon_host=192.168.1.31\:6789\;192.168.1.33\:6789\;192.168.1.38\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=dd315dda-b22a-4cf8-8b77-7c2b2f163155,cache=none: could not open disk image rbd:ceph-volumes/volume-dd315dda-b22a-4cf8-8b77-7c2b2f163155:id=volumes:key=AQC804xS8HzFJxAAD/zzQ8LMzq9wDLq/5a472g==:auth_supported=cephx\;none:mon_host=192.168.1.31\:6789\;192.168.1.33\:6789\;192.168.1.38\:6789: No such file or directory
2013-12-05 12:42:29.544+0000: shutting down

Run qemu-img -h command to check for the supported format , here i found rbd format is not supported by qemu , so there is something fishy in this

Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2 qed vhdx parallels nbd blkdebug host_cdrom host_floppy host_device file gluster gluster gluster gluster

Check the installed qemu version

[root@rdo qemu(keystone_admin)]# rpm -qa | grep -i qemu
qemu-img-0.12.1.2-2.415.el6_5.3.x86_64
qemu-guest-agent-0.12.1.2-2.415.el6_5.3.x86_64
gpxe-roms-qemu-0.9.7-6.10.el6.noarch
qemu-kvm-0.12.1.2-2.415.el6_5.3.x86_64
qemu-kvm-tools-0.12.1.2-2.415.el6_5.3.x86_64
[root@rdo qemu(keystone_admin)]#

Have a look on previous post to see the installation of correct version of qemu . After this your nova instance should boot from volume

[root@rdo qemu(keystone_admin)]# rpm -qa | grep -i qemu
qemu-img-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64
qemu-guest-agent-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64
qemu-kvm-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64
gpxe-roms-qemu-0.9.7-6.10.el6.noarch
qemu-kvm-tools-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64
[root@rdo qemu(keystone_admin)]#

[root@rdo /(keystone_admin)]# qemu-img -h | grep -i rbd
Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2 qed parallels nbd blkdebug host_cdrom host_floppy host_device file rbd
[root@rdo /(keystone_admin)]#


[root@rdo qemu(keystone_admin)]# nova list
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| ID                                   | Name                 | Status  | Task State | Power State | Networks            |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2            | SHUTOFF | None       | Shutdown    | public=172.24.4.228 |
| f24a0b29-9f1e-444b-b895-c3c694f2f1bc | myInstanceFromVolume | ACTIVE  | None       | Running     | private=10.0.0.3    |
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu         | ACTIVE  | None       | Running     | public=172.24.4.230 |
+--------------------------------------+----------------------+---------+------------+-------------+---------------------+
[root@rdo qemu(keystone_admin)]#

Ceph + OpenStack :: Part-4

noreply@blogger.com (Anonymous) — Thu, 05 Dec 2013 17:55:00 +0000

Testing OpenStack Glance + RBD

To allow glance to keep images on ceph RBD volume , edit /etc/glance/glance-api.conf

default_store = rbd
# ============ RBD Store Options =============================

# Ceph configuration file path
# If using cephx authentication, this file should
# include a reference to the right keyring
# in a client. section
rbd_store_ceph_conf = /etc/ceph/ceph.conf

# RADOS user to authenticate as (only applicable if using cephx)
rbd_store_user = images         ## This is the ceph user that we have created above in this document

# RADOS pool in which images are stored
rbd_store_pool = ceph-images   ## This is the ceph pool for images that we have created above in this document

# Images will be chunked into objects of this size (in megabytes).
# For best performance, this should be a power of two
rbd_store_chunk_size = 8

Check ceph auth to make sure keys are present for client.images user . This should present here as we have created them earlier in this document.

[root@rdo ceph(keystone_admin)]# ceph auth list
installed auth entries:

mds.ceph-mon1
   key: AQAxp35ScNUxOBAAfAXc+J5F3/v7jUrpztVRBQ==
   caps: [mds] allow
   caps: [mon] allow profile mds
   caps: [osd] allow rwx
osd.0
   key: AQCOvWpSsKN4JBAA015Uf53JjGCJS4cgzhxGFg==
   caps: [mon] allow profile osd
   caps: [osd] allow *
osd.1
   key: AQCn+mtSULePJxAACKvSkIqF39f5MaFiwsVR6Q==
   caps: [mon] allow profile osd
   caps: [osd] allow *
osd.10
   key: AQCjNIZSOF7AFxAA3vwLvgaB3PI+WAZPt2eIlQ==
   caps: [mon] allow profile osd
   caps: [osd] allow *
osd.2
   key: AQDHBmxSwKTZBxAAyWlQGj8H48sdPGl4PzlFbQ==
   caps: [mon] allow profile osd
   caps: [osd] allow *
osd.3
   key: AQBv/WtSwH5gOBAAHrSWblzq/n/qPbaurBMC2g==
   caps: [mon] allow profile osd
   caps: [osd] allow *
osd.4
   key: AQCiE2xSgDLQMRAAjWotlPtyqaSgpll1P6NTfw==
   caps: [mon] allow profile osd
   caps: [osd] allow *
osd.5
   key: AQCrFGxSOEnjMRAAnrqLcMR8UHu3rTTTQ5DHjw==
   caps: [mon] allow profile osd
   caps: [osd] allow *
osd.6
   key: AQAXFmxSUAmsJxAA83qr0mZ3sGLQbi+C59LXgw==
   caps: [mon] allow profile osd
   caps: [osd] allow *
osd.7
   key: AQBpFmxSOCZFNBAAONPg5I3QnB3Wd/pr7rSkEg==
   caps: [mon] allow profile osd
   caps: [osd] allow *
osd.8
   key: AQC7M4ZSSP9dMhAAh4HQ0uvKFs9yHiQrobXzUA==
   caps: [mon] allow profile osd
   caps: [osd] allow *
osd.9
   key: AQBmNIZSkAIjMRAA3FFGaMhGiPCmYmQ9REisRQ==
   caps: [mon] allow profile osd
   caps: [osd] allow *
client.admin
   key: AQBSt2pS4M5cCBAAUd4jWA1vxJT+y5C9X6juzg==
   caps: [mds] allow
   caps: [mon] allow *
   caps: [osd] allow *
client.bootstrap-mds
   key: AQBSt2pS8IirKxAAQ27MWZ4pEEBuNhCDrj/FRw==
   caps: [mon] allow profile bootstrap-mds
client.bootstrap-osd
   key: AQBSt2pSYLXVGRAAYs0R8gXKSEct6ApEy4h6dQ==
   caps: [mon] allow profile bootstrap-osd
client.images
   key: AQDS04xSEJEYABAA8Kl9eEqIr3Y8pyz+tPRpvQ==
   caps: [mon] allow r
   caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=ceph-images
client.volumes
   key: AQC804xS8HzFJxAAD/zzQ8LMzq9wDLq/5a472g==
   caps: [mon] allow r
   caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=ceph-volumes, allow rx pool=ceph-images

[root@rdo ceph(keystone_admin)]#

Copy the keyrings file to glance directory. This is the same file that we have generated above in this document.

cp /etc/ceph/ceph.client.images.keyring /etc/glance
chown glance:glance /etc/glance/ceph.client.images.keyring

 #  service openstack-glance-api restart
 #  service openstack-glance-registry restart
 #  service openstack-glance-scrubber restart

Before creating a new glance image on ceph volume , check the ceph pool content ( in my case its empty and it should be , this is for the first time we are using this volume )

[root@rdo init.d(keystone_admin)]# rbd -p ceph-images ls
rbd: pool ceph-images doesn't contain rbd images
[root@rdo init.d(keystone_admin)]#

Download a new image or use if you have existing.

[root@rdo var(keystone_admin)]# wget http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
[root@rdo var(keystone_admin)]# glance add name="ceph-glance-image" is_public=True disk_format=qcow2 container_format=ovf architecture=x86_64 <  ubuntu-12.04.3-desktop-amd64.iso
Added new image with ID: fcc07414-bbb3-4473-a8df-523664c8c9df
[root@rdo var(keystone_admin)]# glance index
ID                                   Name                           Disk Format          Container Format     Size
------------------------------------ ------------------------------ -------------------- -------------------- --------------
fcc07414-bbb3-4473-a8df-523664c8c9df ceph-glance-image              qcow2                ovf                       742391808
3c2db0ad-8d1e-400d-ba13-a506448f2a8e precise-server-cloudimg        qcow2                ovf                       254738432
f61edc8d-c9a1-4ff4-b4fc-c8128bd1a10b Ubuntu 12.04 cloudimg amd64    qcow2                ovf                       254738432
[root@rdo var(keystone_admin)]#

Now check your ceph pool , it will have image ( even match the glance image ID with pool objects , also compare image size with object size )

[root@rdo var(keystone_admin)]# rbd -p ceph-images ls
fcc07414-bbb3-4473-a8df-523664c8c9df
[root@rdo var(keystone_admin)]#

[root@rdo var(keystone_admin)]# du ubuntu-12.04.3-desktop-amd64.iso
724996   ubuntu-12.04.3-desktop-amd64.iso
[root@rdo var(keystone_admin)]#

[root@rdo var(keystone_admin)]# rados df
pool name       category                 KB      objects       clones     degraded      unfound           rd        rd KB           wr        wr KB
ceph-images     -                     724993           92            0            0           0           63           50           98       724993
ceph-volumes    -                          1            9            0            0           0          284          212           72            8
data            -                  141557761        34563            0            0           0        71843    131424295        71384    146013188
metadata        -                       9667           23            0            0           0           72        19346          851        10102
rbd             -                          1            1            0            0           0         2117        21883          305       226753
  total used       287309244        34688
  total avail     6222206348
  total space     6509515592

Feeling Happy , you should be , now glance will use ceph to retrieve / store images

Please Follow Ceph + OpenStack :: Part-5 for next step in installation

Ceph + OpenStack :: Part-3

noreply@blogger.com (Anonymous) — Thu, 05 Dec 2013 17:51:00 +0000

Testing OpenStack Cinder + RBD

Creating a cinder volume provided by ceph backend

[root@rdo /]#
[root@rdo /]# cinder create --display-name cinder-ceph-vol1 --display-description "first cinder volume on ceph backend" 10
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2013-11-27T19:35:39.481075      |
| display_description | first cinder volume on ceph backend  |
|     display_name    |           cinder-ceph-vol1           |
|          id         | 10cc0855-652a-4a9b-baa1-80bc86dc12ac |
|       metadata      |                  {}                  |
|         size        |                  10                  |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+
[root@rdo /]#
[root@rdo /]#

[root@rdo /(keystone_admin)]# cinder list
+--------------------------------------+-----------+------------------+------+--------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------------------+------+--------------+----------+-------------+
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | cinder-ceph-vol1 | 5 | ceph-storage | false | |
| 9671edaa-62c8-4f98-a36c-d6e59612141b | available | boot_from_volume | 20 | None | false | |
+--------------------------------------+-----------+------------------+------+--------------+----------+-------------+
[root@rdo /(keystone_admin)]#

[root@rdo /]#

[root@rdo /]# rados lspools
data
metadata
rbd
ceph-images
ceph-volumes
[root@rdo /]#
[root@rdo /]#
[root@rdo /]# rbd -p ceph-volumes ls
volume-10cc0855-652a-4a9b-baa1-80bc86dc12ac
[root@rdo /]#

Attaching cinder volume to Instance

[root@rdo /(keystone_admin)]# nova list
+--------------------------------------+------------------+---------+--------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------+---------+--------------+-------------+---------------------+
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2 | SHUTOFF | None | Shutdown | public=172.24.4.228 |
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu | ACTIVE | None | Running | public=172.24.4.230 |
| 10d1c49f-9fbc-455f-b72d-f731338b2dd5 | small-ubuntu-pwd | ACTIVE | powering-off | Shutdown | public=172.24.4.231 |
+--------------------------------------+------------------+---------+--------------+-------------+---------------------+
[root@rdo /(keystone_admin)]#

[root@rdo /(keystone_admin)]# nova show 9d3c327f-1893-40ff-8a82-16fad9ce6d91
+--------------------------------------+--------------------------------------------------------------------+
| Property                             | Value                                                              |
+--------------------------------------+--------------------------------------------------------------------+
| status                               | ACTIVE                                                             |
| updated                              | 2013-12-03T15:58:31Z                                               |
| OS-EXT-STS:task_state                | None                                                               |
| OS-EXT-SRV-ATTR:host                 | rdo                                                                |
| key_name                             | RDO-admin                                                          |
| image                                | Ubuntu 12.04 cloudimg amd64 (f61edc8d-c9a1-4ff4-b4fc-c8128bd1a10b) |
| hostId                               | 4a74aa79a23a084f73f49a4fedba7447c132ab45c4701ed7fbbb2286           |
| OS-EXT-STS:vm_state                  | active                                                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000018                                                  |
| public network                       | 172.24.4.230                                                       |
| OS-SRV-USG:launched_at               | 2013-12-03T08:55:46.000000                                         |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | rdo                                                                |
| flavor                               | m1.small (2)                                                       |
| id                                   | 9d3c327f-1893-40ff-8a82-16fad9ce6d91                               |
| security_groups                      | [{u'name': u'default'}]                                            |
| OS-SRV-USG:terminated_at             | None                                                               |
| user_id                              | 99f8019ba2694d78a680a5de46aa1afd                                   |
| name                                 | small-ubuntu                                                       |
| created                              | 2013-12-03T08:55:39Z                                               |
| tenant_id                            | 0dafe42cfde242ddbb67b681f59bdb00                                   |
| OS-DCF:diskConfig                    | MANUAL                                                             |
| metadata                             | {}                                                                 |
| os-extended-volumes:volumes_attached | []                                                                 |
| accessIPv4                           |                                                                    |
| accessIPv6                           |                                                                    |
| progress                             | 0                                                                  |
| OS-EXT-STS:power_state               | 1                                                                  |
| OS-EXT-AZ:availability_zone          | nova                                                               |
| config_drive                         |                                                                    |
+--------------------------------------+--------------------------------------------------------------------+
[root@rdo /(keystone_admin)]#

[root@rdo /(keystone_admin)]# virsh list
 Id    Name                           State
----------------------------------------------------
 2     instance-00000018              running

[root@rdo /(keystone_admin)]# cat disk.xml
<disk type='network'>
 <driver name="qemu" type="raw"/>
 <source protocol="rbd" name="ceph-volumes/volume-10cc0855-652a-4a9b-baa1-80bc86dc12ac">
 <host name='192.168.1.38' port='6789'/>
 <host name='192.168.1.31' port='6789'/>
 <host name='192.168.1.33' port='6789'/>
 </source>
 <target dev="vdf" bus="virtio"/>
<auth username='volumes'>
<secret type='ceph' uuid='801a42ec-aec1-3ea8-d869-823c2de56b83'/>
</auth>
</disk>
[root@rdo /(keystone_admin)]#

Things you should know about this file
- source name=<ceph_pool_name/volume_name> ## ceph pool that we have created above and its cinder volume
- host name=<Your_Monitor_nodes>
- auth username=<user_you_created_in_Ceph_having_rights_to_pools_that_will_be_used_with_OS> ## we have created 2 users client.volumes and client.images for ceph that will have access to pools for openstack
- secret uuid=<secret_generated_by_virsh_secrec_define_command> ## refer above that we have generated.

Attaching disk device to instance

[root@rdo /(keystone_admin)]# virsh attach-device instance-00000018 disk.xml
Device attached successfully
[root@rdo /(keystone_admin)]#

Now the ceph volume is attached to your openstack instance , you can use this as a regular block disk.

Making integration more seamless

To allow openstack create and attach ceph volumes using nova / cinder CLI as well as horizon dashboard , we need to add in /etc/nova/nova.conf the following values

rbd_user=volumes
rbd_secret_uuid=801a42ec-aec1-3ea8-d869-823c2de56b83

After updating nova.conf , try creating volume from nova cli and attach to instance

[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]# nova volume-create --display_name=nova-vol_1 2
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| status | creating |
| display_name | nova-vol_1 |
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2013-12-04T14:13:07.265831 |
| display_description | None |
| volume_type | None |
| snapshot_id | None |
| source_volid | None |
| size | 2 |
| id | 0e2bfced-be6a-44ec-a3ca-22c771c66cdc |
| metadata | {} |
+---------------------+--------------------------------------+
[root@rdo nova(keystone_admin)]#

[root@rdo nova(keystone_admin)]# nova volume-list
+--------------------------------------+-----------+------------------+------+--------------+-------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+------------------+------+--------------+-------------+
| 0e2bfced-be6a-44ec-a3ca-22c771c66cdc | available | nova-vol_1 | 2 | None | |
| 9671edaa-62c8-4f98-a36c-d6e59612141b | available | boot_from_volume | 20 | None | |
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | ceph-vol1 | 5 | ceph-storage | |
+--------------------------------------+-----------+------------------+------+--------------+-------------+
[root@rdo nova(keystone_admin)]#

[root@rdo nova(keystone_admin)]# nova list
+--------------------------------------+------------------+---------+--------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------+---------+--------------+-------------+---------------------+
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2 | SHUTOFF | None | Shutdown | public=172.24.4.228 |
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu | ACTIVE | None | Running | public=172.24.4.230 |
| 10d1c49f-9fbc-455f-b72d-f731338b2dd5 | small-ubuntu-pwd | ACTIVE | powering-off | Shutdown | public=172.24.4.231 |
+--------------------------------------+------------------+---------+--------------+-------------+---------------------+
[root@rdo nova(keystone_admin)]#

[root@rdo nova(keystone_admin)]# nova volume-attach 9d3c327f-1893-40ff-8a82-16fad9ce6d91 0e2bfced-be6a-44ec-a3ca-22c771c66cdc /dev/vdi
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdi |
| serverId | 9d3c327f-1893-40ff-8a82-16fad9ce6d91 |
| id | 0e2bfced-be6a-44ec-a3ca-22c771c66cdc |
| volumeId | 0e2bfced-be6a-44ec-a3ca-22c771c66cdc |
+----------+--------------------------------------+
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]# nova volume-list
+--------------------------------------+-----------+------------------+------+--------------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+------------------+------+--------------+--------------------------------------+
| 0e2bfced-be6a-44ec-a3ca-22c771c66cdc | in-use | nova-vol_1 | 2 | None | 9d3c327f-1893-40ff-8a82-16fad9ce6d91 |
| 9671edaa-62c8-4f98-a36c-d6e59612141b | available | boot_from_volume | 20 | None | |
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | ceph-vol1 | 5 | ceph-storage | |
+--------------------------------------+-----------+------------------+------+--------------+--------------------------------------+
[root@rdo nova(keystone_admin)]#

Please Follow Ceph + OpenStack :: Part-4 for next step in installation

Ceph + OpenStack :: Part-2

noreply@blogger.com (Anonymous) — Thu, 05 Dec 2013 17:45:00 +0000

Configuring OpenStack

Two parts of openstack integrates with Ceph’s block devices:

Images: OpenStack Glance manages images for VMs.
Volumes: Volumes are block devices. OpenStack uses volumes to boot VMs, or to attach volumes to running VMs. OpenStack manages volumes using Cinder services.
- Create pools for volumes and images:

ceph osd pool create volumes 128
ceph osd pool create images 128

Configure OpenStack Ceph Client - The nodes running glance-api and cinder-volume act as Ceph clients. Each requires the ceph.conf file:

[root@ceph-mon1 ceph]# scp ceph.conf openstack:/etc/ceph

Installing ceph client packages on openstack node
- First install Python bindings for librbd

yum install python-ceph

- Install ceph

[root@ceph-mon1 ceph]# ceph-deploy install openstack

Setup Ceph Client Authentication for both pools along with keyrings
- Create a new user for Nova/Cinder and Glance.

ceph auth get-or-create client.volumes mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images'
ceph auth get-or-create client.images mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'

- Add these keyrings to glance-api and cinder-volume nodes.

ceph auth get-or-create client.images | ssh openstack tee /etc/ceph/ceph.client.images.keyring
ssh openstack chown glance:glance /etc/ceph/ceph.client.images.keyring
ceph auth get-or-create client.volumes | ssh openstack tee /etc/ceph/ceph.client.volumes.keyring
ssh openstack chown cinder:cinder /etc/ceph/ceph.client.volumes.keyring

- Hosts running nova-compute do not need the keyring. Instead, they store the secret key in libvirt. To create libvirt secret key you will need key from client.volumes.key

ceph auth get-key client.volumes | ssh openstack tee client.volumes.key

- on the compute nodes, add the secret key to libvirt create a secret.xml file

cat > secret.xml < <EOF
<secret ephemeral='no' private='no'>
  <usage type='ceph'>
    <name>client.volumes secret</name>
  </usage>
EOF

- Generate secret from created secret.xml file , make a note of uuid of secret output

# virsh secret-define --file secret.xml

- Set libvirt secret using above key

# virsh secret-set-value --secret {uuid of secret} --base64 $(cat client.volumes.key) && rm client.volumes.key secret.xml

Configure OpenStack-Glance to use CEPH
- Glance can use multiple back ends to store images. To use Ceph block devices by default, edit /etc/glance/glance-api.conf and add:

default_store=rbd
rbd_store_user=images
rbd_store_pool=images

- If want to enable copy-on-write cloning of images into volumes, also add:

show_image_direct_url=True

Configure OpenStack - Cinder to use CEPH
- OpenStack requires a driver to interact with Ceph block devices. You must specify the pool name for the block device. On your OpenStack node, edit/etc/cinder/cinder.conf by adding:

volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_pool=volumes
glance_api_version=2

If you’re using cephx authentication also configure the user and uuid of the secret you added to libvirt earlier:

rbd_user=volumes
rbd_secret_uuid={uuid of secret}

Restart Openstack

service glance-api restart
service nova-compute restart
service cinder-volume restart

Once OpenStack is up and running, you should be able to create a volume with OpenStack on a Ceph block device.
NOTE : Make sure /etc/ceph/ceph.conf file have sufficient rights to be ready by cinder and glance users.

Please Follow Ceph + OpenStack :: Part-3 for next step in installation

Ceph + OpenStack :: Part-1

noreply@blogger.com (Anonymous) — Thu, 05 Dec 2013 17:30:00 +0000

Ceph & OpenStack Integration

We can use Ceph Block Device with openstack through libvirt, which configures the QEMU interface tolibrbd. To use Ceph Block Devices with openstack , we must install QEMU, libvirt, and openstack first. ( we will not cover openstack installation in this document , you can use your existing openstack infrastructure ) The following diagram explains openstack /Ceph technology stack.

Installing QEMU

qemu-img version 0.12.1 does not have RBD support , so we need install packages with async . On Openstack Node , create 3 YUM repo files ceph-extras-source.repo , ceph-extras.repo, ceph-extras-noarch.repo

  [ceph-extras]
  name=Ceph Extra Packages and Backports $basearch
  baseurl=http://ceph.com/packages/ceph-extras/rpm/centos6/$basearch
  enabled=1
  gpgcheck=1
  type=rpm-md
  gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc

  [ceph-extras-noarch]
  name=Ceph Extra Packages and Backports noarch
  baseurl=http://ceph.com/packages/ceph-extras/rpm/centos6/noarch
  enabled=1
  gpgcheck=1
  type=rpm-md
  gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc

 [ceph-extras-source]
 name=Ceph Extra Packages and Backports Sources
 baseurl=http://ceph.com/packages/ceph-extras/rpm/centos6/SRPMS
 enabled=1
 gpgcheck=1
 type=rpm-md
 gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
 centos-extras

#yum update
#yum remove qemu-img
#yum --disablerepo=* --enablerepo=ceph-extras install -y qemu-img
# yum --disablerepo=* --enablerepo=ceph-extras install -y qemu-kvm
# yum --disablerepo=* --enablerepo=ceph-extras install -y qemu-guest-agent
# yum --disablerepo=* --enablerepo=ceph-extras install -y qemu-kvm-tools

--> Check creating a QEMU image it should work

[root@rdo yum.repos.d]# qemu-img create -f rbd rbd:data/foo 10G
Formatting 'rbd:data/foo', fmt=rbd size=10737418240 cluster_size=0
[root@rdo yum.repos.d]#

[root@rdo yum.repos.d]# qemu-img info -f rbd rbd:data/foo
image: rbd:data/foo
file format: rbd
virtual size: 10G (10737418240 bytes)
disk size: unavailable
cluster_size: 4194304
[root@rdo yum.repos.d]#

Installing LIBVIRT

To use libvirt with Ceph, we must have a running Ceph Storage Cluster, and have installed and configured QEMU

yum install libvirt

Please Follow Ceph + OpenStack :: Part-2 for next step in installation

Ceph Installation :: Part-3

noreply@blogger.com (Anonymous) — Thu, 05 Dec 2013 17:22:00 +0000

Creating Block Device from Ceph

From monitor node , use ceph-deploy to install Ceph on your ceph-client1 node.

[root@ceph-mon1 ~]# ceph-deploy install ceph-client1
[ceph_deploy.cli][INFO  ] Invoked (1.3): /usr/bin/ceph-deploy install ceph-client1
[ceph_deploy.install][DEBUG ] Installing stable version dumpling on cluster ceph hosts ceph-client1
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-client1 ...
[ceph-client1][DEBUG ] connected to host: ceph-client1
[ceph-client1][DEBUG ] detect platform information from remote host
[ceph-client1][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Ubuntu 13.04 raring
[ceph-client1][INFO  ] installing ceph on ceph-client1
[ceph-client1][INFO  ] Running command: env DEBIAN_FRONTEND=noninteractive apt-get -q install --assume-yes ca-certificates
[ceph-client1][DEBUG ] Reading package lists...
[ceph-client1][DEBUG ] Building dependency tree...
[ceph-client1][DEBUG ] Reading state information...
[ceph-client1][DEBUG ] ca-certificates is already the newest version.
[ceph-client1][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 105 not upgraded.
[ceph-client1][INFO  ] Running command: wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | apt-key add -
[ceph-client1][WARNIN] command returned non-zero exit status: 4
[ceph-client1][DEBUG ] add ceph deb repo to sources.list
[ceph-client1][INFO  ] Running command: apt-get -q update
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security Release.gpg
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security Release
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring Release.gpg
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates Release.gpg
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring Release
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates Release
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/main Sources
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/main Sources
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/universe Sources
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/main amd64 Packages
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/universe Sources
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/universe amd64 Packages
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/main i386 Packages
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/universe i386 Packages
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/main amd64 Packages
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/main Translation-en
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring/universe Translation-en
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/universe amd64 Packages
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/main Sources
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/universe Sources
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/main amd64 Packages
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/main i386 Packages
[ceph-client1][DEBUG ] Get:1 http://ceph.com raring Release.gpg [836 B]
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/universe amd64 Packages
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/universe i386 Packages
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/main i386 Packages
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/universe i386 Packages
[ceph-client1][DEBUG ] Hit http://ceph.com raring Release
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/main Translation-en
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/main Translation-en
[ceph-client1][DEBUG ] Hit http://nova.clouds.archive.ubuntu.com raring-updates/universe Translation-en
[ceph-client1][DEBUG ] Hit http://security.ubuntu.com raring-security/universe Translation-en
[ceph-client1][DEBUG ] Hit http://ceph.com raring/main amd64 Packages
[ceph-client1][DEBUG ] Hit http://ceph.com raring/main i386 Packages
[ceph-client1][DEBUG ] Ign http://ceph.com raring/main Translation-en
[ceph-client1][DEBUG ] Fetched 836 B in 15s (55 B/s)
[ceph-client1][DEBUG ] Reading package lists...
[ceph-client1][INFO  ] Running command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get -q -o Dpkg::Options::=--force-confnew --no-install-recommends --assume-yes install -- ceph ceph-mds ceph-common ceph-fs-common gdisk
[ceph-client1][DEBUG ] Reading package lists...
[ceph-client1][DEBUG ] Building dependency tree...
[ceph-client1][DEBUG ] Reading state information...
[ceph-client1][DEBUG ] gdisk is already the newest version.
[ceph-client1][DEBUG ] The following extra packages will be installed:
[ceph-client1][DEBUG ]   binutils libaio1 libboost-thread1.49.0 libgoogle-perftools4 libjs-jquery
[ceph-client1][DEBUG ]   libleveldb1 libnspr4 libnss3 librados2 librbd1 libreadline5 libsnappy1
[ceph-client1][DEBUG ]   libtcmalloc-minimal4 libunwind8 python-ceph python-flask python-jinja2
[ceph-client1][DEBUG ]   python-markupsafe python-werkzeug xfsprogs
[ceph-client1][DEBUG ] Suggested packages:
[ceph-client1][DEBUG ]   binutils-doc javascript-common python-jinja2-doc ipython python-genshi
[ceph-client1][DEBUG ]   python-lxml python-memcache libjs-sphinxdoc xfsdump acl attr quota
[ceph-client1][DEBUG ] Recommended packages:
[ceph-client1][DEBUG ]   btrfs-tools ceph-fuse libcephfs1
[ceph-client1][DEBUG ] The following NEW packages will be installed:
[ceph-client1][DEBUG ]   binutils ceph ceph-common ceph-fs-common ceph-mds libaio1
[ceph-client1][DEBUG ]   libboost-thread1.49.0 libgoogle-perftools4 libjs-jquery libleveldb1 libnspr4
[ceph-client1][DEBUG ]   libnss3 librados2 librbd1 libreadline5 libsnappy1 libtcmalloc-minimal4
[ceph-client1][DEBUG ]   libunwind8 python-ceph python-flask python-jinja2 python-markupsafe
[ceph-client1][DEBUG ]   python-werkzeug xfsprogs
[ceph-client1][DEBUG ] 0 upgraded, 24 newly installed, 0 to remove and 105 not upgraded.
[ceph-client1][DEBUG ] Need to get 40.9 MB of archives.
[ceph-client1][DEBUG ] After this operation, 192 MB of additional disk space will be used.
[ceph-client1][DEBUG ] Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libaio1 amd64 0.3.109-3 [6328 B]
[ceph-client1][DEBUG ] Get:2 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libsnappy1 amd64 1.0.5-2 [13.2 kB]
[ceph-client1][DEBUG ] Get:3 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libleveldb1 amd64 1.9.0-1 [138 kB]
[ceph-client1][DEBUG ] Get:4 http://ceph.com/debian-dumpling/ raring/main librados2 amd64 0.67.4-1raring [1635 kB]
[ceph-client1][DEBUG ] Get:5 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libnspr4 amd64 2:4.9.5-1ubuntu1 [134 kB]
[ceph-client1][DEBUG ] Get:6 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libnss3 amd64 2:3.14.3-0ubuntu1 [1044 kB]
[ceph-client1][DEBUG ] Get:7 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libreadline5 amd64 5.2+dfsg-1 [131 kB]
[ceph-client1][DEBUG ] Get:8 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libunwind8 amd64 1.0.1-4ubuntu2 [55.9 kB]
[ceph-client1][DEBUG ] Get:9 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main binutils amd64 2.23.2-2ubuntu1 [2393 kB]
[ceph-client1][DEBUG ] Get:10 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libboost-thread1.49.0 amd64 1.49.0-3.2ubuntu1 [41.6 kB]
[ceph-client1][DEBUG ] Get:11 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libjs-jquery all 1.7.2+debian-1ubuntu1 [115 kB]
[ceph-client1][DEBUG ] Get:12 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main python-werkzeug all 0.8.3+dfsg-1 [1333 kB]
[ceph-client1][DEBUG ] Get:13 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main python-markupsafe amd64 0.15-1build3 [13.8 kB]
[ceph-client1][DEBUG ] Get:14 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main python-jinja2 amd64 2.6-1build3 [158 kB]
[ceph-client1][DEBUG ] Get:15 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main python-flask all 0.9-1 [56.1 kB]
[ceph-client1][DEBUG ] Get:16 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main xfsprogs amd64 3.1.9 [1238 kB]
[ceph-client1][DEBUG ] Get:17 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libtcmalloc-minimal4 amd64 2.0-4ubuntu1 [163 kB]
[ceph-client1][DEBUG ] Get:18 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main libgoogle-perftools4 amd64 2.0-4ubuntu1 [412 kB]
[ceph-client1][DEBUG ] Get:19 http://ceph.com/debian-dumpling/ raring/main librbd1 amd64 0.67.4-1raring [276 kB]
[ceph-client1][DEBUG ] Get:20 http://ceph.com/debian-dumpling/ raring/main python-ceph amd64 0.67.4-1raring [39.7 kB]
[ceph-client1][DEBUG ] Get:21 http://ceph.com/debian-dumpling/ raring/main ceph-common amd64 0.67.4-1raring [6090 kB]
[ceph-client1][DEBUG ] Get:22 http://ceph.com/debian-dumpling/ raring/main ceph amd64 0.67.4-1raring [22.8 MB]
[ceph-client1][DEBUG ] Get:23 http://ceph.com/debian-dumpling/ raring/main ceph-fs-common amd64 0.67.4-1raring [28.2 kB]
[ceph-client1][DEBUG ] Get:24 http://ceph.com/debian-dumpling/ raring/main ceph-mds amd64 0.67.4-1raring [2676 kB]
[ceph-client1][DEBUG ] Fetched 40.9 MB in 12s (3212 kB/s)
[ceph-client1][DEBUG ] Selecting previously unselected package libaio1:amd64.
[ceph-client1][DEBUG ] (Reading database ... 58918 files and directories currently installed.)
[ceph-client1][DEBUG ] Unpacking libaio1:amd64 (from .../libaio1_0.3.109-3_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libsnappy1.
[ceph-client1][DEBUG ] Unpacking libsnappy1 (from .../libsnappy1_1.0.5-2_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libleveldb1:amd64.
[ceph-client1][DEBUG ] Unpacking libleveldb1:amd64 (from .../libleveldb1_1.9.0-1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libnspr4:amd64.
[ceph-client1][DEBUG ] Unpacking libnspr4:amd64 (from .../libnspr4_2%3a4.9.5-1ubuntu1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libnss3:amd64.
[ceph-client1][DEBUG ] Unpacking libnss3:amd64 (from .../libnss3_2%3a3.14.3-0ubuntu1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libreadline5:amd64.
[ceph-client1][DEBUG ] Unpacking libreadline5:amd64 (from .../libreadline5_5.2+dfsg-1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libunwind8.
[ceph-client1][DEBUG ] Unpacking libunwind8 (from .../libunwind8_1.0.1-4ubuntu2_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package binutils.
[ceph-client1][DEBUG ] Unpacking binutils (from .../binutils_2.23.2-2ubuntu1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libboost-thread1.49.0.
[ceph-client1][DEBUG ] Unpacking libboost-thread1.49.0 (from .../libboost-thread1.49.0_1.49.0-3.2ubuntu1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package librados2.
[ceph-client1][DEBUG ] Unpacking librados2 (from .../librados2_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package librbd1.
[ceph-client1][DEBUG ] Unpacking librbd1 (from .../librbd1_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libjs-jquery.
[ceph-client1][DEBUG ] Unpacking libjs-jquery (from .../libjs-jquery_1.7.2+debian-1ubuntu1_all.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package python-werkzeug.
[ceph-client1][DEBUG ] Unpacking python-werkzeug (from .../python-werkzeug_0.8.3+dfsg-1_all.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package python-markupsafe.
[ceph-client1][DEBUG ] Unpacking python-markupsafe (from .../python-markupsafe_0.15-1build3_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package python-jinja2.
[ceph-client1][DEBUG ] Unpacking python-jinja2 (from .../python-jinja2_2.6-1build3_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package python-flask.
[ceph-client1][DEBUG ] Unpacking python-flask (from .../python-flask_0.9-1_all.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package python-ceph.
[ceph-client1][DEBUG ] Unpacking python-ceph (from .../python-ceph_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package ceph-common.
[ceph-client1][DEBUG ] Unpacking ceph-common (from .../ceph-common_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package xfsprogs.
[ceph-client1][DEBUG ] Unpacking xfsprogs (from .../xfsprogs_3.1.9_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libtcmalloc-minimal4.
[ceph-client1][DEBUG ] Unpacking libtcmalloc-minimal4 (from .../libtcmalloc-minimal4_2.0-4ubuntu1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package libgoogle-perftools4.
[ceph-client1][DEBUG ] Unpacking libgoogle-perftools4 (from .../libgoogle-perftools4_2.0-4ubuntu1_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package ceph.
[ceph-client1][DEBUG ] Unpacking ceph (from .../ceph_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package ceph-fs-common.
[ceph-client1][DEBUG ] Unpacking ceph-fs-common (from .../ceph-fs-common_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Selecting previously unselected package ceph-mds.
[ceph-client1][DEBUG ] Unpacking ceph-mds (from .../ceph-mds_0.67.4-1raring_amd64.deb) ...
[ceph-client1][DEBUG ] Processing triggers for man-db ...
[ceph-client1][DEBUG ] Processing triggers for ureadahead ...
[ceph-client1][DEBUG ] Setting up libaio1:amd64 (0.3.109-3) ...
[ceph-client1][DEBUG ] Setting up libsnappy1 (1.0.5-2) ...
[ceph-client1][DEBUG ] Setting up libleveldb1:amd64 (1.9.0-1) ...
[ceph-client1][DEBUG ] Setting up libnspr4:amd64 (2:4.9.5-1ubuntu1) ...
[ceph-client1][DEBUG ] Setting up libnss3:amd64 (2:3.14.3-0ubuntu1) ...
[ceph-client1][DEBUG ] Setting up libreadline5:amd64 (5.2+dfsg-1) ...
[ceph-client1][DEBUG ] Setting up libunwind8 (1.0.1-4ubuntu2) ...
[ceph-client1][DEBUG ] Setting up binutils (2.23.2-2ubuntu1) ...
[ceph-client1][DEBUG ] Setting up libboost-thread1.49.0 (1.49.0-3.2ubuntu1) ...
[ceph-client1][DEBUG ] Setting up librados2 (0.67.4-1raring) ...
[ceph-client1][DEBUG ] Setting up librbd1 (0.67.4-1raring) ...
[ceph-client1][DEBUG ] Setting up libjs-jquery (1.7.2+debian-1ubuntu1) ...
[ceph-client1][DEBUG ] Setting up python-werkzeug (0.8.3+dfsg-1) ...
[ceph-client1][DEBUG ] Setting up python-markupsafe (0.15-1build3) ...
[ceph-client1][DEBUG ] Setting up python-jinja2 (2.6-1build3) ...
[ceph-client1][DEBUG ] Setting up python-flask (0.9-1) ...
[ceph-client1][DEBUG ] Setting up python-ceph (0.67.4-1raring) ...
[ceph-client1][DEBUG ] Setting up ceph-common (0.67.4-1raring) ...
[ceph-client1][DEBUG ] Setting up xfsprogs (3.1.9) ...
[ceph-client1][DEBUG ] Setting up libtcmalloc-minimal4 (2.0-4ubuntu1) ...
[ceph-client1][DEBUG ] Setting up libgoogle-perftools4 (2.0-4ubuntu1) ...
[ceph-client1][DEBUG ] Setting up ceph (0.67.4-1raring) ...
[ceph-client1][DEBUG ] ceph-all start/running
[ceph-client1][DEBUG ] Setting up ceph-fs-common (0.67.4-1raring) ...
[ceph-client1][DEBUG ] Processing triggers for ureadahead ...
[ceph-client1][DEBUG ] Setting up ceph-mds (0.67.4-1raring) ...
[ceph-client1][DEBUG ] ceph-mds-all start/running
[ceph-client1][DEBUG ] Processing triggers for libc-bin ...
[ceph-client1][DEBUG ] ldconfig deferred processing now taking place
[ceph-client1][DEBUG ] Processing triggers for ureadahead ...
[ceph-client1][INFO  ] Running command: ceph --version
[ceph-client1][DEBUG ] ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
[root@ceph-mon1 ~]#

From monitor node , use ceph-deploy to copy the Ceph configuration file and the ceph.client.admin.keyring to the ceph-client1.

[root@ceph-mon1 ceph]# ceph-deploy admin ceph-client1
[ceph_deploy.cli][INFO  ] Invoked (1.3): /usr/bin/ceph-deploy admin ceph-client1
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-client1
[ceph-client1][DEBUG ] connected to host: ceph-client1
[ceph-client1][DEBUG ] detect platform information from remote host
[ceph-client1][DEBUG ] detect machine type
[ceph-client1][DEBUG ] get remote short hostname
[ceph-client1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[root@ceph-mon1 ceph]#

On the ceph-client1 node, create a block device image.

rbd create block1-ceph-client1 --size 200

On the ceph-client1 node, load the rbd client module.

modprobe rbd

On the ceph-client1 node, map the image to a block device.

root@ceph-client1:~# rbd map block1-ceph-client1
root@ceph-client1:~#
root@ceph-client1:~#
root@ceph-client1:~# rbd showmapped
id pool image               snap device
1  rbd  block1-ceph-client1 -    /dev/rbd1
root@ceph-client1:~#

Use the block device by creating a file system on the ceph-client1 node.

root@ceph-client1:/dev/rbd/rbd# mkfs.ext4 /dev/rbd/rbd/block1-ceph-client1
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=4096 blocks, Stripe width=4096 blocks
51200 inodes, 204800 blocks
10240 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
25 block groups
8192 blocks per group, 8192 fragments per group
2048 inodes per group
Superblock backups stored on blocks:
   8193, 24577, 40961, 57345, 73729

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

root@ceph-client1:/dev/rbd/rbd#

Mount the file system on the ceph-client1 node.

root@ceph-client1:/dev/rbd/rbd# mount /dev/rbd/rbd/block1-ceph-client1 /rbd-1/
root@ceph-client1:/dev/rbd/rbd#
root@ceph-client1:/dev/rbd/rbd#
root@ceph-client1:/dev/rbd/rbd#
root@ceph-client1:/dev/rbd/rbd# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       9.8G  1.3G  8.0G  14% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            487M  8.0K  487M   1% /dev
tmpfs           100M  256K  100M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            498M     0  498M   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/vdb        109G  188M  103G   1% /mnt
/dev/rbd1       190M  1.6M  179M   1% /rbd-1
root@ceph-client1:/dev/rbd/rbd#

Ceph Installation :: Part-2

noreply@blogger.com (Anonymous) — Thu, 05 Dec 2013 16:41:00 +0000

CEPH Storage Cluster

Installing Ceph Deploy ( ceph-mon1 )

Update your repository and install ceph-deploy on ceph-mon1 node

[ceph@ceph-mon1 ~]$ sudo yum update && sudo yum install ceph-deploy
Loaded plugins: downloadonly, fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: ftp.funet.fi
 * epel: www.nic.funet.fi
 * extras: ftp.funet.fi
 * updates: mirror.academica.fi
Setting up Update Process
No Packages marked for Update
Loaded plugins: downloadonly, fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: ftp.funet.fi
 * epel: www.nic.funet.fi
 * extras: ftp.funet.fi
 * updates: mirror.academica.fi
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package ceph-deploy.noarch 0:1.2.7-0 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===============================================================================================================================================
 Package                             Arch                           Version                          Repository                           Size
===============================================================================================================================================
Installing:
 ceph-deploy                         noarch                         1.2.7-0                          ceph-noarch                         176 k

Transaction Summary
===============================================================================================================================================
Install       1 Package(s)

Total download size: 176 k
Installed size: 553 k
Is this ok [y/N]: y
Downloading Packages:
ceph-deploy-1.2.7-0.noarch.rpm                           75% [===================================            ]  64 kB/s | 133 kB     00:00 ETA 
 Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
  Installing : ceph-deploy-1.2.7-0.noarch [############################################################################################# ] 1/1
Verifying  : ceph-deploy-1.2.7-0.noarch                                                                                                  1/1 
Installed:
  ceph-deploy.noarch 0:1.2.7-0                                                                                                                 
Complete!
root@ceph-mon1 /]# ce      rpm -qa | grep -i ceph
ceph-release-1-0.el6.noarch
ceph-deploy-1.2.7-0.noarch
[root@ceph-mon1 /]#

Creating Ceph Cluster

As a first exercise, create a Ceph Storage Cluster with one Ceph Monitor and two Ceph OSD nodes. Post this will expand it by adding two more Ceph Monitors.

Tip : The ceph-deploy utility will output files to the current directory. Ensure you are in /etc/ceph directory when executing ceph-deploy.

Important Do not call ceph-deploy with sudo or run it as root if you are logged in as a different user, because it will not issue sudo commands needed on the remote host.

Create the cluster using ceph-deploy , Check the output of ceph-deploy with ls and cat in the current directory. You should see a Ceph configuration file, a keyring, and a log file for the new cluster.

 [root@ceph-mon1 ceph]# ceph-deploy new ceph-mon1
[ [1mceph_deploy.cli [0m][ [1;37mINFO [0m  ] Invoked (1.2.7): /usr/bin/ceph-deploy new ceph-mon1
[ [1mceph_deploy.new [0m][ [1;34mDEBUG [0m ] Creating new cluster named ceph
[ [1mceph_deploy.new [0m][ [1;34mDEBUG [0m ] Resolving host ceph-mon1
[ [1mceph_deploy.new [0m][ [1;34mDEBUG [0m ] Monitor ceph-mon1 at 192.168.1.38
[ [1mceph_deploy.new [0m][ [1;34mDEBUG [0m ] Monitor initial members are ['ceph-mon1']
[ [1mceph_deploy.new [0m][ [1;34mDEBUG [0m ] Monitor addrs are ['192.168.1.38']
[ [1mceph_deploy.new [0m][ [1;34mDEBUG [0m ] Creating a random mon key...
[ [1mceph_deploy.new [0m][ [1;34mDEBUG [0m ] Writing initial config to ceph.conf...
[ [1mceph_deploy.new [0m][ [1;34mDEBUG [0m ] Writing monitor keyring to ceph.mon.keyring...
 [root@ceph-mon1 ceph]# ll
-rw-r--r-- 1 root root 189 Oct 25 13:06 ceph.conf
-rw-r--r-- 1 root root 785 Oct 25 13:06 ceph.log
-rw-r--r-- 1 root root  73 Oct 25 13:06 ceph.mon.keyring
[root@ceph-mon1 ceph]#

Install Ceph on ceph-mon1 node Dont panic if you see errors due to wget , centos does not understand wget and throws some errors that can be ignored , but finally it install ceph on your node and shows the installed version.

[root@ceph-mon1 ceph]# ceph-deploy install ceph-mon1
[ [1mceph_deploy.cli [0m][ [1;37mINFO [0m  ] Invoked (1.2.7): /usr/bin/ceph-deploy install ceph-mon1
[ [1mceph_deploy.install [0m][ [1;34mDEBUG [0m ] Installing stable version dumpling on cluster ceph hosts ceph-mon1
[ [1mceph_deploy.install [0m][ [1;34mDEBUG [0m ] Detecting platform for host ceph-mon1 ...
[ [1mceph_deploy.sudo_pushy [0m][ [1;34mDEBUG [0m ] will use a local connection without sudo
[ [1mceph_deploy.lsb [0m][ [1;33mWARNIN [0m] lsb_release was not found - inferring OS details
[ [1mceph_deploy.install [0m][ [1;37mINFO [0m  ] Distro info: CentOS 6.4 Final
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] installing ceph on ceph-mon1
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] adding EPEL repository
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] Running command: wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ] Traceback (most recent call last):
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py", line 77, in install_epel
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, in inner
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]     def inner(*args, **kwargs):
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 6, in remote_call
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]     This allows us to only remote-execute the actual calls, not whole functions.
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib64/python2.6/subprocess.py", line 500, in check_call
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]     retcode = call(*popenargs, **kwargs)
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib64/python2.6/subprocess.py", line 478, in call
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]     p = Popen(*popenargs, **kwargs)
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]     errread, errwrite)
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]     raise child_exception
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] Running command: rpm -Uvh --replacepkgs epel-release-6*.rpm
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ] Traceback (most recent call last):
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib/python2.6/site-packages/ceph_deploy/util/pkg_managers.py", line 69, in rpm
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, in inner
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]     def inner(*args, **kwargs):
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 6, in remote_call
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]     This allows us to only remote-execute the actual calls, not whole functions.
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]     raise CalledProcessError(retcode, cmd)
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] Running command: su -c 'rpm --import "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc"'
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] Running command: rpm -Uvh --replacepkgs http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] Retrieving http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] Preparing...                ##################################################
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] ceph-release                ##################################################
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] Running command: yum -y -q install ceph
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] Running command: ceph --version
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
 ]0;root@ceph-mon1:/etc/ceph [root@ceph-mon1 ceph]#

Adding node ceph-mon1 as our first monitor node

 [root@ceph-mon1 ceph]# ceph-deploy mon  create ceph-mon1
[ [1mceph_deploy.cli [0m][ [1;37mINFO [0m  ] Invoked (1.2.7): /usr/bin/ceph-deploy mon create ceph-mon1
[ [1mceph_deploy.mon [0m][ [1;34mDEBUG [0m ] Deploying mon, cluster ceph hosts ceph-mon1
[ [1mceph_deploy.mon [0m][ [1;34mDEBUG [0m ] detecting platform for host ceph-mon1 ...
[ [1mceph_deploy.sudo_pushy [0m][ [1;34mDEBUG [0m ] will use a local connection without sudo
[ [1mceph_deploy.lsb [0m][ [1;33mWARNIN [0m] lsb_release was not found - inferring OS details
[ [1mceph_deploy.mon [0m][ [1;37mINFO [0m  ] distro info: CentOS 6.4 Final
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] determining if provided host has same hostname in remote
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] deploying mon to ceph-mon1
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] remote hostname: ceph-mon1
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] write cluster configuration to /etc/ceph/{cluster}.conf
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ] Traceback (most recent call last):
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, in inner
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]     def inner(*args, **kwargs):
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]   File "/usr/lib/python2.6/site-packages/ceph_deploy/conf.py", line 12, in write_conf
[ [1mceph-mon1 [0m][ [1;31mERROR [0m ]     line = self.fp.readline()
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] checking for done path: /var/lib/ceph/mon/ceph-ceph-mon1/done
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] done path does not exist: /var/lib/ceph/mon/ceph-ceph-mon1/done
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] creating keyring file: /var/lib/ceph/tmp/ceph-ceph-mon1.mon.keyring
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] create the monitor keyring file
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] Running command: ceph-mon --cluster ceph --mkfs -i ceph-mon1 --keyring /var/lib/ceph/tmp/ceph-ceph-mon1.mon.keyring
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] ceph-mon: mon.noname-a 192.168.1.38:6789/0 is local, renaming to mon.ceph-mon1
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] ceph-mon: set fsid to 91ad085b-81ad-43db-9aa0-f3895a53613e
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph-mon1 for mon.ceph-mon1
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] unlinking keyring file /var/lib/ceph/tmp/ceph-ceph-mon1.mon.keyring
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] create a done file to avoid re-doing the mon deployment
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] create the init path if it does not exist
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] locating `service` executable...
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] found `service` executable: /sbin/service
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] Running command: /sbin/service ceph -c /etc/ceph/ceph.conf start mon.ceph-mon1
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] === mon.ceph-mon1 === 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] Starting Ceph mon.ceph-mon1 on ceph-mon1...
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] Starting ceph-create-keys on ceph-mon1...
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] Running command: ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon1.asok mon_status
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] ********************************************************************************
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] status for monitor: mon.ceph-mon1
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] {
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]   "election_epoch": 2, 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]   "extra_probe_peers": [], 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]   "monmap": {
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]     "created": "0.000000", 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]     "epoch": 1, 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]     "fsid": "91ad085b-81ad-43db-9aa0-f3895a53613e", 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]     "modified": "0.000000", 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]     "mons": [
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]       {
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]         "addr": "192.168.1.38:6789/0", 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]         "name": "ceph-mon1", 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]         "rank": 0
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]       }
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]     ]
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]   }, 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]   "name": "ceph-mon1", 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]   "outside_quorum": [], 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]   "quorum": [
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]     0
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]   ], 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]   "rank": 0, 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]   "state": "leader", 
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ]   "sync_provider": []
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] }
[ [1mceph-mon1 [0m][ [1;34mDEBUG [0m ] ********************************************************************************
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] monitor: mon.ceph-mon1 is running
[ [1mceph-mon1 [0m][ [1;37mINFO [0m  ] Running command: ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon1.asok mon_status

This would generate (cluster_name).mon.keyring and (cluster_name).client.admin.keyring files in current directory

[root@ceph-mon1 ceph]# ll
total 28
-rw------- 1 root root    64 Oct 25 16:32 ceph.client.admin.keyring
-rw-r--r-- 1 root root   189 Oct 25 16:25 ceph.conf
-rw-r--r-- 1 root root 10937 Oct 25 16:32 ceph.log
-rw-r--r-- 1 root root    73 Oct 25 16:25 ceph.mon.keyring
-rwxr-xr-x 1 root root    92 Oct  4 14:39  rbdmap

You can now check your cluster status , but you will see health Errors that will be resolved later by adding monitor and osd.

[root@ceph-mon1 ceph]# ceph status
  cluster 91ad085b-81ad-43db-9aa0-f3895a53613e
   health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds
   monmap e1: 1 mons at {ceph-mon1=192.168.1.38:6789/0}, election epoch 2, quorum 0 ceph-mon1
   osdmap e1: 0 osds: 0 up, 0 in
    pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB avail
   mdsmap e1: 0/0/1 up
 [root@ceph-mon1 ceph]#

Gather keys Once you have gathered keys, your local directory should have the following keyrings:
- {cluster-name}.client.admin.keyring
- {cluster-name}.bootstrap-osd.keyring
- {cluster-name}.bootstrap-mds.keyring

 [root@ceph-mon1 ceph]# ceph-d e ploy gatherkeys ceph-mon1
[ [1mceph_deploy.cli [0m][ [1;37mINFO [0m  ] Invoked (1.2.7): /usr/bin/ceph-deploy gatherkeys ceph-mon1
[ [1mceph_deploy.gatherkeys [0m][ [1;34mDEBUG [0m ] Have ceph.client.admin.keyring
[ [1mceph_deploy.gatherkeys [0m][ [1;34mDEBUG [0m ] Have ceph.mon.keyring
[ [1mceph_deploy.gatherkeys [0m][ [1;34mDEBUG [0m ] Checking ceph-mon1 for /var/lib/ceph/bootstrap-osd/ceph.keyring
[ [1mceph_deploy.sudo_pushy [0m][ [1;34mDEBUG [0m ] will use a local connection without sudo
[ [1mceph_deploy.gatherkeys [0m][ [1;34mDEBUG [0m ] Got ceph.bootstrap-osd.keyring key from ceph-mon1.
[ [1mceph_deploy.gatherkeys [0m][ [1;34mDEBUG [0m ] Checking ceph-mon1 for /var/lib/ceph/bootstrap-mds/ceph.keyring
[ [1mceph_deploy.sudo_pushy [0m][ [1;34mDEBUG [0m ] will use a local connection without sudo
[ [1mceph_deploy.gatherkeys [0m][ [1;34mDEBUG [0m ] Got ceph.bootstrap-mds.keyring key from ceph-mon1.

[root@ceph-mon1 ceph]# ll
total 36
-rw-r--r-- 1 root root    72 Oct 25 16:33 ceph.bootstrap-mds.keyring
-rw-r--r-- 1 root root    72 Oct 25 16:33 ceph.bootstrap-osd.keyring
-rw------- 1 root root    64 Oct 25 16:32 ceph.client.admin.keyring
-rw-r--r-- 1 root root   189 Oct 25 16:25 ceph.conf
-rw-r--r-- 1 root root 11867 Oct 25 16:33 ceph.log
-rw-r--r-- 1 root root    73 Oct 25 16:25 ceph.mon.keyring
-rwxr-xr-x 1 root root    92 Oct  4 14:39  rbdmap 
[root@ceph-mon1 ceph]#

Modify ceph configuration file /etc/ceph/ceph.conf , add the below entries

[global] 
auth_service_required = cephx
auth_client_required = cephx
auth_cluster_required = cephx

[mon.ceph-mon1]
mon_addr = 192.168.1.38:6789
host = ceph-mon1

[osd]
filestore_xattr_use_omap = true
osd_data = /var/lib/ceph/osd/$cluster-$id
osd_journal_size = 1024

Restart the ceph service on the server and check that your monitor is restarting well. Note : updating ceph.conf file does not require ceph service bounce back , we are here just testing if monitor services are OK

[root@ceph-mon1 ceph]# service ceph restart
=== mon.ceph-mon1 === 
=== mon.ceph-mon1 === 
Stopping Ceph mon.ceph-mon1 on ceph-mon1...kill 27965...done
=== mon.ceph-mon1 === 
Starting Ceph mon.ceph-mon1 on ceph-mon1...
Starting ceph-create-keys on ceph-mon1...
=== mon.ceph-mon1 === 
=== mon.ceph-mon1 === 
Stopping Ceph mon.ceph-mon1 on ceph-mon1...kill 28439...done
=== mon.ceph-mon1 === 
Starting Ceph mon.ceph-mon1 on ceph-mon1...
Starting ceph-create-keys on ceph-mon1...
[root@ceph-mon1 ceph]# 
[root@ceph-mon1 ceph]# service ceph status
=== mon.ceph-mon1 === 
mon.ceph-mon1: running {"version":"0.67.4"}
=== mon.ceph-mon1 === 
mon.ceph-mon1: running {"version":"0.67.4"}
 [root@ceph-mon1 ceph]#

Preparing OSD node ( ceph-node1 & ceph-node2 )

Use ceph-deploy node ( ceph-mon1 ) to list available disk on ceph-node1 that will be used as OSD

[root@ceph-mon1 ceph]# ceph-d e ploy disk list ceph-node1
[ [1mceph_deploy.cli [0m][ [1;37mINFO [0m  ] Invoked (1.2.7): /usr/bin/ceph-deploy disk list ceph-node1
[ [1mceph_deploy.sudo_pushy [0m][ [1;34mDEBUG [0m ] will use a remote connection without sudo
[ [1mceph_deploy.lsb [0m][ [1;33mWARNIN [0m] lsb_release was not found - inferring OS details
[ [1mceph_deploy.osd [0m][ [1;37mINFO [0m  ] Distro info: CentOS 6.4 Final
[ [1mceph_deploy.osd [0m][ [1;34mDEBUG [0m ] Listing disks on ceph-node1...
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] Running command: ceph-disk list
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] /dev/vda :
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ]  /dev/vda1 swap, swap
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ]  /dev/vda2 other, ext4, mounted on /
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] /dev/vdb other, ext3
[root@ceph-mon1 ceph]#

To zap a disk (delete its partition table) in preparation for use with Ceph, execute the following . This will delete all data.

[root@ceph-mon1 ceph]# ceph-deploy disk zap ceph-node1:vdb
[ [1mceph_deploy.cli [0m][ [1;37mINFO [0m  ] Invoked (1.2.7): /usr/bin/ceph-deploy disk zap ceph-node1:vdb
[ [1mceph_deploy.osd [0m][ [1;34mDEBUG [0m ] zapping /dev/vdb on ceph-node1
[ [1mceph-node1 [0m][ [1;34mDEBUG [0m ] detect platform information from remote host
[ [1mceph_deploy.osd [0m][ [1;37mINFO [0m  ] Distro info: CentOS 6.4 Final
[ [1mceph-node1 [0m][ [1;34mDEBUG [0m ] zeroing last few blocks of device
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] Running command: sgdisk --zap-all --clear --mbrtogpt -- /dev/vdb
[ [1mceph-node1 [0m][ [1;34mDEBUG [0m ] Creating new GPT entries.
[ [1mceph-node1 [0m][ [1;34mDEBUG [0m ] GPT data structures destroyed! You may now partition the disk using fdisk or
[ [1mceph-node1 [0m][ [1;34mDEBUG [0m ] other utilities.
[ [1mceph-node1 [0m][ [1;34mDEBUG [0m ] The operation has completed successfully.
 [root@ceph-mon1 ceph]#

Prepare the OSDs and deploy them to the OSD node

[root@ceph-mon1 ceph]# ceph-deploy osd prepare ceph-node1:vdb
[ [1mceph_deploy.cli [0m][ [1;37mINFO [0m  ] Invoked (1.2.7): /usr/bin/ceph-deploy osd prepare ceph-node1:vdb
[ [1mceph_deploy.osd [0m][ [1;34mDEBUG [0m ] Preparing cluster ceph disks ceph-node1:/dev/vdb:
[ [1mceph_deploy.sudo_pushy [0m][ [1;34mDEBUG [0m ] will use a remote connection without sudo
[ [1mceph_deploy.lsb [0m][ [1;33mWARNIN [0m] lsb_release was not found - inferring OS details
[ [1mceph_deploy.osd [0m][ [1;37mINFO [0m  ] Distro info: CentOS 6.4 Final
[ [1mceph_deploy.osd [0m][ [1;34mDEBUG [0m ] Deploying osd to ceph-node1
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] write cluster configuration to /etc/ceph/{cluster}.conf
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] keyring file does not exist, creating one at: /var/lib/ceph/bootstrap-osd/ceph.keyring
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] create mon keyring file
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] Running command: udevadm trigger --subsystem-match=block --action=add
[ [1mceph_deploy.osd [0m][ [1;34mDEBUG [0m ] Preparing host ceph-node1 disk /dev/vdb journal None activate False
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/vdb
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] The operation has completed successfully.
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] The operation has completed successfully.
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] meta-data=/dev/vdb1              isize=2048   agcount=4, agsize=28770239 blks
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ]          =                       sectsz=512   attr=2, projid32bit=0
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] data     =                       bsize=4096   blocks=115080955, imaxpct=25
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ]          =                       sunit=0      swidth=0 blks
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] naming   =version 2              bsize=4096   ascii-ci=0
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] log      =internal log           bsize=4096   blocks=56191, version=2
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ]          =                       sectsz=512   sunit=0 blks, lazy-count=1
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] realtime =none                   extsz=4096   blocks=0, rtextents=0
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] The operation has completed successfully.
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] INFO:ceph-disk:Will colocate journal with data on /dev/vdb
[ [1mceph_deploy.osd [0m][ [1;34mDEBUG [0m ] Host ceph-node1 is now ready for osd use.
 [root@ceph-mon1 ceph]#

Once you prepare an OSD you may activate it. The activate command will cause your OSD to come up and be placed in the cluster. The activate command uses the path to the partition created when running the prepare command.

[root@ceph-mon1 ceph]# ceph-de ploy osd activate ceph-node1: vdb1
[ [1mceph_deploy.cli [0m][ [1;37mINFO [0m  ] Invoked (1.2.7): /usr/bin/ceph-deploy osd activate ceph-node1:vdb
[ [1mceph_deploy.osd [0m][ [1;34mDEBUG [0m ] Activating cluster ceph disks ceph-node1:/dev/vdb:
[ [1mceph_deploy.sudo_pushy [0m][ [1;34mDEBUG [0m ] will use a remote connection without sudo
[ [1mceph_deploy.lsb [0m][ [1;33mWARNIN [0m] lsb_release was not found - inferring OS details
[ [1mceph_deploy.osd [0m][ [1;37mINFO [0m  ] Distro info: CentOS 6.4 Final
[ [1mceph_deploy.osd [0m][ [1;34mDEBUG [0m ] activating host ceph-node1 disk /dev/vdb
[ [1mceph_deploy.osd [0m][ [1;34mDEBUG [0m ] will use init type: sysvinit
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] Running command: ceph-disk-activate --mark-init sysvinit --mount /dev/vdb

During this step if you encounter error like below , then start troubleshooting

[ [1mceph-node1 [0m][ [1;31mERROR [0m ] 2013-10-25 20:46:47.433307 7f6adc1d9700  0 -- :/1010803 >> 192.168.1.38:6789/0 pipe(0x7f6acc003d40 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f6acc003270).fault
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] 2013-10-25 20:46:50.433780 7f6ad43f9700  0 -- :/1010803 >> 192.168.1.38:6789/0 pipe(0x7f6acc00b860 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f6acc005090).fault
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] 2013-10-25 20:46:53.378985 7f6add314700  0 monclient(hunting): authenticate timed out after 300
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] 2013-10-25 20:46:53.379057 7f6add314700  0 librados: client.bootstrap-osd authentication error (110) Connection timed out
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] Error connecting to cluster: Error
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] ERROR:ceph-disk:Failed to activate
[root@ceph-mon1 ceph]#

On ceph-mon1 , cd /etc/ceph
scp ceph.c lient.admin.keyring ceph-node1:/etc/ceph
scp /var/lib/ceph/bootstrap-osd/ceph.keyring ceph-node1:/var/lib/ceph/bootstrap-osd
check firewall between nodes
Again try to activate your OSD , it should work

[root@ceph-mon1 ceph]# ceph-deploy osd prepare ceph-node1:/dev/vdb1
[ [1mceph_deploy.cli [0m][ [1;37mINFO [0m  ] Invoked (1.2.7): /usr/bin/ceph-deploy osd activate ceph-node1:/dev/vdb1
[ [1mceph_deploy.osd [0m][ [1;34mDEBUG [0m ] Activating cluster ceph disks ceph-node1:/dev/vdb1:
[ [1mceph_deploy.sudo_pushy [0m][ [1;34mDEBUG [0m ] will use a remote connection without sudo
[ [1mceph_deploy.lsb [0m][ [1;33mWARNIN [0m] lsb_release was not found - inferring OS details
[ [1mceph_deploy.osd [0m][ [1;37mINFO [0m  ] Distro info: CentOS 6.4 Final
[ [1mceph_deploy.osd [0m][ [1;34mDEBUG [0m ] activating host ceph-node1 disk /dev/vdb1
[ [1mceph_deploy.osd [0m][ [1;34mDEBUG [0m ] will use init type: sysvinit
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] Running command: ceph-disk-activate --mark-init sysvinit --mount /dev/vdb1
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] === osd.0 === 
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] Starting Ceph osd.0 on ceph-node1...
[ [1mceph-node1 [0m][ [1;37mINFO [0m  ] starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] got latest monmap
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] 2013-10-25 21:50:53.768569 7fc6aff527a0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] 2013-10-25 21:50:54.268147 7fc6aff527a0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] 2013-10-25 21:50:54.269488 7fc6aff527a0 -1 filestore(/var/lib/ceph/tmp/mnt.DBPdBc) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] 2013-10-25 21:50:54.611784 7fc6aff527a0 -1 created object store /var/lib/ceph/tmp/mnt.DBPdBc journal /var/lib/ceph/tmp/mnt.DBPdBc/journal for osd.0 fsid 0ff473d9-0670-42a3-89ff-81bbfb2e676a
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] 2013-10-25 21:50:54.611876 7fc6aff527a0 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.DBPdBc/keyring: can't open /var/lib/ceph/tmp/mnt.DBPdBc/keyring: (2) No such file or directory
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] 2013-10-25 21:50:54.612049 7fc6aff527a0 -1 created new key in keyring /var/lib/ceph/tmp/mnt.DBPdBc/keyring
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] added key for osd.0
[ [1mceph-node1 [0m][ [1;31mERROR [0m ] create-or-move updating item name 'osd.0' weight 0.43 at location {host=ceph-node1,root=default} to crush map
 [root@ceph-mon1 ceph]# 
[root@ceph-mon1 ceph]# ceph status
  cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a
   health HEALTH_WARN 192 pgs degraded; 192 pgs stuck unclean
   monmap e1: 1 mons at {ceph-mon1=192.168.1.38:6789/0}, election epoch 2, quorum 0 ceph-mon1
   osdmap e5: 1 osds: 1 up, 1 in
    pgmap v8: 192 pgs: 192 active+degraded; 0 bytes data, 1057 MB used, 438 GB / 439 GB avail
   mdsmap e1: 0/0/1 up
 [root@ceph-mon1 ceph]#

Similarly add 3 more OSD on ceph-node1 , after successful addition of OSD your ceph status will look like

[root@ceph-mon1 ceph]#  ceph status
  cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a
   health HEALTH_WARN 192 pgs stuck unclean
   monmap e1: 1 mons at {ceph-mon1=192.168.1.38:6789/0}, election epoch 1, quorum 0 ceph-mon1
   osdmap e63: 4 osds: 4 up, 4 in
    pgmap v112: 192 pgs: 192 active+remapped; 0 bytes data, 2188 MB used, 1755 GB / 1757 GB avail
   mdsmap e1: 0/0/1 up
 [root@ceph-mon1 ceph]#

Once you have added 4 OSD on ceph-node1 , repeat these steps and add 4 OSD on ceph-node2 , before OSD make you install ceph packages on ceph-node2
After these steps you cluster should have 8 OSD running and cluster health must be OK , PGmap=ACTIVE+CLEAN

[root@ceph-mon1 ceph]# ceph status
  cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a
   health HEALTH_OK
   monmap e1: 1 mons at {ceph-mon1=192.168.1.38:6789/0}, election epoch 1, quorum 0 ceph-mon1
   osdmap e87: 8 osds: 8 up, 8 in
    pgmap v224: 192 pgs: 192 active+clean; 0 bytes data, 2363 MB used, 3509 GB / 3512 GB avail
   mdsmap e1: 0/0/1 up
[root@ceph-mon1 ceph]#

Scaling the cluster by adding monitors ( ceph-mon2 & ceph-mon3 )

Create monitors on the ceph-mon2 and ceph-mon3 , before creating monitors make sure you install ceph packages on ceph-mon2 & ceph-mon3

[root@ceph-mon1 ceph]# ceph-deploy mon create ceph-mon2                        
[ [1mceph_deploy.cli [0m][ [1;37mINFO [0m  ] Invoked (1.2.7): /usr/bin/ceph-deploy mon create ceph-mon2
[ [1mceph_deploy.mon [0m][ [1;34mDEBUG [0m ] Deploying mon, cluster ceph hosts ceph-mon2
[ [1mceph_deploy.mon [0m][ [1;34mDEBUG [0m ] detecting platform for host ceph-mon2 ...
[ [1mceph_deploy.sudo_pushy [0m][ [1;34mDEBUG [0m ] will use a remote connection without sudo
[ [1mceph_deploy.lsb [0m][ [1;33mWARNIN [0m] lsb_release was not found - inferring OS details
[ [1mceph_deploy.mon [0m][ [1;37mINFO [0m  ] distro info: CentOS 6.4 Final
[ [1mceph-mon2 [0m][ [1;34mDEBUG [0m ] determining if provided host has same hostname in remote
[ [1mceph-mon2 [0m][ [1;34mDEBUG [0m ] deploying mon to ceph-mon2
[ [1mceph-mon2 [0m][ [1;34mDEBUG [0m ] remote hostname: ceph-mon2
[ [1mceph-mon2 [0m][ [1;37mINFO [0m  ] write cluster configuration to /etc/ceph/{cluster}.conf
[ [1mceph-mon2 [0m][ [1;34mDEBUG [0m ] checking for done path: /var/lib/ceph/mon/ceph-ceph-mon2/done
[ [1mceph-mon2 [0m][ [1;37mINFO [0m  ] create a done file to avoid re-doing the mon deployment
[ [1mceph-mon2 [0m][ [1;37mINFO [0m  ] create the init path if it does not exist
[ [1mceph-mon2 [0m][ [1;37mINFO [0m  ] locating `service` executable...
[ [1mceph-mon2 [0m][ [1;37mINFO [0m  ] found `service` executable: /sbin/service
[ [1mceph-mon2 [0m][ [1;37mINFO [0m  ] Running command: /sbin/service ceph -c /etc/ceph/ceph.conf start mon.ceph-mon2
[ [1mceph-mon2 [0m][ [1;34mDEBUG [0m ] === mon.ceph-mon2 === 
[ [1mceph-mon2 [0m][ [1;34mDEBUG [0m ] Starting Ceph mon.ceph-mon2 on ceph-mon2...
[ [1mceph-mon2 [0m][ [1;34mDEBUG [0m ] failed: 'ulimit -n 32768;  /usr/bin/ceph-mon -i ceph-mon2 --pid-file /var/run/ceph/mon.ceph-mon2.pid -c /etc/ceph/ceph.conf '
[ [1mceph-mon2 [0m][ [1;34mDEBUG [0m ] Starting ceph-create-keys on ceph-mon2...
[ [1mceph-mon2 [0m][ [1;33mWARNIN [0m] No data was received after 7 seconds, disconnecting...
[ [1mceph-mon2 [0m][ [1;37mINFO [0m  ] Running command: ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon2.asok mon_status
[ [1mceph-mon2 [0m][ [1;31mERROR [0m ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ [1mceph-mon2 [0m][ [1;33mWARNIN [0m] monitor: mon.ceph-mon2, might not be running yet
[ [1mceph-mon2 [0m][ [1;37mINFO [0m  ] Running command: ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon2.asok mon_status
[ [1mceph-mon2 [0m][ [1;31mERROR [0m ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ [1mceph-mon2 [0m][ [1;33mWARNIN [0m] ceph-mon2 is not defined in `mon initial members`
[ [1mceph-mon2 [0m][ [1;33mWARNIN [0m] monitor ceph-mon2 does not exist in monmap
[ [1mceph-mon2 [0m][ [1;33mWARNIN [0m] neither `public_addr` nor `public_network` keys are defined for monitors
[ [1mceph-mon2 [0m][ [1;33mWARNIN [0m] monitors may not be able to form quorum
[root@ceph-mon1 ceph]#

You might encounter some warnings or errors that need to be fixed
check monitor logs from ceph-mon1 , /var/lib/ceph directory
You might need to manually add monitor in cluster

#### Manually login on monitor node and execute commands like below 
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph

In case ceph service is not showing as running in service ceph status command , however ps -ef | grep ceph , shows monitor running , kill the process manually and restart ceph services , monitor now should start working

# service ceph status
# ps -ef | grep ceph
# kill -9 6554

### Finally you would see your ceph cluster is healthy with all the OSD and monitors UP and Running

[root@ceph-mon1 ~]# ceph status
  cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a
   health HEALTH_OK
   monmap e3: 3 mons at {ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0}, election epoch 10, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
   osdmap e97: 8 osds: 8 up, 8 in
    pgmap v246: 192 pgs: 192 active+clean; 0 bytes data, 2352 MB used, 3509 GB / 3512 GB avail
   mdsmap e1: 0/0/1 up
[root@ceph-mon1 ~]#

Please Follow Ceph Installation :: Part-3 for next step in installation

Ceph Installation :: Part-1

noreply@blogger.com (Anonymous) — Thu, 05 Dec 2013 16:16:00 +0000

Ceph Installation Step by Step

This quick start setup helps to deploy ceph with 3 Monitors and 2 OSD nodes with 4 OSD each node. In this we are using commodity hardware running CentOS 6.4

Ceph-mon1 : First Monitor + Ceph-deploy machine (will be used to deploy ceph to other nodes )

Ceph-mon2 : Second Monitor ( for monitor quorum )

Ceph-mon3 : Third Monitor ( for monitor quorum )

Ceph-node1 : OSD node 1 with 10G X 1 for OS , 440G X 4 for 4 OSD

Ceph-node2 : OSD node 2 with 10G X 1 for OS , 440G X 4 for 4 OSD

Ceph-Deploy Version is 1.3.2 , Ceph Version 0.67.4 ( Dumpling )

Preflight Checklist

All the Ceph Nodes may require some basic configuration work prior to deploying a Ceph Storage Cluster.

CEPH node setup

Create a user on each Ceph Node.

sudo useradd -d /home/ceph -m ceph
sudo passwd ceph

Add root privileges for the user on each Ceph Node.

echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers
sudo chmod 0440 /etc/sudoers

Configure your ceph-deploy node ( ceph-mon1) with password-less SSH access to each Ceph Node. Leave the passphrase empty , repeat this step for CEPH and ROOT users.

ceph@ceph-admin:~ [ceph@ceph-admin ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ceph/.ssh/id_rsa): yes         
Created directory '/home/ceph/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ceph/.ssh/id_rsa.
Your public key has been saved in /home/ceph/.ssh/id_rsa.pub.
The key fingerprint is:
48:86:ff:4e:ab:c3:f6:cb:7f:ba:46:33:10:e6:22:52 ceph@ceph-admin.csc.fi
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|    E.  o        |
|   .. oo .       |
|  . .+..o        |
|   . .o.S.       |
|       .  +      |
|     .  o. o     |
|      ++ .. .    |
|     ..+*+++     |
+-----------------+

Copy the key to each Ceph Node. ( Repeat this step for ceph and root users )

[ceph@ceph-mon1 ~]$ ssh-copy-id ceph@ceph-node2
The authenticity of host 'ceph-node2 (192.168.1.38)' can't be established.
RSA key fingerprint is ac:31:6f:e7:bb:ed:f1:18:9e:6e:42:cc:48:74:8e:7b.
Are you sure you want to continue connecting (yes/no)? y
Please type 'yes' or 'no': yes
Warning: Permanently added 'ceph-node2,192.168.1.38' (RSA) to the list of known hosts.
ceph@ceph-node2's password: 
Now try logging into the machine, with "ssh 'ceph@ceph-node2'", and check in:  .ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
[ceph@ceph-mon1 ~]$

Ensure connectivity using ping with hostnames , for convenience we have used local host file , update host file of every node with details of other nodes. PS : Use of DNS is recommended
Packages are cryptographically signed with the release.asc key. Add release key to your system’s list of trusted keys to avoid a security warning:

sudo rpm --import 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc'

Ceph may require additional additional third party libraries. To add the EPEL repository, execute the following:

su -c 'rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm'
sudo yum install snappy leveldb gdisk python-argparse gperftools-libs

Installing Release packages , Dumpling is the most recent stable release of Ceph. ( by the time i am creating this wiki )

su -c 'rpm -Uvh http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm'

Adding ceph to YUM , create repository file for ceph /etc/yum.repos.d/ceph.repo

[ceph]
name=Ceph packages for $basearch
baseurl=http://ceph.com/rpm-dumpling/el6/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=http://ceph.com/rpm-dumpling/el6/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=http://ceph.com/rpm-dumpling/el6/SRPMS
enabled=0
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc

For best results, create directories on your nodes for maintaining the configuration generated by ceph . These should get auto created by ceph however in may case it gave me problems. So creating manually.

mkdir -p /etc/ceph /var/lib/ceph/{tmp,mon,mds,bootstrap-osd} /var/log/ceph

By default, daemons bind to ports within the 6800:7100 range. You may configure this range at your discretion. Before configuring your IP tables, check the default iptables configuration. ::ports within the 6800:7100 range. You may configure this range at your discretion. Since we are performing test deployment we can disable iptables on ceph nodes . For moving to production this need to be attended.

Please Follow Ceph Installation :: Part-2 for next step in installation

Ceph Storage :: Introduction

noreply@blogger.com (Anonymous) — Thu, 05 Dec 2013 15:37:00 +0000

What is CEPH

Ceph is an open-source, massively scalable, software-defined storage system which provides object, block and file system storage from a single clustered platform. Ceph's main goals is to be completely distributed without a single point of failure, scalable to the exabyte level, and freely-available. The data is replicated, making it fault tolerant. Ceph software runs on commodity hardware. The system is designed to be both self-healing and self-managing and self awesome :-)

CEPH Internals

OSD: A Object Storage Daemon (OSD) stores data, handles data replication, recovery, backfilling, rebalancing, and provides some monitoring information to Ceph Monitors by checking other Ceph OSD Daemons for a heartbeat. A Ceph Storage Cluster requires at least two Ceph OSD Daemons to achieve an active + clean state when the cluster makes two copies of your data .
Monitor: A Ceph Monitor maintains maps of the cluster state, including the monitor map, the OSD map, the Placement Group (PG) map, and the CRUSH map. Ceph maintains a history (called an “epoch”) of each state change in the Monitors, Ceph OSD Daemons, and PGs.
MDS: A Ceph Metadata Server (MDS) stores metadata on behalf of the Ceph Filesystem . Ceph Metadata Servers make it feasible for POSIX file system users to execute basic commands like ls, find, etc. without placing an enormous burden on the Ceph Storage Cluster.

Note :: Please use http://ceph.com/docs/master/ and other official InkTank and ceph community resources as a primary source of information on ceph . This entire blog is an attempt help beginners in setting up ceph cluster and sharing my troubleshooting with you.