How to install Calamari for Ceph Cluster on Ubuntu 14.04

ceph calamari dashboard
Calamari is a web-based monitoring and management for Ceph. In this post we will install Calamari on a working ceph cluster. Calamari node and all Ceph nodes are running ubuntu 14.04. We will use ceph-deploy utility to install packages. This article is just for test purposes and give you an idea about Calamari installation.
We have 3 nodes in Ceph Cluster
cpm01 – Ceph Mon
cpm02 – Ceph OSD
cpm03 – Ceph OSD

Prepare a Ubuntu 14.04 machine ( can be VM ) and follow the steps below. Default installation.
Step 1
edit your /etc/hosts file and add all of your ceph nodes
10.4.4.1 cpm01
10.4.4.2 cpm02
10.4.4.3 cpm03
etc
.
.

create a user on your ceph nodes ( same username as you use in Calamari node )
ssh user@ceph-server
sudo useradd -d /home/calamariuser -m calamariuser
sudo passwd calamariuser

On ceph nodes, add your new user to sudoers in order to run without password prompt
echo "calamariuser ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/calamariuser
sudo chmod 0440 /etc/sudoers.d/calamariuser

On Calamari node, generate ssh key and copy it to all your ceph nodes.
do not run this command with sudo or root user. run it under calamariuser. Enter with defaults no password.
ssh-keygen

now copy the generated key to your ceph nodes
ssh-copy-id calamariuser@cpm01
ssh-copy-id calamariuser@cpm02
ssh-copy-id calamariuser@cpm03
etc
.
.

to make things tidy and in place
cd ~
mkdir calamarifiles
cd calamarifiles

Now download and install “ceph-deploy” utility. Don’t install ceph-deploy from default ubuntu repos. Because it doesn’t have calamari commands.( ver 1.40 ). So we will install latest deb packages from ceph.
wget http://download.ceph.com/debian-firefly/pool/main/c/ceph-deploy/ceph-deploy_1.5.28trusty_all.deb
sudo dpkg -i ceph-deploy_1.5.28trusty_all.deb
run "sudo apt-get -f install" if you meet any dependency problem.

Download the calamari deb packages
wget http://download.ceph.com/calamari/1.3.1/ubuntu/trusty/pool/main/c/calamari/calamari-server_1.3.1.1-1trusty_amd64.deb
wget http://download.ceph.com/calamari/1.3.1/ubuntu/trusty/pool/main/c/calamari-clients/calamari-clients_1.3.1.1-1trusty_all.deb
wget http://download.ceph.com/calamari/1.3.1/ubuntu/trusty/pool/main/d/diamond/diamond_3.4.67_all.deb

Step 2
Installing Salt packages
wget -O - https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add -
add the line in /etc/apt/sources.list
deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest trusty main
Then run the commands below
sudo apt-get update
sudo apt-get install salt-syndic
sudo apt-get install salt-minion
sudo apt-get install salt-master
sudo apt-get install -y apache2 libapache2-mod-wsgi libcairo2 supervisor python-cairo libpq5 postgresql

sudo dpkg -i calamari-server*.deb calamari-clients*.deb
After this step you may see some python dependency errors. like “python-gevent python-twisted python-greenlet python-txamqp”
It’s normal. just run sudo apt-get -f install to fix dependencies.

Once, everything is OK. run the following command and follow the instructions. It will ask you some account information for management.
sudo calamari-ctl initialize
Finally, open your web browser and visit Calamari node address then log into the Calamari user interface with the account you created.

Now,you should see a screen like below
calamarifirst

Let’s connect our nodes.

Step 1
As mentioned in the web interface we will use ceph-deploy to connect and install required packages to the nodes.
Edit cephdeploy.conf and add the master definition in the file. This information will be used by your ceph nodes to connect.

On calamari node,
nano ~/.cephdeploy.conf
add these lines top of the file. Change the master address. Save and exit.
[ceph-deploy-calamari]
master = your calamari FQDN address

Now run the following command and follow debug messages. Everything should be done without error.
ceph-deploy calamari connect [ ...]
Example
ceph-deploy calamari connect cpm01 copm02 cpm03

Step 2
Now we should copy the diamond_3.4.67_all.deb file to all ceph nodes and install it.Diamond is a python daemon that collects system metrics.

On calamari node,
cd ~/calamarifiles
scp diamond_3.4.67_all.deb calamariuser@cephnode:/tmp/

Now SSH to each ceph node and install it.
cd /tmp
dpkg -i diamond_3.4.67_all.deb

fix any dependencies if met as before.

Step 3

SSH and check each ceph node if they are all have the master: calamarinodeFQDN the following config file. Otherwise, add and restart minion service.
sudo nano /etc/salt/minion.d/calamari.conf
sudo service salt-minion restart

Make sure ceph nodes solve your calamari node FQDN. Otherwise add it to their /etc/hosts file

Step 4
Now refresh your calamari web interface. You should see your nodes requesting to register. Follow the screen instructions.
That’s all. you have a working monitoring and management system for ceph.
Feel free to ask any question

Veeam Failed to connect to the port [xx.xx.xx.xx:2501].

If you are using Veeam for HyperV and onhost mode enabled you may receive an error like below.

Processing ‘VMName’ Error: Client error: Failed to connect to the port [ProxyIPAddress:2501].
Guest processing skipped (check guest OS VSS state and integration components version)
Error: Client error: Failed to connect to the port [ProxyIPAddress:2501].

Actually it is strange that it was working yesterday but today I met this error for the same job. anyway, to investigate the issue I check firewall traffic to see what traffic is passing through veeam proxy.
Continue reading “Veeam Failed to connect to the port [xx.xx.xx.xx:2501].”

Veeam – Unable to truncate transaction logs RPC 1726

After Veeam 6.5 to 7.0 upgrade, some VMs which have application aware enabled gave the following error.

Unable to truncate transaction logs. Details: RPC function call failed. Function name: [BlobCall]. Target machine:
[xx.xx.xx.xx]. RPC error:The remote procedure call failed. Code: 1726

Also you may see Veeam guest agent services are terminating unexpectedly in your windows event logs..

Faulting application name: VeeamGuestAgent.exe, version: 7.0.0.690, time stamp: 0x520b81e7
Faulting module name: dbghelp.dll, version: 6.1.7601.17514, time stamp: 0x4ce7c5ac
Exception code: 0xc00000fd

I checked ( permissions, VSS writers etc etc ) on VM side but no success.

In my solution,
I disabled application aware processing on backup job. Took a backup. Then re-enabled application aware processing and it is now working fine. I think veeam server pushed a new guest agent on VM side. Now I see no more service termination events and logs are truncated successfully. Hope this helps in your case.

The federation certificate expiration in VMware vDirector Orgs

Organization Administrators can receive a message like below

The federation certificate expiration is mm/dd/yyyy hh:mm:ss . An expired certificate may disable federation with the identity provider setup with your organization. The certificate can be regenerated from the Federation Settings page.

So you may enter Organization and Regenerate certificate.

Corresponding Organization/Administration/federation
Click Regenerate

 

regeneratecertificate

 

Vmware Tools Event log RpcIn: Unable to send

After vmware tools upgrade you may see hundreds of application event warnings in your windows VM.
Goto
Windows XP and Windows Server 2000/2003 – C:\Documents and Settings\AllUsers\ApplicationData\VMware\VMware Tools\
Windows Vista, Windows 7, and Windows Server 2008 – C:\ProgramData\VMware\VMware Tools\

and edit tools.conf and add the following lines.If there is no tools.conf there just create an empty one and edit.
[logging]
vmusr.level = error

Continue reading “Vmware Tools Event log RpcIn: Unable to send”