Thursday, November 3, 2016

Configure Puppet Master and Slave

In my case following are the scenarios

Puppet Master - skynet.dc
Pupper Client - moonclouds

Enable RPM for puppet on both master and slave

RHEL/CentOS 7 - rpm -ivh
RHEL/CentOS 6 - rpm -ivh
RHEL/CentOS 5 - rpm -ivh

On Puppet Master, install the following

yum install puppet-server
puppet resource package puppet-server ensure=latest
/etc/init.d/puppetmaster start

On Puppet Client, install the following

yum install puppet
puppet resource package puppet ensure=latest
/etc/init.d/puppet start

On Puppet Master, add the configuration

vi /etc/puppet/puppet.conf
add these names under [master] heading

dns_alt_names = puppet, master.local, puppet.skynet.dc

service puppetmaster restart

On Puppet Slave, add the configuration

vi /etc/puppet/puppet.conf

Add master DNS name under [agent] heading. Puppet will connect to server.

server = puppet.skynet.dc

vi /etc/default/puppet
service puppet restart

Sign Slave Certificate on Master

[root@skynet ~]# puppet cert --list
  "" (SHA256) 6C:0B:24:CE:EA:F1:97:BF:89:39:FF:AB:6A:B9:AB:FF:8E:B3:EE:0D:E6:73:DD:9A:D5:05:6C:CE:24:0C:3D:DC
[root@skynet ~]#

[root@skynet ~]# puppet cert --sign
Notice: Signed certificate request for
Notice: Removing file Puppet::SSL::CertificateRequest at '/var/lib/puppet/ssl/ca/requests/'
[root@skynet ~]#

Location of certificates on puppet: /var/lib/puppet/ssl/

The above preparation will help the puppet clients to communicate with master. The next you need to code your requirement as code.

Thursday, October 27, 2016

How to configure Postfix as a relay server

I have a Redmine instance, where email configuration is working fine without "TLS" (encrypted). Requirement is to configure the email emission part with TLS enabled?

 I tried to change the default file of Redmine (/usr/share/redmine/config/configuration.yml) to add the TLS enabled, with our relay server and all. But still the out put is emails without TLS.

Later I analyzed that, in that box, postfix is running and is responsible for making the encryption. Then i configure postix as a relay server to our remote mail server. Following steps helps to solve the issue

postconf -e 'relayhost ='
postconf -e 'smtp_sasl_auth_enable = yes'
postconf -e 'smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd'
postconf -e 'smtp_sasl_security_options ='
echo "" > /etc/postfix/sasl_passwd
chown root:root /etc/postfix/sasl_passwd
chmod 600 /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
/etc/init.d/postfix restart

Once I complete the above step, i got TLS enabled mails.

If, you need to read a related blog -

Sunday, October 23, 2016

Error with first step to install 11g R2 RAC

When i try to install yum install oracle-rdbms-server-11gR2-preinstall  i am getting an error

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

OS: RedHat 6.5
rpm –import
rpm -q gpg-pubkey-ec551f03-4c2d256a
yum install oracle-rdbms-server-11gR2-preinstall

This helps to solve the issue.

What is Puppet, why we need puppet?

10 years back, the job of a System Administrator was a tedious one. He need to spend a lot of time to keep the systems harmonized. He needs to know whats going on each server even-though he was not in his work time. Other main challenge in his/her job was replacing a faulty server or scaling an additional server to the existing infrastructure. System Admins need to know the exact OS, version, softwares, applications, patches, disk size and its partitions etc. They need to first bring a bare metal server, then OS, then necessary services, softwares and application deployments. The entire process will consume a good amount of time.Not only this, a System Admin need to understand the different commands that are using in different Operating Systems. For example, In RHEL we use rpm, in Debian we use dep, in SUSE linux we use yast etc. Other main problems are to roll out a change to all the servers in a data center. Suppose, in my case, my  webserver farm has 15 webservers, running apache on port 80. I need to change the port from 80 to 81. As you all know, we need to login to server and change the listening port from 80 to 81 and then restart the apache. You need to do the same steps in all 15 web servers. Do you think its a duplicate of effort? Yes, it is. These all make the life of a System Admin worse and they can't think of going home early :(

How we can solve these problems?
One method is the "manual way", yes, accepting the facts that, these are my jobs and i need to do it. This method of thinking is good for only limited sets of machines. Can we do this for 100's of servers? BTW, we can think of some auto config scripts or kickstart methods to address these 100's server. Still what you do when it comes to 1000's or lakhs of servers? Wooww... Another issue we need to address here are the following

a) Who push the change?
b) Do we have any history of change?
c) Do we have a audit logs?

Here comes the solution for all these issues. What is Puppet?
  • Puppet is a GPL Open Source Project written in Ruby language. 
  • It uses a declarative language for expressing system configurations. 
  • It's a client server model. We have at least one server and "n" clients can connect to that server.
  • Puppet is used for infrastructure automation and configuration management. It helps to define our infrastructure as a line of code. 
  • Puppet is build to be for cross platform. We can use linux, unix or windows machines as clients for puppet.
  • Puppet client runs at every 30 min by default to check the server for any changes. We can adjust this default time as we like.
Puppet Resource Type -
A Type is a particular element that Puppet knows how to configure. There are so many resource types are available in puppet. The commonly used resource types are the following:
  • Files (content, permissions, ownership)
  • Packages (ensure installed or absent)
  • Services (enabled/disabled, running/stopped)
  • Exec (run commands)
Puppet Terminology -
I will try to understand the basic terminology used in Puppet which will help us understand puppet more in deep.

Puppet master - This is our Puppet server which takes care of puppet clients which connect to it. Here the puppet master code runs. This acts as a centralized machine for all your Puppet activity by default.

Puppetmasterd - Running state of Puppet server is called as puppetmasterd

Puppet Agent - Running state of Puppet client software. This software is useful for

1)  Gathering facts: Getting vast system details like OS type, version, RAM details, CPU details and much more using installed facter tool.

2)  Send those details to master for getting required compiled configuration details(catalogs)

3) Getting compiled configuration details from puppet master

4) Executing those compiled configurations at a designated time.

5) Collecting and sending reports of this expected configuration on the client.

Puppet Node - Puppet clients are called as nodes.

Resources - These are files/configs which can be used for changing settings at nodes.

Facter - Reporting tool at nodes, which reports about OS/HW/SW types etc to puppet master about the node where it is installed.

Clipped output of facter command:
swapfree => 13.67 GB
swapsize => 13.67 GB
timezone => PDT
type => Other
uniqueid => c40a660c
uptime => 67 days
virtual => xenhvm

Attribute - Attributes are used to specify the state desired for a given configuration resource. We will see more about this in our coming posts.

Puppet Manifests - A file containing code written in the Puppet DSL language, and named with the .pp file extension. All manifests are stored in /etc/puppet/manifests or /etc/puppet/modules/<name-of-module>/manifests.

Catalogs - Compiled form of manifests.

Puppet Modules - Modules are a bigger form of manifests. Every manifest in a module should define a single class or defined type.

Files - Physical files you can serve out to your agents through puppet. These are stored in /etc/puppet/files or /etc/puppet/modules/<name-of-module>/files.

Templates - Template files contain puppet code with variables which can populate variable values depending on node details.

Classes - Collections of resources

Definitions - Composite collections of resources

If you are interested more, you will get the complete list at

Tuesday, October 4, 2016

NFS mount issue on client machine - clnt_create: RPC: Program not registered

Getting the following error
"clnt_create: RPC: Program not registered" on terminal for command showmount

[root@servername ~]# showmount -e
clnt_create: RPC: Program not registered

Login to the NFS server
Recheck the /etc/export file
/etc/init.d/nfs stop
/etc/init.d/nfs start

If the service nfs started fine, you need to login to the client and issue the previous command showmount, it should work.

Tuesday, July 12, 2016

How to run xfs_repair to resolve Input/output error

/data2 is reporting I/O error.

root@shivinlocal data2]# ls
 ls: reading directory .: Input/output error

Checked the df -h and identify the /data2 partition is on /dev/sda1

Login to the server as root
umount /data2

If the unmount is showing error like this 
[root@searchlb01 /]# umount /data2
umount: /data2: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1)) 

[root@searchlb01 /]# lsof /dev/sda1
lsof: WARNING: can't stat() xfs file system /data2
      Output information may be incomplete.
[root@searchlb01 /]# umount -l /data2

[root@shivinlocal ~]# xfs_repair /dev/sda1
xfs_repair: /dev/sda1 contains a mounted filesystem
fatal error -- couldn't initialize XFS library

How can we solve this error?

Comment out the fstab entry for /data2
Reboot the server

[root@shivinlocal ~]# xfs_repair /dev/sda1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
 - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed. Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair. If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

For fixing the above error

[root@sdw6 ~]# xfs_repair -L /dev/sda1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
 - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
destroyed because the -L option was used.
 - scan filesystem freespace and inode maps...
 - found root inode chunk
Phase 3 - for each AG...
 - scan and clear agi unlinked lists...
 - process known inodes and perform inode discovery...
 - agno = 0
4461d940: Badness in key lookup (length)
bp=(bno 179821952, len 16384 bytes) key=(bno 179821952, len 8192 bytes)
 - agno = 1
 - agno = 2
46e21940: Badness in key lookup (length)
bp=(bno 539478608, len 16384 bytes) key=(bno 539478608, len 8192 bytes)
 - agno = 3
 - agno = 4
 - agno = 5
 - agno = 6
4be29940: Badness in key lookup (length)
disconnected inode 10737418540, moving to lost+found
Phase 7 - verify and correct link counts...

Uncomment the fastab entry
[root@shivinlocal ~]# mount -a /data2
 [root@shivinlocal data2]# ls
folder1 lost+found search99 secondfolder
Make sure all the services are UP and running. 

Saturday, July 2, 2016

Sendmail Error in start

While starting a sendmail i am getting the below error

[root@ip-132-31-52-30 ~]# /etc/init.d/sendmail start
Starting sendmail: 451 4.0.0 /etc/mail/ line 87: fileclass: cannot open '/etc/mail/local-host-names': World writable directory
451 4.0.0 /etc/mail/ line 596: fileclass: cannot open '/etc/mail/trusted-users': World writable directory

check your directory permissions

ls -ld / /etc /etc/mail

drwxr-xr-x 28 root root 4096 Jun 11 16:16 /
drwxr-xr-x 96 root root 12288 Jun 26 04:12 /etc
drwxr-xr-x 5 root root 4096 Jun 17 18:13 /etc/mail

if you have different permissions do

chmod 755 / /etc /etc/mail
and try starting sendmail.It will start smoothly without any error.