Getting yourself familiar with basic programming skills doesn’t hurt. In fact, there are times when you will desperately need to automate a task for some seemingly simple job, and yet not find the right tools out there which cater your needs. It is not about writing thousands of lines of code and designing some user interface, just a dozen or two lines might serve well at times. Here is a list of C codes taken from the book ‘C programming Language’ by K&R and some of the codes might have been changed by me while practicing.

1. Introductory Tutorial – Input/output, characters, strings

2. Types, Operators and Experessions – Upper/lower case conversion, binary operators

3.  Control Flow – If/else, do/while, binary search, sorting, argument list

4.  Functions and Program structure – macros, polish calculator, pattern searching, quick sort

5. Pointers and Arrays – command line argument, find, sort, memory allocation

6.  Structures – self referential arrays, word key counter

7.  Input/Output – file copying, calculator, sscanf

8.  The UNIX System Interface – memory allocations, file & directory listing

The date command in Linux boxes is one of the most powerful open source utilities. It is not just for setting the clock on your PC or server, or showing you what the current time is, it can do amazingly more. It can virtually answer all of your chronological questions.

The simplest use case of date command is to view current time, possibly in different time formats –

$ date
Sat Dec 17 00:45:35 EST 2016

$ date '+%Y-%m-%d'
2016-12-17

$ date '+%c'
Sat 17 Dec 2016 12:45:51 AM EST

It is useful in converting time to/from epoch as well –

$ date '+%s'
1481953669

$ date --date='@1481953669'
Sat Dec 17 00:47:49 EST 2016

The most user friendly use case of the date command is the ‘-d’ or ‘–date’ options, which accepts free format human readable date string such as “yesterday”, “last week”, “next year”, “3 min ago”, “last friday + 2 hours” etc. Here is an excerpt from the man page of the GNU date command –

DATE STRING
The --date=STRING is a mostly free format human readable date string such as "Sun, 29 Feb 2004 16:21:42 -0800" or "2004-02-29 16:21:42" or even "next Thursday". A date string may con?
tain items indicating calendar date, time of day, time zone, day of week, relative time, relative date, and numbers. An empty string indicates the beginning of the day. The date
string format is more complex than is easily documented here but is fully described in the info documentation.

Let us play with it –

$ date -d '2 hours ago'
Fri Dec 16 22:51:25 EST 2016

$ date -d '2 hours ago' '+%c'
Fri 16 Dec 2016 10:51:30 PM EST

$ env TZ=America/Los_Angeles date -d '2 hours ago' '+%c'
Fri 16 Dec 2016 07:52:33 PM PST

$ date -d 'jan 2 1990'
Tue Jan  2 00:00:00 EST 1990

$ date -d 'yesterday'
Fri Dec 16 00:53:04 EST 2016

$ date -d 'next year + 2 weeks'
Sun Dec 31 00:53:27 EST 2017

To give a practical example, let us use the date command to get, on which day all the birth days of someone fall, given their date of birth. This can be for past birth days as well as the future. For this example, we will do it from date of birth to this date. Let us pick someone who was born on Feb 29, 1988. This is an edge case. The date command should be smart enough to figure out the leap years.

for year in {1988..2016}; do 
  date -d "feb 29 $year" &>/dev/null
  if [ $? -eq 0 ]; then
    echo -n "Year: $year   " ; date -d "feb 29 $year" '+%c'
  fi
done

Year: 1988   Mon 29 Feb 1988 12:00:00 AM EST
Year: 1992   Sat 29 Feb 1992 12:00:00 AM EST
Year: 1996   Thu 29 Feb 1996 12:00:00 AM EST
Year: 2000   Tue 29 Feb 2000 12:00:00 AM EST
Year: 2004   Sun 29 Feb 2004 12:00:00 AM EST
Year: 2008   Fri 29 Feb 2008 12:00:00 AM EST
Year: 2012   Wed 29 Feb 2012 12:00:00 AM EST
Year: 2016   Mon 29 Feb 2016 12:00:00 AM EST

A typical case would be, say for someone born on Jan 8 1990 –

age=0
for year in {1990..2016}; do 
  echo -n "Age: $age  "; date -d "Jan 8 $year" '+%A %d %B %Y'
  age=$((age+1))
done

Age: 0  Monday 08 January 1990
Age: 1  Tuesday 08 January 1991
Age: 2  Wednesday 08 January 1992
Age: 3  Friday 08 January 1993
Age: 4  Saturday 08 January 1994
Age: 5  Sunday 08 January 1995
Age: 6  Monday 08 January 1996
Age: 7  Wednesday 08 January 1997
Age: 8  Thursday 08 January 1998
Age: 9  Friday 08 January 1999
Age: 10  Saturday 08 January 2000
Age: 11  Monday 08 January 2001
Age: 12  Tuesday 08 January 2002
Age: 13  Wednesday 08 January 2003
Age: 14  Thursday 08 January 2004
Age: 15  Saturday 08 January 2005
Age: 16  Sunday 08 January 2006
Age: 17  Monday 08 January 2007
Age: 18  Tuesday 08 January 2008
Age: 19  Thursday 08 January 2009
Age: 20  Friday 08 January 2010
Age: 21  Saturday 08 January 2011
Age: 22  Sunday 08 January 2012
Age: 23  Tuesday 08 January 2013
Age: 24  Wednesday 08 January 2014
Age: 25  Thursday 08 January 2015
Age: 26  Friday 08 January 2016

How do you find out the number of CPU cores available in your Linux system? Here are a number of way, pick the one which works for you –

1. nproc command –

[daniel@kauai tmp]$ nproc
2

2. /proc/cpuinfo

[daniel@kauai tmp]$ grep proc /proc/cpuinfo 
processor	: 0
processor	: 1

3. top – run top command and press ‘1’ (number 1), you will see the list of cores at the top, right below tasks.

Cpu0 : 0.7%us, 0.3%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 2.7%us, 1.0%sy, 0.0%ni, 96.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

4. lscpu – display information about the CPU architecture. Count Sockets times Core(s) per socket, in this case 2 x 1=2 –

[daniel@kauai tmp]$ lscpu 
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             AuthenticAMD
CPU family:            16
Model:                 6
Model name:            AMD Athlon(tm) II X2 250 Processor
Stepping:              3
CPU MHz:               3000.000
BogoMIPS:              6027.19
Virtualization:        AMD-V
L1d cache:             64K
L1i cache:             64K
L2 cache:              1024K
NUMA node0 CPU(s):     0,1

5. Kernel threads – pick one of the kernel house keeping threads, such as “migration” or “watchdog” and see on how many cores it is running –

[daniel@kauai tmp]$ ps aux |grep '[m]igration'
root         3  0.0  0.0      0     0 ?        S    Dec09   0:02 [migration/0]
root         7  0.0  0.0      0     0 ?        S    Dec09   0:02 [migration/1]

[daniel@kauai tmp]$ ps aux |grep '[w]atchdog'
root         6  0.0  0.0      0     0 ?        S    Dec09   0:00 [watchdog/0]
root        10  0.0  0.0      0     0 ?        S    Dec09   0:00 [watchdog/1]

Sooner or later, you will find yourself adding sensitive data into Ansible playbooks, host or group vars files.Such information might include MySQL DB credentials, AWS secret keys, API credentials etc. Including such sensitive information in plain text might not be acceptable for security compliance reasons or even lead to your systems being owned when your company hires a third party to do pen testing and worst yet by outside hackers. In addition to this, sharing such playbooks to public repositories such as github won’t be easy as you have to manually search and redact all the sensitive information from all your playbooks, and as we know manual procedure is not always error prone. You might ‘forget’ to remove some of the paswords.

One solution for this is a password vault to hold all your sensitive data, and Ansible provides a utitility called ansible-vault to create this encrypted file and the data can be extracted when running your playbooks with a single option. This is equivalent to Chef’s data bag.

In this blog post, I will share with you how to use a secret key file to protect sensitive data in Ansible with ansible-vault utility. The simplest use case is to protect the encrypted file with a password or passphrase, but that is not convinient as you have to type the password everytime you run a playbook and is not as strong as a key file with hundreds or thousands of random characters. Thus the steps below describe only the procedure for setting up a secret key file rather than a password protected encrypted file. Let us get started.

The first step is to generate a key file containing a random list of characters –

#openssl rand -base64 512 |xargs > /opt/ansible/vaultkey

Create or initialize the vault with the key file generated above –

#ansible-vault create --vault-password-file=/opt/ansible/vaultkey /opt/ansible/lamp/group_vars/dbservers.yml

Populate your vault, refer to Ansible documentation on the format of the vault file –

#ansible-vault edit --vault-password-file=/opt/ansible/vaultkey /opt/ansible/lamp/group_vars/dbservers.yml

You can view the contents by replacing ‘edit’ with ‘view’ –

#ansible-vault view --vault-password-file=/opt/ansible/vaultkey /opt/ansible/lamp/group_vars/dbservers.yml

That is it, you have a secret key file to protect and encrypt a YAML file containing all your sensitive variables to be used in your ansible playbooks.

There comes a time though when you have to change the secret key file, say an admin leaves the company after winning the Mega jackbot lottery 🙂 We have to generate a new key file and rekey the encrypted file as soon as possible –

Generate a new key file –

#openssl rand -base64 512 |xargs > /opt/ansible/vaultkey.new

Rekey to new key file –

#ansible-vault rekey --new-vault-password-file=/opt/ansible/vaultkey.new --vault-password-file=/opt/ansible/vaultkey
Rekey successful

Verify –

#ansible-vault view --vault-password-file=/opt/ansible/vaultkey.new /opt/ansible/lamp/group_vars/dbservers.yml

Last but not least, make sure the secret key file is well protected and is readable only by the owner.

#chmod 600 /opt/ansible/vaultkey.new

Finally, you can use the vault with ansible-playbook. In this case, I am running it against site.yml which is a master playbook to setup a LAMP cluster in AWS (pulling the AWS instances using ec2.py dynamic inventory script) –

#ansible-playbook -i /usr/local/bin/ec2.py site.yml --vault-password-file /opt/ansible/vaultkey.new

Web sites store information on local machines of site visitors using cookies. On subsequent visits, the browser sends the data from the cookies on the visitors machine to the web server, which might then use that information as a historical record of the users activity on the site – on the minimum the time the cookie was created, when it is set to expire and last access time or last time user visited site. Cookies are also used by sites to ‘remember’ user acitivity , say the shopping cart items or login/session information to address the shortcomings of the stateless HTTP protocol.

Most users think that only the sites they had directly visited store cookies on their computers, in reality the number is way higher than that. A single site you visit, usually has lots of links in it, especially ads, that store cookies in your computer. In this post, i will demonstrate how to list the list of all sites that left cookies in your computer, as well as extract additional information from the cookies. When i ran the script and did a count of the 10 top sites which left largest number of entries in the cookies sqlite DB, none of them except for one or two were sites I directly visited!

This Python script was written to extract cookies information on a Linux box running Firefox. The cookies information is stored as a sqlite file and thus you will need the sqlite3 python module to read the sqlite file.

The script takes the path to the cookies file as well as the path to the output file, it will write the output to this file. It will also dump the output to the screen.

root@dnetbook:/home/daniel/python# python cookie_viewer.py 
cookie_viewer.py cookie-fullpath output-file

root@dnetbook:/home/daniel/python# python /home/daniel/python/cookie_viewer.py $(find /home/daniel/ -type f -name 'cookies.sqlite' | head -1) /tmp/test.txt
doubleclick.net,Thu Feb 11 17:56:01 2016,Thu Apr 23 20:46:58 2015,Tue Feb 11 17:56:01 2014
twitter.com,Thu Feb 11 17:56:05 2016,Tue Apr 21 22:27:46 2015,Tue Feb 11 17:56:05 2014
imrworldwide.com,Thu Feb 11 17:56:12 2016,Tue Apr 21 22:19:35 2015,Tue Feb 11 17:56:12 2014
quantserve.com,Thu Aug 13 19:32:02 2015,Thu Apr 23 20:46:57 2015,Tue Feb 11 18:32:0

The output will be the domain name of the site, cookie expiry date, access time and creation time.

Code follows –

#!/usr/bin/env python

''' Given a location to firefox cookie sqlite file
    Write its date param - expiry, last accessed,
    Creation time to a file in plain text.
    id
    baseDomain
    appId
    inBrowserElement
    name
    value
    host
    path
    expiry
    lastAccessed
    creationTime
    isSecure
    isHttpOnly
    python /home/daniel/python/cookie_viewer.py $(find /home/daniel/ -type f -name 'cookies.sqlite' | head -1) /tmp/test.txt 
'''

import sys
import os
from datetime import datetime
import sqlite3

def Usage():
    print "{0} cookie-fullpath output-file".format(sys.argv[0])
    sys.exit(1)

if len(sys.argv)<3:
    Usage()

sqldb=sys.argv[1]
destfile=sys.argv[2]
# Some dates in the cookies file might not be valid, or too big
MAXDATE=2049840000

# cookies file must be there, most often file name is cookies.sqlite
if not os.path.isfile(sqldb):
    Usage()

# a hack - to convert the epoch times to human readable format
def convert(epoch):
    mydate=epoch[:10]
    if int(mydate)>MAXDATE:
        mydate=str(MAXDATE)
    if len(epoch)>10:
        mytime=epoch[11:]
    else:
        mytime='0'
    fulldate=float(mydate+'.'+mytime)
    x=datetime.fromtimestamp(fulldate)
    return x.ctime()

# Bind to the sqlite db and execute sql statements
conn=sqlite3.connect(sqldb)
cur=conn.cursor()
try:
    data=cur.execute('select * from moz_cookies')
except sqlite3.Error, e:
    print 'Error {0}:'.format(e.args[0])
    sys.exit(1)
mydata=data.fetchall()

# Dump results to a file
with open(destfile, 'w') as fp:
    for item in mydata:
        urlname=item[1]
        urlname=item[1]
        expiry=convert(str(item[8]))
        accessed=convert(str(item[9]))
        created=convert(str(item[10]))
        fp.writelines(urlname + ',' + expiry + ',' + accessed + ',' + created)
        fp.writelines('\n')

# Dump to stdout as well
with open(destfile) as fp:
    for line in fp:
        print line

TOP 10 sites with highest number of enties in the cookies file –

root@dnetbook:/home/daniel/python# awk -F, '{print $1}' /tmp/test.txt  | sort | uniq -c | sort -nr | head -10
     73 taboola.com
     59 techrepublic.com
     43 insightexpressai.com
     34 pubmatic.com
     33 2o7.net
     31 rubiconproject.com
     28 demdex.net
     27 chango.com
     26 yahoo.com
     26 optimizely.com

In Python, you can read from and write to files without import any modules. Python has built-in function “open” which can be used to view and manipulate file objects. Let us see two ways of opening a file for reading/writing, for instance –

   fp_in = open('/etc/hosts', 'r')  # default is 'r', we can omit it.
   fp_out = open('/tmp/hosts', 'w')
   for line in fp_in:
       fp_out.write(line)

   fp_in.close()
   fp_out.close()


   with open('/etc/hosts') as fp_in:
       with open('/tmp/hosts') as fp_out:
       for line in fp_in:
           fp_out.write(line)
   # No need to close file, it is automatically closed at end of block.

One of the most common reasons given why you have to close the file object in the first case is to free up resources. But there is a second reason why you should always use ‘with’ keyword. After writing to a file object, and before closing it, the whole content from the source file might not appear in the destination file. This is because write uses buffering, and the changes will not be reflected until you run flush() or close() on the file object. Here is the help page for ‘write’ –

write(...)
    write(str) -> None.  Write string str to file.
    
    Note that due to buffering, flush() or close() may be needed before
    the file on disk reflects the data written.

Let me demonstrate this by copying the /var/log/messages file to /tmp/message, the bigger the file, the more likely you will witness the effect of buffering. First i will take a copy of /var/log/messages to /var/log/messages.orig, and work with messages.orig as the former will most likely change in size as work along.

[root@kauai ~]# wc -l /var/log/messages.orig 
10544 /var/log/messages.orig

[root@kauai ~]# wc -l /tmp/messages 
10542 /tmp/messages
[root@kauai ~]# tail -1 /tmp/messages 
Nov 16 02:36:02 kauai syslog-ng[1605]: Log statistics; processed='src.internal(s_sys[root@kauai ~]# 

[root@kauai ~]# tail -1 /var/log/messages
Nov 16 02:46:02 kauai syslog-ng[1605]: Log statistics; processed='src.internal(s_sys#2)=1787', stamp='src.internal(s_sys#2)=1416123362', processed='source(s_name_servers)=0', processed='destination(d_mesg)=7693', processed='destination(d_auth)=210', processed='source(s_sys)=12643', processed='global(payload_reallocs)=3568', processed='destination(d_mail)=12', processed='destination(d_kern)=5176', processed='destination(d_mlal)=0', processed='destination(d_ns_filtered)=0', processed='global(msg_clones)=0', processed='destination(d_spol)=0', processed='destination(hosts)=12643', processed='destination(d_boot)=0', processed='global(sdata_updates)=0', processed='center(received)=0', processed='destination(d_cron)=3653', processed='center(queued)=0'

Notice how the destination file /tmp/messages got truncated, it doesn’t even have a newline character at the end.

fp_out.close()
[root@kauai ~]# wc -l /tmp/messages 
10544 /tmp/messages

[root@kauai ~]# tail -1 /var/log/messages
Nov 16 02:56:02 kauai syslog-ng[1605]: Log statistics; processed='src.internal(s_sys#2)=1788', stamp='src.internal(s_sys#2)=1416123962', processed='source(s_name_servers)=0', processed='destination(d_mesg)=7694', processed='destination(d_auth)=211', processed='source(s_sys)=12646', processed='global(payload_reallocs)=3570', processed='destination(d_mail)=12', processed='destination(d_kern)=5176', processed='destination(d_mlal)=0', processed='destination(d_ns_filtered)=0', processed='global(msg_clones)=0', processed='destination(d_spol)=0', processed='destination(hosts)=12646', processed='destination(d_boot)=0', processed='global(sdata_updates)=0', processed='center(received)=0', processed='destination(d_cron)=3654', processed='center(queued)=0'

This problem would not have happened if we had used the ‘with’ keyword, as it automatically does the flush() and close() for us at the end of the block statement –

    with open('/var/log/messages.orig') as fp_in:
    with open('/tmp/messages','w') as fp_out:
        for line in fp_in:
            fp_out.write(line)

[root@kauai ~]# wc -l /var/log/messages.orig 
10544 /var/log/messages.orig
[root@kauai ~]# wc -l /tmp/messages 
10544 /tmp/messages

There you go, both source and destination files synced immediately.

This script is written based on the list of U.S. federal holidays I found in Wikipedia – Wikipedia – U.S. Federal holidays. Some of the dates, such as New Year, are straight forward, as the date and month are fixed. While others require some effort, take for instance Thanksgiving, which is on the fourth Thursday of November OR Memorial day – last Monday of May.

The script is written in bash, and only tested in 32 bit Ubuntu netbook. It will exit with an error message if you try to get the holidays for the year 2038 or above. This is a know issue with UNIX dates on 32 bit Operating Systems – UNIX: Year 2038 problem

Sample output

daniel@dnetbook:~$ /usr/local/bin/federalholidays.sh
Usage: federalholidays.sh Year
Eg. federalholidays.sh 2014


daniel@linubuvma:~$ ./federalholidays.sh 1500
New Year's Day:               Monday, January 01, 1500
Martin Luther King, Jr. Day:  Monday, January 15, 1500
Washington's Birthday:        Monday, February 19, 1500
Memorial Day:                 Monday, May 28, 1500
Independence Day:             Wednesday, July 04, 1500
Labor Day:                    Monday, September 03, 1500
Columbus Day:                 Monday, October 08, 1500
Veteran's Day:                Sunday, November 11, 1500
Thanksgiving:                 Thursday, November 22, 1500
Christmas Day:                Tuesday, December 25, 1500


daniel@linubuvma:~$ ./federalholidays.sh 2014
New Year's Day:               Wednesday, January 01, 2014
Martin Luther King, Jr. Day:  Monday, January 20, 2014
Washington's Birthday:        Monday, February 17, 2014
Memorial Day:                 Monday, May 26, 2014
Independence Day:             Friday, July 04, 2014
Labor Day:                    Monday, September 01, 2014
Columbus Day:                 Monday, October 13, 2014
Veteran's Day:                Tuesday, November 11, 2014
Thanksgiving:                 Thursday, November 27, 2014
Christmas Day:                Thursday, December 25, 2014


daniel@linubuvma:~$ ./federalholidays.sh 2500
New Year's Day:               Friday, January 01, 2500
Martin Luther King, Jr. Day:  Monday, January 18, 2500
Washington's Birthday:        Monday, February 15, 2500
Memorial Day:                 Monday, May 31, 2500
Independence Day:             Sunday, July 04, 2500
Labor Day:                    Monday, September 06, 2500
Columbus Day:                 Monday, October 11, 2500
Veteran's Day:                Thursday, November 11, 2500
Thanksgiving:                 Thursday, November 25, 2500
Christmas Day:                Saturday, December 25, 2500

Here is the whole script, feel free to modify it or report any problem –


#!/bin/bash

ARCH=$(arch)
ARGC=$#

function Usage
{

echo "Usage: $(basename $0) Year"
echo "Eg. $(basename $0) 2014"
exit 1

}

# we will need the year as argument in YYYY format
[[ $ARGC -ne 1 ]] &&  Usage

myyear="$1"
dformat='+%A, %B %d, %Y'

[[ "$myyear" -ge 2038 ]] && [[ "$ARCH" = "i686" ]] && echo 'Year 2038 problem : http://en.wikipedia.org/wiki/Year_2038_problem ' && exit 1

#We will ignore any year below 1902
[[ "$myyear" -lt 1902 ]] && [[ "$ARCH" = "i686" ]] && exit 1

##Function to get the nth day week of the month, for instance, Third Monday of March.

function nth_xday_of_month
{

my_nth=$1
my_xday=$2
my_month=$3
my_year=$4

case "$my_nth" in

1)  mydate=$(echo {01..07})
    ;;
2)  mydate=$(echo {08..14})
    ;;
3)  mydate=$(seq 15 21)
    ;;
4)  mydate=$(seq 22 28)
   ;;
5)  mydate=$(seq 29 31)
    ;;
*) echo "Echo wrong day of the week"
   exit 1
   ;;
esac


for x in $mydate; do
  nthday=$(date '+%u' -d "${my_year}${my_month}${x}")
  if [ "$nthday" -eq "$my_xday" ]; then
   date "${dformat}" -d "${my_year}${my_month}${x}"
  fi
done
}


##Memorial day - Last Monday of May.

for x in {31..01}; do y=$(date '+%u' -d "${myyear}05${x}"); if [ "$y" -eq 1 ]; then memday="${x}" ; break; fi ; done

echo "New Year's Day:              " $(date "${dformat}"  -d "${myyear}0101")
echo "Martin Luther King, Jr. Day: " $(nth_xday_of_month 3 1 01 ${myyear})
echo "Washington's Birthday:       " $(nth_xday_of_month 3 1 02 ${myyear})
echo "Memorial Day:                " $(date "${dformat}" -d "${myyear}05${memday}")
echo "Independence Day:            " $(date "${dformat}" -d "${myyear}0704")
echo "Labor Day:                   " $(nth_xday_of_month 1 1 09 ${myyear})
echo "Columbus Day:                " $(nth_xday_of_month 2 1 10 ${myyear})
echo "Veteran's Day:               " $(date "${dformat}" -d "${myyear}1111")
echo "Thanksgiving:                " $(nth_xday_of_month 4 4 11 ${myyear})
echo "Christmas Day:               " $(date "${dformat}" -d "${myyear}1225")

: <<'federal_holidays_comment'

http://en.wikipedia.org/wiki/Federal_holidays_in_the_United_States

Jan 1 - New Year's Day - 1st day of the year
Third Monday of January - Martin Luther King, Jr. Day 
Third Monday of February - Washington's Birthday
Last Monday of May - Memorial Day.
July 4 - Independence Day.
First Monday of September - Labor Day.
Second Monday of October - Columbus Day.
November 11 - Veteran's Day.
Fourth Thursday of November - Thanksgiving
December 25 - Christmas Day
federal_holidays_comment

The strace command allows us to trace the system calls made by a program. In this blog, I will show you how you can use strace to capture some of the syscalls made by Apache when clients make http requests. strace has several options, but here we will consider only the following options –

-p : attach to the process with the process ID pid
-o filename : write the strace output to the file filename rather than to stderr
-ff : If the -o filename option is in effect, each processes trace is written to filename.pid where pid is the numeric process id of each process. 
-e expr : to filter only specific syscalls (eg. open, fstat etc.)

We will attach strace to the parent process for the apache threads. With -ff specified, strace will trace all children of the parent process and saves the trace output to a file named filename.PID. We will be using ab(Apache HTTP server benchmarking tool) to generate traffic to the web server and see which files apache opens during client requests by explicitly looking for open syscall.

1. Let us find the parent process –

ns1 strace # ps xo comm,pid,ppid | grep apache2
apache2          2062     1

The PID to trace in this case is 2062.

2. Run strace command, while this is running, launch another session and run the ab command –

ns1 strace # strace -ff -o wiki.home.net -e trace=open,close -p 2062
Process 2062 attached - interrupt to quit
Process 18526 attached
Process 18531 attached
Process 18532 attached
Process 18536 attached
Process 18537 attached
Process 18538 attached
Process 18539 attached
Process 18526 detached
Process 18531 detached
Process 18532 detached
Process 18538 detached
^CProcess 2062 detached
Process 18536 detached
Process 18537 detached
Process 18539 detached


140706133405:root:homevm:/home/daniel:# ab -n 25 -c 10 http://wiki.home.net./
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking wiki.home.net. (be patient).....done


Server Software:        Apache/2.2.22
Server Hostname:        wiki.home.net.
Server Port:            80

Document Path:          /
Document Length:        0 bytes

Concurrency Level:      10
Time taken for tests:   20.595 seconds
Complete requests:      25
Failed requests:        0
Write errors:           0
Non-2xx responses:      25
Total transferred:      11300 bytes
HTML transferred:       0 bytes
Requests per second:    1.21 [#/sec] (mean)
Time per request:       8237.856 [ms] (mean)
Time per request:       823.786 [ms] (mean, across all concurrent requests)
Transfer rate:          0.54 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1  282 459.3      2    1004
Processing:  5372 6551 733.6   6846    7631
Waiting:      367 1546 734.1   1842    2629
Total:       5374 6833 991.9   7199    8049

Percentage of the requests served within a certain time (ms)
  50%   7170
  66%   7616
  75%   7819
  80%   7933
  90%   8005
  95%   8013
  98%   8049
  99%   8049
 100%   8049 (longest request)

3. Once ab completes, stop the strace command and do ls in current directory to see the output of strace command for each apache thread which was serving the http request as well as the strace output for the parent process

ns1 strace #  ls -l
total 88
-rw-r--r-- 1 root root 10617 Jul  6 13:34 wiki.home.net.18526
-rw-r--r-- 1 root root 10617 Jul  6 13:34 wiki.home.net.18531
-rw-r--r-- 1 root root 10617 Jul  6 13:34 wiki.home.net.18532
-rw-r--r-- 1 root root 10441 Jul  6 13:34 wiki.home.net.18536
-rw-r--r-- 1 root root 10441 Jul  6 13:34 wiki.home.net.18537
-rw-r--r-- 1 root root 10617 Jul  6 13:34 wiki.home.net.18538
-rw-r--r-- 1 root root 10441 Jul  6 13:34 wiki.home.net.18539
-rw-r--r-- 1 root root   581 Jul  6 13:34 wiki.home.net.2062

4. As you can see the apache parent process doesn’t serve any client requests, the child threads are the ones serving the client requests and in each strace output for the child threads we can see the files accesses/opend –

strace output for parent process - 

ns1 strace # cat wiki.home.net.2062
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---


ns1 strace # head -25 wiki.home.net.18526
open("/proc/sys/kernel/ngroups_max", O_RDONLY) = 20
close(20)                               = 0
open("/etc/group", O_RDONLY|O_CLOEXEC)  = 20
close(20)                               = 0
open("/.htaccess", O_RDONLY|O_CLOEXEC)  = -1 ENOENT (No such file or directory)
open("/var/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/var/www/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(22)                               = 0
close(22)                               = 0
close(22)                               = 0
open("/var/www/wiki/index.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/WebStart.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/Init.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/AutoLoader.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/profiler/Profiler.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/Defines.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/normal/UtfNormalDefines.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/DefaultSettings.php", O_RDONLY) = 22

In [1]: import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Users accounts usually get created and removed on most Development or Production servers. It is not uncommon to simply delete the users and yet not either delete or change the ownership of all files and directories associate with that user or user/group id. Some of the files might not be in the home directory of that user, so it is a good idea to search the whole file system for any files not owned by non-existent user or group. This is a big security issue, as an account might be created in the future with the same user or group id of the deleted account and end up having complete ownership of the files which don’t belong to them.

Solution – search ‘un-owned’ files and either change their ownership to ‘root:root’ or move them to some backup storage.


[root@danasmera ~]# declare -a no_user_files
[root@kauai ~]# for myfile in $(egrep '(ext2|ext3|ext4)' /etc/fstab | awk '{print $2}')
do
find $myfile -xdev \( -type f -o -type d \) -nouser -print
done

[root@danasmera ~]#for myfile in ${no_user_files[@]}; do chown  root:root $myfile;done

Follow similar steps for files/directories owned by non-existent domains.

[root@danasmera ~]# declare -a no_group_files
[root@danasmera ~]# for myfile in $(egrep '(ext2|ext3|ext4)' /etc/fstab | awk '{print $2}')
do
find $myfile -xdev \( -type f -o -type d \) -nogroup -print
done

[root@danasmera ~]#for myfile in ${no_group_files[@]}; do chown  root:root $myfile;done

For more information on hardening your Operating system or application, go to the Center for Internet Security website, an download the freely available Benchmarks. The Benchmarks are ‘scorable’, easy to follow steps by step instructions on how to secure you box.