Linux – Mount partition from a raw disk image : dd and mount

In this post, I will share how you can mount a raw disk image such as an image generated with dd. Raw disk image or RAW Image Format is a bit-for-bit copy of disk data, without any metadata information on files. In Linux, dd is a popular tool for data transfer by duplicating entire disk for instance. Let us create a disk image of a mount with an EXT3 file system –

[root@kauai src]# dd if=/dev/sdb of=disk.img 

7233761+0 records in
7233760+0 records out
3703685120 bytes (3.7 GB) copied, 236.166 s, 15.7 MB/s

[root@kauai src]# ls -alh disk.img 
-rw-r--r--. 1 root root 3.5G Jan 15 18:44 disk.img

We have copied a mount with multiple files into a single disk.img file which we can copy to another system. Now let us examine the raw disk layout, that we can use to mount as a file system –

[root@kauai src]# fdisk -lu disk.img 
You must set cylinders.
You can do this from the extra functions menu.

Disk disk.img: 0 MB, 0 bytes
124 heads, 62 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xdebbbd93

   Device Boot      Start         End      Blocks   Id  System
disk.img          630416      945623      157604   83  Linux

As we can see the raw disk has 512 byte size sectors and it starts at offset 630416, given this information we can use mount command to mount the disk image –

[root@kauai src]# mount -o loop,offset=$((630416*512)) disk.img /mnt/hdisk/
[root@kauai src]# ls -al /mnt/hdisk/
total 37
drwxr-xr-x. 3 root root  1024 Jan 15 18:39 .
drwxr-xr-x. 4 root root  4096 Nov 17 20:04 ..
-rw-r--r--. 1 root root    15 Jan 15 18:39 file21
-rw-r--r--. 1 root root    15 Jan 15 18:39 file22
-rw-r--r--. 1 root root    15 Jan 15 18:39 file23
-rw-r--r--. 1 root root    15 Jan 15 18:39 file24
-rw-r--r--. 1 root root    15 Jan 15 18:39 file25
-rw-r--r--. 1 root root    15 Jan 15 18:39 file26
-rw-r--r--. 1 root root    15 Jan 15 18:39 file27
-rw-r--r--. 1 root root    15 Jan 15 18:39 file28
-rw-r--r--. 1 root root    15 Jan 15 18:39 file29
-rw-r--r--. 1 root root    15 Jan 15 18:39 file30
drwx------. 2 root root 12288 Jan 15 18:37 lost+found

[root@kauai src]# cat /mnt/hdisk/file26 
File number 26

Here we were able to mount the disk image and be able to read the content of one of the text files.

References –

https://en.wikipedia.org/wiki/Dd_(Unix)

https://linux.die.net/man/8/mount

How to copy to a clipboard in Linux

Problem statement – You have a file with hundreds or thousands of lines and you want to copy the contents of this file and paste it to an external application, for instance to a browser.

Solution – The first attempt is to try to cat the file and scroll down with your mouse to select each line. This is time consuming or in some cases might not work if there are too many lines as some of the lines will ‘scroll out of the terminal’. One way of getting around this is to use “xclip” – a command line interface to X selections (clipboard).

In my case I wanted to copy the contents of ‘/tmp/ipaddresses.txt’ file to a browser for blogging. The file had 10000 lines. I used the following commands, first to install xclip and then to copy the file contents to a clipboard –

apt-get -y install xclip
xclip -sel cli < /tmp/ipaddresses.txt

The xclip command basically does a selection (-sel) from the file into the clipboard(-cli), where you can copy paste to any other external application.

References
https://linux.die.net/man/1/xclip

https://stackoverflow.com/questions/5130968/how-can-i-copy-the-output-of-a-command-directly-into-my-clipboard

A project to track all active financial blogs.

No question there are too many personal finance, early retirement, deb etc. related blogs out there. It is very difficult to keep track of all these blogs. Luckily there is a project by Rockstar Finance not only keeps track of these blogs but also updates the list on a daily basis. Here is the link to the personal finance blog directory.

Looking at the list the categories of these blogs is very diverse, as of today it includes these –

    Budgeting
    Debt
    Early Retirement
    Family
    Frugality
    General Finance
    Investing
    Lifestyle
    Making Money
    Military
    Millennial
    Minimalism
    Real Estate
    Traditional Retirement

The directory also allows you to easily filter the list by –

    Category
    Age
    Sex
    Race
    City/State/Country
    profession

I found this directory quite useful in filtering out the relevant blogs that I am interested in based on the category. If you manage a personal finance blog, you can also submit your blog to the directory admins so that they can post it there.

How to fake or spoof x-forwarded-for header

The x-forwarded-for header is a way of identifying the IP address of the original client when a web server is sitting behind a proxy or load-balancer. The load-balancer does get the actual client IP as it directly sets up the TCP session with the load-balancer. But the x-forwarded-for address might contain a list of comma separated IP addresses in addition to the immediate client IP. It is these extra IPs that we can spoof and the procedure is similar to modifying any HTTP header such as user agent.

import requests
headers={'X-Forwarded-For':'1.1.1.1'}
r = requests.get('http://web.home.net/index.html', headers=headers)
if r.ok:
    print('Success.')

How the log likes like on an nginx access log –

1.1.1.1, 192.168.10.206 - - [19/Mar/2017:16:43:55 -0700] "GET /index.html HTTP/1.0" 200 1311 "-" "python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-121-generic"
1.1.1.1, 192.168.10.206 - - [19/Mar/2017:16:53:55 -0700] "GET /index.html HTTP/1.0" 200 1311 "-" "python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-121-generic"
1.1.1.1, 192.168.10.206 - - [19/Mar/2017:16:58:55 -0700] "GET /index.html HTTP/1.0" 200 1311 "-" "python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-121-generic"

The take away is not to trust any IPs in the x-forwarded-for list apart from the load balancer IP and the immediate client IP which made a direct call to the load balancer. If we trust our load balancer, we can also reliably identify the immediate client IP. The rest of the IPs in the x-forwarded-for list can be ignored.

References –

https://en.wikipedia.org/wiki/X-Forwarded-For

git – add local files to a git repository in local file system (bare git repo).

In this blow, I will show you how you can turn your local files into a github style repository. In my case I had files in `/etc/puppet` that I wanted to version control, but I wanted to push to a bare repository in the same machine or localhost. Here are the steps I followed –

Files to version control : /etc/puppet
Bare git repository that we will push changes in /etc/puppet : /var/lib/puppet/gitrepo/

1. Create a github style git repository in /var/lib/puppet/gitrepo

root@linubuvmb:/# mkdir -p /var/lib/puppet/gitrepo && cd /var/lib/puppet/gitrepo
root@linubuvmb:/var/lib/puppet/gitrepo# git --bare init
Initialized empty Git repository in /var/lib/puppet/gitrepo/

2. Initialize files as git repository

root@linubuvmb:/# cd /etc/puppet
root@linubuvmb:/etc/puppet# git init
Initialized empty Git repository in /etc/puppet/.git/
root@linubuvmb:/etc/puppet# git add .
root@linubuvmb:/etc/puppet# git commit -m 'First commit'
[master (root-commit) b71ef42] First commit
 50 files changed, 3913 insertions(+)
 create mode 100644 auth.conf
 create mode 100644 environments/example_env/README.environment
 create mode 100755 etckeeper-commit-post
 create mode 100755 etckeeper-commit-pre
 create mode 100644 fileserver.conf
 create mode 100644 manifests/base.pp
 create mode 100644 manifests/nodes.pp
 create mode 100644 manifests/site.pp
 create mode 100644 modules/apache/manifests/init.pp
...

3. Add bare repo as remote

root@linubuvmb:/etc/puppet# git remote add origin file:///var/lib/puppet/gitrepo/

4. Push to local git repository

root@linubuvmb:/etc/puppet# git push -u origin master
Counting objects: 84, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (70/70), done.
Writing objects: 100% (84/84), 129.33 KiB | 0 bytes/s, done.
Total 84 (delta 6), reused 0 (delta 0)
To file:///var/lib/puppet/gitrepo/
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin.
root@linubuvmb:/etc/puppet# git status
On branch master
Your branch is up-to-date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   puppet.conf

no changes added to commit (use "git add" and/or "git commit -a")
root@linubuvmb:/etc/puppet# git commit -a
[master f57997d] test
 1 file changed, 1 deletion(-)
root@linubuvmb:/etc/puppet# git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)

nothing to commit, working directory clean

Reference –

https://git-scm.com/documentation

List shared or dynamic libraries required by a program

In Linux, the

ldd

is used to find out the shared libraries or dependencies required by a program if it is a dynamic executable. ldd requires the full path to the executable as input.

For instance, the Linux ps command depends on the following shared or dynamic libraries –

[root@kauai rtc0]# ldd $(which ps)
	linux-vdso.so.1 =>  (0x00007ffeb6277000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003ef6200000)
	libproc-3.2.8.so => /lib64/libproc-3.2.8.so (0x0000003ef4e00000)
	libc.so.6 => /lib64/libc.so.6 (0x0000003ef4a00000)
	libdl.so.2 => /lib64/libdl.so.2 (0x0000003ef5600000)
	/lib64/ld-linux-x86-64.so.2 (0x0000003ef4600000)

You can also use the ldd command to find out if an executable has an expected dependencies. In this case, we expect that the htpasswd, login and sshd commands depend on the crypt library as they prompt a user for a password for authentication purposes –


[root@kauai rtc0]# ldd $(which htpasswd) |grep crypt
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f010c8ab000)

[root@kauai rtc0]# ldd $(which login) | grep crypt
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x0000003efd200000)

[root@kauai rtc0]# ldd $(which sshd) | grep crypt
	libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007ffb0b1f2000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007ffb0a988000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007ffb0a015000)

References –

http://man7.org/linux/man-pages/man1/ldd.1.html

Getting date from Real time clock (RTC) without using the date command or any other Linux time related commands.

In Linux, the “Real Time Clock” tracks wall clock time and is battery backed so that it works even with system power off. The RTC has no concept of time zone or daylight saving, it defaults to UTC. One of the user interfaces that the Linux Kernel exposes is

 /sys/class/rtc/rtc{N} 

and we will use the files in that directory to directly read time related data from the RTC.

* Files –

[root@ns3 rtc0]# ls /sys/class/rtc/rtc0
date  dev  device  hctosys  max_user_freq  name  power  since_epoch  subsystem  time  uevent  wakealarm

* Date and time in UTC

[root@ns3 rtc0]# cat date
2015-01-19
[root@ns3 rtc0]# cat time
23:05:05

* The maximum interrupt rate an unprivileged user may request from this RTC.

# cat max_user_freq
64

* The name of the RTC corresponding to this sysfs directory

[root@ns3 rtc0]# cat name
rtc_cmos

* The number of seconds since the epoch according to the RTC

[root@ns3 rtc0]# cat since_epoch
1421708627

* Status information is reported through the pseudo-file /proc/driver/rtc

[root@ns3 rtc0]# cat /proc/driver/rtc
rtc_time        : 23:06:58
rtc_date        : 2015-01-19
alrm_time       : 01:00:02
alrm_date       : ****-**-**
alarm_IRQ       : no
alrm_pending    : no
24hr            : yes
periodic_IRQ    : no
update_IRQ      : no
HPET_emulated   : no
DST_enable      : no
periodic_freq   : 1024
batt_status     : okay

References –

Real Time Clock (RTC) Drivers for Linux

Update – Eritrean Ethiopian Internet radio

The server hosting the Internet radio for Eritrean and Ethiopian mostly Tigrigna music has been migrated to a new infrastructure and thus the public Internet IP address of the streaming radio has changed. Please use this URL to get the latest streaming address or save the below updated streaming playlist file –


NumberOfEntries=1
File1=http://162.247.79.20:8000/test.mp3
Title1=Eritrean Ethiopian - Tigrigna
Length1=-1
Version=2

Download the stream playlist in Linux –

wget -O tigrigna-music.pls http://linuxfreelancer.com:8000/listen.pls

Nginx / Apache – log real client IP or x-forwarded-for address.

Web servers such as Nginx or Apache when configured as reverse proxy behind a load balancer, they log the IP address of the load balancer in the access logs as the source IP. For practical use cases, you will usually want to log the actual client IP addresses.

In this setup, Nginx is setup to mimic a load balancer (reverse proxy) with multiple Apache web servers as backend.

1. Nginx snippet configuration to set x_forwarded_for proxy header –


server {
  listen 80;
  listen 443 default ssl;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   .....
   }

2. Apache snippet Configuration to capture x_forwarded_for header in the access logs –


<VirtualHost *:443>
    ServerAdmin webmaster@home.net
    DocumentRoot /var/www/homenet
    ServerName todo.home.net
...
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy
    SetEnvIf X-Forwarded-For "^.*\..*\..*\..*" forwarded
    CustomLog "logs/todo.home.net-ssl_access_log" combined env=!forwarded
    CustomLog "logs/todo.home.net-ssl_access_log" proxy env=forwarded
 </Virtual>

Before making the above custom changes , the logs showed the load balancer IP 192.168.10.162 only –

192.168.10.162 - - [19/Mar/2015:16:21:10 -0700] "GET /signup.php HTTP/1.0" 200 1237
192.168.10.162 - - [19/Mar/2015:16:21:11 -0700] "GET /login.php HTTP/1.0" 200 1715

After the change the client IP (192.168.10.105) was logged –

192.168.10.105 - - [19/Mar/2015:16:26:43 -0700] "GET / HTTP/1.0" 200 1311 "https://todo.home.net/login.php" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:35.0) Gecko/20100101 Firefox/35.0"
192.168.10.105 - - [19/Mar/2015:16:26:44 -0700] "GET /signup.php HTTP/1.0" 200 1237 "https://todo.home.net/" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:35.0) Gecko/20100101 Firefox/35.0"

References –

https://www.nginx.com/resources/wiki/start/topics/examples/likeapache/

Google Chrome browse is not using my host DNS settings.

Environment –
Operating System : CentOS release 6.8
Google Chrome: Version 55.0.2883.87 (64-bit)

I have an Intranet DNS server with internal domain name. The domain name is resolved internally by my DNS server to an internal private IP address. With Firefox I could always visit my internal site without issues, but recently I installed Chrome browser into my CentOS desktop and when I tried to visit my internal site, it was directing me to an Internet site which I don’t own. Apparently Google Chrome was ignoring my dns setting and using its own name servers.

My first attempt – flush DNS on Chrome (failed)
I went to the DNS configuration for chrome and cleared host cache. The dns settings clearly showed my ‘nameservers’ as the 192.168.1.1 (my internal name servers) and yet Chrome was not using it. Even after cleaning the host cache and flushing sockets, it didn’t help.

Chrome setting to view and manage DNS settings –

chrome://net-internals/#dns

My second attempt – block IPv6 (worked)
After running tcpdump on port 53, Chrome was calling an IPv6 address 2001:4860:4860::8888 to resolve domains.

Added below lines to

 /etc/sysctl.conf 

in order to disable IPv6 temporarily for a test –

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

And executed the command

 sysctl -p 

to apply the new kernel settings. After this change and flushing dns, I was able to visit my internal site.

I am guessing the DNS IPs used by Chrome as somehow internally coded, and haven’t been able to find those settings.

References –

https://www.reddit.com/r/chrome/comments/1xj69t/chrome_ignoring_system_dns_and_using_google_dns/

https://www.howtogeek.com/197804/how-to-clear-the-google-chrome-dns-cache-on-windows/