Just more thoughts

Raspberry pi - Introduction

2014-12-02T16:55:00.002+01:00

Hello once again.

This is the first of series of Raspberry pi related posts. I believe that the Raspberry pi (rpi from now on) can help us learn from basic electronics to coding, presenting alternative solutions to existing or not problems. In this series I will make a step-by-step installation of a simple, descent LAMP (Linux Apache MySql PHP) server.

This is what a rpi looks like, compared to a D, a AAA battery and a SD card.

Introduction
For those not familiar with the pi, I think it will enough to start with the following bullets:

it is a single-board processor
it has low cost, (approximately 35$)
low performance, ARM6 700Mhz single core processor with 256/512Mb of RAM (type A/B)
runs a distribution of GNU/linux based on Debian, called raspbian
has some GPIO (General Purpose Input Output) pins, used to connect sensors, devices etc in order to control them, get data or output data to
it has a large community for project ideas/solutions

I believe those are enough for now. In a few words the rpi is a small pc that is not meant to be used as a pc (what th.. ??).

If you don`t have one, then I hope after reading this post, you will consider getting your hands on one.

No arduino

Arduino is a microcontroller, it is not to be confused with the pi. They can work together, but they are not the same. Both projects are very popular, which is the reason why there are clones. In the case of the pi, a popular clone is the banana pi, shown in the pic below, on the right. It is in the same price range, faster (1 GHz dual core ARM7) with more RAM (1 GB), but still ... Not so well supported. More power does not always mean better.

Why bothering?

The best answer is why not? I own mine several months and I have tried a few projects. Through those projects I had to read and learn a few things about electronics (remember that Ohm's law you have learnt in school, that R=I/V?). Moreover I got my hands dirty with solder, cables, LED's, but I also had to use python, setup an apache web server, remember the mysql syntax and of course so many more. So, yes, I believe I have learnt a thing or two.

Disclaimer

I am not an expert, so throughout these series I am only putting together pieces I have found online, I have tried and they work. At least in the cases mentioned. Maybe there are better ways (most likely) to get the same result. Maybe mine suck, but at least, so far, I do know that they do not suck hard. If so please make your point in a comment.

Getting started

If you about to order your slice of pi, then keep in mind that you will need:

one SD card
one USB type A to male micro USB type B (the standard USB cable that comes with a smartphone)
one USB charger (5V, 1A)
one USB keyboard *
one USB mouse * (just to login in the GUI and play around)
one HDMI cable *
one monitor that supports HDMI *
one UTP Ethernet cable

* you will need them only once, to boot the first time, then SSH will be used

If you have put things together, then check again in a few days for the next post.

C u.

sendmail app - GUI and CLI versions

2014-03-16T20:38:00.005+01:00

Hello there.
I recently needed to automate a process of sending emails from MS windows machine, to lots of recipients in order to help a good friend at his work. Think of it as newsletter emails. This is why I made a small .exe file in VB.net . I had little experience with VB.net, this is why I started from the .exe file with GUI which was easier from me to visualize and debug. To cut story short. I made it work. However, without CLI how is it possible to automate it? I know there is autoit program, but ... come on. I 'd spent quite some time to make this work. So I made a CLI version too.

The use is simple.
In the GUI version there are listboxes to add attachments and recipients. Labels are self-explanatory so no need to say much.

The CLI version takes these arguments:
-u=[username]
-p=[password]
-s=[server]
-r=[port]
-b=[subject]
-t=[mail text]
-l=[.txt file path for recipients]
-a=[path of the attachment]
-o=[path of the txt log file to write]
--help for help
e.g. for using gmail
sendmail_cli -u=[yourusername@domain.com] -p=[yourpassword] -s=smtp.gmail.com -r=587 -b=newsletter -t=newsletter -l=[absolute path to the txt file with recipients]t -a=[absolute path to attachment] -o=[absolute path to the txt log file]
The messaging is up to minimum, because I know how to work with it and so does my pal. Recipients are added to bcc. You can use any "antivirus / analyzer / anti-malware / sandbox / virtual machine / local mail server" to make sure there is no malicious code there.

Download the sendmail.exe file here and the CLI version here.

Simple password generator from terminal

2014-01-18T20:46:00.001+01:00

Hello again.

I wanted to get several WPA passwords and test them by trying to hack them for educational purposes.

Since I didn`t want to use an online generator (because when you don`t have a connection they are useless) I though I should have them taken from terminal.

The procedure is simple and it worked. At least for me.

1st open your bash source file
gksudo ~/.bashrc

2nd add this small script

genpasswd() {
    local l=$1
           [ "$l" == "" ] && l=16
          tr -cd 'a-zA-Z0-9\<>!".;:$%&/()=?|@#[]}{'< /dev/urandom | head -c ${l} | xargs
}

3rd save and exit

4th source it
source ~/.bashrc

and that`s it.

try it by typing
genpasswd 10

to get a password with 10 characters

GL and HF

Conky has been polished and refurbished

2012-03-11T21:50:00.002+01:00

Hello again.

Conky is changing. Remember this article? Get ready to see something different. Compare it with this one.


Old one	Polished version

I have left out several indicator, but it looks really nice.

Here is how to get it.

If you don't have conly, install it.
$sudo apt-get install conky

Donwload the new source file, as well as the logo.png image the rings from here and extract them in a folder, let;s call it conky2 in your home folder.

First step is to install the new font, called neuropolitical by double clicking in the neuropolitical.tif file in the neuropolitical.zip file and then the install button.

Then copy the clock_rings.lua to ~/.lua/scripts folder. These two folders do not exist so create them.
$sudo mkdir ~/.lua
$sudo mkdir ~/.lua/scripts

and then copy the clock_rings.lua file
$cd ~/
$sudo cp ~/conky2/Ubuntu/clock_rings.lua ~/.lua/scripts

Create a folder to put the logo.png file.
$sudo mkdir ~/.conky
$sudo cp ~/conky2/Ubuntu/logo.png ~/.conky

Inside the same folder put the conyrc file.

Last change the code for your town to show temp for it. Visit http://weather.noaa.gov, find your town and note the code in bold and in parenthesis up right, as shown below (LGLM for my case).

Find the line below in the ~/.conky/conkyrc file

${font Neuropolitical:size=13}${color FFFFFF}${alignr}temp: ${weather http://weather.noaa.gov/pub/data/observations/metar/stations/ LGLM temperature temperature 30} °C${font}

NOTE
The neuropolitical font does not support all UTF-8 characters, so if you use a different language, greek in my case, change the font for those lines in the conkyrc file with arial, to be sure for the result.

Trigger conky by typing
$conky -c ~/.conky/conkyrc

Good luck.

New AMD ATI catalyst drivers. The scenery is changing.

2012-03-11T21:18:00.000+01:00

Hi there once more.

It has been a while. Remember the previous post how to install the ATI drivers in Ubuntu? Well you can forget all about that.

It seems that something has changed and all the necessary actions are included in a wizard type installation for the AMDCatalyst 12.2 proprietary display drivers for Linux.

First, go here (AMD support site) to get the drivers for your OS and card.

I can say that you are almost done. You must only make this *.run file executable and then run it.
(I won't open terminal to show howeasy the whole process is)

Enter your password to allow it to run as superuser and a wizard will start as shown below. Choose according to yor case and the rest are being taken care of. A reboot however is necessary.

Has this new policy anything to do with the fact that nvidia has joined The Linux Foundation?

Read more here (endgadget.com)

What do you think?

Install the new ATI drivers in Ubuntu 11.10 (and make them work)

2011-11-18T20:41:00.001+01:00

Hi there once more.

Following the previous post about problems with ATI drivers, amdcccle (see it here), I' ve looked it into a little bit more.

I was lucky since guys at osarena.net (they rock), had the solution ready for me.

New ATI catalyst 11.8 to the rescue and a proper easy how-to.

This is basically a re-post of the one found in osarena.net (check it here) from the noobie perspective.

First download the new drivers from here. (I got the 64bit one)

Put the ati-driver-installer-11-11-x86.x86_64.run file in a folder (lets name it ati in the home folder) and make it executable.
$cd ~/ati
$sudo chmod +x ati-driver-installer-11-11-x86.x86_64.run

Then create the necessary packages, based on the 11.10 Ubuntu version

$sudo ./ati-driver-installer-11-11-x86.x86_64.run --buildpkg Ubuntu/oneiric

Before continuing, it is very likely that you might be missing some packages that are necessary to create the packages and cpontinue with this procedure. I've listed below those that I was missing. I have that covers you. One way or another you' get a message to install missing ones.

$sudo apt-get install build-essential cdbs fakeroot dh-make debhelper debconf libstdc++6 dkms libqtgui4 execstack libelfg0 dh-modaliases

Now open the tty session (Ctrl+Alt+F1) and login with your username and password. This is because we are about to uninstall the currently used drivers and we prefer a Command Line Interface (CLI) instead of a graphical one that is using a lot more of the resources of the card.

List folder ati and see the packages.

$ls

fglrx_8.911-0ubuntu1_amd64.deb
fglrx-amdcccle_8.911-0ubuntu1_amd64.deb
fglrx-dev_8.911-0ubuntu1_amd64.deb

Remove old drivers
$sudo apt-get remove --purge fglrx fglrx_* fglrx-amdcccle* fglrx-dev*
Install the new packages (overwriting -f)
$sudo dpkg --force-overwrite -i *.deb

Create the xorg.conf file
$sudo aticonfig --initial -f

and restart

$reboot

After the reboot, when you log in, you might receive an error message that previous settings could not be restored.

$gksudo amdcccle

to open the catalyst and set your card again.

You are done.

I've have noticed 2 degrees Celsius down in the adapter temperature from my conky file (see what I mean) and a lot smoother sense of the card.

Thank you again osarena.net guys.

Good luck to you all.

Ubuntu 11.10 ATI drivers and NTFS mounts problem

2011-11-17T18:04:00.001+01:00

Hi there again Ubuntu users.

This will be a small post about a problem I faced when I upgraded to 11.10 and it has to do with ATI drivers and NTFS mounts. I ' ve googled them around and I see that there are quite some people facing similar problems.

(I run the 64-bit version, so I do not know if the same applies for the 32-bit one.)

I use two monitors and when the PC rebooted, after the update, the monitor mode had been switched to mirror monitors. I though OK, no problem. It is something that has happened before in the past and I thought that AMD Catalyst Control Center will give the solution. Unfortunately, even though I was using the AMD Catalyst Control Center (administrative version) it didn't save any changes and simply terminated. Each time.

So I tried to "kill my time" and watch a movie instead. I tried to cut and paste a movie to an external media center. No go. I didn't have write privs in the NTFS external HDD. That was new. It took a while (reading around the community forums etc) and I came to the conclusion that must be a policy problem. This triggered me and I thought that it might also be the solution to the ATI drivers problem.

Enough said (damn it was long)

I am rookie andI didn't want to mess around any /etc/ .... conf files, so i did it manually the "newbie" way. I suggest try it only since it might not be the solution to your problem. I believe the community and Canonical will provide solution to both problem in future updates. Until then we have to be operational at least.

ATI problem solution
Simply download the proprietary drivers form the system settings section and run the AMD Catalyst Control Center (administrative version) from terminal
$gksudo amdcccle

That's all. It will work just fine (use gksudo, not sudo since it is a GUI app).

NTFS mounts problem
The solution is seperated in two parts. Mounting and unmounting.

Mounting
First you will need to know which device to mount. To list the connected HDD devices type
$sudo fdisk -l

It will list several devices and their partitions. The format will like this:

Disk /dev/sda: 60.0 GB, 60022480896 bytes
255 heads, 63 sectors/track, 7297 cylinders, total 117231408 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00006d97

Device	Boot	Start	End	Blocks	Id	System
/dev/sda1	*	2048	391167	194560	83	Linux
/dev/sda2		393214	117229567	58418177	5	Extended
/dev/sda5		393216	16015359	7811072	83	Linux
/dev/sda6		16017408	113324031	48653312	83	Linux
/dev/sda7		113326080	117229567	1951744	82	Linux swap/SOlaris

/dev/sda is the device

The rest in the table are the partitions.

Type

$sudo mount .... partition to mount it.

Done.

Unmount

You' ll need same privs to unmount the devices mounted before.

First list the mounted devices. Type

$mount | grep /dev/

and the mounted devices will be highlighted. Find the one you want and then

$sudo umount [device]

That's all.

Maybe fstab can be edited, but I am not in position to recommend it. I hope a near future update will deal with these problems.

Try it and I hope it will help. Additional info about Ubuntu device management can be found here.

Various 802.11 Wireless Standards Research

2011-11-10T10:52:00.000+01:00

Scope

The scope of this paper is to research through the IEEE 802.11 wireless standards identifying range, security, throughput, costs and the future prospects of this technology. Since the number of specifications, ratified and draft amendments to the 802.11 Prime is big, we will not include those that are addressed to manufacturers, and we will cover 802.11 a/b/g/n specifications (The book of wireless, p.32) referring to the rest necessary amendments for the security section only.

Introduction

IEEE 802.11 Wireless Standard defines the technology used for local area network (LAN) communications, using radio frequencies (RF). Institute of Electrical and Electronics Engineers (IEEE) is the professional society that creates and Maintains standards that we use for communications (CWNA, p. 154). The original IEEE 802.11 standard was published in

1999 and reaffirmed in 2003. The latest revision of the standard was published in 2007. The publication date of the specifications to be covered in this paper, are shown below:

IEEE Std 802.11-1999 (R2003)
IEEE Std 802.11a-1999
IEEE Std 802.11b-1999
IEEE Std 802.11g-2003
IEEE Std 802.11n-2009

(IEEE Std 802.11™-2007 Revision of IEEE Std 802.11-1999)

“IEEE specifically defines 802.11 technologies at the Physical layer and MAC sublayer of the Data-Link layer. By design, the 802.11 standard does not address the upper layers of the OSI model, although there are interactions between the 802.11 MAC layer and the upper layers for parameters such as quality of service”. (CWMA, p.155)

A brief comparison and overview of the IEEE 802.11 Legacy/a/b/g/n is shown below, in order to be used as a visual reference thoughout the rest of the paper.

	802.11 Legacy	802.11 a	802.11 b	802.11 g	802.11 n
Frequency	2.4 GHz ISM band	5 GHz UNII-1, UNII-2 and UNII-3 bands	2.4 GHz ISM band	2.4 GHz ISM band	2.4 GHz ISM band – 5 GHz UNII band
Spread Spectrum Technology	FHSS or DSSS	OFDM	HR-DSSS (PBCC is optional)	ERP:ERP-OFDM and ERP-DSSS.CKK are mandatory ERP-PBCC and DSSS-OFDM are optional	HT MIMO with OFDM. HR-DSSS, OFDM, ERP.
Data Rates	1, 2 Mbps	6,12 and 24 Mbps are madnatory. Also supported are 9, 18, 36, 48 and 54 Mbps.	DSSS:1, 2Mbps HR-DSSS:5.5 and 11 Mbps	ERP-DSSS/CCK: 1,2, 5.5 and 11 Mbps ERP-OFDM:6, 12 and 24 Mbps are mandatory Also supported are 9, 18, 36, 48 and 54 Mbps. ERP-PBCC:22 and 33 Mbps	Data rates vary from 6,5 Mbps to 600 Mbps due to different type of modulation and Channel width (20/40MHz) (IEEE 802.11 n-2009 pages 345-354)
Backward Compatibility	N/A	None	802.11 DSSS only	802.11 HR-DSSS and 802.11 DSSS

(CWNA, p. 162, p.590 and IEEE 802.11 n-2009 pages 345-354)

Range

The coverage area of the standards above is related to the part of the spectrum they operate, the Radio Frequency behaviors of absorption, reflection, attenuation and multipath.

802.11a and 802.11n operate in 5 GHz Unlicensed National Information Infrastructure (UNII-USA) spectrum. On the other hand the 802.11 b/g/n (when operating in 2,4 GHz) operate in 2,4 GHz Industrial, Scientific and Medical (ISM) band. This means that 802.11 b/g/n (when operating in 2,4 GHz) suffer from less attenuation compared to the 5 GHz band, because of the longer wavelength. This allows signal to reach theoretically further. Table 2 shows the Attenuation Comparison of Materials for 2,4 GHz.

Material	2,4 GHz
Foundation wall	-15dB
Brick, concrete, concrete blocks	-15dB
Elevator or metal obstacle	-10dB
Metal Rack	-6dB
Drywall of Sheetrock	-3dB
Nontinted glass windows or door	-3dB
Wood door	-3dB
Cubicle wall	-2dB

Table 2. Attenuation Comparison of Materials (CWNA, p.47)

Moreover under the consideration that wireless networks are mostly used indoors, the RF behavior of absorption affects the coverage area. Further the 2,4 GHz band is not used exclusively by the 802.11 standards wireless devices, but it is used by other home appliances, as well as bluetooth devices. This means that there is danger of interference, since the channels may be already used, leading to loss of packets, which need to be resent for transmission to be acknowledged. This does not apply to the 5GHz band.

In addition, 802.11n standard multiple input-multiple output (MIMO) amendment to the standard allows it to take advantage of the reflection of waves in furniture, halls, corridors etc converting the multipath RF behavior form enemy to asset.

Outcome:

802.11 b/g/n (operating in the 2,4 GHz band) provide a larger coverage area comparing them to the ones operating under the 5 GHz band. They are even more effective when we have knowledge about the surrounding transmitting devices (home appliances and neighbor wireless devices) and the channels they operate. Channels 1-6-11 are recommended for the 2,4 GHz band because they do not interfere with one another. (The book of wireless, p.35) A practical coverage area for these standards should be estimated approximately to 17m (~50ft) (Wireless Networks for Dummies, p.30)

Security

802.11a/b standards where published in 1999 using the Wired Equivalent Privacy (WEP) as their cryptographic solution for security. WEP provides authentication, ensuring that information is accessible only to those authorized to have access and encryption, meaning that the data is intact and that no one has tampered with it, with 40 to 128 bit key length. It uses RC4 algorithm for confidentiality and CRC algorithm for integrity. CRC-32 is a hash function whose goal is to prevent accidental changes, but since no authentication exists for the AP any intentional intervention from an intruder is not detected. This makes WEP key insecure for replay attacks. Replays attacks are repeat transmission of valid data, previously intercepted by a valid user by a man-in-the-middle during a conversation. Replay attacks aim in gaining authentication credentials or session initialization messages.

Table 4. WEP key basic vulnerabilities

1	AP are not authenticated
2	There are patterns in the final key production making it predictable
3	Key is the same for all mobile nodes
4	Intentional intervention in packets is not detectable
5	Keys are not updated automatically

Result of the weaknesses above was the WPA (1.0 with TKIP and WPA 2 keys introduced with 802.11i in July 2004.

Even though compatibility issues have been taken under consideration, firmware updates are required to old devices, in order to implement the new directives of the 802.11i standard. However, such updates are not always an option, narrowing the security level.

Finally, 802.11 g/n standards have protection mechanisms that implement the backward compatibility with older devices, allowing Access Points to support different standards at the same time.

Throughput

Data rates for each of the standards are shown in Table 1. Even though these data rates are the theoretical speed they can give us a visual comparison for the throughput taking in consideration that the “aggregate throughput is typically one-half or less of the available data rate speed” (CWNA, p.156).

Moreover, it must be stated that protection mechanisms that for compatibility issues, allows the 802.11 b/g/n to coexist at the same time affects the throughput as well. “An 802.11 b/g access point with a data rate of 54 Mbps might see a decrease in aggregate throughput from 20Mbps down to as little as 8 Mbps the instant the protection mechanism is enabled.” (CWNA, p. 161)

Costs - Future prospects of this technology

Devices supporting IEEE 802.11b were the first to be widely accepted in market. Their success is shown in by the backward compatible standards IEEE 802.11g/n. It is obvious that new devices are more expensive and even though new standards offer improvements, they come with a price as well. The upgrade to new devices for companies depends on the value for money they offer and according to certain needs they cover, since it is part of a bigger plan that is affected by the current infrastructure.

Taking under consideration different scenarios and different variables such as compatibility, battery consumption for smartphones-tablets and data rates, devices that support different 802.11 standard can find their place in the market. For example 802.11b offers lower power consumption, but 802.11n devices, implement a new technology that offers more and better services with backward compatibility.

Trends so far have shown that it takes a while for a standard to gain its part in the market after it has been published. This gives time to older devices, while offering more time for testing and debugging the new ones.

Conclusion

Since there is no given scenario to which we must adapt our comparisons and provide a solid direction towards which standard is better, it must be stated that each IEEE 802.11 standard can be presented as the best solution according to certain given facts. We should filter our needs through the basic guidelines above, in order to reach to a conclusion.

References

The book of Wireless 2^nd Edition, John Ross, No Strach Press, 2008
CWNA, Certified Wireless Network Administrator, Official Study Guide, David Coleman, David Westcott, Wiley 2009
IEEE Std 802.11™-2007 (Revision of IEEE Std 802.11-1999) as found at http://standards.ieee.org/getieee802/download/802.11-2007.pdf on 30 Oct 2011.

IEEE Std 802.11n™-2009 (Amendment to IEEE Std 802.11™-2007 as amended by IEEE Std 802.11k™ -2008, IEEE Std 802.11r™-2008, IEEE Std 802.11y™-2008, and IEEE Std 802.11w™-2009) as found at http://standards.ieee.org/getieee802/download/802.11n-2009.pdf on 30 Oct 2011.

Wireless networks for Dummies, Barry D. Lweis, Peter T. Davis, Wiley, 2004

My Conky conf file

2011-08-08T21:59:00.002+02:00

Updated - Weather forecast included

Conky is a well knwon, light-weight system monitor for X. It is highly customizable and you can use it to display useful information in your desktop. The configuration file can be found in ~/.conkyrc.

The package for Ubuntu can be found in the software control centre or type

sudo apt-get install conky

This is a print screen of how I use it.

curl, tail and awk packages are used in the example above. If conky fails, check if you have them installed.

I have included my conf file below. Advanced readers should give a look in the links below to modify the conf file according to their needs.

Official site - http://conky.sourceforge.net/

Variables info - http://conky.sourceforge.net/variables.html

Configuarion info - http://conky.sourceforge.net/config_settings.html

To use the weather forecast you need to go here, to ubuntuforums, and follow the instructions according to your needs.

In a few words you' ll need to find your location in accuweather and copy the URL, place the accuweather_conky folder in your home folder and then add/modify the lines from the conky_acc_int to your .conkyrc file.

TeoBigusGeekus has created full how-to pdf files that will guide you through.

Good luck.

~/.confrc file

background yes
use_xft yes
xftfont HandelGotD:size=9
xftalpha 0.5
update_interval 0.5
total_run_times 0
own_window yes
own_window_type normal
own_window_transparent yes
own_window_hints undecorated,below,sticky,skip_taskbar,skip_pager
double_buffer yes
minimum_size 450
maximum_width 200
draw_shades no
draw_outline no
draw_borders no
draw_graph_borders no
default_color gray
default_shade_color black
default_outline_color green
alignment bm
gap_x -230
gap_y 150
no_buffers yes
uppercase no
cpu_avg_samples 2
override_utf8_locale no

TEXT
$sysname $kernel on $alignr $machine
Uptime $alignr $uptime
Load $alignr $loadavg
Hostname $alignr $nodename
${hr 2}
$alignc${color 8e8e8e}Lemnos${execi 600 bash ~/accuweather_conky/accuw_script}
${font conkyweather:size=40}${execi 600 sed -n '6p' ~/accuweather_conky/images}${font}${goto 75}${voffset -40}${execpi 600 sed -n '1p' ~/accuweather_conky/days}
${goto 75}High: ${execpi 600 sed -n '2p' ~/accuweather_conky/temperatures}C Low: ${execpi 600 sed -n '3p' ~/accuweather_conky/temperatures}C

${font conkyweather:size=40}${execi 600 sed -n '7p' ~/accuweather_conky/images}${font}${goto 75}${voffset -40}${execpi 600 sed -n '3p' ~/accuweather_conky/days}
${goto 75}High: ${execpi 600 sed -n '4p' ~/accuweather_conky/temperatures}C Low: ${execpi 600 sed -n '5p' ~/accuweather_conky/temperatures}C

${font conkyweather:size=40}${execi 600 sed -n '8p' ~/accuweather_conky/images}${font}${goto 75}${voffset -40}${execpi 600 sed -n '5p' ~/accuweather_conky/days}
${goto 75}High: ${execpi 600 sed -n '6p' ~/accuweather_conky/temperatures}C Low: ${execpi 600 sed -n '7p' ~/accuweather_conky/temperatures}C

${hr 2}
$alignc NETWORK
External IP $alignr ${exec curl -s http://checkip.dyndns.org | grep -i address | sed s'/<[^>]*>//g' | sed s'/Current IP CheckCurrent IP Address: //' | sed 's/$.*$./\1/'}
eth0 $alignr ${addr eth0}
Inbound $alignr ${downspeed eth0} kb/s
${downspeedgraph eth0}
Outbound $alignr ${upspeed eth0} kb/s
${upspeedgraph eth0}
${hr 2}
$alignc CPU
CPU1 running at $alignr ${freq_g cpu1} GHz
CPU2 running at $alignr ${freq_g cpu2} GHz
CPU1 ${cpu cpu1}% ${alignc 35}${cpubar cpu1}
CPU2 ${cpu cpu2}% ${alignc 35}${cpubar cpu2}
${hr 2}
$alignc ${exec aticonfig --adapter=0 --od-gettemperature | sed s'/Adapter 0 - //' | grep ATI | sed s'/ Series//'}
GPU load $alignr ${execi 2 aticonfig --adapter=0 --od-getclocks | grep -i GPU | cut -c 30-}
GPU Temp $alignr ${execi 2 aticonfig --adapter=0 --od-gettemperature | tail -n1 | awk '{ printf $5 } '} Celsius
${hr 2}
$alignc MEMORY
$mem / $memmax $alignr $memperc%
$membar
swap $alignc $swap / $swapmax $alignr $swapperc%
${swapbar}
${hr 2}
$alignc FILE SYSTEMS
/ $alignc ${fs_used /} / ${fs_size /} $alignr ${fs_free_perc /}%
${fs_bar /}
/home $alignc ${fs_used /home} / ${fs_size /home} $alignr ${fs_free_perc /home}%
${fs_bar /home}
${hr 2}
$alignc PROCESSES (${running_processes} / ${processes})
NAME $alignr PID CPU
${top name 1} $alignr ${top pid 1} ${top cpu 1}
${top name 2} $alignr ${top pid 2} ${top cpu 2}
${top name 3} $alignr ${top pid 3} ${top cpu 3}
${top name 4} $alignr ${top pid 4} ${top cpu 4}
${top name 5} $alignr ${top pid 5} ${top cpu 5}
${hr 2}

How to find your external IP from terminal

2011-08-08T21:44:00.002+02:00

Hi there Linux fans once more.

If you don't have a static IP from your ISP, then the line below can help you. It's a small combination of terminal commands that return the WAN IP you have.

Type in terminal.
curl -s http://checkip.dyndns.org | grep -i address | sed s'/<[^>]*>//g' | sed s'/Current IP CheckCurrent IP Address: //' | sed 's/$.*$./\1/'

Short explanation
The idea is to use a DNS server to tell the IP you have.
I use checkip.dyndns.org because the html result is easy to handle.
1. curl command scans the html.
2. The result is piped to grep for the first instance of address.
3. This results is again piped to sed editor that removes the < > symbols from the html tags.
4. (Guess what) the result is again piped to sed again to remove the strings that we don't want. Result = IP.
5. Last pipe removes the last symbol added at the end.

Give it a try. It comes handy.

I have included it in my conky source file. How? Check here.

Good luck.

How to recover deleted files in Ubuntu

2011-07-06T01:45:00.003+02:00

Hello there Ubuntu users once more.

How you deleted a file by mistake that you must desperately recover ?

I had and I searched around a way to recover that file, and many more, through terminal, because let's face it. When you are in black screen your options and knowledge is limited.

My suggestion in scalpel.

sudo apt-get install scalpel

to install the package.

The configuration file is the

/etc/scalpel/scalpel.conf file

where you must uncomment the type of files that you wish to recover. Once you have them uncommented,
run:

sudo scalpel <partition> -o <destination dir>
eg

sudo scalpel /dev/sda1 -o /home/user/recovered.

I recommend creating a directory, since is must be empty.

The whole process will take a while, since it goes through each block of the partition in the HDD, but it worths the time.

Good luck.

PS
Advanced readers are encouraged to look up more here.

Manage firewall in Ubuntu 11.04

2011-07-03T12:30:00.000+02:00

Hi the Ubuntu users.

I suggest ufw to manage the firewall. Check the status
sudo ufw status

enable it or disable it by typing
sudo ufw enable / disable

There are two default policies for the firewall. Either block all incoming unless otherwise expressed with rules, or allow all incoming connections unless otherwise blocked by rules.
I suggest to block all incoming and to add rules to allow specific ports, protocols or services to access you. However, this can be an issue for common use. Choose your policy wise.

sudo ufw default deny (or) allow
according to your needs. I'll continue assuming I have chosen the deny one.

sudo ufw status
will list any existing rules. Type

sudo ufw app list
to check the allowed services. Now type

sudo cat /etc/service | grep ftp | cat
to list the services for eg. FTP. To add a rule for port 20 and TCP protocol in ufw type
sudo ufw allow 20/tcp

Check it by typing sudo ufw status
Finally make sure to enable the firewall logging by typing
sudo ufw logging on.

Good luck.

Share folders through Samba in Ubuntu 11.04

2011-07-03T12:25:00.001+02:00

Hi there Ubuntu users.

This is a small and fast how to guide to enable sharing through sabma in Ubuntu 11.04.

First
sudo apt-get install smbfs
to install smb packages if you already ahven't.

Then create a password for the user whose folder you wish to share. In our case it the the "shareuser" user, so that we share his home directory and throw files locally.

Create a samba password for that user
sudo smbpasswd -a shareuser

Then add some lines to the smb.conf file
sudo nano /etc/samba/smb.conf

Add those lines

comment = shareuser Private Files
path = /home/shareuser
valid users = shareuser
public = no
writable = yes

Restart samba service and your done.
sudo restart smbd && nmbd

Make sure that your firewall is not blocking connections (check this for a firewall guide) and you are done.

Good luck

Setup FTP server in Ubuntu server 11.04 (Terminal only)

2011-07-03T11:47:00.005+02:00

Hey there Ubuntu users. This is a small how to guide, in order to setup a FTP server in Ubuntu 11.04 Server Edition using terminal only.

First you ' ll need to download vsftpd package from repos. Make sure that your list is updated.
sudo apt-get update
sudo apt-get install vsftpd

The service that will be used for the FTP server is vsftp under /etc/init.d/vsftpd

The configuration file for vsftpd can be accessed by typing
sudo nano /etc/vsftpd.conf

It is there where all configuration actions will take place. There many issues to be covered here.
Inside the conf file there are examples of what to allow and what to tweak. By uncommenting or uncommenting (#) each particular line will be read or not. Give it a look. If there is something you don't understand, just leave it as it is. Maybe by the end of this guide you' ll figure out what it does.

First by default anonymous users are not allowed.
anonymous-enable=NO
I suggest leaving that this way.

In addition by default local system users are allowed.
local_enable=YES

If you and the users to access through FTP know what they are doing you can uncomment this line.
#write_enable=YES
You can get the list of supported FTP commands here from wiki.

vsftp uses by default port 20
connect_from_port_20=YES
Leave as it is. If you change it, make sure to port forward the same port in you router and to allow the same port in your firewall. My suggestion is to leave it as is, setup the server and then when everything runs smooth you can experiment. For this guide, port 20 is used.
There more options below, such as to allow users to create directories etc. We stick to the basic issues since most of the other options depend on one's needs.

Find the line
nopriv_user=ftpsecure
This is the default user to access through ftp. This user is not created or exist by default in your system. We must create it. First list the current available users in your system to get a clue of how many they are.

cat /etc/passwd |cut -d":" -f1

As you can see there is no ftpsecure one. Let's create it.

OK hold it. One might say "Why create a user and not use one that already exists?"
The answer is simple. It will save time to check and change the privileges for each one of them. In addition some of them are blocked vsftpd by default.

sudo cat /etc/ftpusers
to list of the users BLOCKED by default. You can add the users from the previous list (cat /etc/passwd |cut -d":" -f1) in this file to block them, or some of them for security reasons.

OK. Let's proceed to create the user

sudo adduser -m /home/ftpsecure ftpsecure
This will create the user ftpsecure and create his home folder in /home/ftpsecure.

Let's move back to the conf file (sudo nano /etc/vsftpd.conf)
Find the line

chroot_local_user=YES
and I suggest to uncomment it. This wil resctrict the ftpsecure user to his home directory, otherwise the user will be able to go through the filesystem. This means that any files you want to share through FTP should be placed in this directory.
If you have enabled the ftp commands (check above), or not, this can be a security issue.

Save the changes done.
In order for them to take effect, restart the service
sudo restart vsftpd

However, we are not done. Or better say, we must not end here. For those who have not setup the firewall, suggestions are listed below.

I suggest ufw to manage the firewall. Check the status
sudo ufw status

enable it or disable it by typing
sudo ufw enable / disable

There are two default policies for the firewall. Either block all incoming unless otherwise expressed with rules, or allow all incoming connections unless otherwise blocked by rules.
I suggest to block all incoming and to add rules to allow specific ports, protocols or services to access you.

sudo ufw default deny (or) allow
according to your needs. I ll continue assuming I have chosen the deny one.

sudo ufw status
will list any existing rules. Type

sudo ufw app list
to check the allowed services. Now type

sudo cat /etc/service | grep ftp | cat
to list the services for FTP. To add a rule for port 20 and TCP protocol in ufw type
sudo ufw allow 20/tcp

Check it by typing sudo ufw status
Finally make sure to enable the firewall logging by typing
sudo ufw logging on.

This is where the guide should end, but some may have problems sharing files though Samba to put them in the /home/ftpsecure directory and share them though FTP. What's the point to create an FTP server if its hard to share files between network ? :)

First
sudo apt-get install smbfs
to install smb packages. Then create a password for the user whose folder you wish to share. In our case it the the ftpsecure user, so that we share his home directory and through files locally.

Create a samba password for that user
sudo smbpasswd -a ftpsecure

Then add some lines to the smb.conf file
sudo nano /etc/samba/smb.conf

Add those lines

comment = ftpsecure Private Files
path = /home/ftpsecure
valid users = ftpsecure
public = no
writable = yes

Exit and save.
Restart samba service and your done.
sudo restart smbd && nmbd

(if you have chosen the deny policy for ufw, then you must add the samba service in the allow rules, as shown above)

Good luck.

Ubuntu 11.04 - Network and Security (6/6)

2011-06-27T22:22:00.010+02:00

General Information
This is the result of personal research through community and official documentation about GNU/Linux and Ubuntu. References used are provided also. It will be presented in 6 posts as listed below:

Network

Linux support standard Internet protocols used for Unix-to-Unix communication, but also implements a number of protocols native to other, non-Unix operating systems. “The system currently supports the Internet protocols, the Xerox Network Systems(tm) protocols, and some of the ISO OSI protocols” (http://manpages.ubuntu.com/manpages/natty/en/man4/net.4freebsd.html) TCP/IP is the most important set of protocols in Linux networking system. (Operating System Concepts, p.838-9)

Internally, networking in the Linux kernel is implemented by three layers of software:

The socket interface

All user applications are performed through the socket interface.

Protocol drivers

Data can reach at this layer through network devices or user applications. Using the already existing tag in a packet, it decides which device or which application will be used. It has the ability to rewrite, create, split etc packets.

Network-device drivers

This layer is responsible for the routing. The routing of packets takes place by using a set of pointers in memory (socket buffer), with tags as mentioned before, linking the device with the appropriate protocol from a hash table. Two tables are used for the implementation of routing.

The forwarding information base (FIB), which uses a specific destination address or by specifying destinations according to wildcards.
A cache of recent routing decisions

(Operating System Concepts, p.838-9)

Supported Protocols

The command

cat /etc/protocols

provides us with a full list of the supported Internet Protocols

Port usage is described in

“/etc/services” and can be listed using cat command

Overview of the available network applications

6.1. Web browsers
6.2. The mail system
- SMTP
- Smarthost – SMTP AUTH
6.3. Mail transport agent (MTA)
6.4. Mail user agent (MUA)
6.5. The remote mail retrieval and forward utility
- POP3-APOP-IMAP
- PO3-IMAP4-SDPS
- mail retriever (POP3) with regex filtering capability
- mail retriever (POP3) and DMA with filtering capability
6.6. Mail delivery agent (MDA) with filter
- DMA with filter
- DMA with Perl filter
- DMA with structured filtering language
6.7. POP3/IMAP4 server
6.8. The print server and utilities
- Line printer daemon
- Common UNIX Printing System
6.9. The remote access server and utility (SSH)
6.10. Other network application servers
- TELNET Server (telnet protocol)
- TELNET SSL support
- Unix file sharing (NFS)
- Windows file sharing (SMB)
- Appla/Mac file and printer sharing (AppleTalk) - (ATP)
- General file download (FTP)
- General web server (HTTP)
- General proxy server
- OpenSLP Server as LDAP server (SLP)
- IP address of client itself (DHCP)
- IP address for other hosts (DNS)
- VPN (PPTP-Cisco VPN-OpenVPN-IPSec PN) (https://wiki.ubuntu.com/VPN)
6.11. Diagnosis of the system daemons

(http://www.debian.org/doc/manuals/debian-reference/ch06.en.html)

Available Networking tools

In Ubuntu GNU/Linux, a user in order to use administrative tools, he must log in as root. The tools for normal user and administrators are the same, but if a simple user does not have root password, then certain tools cannot be accessed or some options are disabled.

Graphical tools

Available graphical network tools can be found in the system settings.
There is a whole category named “Internet and Networks” in “System settings”. This category includes the following sub-categories :

Bluetooth settings
Personal file sharing settings
Remote Desktop settings
Network proxy settings and
Network connections settings
Bluetooth settings

Using this tool, root can add bluetooth devices and share files. The GUI is simple and straightforward.

Personal file sharing

Using this tool, root can determine which folders can be shared over the network or via bluetooth, setting privileges and passwords.

Remote Desktop

Using this tool, we can determine if we wish to allow others to use the desktop. We can set passwords, for those we wish to allow, and we also determine the way we will be notified in the main panel, if someone from the list above, is currently logged in.
Network Proxy

Using this tool we can setup the settings to connect to proxy servers.

Network Connections
This tool allow root to create connections and to edit existing ones.

Adding to the tools above, as far networking is concerned, there are two more that must be mentioned.

“gconf-editor”

It is a configuration tool for the system in general (applications, graphic settings. nautilus etc), there are also preferences for networks. The important thing in this tool is that it is the only place where “WORKGROUP” can be setup for samba servers, which affects performance with MS Windows PCs.

“Network tools”
Under system settings in control centre, in the system category there is the “network tools” option. With this tool, users can benefit from the use of commands, normally found only through terminal, as listed below, through an easy GUI:

ping
netstat
traceroute
postscan
lookup
whois
finger

Firewall

Ubuntu Linux has a built-in firewall called netfilter. (Ubuntu pocket Guide and Reference, 2009). There is no default installed GUI for firewall and mainly it is used through terminal. The recommended GUI tool is the “firestarter” (Ubuntu pocket Guide and Reference, 2009) package which can be found through the control software or though terminal by typing
sudo apt-get install firestarter

Otherwise the command to configure the firewall through terminal is “ufw”.

Firestarter allows two policies. The “whitelist” policy and the “blacklist” one.

If white list policy is used, then all traffic is blocked unless otherwise determined by a rule. Using “blacklist” policy, every transmission in allowed unless otherwise determined by a rule. (Ubuntu pocket Guide and Reference, 2009)
TerminalUsing the terminal offers an advance user ful access to the options available for networking configuration in Ubuntu. Listed below are some of the basic commands for network configuration and information listing.

lspci -nn | grep -i net, network pci controller information
iwconfig, wireless connection overview
ifconfig, connection overview
ufw, configures the firewall
ip addr, reports on your system's network interfaces
More commands can be found here, http://manpages.ubuntu.com/manpages/natty/en/man5/interfaces.5.html

/proc/sys/net/ipv4 folder

In this folder in the virtual fs of /proc, an advance user can redetermine the TCP’s values, such as the use of acknowledgements, in order to improve performance. eg. if /proc/sys/net/ipv4/tcp_sack value is 1, then acknowledgement is enabled. (Pro Ubuntu server administration, 2009)

Security

Ubuntu security model is closely related to typical Unix security mechanisms (Operating system Concepts, p.840)

It can be classified in two groups:

Authentication, and
Access control
Authentication

Authentication is making sure that only authorised people can access the system, proving their identity. Ubuntu uses the pluggable authentication modules (PAM) system, which a mechamism on a shared library making it possible to be used by any system component. (Operating system Concepts, p.840-841)

Access control is providing a mechanism to check whether a user has the right to access a certain object and preventing access if required. In Ubuntu there are three categories of users.

User/owner
Group
The rest

A user is identified by the UID. We can also determine privileges for a group of user, who will be identify by the GID. Root is a special type of user. It is the equivalent of administrator and has full system access.

There are 3 types for permissions in file/folders.

Execute (x)
Write (w)
Read only (r)

The implementation of security scheme in Linux is directly linked, on the physical layer, with the file system. The file system must be able to store the privileges for users and groups in each file. The speciality of Ubuntu, in the implementation of security policy of linux, is that it allows normal users to “borrow” root privileges. Ubuntu encourages a simple user to log in with the minimum privileges required. However, when required, eg to access system tools, files or perform administrative actions in general, he does not have to log out and relogin. A pop up window shows up and the root password is required to proceed, otherwise no action is taken. In terminal, the $ symbol determine a simple user and # character determines that superuser is logged in.

Encyption

Ubuntu offers the ability to encypt the /home folder so that only the current user and the root will have access to the personal files. Additionally the ability to encrypt a file or folder is also available. PGP protocol is used to do so and the keys are 2024-bit long. (Ubuntu Pocket Guide and Reference, p126-128)

Keyring

Keyring a repository for passwords. It is there to help us with password control. Since a simple user may have several passwords for several objects (browsers, sites etc), keying is there to gather all of them. When an application needs to access the keyring to get the correct password, then a popup window appears to enter the default keyring password. It is a helpful attribute of the OS.

Security tools

Graphical

Security tools can be found in the Control center, under the “Other” and “System” category.

User and Groups tool, allow us to create users and groups, assigning them rights.

Passwords and Encryptions keys is where keying preferences are stored and encryption options can be found.

Terminal

Below are listed some basic commands to manipulate users and permissions through the terminal

- sudo adduser --home /home/john john

command creates a user with the name john and also creates the Home folder

- deluser --remove-all-files, command will delete the user john as well as all the file sand the Home folder

- sudo adduser john accounts, will add john to the accounts group

- sudo delgroup accounts, will remove accounts group
- sudo deluser john accounts, will remove john from the accounts group
- ls -la, will list all files and folder in a folder with permissions and ownerships

chmod {options} filename, command to change ownership and permissions in files and folders

Options	Definition
u	owner
g	group
o	other
a	all (same as ugo)
x	execute
w	write
r	read
+	add permission
-	remove permission
=	set permission

References

http://manpages.ubuntu.com/manpages/natty/en/man4/net.4freebsd.html, as found on 21^st June 2011
Operating System Concepts, 8^th Edition, Abraham Silberschatz, Peter B. Galvin, Greg Gange, 2010, Jonh Wiley and Sons
https://wiki.ubuntu.com/VPN, as found on 22^nd June 2011
(http://www.debian.org/doc/manuals/debian-reference/ch06.en.html, as found on June 23^rd 2011
Ubuntu Pocket Guide and Reference, Keir Thomas, 2009, MacFreda
Pro Ubuntu Server Administration, Sander van Vugt, Apress, 2009
https://help.ubuntu.com/community/UsingTheTerminal as found on 23^rd June 2011
https://help.ubuntu.com/6.06/ubuntu/desktopguide/C/linux-basics.html as found on June 23^rd 2011

Ubuntu 11.04 - File Systems (5/6)

2011-06-27T21:54:00.003+02:00

General overview of the file system (FS) implementation in Linux

Linux uses the UNIX's standard file-system model, where everything appear in the form of files for the user. However a file does not have to be stored as object on a disk, it can be a stream of data. (Operating System Concepts, p.828) The kernel handles all these types of files by hiding the implementation details of any type behind a layer of software, the virtual file system (VFS). (Operating System Concepts, p.828)

The VFS is designed with the object-oriented principles. It has two components. One the specifies what file-system objects are allowed to look like and a layer of software to manipulate the objects. (Operating System Concepts, p.829) . The VFS works with objects each one contains a pointer to a function table. (Operating System Concepts, p.469)

The main VFS objects are:
the inode object and the file object
Both are mechanisms to access files. Inode objects contain pointers to the disk blocks and the file object contains pointers to the actual file data.
the superblock object
Its main responsibility is to provide access to inodes. It is a connected set of files that form a self-contained file-system. The VFS keeps in pairs the file-system/inode numbers to identify each node and it used the superblock object to get the inode with that pair.
the dentry object
It represents the directory entries found in a path name.
(Operating System Concepts, p.469 / p.830)

File systems in Linux are either loaded during boot time, or manually.
File systems and UBUNTU 11.04

Ubuntu 11.04 support the Linux Logical Volume Manager, which allows us to increase the size of a partition online, while the system is running.

In addition, Ubuntu 11.04 allows us to use different devices as mount points for several “heavy/crucial” directories. These are:

/boot, the directory containing the basic instructions to load the OS
/ (root) directory
/home, the user(s) home directory, equivalent to “My documents” folder in MS Windows

A full list of the supported file systems in UBUNTU 11.04 can be found using the “cat” command in the virtual file system /proc as shown below.

cat /proc/filesystems

Table 1. Supported file systems in UBUNTU 11.04 according to Ubuntu natty manpages.

minix	is the file system used in the Minix operating system, the first to run under Linux. It has a number of shortcomings: a 64MB partition size limit, short filenames, a single timestamp, etc. It remains useful for floppies and RAM disks.
ext	an elaborate extension of the minix file system. It has been completely superseded by the second version of the extended file system (ext2) and has been removed from the kernel (in 2.1.21).
ext2	the high performance disk file system used by Linux for fixed disks as well as removable media. The second extended file system was designed as an extension of the extended file system (ext). ext2 offers the best performance (in terms of speed and CPU usage) of the file systems supported under Linux.
ext3	a journaling version of the ext2 file system. It is easy to switch back and forth between ext2 and ext3.
ext4	a set of upgrades to ext3 including substantial performance and reliability enhancements, plus large increases in volume, file, and directory size limits.
Reiserfs	a journaling file system, designed by Hans Reiser, that was integrated into Linux in kernel 2.4.1.
XFS	a journaling file system, developed by SGI, that was integrated into Linux in kernel 2.4.20.
JFS	a journaling file system, developed by IBM, that was integrated into Linux in kernel 2.4.24.
msdos	the file system used by DOS, Windows, and some OS/2 computers. msdos filenames can be no longer than 8 characters, followed by an optional period and 3 character extension.
umsdos	an extended DOS file system used by Linux. It adds capability for long filenames, UID/GID, POSIX permissions, and special files (devices, named pipes, etc.) under the DOS file system, without sacrificing compatibility with DOS.
vfat	an extended DOS file system used by Microsoft Windows95 and Windows NT. VFAT adds the capability to use long filenames under the MSDOS file system.
ntfs	replaces Microsoft Window's FAT file systems (VFAT, FAT32). It has reliability, performance, and space-utilization enhancements plus features like ACLs, journaling, encryption, and so on.
proc	a pseudo file system which is used as an interface to kernel data structures rather than reading and interpreting /dev/kmem. In particular, its files do not take disk space.
iso9660	a CD-ROM file system type conforming to the ISO 9660 standard.
High Sierra	Linux supports High Sierra, the precursor to the ISO 9660 standard for CD-ROM file systems. It is automatically recognized within the iso9660 file-system support under Linux.
Rock Ridge	Linux also supports the System Use Sharing Protocol records specified by the Rock Ridge Interchange Protocol. They are used to further describe the files in the iso9660 file system to a Unix host, and provide information such as long filenames, UID/GID, POSIX permissions, and devices. It is automatically recognized within the iso9660 file-system support under Linux.
hpfs	the High Performance Filesystem, used in OS/2. This file system is read-only under Linux due to the lack of available documentation.
sysv	an implementation of the SystemV/Coherent file system for Linux. It implements all of Xenix FS, SystemV/386 FS, and Coherent FS.
nfs	the network file system used to access disks located on remote computers.
smb	a network file system that supports the SMB protocol, used by Windows for Workgroups, Windows NT, and Lan Manager.
ncpfs	a network file system that supports the NCP protocol, used by Novell NetWare.

(http://manpages.ubuntu.com/manpages/natty/man5/filesystems.5.html)

However, support for additional file systems can be added by adding additional modules to kernel.

As it can be seen above, UBUNTU 11.04 supports several file systems. The result when using the cat /proc/filesystems – command, will include several virtual file systems connected to no devices, that are used from Ubuntu kernel for administrative actions, such as memory caching. e.g.

“ramfs, a very simple filesystem that exports Linux's disk caching mechanisms (the page cache and dentry cache) as a dynamically resizable RAM-based filesystem.

rootfs, a special instance of ramfs (or tmpfs, if that's enabled), which is

always present in 2.6 systems. You can't unmount rootfs for approximately the

same reason you can't kill the init process; rather than having special code

to check for and handle an empty list, it's smaller and simpler for the kernel

to just make sure certain lists can't become empty.”

(http://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt)

Adding to those above, the swap file system must be noted as well, even though it is a raw file system, used for memory caching. Swap is a very important parameter in Linux.

The ones available to use in a physical drive during a clean installation, using the default partitioning tool are:

ext2
ext3
ext4
reiserfs
xfs
jfs
btrfs
(http://www.raiden.net/node/576)

A very brief comparison of the File systems most commonly found in UBUNTU.

File System	Max File Size	Max Partition Size	Journaling	Notes
Fat16	2 GB	2 GB	No	Legacy
Fat32	4 GB	8 TB	No	Legacy
NTFS	2 TB	256 TB	Yes	For Windows Compatibility) NTFS-3g is installed by default in Ubuntu, allowing Read/Write support)
ext2	2 TB	32 TB	No	Legacy
ext3	2 TB	31 TB	Yes	Standard linux filesystem for many years. Best choice for super-standard installation.
ext4	16 TB	1EB	Yes	Modern iteration of ext3. Best choice for new installations where super-standard isn't necessary.
reiserfs	8 TB	16 TB	Yes	No longer well-maintained.
JFS	4 PB	32 PB	Yes (metadata)	Created by IBM - Not well maintained.
XFS	8 EB	8 EB	Yes (metadata)	Created by SGI. Best choice for a mix of stability and advanced journaling.
GB = Gigabyte (1024 MB) :: TB = Terabyte (1024 GB) :: PB = Petabyte (1024 TB) :: EB = Exabyte (1024 PB)

(https://help.ubuntu.com/community/LinuxFilesystemsExplained)

However, as mentioned before, not all are available during installation to be used in a physical drive, since not all support the read/write and file permissions required for Ubuntu to work properly. NTFS cannot be used to install Linux at all, and FAT16/32 are not recommended to install Linux, since FAT does not have any of the permissions of a true Unix FS. They are used for compatibility reasons with Microsoft Windows.
From the file systems described above, the most commonly used for Ubuntu installation, are:

ext2/ext3
ext4
and
XFS

Ext2 as shown in table 1, is not a journaling FS. It is still commonly used however, to mount in boot time the /boot directory. Since it is not a journaling system, there are less read/write operations and since the /boot directory is only used in boot time to load the kernel, ext2 can result in faster OS boot.

Ext3 is the successor of ext2. However, it is a journaling system. It has been the default file system for a lot of years and since the LTS versions starting from 6.04 are only now starting to expire, many still use this file system for their / (root) and /home directories. Ubuntu supports the conversion from ext3 to ext4, the new default journaling default file system. The possible limit for a sub directory is 32000. It uses an indirect block mapping scheme to keep track of each block used for the blocks corresponding to the data of a file.

(http://kernelnewbies.org/Ext4)

Ext4 is the evolution of ext3 and it is a journaling file system, but it is an attribute that can be disabled. Ext4 breaks sub directory limit and allows an unlimited number of sub directories. Moreover, it uses “extents” to improve performance for large files. “For example, a 100 MB file can be allocated into a single extent of that size, instead of needing to create the indirect mapping for 25600 blocks (4 KB per block). Huge files are split in several extents. Extents improve the performance and also help to reduce the fragmentation, since an extent encourages continuous layouts on the disk. When Ext3 needs to write new data to the disk, there's a block allocator that decides which free blocks will be used to write the data. But the Ext3 block allocator only allocates one block (4KB) at a time. That means that if the system needs to write the 100 MB data mentioned in the previous point, it will need to call the block allocator 25600 times (and it was just 100 MB!). Not only this is inefficient, it doesn't allow the block allocator to optimize the allocation policy because it doesn't knows how many total data is being allocated, it only knows about a single block. Ext4 uses a "multiblock allocator" (mballoc) which allocates many blocks in a single call, instead of a single block per call, avoiding a lot of overhead. This improves the performance, and it's particularly useful with delayed allocation and extents.” (http://kernelnewbies.org/Ext4)

In addition, delayed allocation procedure is used in ext4 improving performance and reducing fragmentation and it is supported for online de fragmentation.

XFS is a journaled 64-bit file system. It is organized in balanced B+Trees and uses Extends to store data. XFS is fully supported by all Ubuntu-Versions. It is not recommended to be used to mount the /boot directory, due to GRUB support. It I sused to mount the root directory and the /home one.

Advantages

Part of official Kernel
Online optimizing (defragmenting) filesystem using "xfs_fsr" (part of xfsdump-package)
Ability to grow an existing filesystem using "xfs_growfs" (part of xfsprogs-package)
Own dump- and restore-utilities to keep fs-specific info
Fast seeking through directories
Efficient file storage, low fragmentation
Relatively low memory usage
Journal (internal or external)
Balance between speed and safety
Delayed allocation and improved sparse file handling

Disadvantages

Slower than non-journaled fs (like ext2)
Due to balance of speed and safety, there is the possibility to lose data on aprupt power loss (outdated journal metadata), but not filesystem consistence
Currently there is no ability to shrink an xfs filesystem. However, there are some people working on this issue
GRUB-Support is in an early stage
The journal is currently not 32-bit / 64-bit portable. Before mounting a xfs prevously used on 32-bit linux in 64-bit linux (or different arch) the journal has to be emptied using xfs_repair.

(https://wiki.ubuntu.com/XFS and http://oss.sgi.com/projects/xfs)

Tools for the ext2,ext3,ext4 and XFS file systems in Ubuntu.

The available tools for file system information and modifications, are separated ion this paper in two parts. The one where the only the graphical tools are explained, and the second one where the commands in terminal are briefly explained.

GUI tools

A fast overview of the available file systems can be found in the system monitor, in the file systems tab.

Further actions and information for the available storage devices can be found in the Disk Utility in the Control Center.

Disk Utility provide us with an overview of the storage devices currently connected in the system. It also provides us with an easy graphical interface to:

get hardware information
mount devices
unmount devices
check the filesystem in the device
edit partitions
delete partitions
format volumes

Graphical tools for fie systems, as for any other administrative actions, are poorer comparing to the equivalents in terminal. They addressed to normal users and available actions are limited, e.g define block size, cannot be processed thought them.

Terminal Commands

They are addressed to advanced users and the can modify any available option for a file system. This is why they must be used as superuser, or by borrowing su privileges. Additional information, about parameters and syntax, are provided in the links below.

blkid, used to find the Universally Unique Identifier (UUID) for a device, which is prefered from Ubuntu over the deveice path, eg /dev/sda, since it can change during boot.

mkfs, used to create a file system

(http://manpages.ubuntu.com/manpages/natty/man8/mkfs.8.html)

fsck, used to check and repair file system

(http://manpages.ubuntu.com/manpages/natty/man8/fsck.8.html)

fdisk

(http://manpages.ubuntu.com/manpages/natty/man8/fdisk.8.html)

df, used to report file system usage

(http://manpages.ubuntu.com/manpages/natty/man1/df.1.html)

badblocks, used to check a device for bad blocks

(http://manpages.ubuntu.com/manpages/natty/man8/badblocks.8.html)

umount and mount, used to unmount and mount devices

(https://help.ubuntu.com/community/Mount)

References

Operating System Concepts, 8^th Edition, Abraham Silberschatz, Peter B. Galvin, Greg Gange, 2010, Jonh Wiley and Sons

http://manpages.ubuntu.com/manpages/natty/man5/filesystems.5.html, as found on June 14^th 2011

http://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt as found on June 15^th 2011.

http://www.raiden.net/node/576, as found on June 15^th 2011

Pro Ubuntu Server Administration, Sander van Vugt, 2009, Apress

https://help.ubuntu.com/community/LinuxFilesystemsExplained, as found on June 11^th 2011

http://kernelnewbies.org/Ext4, as found on June 14th 2011

https://wiki.ubuntu.com/XFS , as found on June 15^th 2011

http://oss.sgi.com/projects/xfs , as found on June 15^th 2011

http://manpages.ubuntu.com/manpages/natty/man8/mkfs.8.html, as found on June 15^th 2011

http://manpages.ubuntu.com/manpages/natty/man8/fsck.8.html, as found on June 15^th 2011

http://manpages.ubuntu.com/manpages/natty/man8/fdisk.8.html, as found on June 15^th 2011

http://manpages.ubuntu.com/manpages/natty/man1/df.1.html, as found on June 15^th 2011

http://manpages.ubuntu.com/manpages/natty/man8/badblocks.8.html, as found on June 15^th 2011

https://help.ubuntu.com/community/Mount, as found on June 15^th 2011

Ubuntu 11.04 - Device management (4/6)

2011-06-27T21:41:00.006+02:00

Introduction

UBUNTU has support for all types of hardware built into the kernel. (Ubuntu pocket guide and reference, p.19). To the extent possible, all device drivers appear as normal files. Devices are separated in three classes:

block devices

Here belong all the devices that allow random access to completely independent, ficed-size blocks of data, such as HDDs, floppy disks, CD-ROMs, flash memory. They are accessed randomly.

character devices

In this category belong most other devices, most of which are pointing devices such as keyboards, mice etc. They are accessed serially.

network devices

Here we find the network devices that users cannot communicate directly. Their transfer data by opening connection with the kernel's network subsystem. (Operating System Concepts, p.835)

A fundamental difference in Ubuntu 11.04, based on their kernel version, 2.6.38 is the use of deadline I/O shceduler in order address the starvation issue. The deadline for read request is 0,5 second and for write requests is 5 seconds. It maintains a sorted queue, a read and a write queue. (Operating System Concepts, p.836)

Four are the key folders for devices in UBUNTU, show in the table below:

Folder	Description
/dev	Virtual files representing hardware devices
/media	Contains folder where storage devices can be mounted
/proc	Virtual folder containing file representing stats and settings
/etc	System (global) configuration files

(Ubuntu pocket guide and reference, p.53)

UBUNTU does not have a “Computer” view like windows. Storage devices are mounted in folders inside the /media folder. Moreover, for HDDs Master Boot record is important, because bootloader (GRUB 2.0) cannot be after the 1024th cyclinder, since it may cause system to halt at boot.

Through kernel updates the drivers are updated and more are included. However, it is possible to compile drivers in kernel. However, it is addressed to advanced users only. The administration of the devices using graphical tools is limited, compared to the tools available in terminal. The utilities can be found in the System settings → “Hardware” and “System” section.

Additionally the gnome-device-manager package can be installed through the synaptic manager or through terminal,

$sudo apt-get install gnome-device-manager

in order to graphically list all the available connected devices. “Connected” devices does not mean “operational” too.

System administrator must count in terminal to list and compare values from the /proc folder and /dev for the problematic device, so as to verify the amount of interrupts, mapping, memory allocation addresses, pages etc to figure out where the problem is located.

References

Ubuntu pocket guide and reference, Keir Thomas, MacFreda publishing, 2009
Operating System Concepts, 8^th Edition, Abraham Silberschatz, Peter B. Galvin, Greg Gange, 2010, Jonh Wiley and Sons

Ubuntu 11.04 - Memory management (3/6)

2011-06-27T21:34:00.004+02:00

Overview – Allocation

UBUNTU as a GNU/Linux distribution is designed for different architectures. Due to this characteristic, UBUNTU separates physical memory into three different zones:

zone_DMA
zone_NORMAL
zone_HIGHMEM

Zone_DMA is the zone that includes the first 16MB of physical memory. It is consisted of the DMA-able pages (Direct Access Memory). This happens because certain hardware, Industry Standard Architecture (ISA) devices, can only access the lower 16MB of physical memory. (Operating System Concepts, p.820)

Zone_NORMAL is the zone that includes the normally addressable pages up to the first 896 MB of physical memory. The remaining physical memory (>896 MB) is allocated from zone_HIGHMEM. It is referred as high memory and includes the dynamically mapped pages.

“The primary physical-memory manager in the Linux kernel is the page allocator” (Operating System Concepts, p.820). There is one allocator for each zone, responsible for allocating, freeing all physical pages for each zone, and capable of allocating ranges of physically contiguous pages on request, but even though some allocations may require a page from a particular zone, a normal allocation can come form zone_DMA or zone_NORMAL. However, it is preferable to use a page from the particular zone, mostly to save the DMA space, but if needed and memory is low, Kernel can use whichever zone is available and suitable. (Robert Love, Linux Kernel Development Second Edition). It is important to note that many kernel operations can only take place using zone_NORMAL so it is the most critical zone about performance. (http://www.kernel.org/doc/gorman/html/understand/understand005.html)

UBUNTU 11.04 uses 4096-byte page size (http://manpages.ubuntu.com/manpages/natty/man1/pagesize.1.html) that utilises available physical memory, reducing the percentage of internal fragmentation. However, as of 2.6.38 kernel, the one UBUNTU 11.04 uses (https://wiki.ubuntu.com/NattyNarwhal/ReleaseNotes), the transparent huge page (THP) feature is added, which takes advantage of the page sizes that contemporary processors can handle simultaneously. For example, the Intel® Pentium® Processor E5700 has a 2M Cache. (http://ark.intel.com/Product.aspx?id=42801) This means that a 2M page requires one entry in the processor's translation lookaside buffer. Instead, with page size 4096, it needs 512 entries. “THP simply substitutes huge pages for the regular kind whenever they are available and it appear to make sense. The result is that Linux systems take advantage of this hardware feature and run faster on almost all workloads” (http://www.linux.com/news/featured-blogs/171-jonathan-corbet/421304-2638-making-things-just-work).

There are two ways that memory is allocated in UBUNTU. Statically, during boot time for drivers that deserve contiguous areas in memory, and dynamically by the page allocator.

There are two mechanisms for allocating Kernel memory:

The buddy service technique, and
The slab allocation

kmalloc is the normal method of allocating memory in the kernel. (http://manpages.ubuntu.com/manpages/dapper/en/man9/__kmalloc.9.html)

Using a “buddy service” type technique of allocation, the kmalloc() service is used, when the size of the request is not known in advance and it can be a few bytes, to allocate entire pages on demand and then splitting them into smaller pieces. The allocation by kmalloc() service is permanent until they are freed explicitly. The kernel maintains a list with the pages used by the kmalloc() service. (Operating System Concepts, p.820-821)

The slab technique is used “for allocating memory for Kernel data structures and is made up for one or more physically contiguous pages”. (Operating System Concepts, p.821) The slab allocation algorithm uses caches to store kernel objects. A cache consists of one or more slabs. Each kernel structure has its own cache. “A slab may be in one of three possible states:

Full. All objects in the slab are marked as used.
Empty. All objects in the slab are marked as free.
Partial. The slab consists of both used and free objects.”

(Operating System Concepts, p.823) Priority is first given to partial slabs. If one does not exist, then a free object is assigned from a free one.

Except of these two main memory managers, there are two more subsystems that do their own management of physical-memory: the page cache and the virtual memory system.

“The page cache is the kernel's main cache for block devices and memory-mapped files and is the main mechanism though which I/O to these devices is performed.” (Operating System Concepts, p.823). The page cache can also cahce networked data. Block devices provide the main interface for disks in a system. UBUNTU 11.04 use a deadline I/O scheduler which provide deadline for read requests of 0.5 second and 5 seconds deadline for write requests. The deadline scheduler maintains a sorted queue, a read and a write queue. (Operating System Concepts, p.836)

UBUNTU do very high use of virtual memory. mmap() service is responsible for mapping files or devices into memory. “Linux reserves for its own internal use a constant, architecture-dependent region of the virtual memory address space of every process. The page-table entries that map to these kernel pages are marked as protected” (Operating System Concepts, p.826) Two regions are used, one where the core of kernel, along with all pages allocated by the normal page allocator reside, and another which is reserved for no specific purpose. The vmalloc() service is used to allocate an arbitrary number of physical pages of memory that may not be contuguous into a single region of virtually contiguous kernel memory. (Operating System Concepts, p.826)

The mmap() service has flag arguments that transform the mapping to shared with other processes, or private creating a private copy-on0write mapping. (http://manpages.ubuntu.com/manpages/lucid/man2/mmap.2.html) The virtual mapping procedure is directly connected to the way the process was created. If it is created through the exec() servicem then a new mapping address is provided. If it is created through fork() then the parent's pages are directly copied to the child's, resulting that the parent and child share physical pages in memory. UBUNTU 11.04 use the least frequently used (LFU) as page out policy (Operating System Concepts, p.825). It is through the age variable of a page that it is chosen which page to swap.

UBUNTU extends the physical memory of the system using a so called “swap” partition in the HDD. It is a raw type file-system where pages are swapped when faulting. Due to the type of the file system the I/O are faster than normal file systems.

It also important to note that to the available architecture of the OS, the physical limitation of a 32-bit version of UBUNTU would normally be limited to a ~4GB of usable RAM. However, the Physical Address Extension kernel option is available for UBUNTU 9.10 and above. This technology allows 32-bit OS to use up to 64 GB of RAM (https://help.ubuntu.com/community/EnablingPAE). Even though the additional RAM is slower, it is still faster than a HDD. In general in these cases a 64-bit OS is prefered, however, so far drivers and applications are still better supported in a 32-bit OS.

Memory Management OS Utilities

The only graphical tool about memory management that comes with UBUNTU installation is the System Monitor. One can reach it from System Settings → System Monitor. As it can be seen, the only available information one can get is the amount and percentage of memory and swap currently being used. A graphical representation over time is created helping us to visualize the constant chagnes in the values. However, these information are poor for an advanced user and even less for the system root user. More detailed utilities are available through the command-line, most of which are only available for the root group.

At this point it must be noted, as it has been mentioned in assignment 1, there is a /proc folder in the file system. It is a pseudofile system which is used as an interface to kernel data structures. It is commonly mounted at /proc.Most of it is read-only, but some files allow kernel variables to be chagned.” (http://manpages.ubuntu.com/manpages/natty/man5/proc.5.html)

It can seen that the “folder” includes several sub folders with numerical names and some files.

The folders with numerical names, correspond to PIDs. Each folder contains several files that carry information about that process. It must be made clear that when we open one such file, root privileges are required, so a $sudo should be used. In addition, what we see is an instance of the constant memory changes that take place. Since we cannot recognise a task by the PID, $top should be used.

PID : Process ID
USER: User the process belongs to
PR: Priority
NI: Nice value
VIRT: Virtual Image
RES: Resident size
SHR:Shared memory size
%CPU: CPU usage
%MEM: Memory usage
TIME+:CPU Time
COMMAND: command name/line

(Sander van Vugt, 2009)

$top command can be customized about the available shown options. In addition, it provides updates about swap, cache and memory usage.

Combining the $top command with the proc virtual file system, we can get all the available information about a process. First we get an overview with top of the current situation, we identify the terget process, get the PID and then we move to the proc file at /proc for further investigation. http://manpages.ubuntu.com/manpages/natty/man5/proc.5.html provides a full overview of the information we can obtain through the use of proc file system and an text editor, such as vi.

Additional commands and files containing memory information are shown below:

free -m	Limited memory information
/proc/meminfo	Full memory information
/proc/interrupts	System interrups served per CPU
uptime	System uptime
vmstat -s	Virtual memory information
sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"	Drops caches
pagesize	Page size currently used
uname -r	Kernel version (PAE included)
sudo swapoff -a	Turning swap off
sudo swapon -a	Turning swap on

References

Operating System Concepts, 8^th Edition, Abraham Silberschatz, Peter B. Galvin, Greg Gange, 2010, Jonh Wiley and Sons
Linux Kernel Development Second Edition, Robert Love, Sams Publishing, 2005 as found at http://book.chinaunix.net/special/ebook/Linux_Kernel_Development/0672327201/ch11lev1sec2.html on June 1^st 2011.
https://wiki.ubuntu.com/NattyNarwhal/ReleaseNotes as found on May 2^nd 2011.
http://ark.intel.com/Product.aspx?id=42801, as found on June 1^st 2011.
http://www.kernel.org/doc/gorman/html/understand/understand005.html found on June 1^st 2011
http://www.linux.com/news/featured-blogs/171-jonathan-corbet/421304-2638-making-things-just-work found on MAY 31^st 2011
http://manpages.ubuntu.com/manpages/natty/man1/pagesize.1.html as found on May 31^st 2011.
http://manpages.ubuntu.com/manpages/dapper/en/man9/__kmalloc.9.html found on June 1^st 2011.
http://manpages.ubuntu.com/manpages/lucid/man2/mmap.2.html as found on June 2^nd 2011.
https://help.ubuntu.com/community/EnablingPAE found on June 3^rd 2011.
http://manpages.ubuntu.com/manpages/natty/man5/proc.5.html ofund on June 2^nd 2011.
Pro Ubuntu Server Administration, Sander van Vugt, 2009, Apress

Ubuntu 11.04 - Process management (2/6)

2011-06-27T21:23:00.013+02:00

General Information

This is the result of personal research through community and official documentation about GNU/Linux and Ubuntu. References used are provided also. It will be presented in 6 posts as listed below:

Introduction

UBUNUT Linux11.04 is based on 2.6.38 Linux kernel (https://wiki.ubuntu.com/NattyNarwhal/ReleaseNotes). As far as the process management is concerned, UBUNTU Linux11.04 has preemptive kernel and supports Symmetric Multiprocessing (SMP).

There are two spaces that we identify in process circle. The kernel space and the user space. The process management takes place in the kernel space.

The scope of this paper is to present how the process management takes place in UBUNTU 11.04 GNU/Linux in both User Interfaces available, and how to admiister the system using the standard tools that come with the OS. Since the OS is developed under the GNU GPL (http://www.gnu.org/licenses/gpl.html), there are a lot more tools available , but they are outside the scope of this paper.

Before going through the presentation of the tools, a small introductional information is provided I norder to better ubderstand abbreviations and procedures to be discussed.

Basic aknowledgements

Components of the UBUNTU/Linux system

As every Linux system, UBUNTU is composed of three main parts of code.

The kernel

It is the heart of the OS. “It is responsible for mainting all the important abstractions of the operating system, including such things as virtual memory and processes” (Operating Systems Concepts, p. 807)

System libraries

They include sets of fucntions that do not need kernel priviledges in order applications to interact with the kernel.

System utilies

They include programs that perform individual management tasks, eg initialize, configure aspects of the system. In addition there are the so called deamons, that “may run permanently, handling such tasks as responding to incoming network connections, accepting logon requests from terminal and updating log files” (Operating Systems Concepts, p. 807).

Processes and threads.

Processes and threads are not that different for linux. Both fork() and exec() are used to create processes. Threads are created by the clone() system call. Clone() creates a child process allowing it to use shared information with the parent process by sets of flags that are passed to it. This how a thead is described in other Operating Systems. Exec() is the system call used to run new program whereas fork() is the system call to create a new process. In general it can be said that fork() is similar to clone() when no flags are passed to the new process. (Operating Systems Concepts, p. 815)

Process properties

Process poperties fall into three groups:

Process identity

Process ID (PID)

It is the unique identifier of the process. It is used to specify the process to the OS. It does not change until the process termination.

Credentials

Also known as User ID (UID) and Group ID (GID). It determines the rights and privilegdes of the process to access system resources.

Personality

It can modify the semantics of certain system calls. It is used primarly by emulation libraries in order to request the combatible system call with certain varieties of UNIX.

Process environment

It “is inherited from its aprent process and is composed of two null-terminated vectors; the argument vector and the environment vector (Operating System Concepts, p.813). The environment vector is a list of “NAME=VALUE” paris that associate named enviroment variables with arbitary textual values. eg. The LANG variable determines the language that is to be used to display system messages.

Process Context

It represents the state of the process and it constantly changes. It inculdes the parts listed in the table 1 below.

	Description
1	Scheduling Context
2	Accoutning
3	File table
4	File-sysstem context
5	Signal-handler table
6	Virtual memory context
Table 1. Parts of the Process context (Operating System Concepts, p.814)

Scheduling

UBUNTU Linux 11.04 have two seperate process-scheduling algorithms. “One is time-sharing for fair, preemptive scheduling among multiple processes; the other is designed fo real-time tasks.” (Operating System Concepts, p.815) “Real-time algorithms are also supported, for special time-critical applications that need precise control over the way in which runnable processes are selected for execution.” (http://manpages.ubuntu.com/manpages/natty/man2/sched_setscheduler.2.html)

The time sharing algorithms are three.

```
SCHED_OTHER
```
```
SCHED_BATCH
```
```
SCHED_IDLE 
```

For the time sharing algorithms, the standard Round-Robin method is used. It is a First-Comes-First-Served (FCFS) method, but also assingning a time quanta for each process, creating a loop ending with the init() system call, which is the parent of all processes. (http://manpages.ubuntu.com/manpages/natty/man8/init.8.html)

Two seperate priority ranges are used. A real time range from 0-99 and a nice value ranging from 100 to 140. In this scheme the numerically lower processes have higher priority. When a task is runnable, it enters a runqueue, mainted by the kernel. The runqueue includes two arrays. The active list, which contains the processes to be executed, and the expired list, which contains the tasks that have expired. If the time quanta does not allow the task to be completed, then it is considerd as expired and it goes back to the expired array list of the runqueue. When the acitve array of the active list = 0 then the expired list becomes the active one and vice versa. The runqueue is processor indepented.

The nice priority value is determined by how much time the task has been sleeping. It is dynamically changed and the scheduler infavors interactive tasks, which are tasks that have longer sleep times. The nice priority value is recalculated when the task enters the expired list.

The real time algorithms are the FCFS and the Round-Robin, which was described above, with the addition of a priority to the one of its scheduling class. When two processes have the same class, the one waiting in queue the lognest has higher priority. For processesof different class the one with the hight class has the higher priority.

Kernel synchronization

Tasks may try to access the same internal data structures. Since UBUNTU Linux are preemptive, if another task with higher priority is ready to use CPU time, the running task will be preempted even if running in kernel.

In order to lock the kernel, UBUNTU provides spinlocks and semaphores.For Symmetric Multiprocessing machines, spinslocks are the fundamental mechanism. This cannot happen in single processor machines since there is only one CPU time and locking for short periods, which is what spinslocks do, would have high impact to the system performance. Instead of spinlocks, in single processor machines, the preemptive ability is enabled and disabled instead.

The most important aspect in preemptive mechanism is what happens with interrups. IN UBUNTU the interrupts are prioritized and seperated into two parts. The top-half one and the bottom-half one. An interrupt form the first class can interrupt one form the second class, but cannot interrupt another of the same class.

A simplified example overviewing what has been discussed so far is presented in figure 1 below.

Administration utilities found in the GUI (UNITY)

The graphical utility can be found in “system settings” → “system monitor”. The second tab called “resources” shows the running processes in queue.

The monitor informs us about:The process name

The status of the proces
The CPU % usage
The Nice priority value
The PID
The memory usage

By right clicking in a entry, a menu appears, allowing us:

to stop a proces
to continue a stopped proces
to end a process (asking to end)
to kill a proces (forcing to terminate)
to change priority
the memory maps (VM start and end values)
and to show the linked files

Administration utilities used from the command-line

There are several commands that be used to monitor and manage the running processes of the system. Since the terminal is addressed to advanced users everything cannot be presented in this paper.

In order to interact with a task in terminal, we need to know either the PID or the name of the task. The table below summarises the most useful commands for process management through the terminal. The description of each command as well as the available arguments can be viewed by typing (command) --help, or man (the name of the command)

	Name	Description
1	ps	With the appropriate arguments it can show all the running processes, the process runnning from a user
2	pstree	Shows the runnning processes in a tree scheme
3	kill	Kill a process by PID
4	killall	Kill a process by name
5	nice	Sets the priority of new processe
6	renice	Changes the priority of a currently running process
7	skill	Reports process status

However, I believe that the most usefull command for the process management throught the terminal is the command top. It is a real time non graphical “System Monitor”.

Althought the number of the taks running is bigger and because it changes dynamically, the best proposed use of it is to use the results of the top command exporting them in home folder in a text file, like this

top >> ~/top.txt, which will overwrite a text file in the home folder with the name top.

It will provide with all the necessary information to manipulate the running tasks in the system. The “h” key shows the help menu na the “q” is used to exit the top command.

Conclusion

Ubuntu Linux 11.04 is cooperative or preemptive OS. There is practically no difference between processes and threads. The process management takes place in the kernel space and it works in SMP machines . Finally it provides us with tools to manipulate the processes running in both the graphical and the terminal interface.

References

https://wiki.ubuntu.com/NattyNarwhal/ReleaseNotes as found on May 2^nd 2011.
http://www.gnu.org/licenses/gpl.html as found on May 2^nd 2011
Operating System Concepts, 8^th edition, International stucent version, Abraham Silberschatz, Peter B. Galvin, Greg Gange, Wiley 2010
http://manpages.ubuntu.com/manpages/natty/man2/sched_setscheduler.2.html as found on May 10^th 2011.

Ubuntu 11.04 - General Overview (1/6)

2011-06-27T21:09:00.009+02:00

Introduction

This is the result of personal research through community and official documentation about GNU/Linux and Ubuntu. References used are provided also. It will be presented in 6 posts as listed below:

General Information.

UBUNTU Linux 11.04 (with code name natty narwhal) is a GNU/LINUX stable distribution provided by Canonical Ltd. and released on April 2011. It supports Intel x86 (IBM-compatible PC), AMD64 (Hammer) and PowerPC (Apple iBook and Powerbook, G4 and G5) architectures. It is available in both 32-bit and 64-bit versions. It is based on 2.6.38.2. kernel release (https://wiki.ubuntu.com/NattyNarwhal/ReleaseNotes) and comes with UNITY graphical user interface. (www.help.ubuntu.com)

UNITY has come to “unite” into one, all the previous UBUNTU distribution editions. The user interface for server and desktop editions (the notebook and desktop editions are now one) is now the same. This provide one solid platform for users, making it easier for them to navigate across different editions. Except of the UNITY GUI, command line (CLI) user interface, also know as terminal, is also available.

Hardware Requirements

The recommended by Canonical Ltd. minimum hardware requirements are shown below:

1 GHz x86 processor
1GB of system memory (RAM)
15GB of hard-drive space (although this can be split onto 2 drives, a 5Gb / and a 10Gb /home fairly easily)
Graphics card and monitor capable of 1024 by 768
Either a CD/DVD Drive or a USB port (or both)
Internet access is helpful

(www.help.ubuntu.com)Canonical provides a list of certified hardware for the version at http://www.ubuntu.com/certification/release/11.04.

Features against competition - Target group.

UBUNTU 11.04 Desktop edition, is a free Operating System committed to the principles of open source software development. It is and will always be free of charge, including updates. The release will be supported for 18 months. (https://help.ubuntu.com/11.04/installation-guide/i386/what-is-ubuntu.html) But this is not all. Through “UBUNTU software Center”, it provides a database for easy and fast access to free, compatible, open source applications that fit almost all needs.

Canonical is the enterprise that sponsors and provides the official and professional development of UBUNTU. However, ubuntuforums.com is the official forum where the UBUNTU community contributes to troubleshooting.

In UBUNTU 11.04 Desktop edition, “support for all types of hardware is built into the central kernel program.” (Ubuntu Pocket Guide and Reference, p. 19) This means that no additional drivers are needed for hardware to be functional (even though proprietary drivers may be required for additional effects). This this the reason of the expression, “UBUNTU simply works” (Ubuntu Pocket Guide and Reference, p. 19).

Moreover, it provides almost a ‘computer virus’ free environment, because of the way user accounts and groups are used. During installation, two login accounts are created. “A standard user and the root user. “The root user is a special account gifted with the ability to do anything, such as delete system files, or install software.” (Ubuntu Pocket Guide and Reference, p. 54).

“Althought the rot account is there in the backround, the user is discouraged from directly logging in as root. Instead, the user “borrows” root powers to administer the system when necessary. Usually this is done by simply entering your login password when prompted. When working at the command-line any command needing administrative powers must be preceded by sudo” (Ubuntu Pocket Guide and Reference, p. 54).

“Although the root user account has a back seat in UBUNTU, its influence is felt everywhere. Most operating system files are “owned” by the root user, and have eprmissions so that only root can edit them.” (Ubuntu Pocket Guide and Reference, p. 54). “This simple mechanism of protecting operating system files though root user ownership is how Linux enforces security and system protection.” (Ubuntu Pocket Guide and Reference, p. 55).

According to Steven J. Vaughan-Nichols, Canonical aims to “win over casual Windows users” with 11.04. (www.zdnet.com). UNITY provides a stable standard graphical interface with cosmetic additions that aims to provide more visual space and reduce the time required to access applications. (https://help.ubuntu.com/11.04/ubuntu-help/unity-introduction.html). Although, a casual Microsoft Windows User will need to spend a lot of time in order to understand the way UBUNTU works and how important the terminal is, in order to fully use the OS capabilities.

The concluding great asset of UBUNTU 11.04 Desktop Edition, is the opportunity it offers to its users to customize the OS literally according to one’s needs. From windows colors to cleaning cache memory, everything is available though the terminal. Summarizing, UBUNTU 11.04 Desktop edition:1. is Open Source based GNU/LINUX OS2. is free of charge3. is supported both professionally and by a wide community4. it simply works (real plug and play hardware functionality)5. provides free/open source applications access6. virus free environment7. is fully customizable 8. requires considerable time to adapt and requires fundamental knowledge of terminal commands to overcome issues that will arise

Available UIs in depth

As mentioned in the beginning, there are two UIs available by default in Natty Narwhal. The Graphical UI is UNITY and the other is the command line, or terminal.

UNITY

By default there is a panel in the top of the screen. This panel acts as a title bar for applications as well as menu bar for the application where mouseover. This way two lines are merged into one, providing more space in screen.

In the right side of the top panel, there are several icons for network, sound , user and system settings shortcuts.

A second sliding launcher is on the left where minimized and pinned programs can by found. The windows start button alternative is the UBUNTU trademark in the top left. If clicked, it provides panel with pinned and mostly used applications as well as a search bar for software installed, either by name or by extension.

Nautilus is used as file browsing program. The window, which is handled as in other OS, uses the top panel as title and menu bar. It is also consisted of a tool bar, an address bar, a side bar and the main browsing space containing folders and files. Tabs are available by Ctrl+T.

Paths and Files

“There are no drive letters under UBUNTU. Instead, the root of the file system, know as C:\ under windows, is identified simply by a forward slash (/). Additionally, whereas Windows uses a backslash (\) to indicate the root of the filesystem, and to separate folders in a path listing, UBUNTU uses a forward slash.” (UBUNTU Pocket Guide and References, p. 49)

NO drive letters means no problem for CD/DVD drives. Instead of letters, when inserting a disk in the drive, an image is “mounted” and is shown under the /media/ folder.

File and folder names can contain spaces and any letter, number and symbol, with the exception of the forward slash, but they are case sensitive. Document.doc is different than document.doc file.

File extensions

“Generally speaking, the trend in Linux is not to use file extensions for systems files.” (UBUNTU Pocket Guide and References, p. 53). However, in personal folders, extensions are as important as in Windows. Executable programs under UBUNTU dont' have a .exe file extension. Instead, they executable attribute indicates that they are programs and not ordinary data files. This attibute can be checked in the the right click menu of the file under the label “Allow executing file as program”.

Hidden files and folders

Hidden files in UBUNTU are determined by a dot in front of the filename. “.document.doc” is hidden form the folder list, unless all files of the folder are requested. The tide (~) symbol at the end of the file name makes the file hidden in file listing provided by nautilus file manager.

Important filesystem locations

Location	Details
/bin	Essential software, typically needed to get the system running
/boot	Files related to the boot menu/loader
/dev	Virtual files representing hardware devices
/etc	System (global) configuration files
/home	User's personal folders
/lib	Support (library) files required by software
/media	Contains subfolders where storage devices can be mounted
/proc	Virtual folder containing files representing stats and settings
/root	Personal folder of the root user
/sbin	Essential software for system maintenance, usually used only by the root user
/tmp	Temporary files/folders
/usr	Essentially, subfolders containing most software used on the system, including system libraries and documentation
/var	Data that is vital to the running of the system and that is constantly being updated

Table 1. Key locations in the UBUNTU filesystem. (UBUNTU Pocket Guide and References, p.53)

UBUNTU software center

UBUNTU Software Center provides a fast and easy way to search for open source programs UBUNTU compatible. One can provide the search terms and the results will appear on screen.

Synaptic package manager

Synaptic is a package manager providing list of available and installed packages. It also provides information about available updates and the repository configuration information. Repositories can be considered as libraries of available software.

Gedit

Gedit is the notepad equivalent. It is a plain text graphical editor, that is heavily used in UBUNTU. As mentioned before, system files, in general, do not use extensions. They are plain text files that can be viewed and modified using the appropriate program. Gedit is the one for the graphical environment.

System Settings

Control Center provides a graphical overview of the system settings. They are categorized in groups as seen below:

Personal. Settings to personalize the appearance of windows, desktop, personal user information and accounts, preferred programs and shortcuts.
Internet and Network. Settings about bluetooth, network connections, proxies, remote access and file sharing.
Hardware. Settings about hardware information, drivers, video and sound configuration, peripherals, and power management.
System. Settings about networking tools and utilities, system monitor, packages information and system logging.
Other. Settings about Cloud settings, keyboard layouts, passwords and encryption.

Terminal UI – Non graphical environment

Command line is a powerful tool in UBUNTU 11.04. The same file and folder rules about paths and names described above apply here too.

General view

When thte terminal appears, it starts with something like this:

paladin@paladin-Ubuntu:

paladin is the user currently logged and used. paladin-Ubuntu is the computer name that the user is logged into.

The dollar symbol indicates that the user logged is an ordinary user. A root user is indicated by the hash symbol

Using terminal to navigate – Basic commands

Arguments

Some commands work on their own. Most need to be told what files or folders to work with. They are called arguments.

Command options

Most commands also have options. Options follow the command with a preceded hyphen.

Useful everyday commands and symbols

Command	Description
ls	List files and folders. Example: ls -l. Shows all files in folder, including hidden ones.
cd	Change folder. Example: 'cd ~/Documents' will move to Documents folder in the /home/”current user” folder
cp	Copy file or folder. Example: cp file.doc Desktop/, will copy file.doc from current path to /home/”current user”/desktop
mv	Move files or folders. If no destination path is provided, renames current file/folder
rm	Deletes files
nano and vi	Non graphical editors, Gedit equievelant
man	View the manual for a specified command
locate	Finds a specified file. It used a database which can be updated manually using ”sudo updatedb”
rmdir	Deletes folder
mkdir	Creates folder
wget	Powerful download program
apt-get	(Advanced Packaging Tool)
>	Redirects the output of a command to a specified location/file
\|	Pipe. Uses the output of one command as input for another

Since ternimal is actuals written commands, the administration offered in the graphical UI cannot been fully explained here. Moreover, the configuration files are different for each hardware device that is to be modified. In most cases the command that is to be used to edit a .conf file look look like this,

sudo vi /etc/00_grub.d, where /etc/00_grub.d will be the file to be edited.

Basic Administration Commands

Command	Description
ps	Shows the processes running and their pid
chown	Changes the owner of a file / folder
chgrp	Changes the owner group of a file
chmod	Changes file and directory permissions
mount	Mounts storage device from specified path in /media folder
unmount	Unmounts specified device
du	Shows a summary of free space in a partition
fsck	Finds and repairs filesystem corruption
alsamixer	Shows the alsamixer sound properties in terminal
--help	Shows available help information about current command
at	Runs a command at periodical intervals

Alternative GUIs

UBUNTU 11.04 Desktop edition comes with UNITY as default graphical UI. However, there are alternatives such as KDE, GNOME, LxDE and Xfce. Although they are based in the kernel, these GUIs require different tools and utilities for administration, even thought the same basic rules apply. They vary from more customizable interface to lighter ones, so they result in different versions and they are outside the scope of this paper. They can be found as seen below.

KDE	http://www.kde.org/	KUBUNTU
LxDE	http://lxde.org/	LUBUNTU
Xfce	http://www.xfce.org/	XUBUNTU

References

1. help.ubuntu.com as found at https://help.ubuntu.com/11.04/installation-guide/i386/what-is-ubuntu.html on May 2nd 2011.

2. https://wiki.ubuntu.com/NattyNarwhal/ReleaseNotes as found on May 2nd 2011.

3. help.ubuntu.com as found at https://help.ubuntu.com/community/Installation/SystemRequirements on May 2nd 2011.

4. http://www.ubuntu.com/certification/release/11.04 as found on May 2nd 2011.

5. www.zdnet.com as found at http://www.zdnet.com/blog/open-source/ubuntu-linux-1104s-target-audience-casual-windows-users/8723 on May 2nd 2011.

6. https://help.ubuntu.com/11.04/installation-guide/i386/what-is-ubuntu.html as found on May 2nd 2011.

7. https://help.ubuntu.com/11.04/ubuntu-help/unity-introduction.html as found on May 2nd 2011.

8. Ubuntu Pocket Guide and Reference, Keir Thomas, MacFreda Publishing, 2009

9. UBUNTU 11.04 Desktop edition desktop preview pic as found at https://help.ubuntu.com/11.04/ubuntu-help/unity-introduction.html on May 2^nd 2011.

How to lock a domument in MS Word 2003

2011-06-10T23:00:00.002+02:00

This is a small walk through guide how to lock a document using MS Word 2003 (a version with Microsft Word 2007 can be found here). One might say why lock and not convert to .pdf.

The main difference is that is easier and it does not require any other programs to manipulate .pdf files. Locking is helpful when we wish to control who is allowed to alter the data. In most cases it is used with forms and macros.

Microsoft Word 2003 version (go to Microsoft Word 2007 version)

Locking the document.

Let's suppose we wish to lock the document so that other will only be able to use forms in it.

Step 1

Step 2

A popup window will appear. Select the “security” tab. Write a password in the section shown in the image below.

Step 3

Then click on the “Protect Document” button as seen below.

Step 4

A frame in the right of the document will appear.

DONE. Now only a person who knows the password above can alter the document.

Give it a try.

(go to the Microsoft Word 2007 version)

How to lock a document in MS Word 2007

2011-06-10T22:50:00.005+02:00

This is a small walk through guide how to lock a document using MS Word 2007 (a version with Microsft Word 2003 can be found here). One might say why lock and not convert to .pdf.

The main difference is that is easier and it does not require any other programs to manipulate .pdf files. Locking is helpful when we wish to control who is allowed to alter the data. In most cases it is used with forms and macros.

Microsoft Word 2007 version (go to Microsoft Word 2003 version)

Part 1: Activate the developer toolbar.

It is deactivated by default. Folow the steps shown below to activate it.

Step 1

Step 2

Step 3

Part 1 is done. The developer toolbar should appear on the right of the toolbars, right after “view”, like this:

It is inside there where the locking options are found. There more options which you encouraged to check.

Part 2. Locking the document.

Let's suppose we wish to lock the document so that other will only be able to use forms in it.

Step 1

Step 2

A frame will appear on the right. Check the option “Allow only this type of editing in the document” and select filling in forms from the drop down menu. Then click on “Yes, Start Enforcing Protection”

Step 3

Step 4

A window will popup to enter a password.

DONE. Now only a person who knows the password above can alter the document.

Give it a try.

(go to the Microsoft Word 2003 version)

Ubuntuone and Ubuntuone music

2011-06-10T00:34:00.003+02:00

Hi there Ubuntu users.

Lately I've decided to use Ubuntu 11.04 as primary workplace in a new HDD. During the process of backing-up staff from the previous "experimental" installation, I have found very helpful the "Ubuntuone" option.

Using the cloud option of Ubuntuone, with 2 GB for free, the whole process has been made a lot easier, since I had nothing to back-up. Everything was already up in the air. It is a bit slow, but for free it is more than fine.

I was convinced and decided to buy more space for future use and try the music option to share music with my android phone. There is a Ubuntuone music app in the market to do so. Since I had a basic plan, I could only download the files in the phone, by using the username and password that I already had for my Ubuntuone account.

I though, why not, so I bought the mobile package for a month to stream music in the android phone, to give it a try. At first I was dissapointed, since the error

"Cannot parse 'none' as integer"

kept popping up. There were no files to play and I couldn't create a playlist to try and load some.

I gave it a look and the solution is simple. The problem was the mp3 files I was trying to share didn't have ID3 tags. So I used the 'EAYTAG' app from the Ubuntu Software Center to enter tags for title, artist and album. After that everything went fine.

I suggest you give it a try.