Ray´s Guide to Linux Command Line

archiving & Compressing

noreply@blogger.com (Andrée ray) — Thu, 23 Jun 2011 14:06:00 +0000

tar short for Tape Archive commonly referred to as tarball is both a file format and a program used to handle such files. The format was developed in the early days of Unix and made standard POSIX.1-1998 and later POSIX.1-2001.

Its easy to compress a whole directory in Unix/Linux´sBSD. Its useful when you are backing-up, transferring files over the Internet.

To make a tarball you need to follow the syntax and,

tar -zcvf archive_name directory_name

z compress a archive using gzip program
c create archive
v verbose - displays progress
f archive file name

To de-compress a file follow this syntax,

tar -zxvf archive_name directory_name

where

x Extract files

-----------------------------------
Read on

The basics of file compression in Unix
Why small files may be larger than large files. Why one file may be better than two, and how you can get more space on your disk by compressing your files using Tape Archiove (tar, tarball).

If you list files on your Unix system using ls -l you get the file size, however not the amount of space used on the disk. To se the about used by the disk you set a flag to -s, so ls -ls. This will display a initial number representing blocks used by the file. A block is a unit of 512 bytes, so 4 blocks would be 2048 bytes

The reason a filsystem uses blocks is that if it would be using the actual size would creata a overhead that would become a load on the system. Also keep in mind that you need some information about the file, like name and other information. Also a file would become heavily fragmentet (parts of the file spread across the filesystem) leading to slow performance.

To handle the problem a compromise was reached in disk organization. A convenient number of bytes was selected as the minimum amount that could be allocated to a file, the allocation unit. If a file don´t use the allocated unit, this would be reserved for future expansion.

Early Unix system used 512 bytes, however this was expanded as memory increased to 1024 bytes on most Unix systems (larger on some), but many utilities (like ls) continues to report in 512 bytes-blocks even though the actual size increased.

Black holes
With this in mind its easy to understand that a 3 byte file will occupy 512 bytes in the momory, 1-block on the disk. If the file is expanded over 512 bytes it will occupy another block though now using 1024 bytes on your disk.

This makes for a lot of space on your system. If you have a file that is 1201 bytes therefor occupy 3 blocks on a system that uses the 1024 byte-block 99.7 percent of the space allocated is unused, and 41.4 percent is wasted. Multiply this by the number of files on your system and you could imagean the vast black holes of disk space.

Remember that this kind of waste only occurs on very small files, so the larger files are more efficient in allocation space on your disk. If you allow that your system is probably working well your allocation is a good compromise between allocation and speed of disk access.

To establish the allocation unit of your system. You can read the man-page for the ls command. Establish the block size used by the -s flag.

Compression
Pack, compress, and gzip compression utilities work well on large files, but perform poorly on small files. In the sample below, compress is applied on the file and the result is displayed. The compress utility recognizes that it can´t do any good on the file and leaves it alone. If it would to a do a job it would append the .z to a file when it compresses it. To reverse the process use the uncompress command.

The smallest blocks is a pair of two (or the default allocation of units on your system).

Compressing files with tar
If you have a directory of small files that are little used, but need to remain on the system. One way of doing it is to combine the files and compress them. The obvious candidate for combining files are tar (tape archive) utility program.

Tar informs you that it has appended a file and how many tape blocks it uses. tar reprots in 512 bytes blocks.

Having a look at the files produced you might get a shock! The files may now take even more blocks than prior. Fortunately, tar fills those empty spaces in the blocks with garbage (takes form of hex zeros or NULLs.

Proceeding the next logical steps, is to compress those tarballs (tar). This results in a .tar.Z file that is smaller than the original file.

Just reverse the process starting with uncompress, and then use tar with the flag -xvf

It might be easier to just follow the tar command as described at the beginning of this guide. However if you would like to use archiving and compressing of your choice: That makes this process more powerful - and operating from the command line, is that not why we prefer shells infront its desktop alternative? Now we can perform a tailored tarball for best performance of a particular file.

FTP

noreply@blogger.com (Andrée ray) — Sun, 19 Jun 2011 21:05:00 +0000

The user interface to the Internet Standard File Transfer Protocol.

Its a program commonly used for copying files to and from systems on a network. These systems may be on the same site or anywhere in the world.

For the purpose of this tutorial, your machine is the local, where you are issuing the commands. The remote machine is the one holding the FTP server, the machine that performs the desired actions on its system and transfers the file to you.

Getting Started

Connecting to a remote machine, type

ftp machine_name.domain

Where machine_name is the full path to the server ex, domain.something.something
Or you could enter a IP-address, type

ftp 192.168.1.0

Sometimes you may which to access a system that you don´t have an account. You may then use the anonymous login_name (anonymous must be active on the system). Normally you are asked for your username and passwords. If you should type in the wrong data you can use the USER to relogin

USER

I would recommend that you compress directories when you want to transfer multiple files, it makes transfer easy and safe. Please follow the guide on tape archive (tar, tarball)

Finally here are the common commands with FTP

Common commands

`?`	to request `help` or information about the FTP commands
`ascii`	to set the mode of file transfer to ASCII (this is the default and transmits seven bits per character)
`binary`	to set the mode of file transfer to binary (the binary mode transmits all eight bits per byte and thus provides less chance of a transmission error and must be used to transmit files other than ASCII files)
`bye`	to exit the FTP environment (same as `quit`)
`cd`	to change directory on the remote machine
`close`	to terminate a connection with another computer
	`close brubeck`	closes the current FTP connection with `brubeck`, but still leaves you within the FTP environment.
`delete`	to delete (remove) a file in the current remote directory (same as `rm` in UNIX)
`get`	to copy one file from the remote machine to the local machine
	`get ABC DEF`	copies file `ABC` in the current remote directory to (or on top of) a file named `DEF` in your current local directory.
	`get ABC`	copies file `ABC` in the current remote directory to (or on top of) a file with the same name, `ABC`, in your current local directory.
`help`	to request a list of all available FTP commands
`lcd`	to change directory on your local machine (same as UNIX `cd`)
`ls`	to list the names of the files in the current remote directory
`mkdir`	to make a new directory within the current remote directory
`mget`	to copy multiple files from the remote machine to the local machine; you are prompted for a `y`/`n` answer before transferring each file
	`mget *`	copies all the files in the current remote directory to your current local directory, using the same filenames. Notice the use of the wild card character, `*`.
`mput`	to copy multiple files from the local machine to the remote machine; you are prompted for a `y`/`n` answer before transferring each file
`open`	to open a connection with another computer
	`open brubeck`	opens a new FTP connection with `brubeck`; you must enter a username and password for a `brubeck` account (unless it is to be an anonymous connection).
`put`	to copy one file from the local machine to the remote machine
`pwd`	to find out the pathname of the current directory on the remote machine
`quit`	to exit the FTP environment (same as `bye`)
`rmdir`	to to remove (delete) a directory in the current remote directory

Screeen

noreply@blogger.com (Andrée ray) — Sat, 18 Jun 2011 02:02:00 +0000

Screen is a GNU project program that will allow your CLI applications to run in the background without x running. Using the terminal to run many programs, it will fast get messy having all the terminal windows in view. This is easy to take care of by installing screen.

When screen is called, it creates a single window with a shell in it and then gets out of the way so that you can use the program as you normally would. At any time you can create a new window with other programs in them, kill the current, view scrollback, switch between windows, etc. As i stated before you programs still run when you are not viewing them. Everything you type is sent to the one keystroke that is used to initiate a command to the window manager. By default, each command begins with a control-a (abbriviated C-a from now on)

You will probably have screen installed on your system already if not, in Ubuntu do

$ sudo apt-get install screen

Then copy screenrc to your home directory

$ cp /etc/screenrc ~/.screenrc

If you would like to have screen always run with rtorrent (What I am using it for right now) add the following to your screenrc file

# Always start rtorrent with screen
screen -t rtorrent rtorrent

To start screen and rtorrent, simply run screen from a terminal . Control-a followed by d will detach screen and running screen -r will open screen again

Tip - Its common to use screen with ssh to use rtorrent on a remote machine. I will not get in to that just yet. Stay tuned for a article about ssh, x11 and the use of rtorrent on a remote machine.

rtorrent

noreply@blogger.com (Andrée ray) — Fri, 17 Jun 2011 23:54:00 +0000

rtorrent is a Bittorrent client, with emphasis on speed (thats what we love about the command line:). Even tho its faster than a GUI torrent client it dose not lack the features of one, its a full-blown client that suit your every need.

You install it in Ubuntu by issuing the command

$ sudo apt-get install rtorrent

To start your saved torrents issue the command

$ rtorrent Torrent/atorrent.torrent

Your torrent, both finished and unfinished will be saved in your home directory by default. To set another directory by using the -d flag

$ rtorrent -D Downloads/ Torrents/atorrent.torrent

I would however download the default config file and make the changes you want before you open rtorrent for the first time. You can get it here or use the wget command like the example below. Save it in your home directory as .rtorrent.rc. The reason I would never go without this is because I like to have a folder where I download torrents to and then have the client auto check for .torrent files in that directory

$ wget http://libtorrent.rakshasa.no/browser/trunk/rtorrent/doc/rtorrent.rc?format=raw

Open it with nano and configure rtorrents to suite your need, you get directions in the .rc file. Don´t forget to save it as rtorrent.rc!

$ nano /home/your_account/rtorrent.rc?format=raw

To make a directory that the client watches and adds, edit the rc file like bellow.

# Watch a directory for new torrents, and stop those that have been
# deleted.
#schedule = watch_directory,5,5,load_start=./watch/*.torrent
#schedule = untied_directory,5,5,stop_untied=
schedule = watch_directory,5,5,load_start=/home/[user]/watch/*.torrent
schedule = untied_directory,5,5,stop_untied=
schedule = tied_directory,5,5,start_tied=

Note - you need to mkdir the watch directory yourself.

Adding and removing torrents
Backspace -- Add torrent using an URL or file path. Use tab to view directory content and do auto-complete. Also, wildcards can be used. For example: ~/torrent/*
Return/Enter -- Same as backspace, except the torrent remains inactive. (Use ^s to activate)
CTRL-O -- Set new download directory for selected torrent. Only works if torrent has not yet been activated.
CTRL-S -- Start download. Runs hash first unless already done.
CTRL-D -- Stop an active download, or remove a stopped download.
CTRL-R -- Initiate hash check of torrent. Without starting to download/upload.

Throttling
a/s/d -- Increase the upload throttle by 1/5/50 KB.
z/x/c -- Decrease the upload throttle by 1/5/50 KB.
A/S/D -- Increase the download throttle by 1/5/50 KB.
Z/X/C -- Decrease the download throttle by 1/5/50 KB.

Navigating
Main View Keys:
right -- Switch to Download View.
CTRL-R -- Initiate hash check of torrent.
+/- -- Change priority of torrent.
l -- View log. Exit by pressing the space-bar.
1 -- Show all downloads.
2 -- Show all downloads, ordered by name.
3 -- Show started downloads.
4 -- Show stopped downloads.
5 -- Show complete downloads.
6 -- Show incomplete downloads.
7 -- Show hashing downloads.
8 -- Show seeding downloads.

Download View Keys
right -- Switch to selected view.
left -- Switch to view selection or back to main view.
1/2 -- Adjust max uploads.
3/4 -- Adjust min peers.
5/6 -- Adjust max peers.
p -- Display peer list.
o -- Display torrent info.
i -- Display file list.
u -- Display tracker list.
t/T -- Initiate tracker request. Use capital T to force the request, ignoring the "min interval" set by the tracker.

Peer list View Keys
left -- Switch to view selection.
right -- Show file details.
space -- Change the file priority; applies recursively when done on a directory.
* -- Change the priority of all files.
/ -- Collapse directories. While collapsed, press right to expand the selected directory.

Tracker list View Keys
left -- Switch to view selection.
* -- Enable/disable tracker.
space -- Rotate trackers in a group.

rtorrent wiki here