e-maknyus

Lecture #21 - Asynchronous Transfer Mode (ATM)

emaknyus@gmail.com (tamvan & verani) — Fri, 16 Sep 2011 14:22:00 -0700

ATM is a major traffic carrier used by telecommunication companies. ATM is also the core networking technology used to aggregate and connect ADSL tail circuits to ISPs. ATM provides for virtual circuits and can guarantee Quality of Service (QoS).
Some of the key concepts involved in ATM include:

The basic unit of transmission is the cell. A cell is 53 bytes in length - 5 bytes of header and 48 bytes of payload. These small, fixed-format cells can be switched very quickly.
ATM networks are based on connection-oriented cell relay - cells are multiplexed onto higher bitrate channels, but there is no timing relationship between individual cells within a particular connection. Cells may be dropped but are never delivered out of order.
The telecommunications industry uses ATM as a basis for providing converged services, allowing for the seamless integration of voice (telephony), audio, video-on-demand, video conferencing, video broadcast and data, on the one network.
ATM is complex. Even the telcos admit it has many unsolved problems.

ATM Background

Designed in early 1990s, when both bandwidth and networking equipment was relatively expensive, and voice and data convergence was a hot topic in the industry.
A technology evolved more from the telecommunications industry than the computer/data industry.
Designed for high-performance mixed network traffic (voice, data, video, and future media applications).
Designed for bursty traffic.
Designed to replace all existing analog and digital communication backbone systems.
A relatively expensive networking technology, mainly used among tier 1 and tier 2 telcos.
Gigabit Ethernet (a much cheaper technology) has replaced ATM on many networks. ATM is Less popular now.
Gigabit Ethernet is a natural upgrade path for Fast Ethernet.
ATM offers QoS, but QoS issues can simply be overcome by providing a bigger pipe?

"Arguably, ATM is unnecessary if bandwidth is cheap and plentiful, and there is no congestion in the network."
"Experience has also shown that over-provisioning bandwidth does not resolve all QoS issues. Hence Gigabit Ethernet is not expected to replace ATM at this time."

ATM Features

Cell based switching technique
Fixed-size cells (53-Bytes) - small cells
Asynchronous time-division multiplexing (Asyn TDM)
Connection oriented
Virtual Circuit (VPI VCI)
QoS as part of the technology

ATM Speed

Originally designed to work over Synchronous Optical Networking (SONET)
Expresses in OC (Optical Carrier) levels.
OC-3: 155 Mbps (3 x 51.84 Mbps)
OC-12: 622 Mbps
OC-192: upto 10Gbps bit rate
In early 1990s, 155 Mbps was considered very fast.

ATM - Circuit Switching or Packet Switching

ATM is a packet switching technology where fixed size small packets are switched efficiently.
ATM uses virtual circuits - provides a reliable dedicated path during the whole session of transmission, offering the advantage of circuit switching, i.e. reliable connection, and can guarantee a specific QoS.
ATDM - how ATM does multiplexing.

Advantages

Fast switching of small cells
Better utilisation than circuit switching
No fragmentation problem
Reduce jitter (variance in delays)
Easier to allocate bandwidths and delays to VCs
Easier to implement priorities to traffic
Enables traffic management and congestion control
Suitable for time-critical information such as voice or video

Disadvantages

Extra overhead as IP datagrams are segmented into small cells
Expensive technology

ATM Virtual Channels

ATM is connection-oriented and switching is based on the idea of a virtual channel or VC. A virtual channel is set up between a pair of hosts connected to a network of switches:

The existence of a VC is based on entries in routing tables in each of the switches (which are really routers), set up at VC establishment time. Each switch compares its table entries to a field in the cell header to decide where the cell is to be sent.

Permanent and Switched Virtual Circuits

Permanent Virtual Circuit (PVC): dedicated virtual circuit established for continuous use, and no need for repeated call setup and clearing. Usually preconfigured by a service provider. PVC is similar to a leased line.
Switched Virtual Circuit (SVC): setup on a per-call basis and disconnected when the call is terminated. SVC is dynamically established on demand and is torn down when transmission is finished. SVC is similar to a normal PSTN phone line.

ATM - connection-oriented protocol

Connection-oriented establishes a session, then delivers a stream of data in the same order as it was sent.
Can be a circuit switched connection, or a virtual circuit switched connection in a packet switched network like ATM.
Connection-oriented protocols are not necessarily reliable protocols. ATM is an example of a connection-oriented, unreliable protocol.

ATM Virtual Channel Setup

All cells relating to a connection (call, circuit) follow the same path through the network. During call setup, a separate Protocol Connection Identifier is assigned to the connection on each link through the network.
The numeric value of the PCI identifier has only local significance at each link, and changes from one link to another as cells relating to a particular call (or connection) pass through the network.
Associated with each incoming link/port at a switch is a routing table that contains, for each incoming PCI, the corresponding outgoing link/port and the new PCI to be used.
The PCI is made up of two components: the Virtual Path Identifier (VPI) and the Virtual Channel Identifier (VCI). Routing can be on the basis of the value of either, or both, of these subfields.
The VPI can be used where multiple connections are set up between the same pair of hosts. The VCI field is used where connections are not between the same pair of hosts.

ATM VPI/VCI

Transmission Path (TP): Physical connection, usually fibers.
Virtual Path (VP): Connection between two ATM switches. VPI identifies the cell's next VP to pass through.
VC: Virtual Channel/Circuit (VC): VC must first established before any data transfer. VCI identifies the cell's next VC inside the VP.

ATM and ADSL

DSL is an access method for an ATM network, allowing a DSL tail to connect to many ISPs through a wholesale ATM network (e.g. Telstra wholesale).
ADSL uses ATM as an intermediate layer between the physical layer and a layer 2 protocol like PPP.

High Level View of an ADSL / ATM Network

ADSL access tails connecting to DSLAMs (Digital Subscriber Line Access Multiplexer).
DSLAMs connecting to a Tier 1 telco's ATM backbone network.
Tier 1 telco aggregates all the ADSL tails and distributes the traffic to the smaller providers.

ATM Switching

ATM routes a cell using both the VPI and the VCI.

ATM Switches

ATM switches are typically built of a network of Banyan switches.
Each element of the switch can either pass a cell straight through or cross the path over. This decision is made directly by the corresponding bit in the cell's destination address (VPI/VCI bits). The first bit controls the first switch, the second bit controls the second switch etc.

All ATM switches have two common goals:

Switch all cells with as low a discard rate as possible, preferably nil.
Never (ever) re-order the cells on a virtual connection.

A problem occurs when two cells arrive at a given time, both destined for the same output. The general solution is:

Pre-sort the cells before feeding them into the Banyan switch (this avoids switching contention) and
Provide a queue on each outgoing line (this avoids "output" contention).

ATM Architecture

ATM Switches: reads and updates the cell header information and switches the information cell towards its destination.
ATM Endpoints: contains the ATM network interface adaptor to enable data entering or leaving the ATM network to interface to the external world. Examples of these end points include workstations, LAN switches, video codecs and many more items.
NNI: Network-to-network interface (between two switches, inside an ATM network).
UNI: User-to-network interface (between a user access device and a switch, on the edge of an ATM network).

ATM Cell Format

GFC: Generic Flow Control, only present in cells passing over the User-Network Interface (UNI). On Network-Network Interfaces (NNI), this is part of the VPI field. GFC provides local function, such as identifying multiple stations that share a single ATM interface. Typically not used and is set to default (binary 0000).
VPI, VCI: See previous slide.
PTI: Payload Type Identifier. MSB (1st bit) indicates data type - (0 = user data, 1 = control data). 2nd bit indicates congestion (0 = no congestion, 1 = congestion). 3rd bit indicates if the cell is the last of the frame (0 = No, 1 = Yes).
CLP: Cell loss priority. 1 = discard the cell under extreme congestions.
HEC: Header Error Correction, or checksum.

The HEC field

Calculates checksum only on the first 4 bytes of the header. HEC can detect and correct a single bit error in these bytes.
The HEC field is an 8-bit CRC checksum of the first 4 bytes (octets) of the header. In fact, it consists of the remainder after the 32 bits have been divided by the magic number 100000111, to which is added the constant 01010101, to make the checksum more resilient where headers contain mostly zeroes.
The checksum covers the header only for speed of computation, and because for many telco applications incorrect data is not catastrophic...
The HEC algorithm is effective where most errors are single bit, which is usually the case for optical fibre.
The HEC comes in handy when multiple cells are supplied to a switch port in strict succession. The protocol provides no way to determine cell boundaries (eg. no FLAG or SOH fields). As bits arrive, the software maintains a 40-bit shift register. If the last 8 bits form a valid HEC checksum of the first 32, then this may be a valid cell boundary. If not, load one more bit and try again until a match is found.

ATM Traffic Control and QoS

ATM supports a set of QoS parameters and classes for all ATM services.
ATM minimises network complexity while maximising utilisation.

CBR - Constant bit rate

a Peak Cell Rate (PCR) is specified, which is constant.
for connections that require a static amount of bandwidth that is continuously available during the connection time.
for applications such as telephone, video conferencing, interactive Audio, TV.

VBR - Variable bit rate

nrt-VBR (non-real time) and rt-VBR (real time).
an average cell rate is specified, which can peak at a certain level for a maximum interval before being problematic.
for bursty traffic, VoIP, interactive compressed voice/video.

ABR - Available bit rate

a minimum guaranteed rate is specified.
provides rate-based flow control.
depending on the state of congestion in the network, rate controlled by source.
used for data traffic such as file transfer and email.

UBR Unspecified bit rate:

traffic is allocated to all remaining transmission capacity.
for non-real time applications. Delay and delay variations can be tolerated.
most TCP/IP applications.

ATM Reference Model

ATM Layer

Defines cell layout
Defines ATM header
Establishment and release VC
Switching
Multiplexing
Congestion control

ATM Adaptation Layers (AAL)

ATM was designed to replace ALL existing digital and analog telecommunication systems.
Therefore ATM must be able to accept ANY type of services from upper layers.
AAL is responsible for fragmenting them into small fixed size ATM cells.
And to reassembles cells.
5 adaptation layers have been defined.

AAL1 to AAL5

AAL1 is used for CBR services such as video and voice from upper layer. Require synchronisation of source and destination.
AAL2 is used for VBR services such as compressed video, voice, and fax. Bursty in nature.
AAL3/4 support connection-oriented and connectionless data services, replaced by AAL5.
AAL5 is for data. Also known as the Simple and Efficient Adaptation Layer (SEAL).

More on ATM AAL

ATM is all things to all people

ATM was designed to replace all existing digital and analog telecommunication systems. In order to do this, the underlying lightweight packet stream had to be adapted to provide a number of different types of service. This was achieved by placing an adaption layer on top of the lightweight packet stream. The adaption layer hides the underlying packet stream and presents an appropriate interface to the higher layer(s). Various adaption layers have been defined, as follows:

AAL1 defines a Constant Bit Rate (CBR) stream between the ends of the ATM virtual circuit (VC). AAL1 emulates a traditional phone call. AAL1 steals one byte from the payload field to extend the cell header.
AAL2 as for AAL1, except the bit rate can vary. Nevertheless, the ends of the VC still have a fixed timing relationship. This is appropriate for transfer of compressed video. AAL2 steals 3 bytes from the payload field for administration.
AAL3/4 originally specified as two distinct adaption layers, this is now a complex specification for a connectionless data service over ATM. Higher layer data packets (up to 64 Kbytes) have 4 bytes each of header and trailer information added, then are diced into ATM cells from which 4 bytes of payload are stolen for admin purposes.
AAL5 (designed to replace AAL3/4) accepts data packets of up to 64 Kbytes, adds 8 trailer bytes, then dices this up to fit in ATM cells (without any further headers).

ATM AAL5 and Encapsulation of Internet Data

To send Internet Protocol (IP) datagrams over an ATM service, using AAL5, the data undergoes a complicated encapsulation process:

Since AAL5 has no Type field, the IP Datagram is first encapsulated in an 802.2 LLC frame, using connectionless SNAP.
The 802.2 LLC frame is placed in an AAL5 Convergence sub-layer frame. It is padded to make the AAL5 frame a multiple of 48 bytes long.
The AAL5 Convergence Layer frame is chopped into 48 byte packets by the AAL5 Segmentation and Reassembly Sub-layer.
The resulting pieces become the contents of ATM cells.

Here's a closer look at the AAL5 Convergence sub-layer (CS) Protocol Data Unit (PDU):

UU: User-to-User field is not used in AAL5 but is available for higher layer protocols for eg. sequence information.
RFU: Reserved for future use.
Length: The actual length of the data. The payload field is padded to make the total CS PDU a multiple of 48 bytes.
CRC: A standard 32-bit CRC checksum over the entire CS PDU.

LANs over ATM

Where ATM is to be used to form a LAN, fundamental differences between ATM and IP becomes troublesome:

IP is connectionless and has provision for broadcasting built in. Everything is carried by IP.
ATM is connection oriented (broadcasting is difficult).
ATM and IP addressing are incompatible.

A number of different approaches have been proposed to solve these problems:

CLIP - Classical IP over ATM
LANE - LAN Emulation
NHRP - Next Hop Resolution Protocol
MPOA - Multi-Protocol over ATM

CLIP - Classical IP over ATM

This is an IETF specification whereby IP treats ATM as another subnetwork to which routers and hosts are attached. In the CLIP model, multiple Logical IP subnetworks (LIS) are overlaid on top of the ATM network. Machines within a LIS can communicate directly using ATM Virtual Channel Connections (VCCs). Machines in different LISs can communicate only through routers, even though direct connections using VCCs may be possible.

LANE - LAN Emulation

This is an ATM Forum (now the Broadband Forum) competitor to CLIP. It is essentially similar, except that where CLIP handles only IP, LANE handles other network level protocols as well.
ATM LAN Emulation (LANE) looks like the following:

A virtual circuit can be set up between two hosts connected to the ATM network. However, as in conventional LANs (eg. 802.3/Ethernet), communication is based on first knowing the MAC-level address of the destination.

LES: The LAN Emulation Server supplies network layer to MAC (in this case, ATM) address mappings on request.
BUS: ATM does not support broadcast. The "Broadcast and Unknown Server" accepts higher layer broadcast messages and sends them to all hosts participating in the LANE, one at a time.

NHRP - Next Hop Resolution Protocol

In CLIP LANs, machines in different LISs can communicate only through routers, even though direct connections using VCCs may be possible. LANE appears to be similar.
NHRP uses "Next Hop" clients to initiate connections to "Next Hop" servers. Together, these act to discover an ATM destination address corresponding to an IP address. The originating client then establishes an ATM connection directly to the destination host. Intermediate routers act as both NHRP servers and NHRP clients during call set-up.
NHRP and CLIP compared

MPOA - Multi-Protocol over ATM

MPOA was designed by the ATM Forum to provide an internetworking service over ATM. Basically, MPOA integrates LANE and NHRP. MPOA uses LANE to establish layer 2 connections (ie. within the same LIS) and NHRP to establish layer 3 connections (to another LIS).
Short lived connections between a host in one LIS and another host in a different LIS are passed router to router, as for CLIP. ATM virtual circuits are established for longer lived connections.
It is rumoured that this constant setting up and tearing down of ATM virtual circuits stresses the ATM switches. As a result, MPOA does not scale well in a system with many hosts.

IP Switching

A variation on Classical IP over ATM originally introduced by a company called "Ipsilon" (and now various others), who have announced a combination IP router and ATM switch which combines the best of both approaches.
This technique relies on the fact that where Classical IP over ATM is in use, the first cell of an encapsulated datagram will contain (in addition to the 802.2 SNAP headers) the entire IP and TCP (or UDP) headers, presuming minimum length headers.
This allows the ATM switch to make a routing decision (on the basis of the IP destination address) and a stream decision (on the basis of the TCP header) on the basis of the contents of the first cell.
If the ATM switch is also a router (ie. connected to more than one LIS), it can immediately begin sending the cells of a datagram to its destination, instead of waiting for the entire datagram to be accumulated, then forwarded. Thus, datagrams are effectively forwarded through the device at switch speeds, instead of the (usually) slower router speed.

Multi-Protocol Label Switching (MPLS)

The big problem with all the previous approaches is the duplicated network infrastructures (IP and ATM) with their incompatible addressing models and routing techniques.
MPLS routers are intended to integrate layer 2 switching functions with layer 3 addressing. When a layer 3 packet enters an MPLS network, the Label Edge Router (LER) prepends a MPLS label onto the incoming packet, before forwarding it to the next hop Label Switch Router (LSR). The MPLS label is based on the destination IP address and identifies the virtual circuit to be used between adjacent LSRs. When delivered by ATM, the label becomes equivalent to the ATM Virtual Circuit Identifier. The established path, unlike ATM though, is unidirectional.
Like ATM, MPLS is connection-oriented and a path needs to be established through the MPLS network before communication can occur. This is achieved using a Label Distribution Protocol (LDP). Quality of Service (QoS) can also be achieved by using an LDP that supports traffic engineering. Examples are the Constraint-based Label Distribution Protocol (CR-LDP) or the Resource Reservation Protocol - Traffic Engineering (RSVP-TE).
MPLS is an IETF initiative and is defined in RFC3031. For more information on MPLS, refer to this lecture in our companion subject, Internetworking.

Three (Four?) Letter Acronyms (TLAs)

CLIP - Classical IP over ATM
LANE - LAN Emulation
NHRP - Next Hop Resolution Protocol
MPOA - Multi-Protocol Over ATM
MPLS - Multi-Protocol Label Switching
VCC - ATM Virtual Channel Connection
LDP - Label Distribution Protocol LIS - Logical IP subnetwork
LER - Label Edge Router
LSR - Label Switch Router
QoS - Quality of Service
IETF - Internet Engineering Task Force
ATM Forum - The telco's answer to IETF

http://ironbark.bendigo.latrobe.edu.au/subjects/DC/lectures/21/

ADSL Information

emaknyus@gmail.com (tamvan & verani) — Fri, 16 Sep 2011 14:21:00 -0700

~ New! Broadband Availability Checker - Checks adsl line and exchange status. ~ Max Speed Calculator - Indication of your maximum speed. ~ How to get your line stats from your router ~ How to understand your line stats ~ Connecting and adsl filters. ~ Networking and setting up a router ~ How adsl works. ~ Different types of adsl - IPStream, Datastream + LLU ~ DSL Max & Interleaving ~ How DSLMax works & your bRAS IP Profile information ~ Troubleshooting and adsl connection problems ~ Common Faults - Low SNR and REIN faults ~ Understanding SNR and attenuation ~ Slow speed problems ~ MTU Problems and MTU tweaking

http://www.kitz.co.uk/adsl/index.htm

What is Clonezilla?

emaknyus@gmail.com (tamvan & verani) — Fri, 16 Sep 2011 14:09:00 -0700

You're probably familiar with the popular proprietary commercial package Norton Ghost®. The problem with these kind of software packages is that it takes a lot of time to massively clone systems to many computers. You've probably also heard of Symantec's solution to this problem,Symantec Ghost Corporate Edition® with multicasting. Well, now there is an OpenSource clone system (OCS) solution called Clonezilla with unicasting and multicasting!
Clonezilla, based on DRBL, Partclone and udpcast, allows you to do bare metal backup and recovery. Two types of Clonezilla are available,Clonezilla live and Clonezilla SE (server edition). Clonezilla live is suitable for single machine backup and restore. While Clonezilla SE is for massive deployment, it can clone many (40 plus!) computers simultaneously. Clonezilla saves and restores only used blocks in the harddisk. This increases the clone efficiency. At the NCHC's Classroom C, Clonezilla SE was used to clone 41 computers simultaneously. It took only about 10 minutes to clone a 5.6 GBytes system image to all 41 computers via multicasting!

Perbedaan redirect to proxy memakai dst-nat dan mark-routing

emaknyus@gmail.com (tamvan & verani) — Fri, 16 Sep 2011 09:56:00 -0700

Mungkin selama ini anda tidak pernah menyadari traffic sebuah server yang mengirimkan data atas permintaan ip klien jika menggunakan dst-nat to proxy itu server akan mengirim ke ip proxy bukan langsung ke ip klien yang telah melakukan permintaan tadi

Jadi kita akan mengalami kesulitan jika ingin membuat rule bypass/limit untuk ip server (port 80) yang dari luar yang telah masuk ke ip proxy

Contohnya ip klien 192.168.100.124 mengirim permintaan ke ip server 119.2.24.66:80 (server4.asixfiles.com) jika memakai dst-nat maka permintaan 119.2.24.66:80 ini akan di mapping ke ip proxy (contohnya 192.168.1.10:8080) sesuai NAT redirect pada mikrotik :

1/ip firewall nat

2add action=dst-nat chain=dstnat comment="" disabled=no dst-port=80 protocol=\

3    tcp src-address=192.168.100.0/24 to-addresses=192.168.1.10 to-ports=8080

Secara otomatis ip server 119.2.24.66:80 (server4.asixfiles.com) mengirimkan data kepada ip proxynya bukan ke klien langsung karena yang melakukan permintaan adalah ip proxy bukan ip klien dan ip klien baru menerima data dari ip server melalui ip proxynya

dstnat

Jadi kesimpulannya kita tidak bisa memisahkan ip server ini karena telah masuk ke ip proxy, seandainya kita mau melakukan bypass/limitter server tertentu yang dari port 80 akan mengalami kesulitan.

Dan disinilah kelebihan jika memakai redirect to proxy memakai mark-routing, sebab ip klien bukan lagi di mapping melainkan dikirim langsung ke ip server melalui GATEWAY. jadi systemnya adalah original id’s seolah-olah tidak memakai proxy/mapping untuk permintaan dari ip klien.

Contohnya ip klien melakukan permintaan ke server dan langsung dikirim oleh router melalui GATEWAY dan GATEWAY ini meneruskan ip klien tadi ke ip server baru ip server mengirimkan data atas permintaan ip klien.
Jadi kita mempunyai 2 GATEWAY dalam satu router dong?
Benar.. kita punya 2 GATEWAY satu untuk ke internet dan satu lagi ke proxy sebagai transparent proxynya dan itu dilakukan melalui mark-routing di mikrotik.

contoh mark-routingnya:

1/ip firewall mangle

2add action=mark-routing chain=prerouting comment="To Proxy (Expert ONLY!)" \

3    disabled=no dst-port=80 new-routing-mark=proxy passthrough=yes protocol=\

4    tcp src-address=192.168.100.0/24

dan untuk route-nya

1/ip route

2add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.10 \

3    routing-mark=proxy scope=30 target-scope=10

Dan hasilnya seperti pada gambar di bawah ini:

mark-routing

Anda bisa melihat sendiri khan interface PROXY mengirimkan data langsung dari ip server ke ip klien

Permasalahannya bukan sampai disini saja, anda harus mengubah server proxy anda menjadi sebuah GATEWAY biar bisa melakukan forward ke internet dan ke klien (LAN). Dan untuk mesin FreeBSD dan Linux anda harus melakukan beberapa hal seperti berikut:

FreeBSD
tambahkan gateway_enable=”YES” pada “/etc/rc.conf” artinya diubah sebagai gateway
dan bikin autoexec untuk redirect to proxy pada “/etc/rc.local” (kalau tidak ada anda dapat create file baru) dan tambahan command berikut:
ipfw add 100 fwd 192.168.1.10,8080 tcp from any to any 80 recv em0
atau bisa juga membuat bin/.sh pada “/usr/local/etc/rc.d/”
jangan lupa sesuikan ip dan em0 sesuai network anda.
terakhir recompile kernel FreeBSD anda dengan menambahkan
“options IPFIREWALL_FORWARD”
“options IPFIREWALL_DEFAULT_TO_ACCEPT” # ini opsi untuk jaga-jaga kalau macet saja biar tidak ada yg ke blokir karena server ini bukan untuk IP PUBLIK

Linux
tentunya harus di install dulu paket ip-tables (silahkan google cara installnya sesuai distro anda)
untuk enable forward edit/tambahkan value net.ipv4.ip_forward = 1 pada “/etc/sysctl.conf”
masquerade menggunakan ip-tables via terminal

1iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

redirect port menggunakan ip-tables

1iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

eth0 dan port 8080 sesuaikan dengan network anda.
lalu save dan restart ip-tablesnya
service iptables save
service iptables restart
atau silahkan mencari referensi sendiri sesuai distro masing-masing

Selamat mencoba!

diantaranya yang sudah sukses mencoba di FMI
Forummikrotik
wiki.warneter.net

dan inilah cuplikan bw management yang simple gk pake ribet kalau di lihat juga gak bikin mumet
klik untuk memperbesar gambar

http://reges007.wordpress.com/2011/05/22/perbedaan-redirect-to-proxy-memakai-dst-nat-dan-mark-routing/

td 8117 exploit

emaknyus@gmail.com (tamvan & verani) — Sat, 10 Sep 2011 01:16:00 -0700

http://carnal0wnage.attackresearch.com/
http://www.exploit-db.com/exploits/10276/
http://www.securityfocus.com/archive/1/349085
http://www.cvedetails.com

ultimate.sh

emaknyus@gmail.com (tamvan & verani) — Thu, 8 Sep 2011 16:52:00 -0700

#!/bin/bash
# ultimate.sh for duron
# Based on Wonder Shaper v1.1a
echo "/usr/sbin/ultimate.sh:  "

# Please read the README before filling out these values.
# Set the following values to somewhat less than your actual download
# and upload speed in kilobits.  Also set the device that is to be shaped.

# Run a speed test from netspeed.stanford.edu with no shaping enabled.
# Multiply the reported rates in Kb/s by ~.95 and enter them here:
# Example: DNLINK=4.76M*.95 = 4522
#          UPLINK=444.26K*.95 = 422
# Run the stanford test again.  Note: stanford uses high ports.
# Tweak during heavy upload so that each affected class has a small backlog but
# as few dropped packets as possible.  Interactive must NEVER backlog!
# If heavy downloads affect uploads, tweak IMQ so there is a small backlog but
# as few dropped packets as possible.
# Watch the logs for "some class has too small rate" even with quantum set.
# Stanford is in class 30; egress rate 36% ceil 92%, ingress rate 57% ceil 92%
# Download w/HTB is plenty (5.2x), don't tweak.
# Upload w/UPLINK=470 -> range 416 - 420 = ~85% of uCEIL.  Try for 90% - 91%.


dCEIL=9740	# Stanford max down with no shaping (best of 3 tests)
DNLINK=9350	# (.95*dCEIL, rounded down)
uCEIL=974	# Stanford max up (best of 3 tests).
UPLINK=952	# Tweak.  (was 470=.95*uCEIL, rounded up)
# Percentages, egress:
# sum(Ie + Ae + Be + Pe) _MUST_ = 100
Ie=7			# Interactive
Ae=50			# Accelerated
Be=36			# Bulk
Pe=7			# Penalized

# Percentages, ingress (IMQ):
# sum(Ii + Ai + Bi) _MUST_ = 100
Ii=5			# Interactive
Ai=38			# Accelerated
Bi=57			# Bulk

IFE=eth1		# Egress NIC; External Interface
IMQ=imq0		# Egress NIC - shape incoming

# Local IPs:
ATJ=`cat /etc/firewall/Duron`

# Remote IPs:
#ADE="216.196.97.0/24"	# adelphia.net
CHS=""	# chsoft.biz
DMF=""	# Diana's
NGUY="74.209.0.80/28"	# Newsguy
#HBED1="62.146.66.160/27" # H+BEDV (antivirus)
#HBED2="217.11.60.0/27"	# H+BEDV (antivirus)
#ISW1="207.178.128.0/24"	# iswest
#ISW2="216.166.71.0/24"	# iswest
#ISW3="216.196.105.0/24"	# iswest (giganews)
LHD1=""	# L. H. Dottie
LHD2=""	# L. H. Dottie
#NEWS=""	# news.chsoft.biz
NSAD="67.21.15.0/24"	# Adelphia dns servers
NSTW="66.75.164.0/24"	# Time Warner dns servers
#QUE1="63.231.95.0/24"	# quest
#QUE2="207.225.159.0/24"	# quest
SC=`cat /etc/firewall/SupportJ`	# Note that ports 5500 and 5900 have priority
#TERA="66.150.105.0/24"	# teranews
WALK=""	# Walker Foods
#YIC=""	# yesican.chsoft.biz

###### EXPLANATION
# See 'INGRESS' below for information about incoming packets.  All of this
# except for that refers to queueing outgoing packets.
#
# Most of what is sent goes out on a randomly selected high SOURCE port 
# ('sport') to a specific DESTINATION port ('dport').  For example, an http 
# request is sent with 'dport' = 80 and 'sport' = random.  Therefore, 'sport'
# specifications are rarely applicable.  The port number is located in the
# packet header and is specified thus:
# "u32 match ip dport PORT# 0xffff" or "u32 match ip sport PORT# 0xffff"
#
# Since shaping means quequeing OUTGOING packets, your IP Address(es) are 
# SOURCEs ('src') and the remote machine's IP Address(es) are DESTINATIONs 
# ('dst') - so, as with 'sport', 'src' specifications are rarely applicable. 
# The IP Address is located in the packet header and is specified thus:
# "u32 match ip src IP-ADDRESS" or "u32 match ip dst IP-ADDRESS"
#
# Read the HOWTO "All the filtering commands you will normally need" for
# protocol information (TCP, UDP, ICMP, GRE, IPSEC).  Also see the TOS and
# ICMP sections below.
######

# Sometimes you may notice low priority OUTGOING traffic slowing down important
# traffic. In that case, the following eLoPrio options may help you:

## * * * * Structure:
## {1} * * * * * IFE DEFINITIONS
## {2} * * * * * EGRESS on IFE (HERE --> INTERNET)
## {3} * * * * * IMQ INGRESS shaping on IFE (HERE <-- INTERNET)

###### {1} External Interface DEFINITIONS:
#			Ports:
# eLoPrioSPORT ('sport')
#	Set this to source ports that should have low priority.  If you have
#	an unimportant webserver on your traffic, set this to 80.
# These go into class 40
#   (some ports        ) 20   21    22 23 25   53  80   113   119  873 
#   (and what they are:) FTPd FTPc  SSH   SMTP DNS HTTP IDENT NNTP Rsync
# 9001 is tor Server, 9030 is tor Directory
# Search 6881 for torrent special handling (1:40)
# Matches FROM me:PORT 
eLoPrioSPORT="8 21 23 67 68 79 110 135 137 138 139 389 445 446 901 1026 1027 1028 1029 1234 5018"

# eHiPrioSPORT ('sport')
# These go into class 20.  SSH and scp are interactive (class 10).
# *** Make sure no eLoPrioSPORTs are duplicated here! ***
# Matches (apparently) nothing.
eHiPrioSPORT="113 119 123 443 563 873 5500 5900"

# eLoPrioDPORT ('dport')
#	Set this to destination ports that should have low priority. 
# The following are from my TARPIT list: 79 135 137 138 139 445 901
# Matches DIR=FROM? remote:PORT
# These go into class 40
eLoPrioDPORT="8 21 23 67 68 79 110 135 137 138 139 389 445 446 901 1026 1027 1028 1029 1234 5018"

# eHiPrioDPORT ('dport')
# *** Make sure no eLoPrioDPORTs are duplicated here! ***  ssh is interactive.
# Matches DIR=FROM? ?:PORT
# These go into class 20
eHiPrioDPORT="113 119 123 443 563 873 5500 5900"

#			IPs / Netmasks:
# Set this to hosts or netmasks in your network that should have low priority.
# Low priority OUTGOING traffic.  You can leave this blank if you want.
# Matches TO REMOTE
# These go into class 40
eLoPrioCIDR_Src=""
#eLoPrioCIDR_Src="$ATJ"

# See EXPLANATION above; these do not need to be here.
# Matches (apparently) nothing.
# These go into class 20
eHiPrioCIDR_Src="$CHS $LHD1 $NGUY $SC $WALK"

# Set this to hosts or netmasks on the internet that should have low priority.
# Matches FROM REMOTE
# These go into class 40
eLoPrioCIDR_Dst="221.0.0.0/8"

# high priority destination netmasks ('dst'):
# Matches DIR? REMOTE:IP
# These go into class 20
eHiPrioCIDR_Dst="$CHS $LHD1 $NGUY $SC $WALK"
###### End External Interface DEFINITIONS

if [ "$1" = "status" ]; then
	echo "Sent on $IFE:"
# 	tc -s filter show dev $IFE
	tc -d qdisc ls dev $IFE
	tc -s qdisc ls dev $IFE
	tc -s class ls dev $IFE
	echo "'rate #bit' means Bytes per Second"
	echo "Received on $IMQ:"
#	tc -s filter show dev $IMQ
	tc -d qdisc ls dev $IMQ
	tc -s qdisc ls dev $IMQ
	tc -s class ls dev $IMQ
	echo "'rate #bit' is meaningless.  Measure by Backlog and Dropped."
	exit
fi

# Clean existing setup, hiding errors
tc qdisc del dev $IFE root	2> /dev/null > /dev/null
tc qdisc del dev $IFE ingress	2> /dev/null > /dev/null
iptables -t mangle -D PREROUTING -i $IFE -j IMQ --todev 0	2>/dev/null > /dev/null
tc qdisc del dev $IMQ root	2> /dev/null > /dev/null
ip link set $IMQ down		2> /dev/null > /dev/null
modprobe -r ipt_IMQ		# iptables
modprobe -r imq			# device
modprobe -r cls_u32
modprobe -r sch_esfq		# requires modified tc
modprobe -r sch_sfq
modprobe -r sch_htb
modprobe -r sch_ingress

if [ "$1" = "stop" ]; then 
	exit
fi

###### {2} uplink (EGRESS) on IFE
# This part shapes on the external interface:  (HERE --> INTERNET)
# Install root HTB, point default traffic to 1:30:
tc qdisc add dev $IFE root handle 1: htb default 30

# Shape everything at $UPLINK speed - this prevents huge queues in your
# DSL modem which destroy latency:
tc class add dev $IFE parent 1: classid 1:1 htb rate ${uCEIL}kbit burst 32k cburst 20k

# High prio (interactive) class 1:10:
tc class add dev $IFE parent 1:1 classid 1:10 htb rate $[Ie*$UPLINK/100]kbit \
   ceil ${uCEIL}kbit burst 32k cburst 20k quantum 1514 prio 1

# Accelerated class 1:20 - the HIPRIO stuff:
tc class add dev $IFE parent 1:1 classid 1:20 htb rate $[Ae*$UPLINK/100]kbit \
   ceil $[96*$uCEIL/100]kbit burst 16k cburst 10k quantum 1514 prio 2

# Bulk & default class 1:30 - gets a low priority:
tc class add dev $IFE parent 1:1 classid 1:30 htb rate $[Be*$UPLINK/100]kbit \
   ceil $[92*$UPLINK/100]kbit burst 8k cburst 5k quantum 1514 prio 3
#   ceil $[Ae*$UPLINK/100]kbit burst 8k cburst 5k quantum 1514 prio 3

# Penalized class 1:40 - the LOPRIO stuff gets the lowest rate and priority:
tc class add dev $IFE parent 1:1 classid 1:40 htb rate $[Pe*$UPLINK/100]kbit \
   ceil $[80*$UPLINK/100]kbit burst 2k quantum 1514 prio 4
#   ceil $[Be*$UPLINK/100]kbit burst 2k quantum 1514 prio 4

# Some get Stochastic Fairness:
# LARTC mailing list indicates that interactive should not be included.  Andy
# Furniss suggests that bulk be the only thing subject to SF.  I set a long
# perturb for accelerated and exclude interactive.
# Increased limit from 64 to reduce # of dropped packets 7Dec07
# 11Dec07: esfq limit must be less than or equal depth; depth max = 1024:
#tc qdisc add dev $IFE parent 1:10 handle 10: esfq limit 64 depth 64 divisor 10 hash classic perturb 20
tc qdisc add dev $IFE parent 1:20 handle 20: esfq limit 256 depth 256 divisor 10 hash classic perturb 99
tc qdisc add dev $IFE parent 1:30 handle 30: esfq limit 1024 depth 1024 divisor 10 hash classic perturb 20
tc qdisc add dev $IFE parent 1:40 handle 40: esfq limit 32 depth 32 divisor 10 hash classic perturb 10

# ARP:
tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
   match u16 0x0806 0xffff at -2 flowid 1:40

###### Accelerate (interactive) the following:
# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class 1:10:
# IP header length 0x5 (32 bit words)
# IP total length 0x34 (ACK + 12 bytes of TCP options)
# TCP ACK set (bit 5, offset 33)
# ("at nexthdr+33" = "at 13")

## Match ACK on all TCP packets with the ACK bit set:
## Caveat!: This can match packets up to 64K; don't do it.  Match small.
#tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
#    match ip protocol 6 0xff \
#    match u8 0x10 0xff at nexthdr+13 \
#    flowid 1:10

# Match TCP packets smaller than 128 bytes:
# Because of the mask, can only match powers of 2 (32, 64, 128...)
# 0xfff8 - 8
# 0xfff0 - 16
# 0xffe0 - 32
# 0xffc0 - 64
# 0xff80 - 128
# Remarked out ACK, normally 5th line:  match u8 0x10 0xff at 33 \
# "match u8 0x05 0x0f at 0" makes sure the IP header is 20 bytes.
tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
    match ip protocol 6 0xff \
    match u8 0x05 0x0f at 0 \
    match u16 0x0000 0xff80 at 2 \
    flowid 1:10

# SSH and scp are interactive.
tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
    match ip dport 22 0xffff flowid 1:10
tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
    match ip sport 22 0xffff flowid 1:10

# dns is interactive:
tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
    match ip dport 53 0xffff flowid 1:10
tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
    match ip sport 53 0xffff flowid 1:10

###### Some traffic is preferred, so in our class 1:20:
for a in $eHiPrioDPORT; do
 	tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
	   match ip dport $a 0xffff flowid 1:20
done
for a in $eHiPrioSPORT; do
 	tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
	   match ip sport $a 0xffff flowid 1:20
done
for a in $eHiPrioCIDR_Src; do
 	tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
	   match ip src $a flowid 1:20
done
for a in $eHiPrioCIDR_Dst; do
 	tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
	   match ip dst $a flowid 1:20
done

###### Give these short shrift:
# Penalized traffic suffers a worse fate in our slowest class 1:40:
# Special for torrents ports 6880-6887:
# The mask determines the number of ports: ffff=1 fffe=2 fffc=4 fff8=8 fff0=16
tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
   match ip dport 6881 0xfff8 flowid 1:40
tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
   match ip sport 6881 0xfff8 flowid 1:40
for a in $eLoPrioDPORT; do
	tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
	   match ip dport $a 0xffff flowid 1:40
done
for a in $eLoPrioSPORT; do
 	tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
	   match ip sport $a 0xffff flowid 1:40
done
for a in $eLoPrioCIDR_Src; do
 	tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
	   match ip src $a flowid 1:40
done
for a in $eLoPrioCIDR_Dst; do
 	tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
	   match ip dst $a flowid 1:40
done
if [ -f /etc/firewall/shitlist ]; then
   while read SLIST; do
      SLIST=$(echo "$SLIST" | awk '{print $1}')
      tc filter add dev $IFE parent 1: protocol all prio 5 u32 \
         match ip dst $SLIST flowid 1:40
   done <-- INTERNET)
modprobe imq numdevs=1
# Since class 1:10 uses the default pfifo_fast, we need to give it txqueuelen
# packets because pfifo_fast drops incoming packets when the queue fills up:
ip link set $IMQ up txqueuelen 1024 mtu 1500

# Install root HTB, point default traffic to 1:30:
tc qdisc add dev $IMQ root handle 1: htb default 30

# Shape everything at DNLINK speed:
# Root class
tc class add dev $IMQ parent 1: classid 1:1 htb rate ${dCEIL}kbit burst 32k \
  cburst 20k quantum 1500

# High priority (interactive) class 1:10
tc class add dev $IMQ parent 1:1 classid 1:10 htb rate $[Ii*$DNLINK/100]kbit \
   ceil ${dCEIL}kbit burst 32k cburst 20k quantum 1500 prio 1

# Accelerated class 1:20 
tc class add dev $IMQ parent 1:1 classid 1:20 htb rate $[Ai*$DNLINK/100]kbit \
   ceil $[96*$dCEIL/100]kbit burst 16k cburst 10k quantum 1500 prio 2

# Bulk & default class 1:30
tc class add dev $IMQ parent 1:1 classid 1:30 htb rate $[Bi*$DNLINK/100]kbit \
   ceil $[92*$dCEIL/100]kbit burst 8k cburst 5k quantum 1500 prio 3

# Stochastic Fairness
tc qdisc add dev $IMQ parent 1:20 handle 20: esfq limit 1024 depth 1024 divisor 10 hash dst perturb 99 
tc qdisc add dev $IMQ parent 1:30 handle 30: esfq limit 768 depth 768 divisor 10 hash dst perturb 20 

# Interactive ports:
IMQint="22 53 5500 5900"

# Accelerated ports:
IMQacc="80 113 123 443 873 6502"

for a in $IMQint; do
   tc filter add dev $IMQ parent 1: protocol all prio 5 u32 match \
      ip dport $a 0xffff flowid 1:10
   tc filter add dev $IMQ parent 1: protocol all prio 5 u32 match \
      ip sport $a 0xffff flowid 1:10
done

for a in $IMQacc; do
   tc filter add dev $IMQ parent 1: protocol all prio 5 u32 match \
      ip dport $a 0xffff flowid 1:20
   tc filter add dev $IMQ parent 1: protocol all prio 5 u32 match \
      ip sport $a 0xffff flowid 1:20
done

# Not needed because 1:30 is default:
## Match everything else: 
#tc filter add dev $IMQ parent 1: protocol all prio 5 u32 match u32 0 0 flowid 1:30

iptables -t mangle -I PREROUTING -i $IFE -j IMQ --todev 0

# Done ultimate.sh

fw_function

emaknyus@gmail.com (tamvan & verani) — Thu, 8 Sep 2011 16:51:00 -0700

#-----------------------
# FONCTIONS Firewall
# guibo@guibo.com
# version 1.2
# tested on slk
#-----------------------

load_module() {
IP_MODULES=`/sbin/lsmod | awk '{print $1}' | /bin/grep '^ip' | grep $1`
if [ -z "$IP_MODULES" ]; then
    if [ -e $PATH_modules/$1.$EXTENSION_module ]; then
            /sbin/insmod $PATH_modules/$1.$EXTENSION_module
            else
            echo " - MUST HAVE Compiled kernel $1 support"
    fi
fi
}

load_module_q() {
IP_MODULES=`/sbin/lsmod | awk '{print $1}'`
if [ -z "$IP_MODULES" ]; then
    if [ -e $PATH_modules_q/$1.$EXTENSION_module ]; then
            /sbin/insmod $PATH_modules_q/$1.$EXTENSION_module
            else
            echo " - MUST HAVE Compiled kernel $1 support"
    fi
fi
}

flush_regle() {
echo " - Flushing Table"
$IPTABLES -F
$IPTABLES -F -t mangle
$IPTABLES -t mangle -X
$IPTABLES -t filter -F INPUT
$IPTABLES -t filter -F OUTPUT
$IPTABLES -t filter -F FORWARD
$IPTABLES -t nat --flush PREROUTING
$IPTABLES -t nat --flush POSTROUTING
$IPTABLES -t nat --flush OUTPUT
$IPTABLES -X
}

zero_compteur() {
echo " - Reseting compteur"
$IPTABLES -t filter -Z INPUT
$IPTABLES -t filter -Z OUTPUT
$IPTABLES -t filter -Z FORWARD
$IPTABLES -t nat -Z PREROUTING
$IPTABLES -t nat -Z POSTROUTING
$IPTABLES -t nat -Z OUTPUT
$IPTABLES -t mangle -F
}

config_kernel() {
echo " - Rejecting non routable ip"
for i in $KERNEL_conf/conf/*/rp_filter ; do echo 1 > $i ; done
echo " - Logging Martians"
if [ -e $KERNEL_conf/conf/all/log_martians ]; then
       for i in $KERNEL_conf/conf/*/log_martians ; do
    echo 1 > $i ; done
    fi
echo " - Ignore broadcast PING"
if [ -e $KERNEL_conf/icmp_echo_ignore_broadcasts ]; then
        echo 1 > $KERNEL_conf/icmp_echo_ignore_broadcasts
fi
echo " - Icmp Bogus reponse"
if [ -e $KERNEL_conf/icmp_ignore_bogus_error_responses ]; then
        echo 1 > $KERNEL_conf/icmp_ignore_bogus_error_responses
    fi
echo " - DROP icmp_redirect"
if [ -e $KERNEL_conf/conf/all/accept_redirects ]; then
        echo 0 > $KERNEL_conf/conf/all/accept_redirects
fi
echo " - Active ip_forward"
echo 1 > $KERNEL_conf/ip_forward
if [ -e $KERNEL_conf/conf/all/accept_source_route ];then
        echo 1 > $KERNEL_conf/conf/all/accept_source_route
fi
}

net_redirect() {
IF_EXT=$1        # external interface
PORT_ex=$2        # external port to redirect
IP_REDIR_in=$3        # internal IP
PORT_REDIR_in=$4    # internal port
echo " - Redirecting $IF_EXT:$PORT_ex to $IP_REDIR_in:$PORT_REDIR_in"
$IPTABLES -A FORWARD -p tcp -i $IF_EXT --dport $PORT_ex -j ACCEPT
$IPTABLES -A OUTPUT -p tcp -o $IF_EXT --dport $PORT_ex -j ACCEPT

$IPTABLES -t nat -A PREROUTING -p tcp -i $IF_EXT --dport $PORT_ex \
        -j DNAT --to $IP_REDIR_in:$PORT_REDIR_in
}

default_rule() {
echo " - Default rule set"
$IPTABLES -P INPUT DROP
$IPTABLES -P FORWARD DROP
$IPTABLES -P OUTPUT ACCEPT
# $IPTABLES -A FORWARD -f -j ACCEPT # accept fragment
}

block_nmap() {
echo " - Creating VALID_CHECK rule"
$IPTABLES -N VALID_CHECK
$IPTABLES -F VALID_CHECK

LOGLEVEL=7
# (NMAP) FIN/URG/PSH
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags ALL FIN,URG,PSH -m limit --limit 3/m --limit-burst 5 -j LOG --log-level $LOGLEVEL --log-prefix "Stealth XMAS scan: "

# SYN/RST/ACK/FIN/URG
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -m limit --limit 3/m --limit-burst 5 -j LOG --log-level $LOGLEVEL --log-prefix "Stealth XMAS-PSH scan: "

# ALL/ALL
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags ALL ALL -m limit --limit 3/m --limit-burst 5 -j LOG --log-level $LOGLEVEL --log-prefix "Stealth XMAS-ALL scan: "

# NMAP FIN Stealth
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags ALL FIN -m limit --limit 3/m --limit-burst 5 -j LOG --log-level $LOGLEVEL --log-prefix "Stealth FIN scan: "

# SYN/RST
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags SYN,RST SYN,RST -m limit --limit 3/m --limit-burst 5 -j LOG --log-level $LOGLEVEL --log-prefix "Stealth SYN/RST scan: "

# SYN/FIN (probably)
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags SYN,FIN SYN,FIN -m limit --limit 3/m --limit-burst 5 -j LOG --log-level $LOGLEVEL --log-prefix "Stealth SYN/FIN scan(?): "

# Null scan
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags ALL NONE -m limit --limit 3/m --limit-burst 5 -j LOG --log-level $LOGLEVEL --log-prefix "Stealth Null scan: "

# NMAP FIN/URG/PSH
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags ALL FIN,URG,PSH -j DROP

# SYN/RST/ACK/FIN/URG
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j DROP

# ALL/ALL Scan
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags ALL ALL -j DROP

# NMAP FIN Stealth
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags ALL FIN -j DROP

# SYN/RST
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags SYN,RST SYN,RST -j DROP

# SYN/FIN -- Scan(probably)
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP

# NMAP Null Scan
$IPTABLES -A VALID_CHECK -p tcp --tcp-flags ALL NONE -j DROP

# Port 0 fingerprint attempt
echo " - Detecting Fingerprint attempt"
$IPTABLES -N CHECK
$IPTABLES -F CHECK
$IPTABLES -A CHECK -p tcp --dport 0 -m limit --limit 1/h --limit-burst 1 -j LOG --log-level $LOGLEVEL --log-prefix "TCP port 0 OS fingerprint: "
$IPTABLES -A CHECK -p udp --dport 0 -m limit --limit 1/h --limit-burst 1 -j LOG --log-level $LOGLEVEL --log-prefix "UDP port 0 OS fingerprint: "

# Drop port 0 port scan packets
$IPTABLES -A CHECK -p tcp --dport 0 -j DROP
$IPTABLES -A CHECK -p udp --dport 0 -j DROP
}

set_tos() {
# TOS table
# Options:
#               Normal-Service = 0 (0x00)
#               Minimize-Cost = 2 (0x02)
#               Maximize-Reliability = 4 (0x04)
#               Maximize-Throughput = 8 (0x08)
#               Minimize-Delay = 16 (0x10)
IF_EXT=$1
echo " - Setting Tos"
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport ftp-data -j TOS --set-tos Minimize-Cost
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport ftp -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport ssh -j TOS --set-tos Minimize-Delay
#$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport telnet -j TOS --set-tos Minimize-Delay
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport smtp -j TOS --set-tos Minimize-Cost
$IPTABLES -t mangle -A OUTPUT -p udp -o $IF_EXT --dport domain -j TOS --set-tos Maximize-Throughput
#$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport bootps -j TOS --set-tos Minimize-Delay
# to add http in priority list of QoS, is set in Minimize-Delay not in Maximize-Throughput
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport http -j TOS --set-tos Minimize-Delay
# http internal
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport 81 -j TOS --set-tos Minimize-Delay
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport pop3 -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport auth -j TOS --set-tos Minimize-Delay
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport ntp -j TOS --set-tos Minimize-Delay
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport imap -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport https -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport imaps -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport pop3s -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport socks -j TOS --set-tos Minimize-Delay
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport 6000:6063 -j TOS --set-tos Maximize-Throughput

# rules for mldonkey
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport 4080 -j MARK --set-mark 1
$IPTABLES -t mangle -A OUTPUT -p tcp -o $IF_EXT --dport 4662 -j MARK --set-mark 1

$IPTABLES -t mangle -A PREROUTING -p tcp --dport ftp-data -j TOS --set-tos Minimize-Cost
$IPTABLES -t mangle -A PREROUTING -p tcp --dport ftp -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A PREROUTING -p tcp --dport ssh -j TOS --set-tos Minimize-Delay
#$IPTABLES -t mangle -A PREROUTING -p tcp --dport telnet -j TOS --set-tos Minimize-Delay
$IPTABLES -t mangle -A PREROUTING -p tcp --dport smtp -j TOS --set-tos Minimize-Cost
$IPTABLES -t mangle -A PREROUTING -p udp --dport domain -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A PREROUTING -p tcp --dport bootps -j TOS --set-t Minimize-Delay
# http internal
$IPTABLES -t mangle -A PREROUTING -p tcp --dport 81 -j TOS --set-tos Minimize-Delay
$IPTABLES -t mangle -A PREROUTING -p tcp --dport http -j TOS --set-tos Minimize-Delay
$IPTABLES -t mangle -A PREROUTING -p tcp --dport pop3 -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A PREROUTING -p tcp --dport auth -j TOS --set-tos Minimize-Delay
$IPTABLES -t mangle -A PREROUTING -p tcp --dport ntp -j TOS --set-tos Minimize-Delay
$IPTABLES -t mangle -A PREROUTING -p tcp --dport imap -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A PREROUTING -p tcp --dport https -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A PREROUTING -p tcp --dport imaps -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A PREROUTING -p tcp --dport pop3s -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A PREROUTING -p tcp --dport socks -j TOS --set-tos Minimize-Delay
$IPTABLES -t mangle -A PREROUTING -p tcp --dport 6000:6063 -j TOS --set-tos Maximize-Throughput
}

masquerade_net() {
HOME_NET=$1     # ex: 10.0.1.0/24
echo " - Masquerading $HOME_NET"
$IPTABLES -t nat -A POSTROUTING -s $HOME_NET -j MASQUERADE
}

accept_tcp() {
IN_ETH=$1     # input interface
PORT=$2     # port for network service
echo " - Accept TCP port $PORT on $IN_ETH"
$IPTABLES -A INPUT -p tcp -i $IN_ETH --dport $PORT -j ACCEPT
}

accept_udp() {
IN_ETH=$1     # input interface
PORT=$2     # port for network service
echo " - Accept UDP port $PORT from $IN_ETH"
$IPTABLES -A INPUT -p udp -i $IN_ETH --dport $PORT -j ACCEPT
}

create_block() {
echo " - Creating Block "
$IPTABLES -N block
$IPTABLES -F block
}

accept_interne() {
# accept all traffic from internal network
echo " - Accept traffic coming from internal"
$IPTABLES -A block -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A block -m state --state NEW -i ! $IF_EXT -j ACCEPT
}

reject_host() {
IF_ETH=$1
IP_DENIED=$2
echo " - Deny $IP_DENIED on $IF_ETH"
$IPTABLES -I INPUT -p tcp -s $IP_DENIED -j DROP
$IPTABLES -I INPUT -p udp -s $IP_DENIED -j DROP
}

reject_tcp_udp() {
IF_ETH=$1
IP_DENIED=$2
PORT=$3
echo " - Deny $IP_DENIED on $IF_ETH port $PORT"
$IPTABLES -I INPUT -p tcp -i $IF_ETH --dport $PORT -s $IP_DENIED -j DROP
$IPTABLES -I INPUT -p udp -i $IF_ETH --dport $PORT -s $IP_DENIED -j DROP
}

drop_trojan() {
PORT_DENY=$1
NAME_trojan=$2
PROTOCOL=$3
echo " - Deny $NAME_trojan $PORT_DENY $PROTOCOL"
$IPTABLES -A DENY_PORTS -p $PROTOCOL --dport $PORT_DENY -m limit --limit 5/minute \
                -j LOG --log-level 7 --log-prefix "$NAME_trojan:"
$IPTABLES -A DENY_PORTS -p $PROTOCOL --sport $PORT_DENY -m limit --limit 5/minute \
                -j LOG --log-level 7 --log-prefix "$NAME_trojan:"
$IPTABLES -A DENY_PORTS -p $PROTOCOL --sport $PORT_DENY -j DROP
$IPTABLES -A DENY_PORTS -p $PROTOCOL --dport $PORT_DENY -j DROP
}

reject_all() {
echo " - Reject all with Block"
$IPTABLES -A block -j LOG
$IPTABLES -A INPUT -j block
$IPTABLES -A FORWARD -j block
$IPTABLES -A block -j REJECT
}

reject_classic() {
echo " - Drop Status invalid, flood and PoD"
# invalid state
$IPTABLES -A INPUT -m state --state INVALID -i $IF_EXT -j DROP
$IPTABLES -A FORWARD -m state --state NEW,INVALID -i $IF_EXT -j DROP
# flood
$IPTABLES -A FORWARD -p tcp --syn -m limit --limit 1/s -j ACCEPT
$IPTABLES -A FORWARD -p tcp --tcp-flags SYN,ACK,FIN,RST RST -m limit --limit 1/s -j ACCEPT
# ping of death
$IPTABLES -A FORWARD -p icmp --icmp-type echo-request -m limit --limit 1/s -j ACCEPT
$IPTABLES -A FORWARD -p icmp --icmp-type echo-request -j DROP

echo " - Creating and Applying DENY_PORTS"
$IPTABLES -N DENY_PORTS
$IPTABLES -F DENY_PORTS
# samba
$IPTABLES -A DENY_PORTS -p tcp -i $IF_EXT --dport 137:139 -j DROP
$IPTABLES -A DENY_PORTS -p tcp -i $IF_EXT --sport 137:139 -j DROP
# Drop X
$IPTABLES -A DENY_PORTS -p tcp --dport 5999:6063 -j DROP
$IPTABLES -A DENY_PORTS -p tcp --sport 5999:6063 -j DROP
#Drop VNC
$IPTABLES -A DENY_PORTS -p tcp --dport 5900:5910 -j DROP
$IPTABLES -A DENY_PORTS -p tcp --sport 5900:5910 -j DROP
# Drop NFS
$IPTABLES -A DENY_PORTS -p tcp --dport 2049 -j DROP
$IPTABLES -A DENY_PORTS -p tcp --sport 2049 -j DROP
}

mangle_tc() {
# from http://downloads.thedonkeynetwork.com/dserver_patch/shaped
# This script use QOS (Quality OF Service) to control the bandwidth of the Internet line
# Note : QOS controls only the upload
# CBQ uses timers to control the bandwitdh, and a finer resolution of timer means a finer control of
# the bandwidth
echo " - Setting QoS"
UPLINK=$UPLINK
LOWUP=$LOWUP
DEV=$IF_EXT
LOCALNET=$HOME_NET
AVPKT=200

# clean existing qdiscs, hide errors
tc qdisc del dev $DEV root    >/dev/null 2>&1
# qdisc
tc qdisc add dev $DEV root handle 1: cbq avpkt $AVPKT bandwidth 10mbit allot 1514 cell 8 mpu 96

# classe locale pour notre reseau local, on utilise 9mbit seulement.
# tc class add dev $DEV parent 1: classid 1:1 cbq bandwidth 10mbit rate 9mbit allot 1514 maxburst 40 avpkt $AVPKT prio 5 bounded

# shape everything at $UPLINK speed - this prevents huge queues in your
# DSL modem which destroy latency:
# main class
# tc class add dev $DEV parent 1: classid 1:2 cbq bandwidth 10mbit rate ${UPLINK} allot 1514 cell 8 maxburst 10 avpkt $AVPKT bounded prio 1

# high prio class 1:10:
tc class add dev $DEV parent 1: classid 1:10 cbq rate ${UPLINK} weight 118kbit allot 1514 cell 8 prio 1 avpkt $AVPKT isolated

# low prio class 1:20:
tc class add dev $DEV parent 1: classid 1:20 cbq rate ${LOWUP} weight 10kbit allot 1514 prio 8 mpu 96 avpkt $AVPKT bounded

tc qdisc add dev $DEV parent 1:10 handle 10: sfq quantum 1514b perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq quantum 1514b perturb 10

# comunications on our LAN are at full speed of the link
# Les communications vers le réseau local sont a 10mbit
# tc filter add dev $DEV parent 1:0 protocol ip prio 5 u32 match ip dst $LOCALNET flowid 1:1

# TOS Minimum Delay (ssh) prioritaire
# This is for us, admins, who want to log to our machine remotely with SSH
tc filter add dev $DEV parent 1:0 protocol ip prio 5 u32 match ip tos 0x10 0xff flowid 1:10

# We want SYN packets to be in High priority flow
tc filter add dev $DEV parent 1: protocol ip prio 5 u32 match ip protocol 6 0xff match u8 0x05 0x0f at 0 match u8 0x02 0xff at 33 flowid 1:10

# everything else in Low priority flow
tc filter add dev $DEV parent 1: protocol ip prio 5 u32 match ip dst 0.0.0.0/0 flowid 1:20

# mldonkey rules
tc class add dev $DEV parent 1:0 classid 1:3 cbq bandwidth 10Mbit \
    rate 10Kbit allot 1514 prio 8 maxburst 2 avpkt 200 bounded
tc filter add dev $DEV parent 1:0 protocol ip handle 1 fw flowid 1:3

}

Traffic Control

emaknyus@gmail.com (tamvan & verani) — Thu, 8 Sep 2011 16:49:00 -0700

Introduction

Linux's traffic control functionality offers a lot of capabilities related to influencing the rate of flow, as well as latency, of primarily outgoing but also in some cases incoming network traffic. It is designed to be a "construction kit" rather than a turn-key system, where complex network traffic policing and shaping decisions can be made using a variety of algorithms. The Linux traffic control code is also often used by academia for research purposes, where is it can be a useful mechanism to simulate and explore the impact of a variety of different network behaviors. See netem for an example of a simulation framework that can be used for this purpose.

Of course, Linux traffic control can also be extremely useful in an IT context, and this document is intended to focus on the practical, useful applications of Linux traffic control, where these capabilities can be applied to solve problems that are often experienced on modern networks.

Incoming and Outgoing Traffic

One common use of Linux traffic control is to configure a Linux system as a Linux router or bridge, so that the Linux system sits between two networks, or between the "inside" of the network and the real router, so that it can shape traffic going to local machines as well as out to the Internet. This provides a way to prioritize, shape and police both incoming (from the Internet) and outgoing (from local machines) network traffic, because it is easiest to create traffic control rules for traffic flowing out of an interface, since we can control when the system sends data, but controlling when we receive data requires an additional intermediate queue to be created to buffer incoming data. When a Linux system is configured as a firewall or router with a physical interface for each part of the network, we can avoid using intermediate queues.

A simple way to set up a layer 2 bridge using Linux involves creating a bridge device with brctl, adding two Ethernet ports to this bridge (again using brctl), and then apply prioritization, shaping and policing rules to both interfaces. The rules will apply to outgoing traffic on each interface. One physical interface will be connected to an upstream router on the same network, while the other network port will be connected to a layer 2 access switch to which local machines are connected. This allows powerful egress shaping policies to be created on both interfaces, to control the flows in and out of the network.

Recommended Resources

Resources you should take a look at, in order:

HTB documentation by Martin Devera. Best way to create different priority classes and bandwidth allocations.
Differentiated Services On Linux HOWTO by Leonardo Balliache. Good general docs.
A Practical Guide to Linux Traffic Control by Jason Boxman. Good general docs.
IFB - replacement for Linux IMQ, with examples. This is the official best way to do inbound traffic control, when you don't have dedicated in/out interfaces.
Use of iptables hashlimit - Great functionality in iptables. There's a hashlimit example below as well.

Recommended Approaches

Daniel Robbins has had very good results with the HTB queuing discipline - it has very good features, and also has very good documentation, which is just as important, and is designed to deliver useful results in a production environment. And it works. If you use traffic control under Funtoo Linux, please use the HTB queuing discipline as the root queuing discipline because you will get good results in very little time. Avoid using any other queuing discipline under Funtoo Linux as the root queuing discipline on any interface. If you are creating a tree of classes and qdiscs, HTB should be at the top, and you should avoid hanging classes under any other qdisc unless you have plenty of time to experiment and verify that your QoS rules are working as expected. Please see State of the Code for more info on what Daniel Robbins considers to be the current state of the traffic control implementation in Linux.

State of the Code

If you are using enterprise kernels, especially any RHEL5-based kernels, you must be aware that the traffic control code in these kernels is about 5 years old and contains many significant bugs. In general, it is possible to avoid these bugs by using HTB as your root queueing discipline and testing things carefully to ensure that you are getting the proper behavior. The prio queueing discipline is known to not work reliably in RHEL5 kernels. See Broken Traffic Control for more information on known bugs with older kernels.

If you are using a more modern kernel, Linux traffic control should be fairly robust. The examples below should work with RHEL5 as well as newer kernels.

Inspect Your Rules

If you are implementing Linux traffic control, you should be running these commands frequently to monitor the behavior of your queuing discipline. Replace $wanif with the actual network interface name.

tc -s qdisc ls dev $wanif
tc -s class ls dev $wanif

Matching

Here are some examples you can use as the basis for your own filters/classifiers:

protocol arp u32 match u32 0 0 - match ARP packets
protocol ip u32 match ip protocol 0x11 0xff - match UDP packets
protocol ip u32 match ip protocol 17 0xff - (also) match UDP packets
protocol ip u32 match ip protocol 0x6 0xff - match TCP packets
protocol ip u32 match ip protocol 1 0xff - match ICMP (ping) packets
protocol ip u32 match ip dst 4.3.2.1/32 - match all IP traffic headed for IP 4.3.2.1
protocol ip u32 match ip src 4.3.2.1/32 match ip sport 80 0xffff - match all IP traffic from 4.3.2.1 port 80
protocol ip u32 match ip sport 53 0xffff - match originating DNS (both TCP and UDP)
protocol ip u32 match ip dport 53 0xffff - match response DNS (both TCP and UDP)
protocol ip u32 match ip protocol 6 0xff match u8 0x10 0xff at nexthdr+13 - match packets with ACK bit set
protocol ip u32 match ip protocol 6 0xff match u8 0x10 0xff at nexthdr+13 match u16 0x0000 0xffc0 at 2 - packets less than 64 bytes in size with ACK bit set
protocol ip u32 match ip tos 0x10 0xff - match IP packets with "type of service" set to "Minimize delay"/"Interactive"
protocol ip u32 match ip tos 0x08 0xff - match IP packets with "type of service" set to "Maximize throughput"/"Bulk" (see "QDISC PARAMETERS" in tc-prio man page)
protocol ip u32 match tcp dport 53 0xffff match ip protocol 0x6 0xff - match TCP packets heading for dest. port 53 (my not work)

Sample Traffic Control Code

modemif=eth4
 
iptables -t mangle -A POSTROUTING -o $modemif -p tcp -m tos --tos Minimize-Delay -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -o $modemif -p tcp --dport 53 -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -o $modemif -p tcp --dport 80 -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -o $modemif -p tcp --dport 443 -j CLASSIFY --set-class 1:10
 
tc qdisc add dev $modemif root handle 1: htb default 12
tc class add dev $modemif parent 1: classid 1:1 htb rate 1500kbit ceil 1500kbit burst 10k
tc class add dev $modemif parent 1:1 classid 1:10 htb rate 700kbit ceil 1500kbit prio 1 burst 10k
tc class add dev $modemif parent 1:1 classid 1:12 htb rate 800kbit ceil 800kbit prio 2
tc filter add dev $modemif protocol ip parent 1:0 prio 1 u32 match ip protocol 0x11 0xff flowid 1:10
tc qdisc add dev $modemif parent 1:10 handle 20: sfq perturb 10
tc qdisc add dev $modemif parent 1:12 handle 30: sfq perturb 10

The code above is a working traffic control script that is even compatible with RHEL5 kernels, for a 1500kbps outbound link (T1, Cable or similar.) In this example, eth4 is part of a bridge. The code above should work regardless of whether eth4 is in a bridge or not -- just make sure that modemif is set to the interface on which traffic is flowingout and you wish to apply traffic control.

`tc` code walkthrough

This script uses the tc command to create two priority classes - 1:10 and 1:12. By default, all traffic goes into the low-priority class, 1:12. 1:10 has priority over 1:12 (prio 1vs. prio 2,) so if there is any traffic in 1:10 ready to be sent, it will be sent ahead of 1:12. 1:10 has a rate of 700kbit but can use up to the full outbound bandwidth of 1500kbit by borrowing from 1:12.

UDP traffic (traffic that matches ip protocol 0x11 0xff) will be put in the high priority class 1:10. This can be good for things like FPS games, to ensure that latency is low and not drowned out by lower-priority traffic.

If we stopped here, however, we would get a bit worse results than if we didn't use tc at all. We have basically created two outgoing sub-channels of different priorities. The higher priority class can drown out the lower-priority class, and this is intentional so it isn't the issue -- in this case we want that functionality. The problem is that the high priority and low priority classes can both be dominated by high-bandwidth flows, causing other traffic flows of the same priority to be drowned out. To fix this, two sfqqueuing disciplines are added to the high and low priority classes and will ensure that individual traffic flows are identified and each given a fair shot at sending data out of their respective classes. This should prevent starvation within the classes themselves.

`iptables` code walkthrough

First note that we are adding netfilter rules to the POSTROUTING chain, in the mangle table. This table allows us to modify the packets right before they are queued to be sent out of an interface, which is exactly what we want. At this point, these packets could have been locally-generated or forwarded -- as long as they are on their way to going out of modemif (eth4 in this case), the mangle POSTROUTING chain will see them and we can classify them and perform other useful tweaks.

The iptables code puts all traffic with the "minimize-delay" flag (interactive ssh traffic, for example) in the high priority traffic class. In addition, all HTTP, HTTPS and DNS TCP traffic will be classified as high-priority. Remember that all UDP traffic is being classified as high priority via the tc rule described above, so this will take care of DNS UDP traffic automatically.

Further optimizations

SSH

iptables -t mangle -N tosfix
iptables -t mangle -A tosfix -p tcp -m length --length 0:512 -j RETURN
#allow screen redraws under interactive SSH sessions to be fast:
iptables -t mangle -A tosfix -m hashlimit --hashlimit 20/sec --hashlimit-burst 20 \
--hashlimit-mode srcip,srcport,dstip,dstport --hashlimit-name minlat -j RETURN
iptables -t mangle -A tosfix -j TOS --set-tos Maximize-Throughput
iptables -t mangle -A tosfix -j RETURN
 
iptables -t mangle -A POSTROUTING -p tcp -m tos --tos Minimize-Delay -j tosfix

To use this code, place it near the top of the file, just below the modemif="eth4" line, but before the main iptables and tc rules. These rules will apply to all packets about to get queued to any interface, but this is not necessarily a bad thing, since the TCP flags being set are not just specific to our traffic control functionality. To make these rules specific to modemif, add "-o $modemif" after "-A POSTROUTING" on the last line, above. As-is, the rules above will set the TCP flags on all packets flowing out of all interfaces, but the the traffic control rules will only take effect for modemif, because they are only configured for that interface.

SSH is a tricky protocol. By default, all the outgoing SSH traffic is classified as "minimize-delay" traffic, which will cause it to all flow into our high-priority class, even if it is a bulk scp transfer running in the background. This code will grab all "minimize-delay" traffic such as SSH and telnet and route it through some special rules. Any individual keystrokes (small packets) will be left as "minimize-delay" packets. For anything else, we will run the hashlimit iptables module, which will identify individual outbound flows and allow small bursts of traffic (even big packets) to remain "minimize-delay" packets. These settings have been specifically tuned so that most GNU screen screen changes (^A^N) when logging into your server(s) remotely will be fast. Any traffic over these burst limits will be reclassified as "maximize-throughput" and thus will drop to our lower-priority class 1:12. Combined with the traffic control rules, this will allow you to have very responsive SSH sessions into your servers, even if they are doing some kind of bulk outbound copy, like rsync over SSH.

Code in our main iptables rules will ensure that any "minimize-delay" traffic is tagged to be in the high-priority 1:10 class.

What this does is keep interactive SSH and telnet keystrokes in the high-priority class, allow GNU screen full redraws and reasonable full-screen editor scrolling to remain in the high-priority class, while forcing bulk transfers into the lower-priority class.

ACKs

iptables -t mangle -N ack
iptables -t mangle -A ack -m tos ! --tos Normal-Service -j RETURN
iptables -t mangle -A ack -p tcp -m length --length 0:128 -j TOS --set-tos Minimize-Delay
iptables -t mangle -A ack -p tcp -m length --length 128: -j TOS --set-tos Maximize-Throughput
iptables -t mangle -A ack -j RETURN
 
iptables -t mangle -A POSTROUTING -p tcp -m tcp --tcp-flags SYN,RST,ACK ACK -j ack

To use this code, place it near the top of the file, just below the modemif="eth4" line, but before the main iptables and tc rules.

ACK optimization is another useful thing to do. If we prioritize small ACKs heading out to the modem, it will allow TCP traffic to flow more smoothly without unnecessary delay. The lines above accomplish this.

This code basically sets the "minimize-delay" flag on small ACKs. Code in our main iptables rules will then tag these packets so they enter high-priority traffic class 1:10.

IPTables U32 Match Tutorial

emaknyus@gmail.com (tamvan & verani) — Thu, 8 Sep 2011 16:46:00 -0700

Introduction

IPTables has always been a relatively flexible and modular firewall; if it can't currently test for a particular packet characteristic, you have the option of writing a test or modifying an existing test. The catch - as with so much of open source - is that most of us aren't programmers.
It turns out we do have an option that doesn't require programming. Don Cohen was kind enough to write an IPTables module that pulls any bytes you'd like out of the packet, does some manipulation, and sees if the result is in a particular range. For example, I can grab the Fragmentation information out of the IP header, throw away everything except the More Fragments flag, and see if that flag is set.
Without writing any C code at all. :-)
What I'll do is introduce the core concepts here, and put in hopefully enough annotated examples that you'll be able to write your own tests.
I won't be focusing on what these fields are, or why you'd want to test them; there are lots of (warning - shameless plug for my employer ahead!) resources for doing that. If you simply need a quick reference for the packet headers, see tcpip.pdf.
All byte positions in this article start counting at 0 as the first byte of the header. For example, in the IP header, byte "0" holds the 4 bit "Version" and 4 bit "IP Header Length", byte "1" holds the "TOS" field, etc.

Check the value of a 2 byte field

In it's simplest form, u32 grabs a block of 4 bytes starting at Start, applies a mask of Mask to it, and compares the result to Range. Here's the syntax we'll use for our first examples:


iptables -m u32 --u32 "Start&Mask=Range"

We'll generally pick a "Start" value that's 3 less than the last byte in which you're interested. So, if you want bytes 4 and 5 of the IP header (the IP ID field), Start needs to be 5-3 = 2. Mask strips out all the stuff you don't want; it's a bitmask that can be as large as 0xFFFFFFFF. To get to our target of bytes 4 or 5, we have to discard bytes 2 and 3. Here's the mask we'll use: 0x0000FFFF . We'll actually use the shorter, and equivalent, 0xFFFF instead.
So, to test for IPID's from 2 to 256, the iptables command line is:


iptables -m u32 --u32 "2&0xFFFF=0x2:0x0100"

To read this off from left to right: "Load the u32 module, and perform the following u32 tests on this packet; grab the 4 bytes starting with byte 2 (bytes 2 and 3 are the Total Length field, and bytes 4 and 5 are the IPID), apply a mask of 0x0000FFFF (which sets the first two bytes to all zeroes, leaving the last two bytes untouched), and see if that value - the IPID - falls between 2 and 256 inclusive; if so, return true, otherwise false."
There is no standalone IPID check in IPTables, but this is the equivalent of the "ip[2:2] >= 2 and ip[2:2] <= 256" tcpdump/bpf filter.
I leave off actions in these examples, but you can add things like:


-j LOG --log-prefix "ID-in-2-256 "

-j DROP

or any other action. You can also add other tests, as we'll do in a minute.
Don offers this test to see if the total packet length is greater than or equal to 256. The total length field is bytes 2 and 3 of the IP header, so our starting position is 3-3 = 0. Since we're pulling out two bytes again, the mask will be 0xFFFF here as well. The final test is:


iptables -m u32 --u32 "0&0xFFFF=0x100:0xFFFF"

This is the same as:


iptables -m length --length 256:65535

	or the bpf filter

"len >= 256"

Check the value of a 1 byte field

Much the same, except we'll use a mask of 0x000000FF (or it's shorter equivalent 0xFF) to pull out a single byte from the 4 bytes u32 initially hands us. Let's say I want to test the TTL field for TTL's below 3 to find people tracerouting to us. Yes, there's a ttl module, but let's see how this would be done in u32.
I want to end up with byte 8 of the IP header, so my starting position is 8-3 = 5. Here's the test:


iptables -m u32 --u32 "5&0xFF=0:3"

Which is equivalent to:


iptables -m ttl --ttl-lt 4

	or the bpf filter

"ip[8] <= 3"

Looking at 4 bytes at once

To check a complete destination IP address, we'll inspect bytes 16-19. Because we want all 4 bytes, we don't need a mask at all. Let's see if the destination address is 224.0.0.1:


iptables -m u32 --u32 "16=0xE0000001"

This is equivalent to:


iptables -d 224.0.0.1/32

If we only want to look at the first three bytes (to check if a source address is part of a given class C network), we'll need to use a mask again. The mask we'll use is 0xFFFFFF00 , which throws away the last octet. Let's check if the source address (from bytes 12-15, although we'll ignore byte 15 with the mask) is in the class C network 192.168.15.0 (0xC0A80F00):


iptables -m u32 --u32 "12&0xFFFFFF00=0xC0A80F00"

Which is the same as:


iptables -s 192.168.15.0/24

Inspecting early bytes in the header

Obviously, if I want to look at the TOS field (byte 1 of the IP header), I can't start at byte 1-3 = -2. What we'll do instead is start at byte 0, pull out the byte we want, and then move it down to the last position for easy testing. This isn't the only way we could do this, but it helps demonstrate a technique we'll need in a minute.
To pull out the TOS field, I first ask u32 to give me bytes 0-3 by using an offset of 0. Now, I pull out byte 1 (the second byte in that block) with a mask of 0x00FF0000 . I need to shift the TOS value down to the far right position for easy comparison. To do this, I use a technique called, unsuprisingly, "right shift". The symbol for right shift is ">>"; this is followed by the number of bits right to move the data. If you're unfamiliar with right shift, take a look at this tutorial from Harper College.
I want to move TOS two bytes - or 16 bits - to the right. This is done with ">>16". Now that we have TOS in the correct position, we compare it to 0x08 (Maximize Throughput):


iptables -m u32 --u32 "0&0x00FF0000>>16=0x08"

which is the equivalent of:


iptables -m ttl --tos 8

Inspecting individual bits

I'd like to look at the "More Fragments" flag - a flag which has no existing test in iptables (-f matches 2nd and further fragments, I want to match all fragments except the last). Byte 6 contains this, so I'll start with offset 3 and throw away bytes 3-5. Normally this would use a mask of 0x000000FF, but I also want to discard the other bits in that last byte. The only bit I want to keep is the third from the top (0010 0000), so the mask I'll use is 0x00000020 . Now I have two choices; move that bit down to the lowest position and compare, or leave it in its current position and compare.
To move it down, we'll right shift 5 bits. The final test is:


iptables -m u32 --u32 "3&0x20>>5=1"

If I take the other approach of leaving the bit where it is, I need to be careful about the compare value on the right. If that bit is turned on, the compare value needs to be 0x20 as well.


iptables -m u32 --u32 "3&0x20=0x20"

Both approaches return true if the More Fragments flag is turned on.

Combining tests

If you want to inspect more than one aspect of a packet, use:

&&

between each test.

Moving on to the TCP header

This is a little tricky. Let's say I'd like to look at bytes 4-7 of the TCP header (the TCP sequence number). Let's take the simple approach first, and then look at some ways to improve this.
For our first version, let's assume that the IP header is 20 bytes long - usually a good guess. Our starting point is byte 4 of the tcp header that immediately follows the IP header. Our simplistic test for whether the sequence number is 41 (hex 29) might look like this:


iptables -m u32 --u32 "24=0x29"

For packets where the IP header length is 20, this will actually work, but there are a few problems. Let's fix them one by one.
First, we never check to see if the packet is even a TCP packet. This is stored in byte 9 of the IP header, so we'll pull 4 bytes starting at byte 6, drop 6-8, and check to see if it's 6. The new rule that first checks if this is a TCP packet at all and also checks that the Sequence Number is 41 is:


iptables -m u32 --u32 "6&0xFF=0x6 && 24=0x29"

The second problem we've momentarily ignored is the IP header length. True, it usually is 20 bytes long, but it can be longer, if IP options are used.
Here are the steps. We pull the IP header length (a nibble that shows how many 4 bytes words there are in the header, usually 5) out of the IP header. We multiply it by 4 to get the number of bytes in the IP header. We use this number to say how many bytes to jump to get to the beginning of the TCP header, and jump 4 more bytes to get to the Sequence number.
To get the header length, we need the first byte: "0>>24", but we need to only grab the lower nibble and we need to multiply that number by 4 to get the actual number of bytes in the header. To do the multiply, we'll right shift 22 instead of 24. With this shift, we'll need to use a mask of 0x3C instead of the 0x0F we would have used. The expression so far is: "0>>22&0x3C". On an IP header with no options, that expression returns 20; just what we'd expect. Now we need to tell u32 to use that number and make a jump that many bytes into the packet, a step performed by the "@" operator.


iptables -m u32 --u32 "6&0xFF=0x6 && 0>>22&0x3C@4=0x29"

The "@" grabs the number we created on its left (20, normally) and jumps that many bytes forward (we can even do this more than once - see the TCP payload section below). The 4 to its right tells u32 to grab bytes 4-7, but u32 knows to pull them relative to the 20 bytes it skipped over. This gives us the Sequence Number, even if the IP header grows because of options. *phew*!
The last quirk to handle is fragments. When we were only working with the IP header, this wasn't an issue; IP is designed in such a way that the IP header itself can never be fragmented. The TCP header and application payload technically might be, and if we're handed the second or further fragment, we might be looking not at the Sequence Number in bytes 4-7, but perhaps some other part of the TCP header, or more likely, some application layer data.
What we'll do is check that this is the first fragment (or an unfragmented packet, the test won't care), so that we're sure we're looking at tcp header info. To do this, we test the fragment offset in most (we discard the top three flag bits) of bytes 6 and 7 of the IP header to make sure the offset is 0. The test is: "4&0x1FFF=0"
The final expression (check for TCP, check for unfragmented packet or first fragment, and jump over the IP header, checking that bytes 4-7 of the TCP header are equal to 41) is:


iptables -m u32 --u32 "6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C@4=0x29"

If the packet is, in fact, fragmented, we have one more consideration; the fragment might be so small that the field we're testing might have been put in a future fragment! In this one case, it's not an issue because every IP link should handle packets of at least 68 bytes; even if the IP header was at its maximum of 60 bytes, the first 8 bytes of the TCP header should be included in that first fragment.
When we start testing for things further in to the packet, we'll have to depend on u32's ability to simply return false if we ever try to ask for a value that falls outside of the packet being inspected.

Checking for values in the ICMP header

Let's look for ICMP Host Unreachables (ICMP, type 3, code 1). Just as in the above example, we need to check for the Protocol field (Protocol 1 = ICMP this time) and that we're looking at a complete packet or at least the first fragment:

"6&0xFF=1 &&
4&0x1FFF=0"

To check for the ICMP Type and Code, we skip over the IP header again ("0>>22&0x3C@..."). To grab the first two bytes, we'll start at offset 0 and just right shift 16 bits. The final test is:


iptables -m u32 --u32 "6&0xFF=1 && 4&0x1FFF=0 && 0>>22&0x3C@0>>16=0x0301"

Checking for values in the UDP payload

Lets try going all the way into the packet payload now, and match packets that are UDP DNS queries. Here we're not only going to check for destination port 53, but we're also going to test the top bit of byte 2 of the payload; if set, this is a DNS query.
We start by checking that this is a UDP packet: "6&0xFF=17". We add the now familiar check for first fragment: "4&0x1FFF=0".
To test the destination port, we grab bytes 2 and 3 from the udp header (after jumping over the IP header as in the previous examples): "0>>22&0x3C@0&0xFFFF=53".
If the packet has passed all of the above, we go back to check the payload (remember we have to jump over the variable-length IP and 8 byte UDP headers "0>>22&0x3C@8 ...") to make sure this is a DNS query rather than a response. To grab the high bit from byte 2, I'll use offset 8 to grab the first 4 payload bytes, right shift 15 bits to deposit the Query bit in the lowest position, and throw away all the rest of the bits with a mask of 0x01: "0>>22&0x3C@8>>15&0x01=1"
The final test is:


iptables -m u32 --u32 "6&0xFF=17 && 4&0x1FFF=0 
&& 0>>22&0x3C@0&0xFFFF=53 && 
0>>22&0x3C@8>>15&0x01=1"

Ugh. I've seen stellar noise that had less entropy :-) Note that we're doing the whole thing with u32 checks; we could pull out the "udp", "first/no fragment" and "port 53" checks into other modules, and end up with this slightly more readable version:


iptables -p udp --dport 53 \! -f -m u32 --u32 "0>>22&0x3C@8>>15&0x01=1"

Checking for values in the TCP payload

As with the udp payload example above, this approach is only useful if we're absolutely certain of where our data of interest can be found. In this example, I want to find ssh sessions, even if they're on a port _other_ than 22. But wait - ssh connections are encrypted! Sounds hopeless? Actually, it isn't.
The first thing to happen in an ssh connection is that the server sends a protocol string back to the client. The protocol looks like:


SSH-protoversion-softwareversion comments

The protoversion is generally 1.99, 2.0, or 1.5 . The entire line, including a possible trailing carriage return and linefeed, needs to be smaller than 255 bytes. This line _may_ have another line before it, but this would break compatibility with ssh 1.0 clients, so I'm guessing this is uncommon. The above information comes from draft-ietf-secsh-transport-17.txt , if you want to see more details.
The interesting piece, as far as we're concerned, is that "SSH-" string. It lands in the first 4 bytes of the connection. Bingo! We can use the u32 module to look at only those bytes in the early part of a connection; this is a much less intensive check than checking with the string module, and makes it feasible to look for ssh connections on any port without undue load on the firewall.
Lets get the easy stuff out of the way. I want to check tcp packets, unfragmented ones, I'm only interested in packets in the first 255 bytes of a connection, and established ones, obviously, and the entire packet length needs to be between 45 and 375 bytes (think about the minimums and maximums for IP and tcp headers and ssh protocol string):


iptables -p tcp \! -f -m connbytes --connbytes 0:255 -m state --state ESTABLISHED -m length --length 46:375

With these restrictions, we should be inspecting very few packets; this allows us to look at the first few packets of every connection for legitimate or rogue ssh connections.
Now u32 does its magic. We'll build it up from scratch. Because we've already checked for protocol TCP, we'll skip 6&0xFF=0x6. We skip over the IP header with 0>>22&0x3C@:


0>>22&0x3C@

We now need to skip over the tcp header; we pull the TCP header length from the first half of byte 12. Although we'd need to right shift it 28 bits to get it into the last 4 bits of our 32 bit buffer, we have to multiply this by 4, too, to get the number of 4-byte words this number expresses; this is why we right shift by 26 and mask again with 0x3C:


0>>22&0x3C@ 12>>26&0x3C@

This slides us into the tcp payload portion of the packet. As the bytes we want to test should generally be in the first 4 bytes of the packet, we just grab 4 bytes starting at 0 and compare them to 0x5353482D (the hex equivalent of "SSH-"):


0>>22&0x3C@ 12>>26&0x3C@ 0=0x5353482D

Here's the entire test, line wrapped to save screen space:


iptables -p tcp \! -f -m connbytes --connbytes 0:255 -m state --state ESTABLISHED
 -m length --length 46:375 -m u32 --u32 "0>>22&0x3C@ 12>>26&0x3C@ 0=0x5353482D"

Once I've done this, I can even check for specific SSH protocol versions with -m string --string "SSH-1.99" or even as specific as

-m string --string
"SSH-1.99-OpenSSH_3.7.1p2"

. This string match - normally very expensive in terms of processor time per packet - isn't bad at all if we've already figured out with the previous iptables command that this is almost certainly an ssh protocol string. By essentially requiring "SSH-" in the first 4 bytes of the connection, we avoid the problem of the string match incorrectly matching "SSH-" when someone downloads this article over port 80.

Wrapup

With homage to Douglas Adams, Don't Panic.
The examples we've worked through have been annoyingly complex. The nice part? You don't have to rewrite them every time you want to inspect something else in the icmp header, or tcp payload, or anything else. Just grab the appropriate tests from above - or the table below - and finish them off with the specific field you wanted to inspect.

Tests

First, a recap of the above, then some additional tests.

"2&0xFFFF=0x2:0x0100": Test for IPID's between 2 and 256
"0&0xFFFF=0x100:0xFFFF": Check for packets with 256 or more bytes.
"5&0xFF=0:3": Match packets with a TTL of 3 or less.
"16=0xE0000001": Destination IP address is 224.0.0.1
"12&0xFFFFFF00=0xC0A80F00": Source IP is in the 192.168.15.X class C network.
0&0x00FF0000>>16=0x08: Is the TOS field 8 (Maximize Throughput)?
"3&0x20>>5=1": Is the More Fragments flag set?
"6&0xFF=0x6": Is the packet a TCP packet?
"4&0x1FFF=0": Is the fragment offset 0? (If so, this is either an unfragmented packet or the first fragment).
"0>>22&0x3C@4=0x29": Is the TCP Sequence number 41? (This requires the previous two checks for TCP and First Fragment as well)
"0>>22&0x3C@0>>16=0x0301": Check for ICMP type=3 and code=1 (needs UDP and first fragment tests too)
"0>>22&0x3C@0&0xFFFF=53": Is the UDP destination port 53? (Check for udp and first/no fragment first)
"0>>22&0x3C@8>>15&0x01=1": Check that the UDP DNS Query bit is set (again, check for UDP, first/no fragment, and dest port 53 first).
"0>>22&0x3C@ 12>>26&0x3C@ 0=0x5353482D": Are the first 4 bytes of the TCP payload "SSH-"? (this requires a number of additional checks; the entire line is: iptables -p tcp \! -f -m connbytes --connbytes 0:255 -m state --state ESTABLISHED -m length --length 46:375 -m u32 --u32 "0>>22&0x3C@ 12>>26&0x3C@ 0=0x5353482D" And now, some new tests:
"6&0xFF=1": Is this an ICMP packet? (From Don Cohen's documentation)
"6&0xFF=17": Is this a UDP packet?
"4&0x3FFF=0": Is the fragment offset 0 and MF cleared? (If so, this is an unfragmented packet).
"4&0x3FFF=1:0x3FFF": Is the fragment offset greater than 0 or MF set? (If so, this is a fragment).
0>>22&0x3C@12>>26&0x3C@-3&0xFF=0:255: Is there any payload on this tcp packet (check for tcp and not fragmented first)? This elegant test was contributed by Don Cohen as I fumbled for a way to look for payload on a syn packet. By simply testing to see if payload byte 0 has a value between 0 and 255, we get true if payload byte 0 exists (read: "if there is any payload at all"), and false if we've gone beyond the end of the packet (read: "if there is no payload").
"0>>22&0x3C@8=3000": Is the TCP ack field 3000? (This, too, needs the TCP and First fragment checks)
"0>>22&0x3C@10&0x80=0x80": Is the high order bit (CWR) of the TCP flag byte (13) turned on? (Needs TCP and first fragment checks too). To see if it's off, use =0 at the end instead of =0x80.
"0>>22&0x3C@10&0x40=0x40": Is the next bit down (ECN-Echo) turned on? Likewise, =0 to see if it's off.
"3&0xE0=0x20": Is the More Fragments bit set, and are the Reserved and Don't fragment flags cleared?
"3&0x20=0x20": This also checks if MF is set, but doesn't care about Reserved or Don't fragment.
"3&0xE0=0x60": Are More Fragments and Don't fragment set, and is Reserved cleared?
"3&0xE0=0x80": Is reserved set, and are More Fragments and Don't fragment cleared?
"0>>22&0x3C@2&0xFFFF=0": Is the ICMP ID (inside an echo request or reply packet) 0? This assumes you've already checked that the packet is icmp, unfragmented, and an echo request or reply.
"0>>22&0x3C@4&0xFFFF=35": Is the ICMP echo Sequence number 35?

Don Cohen wrote the u32 module, and also wrote some (if you'll forgive me) somewhat cryptic documentation inside the source code for the module. William Stearns wrote this text, which borrows some examples and concepts from Don's documentation. Many thanks to Don for reviewing an early draft of this article. Thanks also to Gary Kessler and Sans for making the TCP/IP pocket reference guide freely available.

http://www.stearns.org/doc/iptables-u32.v0.1.7.html

unix toolbox

emaknyus@gmail.com (tamvan & verani) — Thu, 8 Sep 2011 16:15:00 -0700

This document is a collection of Unix/Linux/BSD commands and tasks which are useful for IT work or for advanced users. This is a practical guide with concise explanations, however the reader is supposed to know what s/he is doing.

Unix Toolbox revision 14.3

The latest version of this document can be found at http://cb.vu/unixtoolbox.xhtml. Replace .xhtml on the link with .pdf for the PDF version and with .book.pdf for the booklet version. On a duplex printer the booklet will create a small book ready to bind. This XHTML page can be converted into a nice PDF document with a CSS3 compliant application (see the script example). See also the about page.

Error reports and comments are most welcome - c@cb.vu Colin Barschel.

© Colin Barschel 2007-2011. Some rights reserved under Creative Commons.

   1.   System

   2.   Processes

   3.   File System

   4.   Network

   5.   SSH SCP

   6.   VPN with SSH

   7.   RSYNC

   8.   SUDO

   9.   Encrypt Files

10.   Encrypt Partitions

11.   SSL Certificates

12.   CVS

13.   SVN

14.   Useful Commands

15.   Install Software

16.   Convert Media

17.   Printing

18.   Databases

19.   Disk Quota

20.   Shells

21.   Scripting

22.   Programming

23.   Online Help

white

black

System

Hardware | Statistics | Users | Limits | Runlevels | root password | Compile kernel | Repair grub

Running kernel and system information

# uname -a                           # Get the kernel version (and BSD version)
# lsb_release -a                     # Full release info of any LSB distribution
# cat /etc/SuSE-release              # Get SuSE version
# cat /etc/debian_version            # Get Debian version

Use /etc/DISTR-release with DISTR= lsb (Ubuntu), redhat, gentoo, mandrake, sun (Solaris), and so on. See also /etc/issue.

# uptime                             # Show how long the system has been running + load
# hostname                           # system's host name
# hostname -i                        # Display the IP address of the host. (Linux only)
# man hier                           # Description of the file system hierarchy
# last reboot                        # Show system reboot history

Hardware Informations

Kernel detected hardware

# dmesg                              # Detected hardware and boot messages
# lsdev                              # information about installed hardware
# dd if=/dev/mem bs=1k skip=768 count=256 2>/dev/null | strings -n 8 # Read BIOS

Linux

# cat /proc/cpuinfo                  # CPU model
# cat /proc/meminfo                  # Hardware memory
# grep MemTotal /proc/meminfo        # Display the physical memory
# watch -n1 'cat /proc/interrupts'   # Watch changeable interrupts continuously
# free -m                            # Used and free memory (-m for MB)
# cat /proc/devices                  # Configured devices
# lspci -tv                          # Show PCI devices
# lsusb -tv                          # Show USB devices
# lshal                              # Show a list of all devices with their properties
# dmidecode                          # Show DMI/SMBIOS: hw info from the BIOS

FreeBSD

# sysctl hw.model                    # CPU model
# sysctl hw                          # Gives a lot of hardware information
# sysctl hw.ncpu                     # number of active CPUs installed
# sysctl vm                          # Memory usage
# sysctl hw.realmem                  # Hardware memory
# sysctl -a | grep mem               # Kernel memory settings and info
# sysctl dev                         # Configured devices
# pciconf -l -cv                     # Show PCI devices
# usbdevs -v                         # Show USB devices
# atacontrol list                    # Show ATA devices
# camcontrol devlist -v              # Show SCSI devices

Load, statistics and messages

The following commands are useful to find out what is going on on the system.

# top                                # display and update the top cpu processes
# mpstat 1                           # display processors related statistics
# vmstat 2                           # display virtual memory statistics
# iostat 2                           # display I/O statistics (2 s intervals)
# systat -vmstat 1                   # BSD summary of system statistics (1 s intervals)
# systat -tcp 1                      # BSD tcp connections (try also -ip)
# systat -netstat 1                  # BSD active network connections
# systat -ifstat 1                   # BSD network traffic through active interfaces
# systat -iostat 1                   # BSD CPU and and disk throughput
# tail -n 500 /var/log/messages      # Last 500 kernel/syslog messages
# tail /var/log/warn                 # System warnings messages see syslog.conf

Users

# id                                 # Show the active user id with login and group
# last                               # Show last logins on the system
# who                                # Show who is logged on the system
# groupadd admin                     # Add group "admin" and user colin (Linux/Solaris)
# useradd -c "Colin Barschel" -g admin -m colin
# usermod -a -G <group> <user>       # Add existing user to group (Debian)
# groupmod -A <user> <group>         # Add existing user to group (SuSE)
# userdel colin                      # Delete user colin (Linux/Solaris)
# adduser joe                        # FreeBSD add user joe (interactive)
# rmuser joe                         # FreeBSD delete user joe (interactive)
# pw groupadd admin                  # Use pw on FreeBSD
# pw groupmod admin -m newmember     # Add a new member to a group
# pw useradd colin -c "Colin Barschel" -g admin -m -s /bin/tcsh
# pw userdel colin; pw groupdel admin

Encrypted passwords are stored in /etc/shadow for Linux and Solaris and /etc/master.passwd on FreeBSD. If the master.passwd is modified manually (say to delete a password), run # pwd_mkdb -p master.passwd to rebuild the database.

To temporarily prevent logins system wide (for all users but root) use nologin. The message in nologin will be displayed (might not work with ssh pre-shared keys).

# echo "Sorry no login now" > /etc/nologin       # (Linux)
# echo "Sorry no login now" > /var/run/nologin   # (FreeBSD)

Limits

Some application require higher limits on open files and sockets (like a proxy
web server, database). The default limits are usually too low.
Linux

Per shell/script

The shell limits are governed by ulimit. The status is checked
with ulimit -a. For example to change the open files limit from
1024 to 10240 do:

# ulimit -n 10240                    # This is only valid within the shell

The ulimit command can be used in a script to change the limits for the script only.

Per user/process

Login users and applications can be configured in /etc/security/limits.conf. For example:

# cat /etc/security/limits.conf
*   hard    nproc   250              # Limit user processes
asterisk hard nofile 409600          # Limit application open files

System wide

Kernel limits are set with sysctl. Permanent limits are set in /etc/sysctl.conf.

# sysctl -a                          # View all system limits
# sysctl fs.file-max                 # View max open files limit
# sysctl fs.file-max=102400          # Change max open files limit
# echo "1024 50000" > /proc/sys/net/ipv4/ip_local_port_range # port range
# cat /etc/sysctl.conf
fs.file-max=102400                   # Permanent entry in sysctl.conf
# cat /proc/sys/fs/file-nr           # How many file descriptors are in use

FreeBSD

Per shell/script

Use the command limits in csh or tcsh or as in Linux, use ulimit in an sh or bash shell.
Per user/process

The default limits on login are set in /etc/login.conf. An unlimited value is still limited by the system maximal value.
System wide

Kernel limits are also set with sysctl. Permanent limits are set in /etc/sysctl.conf or /boot/loader.conf. The syntax is the same as Linux but the keys are different.

# sysctl -a                          # View all system limits
# sysctl kern.maxfiles=XXXX          # maximum number of file descriptors
kern.ipc.nmbclusters=32768           # Permanent entry in /etc/sysctl.conf
kern.maxfiles=65536                  # Typical values for Squid
kern.maxfilesperproc=32768
kern.ipc.somaxconn=8192              # TCP queue. Better for apache/sendmail
# sysctl kern.openfiles              # How many file descriptors are in use
# sysctl kern.ipc.numopensockets     # How many open sockets are in use
# sysctl net.inet.ip.portrange.last=50000 # Default is 1024-5000
# netstat -m                         # network memory buffers statistics

See The FreeBSD handbook Chapter 11http://www.freebsd.org/handbook/configtuning-kernel-limits.html for details.

Solaris

The following values in /etc/system will increase the maximum file descriptors per proc:

set rlim_fd_max = 4096               # Hard limit on file descriptors for a single proc
set rlim_fd_cur = 1024               # Soft limit on file descriptors for a single proc

Runlevels

Linux

Once booted, the kernel starts init which then starts rc which starts all scripts belonging to a runlevel. The scripts are stored in /etc/init.d and are linked into /etc/rc.d/rcN.d with N the runlevel number.

The default runlevel is configured in /etc/inittab. It is usually 3 or 5:

# grep default: /etc/inittab
id:3:initdefault:

The actual runlevel can be changed with init. For example to go from 3 to 5:

# init 5                             # Enters runlevel 5

    *   0       Shutdown and halt

    *   1       Single-User mode (also S)

    *   2       Multi-user without network

    *   3       Multi-user with network

    *   5       Multi-user with X

    *   6       Reboot

Use chkconfig to configure the programs that will be started at boot in a runlevel.

# chkconfig --list                   # List all init scripts
# chkconfig --list sshd              # Report the status of sshd
# chkconfig sshd --level 35 on       # Configure sshd for levels 3 and 5
# chkconfig sshd off                 # Disable sshd for all runlevels

Debian and Debian based distributions like Ubuntu or Knoppix use the command update-rc.d to manage the runlevels scripts. Default is to start in 2,3,4 and 5 and shutdown in 0,1 and 6.

# update-rc.d sshd defaults          # Activate sshd with the default runlevels
# update-rc.d sshd start 20 2 3 4 5 . stop 20 0 1 6 . # With explicit arguments
# update-rc.d -f sshd remove         # Disable sshd for all runlevels
# shutdown -h now (or # poweroff)    # Shutdown and halt the system

FreeBSD

The BSD boot approach is different from the SysV, there are no runlevels. The final boot state (single user, with or without X) is configured in /etc/ttys. All OS scripts are located in /etc/rc.d/ and in /usr/local/etc/rc.d/ for third-party applications. The activation of the service is configured in /etc/rc.conf and /etc/rc.conf.local. The default behavior is configured in /etc/defaults/rc.conf. The scripts responds at least to start|stop|status.

# /etc/rc.d/sshd status
sshd is running as pid 552.
# shutdown now                       # Go into single-user mode
# exit                               # Go back to multi-user mode
# shutdown -p now                    # Shutdown and halt the system
# shutdown -r now                    # Reboot

The process init can also be used to reach one of the following states level. For example # init 6 for reboot.

    *   0       Halt and turn the power off (signal USR2)

    *   1       Go to single-user mode (signal TERM)

    *   6       Reboot the machine (signal INT)

    *   c       Block further logins (signal TSTP)

    *   q       Rescan the ttys(5) file (signal HUP)

Windows

Start and stop a service with either the service name or "service description" (shown in the Services Control Panel) as follows:

net stop WSearch
net start WSearch                    # start search service
net stop "Windows Search"
net start "Windows Search"           # same as above using descr.

Reset root password

Linux method 1

At the boot loader (lilo or grub), enter the following boot option:

init=/bin/sh

The kernel will mount the root partition and init will start the bourne shell
instead of rc and then a runlevel. Use the command passwd at the prompt to change the password and then reboot. Forget the single user mode as you need the password for that.

If, after booting, the root partition is mounted read only, remount it rw:

# mount -o remount,rw /
# passwd                             # or delete the root password (/etc/shadow)
# sync; mount -o remount,ro /        # sync before to remount read only
# reboot

FreeBSD method 1

On FreeBSD, boot in single user mode, remount / rw and use passwd. You can select the single user mode on the boot menu (option 4) which is displayed for 10 seconds at startup. The single user mode will give you a root shell on the / partition.

# mount -u /; mount -a               # will mount / rw
# passwd
# reboot

Unixes and FreeBSD and Linux method 2

Other Unixes might not let you go away with the simple init trick. The solution is to mount the root partition from an other OS (like a rescue CD) and change the password on the disk.

    *   Boot a live CD or installation CD into a rescue mode which will give you a shell.

    *   Find the root partition with fdisk e.g. fdisk /dev/sda

    *   Mount it and use chroot:

# mount -o rw /dev/ad4s3a /mnt
# chroot /mnt                        # chroot into /mnt
# passwd
# reboot

Kernel modules

Linux

# lsmod                              # List all modules loaded in the kernel
# modprobe isdn                      # To load a module (here isdn)

FreeBSD

# kldstat                            # List all modules loaded in the kernel
# kldload crypto                     # To load a module (here crypto)

Compile Kernel

Linux

# cd /usr/src/linux
# make mrproper                      # Clean everything, including config files
# make oldconfig                     # Reuse the old .config if existent
# make menuconfig                    # or xconfig (Qt) or gconfig (GTK)
# make                               # Create a compressed kernel image
# make modules                       # Compile the modules
# make modules_install               # Install the modules
# make install                       # Install the kernel
# reboot

FreeBSD

Optionally update the source tree (in /usr/src) with csup (as of FreeBSD 6.2 or later):

# csup <supfile>

I use the following supfile:

*default host=cvsup5.FreeBSD.org # www.freebsd.org/handbook/cvsup.html#CVSUP-MIRRORS
*default prefix=/usr
*default base=/var/db
*default release=cvs delete tag=RELENG_7
src-all

To modify and rebuild the kernel, copy the generic configuration file to a new name and edit it as needed (you can also edit the file GENERIC directly). To restart the build after an interruption, add the option NO_CLEAN=YES to the make command to avoid cleaning the objects already build.

# cd /usr/src/sys/i386/conf/
# cp GENERIC MYKERNEL
# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL
# make installkernel KERNCONF=MYKERNEL

To rebuild the full OS:

# make buildworld                    # Build the full OS but not the kernel
# make buildkernel                   # Use KERNCONF as above if appropriate
# make installkernel
# reboot
# mergemaster -p                     # Compares only files known to be essential
# make installworld
# mergemaster -i -U                  # Update all configurations and other files
# reboot

For small changes in the source you can use NO_CLEAN=yes to avoid rebuilding the whole tree.

# make buildworld NO_CLEAN=yes       # Don't delete the old objects
# make buildkernel KERNCONF=MYKERNEL NO_CLEAN=yes

Repair grub

So you broke grub? Boot from a live cd, [find your linux partition under /dev and use fdisk to find the linux partion] mount the linux partition, add /proc and /dev and use grub-install /dev/xyz. Suppose linux lies on /dev/sda6:

# mount /dev/sda6 /mnt               # mount the linux partition on /mnt
# mount --bind /proc /mnt/proc       # mount the proc subsystem into /mnt
# mount --bind /dev /mnt/dev         # mount the devices into /mnt
# chroot /mnt                        # change root to the linux partition
# grub-install /dev/sda              # reinstall grub with your old settings

Processes

Listing | Priority | Background/Foreground | Top | Kill

Listing and PIDs

Each process has a unique number, the PID. A list of all running process is retrieved with ps.

# ps -auxefw                         # Extensive list of all running process

However more typical usage is with a pipe or with pgrep (for OS X install proctools from MacPorts):

# ps axww | grep cron
586 ?? Is     0:01.48 /usr/sbin/cron -s
# ps axjf                            # All processes in a tree format (Linux)
# ps aux | grep 'ss[h]'              # Find all ssh pids without the grep pid
# pgrep -l sshd                      # Find the PIDs of processes by (part of) name
# echo $$                            # The PID of your shell
# fuser -va 22/tcp                   # List processes using port 22 (Linux)
# pmap PID                           # Memory map of process (hunt memory leaks) (Linux)
# fuser -va /home                    # List processes accessing the /home partition
# strace df                          # Trace system calls and signals
# truss df                           # same as above on FreeBSD/Solaris/Unixware

Priority

Change the priority of a running process with renice. Negative numbers have a higher priority, the lowest is -20 and "nice" have a positive value.

# renice -5 586                      # Stronger priority
586: old priority 0, new priority -5

Start the process with a defined priority with nice. Positive is "nice" or weak, negative is strong scheduling priority. Make sure you know if /usr/bin/nice or the shell built-in is used (check with # which nice).

# nice -n -5 top                     # Stronger priority (/usr/bin/nice)
# nice -n 5 top                      # Weaker priority (/usr/bin/nice)
# nice +5 top                        # tcsh builtin nice (same as above!)

While nice changes the CPU scheduler, an other useful command ionice will schedule the disk IO. This is very useful for intensive IO application (e.g. compiling). You can select a class (idle - best effort - real time), the man page is short and well explained.

# ionice c3 -p123                    # set idle class for pid 123 (Linux only)
# ionice -c2 -n0 firefox             # Run firefox with best effort and high priority
# ionice -c3 -p$$                    # Set the actual shell to idle priority

The last command is very useful to compile (or debug) a large project. Every command launched from this shell will have a lover priority. $$ is your shell pid (try echo $$).

FreeBSD uses idprio/rtprio (0 = max priority, 31 = most idle):

# idprio 31 make                     # compile in the lowest priority
# idprio 31 -1234                    # set PID 1234 with lowest priority
# idprio -t -1234                    # -t removes any real time/idle priority

Background/Foreground

When started from a shell, processes can be brought in the background and back to the foreground with [Ctrl]-[Z] (^Z), bg and fg. List the processes with jobs.

# ping cb.vu > ping.log
^Z                                   # ping is suspended (stopped) with [Ctrl]-[Z]
# bg                                 # put in background and continues running
# jobs -l                            # List processes in background
[1] - 36232 Running                       ping cb.vu > ping.log
[2] + 36233 Suspended (tty output)        top
# fg %2                              # Bring process 2 back in foreground

Use nohup to start a process which has to keep running when the shell is closed (immune to hangups).

# nohup ping -i 60 > ping.log &

Top

The program top displays running information of processes. See also the program htop from htop.sourceforge.net (a more powerful version of top) which runs on Linux and FreeBSD (ports/sysutils/htop/). While top is running press the key h for a help overview. Useful keys are:

    *   u [user name] To display only the processes belonging to the user. Use + or blank to see all users

    *   k [pid] Kill the process with pid.

    *   1 To display all processors statistics (Linux only)

    *    R Toggle normal/reverse sort.

Signals/Kill

Terminate or send a signal with kill or killall.

# ping -i 60 cb.vu > ping.log &
[1] 4712
# kill -s TERM 4712                  # same as kill -15 4712
# killall -1 httpd                   # Kill HUP processes by exact name
# pkill -9 http                      # Kill TERM processes by (part of) name
# pkill -TERM -u www                 # Kill TERM processes owned by www
# fuser -k -TERM -m /home            # Kill every process accessing /home (to umount)

Important signals are:

    *   1       HUP (hang up)

    *   2       INT (interrupt)

    *   3       QUIT (quit)

    *   9       KILL (non-catchable, non-ignorable kill)

    *   15     TERM (software termination signal)

File System

Disk info | Boot | Disk usage | Opened files | Mount/remount | Mount SMB | Mount image | Burn ISO | Create image | Memory disk | Disk performance

Permissions

Change permission and ownership with chmod and chown. The default umask can be changed for all users in /etc/profile for Linux or /etc/login.conf for FreeBSD. The default umask is usually 022. The umask is subtracted from 777, thus umask 022 results in a permission 0f 755.

1 --x execute                        # Mode 764 = exec/read/write | read/write | read
2 -w- write                          # For:       |-- Owner --|   |- Group-|   |Oth|
4 r-- read
ugo=a                              u=user, g=group, o=others, a=everyone

# chmod [OPTION] MODE[,MODE] FILE    # MODE is of the form [ugoa]*([-+=]([rwxXst]))
# chmod 640 /var/log/maillog         # Restrict the log -rw-r-----
# chmod u=rw,g=r,o= /var/log/maillog # Same as above
# chmod -R o-r /home/*               # Recursive remove other readable for all users
# chmod u+s /path/to/prog            # Set SUID bit on executable (know what you do!)
# find / -perm -u+s -print           # Find all programs with the SUID bit
# chown user:group /path/to/file     # Change the user and group ownership of a file
# chgrp group /path/to/file          # Change the group ownership of a file
# chmod 640 `find ./ -type f -print` # Change permissions to 640 for all files
# chmod 751 `find ./ -type d -print` # Change permissions to 751 for all directories

Disk information

# diskinfo -v /dev/ad2               # information about disk (sector/size) FreeBSD
# hdparm -I /dev/sda                 # information about the IDE/ATA disk (Linux)
# fdisk /dev/ad2                     # Display and manipulate the partition table
# smartctl -a /dev/ad2               # Display the disk SMART info

Boot

FreeBSD

To boot an old kernel if the new kernel doesn't boot, stop the boot at during the count down.

# unload
# load kernel.old
# boot

System mount points/Disk usage

# mount | column -t                  # Show mounted file-systems on the system
# df                                 # display free disk space and mounted devices
# cat /proc/partitions               # Show all registered partitions (Linux)

Disk usage

# du -sh *                           # Directory sizes as listing
# du -csh                            # Total directory size of the current directory
# du -ks * | sort -n -r              # Sort everything by size in kilobytes
# ls -lSr                            # Show files, biggest last

Who has which files opened

This is useful to find out which file is blocking a partition which has to be unmounted and gives a typical error of:

# umount /home/
umount: unmount of /home             # umount impossible because a file is locking home
   failed: Device busy

FreeBSD and most Unixes

# fstat -f /home                     # for a mount point
# fstat -p PID                       # for an application with PID
# fstat -u user                      # for a user name

Find opened log file (or other opened files), say for Xorg:

# ps ax | grep Xorg | awk '{print $1}'
1252
# fstat -p 1252
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     Xorg        1252 root /             2 drwxr-xr-x     512 r
root     Xorg        1252 text /usr     216016 -rws--x--x 1679848 r
root     Xorg        1252    0 /var     212042 -rw-r--r--   56987 w

The file with inum 212042 is the only file in /var:

# find -x /var -inum 212042
/var/log/Xorg.0.log

Linux

Find opened files on a mount point with fuser or lsof:

# fuser -m /home                     # List processes accessing /home
# lsof /home
COMMAND   PID    USER   FD   TYPE DEVICE    SIZE     NODE NAME
tcsh    29029 eedcoba cwd    DIR   0,18   12288 1048587 /home/eedcoba (guam:/home)
lsof    29140 eedcoba cwd    DIR   0,18   12288 1048587 /home/eedcoba (guam:/home)

About an application:

ps ax | grep Xorg | awk '{print $1}'
3324
# lsof -p 3324
COMMAND   PID    USER   FD   TYPE DEVICE    SIZE    NODE NAME
Xorg    3324 root    0w   REG        8,6   56296      12492 /var/log/Xorg.0.log

About a single file:

# lsof /var/log/Xorg.0.log
COMMAND PID USER   FD   TYPE DEVICE SIZE NODE NAME
Xorg    3324 root    0w   REG    8,6 56296 12492 /var/log/Xorg.0.log

Mount/remount a file system

For example the cdrom. If listed in /etc/fstab:

# mount /cdrom

Or find the device in /dev/ or with dmesg
FreeBSD

# mount -v -t cd9660 /dev/cd0c /mnt # cdrom
# mount_cd9660 /dev/wcd0c /cdrom     # other method
# mount -v -t msdos /dev/fd0c /mnt   # floppy

Entry in /etc/fstab:

# Device                Mountpoint      FStype Options         Dump    Pass#
/dev/acd0               /cdrom          cd9660 ro,noauto       0       0

To let users do it:

# sysctl vfs.usermount=1 # Or insert the line "vfs.usermount=1" in /etc/sysctl.conf

Linux

# mount -t auto /dev/cdrom /mnt/cdrom   # typical cdrom mount command
# mount /dev/hdc -t iso9660 -r /cdrom   # typical IDE
# mount /dev/scd0 -t iso9660 -r /cdrom # typical SCSI cdrom
# mount /dev/sdc0 -t ntfs-3g /windows   # typical SCSI

Entry in /etc/fstab:

/dev/cdrom   /media/cdrom subfs noauto,fs=cdfss,ro,procuid,nosuid,nodev,exec 0 0

Mount a FreeBSD partition with Linux

Find the partition number containing with fdisk, this is usually the root partition, but it could be an other BSD slice too. If the FreeBSD has many slices, they are the one not listed in the fdisk table, but visible in /dev/sda* or /dev/hda*.

# fdisk /dev/sda                     # Find the FreeBSD partition
/dev/sda3   *        5357        7905    20474842+ a5 FreeBSD
# mount -t ufs -o ufstype=ufs2,ro /dev/sda3 /mnt
/dev/sda10 = /tmp; /dev/sda11 /usr   # The other slices

Remount

Remount a device without unmounting it. Necessary for fsck for example

# mount -o remount,ro /              # Linux
# mount -o ro /                      # FreeBSD

Copy the raw data from a cdrom into an iso image:

# dd if=/dev/cd0c of=file.iso

Add swap on-the-fly

Suppose you need more swap (right now), say a 2GB file /swap2gb (Linux only).

# dd if=/dev/zero of=/swap2gb bs=1024k count=2000
# mkswap /swap2gb                    # create the swap area
# swapon /swap2gb                    # activate the swap. It now in use
# swapoff /swap2gb                   # when done deactivate the swap
# rm /swap2gb

Mount an SMB share

Suppose we want to access the SMB share myshare on the computer smbserver, the address as typed on a Windows PC is \\smbserver\myshare\. We mount on /mnt/smbshare. Warning> cifs wants an IP or DNS name, not a Windows name.
Linux

# smbclient -U user -I 192.168.16.229 -L //smbshare/    # List the shares
# mount -t smbfs -o username=winuser //smbserver/myshare /mnt/smbshare
# mount -t cifs -o username=winuser,password=winpwd //192.168.16.229/myshare /mnt/share

Additionally with the package mount.cifs it is possible to store the credentials in a file, for example /home/user/.smb:

username=winuser
password=winpwd

And mount as follow:

# mount -t cifs -o credentials=/home/user/.smb //192.168.16.229/myshare /mnt/smbshare

FreeBSD

Use -I to give the IP (or DNS name); smbserver is the Windows name.

# smbutil view -I 192.168.16.229 //winuser@smbserver    # List the shares
# mount_smbfs -I 192.168.16.229 //winuser@smbserver/myshare /mnt/smbshare

Mount an image

# hdiutil mount image.iso                               # OS X

Linux loop-back

# mount -t iso9660 -o loop file.iso /mnt                # Mount a CD image
# mount -t ext3 -o loop file.img /mnt                   # Mount an image with ext3 fs

FreeBSD

With memory device (do # kldload md.ko if necessary):

# mdconfig -a -t vnode -f file.iso -u 0
# mount -t cd9660 /dev/md0 /mnt
# umount /mnt; mdconfig -d -u 0                         # Cleanup the md device

Or with virtual node:

# vnconfig /dev/vn0c file.iso; mount -t cd9660 /dev/vn0c /mnt
# umount /mnt; vnconfig -u /dev/vn0c                    # Cleanup the vn device

Solaris and FreeBSD

with loop-back file interface or lofi:

# lofiadm -a file.iso
# mount -F hsfs -o ro /dev/lofi/1 /mnt
# umount /mnt; lofiadm -d /dev/lofi/1                   # Cleanup the lofi device

Create and burn an ISO image

This will copy the cd or DVD sector for sector. Without conv=notrunc, the image will be smaller if there is less content on the cd. See below and the dd examples.

# dd if=/dev/hdc of=/tmp/mycd.iso bs=2048 conv=notrunc

Use mkisofs to create a CD/DVD image from files in a directory. To overcome the file names restrictions: -r enables the Rock Ridge extensions common to UNIX systems, -J enables Joliet extensions used by Microsoft systems. -L allows ISO9660 filenames to begin with a period.

# mkisofs -J -L -r -V TITLE -o imagefile.iso /path/to/dir
# hdiutil makehybrid -iso -joliet -o dir.iso dir/       # OS X

On FreeBSD, mkisofs is found in the ports in sysutils/cdrtools.
Burn a CD/DVD ISO image

FreeBSD

FreeBSD does not enable DMA on ATAPI drives by default. DMA is enabled with the sysctl command and the arguments below, or with /boot/loader.conf with the following entries:

hw.ata.ata_dma="1"
hw.ata.atapi_dma="1"

Use burncd with an ATAPI device (burncd is part of the base system) and cdrecord (in sysutils/cdrtools) with a SCSI drive.

# burncd -f /dev/acd0 data imagefile.iso fixate      # For ATAPI drive
# cdrecord -scanbus                  # To find the burner device (like 1,0,0)
# cdrecord dev=1,0,0 imagefile.iso

Linux

Also use cdrecord with Linux as described above. Additionally it is possible to use the native ATAPI interface which is found with:

# cdrecord dev=ATAPI -scanbus

And burn the CD/DVD as above.
dvd+rw-tools

The dvd+rw-tools package (FreeBSD: ports/sysutils/dvd+rw-tools) can do it all and includes growisofs to burn CDs or DVDs. The examples refer to the dvd device as /dev/dvd which could be a symlink to /dev/scd0 (typical scsi on Linux) or /dev/cd0 (typical FreeBSD) or /dev/rcd0c (typical NetBSD/OpenBSD character SCSI) or /dev/rdsk/c0t1d0s2 (Solaris example of a character SCSI/ATAPI CD-ROM device). There is a nice documentation with examples on the FreeBSD handbook chapter 18.7http://www.freebsd.org/handbook/creating-dvds.html.

                       # -dvd-compat closes the disk
# growisofs -dvd-compat -Z /dev/dvd=imagefile.iso     # Burn existing iso image
# growisofs -dvd-compat -Z /dev/dvd -J -R /p/to/data # Burn directly

Convert a Nero .nrg file to .iso

Nero simply adds a 300Kb header to a normal iso image. This can be trimmed with dd.

# dd bs=1k if=imagefile.nrg of=imagefile.iso skip=300

Convert a bin/cue image to .iso

The little bchunk programhttp://freshmeat.net/projects/bchunk/ can do this. It is in the FreeBSD ports in sysutils/bchunk.

# bchunk imagefile.bin imagefile.cue imagefile.iso

Create a file based image

For example a partition of 1GB using the file /usr/vdisk.img. Here we use the vnode 0, but it could also be 1.
FreeBSD

# dd if=/dev/random of=/usr/vdisk.img bs=1K count=1M
# mdconfig -a -t vnode -f /usr/vdisk.img -u 0         # Creates device /dev/md1
# bsdlabel -w /dev/md0
# newfs /dev/md0c
# mount /dev/md0c /mnt
# umount /mnt; mdconfig -d -u 0; rm /usr/vdisk.img    # Cleanup the md device

The file based image can be automatically mounted during boot with an entry in /etc/rc.conf and /etc/fstab. Test your setup with # /etc/rc.d/mdconfig start (first delete the md0 device with # mdconfig -d -u 0).

Note however that this automatic setup will only work if the file image is NOT on the root partition. The reason is that the /etc/rc.d/mdconfig script is executed very early during boot and the root partition is still read-only. Images located outside the root partition will be mounted later with the script /etc/rc.d/mdconfig2.

/boot/loader.conf:

md_load="YES"

/etc/rc.conf:

# mdconfig_md0="-t vnode -f /usr/vdisk.img"          # /usr is not on the root partition

/etc/fstab: (The 0 0 at the end is important, it tell fsck to ignore this device, as is does not exist yet)

/dev/md0                /usr/vdisk      ufs     rw              0       0

It is also possible to increase the size of the image afterward, say for example 300 MB larger.

# umount /mnt; mdconfig -d -u 0
# dd if=/dev/zero bs=1m count=300 >> /usr/vdisk.img
# mdconfig -a -t vnode -f /usr/vdisk.img -u 0
# growfs /dev/md0
# mount /dev/md0c /mnt                                # File partition is now 300 MB larger

Linux

# dd if=/dev/zero of=/usr/vdisk.img bs=1024k count=1024
# mkfs.ext3 /usr/vdisk.img
# mount -o loop /usr/vdisk.img /mnt
# umount /mnt; rm /usr/vdisk.img                      # Cleanup

Linux with losetup

/dev/zero is much faster than urandom, but less secure for encryption.

# dd if=/dev/urandom of=/usr/vdisk.img bs=1024k count=1024
# losetup /dev/loop0 /usr/vdisk.img                   # Creates and associates /dev/loop0
# mkfs.ext3 /dev/loop0
# mount /dev/loop0 /mnt
# losetup -a                                          # Check used loops
# umount /mnt
# losetup -d /dev/loop0                               # Detach
# rm /usr/vdisk.img

Create a memory file system

A memory based file system is very fast for heavy IO application. How to create a 64 MB partition mounted on /memdisk:
FreeBSD

# mount_mfs -o rw -s 64M md /memdisk
# umount /memdisk; mdconfig -d -u 0                   # Cleanup the md device
md     /memdisk     mfs     rw,-s64M    0   0         # /etc/fstab entry

Linux

# mount -t tmpfs -osize=64m tmpfs /memdisk

Disk performance

Read and write a 1 GB file on partition ad4s3c (/home)

# time dd if=/dev/ad4s3c of=/dev/null bs=1024k count=1000
# time dd if=/dev/zero bs=1024k count=1000 of=/home/1Gb.file
# hdparm -tT /dev/hda      # Linux only

Network

Routing | Additional IP | Change MAC | Ports | Firewall | IP Forward | NAT | DNS | DHCP | Traffic | QoS | NIS | Netcat

Debugging (See also Traffic analysis)

Linux

# ethtool eth0              # Show the ethernet status (replaces mii-diag)
# ethtool -s eth0 speed 100 duplex full # Force 100Mbit Full duplex
# ethtool -s eth0 autoneg off # Disable auto negotiation
# ethtool -p eth1           # Blink the ethernet led - very useful when supported
# ip link show              # Display all interfaces on Linux (similar to ifconfig)
# ip link set eth0 up       # Bring device up (or down). Same as "ifconfig eth0 up"
# ip addr show              # Display all IP addresses on Linux (similar to ifconfig)
# ip neigh show             # Similar to arp -a

Other OSes

# ifconfig fxp0             # Check the "media" field on FreeBSD
# arp -a                    # Check the router (or host) ARP entry (all OS)
# ping cb.vu                # The first thing to try...
# traceroute cb.vu          # Print the route path to destination
# ifconfig fxp0 media 100baseTX mediaopt full-duplex # 100Mbit full duplex (FreeBSD)
# netstat -s                # System-wide statistics for each network protocol

Additional commands which are not always installed per default but easy to find:

# arping 192.168.16.254     # Ping on ethernet layer
# tcptraceroute -f 5 cb.vu # uses tcp instead of icmp to trace through firewalls

Routing

Print routing table

# route -n                  # Linux or use "ip route"
# netstat -rn               # Linux, BSD and UNIX
# route print               # Windows

Add and delete a route

FreeBSD

# route add 212.117.0.0/16 192.168.1.1
# route delete 212.117.0.0/16
# route add default 192.168.1.1

Add the route permanently in /etc/rc.conf

static_routes="myroute"
route_myroute="-net 212.117.0.0/16 192.168.1.1"

Linux

# route add -net 192.168.20.0 netmask 255.255.255.0 gw 192.168.16.254
# ip route add 192.168.20.0/24 via 192.168.16.254       # same as above with ip route
# route add -net 192.168.20.0 netmask 255.255.255.0 dev eth0
# route add default gw 192.168.51.254
# ip route add default via 192.168.51.254 dev eth0      # same as above with ip route
# route delete -net 192.168.20.0 netmask 255.255.255.0

Solaris

# route add -net 192.168.20.0 -netmask 255.255.255.0 192.168.16.254
# route add default 192.168.51.254 1                    # 1 = hops to the next gateway
# route change default 192.168.50.254 1

Permanent entries are set in entry in /etc/defaultrouter.
Windows

# Route add 192.168.50.0 mask 255.255.255.0 192.168.51.253
# Route add 0.0.0.0 mask 0.0.0.0 192.168.51.254

Use add -p to make the route persistent.

Configure additional IP addresses

Linux

# ifconfig eth0 192.168.50.254 netmask 255.255.255.0       # First IP
# ifconfig eth0:0 192.168.51.254 netmask 255.255.255.0     # Second IP
# ip addr add 192.168.50.254/24 dev eth0                   # Equivalent ip commands
# ip addr add 192.168.51.254/24 dev eth0 label eth0:1

FreeBSD

# ifconfig fxp0 inet 192.168.50.254/24                     # First IP
# ifconfig fxp0 alias 192.168.51.254 netmask 255.255.255.0 # Second IP
# ifconfig fxp0 -alias 192.168.51.254                      # Remove second IP alias

Permanent entries in /etc/rc.conf

ifconfig_fxp0="inet 192.168.50.254 netmask 255.255.255.0"
ifconfig_fxp0_alias0="192.168.51.254 netmask 255.255.255.0"

Solaris

Check the settings with ifconfig -a

# ifconfig hme0 plumb                                      # Enable the network card
# ifconfig hme0 192.168.50.254 netmask 255.255.255.0 up    # First IP
# ifconfig hme0:1 192.168.51.254 netmask 255.255.255.0 up # Second IP

Change MAC address

Normally you have to bring the interface down before the change. Don't tell me why you want to change the MAC address...

# ifconfig eth0 down
# ifconfig eth0 hw ether 00:01:02:03:04:05      # Linux
# ifconfig fxp0 link 00:01:02:03:04:05          # FreeBSD
# ifconfig hme0 ether 00:01:02:03:04:05         # Solaris
# sudo ifconfig en0 ether 00:01:02:03:04:05     # OS X Tiger, Snow Leopard LAN*
# sudo ifconfig en0 lladdr 00:01:02:03:04:05    # OS X Leopard

*Typical wireless interface is en1 and needs do disassociate from
any network first (osxdaily howto).

# echo "alias airport='/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport'"\
>> ~/.bash_profile         # or symlink to /usr/sbin
# airport -z               # Disassociate from wireless networks
# airport -I               # Get info from wireless network

Many tools exist for Windows. For example etherchangehttp://ntsecurity.nu/toolbox/etherchange. Or look for "Mac Makeup", "smac".

Ports in use

Listening open ports:

# netstat -an | grep LISTEN
# lsof -i                  # Linux list all Internet connections
# socklist                 # Linux display list of open sockets
# sockstat -4              # FreeBSD application listing
# netstat -anp --udp --tcp | grep LISTEN        # Linux
# netstat -tup             # List active connections to/from system (Linux)
# netstat -tupl            # List listening ports from system (Linux)
# netstat -ano             # Windows

Firewall

Check if a firewall is running (typical configuration only):
Linux

# iptables -L -n -v                  # For status
Open the iptables firewall
# iptables -P INPUT       ACCEPT     # Open everything
# iptables -P FORWARD     ACCEPT
# iptables -P OUTPUT      ACCEPT
# iptables -Z                        # Zero the packet and byte counters in all chains
# iptables -F                        # Flush all chains
# iptables -X                        # Delete all chains

FreeBSD

# ipfw show                          # For status
# ipfw list 65535 # if answer is "65535 deny ip from any to any" the fw is disabled
# sysctl net.inet.ip.fw.enable=0     # Disable
# sysctl net.inet.ip.fw.enable=1     # Enable

IP Forward for routing

Linux

Check and then enable IP forward with:

# cat /proc/sys/net/ipv4/ip_forward # Check IP forward 0=off, 1=on
# echo 1 > /proc/sys/net/ipv4/ip_forward

or edit /etc/sysctl.conf with:

net.ipv4.ip_forward = 1

FreeBSD

Check and enable with:

# sysctl net.inet.ip.forwarding      # Check IP forward 0=off, 1=on
# sysctl net.inet.ip.forwarding=1
# sysctl net.inet.ip.fastforwarding=1   # For dedicated router or firewall
Permanent with entry in /etc/rc.conf:
gateway_enable="YES"                 # Set to YES if this host will be a gateway.

Solaris

# ndd -set /dev/ip ip_forwarding 1   # Set IP forward 0=off, 1=on

NAT Network Address Translation

Linux

# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE   # to activate NAT
# iptables -t nat -A PREROUTING -p tcp -d 78.31.70.238 --dport 20022 -j DNAT \
--to 192.168.16.44:22           # Port forward 20022 to internal IP port ssh
# iptables -t nat -A PREROUTING -p tcp -d 78.31.70.238 --dport 993:995 -j DNAT \
--to 192.168.16.254:993-995     # Port forward of range 993-995
# ip route flush cache
# iptables -L -t nat            # Check NAT status

Delete the port forward with -D instead of -A. The program netstat-nathttp://tweegy.nl/projects/netstat-nat is very useful to track connections (it uses /proc/net/ip_conntrack or /proc/net/nf_conntrack).

# netstat-nat -n                # show all connections with IPs

FreeBSD

# natd -s -m -u -dynamic -f /etc/natd.conf -n fxp0
Or edit /etc/rc.conf with:
firewall_enable="YES"           # Set to YES to enable firewall functionality
firewall_type="open"            # Firewall type (see /etc/rc.firewall)
natd_enable="YES"               # Enable natd (if firewall_enable == YES).
natd_interface="tun0"           # Public interface or IP address to use.
natd_flags="-s -m -u -dynamic -f /etc/natd.conf"

Port forward with:

# cat /etc/natd.conf
same_ports yes
use_sockets yes
unregistered_only
# redirect_port tcp insideIP:2300-2399 3300-3399 # port range
redirect_port udp 192.168.51.103:7777 7777

DNS

On Unix the DNS entries are valid for all interfaces and are stored in /etc/resolv.conf. The domain to which the host belongs is also stored in this file. A minimal configuration is:

nameserver 78.31.70.238
search sleepyowl.net intern.lab
domain sleepyowl.net

Check the system domain name with:

# hostname -d                        # Same as dnsdomainname

Windows

On Windows the DNS are configured per interface. To display the configured DNS and to flush the DNS cache use:

# ipconfig /?                        # Display help
# ipconfig /all                      # See all information including DNS

Flush DNS

Flush the OS DNS cache, some application using their own cache (e.g. Firefox) and will be unaffected.

# /etc/init.d/nscd restart           # Restart nscd if used - Linux/BSD/Solaris
# lookupd -flushcache                # OS X Tiger
# dscacheutil -flushcache            # OS X Leopard and newer
# ipconfig /flushdns                 # Windows

Forward queries

Dig is you friend to test the DNS settings. For example the public DNS server 213.133.105.2 ns.second-ns.de can be used for testing. See from which server the client receives the answer (simplified answer).

# dig sleepyowl.net
sleepyowl.net.          600     IN      A       78.31.70.238
;; SERVER: 192.168.51.254#53(192.168.51.254)

The router 192.168.51.254 answered and the response is the A entry. Any entry can be queried and the DNS server can be selected with @:

# dig MX google.com
# dig @127.0.0.1 NS sun.com          # To test the local server
# dig @204.97.212.10 NS MX heise.de # Query an external server
# dig AXFR @ns1.xname.org cb.vu      # Get the full zone (zone transfer)

The program host is also powerful.

# host -t MX cb.vu                   # Get the mail MX entry
# host -t NS -T sun.com              # Get the NS record over a TCP connection
# host -a sleepyowl.net              # Get everything

Reverse queries

Find the name belonging to an IP address (in-addr.arpa.). This can be done with dig, host and nslookup:

# dig -x 78.31.70.238
# host 78.31.70.238
# nslookup 78.31.70.238

/etc/hosts

Single hosts can be configured in the file /etc/hosts instead of running named locally to resolve the hostname queries. The format is simple, for example:

78.31.70.238   sleepyowl.net   sleepyowl

The priority between hosts and a dns query, that is the name resolution order, can be configured in /etc/nsswitch.conf AND /etc/host.conf. The file also exists on Windows, it is usually in:

C:\WINDOWS\SYSTEM32\DRIVERS\ETC

DHCP

Linux

Some distributions (SuSE) use dhcpcd as client. The default interface is eth0.

# dhcpcd -n eth0           # Trigger a renew (does not always work)
# dhcpcd -k eth0           # release and shutdown

The lease with the full information is stored in:

/var/lib/dhcpcd/dhcpcd-eth0.info

FreeBSD

FreeBSD (and Debian) uses dhclient. To configure an interface (for example bge0) run:

# dhclient bge0

The lease with the full information is stored in:

/var/db/dhclient.leases.bge0

Use

/etc/dhclient.conf

to prepend options or force different options:

# cat /etc/dhclient.conf
interface "rl0" {
    prepend domain-name-servers 127.0.0.1;
    default domain-name "sleepyowl.net";
    supersede domain-name "sleepyowl.net";
}

Windows

The dhcp lease can be renewed with ipconfig:

# ipconfig /renew          # renew all adapters
# ipconfig /renew LAN      # renew the adapter named "LAN"
# ipconfig /release WLAN   # release the adapter named "WLAN"

Yes it is a good idea to rename you adapter with simple names!

Traffic analysis

Bmonhttp://people.suug.ch/~tgr/bmon/ is a small console bandwidth monitor and can display the flow on different interfaces.
Sniff with tcpdump

# tcpdump -nl -i bge0 not port ssh and src $192.168.16.121 or 192.168.16.54$
# tcpdump -n -i eth1 net 192.168.16.121           # select to/from a single IP
# tcpdump -n -i eth1 net 192.168.16.0/24          # select traffic to/from a network
# tcpdump -l > dump && tail -f dump               # Buffered output
# tcpdump -i rl0 -w traffic.rl0                   # Write traffic headers in binary file
# tcpdump -i rl0 -s 0 -w traffic.rl0              # Write traffic + payload in binary file
# tcpdump -r traffic.rl0                          # Read from file (also for ethereal
# tcpdump port 80                                 # The two classic commands
# tcpdump host google.com
# tcpdump -i eth0 -X port $110 or 143$          # Check if pop or imap is secure
# tcpdump -n -i eth0 icmp                         # Only catch pings
# tcpdump -i eth0 -s 0 -A port 80 | grep GET      # -s 0 for full packet -A for ASCII

Additional important options:

    *   -A     Print each packets in clear text (without header)

    *   -X     Print packets in hex and ASCII

    *   -l     Make stdout line buffered

    *   -D     Print all interfaces available

On Windows use windump from www.winpcap.org. Use windump -D to list the interfaces.
Scan with nmap

Nmaphttp://insecure.org/nmap/ is a port scanner with OS detection, it is usually installed on most distributions and is also available for Windows. If you don't scan your servers, hackers do it for you...

# nmap cb.vu               # scans all reserved TCP ports on the host
# nmap -sP 192.168.16.0/24 # Find out which IP are used and by which host on 0/24
# nmap -sS -sV -O cb.vu    # Do a stealth SYN scan with version and OS detection
PORT      STATE SERVICE             VERSION
22/tcp    open   ssh                 OpenSSH 3.8.1p1 FreeBSD-20060930 (protocol 2.0)
25/tcp    open   smtp                Sendmail smtpd 8.13.6/8.13.6
80/tcp    open   http                Apache httpd 2.0.59 ((FreeBSD) DAV/2 PHP/4.
[...]
Running: FreeBSD 5.X
Uptime 33.120 days (since Fri Aug 31 11:41:04 2007)

Other non standard but useful tools are hping (www.hping.org) an IP packet assembler/analyzer and fping (fping.sourceforge.net). fping can check multiple hosts in a round-robin fashion.

Traffic control (QoS)

Traffic control manages the queuing, policing, scheduling, and other traffic parameters for a network. The following examples are simple practical uses of the Linux and FreeBSD capabilities to better use the available bandwidth.
Limit upload

DSL or cable modems have a long queue to improve the upload throughput. However filling the queue with a fast device (e.g. ethernet) will dramatically decrease the interactivity. It is therefore useful to limit the device upload rate to match the physical capacity of the modem, this should greatly improve the interactivity. Set to about 90% of the modem maximal (cable) speed.
Linux

For a 512 Kbit upload modem.

# tc qdisc add dev eth0 root tbf rate 480kbit latency 50ms burst 1540
# tc -s qdisc ls dev eth0                         # Status
# tc qdisc del dev eth0 root                      # Delete the queue
# tc qdisc change dev eth0 root tbf rate 220kbit latency 50ms burst 1540

FreeBSD

FreeBSD uses the dummynet traffic shaper which is configured with ipfw. Pipes are used to set limits the bandwidth in units of [K|M]{bit/s|Byte/s}, 0 means unlimited bandwidth. Using the same pipe number will reconfigure it. For example limit the upload bandwidth to 500 Kbit.

# kldload dummynet                                # load the module if necessary
# ipfw pipe 1 config bw 500Kbit/s                 # create a pipe with limited bandwidth
# ipfw add pipe 1 ip from me to any               # divert the full upload into the pipe

Quality of service

Linux

Priority queuing with tc to optimize VoIP. See the full example on voip-info.org or www.howtoforge.com. Suppose VoIP uses udp on ports 10000:11024 and device eth0 (could also be ppp0 or so). The following commands define the QoS to three queues and force the VoIP traffic to queue 1 with QoS 0x1e (all bits set). The default traffic flows into queue 3 and QoS Minimize-Delay flows into queue 2.

# tc qdisc add dev eth0 root handle 1: prio priomap 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 0
# tc qdisc add dev eth0 parent 1:1 handle 10: sfq
# tc qdisc add dev eth0 parent 1:2 handle 20: sfq
# tc qdisc add dev eth0 parent 1:3 handle 30: sfq
# tc filter add dev eth0 protocol ip parent 1: prio 1 u32 \
match ip dport 10000 0x3C00 flowid 1:1          # use server port range
match ip dst 123.23.0.1 flowid 1:1              # or/and use server IP

Status and remove with

# tc -s qdisc ls dev eth0                         # queue status
# tc qdisc del dev eth0 root                      # delete all QoS

Calculate port range and mask

The tc filter defines the port range with port and mask which you have to calculate. Find the 2^N ending of the port range, deduce the range and convert to HEX. This is your mask. Example for 10000 -> 11024, the range is 1024.

# 2^13 (8192) < 10000 < 2^14 (16384)              # ending is 2^14 = 16384
# echo "obase=16;(2^14)-1024" | bc                # mask is 0x3C00

FreeBSD

The max link bandwidth is 500Kbit/s and we define 3 queues with priority 100:10:1 for VoIP:ssh:all the rest.

# ipfw pipe 1 config bw 500Kbit/s
# ipfw queue 1 config pipe 1 weight 100
# ipfw queue 2 config pipe 1 weight 10
# ipfw queue 3 config pipe 1 weight 1
# ipfw add 10 queue 1 proto udp dst-port 10000-11024
# ipfw add 11 queue 1 proto udp dst-ip 123.23.0.1 # or/and use server IP
# ipfw add 20 queue 2 dsp-port ssh
# ipfw add 30 queue 3 from me to any              # all the rest

Status and remove with

# ipfw list                                       # rules status
# ipfw pipe list                                  # pipe status
# ipfw flush                                      # deletes all rules but default

NIS Debugging

Some commands which should work on a well configured NIS client:

# ypwhich                  # get the connected NIS server name
# domainname               # The NIS domain name as configured
# ypcat group              # should display the group from the NIS server
# cd /var/yp && make       # Rebuild the yp database
# rpcinfo -p servername    # Report RPC services of the server

Is ypbind running?

# ps auxww | grep ypbind
/usr/sbin/ypbind -s -m -S servername1,servername2   # FreeBSD
/usr/sbin/ypbind           # Linux
# yppoll passwd.byname
Map passwd.byname has order number 1190635041. Mon Sep 24 13:57:21 2007
The master server is servername.domain.net.

Linux

# cat /etc/yp.conf
ypserver servername
domain domain.net broadcast

Netcat

Netcathttp://netcat.sourceforge.net (nc) is better known as the "network Swiss Army Knife", it can manipulate, create or read/write TCP/IP connections. Here some useful examples, there are many more on the net, for example g-loaded.eu[...]http://www.g-loaded.eu/2006/11/06/netcat-a-couple-of-useful-examples and herehttp://www.terminally-incoherent.com/blog/2007/08/07/few-useful-netcat-tricks.
You might need to use the command netcat instead of nc. Also see the similar command socat.

File transfer

Copy a large folder over a raw tcp connection. The transfer is very quick (no protocol overhead) and you don't need to mess up with NFS or SMB or FTP or so, simply make the file available on the server, and get it from the client. Here 192.168.1.1 is the server IP address.

server# tar -cf - -C VIDEO_TS . | nc -l -p 4444         # Serve tar folder on port 4444
client# nc 192.168.1.1 4444 | tar xpf - -C VIDEO_TS     # Pull the file on port 4444
server# cat largefile | nc -l 5678                      # Server a single file
client# nc 192.168.1.1 5678 > largefile                 # Pull the single file
server# dd if=/dev/da0 | nc -l 4444                     # Server partition image
client# nc 192.168.1.1 4444 | dd of=/dev/da0            # Pull partition to clone
client# nc 192.168.1.1 4444 | dd of=da0.img             # Pull partition to file

Other hacks

Specially here, you must know what you are doing.
Remote shell

Option -e only on the Windows version? Or use nc 1.10.

# nc -lp 4444 -e /bin/bash                        # Provide a remote shell (server backdoor)
# nc -lp 4444 -e cmd.exe                          # remote shell for Windows

Emergency web server

Serve a single file on port 80 in a loop.

# while true; do nc -l -p 80 < unixtoolbox.xhtml; done

Chat

Alice and Bob can chat over a simple TCP socket. The text is transferred with the enter key.

alice# nc -lp 4444
bob # nc 192.168.1.1 4444

SSH SCP

Public key | Fingerprint | SCP | Tunneling

See other tricks 25 ssh cmdhttp://blog.urfix.com/25-ssh-commands-tricks/
Public key authentication

Connect to a host without password using public key authentication. The idea is to append your public key to the authorized_keys2 file on the remote host. For this example let's connect host-client to host-server, the key is generated on the client. With cygwin you might have to create your home directoy and the .ssh directory with # mkdir -p /home/USER/.ssh

    *   Use ssh-keygen to generate a key pair. ~/.ssh/id_dsa is the private key, ~/.ssh/id_dsa.pub is the public key.

    *   Copy only the public key to the server and append it to the file ~/.ssh/authorized_keys2 on your home on the server.

# ssh-keygen -t dsa -N ''
# cat ~/.ssh/id_dsa.pub | ssh you@host-server "cat - >> ~/.ssh/authorized_keys2"

Using the Windows client from ssh.com

The non commercial version of the ssh.com client can be downloaded the main ftp site: ftp.ssh.com/pub/ssh/. Keys generated by the ssh.com client need to be converted for the OpenSSH server. This can be done with the ssh-keygen command.

    *   Create a key pair with the ssh.com client: Settings - User Authentication - Generate New....

    *   I use Key type DSA; key length 2048.

    *   Copy the public key generated by the ssh.com client to the server into the ~/.ssh folder.

    *   The keys are in C:\Documents and Settings\%USERNAME%\Application Data\SSH\UserKeys.

    *   Use the ssh-keygen command on the server to convert the key:

      # cd ~/.ssh
      # ssh-keygen -i -f keyfilename.pub >> authorized_keys2

Notice: We used a DSA key, RSA is also possible. The key is not protected by a password.
Using putty for Windows

Puttyhttp://www.chiark.greenend.org.uk/~sgtatham/putty/download.html is a simple and free ssh client for Windows.

    *   Create a key pair with the puTTYgen program.

    *   Save the public and private keys (for example into C:\Documents and Settings\%USERNAME%\.ssh).

    *   Copy the public key to the server into the ~/.ssh folder:


      # scp .ssh/puttykey.pub root@192.168.51.254:.ssh/

    *   Use the ssh-keygen command on the server to convert the key for OpenSSH:

      # cd ~/.ssh
      # ssh-keygen -i -f puttykey.pub >> authorized_keys2

    * Point the private key location in the putty settings: Connection - SSH - Auth

Check fingerprint

At the first login, ssh will ask if the unknown host with the fingerprint has to be stored in the known hosts. To avoid a man-in-the-middle attack the administrator of the server can send you the server fingerprint which is then compared on the first login. Use ssh-keygen -l to get the fingerprint (on the server):

# ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub      # For RSA key
2048 61:33:be:9b:ae:6c:36:31:fd:83:98:b7:99:2d:9f:cd /etc/ssh/ssh_host_rsa_key.pub
# ssh-keygen -l -f /etc/ssh/ssh_host_dsa_key.pub      # For DSA key (default)
2048 14:4a:aa:d9:73:25:46:6d:0a:48:35:c7:f4:16:d4:ee /etc/ssh/ssh_host_dsa_key.pub

Now the client connecting to this server can verify that he is connecting to the right server:

# ssh linda
The authenticity of host 'linda (192.168.16.54)' can't be established.
DSA key fingerprint is 14:4a:aa:d9:73:25:46:6d:0a:48:35:c7:f4:16:d4:ee.
Are you sure you want to continue connecting (yes/no)? yes

Secure file transfer

Some simple commands:

# scp file.txt host-two:/tmp
# scp joe@host-two:/www/*.html /www/tmp
# scp -r joe@host-two:/www /www/tmp

In Konqueror or Midnight Commander it is possible to access a remote file system with the address fish://user@gate. However the implementation is very slow.

Furthermore it is possible to mount a remote folder with sshfs a file system client based on SCP. See fuse sshfshttp://fuse.sourceforge.net/sshfs.html.

ssh_exchange_identification: Connection closed by remote host

With this error try the following on the server:

echo 'SSHD: ALL' >> /etc/hosts.allow
/etc/init.d/sshd restart

Tunneling

SSH tunneling allows to forward or reverse forward a port over the SSH connection, thus securing the traffic and accessing ports which would otherwise be blocked. This only works with TCP. The general nomenclature for forward and reverse is (see also ssh and NAT example):

# ssh -L localport:desthost:destport user@gate # desthost as seen from the gate
# ssh -R destport:desthost:localport user@gate # forwards your localport to destination
    # desthost:localport as seen from the client initiating the tunnel
# ssh -X user@gate   # To force X forwarding

This will connect to gate and forward the local port to the host desthost:destport. Note desthost is the destination host as seen by the gate, so if the connection is to the gate, then desthost is localhost. More than one port forward is possible.
Direct forward on the gate

Let say we want to access the CVS (port 2401) and http (port 80) which are running on the gate. This is the simplest example, desthost is thus localhost, and we use the port 8080 locally instead of 80 so we don't need to be root. Once the ssh session is open, both services are accessible on the local ports.

# ssh -L 2401:localhost:2401 -L 8080:localhost:80 user@gate

Netbios and remote desktop forward to a second server

Let say a Windows smb server is behind the gate and is not running ssh. We need access to the smb share and also remote desktop to the server.

# ssh -L 139:smbserver:139 -L 3388:smbserver:3389 user@gate

The smb share can now be accessed with \\127.0.0.1\, but only if the local share is disabled, because the local share is listening on port 139.

It is possible to keep the local share enabled, for this we need to create a new virtual device with a new IP address for the tunnel, the smb share will be connected over this address. Furthermore the local RDP is already listening on 3389, so we choose 3388. For this example let's use a virtual IP of 10.1.1.1.

    * With putty use Source port=10.1.1.1:139. It is possible to create multiple loop devices and tunnel. On Windows 2000, only putty worked for me. On Windows Vista also forward the port 445 in addition to the port 139. Also on Vista the patch KB942624 prevents the port 445 to be forwarded, so I had to uninstall this path in Vista.

    * With the ssh.com client, disable "Allow local connections only". Since ssh.com will bind to all addresses, only a single share can be connected.

Now create the loopback interface with IP 10.1.1.1:

    * # System->Control Panel->Add Hardware # Yes, Hardware is already connected
      # Add a new hardware device (at bottom).

    * # Install the hardware that I manually select # Network adapters # Microsoft , Microsoft Loopback Adapter.

    * Configure the IP address of the fake device to 10.1.1.1 mask 255.255.255.0, no gateway.

    * advanced->WINS, Enable LMHosts Lookup; Disable NetBIOS over TCP/IP.

    * # Enable Client for Microsoft Networks. # Disable File and Printer Sharing for Microsoft Networks.

I HAD to reboot for this to work. Now connect to the smb share with \\10.1.1.1 and remote desktop to 10.1.1.1:3388.
Debug

If it is not working:

    * Are the ports forwarded: netstat -an? Look at 0.0.0.0:139 or 10.1.1.1:139

    * Does telnet 10.1.1.1 139 connect?

    * You need the checkbox "Local ports accept connections from other hosts".

    * Is "File and Printer Sharing for Microsoft Networks" disabled on the loopback interface?

Connect two clients behind NAT

Suppose two clients are behind a NAT gateway and client cliadmin has to connect to client cliuser (the destination), both can login to the gate with ssh and are running Linux with sshd. You don't need root access anywhere as long as the ports on gate are above 1024. We use 2022 on gate. Also since the gate is used locally, the option GatewayPorts is not necessary.

On client cliuser (from destination to gate):

# ssh -R 2022:localhost:22 user@gate            # forwards client 22 to gate:2022

On client cliadmin (from host to gate):

# ssh -L 3022:localhost:2022 admin@gate         # forwards client 3022 to gate:2022

Now the admin can connect directly to the client cliuser with:

# ssh -p 3022 admin@localhost                   # local:3022 -> gate:2022 -> client:22

Connect to VNC behind NAT

Suppose a Windows client with VNC listening on port 5900 has to be accessed from behind NAT.
On client cliwin to gate:

# ssh -R 15900:localhost:5900 user@gate

On client cliadmin (from host to gate):

# ssh -L 5900:localhost:15900 admin@gate

Now the admin can connect directly to the client VNC with:

# vncconnect -display :0 localhost

Dig a multi-hop ssh tunnel

Suppose you can not reach a server directly with ssh, but only via multiple intermediate hosts (for example because of routing issues). Sometimes it is still necessary to get a direct client - server connection, for example to copy files with scp, or forward other ports like smb or vnc. One way to do this is to chain tunnels together to forward a port to the server along the hops. This "carrier" port only reaches its final destination on the last connection to the server.

Suppose we want to forward the ssh port from a client to a server over two hops. Once the tunnel is build, it is possible to connect to the server directly from the client (and also add an other port forward).
Create tunnel in one shell

client -> host1 -> host2 -> server and dig tunnel 5678

client># ssh -L5678:localhost:5678 host1        # 5678 is an arbitrary port for the tunnel
host_1># ssh -L5678:localhost:5678 host2        # chain 5678 from host1 to host2
host_2># ssh -L5678:localhost:22 server         # end the tunnel on port 22 on the server

Use tunnel with an other shell

client -> server using tunnel 5678

# ssh -p 5678 localhost                         # connect directly from client to server
# scp -P 5678 myfile localhost:/tmp/            # or copy a file directly using the tunnel
# rsync -e 'ssh -p 5678' myfile localhost:/tmp/ # or rsync a file directly to the server

Autoconnect and keep alive script

I use variations of the following script to keep a machine reacheable over a reverse ssh tunnel. The connection is automatically rebuilt if closed. You can add multiple -L or -R tunnels on one line.

#!/bin/sh
COMMAND="ssh -N -f -g -R 3022:localhost:22 colin@cb.vu"
pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND
exit 0

1 * * * * colin /home/colin/port_forward.sh     # crontab entry (here hourly)

VPN with SSH

As of version 4.3, OpenSSH can use the tun/tap device to encrypt a tunnel. This is very similar to other TLS based VPN solutions like OpenVPN. One advantage with SSH is that there is no need to install and configure additional software. Additionally the tunnel uses the SSH authentication like pre shared keys. The drawback is that the encapsulation is done over TCP which might result in poor performance on a slow link. Also the tunnel is relying on a single (fragile) TCP connection. This technique is very useful for a quick IP based VPN setup. There is no limitation as with the single TCP port forward, all layer 3/4 protocols like ICMP, TCP/UDP, etc. are forwarded over the VPN. In any case, the following options are needed in the sshd_conf file:

PermitRootLogin yes
PermitTunnel yes

Single P2P connection

Here we are connecting two hosts, hclient and hserver with a peer to peer tunnel. The connection is started from hclient to hserver and is done as root. The tunnel end points are 10.0.1.1 (server) and 10.0.1.2 (client) and we create a device tun5 (this could also be an other number). The procedure is very simple:

    *   Connect with SSH using the tunnel option -w

    *   Configure the IP addresses of the tunnel. Once on the server and once on the client.

Connect to the server

Connection started on the client and commands are executed on the server.
Server is on Linux

cli># ssh -w5:5 root@hserver
srv># ifconfig tun5 10.0.1.1 netmask 255.255.255.252   # Executed on the server shell

Server is on FreeBSD

cli># ssh -w5:5 root@hserver
srv># ifconfig tun5 10.0.1.1 10.0.1.2                  # Executed on the server shell

Configure the client

Commands executed on the client:

cli># ifconfig tun5 10.0.1.2 netmask 255.255.255.252   # Client is on Linux
cli># ifconfig tun5 10.0.1.2 10.0.1.1                  # Client is on FreeBSD

The two hosts are now connected and can transparently communicate with any layer 3/4 protocol using the tunnel IP addresses.

Connect two networks

In addition to the p2p setup above, it is more useful to connect two private networks with an SSH VPN using two gates. Suppose for the example, netA is 192.168.51.0/24 and netB 192.168.16.0/24. The procedure is similar as above, we only need to add the routing. NAT must be activated on the private interface only if the gates are not the same as the default gateway of their network.

192.168.51.0/24 (netA)|gateA <-> gateB|192.168.16.0/24 (netB)

    *   Connect with SSH using the tunnel option -w.

    *   Configure the IP addresses of the tunnel. Once on the server and once on the client.

    *   Add the routing for the two networks.

    *   If necessary, activate NAT on the private interface of the gate.

The setup is started from gateA in netA.
Connect from gateA to gateB

Connection is started from gateA and commands are executed on gateB.
gateB is on Linux

gateA># ssh -w5:5 root@gateB
gateB># ifconfig tun5 10.0.1.1 netmask 255.255.255.252 # Executed on the gateB shell
gateB># route add -net 192.168.51.0 netmask 255.255.255.0 dev tun5
gateB># echo 1 > /proc/sys/net/ipv4/ip_forward        # Only needed if not default gw
gateB># iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

gateB is on FreeBSD

gateA># ssh -w5:5 root@gateB                          # Creates the tun5 devices
gateB># ifconfig tun5 10.0.1.1 10.0.1.2               # Executed on the gateB shell
gateB># route add 192.168.51.0/24 10.0.1.2
gateB># sysctl net.inet.ip.forwarding=1               # Only needed if not default gw
gateB># natd -s -m -u -dynamic -n fxp0                # see NAT
gateA># sysctl net.inet.ip.fw.enable=1

Configure gateA

Commands executed on gateA:
gateA is on Linux

gateA># ifconfig tun5 10.0.1.2 netmask 255.255.255.252
gateA># route add -net 192.168.16.0 netmask 255.255.255.0 dev tun5
gateA># echo 1 > /proc/sys/net/ipv4/ip_forward
gateA># iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

gateA is on FreeBSD

gateA># ifconfig tun5 10.0.1.2 10.0.1.1
gateA># route add 192.168.16.0/24 10.0.1.2
gateA># sysctl net.inet.ip.forwarding=1
gateA># natd -s -m -u -dynamic -n fxp0                # see NAT
gateA># sysctl net.inet.ip.fw.enable=1

The two private networks are now transparently connected via the SSH VPN. The IP forward and NAT settings are only necessary if the gates are not the default gateways. In this case the clients would not know where to forward the response, and nat must be activated.

RSYNC

Rsync can almost completely replace cp and scp, furthermore interrupted transfers are efficiently restarted. A trailing slash (and the absence thereof) has different meanings, the man page is good... Here some examples:

Copy the directories with full content:

# rsync -a /home/colin/ /backup/colin/                # "archive" mode. e.g keep the same
# rsync -a /var/ /var_bak/
# rsync -aR --delete-during /home/user/ /backup/      # use relative (see below)

Same as before but over the network and with compression. Rsync uses SSH for the transport per default and will use the ssh key if they are set. Use ":" as with SCP. A typical remote copy:

# rsync -axSRzv /home/user/ user@server:/backup/user/ # Copy to remote
# rsync -a 'user@server:My\ Documents' My\ Documents # Quote AND escape spaces for the remote shell

Exclude any directory tmp within /home/user/ and keep the relative folders hierarchy, that is the remote directory will have the structure /backup/home/user/. This is typically used for backups.

# rsync -azR --exclude=tmp/ /home/user/ user@server:/backup/

Use port 20022 for the ssh connection:

# rsync -az -e 'ssh -p 20022' /home/colin/ user@server:/backup/colin/

Using the rsync daemon (used with "::") is much faster, but not encrypted over ssh. The location of /backup is defined by the configuration in /etc/rsyncd.conf. The variable RSYNC_PASSWORD can be set to avoid the need to enter the password manually.

# rsync -axSRz /home/ ruser@hostname::rmodule/backup/
# rsync -axSRz ruser@hostname::rmodule/backup/ /home/    # To copy back

Some important options:

    *   -a, --archive       archive mode; same as -rlptgoD (no -H)

    *   -r, --recursive       recurse into directories

    *   -R, --relative       use relative path names

    *   -H, --hard-links       preserve hard links

    *   -S, --sparse       handle sparse files efficiently

    *   -x, --one-file-system       don't cross file system boundaries

    *    --exclude=PATTERN       exclude files matching PATTERN

    *    --delete-during       receiver deletes during xfer, not before

    *    --delete-after       receiver deletes after transfer, not before

Rsync on Windows

Rsync is available for Windows through cygwin or as stand-alone packaged in cwrsynchttp://sourceforge.net/projects/sereds. This is very convenient for automated backups. Install one of them (not both) and add the path to the Windows system variables: # Control Panel -> System -> tab Advanced, button Environment Variables. Edit the "Path" system variable and add the full path to the installed rsync, e.g. C:\Program Files\cwRsync\bin or C:\cygwin\bin. This way the commands rsync and ssh are available in a Windows command shell.
Public key authentication

Rsync is automatically tunneled over SSH and thus uses the SSH authentication on the server. Automatic backups have to avoid a user interaction, for this the SSH public key authentication can be used and the rsync command will run without a password.

All the following commands are executed within a Windows console. In a console (Start -> Run -> cmd) create and upload the key as described in SSH, change "user" and "server" as appropriate. If the file authorized_keys2 does not exist yet, simply copy id_dsa.pub to authorized_keys2 and upload it.

# ssh-keygen -t dsa -N ''                   # Creates a public and a private key
# rsync user@server:.ssh/authorized_keys2 . # Copy the file locally from the server
# cat id_dsa.pub >> authorized_keys2        # Or use an editor to add the key
# rsync authorized_keys2 user@server:.ssh/ # Copy the file back to the server
# del authorized_keys2                      # Remove the local copy

Now test it with (in one line):

rsync -rv "/cygdrive/c/Documents and Settings/%USERNAME%/My Documents/" \
'user@server:My\ Documents/'

Automatic backup

Use a batch file to automate the backup and add the file in the scheduled tasks (Programs -> Accessories -> System Tools -> Scheduled Tasks). For example create the file backup.bat and replace user@server.

@ECHO OFF
REM rsync the directory My Documents
SETLOCAL
SET CWRSYNCHOME=C:\PROGRAM FILES\CWRSYNC
SET CYGWIN=nontsec
SET CWOLDPATH=%PATH%
REM uncomment the next line when using cygwin
SET PATH=%CWRSYNCHOME%\BIN;%PATH%
echo Press Control-C to abort
rsync -av "/cygdrive/c/Documents and Settings/%USERNAME%/My Documents/" \
'user@server:My\ Documents/'
pause

SUDO

Sudo is a standard way to give users some administrative rights without giving out the root password. Sudo is very useful in a multi user environment with a mix of server and workstations. Simply call the command with sudo:

# sudo /etc/init.d/dhcpd restart            # Run the rc script as root
# sudo -u sysadmin whoami                   # Run cmd as an other user

Configuration

Sudo is configured in /etc/sudoers and must only be edited with visudo. The basic syntax is (the lists are comma separated):

user hosts = (runas) commands          # In /etc/sudoers



    *   users one or more users or %group (like %wheel) to gain the rights

    *   hosts list of hosts (or ALL)

    *   runas list of users (or ALL) that the command rule can be run as. It is enclosed in ( )!

    *   commands list of commands (or ALL) that will be run as root or as (runas)

Additionally those keywords can be defined as alias, they are called User_Alias, Host_Alias, Runas_Alias and Cmnd_Alias. This is useful for larger setups. Here a sudoers example:

# cat /etc/sudoers
# Host aliases are subnets or hostnames.
Host_Alias   DMZ     = 212.118.81.40/28
Host_Alias   DESKTOP = work1, work2

# User aliases are a list of users which can have the same rights
User_Alias   ADMINS = colin, luca, admin
User_Alias   DEVEL   = joe, jack, julia
Runas_Alias DBA     = oracle,pgsql

# Command aliases define the full path of a list of commands
Cmnd_Alias   SYSTEM = /sbin/reboot,/usr/bin/kill,/sbin/halt,/sbin/shutdown,/etc/init.d/
Cmnd_Alias   PW      = /usr/bin/passwd [A-z]*, !/usr/bin/passwd root # Not root pwd!
Cmnd_Alias   DEBUG   = /usr/sbin/tcpdump,/usr/bin/wireshark,/usr/bin/nmap

# The actual rules
root,ADMINS ALL     = (ALL) NOPASSWD: ALL    # ADMINS can do anything w/o a password.
DEVEL        DESKTOP = (ALL) NOPASSWD: ALL    # Developers have full right on desktops
DEVEL        DMZ     = (ALL) NOPASSWD: DEBUG # Developers can debug the DMZ servers.

# User sysadmin can mess around in the DMZ servers with some commands.
sysadmin     DMZ     = (ALL) NOPASSWD: SYSTEM,PW,DEBUG
sysadmin     ALL,!DMZ = (ALL) NOPASSWD: ALL   # Can do anything outside the DMZ.
%dba         ALL     = (DBA) ALL              # Group dba can run as database user.

# anyone can mount/unmount a cd-rom on the desktop machines
ALL          DESKTOP = NOPASSWD: /sbin/mount /cdrom,/sbin/umount /cdrom

Encrypt Files

OpenSSL

A single file

Encrypt and decrypt:

# openssl aes-128-cbc -salt -in file -out file.aes
# openssl aes-128-cbc -d -salt -in file.aes -out file

Note that the file can of course be a tar archive.

tar and encrypt a whole directory

# tar -cf - directory | openssl aes-128-cbc -salt -out directory.tar.aes      # Encrypt
# openssl aes-128-cbc -d -salt -in directory.tar.aes | tar -x -f -            # Decrypt


tar zip and encrypt a whole directory

# tar -zcf - directory | openssl aes-128-cbc -salt -out directory.tar.gz.aes # Encrypt
# openssl aes-128-cbc -d -salt -in directory.tar.gz.aes | tar -xz -f -        # Decrypt

    * Use -k mysecretpassword after aes-128-cbc to avoid the interactive password request. However note that this is highly insecure.

    * Use aes-256-cbc instead of aes-128-cbc to get even stronger encryption. This uses also more CPU.

GPG

GnuPG is well known to encrypt and sign emails or any data. Furthermore gpg and also provides an advanced key management system. This section only covers files encryption, not email usage, signing or the Web-Of-Trust.

The simplest encryption is with a symmetric cipher. In this case the file is encrypted with a password and anyone who knows the password can decrypt it, thus the keys are not needed. Gpg adds an extention ".gpg" to the encrypted file names.

# gpg -c file                        # Encrypt file with password
# gpg file.gpg                       # Decrypt file (optionally -o otherfile)

Using keys

For more details see GPG Quick Starthttp://www.madboa.com/geek/gpg-quickstart and GPG/PGP Basicshttp://aplawrence.com/Basics/gpg.html and the gnupg documentationhttp://gnupg.org/documentation among others.

The private and public keys are the heart of asymmetric cryptography. What is important to remember:

    * Your public key is used by others to encrypt files that only you as the receiver can decrypt (not even the one who encrypted the file can decrypt it). The public key is thus meant to be distributed.

    * Your private key is encrypted with your passphrase and is used to decrypt files which were encrypted with your public key. The private key must be kept secure. Also if the key or passphrase is lost, so are all the files encrypted with your public key.

    * The key files are called keyrings as they can contain more than one key.

First generate a key pair. The defaults are fine, however you will have to enter at least your full name and email and optionally a comment. The comment is useful to create more than one key with the same name and email. Also you should use a "passphrase", not a simple password.

# gpg --gen-key                      # This can take a long time

The keys are stored in ~/.gnupg/ on Unix, on Windows they are typically stored in

C:/Documents and Settings/%USERNAME%/Application Data/gnupg/.

~/.gnupg/pubring.gpg                 # Contains your public keys and all others imported
~/.gnupg/secring.gpg                 # Can contain more than one private key

Short reminder on most used options:

    *   -e encrypt data

    *   -d decrypt data

    *   -r NAME encrypt for recipient NAME (or 'Full Name' or 'email@domain')

    *   -a create ascii armored output of a key

    *   -o use as output file

The examples use 'Your Name' and 'Alice' as the keys are referred to by the email or full name or partial name. For example I can use 'Colin' or 'c@cb.vu' for my key [Colin Barschel (cb.vu) <c@cb.vu>].
Encrypt for personal use only

No need to export/import any key for this. You have both already.

# gpg -e -r 'Your Name' file                  # Encrypt with your public key
# gpg -o file -d file.gpg                     # Decrypt. Use -o or it goes to stdout

Encrypt - Decrypt with keys

First you need to export your public key for someone else to use it. And you need to import the public say from Alice to encrypt a file for her. You can either handle the keys in simple ascii files or use a public key server.

For example Alice export her public key and you import it, you can then encrypt a file for her. That is only Alice will be able to decrypt it.

# gpg -a -o alicekey.asc --export 'Alice'     # Alice exported her key in ascii file.
# gpg --send-keys --keyserver subkeys.pgp.net KEYID   # Alice put her key on a server.
# gpg --import alicekey.asc                   # You import her key into your pubring.
# gpg --search-keys --keyserver subkeys.pgp.net 'Alice' # or get her key from a server.

Once the keys are imported it is very easy to encrypt or decrypt a file:

# gpg -e -r 'Alice' file                      # Encrypt the file for Alice.
# gpg -d file.gpg -o file                     # Decrypt a file encrypted by Alice for you.

Key administration

# gpg --list-keys                             # list public keys and see the KEYIDS
    The KEYID follows the '/' e.g. for: pub   1024D/D12B77CE the KEYID is D12B77CE
# gpg --gen-revoke 'Your Name'                # generate revocation certificate
# gpg --list-secret-keys                      # list private keys
# gpg --delete-keys NAME                      # delete a public key from local key ring
# gpg --delete-secret-key NAME                # delete a secret key from local key ring
# gpg --fingerprint KEYID                     # Show the fingerprint of the key
# gpg --edit-key KEYID                        # Edit key (e.g sign or add/del email)

Encrypt Partitions

Linux with LUKS | Linux dm-crypt only | FreeBSD GELI | FBSD pwd only | OS X image

There are (many) other alternative methods to encrypt disks, I only show here the methods I know and use. Keep in mind that the security is only good as long the OS has not been tempered with. An intruder could easily record the password from the keyboard events. Furthermore the data is freely accessible when the partition is attached and will not prevent an intruder to have access to it in this state.
Linux

Those instructions use the Linux dm-crypt (device-mapper) facility available on the 2.6 kernel. In this example, lets encrypt the partition /dev/sdc1, it could be however any other partition or disk, or USB or a file based partition created with losetup. In this case we would use /dev/loop0. See file image partition. The device mapper uses labels to identify a partition. We use sdc1 in this example, but it could be any string.
dm-crypt with LUKS

LUKS with dm-crypt has better encryption and makes it possible to have multiple passphrase for the same partition or to change the password easily. To test if LUKS is available, simply type # cryptsetup --help, if nothing about LUKS shows up, use the instructions below Without LUKS. First create a partition if necessary: fdisk /dev/sdc.
Create encrypted partition

# dd if=/dev/urandom of=/dev/sdc1          # Optional. For paranoids only (takes days)
# cryptsetup -y luksFormat /dev/sdc1       # This destroys any data on sdc1
# cryptsetup luksOpen /dev/sdc1 sdc1
# mkfs.ext3 /dev/mapper/sdc1               # create ext3 file system
# mount -t ext3 /dev/mapper/sdc1 /mnt
# umount /mnt
# cryptsetup luksClose sdc1                # Detach the encrypted partition

Attach

# cryptsetup luksOpen /dev/sdc1 sdc1
# mount -t ext3 /dev/mapper/sdc1 /mnt

Detach

# umount /mnt
# cryptsetup luksClose sdc1

dm-crypt without LUKS

# cryptsetup -y create sdc1 /dev/sdc1      # or any other partition like /dev/loop0
# dmsetup ls                               # check it, will display: sdc1 (254, 0)
# mkfs.ext3 /dev/mapper/sdc1               # This is done only the first time!
# mount -t ext3 /dev/mapper/sdc1 /mnt
# umount /mnt/
# cryptsetup remove sdc1                   # Detach the encrypted partition

Do exactly the same (without the mkfs part!) to re-attach the partition. If the password is not correct, the mount command will fail. In this case simply remove the map sdc1 (cryptsetup remove sdc1) and create it again.
FreeBSD

The two popular FreeBSD disk encryption modules are gbde and geli. I now use geli because it is faster and also uses the crypto device for hardware acceleration. See The FreeBSD handbook Chapter 18.6http://www.freebsd.org/handbook/disks-encrypting.html for all the details. The geli module must be loaded or compiled into the kernel:

options GEOM_ELI
device crypto                                       # or as module:
# echo 'geom_eli_load="YES"' >> /boot/loader.conf   # or do: kldload geom_eli

Use password and key

I use those settings for a typical disk encryption, it uses a passphrase AND a key to encrypt the master key. That is you need both the password and the generated key /root/ad1.key to attach the partition. The master key is stored inside the partition and is not visible. See below for typical USB or file based image.
Create encrypted partition

# dd if=/dev/random of=/root/ad1.key bs=64 count=1 # this key encrypts the mater key
# geli init -s 4096 -K /root/ad1.key /dev/ad1       # -s 8192 is also OK for disks
# geli attach -k /root/ad1.key /dev/ad1             # DO make a backup of /root/ad1.key
# dd if=/dev/random of=/dev/ad1.eli bs=1m           # Optional and takes a long time
# newfs /dev/ad1.eli                                # Create file system
# mount /dev/ad1.eli /mnt

Attach

# geli attach -k /root/ad1.key /dev/ad1
# fsck -ny -t ffs /dev/ad1.eli                      # In doubt check the file system
# mount /dev/ad1.eli /mnt

Detach

The detach procedure is done automatically on shutdown.

# umount /mnt
# geli detach /dev/ad1.eli

/etc/fstab

The encrypted partition can be configured to be mounted with /etc/fstab. The password will be prompted when booting. The following settings are required for this example:

# grep geli /etc/rc.conf
geli_devices="ad1"
geli_ad1_flags="-k /root/ad1.key"
# grep geli /etc/fstab
/dev/ad1.eli         /home/private              ufs             rw      0       0

Use password only

It is more convenient to encrypt a USB stick or file based image with a passphrase only and no key. In this case it is not necessary to carry the additional key file around. The procedure is very much the same as above, simply without the key file. Let's encrypt a file based image /cryptedfile of 1 GB.

# dd if=/dev/zero of=/cryptedfile bs=1M count=1000 # 1 GB file
# mdconfig -at vnode -f /cryptedfile
# geli init /dev/md0                                # encrypts with password only
# geli attach /dev/md0
# newfs -U -m 0 /dev/md0.eli
# mount /dev/md0.eli /mnt
# umount /dev/md0.eli
# geli detach md0.eli

It is now possible to mount this image on an other system with the password only.

# mdconfig -at vnode -f /cryptedfile
# geli attach /dev/md0
# mount /dev/md0.eli /mnt

OS X Encrypted Disk Image

Don't know by command line only. See OS X Encrypted Disk Imagehttps://wiki.thayer.dartmouth.edu/display/computing/Creating+a+Mac+OS+X+Encrypted+Disk+Image and Apple supporthttp://support.apple.com/kb/ht1578

SSL Certificates

So called SSL/TLS certificates are cryptographic public key certificates and are composed of a public and a private key. The certificates are used to authenticate the endpoints and encrypt the data. They are used for example on a web server (https) or mail server (imaps).
Procedure

    * We need a certificate authority to sign our certificate. This step is
        usually provided by a vendor like Thawte, Verisign, etc., however we can also create our own.

    * Create a certificate signing request. This request is like an unsigned certificate (the public part) and already contains all necessary information. The certificate request is normally sent to the authority vendor for signing. This step also creates the private key on the local machine.

    * Sign the certificate with the certificate authority.

    * If necessary join the certificate and the key in a single file to be used by the application (web server, mail server etc.).

Configure OpenSSL

We use /usr/local/certs as directory for this example check or edit /etc/ssl/openssl.cnf accordingly to your settings so you know where the files will be created. Here are the relevant part of openssl.cnf:

[ CA_default ]
dir             = /usr/local/certs/CA       # Where everything is kept
certs           = $dir/certs                # Where the issued certs are kept
crl_dir         = $dir/crl                  # Where the issued crl are kept
database        = $dir/index.txt            # database index file.

Make sure the directories exist or create them

# mkdir -p /usr/local/certs/CA
# cd /usr/local/certs/CA
# mkdir certs crl newcerts private
# echo "01" > serial                        # Only if serial does not exist
# touch index.txt

If you intend to get a signed certificate from a vendor, you only need a certificate signing request (CSR). This CSR will then be signed by the vendor for a limited time (e.g. 1 year).
Create a certificate authority

If you do not have a certificate authority from a vendor, you'll have to create your own. This step is not necessary if one intend to use a vendor to sign the request. To make a certificate authority (CA):

# openssl req -new -x509 -days 730 -config /etc/ssl/openssl.cnf \
-keyout CA/private/cakey.pem -out CA/cacert.pem

Create a certificate signing request

To make a new certificate (for mail server or web server for example), first create a request certificate with its private key. If your application do not support encrypted private key (for example UW-IMAP does not), then disable encryption with -nodes.

# openssl req -new -keyout newkey.pem -out newreq.pem \
-config /etc/ssl/openssl.cnf
# openssl req -nodes -new -keyout newkey.pem -out newreq.pem \
-config /etc/ssl/openssl.cnf                # No encryption for the key

Keep this created CSR (newreq.pem) as it can be signed again at the next renewal, the signature onlt will limit the validity of the certificate. This process also created the private key newkey.pem.

Sign the certificate

The certificate request has to be signed by the CA to be valid, this step is usually done by the vendor. Note: replace "servername" with the name of your server in the next commands.

# cat newreq.pem newkey.pem > new.pem
# openssl ca -policy policy_anything -out servernamecert.pem \
-config /etc/ssl/openssl.cnf -infiles new.pem
# mv newkey.pem servernamekey.pem

Now servernamekey.pem is the private key and servernamecert.pem is the server certificate.

Create united certificate

The IMAP server wants to have both private key and server certificate in the same file. And in general, this is also easier to handle, but the file has to be kept securely!. Apache also can deal with it well. Create a file servername.pem containing both the certificate and key.

    *   Open the private key (servernamekey.pem) with a text editor and copy the private key into the "servername.pem" file.

    *   Do the same with the server certificate (servernamecert.pem).

The final servername.pem file should look like this:

-----BEGIN RSA PRIVATE KEY-----
MIICXQIBAAKBgQDutWy+o/XZ/[...]qK5LqQgT3c9dU6fcR+WuSs6aejdEDDqBRQ
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIERzCCA7CgAwIBAgIBBDANB[...]iG9w0BAQQFADCBxTELMAkGA1UEBhMCREUx
-----END CERTIFICATE-----

What we have now in the directory /usr/local/certs/:

    *   CA/private/cakey.pem (CA server private key)

    *   CA/cacert.pem (CA server public key)

    *   certs/servernamekey.pem (server private key)

    *   certs/servernamecert.pem (server signed certificate)

    *   certs/servername.pem (server certificate with private key)

Keep the private key secure!

View certificate information

To view the certificate information simply do:

# openssl x509 -text -in servernamecert.pem      # View the certificate info
# openssl req -noout -text -in server.csr        # View the request info
# openssl s_client -connect cb.vu:443            # Check a web server certificate

CVS

Server setup | CVS test | SSH tunneling | CVS usage

Server setup

Initiate the CVS

Decide where the main repository will rest and create a root cvs. For example /usr/local/cvs (as root):

# mkdir -p /usr/local/cvs
# setenv CVSROOT /usr/local/cvs      # Set CVSROOT to the new location (local)
# cvs init                           # Creates all internal CVS config files
# cd /root
# cvs checkout CVSROOT               # Checkout the config files to modify them
# cd CVSROOT
edit config ( fine as it is)
# cvs commit config
cat >> writers                       # Create a writers file (optionally also readers)
colin
^D                                   # Use [Control][D] to quit the edit
# cvs add writers                    # Add the file writers into the repository
# cvs edit checkoutlist
# cat >> checkoutlist
writers
^D                                   # Use [Control][D] to quit the edit
# cvs commit                         # Commit all the configuration changes

Add a readers file if you want to differentiate read and write permissions Note: Do not (ever) edit files directly into the main cvs, but rather checkout the file, modify it and check it in. We did this with the file writers to define the write access.

There are three popular ways to access the CVS at this point. The first two don't need any further configuration. See the examples on CVSROOT below for how to use them:

    *   Direct local access to the file system. The user(s) need sufficient file permission to access the CS directly and there is no further authentication in addition to the OS login. However this is only useful if the repository is local.

    *   Remote access with ssh with the ext protocol. Any use with an ssh shell account and read/write permissions on the CVS server can access the CVS directly with ext over ssh without any additional tunnel. There is no server process running on the CVS for this to work. The ssh login does the authentication.

    *   Remote access with pserver (default port: 2401/tcp). This is the preferred use for larger user base as the users are authenticated by the CVS pserver with a dedicated password database, there is therefore no need for local users accounts. This setup is explained below.

Network setup with inetd

The CVS can be run locally only if a network access is not needed. For a remote access, the daemon inetd can start the pserver with the following line in /etc/inetd.conf (/etc/xinetd.d/cvs on SuSE):

cvspserver   stream tcp nowait cvs /usr/bin/cvs   cvs \
--allow-root=/usr/local/cvs pserver

It is a good idea to block the cvs port from the Internet with the firewall and use an ssh tunnel to access the repository remotely.

Separate authentication

It is possible to have cvs users which are not part of the OS (no local users). This is actually probably wanted too from the security point of view. Simply add a file named passwd (in the CVSROOT directory) containing the users login and password in the crypt format. This is can be done with the apache htpasswd tool.

Note: This passwd file is the only file which has to be edited directly in the CVSROOT directory. Also it won't be checked out. More info with htpasswd --help

# htpasswd -cb passwd user1 password1 # -c creates the file
# htpasswd -b passwd user2 password2

Now add :cvs at the end of each line to tell the cvs server to change the user to cvs (or whatever your cvs server is running under). It looks like this:

# cat passwd
user1:xsFjhU22u8Fuo:cvs
user2:vnefJOsnnvToM:cvs

Test it

Test the login as normal user (for example here me)

# cvs -d :pserver:colin@192.168.50.254:/usr/local/cvs login
Logging in to :pserver:colin@192.168.50.254:2401/usr/local/cvs
CVS password:

CVSROOT variable

This is an environment variable used to specify the location of the repository we're doing operations on. For local use, it can be just set to the directory of the repository. For use over the network, the transport protocol must be specified. Set the CVSROOT variable with setenv CVSROOT string on a csh, tcsh shell, or with export CVSROOT=string on a sh, bash shell.

# setenv CVSROOT :pserver:<username>@<host>:/cvsdirectory
For example:
# setenv CVSROOT /usr/local/cvs                               # Used locally only
# setenv CVSROOT :local:/usr/local/cvs                        # Same as above
# setenv CVSROOT :ext:user@cvsserver:/usr/local/cvs           # Direct access with SSH
# setenv CVS_RSH ssh                                          # for the ext access
# setenv CVSROOT :pserver:user@cvsserver.254:/usr/local/cvs   # network with pserver

When the login succeeded one can import a new project into the repository:
cd into your project root directory

cvs import <module name> <vendor tag> <initial tag>
cvs -d :pserver:colin@192.168.50.254:/usr/local/cvs import MyProject MyCompany START

Where MyProject is the name of the new project in the repository (used later to checkout). Cvs will import the current directory content into the new project.

To checkout:

# cvs -d :pserver:colin@192.168.50.254:/usr/local/cvs checkout MyProject
or
# setenv CVSROOT :pserver:colin@192.168.50.254:/usr/local/cvs
# cvs checkout MyProject

SSH tunneling for CVS

We need 2 shells for this. On the first shell we connect to the cvs server with ssh and port-forward the cvs connection. On the second shell we use the cvs normally as if it where running locally.

on shell 1:

# ssh -L2401:localhost:2401 colin@cvs_server   # Connect directly to the CVS server. Or:
# ssh -L2401:cvs_server:2401 colin@gateway     # Use a gateway to reach the CVS

on shell 2:

# setenv CVSROOT :pserver:colin@localhost:/usr/local/cvs
# cvs login
Logging in to :pserver:colin@localhost:2401/usr/local/cvs
CVS password:
# cvs checkout MyProject/src

CVS commands and usage

Import

The import command is used to add a whole directory, it must be run from within the directory to be imported. Say the directory /devel/ contains all files and subdirectories to be imported. The directory name on the CVS (the module) will be called "myapp".

# cvs import [options] directory-name vendor-tag release-tag
# cd /devel                          # Must be inside the project to import it
# cvs import myapp Company R1_0      # Release tag can be anything in one word

After a while a new directory "/devel/tools/" was added and it has to be imported too.

# cd /devel/tools
# cvs import myapp/tools Company R1_0

Checkout update add commit

# cvs co myapp/tools                 # Will only checkout the directory tools
# cvs co -r R1_1 myapp               # Checkout myapp at release R1_1 (is sticky)
# cvs -q -d update -P                # A typical CVS update
# cvs update -A                      # Reset any sticky tag (or date, option)
# cvs add newfile                    # Add a new file
# cvs add -kb newfile                # Add a new binary file
# cvs commit file1 file2             # Commit the two files only
# cvs commit -m "message"            # Commit all changes done with a message

Create a patch

It is best to create and apply a patch from the working development directory related to the project, or from within the source directory.

# cd /devel/project
# diff -Naur olddir newdir > patchfile # Create a patch from a directory or a file
# diff -Naur oldfile newfile > patchfile

Apply a patch

Sometimes it is necessary to strip a directory level from the patch, depending how it was created. In case of difficulties, simply look at the first lines of the patch and try -p0, -p1 or -p2.

# cd /devel/project
# patch --dry-run -p0 < patchfile    # Test the path without applying it
# patch -p0 < patchfile
# patch -p1 < patchfile              # strip off the 1st level from the path

SVN

Server setup | SVN+SSH | SVN over http | SVN usage

Subversion (SVN)http://subversion.tigris.org/ is a version control system designed to be the successor of CVS (Concurrent Versions System). The concept is similar to CVS, but many shortcomings where improved. See also the SVN bookhttp://svnbook.red-bean.com/en/1.4/.
Server setup

The initiation of the repository is fairly simple (here for example /home/svn/ must exist):

# svnadmin create --fs-type fsfs /home/svn/project1

Now the access to the repository is made possible with:

    *   file:// Direct file system access with the svn client with. This requires local permissions on the file system.

    *   svn:// or svn+ssh:// Remote access with the svnserve server (also over SSH). This requires local permissions on the file system (default port: 2690/tcp).

    *   http:// Remote access with webdav using apache. No local users are necessary for this method.

Using the local file system, it is now possible to import and then check out an existing project. Unlike with CVS it is not necessary to cd into the project directory, simply give the full path:

# svn import /project1/ file:///home/svn/project1/trunk -m 'Initial import'
# svn checkout file:///home/svn/project1

The new directory "trunk" is only a convention, this is not required.
Remote access with ssh

No special setup is required to access the repository via ssh, simply replace file:// with svn+ssh/hostname. For example:

# svn checkout svn+ssh://hostname/home/svn/project1

As with the local file access, every user needs an ssh access to the server (with a local account) and also read/write access. This method might be suitable for a small group. All users could belong to a subversion group which owns the repository, for example:

# groupadd subversion
# groupmod -A user1 subversion
# chown -R root:subversion /home/svn
# chmod -R 770 /home/svn

Remote access with http (apache)

Remote access over http (https) is the only good solution for a larger user group. This method uses the apache authentication, not the local accounts. This is a typical but small apache configuration:

LoadModule dav_module         modules/mod_dav.so
LoadModule dav_svn_module     modules/mod_dav_svn.so
LoadModule authz_svn_module   modules/mod_authz_svn.so    # Only for access control

<Location /svn>
DAV svn
# any "/svn/foo" URL will map to a repository /home/svn/foo
SVNParentPath /home/svn
AuthType Basic
AuthName "Subversion repository"
AuthzSVNAccessFile /etc/apache2/svn.acl
AuthUserFile /etc/apache2/svn-passwd
Require valid-user
</Location>

The apache server needs full access to the repository:

# chown -R www:www /home/svn

Create a user with htpasswd2:

# htpasswd -c /etc/svn-passwd user1 # -c creates the file

Access control svn.acl example

# Default it read access. "* =" would be default no access
[/]
* = r
[groups]
project1-developers = joe, jack, jane
# Give write access to the developers
[project1:]
@project1-developers = rw

SVN commands and usage

See also the Subversion Quick Reference Cardhttp://www.cs.put.poznan.pl/csobaniec/Papers/svn-refcard.pdf. Tortoise SVNhttp://tortoisesvn.tigris.org is a nice Windows interface.
Import

A new project, that is a directory with some files, is imported into the repository with the import command. Import is also used to add a directory with its content to an existing project.

# svn help import                                # Get help for any command
    # Add a new directory (with content) into the src dir on project1
# svn import /project1/newdir http://host.url/svn/project1/trunk/src -m 'add newdir'

Typical SVN commands

# svn co http://host.url/svn/project1/trunk      # Checkout the most recent version
    # Tags and branches are created by copying
# svn mkdir http://host.url/svn/project1/tags/   # Create the tags directory
# svn copy -m "Tag rc1 rel." http://host.url/svn/project1/trunk \
                             http://host.url/svn/project1/tags/1.0rc1
# svn status [--verbose]                         # Check files status into working dir
# svn add src/file.h src/file.cpp                # Add two files
# svn commit -m 'Added new class file'           # Commit the changes with a message
# svn ls http://host.url/svn/project1/tags/      # List all tags
# svn move foo.c bar.c                           # Move (rename) files
# svn delete some_old_file                       # Delete files

Useful Commands

less | vi | mail | tar | dd | screen | find | Miscellaneous

less

The less command displays a text document on the console. It is present on most installation.

# less unixtoolbox.xhtml

Some important commands are (^N stands for [control]-[N]):

    *   h H       good help on display

    *   f ^F ^V SPACE       Forward one window (or N lines).

    *   b ^B ESC-v       Backward one window (or N lines).

    *   F       Forward forever; like "tail -f".

    *   /pattern       Search forward for (N-th) matching line.

    *   ?pattern       Search backward for (N-th) matching line.

    *   n       Repeat previous search (for N-th occurrence).

    *   N       Repeat previous search in reverse direction.

    *   q       quit

vi

Vi is present on ANY Linux/Unix installation (not gentoo?) and it is therefore useful to know some basic commands. There are two modes: command mode and insertion mode. The commands mode is accessed with [ESC], the insertion mode with i. Use : help if you are lost.

The editors nano and pico are usually available too and are easier (IMHO) to use.
Quit

    *   :w newfilename       save the file to newfilename

    *   :wq or :x       save and quit

    *   :q!       quit without saving

Search and move

    *   /string       Search forward for string

    *   ?string       Search back for string

    *   n       Search for next instance of string

    *   N       Search for previous instance of string

    *   {       Move a paragraph back

    *   }       Move a paragraph forward

    *   1G       Move to the first line of the file

    *   nG       Move to the n th line of the file

    *   G       Move to the last line of the file

    *   :%s/OLD/NEW/g       Search and replace every occurrence

Delete copy paste text

    *   dd (dw)       Cut current line (word)

    *   D       Cut to the end of the line

    *   x       Delete (cut) character

    *   yy (yw)       Copy line (word) after cursor

    *   P       Paste after cursor

    *   u       Undo last modification

    *   U       Undo all changes to current line

mail

The mail command is a basic application to read and send email, it is usually installed. To send an email simply type "mail user@domain". The first line is the subject, then the mail content. Terminate and send the email with a single dot (.) in a new line. Example:

# mail c@cb.vu
Subject: Your text is full of typos
"For a moment, nothing happened. Then, after a second or so,
nothing continued to happen."
.
EOT
#

This is also working with a pipe:

# echo "This is the mail body" | mail c@cb.vu

This is also a simple way to test the mail server.

tar

The command tar (tape archive) creates and extracts archives of file and directories. The archive .tar is uncompressed, a compressed archive has the extension .tgz or .tar.gz (zip) or .tbz (bzip2). Do not use absolute path when creating an archive, you probably want to unpack it somewhere else. Some typical commands are:
Create

# cd /
# tar -cf home.tar home/        # archive the whole /home directory (c for create)
# tar -czf home.tgz home/       # same with zip compression
# tar -cjf home.tbz home/       # same with bzip2 compression

Only include one (or two) directories from a tree, but keep the relative structure. For example archive /usr/local/etc and /usr/local/www and the first directory in the archive should be local/.

# tar -C /usr -czf local.tgz local/etc local/www
# tar -C /usr -xzf local.tgz    # To untar the local dir into /usr
# cd /usr; tar -xzf local.tgz   # Is the same as above

Extract

# tar -tzf home.tgz             # look inside the archive without extracting (list)
# tar -xf home.tar              # extract the archive here (x for extract)
# tar -xzf home.tgz             # same with zip compression (-xjf for bzip2 compression)
                                # remove leading path gallery2 and extract into gallery
# tar --strip-components 1 -zxvf gallery2.tgz -C gallery/
# tar -xjf home.tbz home/colin/file.txt    # Restore a single file

More advanced

# tar c dir/ | gzip | ssh user@remote 'dd of=dir.tgz' # arch dir/ and store remotely.
# tar cvf - `find . -print` > backup.tar              # arch the current directory.
# tar -cf - -C /etc . | tar xpf - -C /backup/etc      # Copy directories
# tar -cf - -C /etc . | ssh user@remote tar xpf - -C /backup/etc      # Remote copy.
# tar -czf home.tgz --exclude '*.o' --exclude 'tmp/' home/

dd

The program dd (disk dump or destroy disk or see the meaning of dd) is used to copy partitions and disks and for other copy tricks. Typical usage:

# dd if=<source> of=<target> bs=<byte size> conv=<conversion>

Important conv options:

    *   notrunc       do not truncate the output file, all zeros will be written as zeros.

    *   noerror       continue after read errors (e.g. bad blocks)

    *   sync       pad every input block with Nulls to ibs-size

The default byte size is 512 (one block). The MBR, where the partition table is located, is on the first block, the first 63 blocks of a disk are empty. Larger byte sizes are faster to copy but require also more memory.
Backup and restore

# dd if=/dev/hda of=/dev/hdc bs=16065b                # Copy disk to disk (same size)
# dd if=/dev/sda7 of=/home/root.img bs=4096 conv=notrunc,noerror # Backup /
# dd if=/home/root.img of=/dev/sda7 bs=4096 conv=notrunc,noerror # Restore /
# dd bs=1M if=/dev/ad4s3e | gzip -c > ad4s3e.gz                  # Zip the backup
# gunzip -dc ad4s3e.gz | dd of=/dev/ad0s3e bs=1M                 # Restore the zip
# dd bs=1M if=/dev/ad4s3e | gzip | ssh eedcoba@fry 'dd of=ad4s3e.gz' # also remote
# gunzip -dc ad4s3e.gz | ssh eedcoba@host 'dd of=/dev/ad0s3e bs=1M'
# dd if=/dev/ad0 of=/dev/ad2 skip=1 seek=1 bs=4k conv=noerror    # Skip MBR
    # This is necessary if the destination (ad2) is smaller.

Recover

The command dd will read every single block of the partition. In case of problems it is better to use the option conv=sync,noerror so dd will skip the bad block and write zeros at the destination. Accordingly it is important to set the block size equal or smaller than the disk block size. A 1k size seems safe, set it with bs=1k. If a disk has bad sectors and the data should be recovered from a partition, create an image file with dd, mount the image and copy the content to a new disk. With the option noerror, dd will skip the bad sectors and write zeros instead, thus only the data contained in the bad sectors will be lost.

# dd if=/dev/hda of=/dev/null bs=1m                   # Check for bad blocks
# dd bs=1k if=/dev/hda1 conv=sync,noerror,notrunc | gzip | ssh \ # Send to remote
root@fry 'dd of=hda1.gz bs=1k'
# dd bs=1k if=/dev/hda1 conv=sync,noerror,notrunc of=hda1.img    # Store into an image
# mount -o loop /hda1.img /mnt                        # Mount the image
# rsync -ax /mnt/ /newdisk/                           # Copy on a new disk
# dd if=/dev/hda of=/dev/hda                          # Refresh the magnetic state
# The above is useful to refresh a disk. It is perfectly safe, but must be unmounted.

Delete

# dd if=/dev/zero of=/dev/hdc                         # Delete full disk
# dd if=/dev/urandom of=/dev/hdc                      # Delete full disk better
# kill -USR1 PID                                      # View dd progress (Linux)
# kill -INFO PID                                      # View dd progress (FreeBSD)

MBR tricks

The MBR contains the boot loader and the partition table and is 512 bytes small. The first 446 are for the boot loader, the bytes 446 to 512 are for the partition table.

# dd if=/dev/sda of=/mbr_sda.bak bs=512 count=1       # Backup the full MBR
# dd if=/dev/zero of=/dev/sda bs=512 count=1          # Delete MBR and partition table
# dd if=/mbr_sda.bak of=/dev/sda bs=512 count=1       # Restore the full MBR
# dd if=/mbr_sda.bak of=/dev/sda bs=446 count=1       # Restore only the boot loader
# dd if=/mbr_sda.bak of=/dev/sda bs=1 count=64 skip=446 seek=446 # Restore partition table

screen

Screen (a must have) has two main functionalities:

    * Run multiple terminal session within a single terminal.

    * A started program is decoupled from the real terminal and can thus run in the background. The real terminal can be closed and reattached later.

Short start example

start screen with:

# screen

Within the screen session we can start a long lasting program (like top).

# top

Now detach with Ctrl-a Ctrl-d. Reattach the terminal with:

# screen -R -D

In detail this means: If a session is running, then reattach. If necessary detach and logout remotely first. If it was not running create it and notify the user. Or:

# screen -x

Attach to a running screen in a multi display mode. The console is thus shared among multiple users. Very useful for team work/debug!

Screen commands (within screen)

All screen commands start with Ctrl-a.

    *   Ctrl-a ? help and summary of functions

    *   Ctrl-a c create an new window (terminal)

    *   Ctrl-a Ctrl-n and Ctrl-a Ctrl-p to switch to the next or previous
      window in the list, by number.

    *   Ctrl-a Ctrl-N where N is a number from 0 to 9, to switch to the corresponding window.

    *   Ctrl-a " to get a navigable list of running windows

    *   Ctrl-a a to clear a missed Ctrl-a

    *   Ctrl-a Ctrl-d to disconnect and leave the session running in the background

    *   Ctrl-a x lock the screen terminal with a password

    *   Ctrl-a [ enter into scrollback mode, exit with esc.

        Use echo "defscrollback 5000" > ~/.screenrc to increase buffer (default is 100)



          o     C-u Scrolls a half page up

          o     C-b Scroll a full page up

          o     C-d Scroll a half page down

          o     C-f Scroll a full page down

          o     / Search forward

          o     ? Search backward



The screen session is terminated when the program within the running terminal is closed and you logout from the terminal.

Find

Some important options:

    *   -x (on BSD) -xdev (on Linux)       Stay on the same file system (dev in fstab).

    *   -exec cmd {} \;       Execute the command and replace {} with the full path

    *   -iname       Like -name but is case insensitive

    *   -ls       Display information about the file (like ls -la)

    *   -size n       n is +-n (k M G T P)

    *   -cmin n       File's status was last changed n minutes ago.

# find . -type f ! -perm -444        # Find files not readable by all
# find . -type d ! -perm -111        # Find dirs not accessible by all
# find /home/user/ -cmin 10 -print   # Files created or modified in the last 10 min.
# find . -name '*.[ch]' | xargs grep -E 'expr' # Search 'expr' in this dir and below.
# find / -name "*.core" | xargs rm   # Find core dumps and delete them (also try core.*)
# find / -name "*.core" -print -exec rm {} \; # Other syntax
      # Find images and create an archive, iname is not case sensitive. -r for append
# find . $ -iname "*.png" -o -iname "*.jpg" $ -print -exec tar -rf images.tar {} \;
# find . -type f -name "*.txt" ! -name README.txt -print # Exclude README.txt files
# find /var/ -size +10M -exec ls -lh {} \;     # Find large files > 10 MB
# find /var/ -size +10M -ls           # This is simpler
# find . -size +10M -size -50M -print
# find /usr/ports/ -name work -type d -print -exec rm -rf {} \; # Clean the ports
      # Find files with SUID; those file are vulnerable and must be kept secure
# find / -type f -user root -perm -4000 -exec ls -l {} \;

Be careful with xarg or exec as it might or might not honor quotings and can return wrong results when files or directories contain spaces. In doubt use "-print0 | xargs -0" instead of "| xargs". The option -print0 must be the last in the find command. See this nice mini tutorial for findhttp://www.hccfl.edu/pollock/Unix/FindCmd.htm.

# find . -type f | xargs ls -l       # Will not work with spaces in names
# find . -type f -print0 | xargs -0 ls -l # Will work with spaces in names
# find . -type f -exec ls -l '{}' \; # Or use quotes '{}' with -exec

Miscellaneous

# which command                      # Show full path name of command
# time command                       # See how long a command takes to execute
# time cat                           # Use time as stopwatch. Ctrl-c to stop
# set | grep $USER                   # List the current environment
# cal -3                             # Display a three month calendar
# date [-u|--utc|--universal] [MMDDhhmm[[CC]YY][.ss]]
# date 10022155                      # Set date and time
# whatis grep                        # Display a short info on the command or word
# whereis java                       # Search path and standard directories for word
# setenv varname value               # Set env. variable varname to value (csh/tcsh)
# export varname="value"             # set env. variable varname to value (sh/ksh/bash)
# pwd                                # Print working directory
# mkdir -p /path/to/dir              # no error if existing, make parent dirs as needed
# mkdir -p project/{bin,src,obj,doc/{html,man,pdf},debug/some/more/dirs}
# rmdir /path/to/dir                 # Remove directory
# rm -rf /path/to/dir                # Remove directory and its content (force)
# rm -- -badchar.txt                 # Remove file whitch starts with a dash (-)
# cp -la /dir1 /dir2                 # Archive and hard link files instead of copy
# cp -lpR /dir1 /dir2                # Same for FreeBSD
# cp unixtoolbox.xhtml{,.bak}        # Short way to copy the file with a new extension
# mv /dir1 /dir2                     # Rename a directory
# ls -1                              # list one file per line
# history | tail -50                 # Display the last 50 used commands
# cd -                               # cd to previous ($OLDPWD) directory

Check file hashes with openssl. This is a nice alternative to the commands md5sum or sha1sum (FreeBSD uses md5 and sha1) which are not always installed.

# openssl md5 file.tar.gz            # Generate an md5 checksum from file
# openssl sha1 file.tar.gz           # Generate an sha1 checksum from file
# openssl rmd160 file.tar.gz         # Generate a RIPEMD-160 checksum from file

Install Software

Usually the package manager uses the proxy variable for http/ftp requests. In .bashrc:

export http_proxy=http://proxy_server:3128
export ftp_proxy=http://proxy_server:3128

List installed packages

# rpm -qa                            # List installed packages (RH, SuSE, RPM based)
# dpkg -l                            # Debian, Ubuntu
# pkg_info                           # FreeBSD list all installed packages
# pkg_info -W smbd                   # FreeBSD show which package smbd belongs to
# pkginfo                            # Solaris

Add/remove software

Front ends: yast2/yast for SuSE, redhat-config-packages for Red Hat.

# rpm -i pkgname.rpm                 # install the package (RH, SuSE, RPM based)
# rpm -e pkgname                     # Remove package

SuSE zypper (see doc and cheet sheet)http://en.opensuse.org/Zypper/Usage

# zypper refresh                     # Refresh repositorie
# zypper install vim                 # Install the package vim
# zypper remove vim                  # Remove the package vim
# zypper search vim                  # Search packages with vim
# zypper update vim                  # Search packages with vim

Debian

# apt-get update                     # First update the package lists
# apt-get install emacs              # Install the package emacs
# dpkg --remove emacs                # Remove the package emacs
# dpkg -S file                       # find what package a file belongs to

Gentoo

Gentoo uses emerge as the heart of its "Portage" package management system.

# emerge --sync                      # First sync the local portage tree
# emerge -u packagename              # Install or upgrade a package
# emerge -C packagename              # Remove the package
# revdep-rebuild                     # Repair dependencies

Solaris

The <cdrom> path is usually /cdrom/cdrom0.

# pkgadd -d <cdrom>/Solaris_9/Product SUNWgtar
# pkgadd -d SUNWgtar                 # Add downloaded package (bunzip2 first)
# pkgrm SUNWgtar                     # Remove the package

FreeBSD

# pkg_add -r rsync                   # Fetch and install rsync.
# pkg_delete /var/db/pkg/rsync-xx    # Delete the rsync package

Set where the packages are fetched from with the PACKAGESITE variable. For example:

# export PACKAGESITE=ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages/Latest/
# or ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-6-stable/Latest/

FreeBSD portshttp://www.freebsd.org/handbook/ports.html

The port tree /usr/ports/ is a collection of software ready to compile and install (see man ports). The ports are updated with the program portsnap.

# portsnap fetch extract             # Create the tree when running the first time
# portsnap fetch update              # Update the port tree
# cd /usr/ports/net/rsync/           # Select the package to install
# make install distclean             # Install and cleanup (also see man ports)
# make package                       # Make a binary package of this port
# pkgdb -F                           # Fix the package registry database
# portsclean -C -DD                  # Clean workdir and distdir (part of portupgrade)

OS X MacPortshttp://guide.macports.org/ (use sudo for all commands)

# port selfupdate                    # Update the port tree (safe)
# port installed                     # List installed ports
# port deps apache2                  # List dependencies for this port
# port search pgrep                  # Search for string
# port install proctools             # Install this package
# port variants ghostscript          # List variants of this port
# port -v install ghostscript +no_x11# -no_x11 for negative value
# port clean --all ghostscript       # Clean workdir of port
# port upgrade ghostscript           # Upgrade this port
# port uninstall ghostscript         # Uninstall this port
# port -f uninstall installed        # Uninstall everything

Library path

Due to complex dependencies and runtime linking, programs are difficult to copy to an other system or distribution. However for small programs with little dependencies, the missing libraries can be copied over. The runtime libraries (and the missing one) are checked with ldd and managed with ldconfig.

# ldd /usr/bin/rsync                 # List all needed runtime libraries
# otool -L /usr/bin/rsync            # OS X equivalent to ldd
# ldconfig -n /path/to/libs/         # Add a path to the shared libraries directories
# ldconfig -m /path/to/libs/         # FreeBSD
# LD_LIBRARY_PATH                    # The variable set the link library path

Convert Media

Sometimes one simply need to convert a video, audio file or document to another format.
Text encoding

Text encoding can get totally wrong, specially when the language requires
special characters like àäç. The command iconv can convert from
one encoding to an other.

# iconv -f <from_encoding> -t <to_encoding> <input_file>
# iconv -f ISO8859-1 -t UTF-8 -o file.input > file_utf8
# iconv -l                           # List known coded character sets

Without the -f option, iconv will use the local char-set, which is usually fine
if the document displays well.
Unix - DOS newlines

Convert DOS (CR/LF) to Unix (LF) newlines and back within a Unix shell. See also dos2unix and unix2dos if you have them.

# sed 's/.$//' dosfile.txt > unixfile.txt                  # DOS to UNIX
# awk '{sub(/\r$/,"");print}' dosfile.txt > unixfile.txt   # DOS to UNIX
# awk '{sub(/$/,"\r");print}' unixfile.txt > dosfile.txt   # UNIX to DOS

Convert Unix to DOS newlines within a Windows environment. Use sed or awk from mingw or cygwin.

# sed -n p unixfile.txt > dosfile.txt
# awk 1 unixfile.txt > dosfile.txt   # UNIX to DOS (with a cygwin shell)

PDF to Jpeg and concatenate PDF files

Convert a PDF document with gs (GhostScript) to jpeg (or png) images for each page. Also much shorter with convert and mogrify (from ImageMagick or GraphicsMagick).

# gs -dBATCH -dNOPAUSE -sDEVICE=jpeg -r150 -dTextAlphaBits=4 -dGraphicsAlphaBits=4 \
-dMaxStripSize=8192 -sOutputFile=unixtoolbox_%d.jpg unixtoolbox.pdf
# convert unixtoolbox.pdf unixtoolbox-%03d.png
# convert *.jpeg images.pdf          # Create a simple PDF with all pictures
# convert image000* -resample 120x120 -compress JPEG -quality 80 images.pdf
# mogrify -format png *.ppm          # convert all ppm images to png format

Ghostscript can also concatenate multiple pdf files into a single one. This only works well if the PDF files are "well behaved".

# gs -q -sPAPERSIZE=a4 -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=all.pdf \
file1.pdf file2.pdf ...              # On Windows use '#' instead of '='

Convert video

Compress the Canon digicam video with an mpeg4 codec and repair the crappy sound.

# mencoder -o videoout.avi -oac mp3lame -ovc lavc -srate 11025 \
-channels 1 -af-adv force=1 -lameopts preset=medium -lavcopts \
vcodec=msmpeg4v2:vbitrate=600 -mc 0 vidoein.AVI

See sox for sound processing.

Copy an audio cd

The program cdparanoiahttp://xiph.org/paranoia/ can save the audio tracks (FreeBSD port in audio/cdparanoia/), oggenc can encode in Ogg Vorbis format, lame converts to mp3.

# cdparanoia -B                      # Copy the tracks to wav files in current dir
# lame -b 256 in.wav out.mp3         # Encode in mp3 256 kb/s
# for i in *.wav; do lame -b 256 $i `basename $i .wav`.mp3; done
# oggenc in.wav -b 256 out.ogg       # Encode in Ogg Vorbis 256 kb/s

Printing

Print with lpr

# lpr unixtoolbox.ps                 # Print on default printer
# export PRINTER=hp4600              # Change the default printer
# lpr -Php4500 #2 unixtoolbox.ps     # Use printer hp4500 and print 2 copies
# lpr -o Duplex=DuplexNoTumble ...   # Print duplex along the long side
# lpr -o PageSize=A4,Duplex=DuplexNoTumble ...

# lpq                                # Check the queue on default printer
# lpq -l -Php4500                    # Queue on printer hp4500 with verbose
# lprm -                             # Remove all users jobs on default printer
# lprm -Php4500 3186                 # Remove job 3186. Find job nbr with lpq
# lpc status                         # List all available printers
# lpc status hp4500                  # Check if printer is online and queue length

Some devices are not postscript and will print garbage when fed with a pdf file. This might be solved with:

# gs -dSAFER -dNOPAUSE -sDEVICE=deskjet -sOutputFile=\|lpr file.pdf

Print to a PDF file even if the application does not support it. Use gs on the print command instead of lpr.

# gs -q -sPAPERSIZE=a4 -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=/path/file.pdf

Databases

PostgreSQL

Change root or a username password

# psql -d template1 -U pgsql
> alter user pgsql with password 'pgsql_password'; # Use username instead of "pgsql"

Create user and database

The commands createuser, dropuser, createdb and dropdb are convenient shortcuts equivalent to the SQL commands. The new user is bob with database bobdb ; use as root with pgsql the database super user:

# createuser -U pgsql -P bob         # -P will ask for password
# createdb -U pgsql -O bob bobdb     # new bobdb is owned by bob
# dropdb bobdb                       # Delete database bobdb
# dropuser bob                       # Delete user bob

The general database authentication mechanism is configured in pg_hba.conf
Grant remote access

The file $PGSQL_DATA_D/postgresql.conf specifies the address to bind to. Typically listen_addresses = '*' for Postgres 8.x.

The file $PGSQL_DATA_D/pg_hba.conf defines the access control. Examples:

# TYPE DATABASE    USER        IP-ADDRESS        IP-MASK          METHOD
host    bobdb       bob        212.117.81.42     255.255.255.255   password
host    all         all        0.0.0.0/0                           password

Backup and restore

The backups and restore are done with the user pgsql or postgres. Backup and restore a single database:

# pg_dump --clean dbname > dbname_sql.dump
# psql dbname < dbname_sql.dump

Backup and restore all databases (including users):

# pg_dumpall --clean > full.dump
# psql -f full.dump postgres

In this case the restore is started with the database postgres which is better when reloading an empty cluster.

MySQL

Change mysql root or username password

Method 1

# /etc/init.d/mysql stop
or
# killall mysqld
# mysqld --skip-grant-tables
# mysqladmin -u root password 'newpasswd'
# /etc/init.d/mysql start

Method 2

# mysql -u root mysql
mysql> UPDATE USER SET PASSWORD=PASSWORD("newpassword") where user='root';
mysql> FLUSH PRIVILEGES;                           # Use username instead of "root"
mysql> quit

Create user and database (see MySQL dochttp://dev.mysql.com/doc/refman/5.1/en/adding-users.html)

# mysql -u root mysql
mysql> CREATE USER 'bob'@'localhost' IDENTIFIED BY 'pwd'; # create only a user
mysql> CREATE DATABASE bobdb;
mysql> GRANT ALL ON *.* TO 'bob'@'%' IDENTIFIED BY 'pwd'; # Use localhost instead of %
                                                   # to restrict the network access
mysql> DROP DATABASE bobdb;                        # Delete database
mysql> DROP USER bob;                              # Delete user
mysql> DELETE FROM mysql.user WHERE user='bob and host='hostname'; # Alt. command
mysql> FLUSH PRIVILEGES;

Grant remote access

Remote access is typically permitted for a database, and not all databases. The file /etc/my.cnf contains the IP address to bind to. (On FreeBSD my.cnf not created per fedault, copy one .cnf file from /usr/local/share/mysql to /usr/local/etc/my.cnf) Typically comment the line bind-address = out.

# mysql -u root mysql
mysql> GRANT ALL ON bobdb.* TO bob@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'PASSWORD';
mysql> REVOKE GRANT OPTION ON foo.* FROM bar@'xxx.xxx.xxx.xxx';
mysql> FLUSH PRIVILEGES;                  # Use 'hostname' or also '%' for full access

Backup and restore

Backup and restore a single database:

# mysqldump -u root -psecret --add-drop-database dbname > dbname_sql.dump
# mysql -u root -psecret -D dbname < dbname_sql.dump

Backup and restore all databases:

# mysqldump -u root -psecret --add-drop-database --all-databases > full.dump
# mysql -u root -psecret < full.dump

Here is "secret" the mysql root password, there is no space after -p. When the -p option is used alone (w/o password), the password is asked at the command prompt.

SQLite

SQLitehttp://www.sqlite.org is a small powerful self-contained, serverless, zero-configuration SQL database.
Dump and restore

It can be useful to dump and restore an SQLite database. For example you can edit the dump file to change a column attribute or type and then restore the database. This is easier than messing with SQL commands. Use the command sqlite3 for a 3.x database.

# sqlite database.db .dump > dump.sql              # dump
# sqlite database.db < dump.sql                    # restore

Convert 2.x to 3.x database

sqlite database_v2.db .dump | sqlite3 database_v3.db

Disk Quota

A disk quota allows to limit the amount of disk space and/or the number of files a user or (or member of group) can use. The quotas are allocated on a per-file system basis and are enforced by the kernel.
Linux setup

The quota tools package usually needs to be installed, it contains the command line tools.

Activate the user quota in the fstab and remount the partition. If the partition is busy, either all locked files must be closed, or the system must be rebooted. Add usrquota to the fstab mount options, for example:

/dev/sda2     /home    reiserfs     rw,acl,user_xattr,usrquota 1 1
# mount -o remount /home
# mount                              # Check if usrquota is active, otherwise reboot

Initialize the quota.user file with quotacheck.

# quotacheck -vum /home
# chmod 644 /home/aquota.user        # To let the users check their own quota

Activate the quota either with the provided script (e.g. /etc/init.d/quotad on SuSE) or with quotaon:

quotaon -vu /home

Check that the quota is active with:

quota -v

FreeBSD setup

The quota tools are part of the base system, however the kernel needs the option quota. If it is not there, add it and recompile the kernel.

options QUOTA

As with Linux, add the quota to the fstab options (userquota, not usrquota):

/dev/ad0s1d    /home    ufs     rw,noatime,userquota    2 2
# mount /home                        # To remount the partition

Enable disk quotas in /etc/rc.conf and start the quota.

# grep quotas /etc/rc.conf
enable_quotas="YES"                  # turn on quotas on startup (or NO).
check_quotas="YES"                   # Check quotas on startup (or NO).
# /etc/rc.d/quota start

Assign quota limits

The quotas are not limited per default (set to 0). The limits are set with edquota for single users. A quota can be also duplicated to many users. The file structure is different between the quota implementations, but the principle is the same: the values of blocks and inodes can be limited. Only change the values of soft and hard. If not specified, the blocks are 1k. The grace period is set with edquota -t. For example:

# edquota -u colin

Linux

Disk quotas for user colin (uid 1007):
Filesystem         blocks       soft       hard     inodes     soft     hard
/dev/sda8            108       1000       2000          1        0        0

FreeBSD

Quotas for user colin:
/home: kbytes in use: 504184, limits (soft = 700000, hard = 800000)
   inodes in use: 1792, limits (soft = 0, hard = 0)

For many users

The command edquota -p is used to duplicate a quota to other users. For example to duplicate a reference quota to all users:

# edquota -p refuser `awk -F: '$3 > 499 {print $1}' /etc/passwd`
# edquota -p refuser user1 user2     # Duplicate to 2 users

Checks

Users can check their quota by simply typing quota (the file quota.user must be readable). Root can check all quotas.

# quota -u colin                     # Check quota for a user
# repquota /home                     # Full report for the partition for all users

Shells

Most Linux distributions use the bash shell while the BSDs use tcsh, the bourne shell is only used for scripts. Filters are very useful and can be piped:

    *   grep   Pattern matching

    *   sed   Search and Replace strings or characters

    *   cut   Print specific columns from a marker

    *   sort   Sort alphabetically or numerically

    *   uniq   Remove duplicate lines from a file

For example used all at once:

# ifconfig | sed 's/ / /g' | cut -d" " -f1 | uniq | grep -E "[a-z0-9]+" | sort -r
# ifconfig | sed '/.*inet addr:/!d;s///;s/ .*//'|sort -t. -k1,1n -k2,2n -k3,3n -k4,4n

The first character in the sed pattern is a tab. To write a tab on the console, use ctrl-v ctrl-tab.

bash

Redirects and pipes for bash and sh:

# cmd 1> file                         # Redirect stdout to file.
# cmd 2> file                         # Redirect stderr to file.
# cmd 1>> file                        # Redirect and append stdout to file.
# cmd &> file                         # Redirect both stdout and stderr to file.
# cmd >file 2>&1                      # Redirects stderr to stdout and then to file.
# cmd1 | cmd2                         # pipe stdout to cmd2
# cmd1 2>&1 | cmd2                    # pipe stdout and stderr to cmd2

Modify your configuration in ~/.bashrc (it can also be ~/.bash_profile). The following entries are useful, reload with ". .bashrc". With cygwin use ~/.bash_profile; with rxvt past with shift + left-click.

# in .bashrc
bind '"\e[A"':history-search-backward # Use up and down arrow to search
bind '"\e[B"':history-search-forward # the history. Invaluable!
set -o emacs                          # Set emacs mode in bash (see below)
set bell-style visible                # Do not beep, inverse colors
    # Set a nice prompt like [user@host]/path/todir>
PS1="\[\033[1;30m\][\[\033[1;34m\]\u\[\033[1;30m\]"
PS1="$PS1@\[\033[0;33m\]\h\[\033[1;30m\]]\[\033[0;37m\]"
PS1="$PS1\w\[\033[1;30m\]>\[\033[0m\]"

# To check the currently active aliases, simply type alias
alias ls='ls -aF'                    # Append indicator (one of */=>@|)
alias ll='ls -aFls'                  # Listing
alias la='ls -all'
alias ..='cd ..'
alias ...='cd ../..'
export HISTFILESIZE=5000              # Larger history
export CLICOLOR=1                     # Use colors (if possible)
export LSCOLORS=ExGxFxdxCxDxDxBxBxExEx

tcsh

Redirects and pipes for tcsh and csh (simple > and >> are the same as sh):

# cmd >& file                         # Redirect both stdout and stderr to file.
# cmd >>& file                        # Append both stdout and stderr to file.
# cmd1 | cmd2                         # pipe stdout to cmd2
# cmd1 |& cmd2                        # pipe stdout and stderr to cmd2

The settings for csh/tcsh are set in ~/.cshrc, reload with "source .cshrc". Examples:

# in .cshrc
alias ls      'ls -aF'
alias ll      'ls -aFls'
alias la      'ls -all'
alias ..      'cd ..'
alias ...     'cd ../..'
set   prompt    = "%B%n%b@%B%m%b%/> " # like user@host/path/todir>
set   history   = 5000
set   savehist = ( 6000 merge )
set   autolist                        # Report possible completions with tab
set   visiblebell                     # Do not beep, inverse colors

# Bindkey and colors
bindkey -e     Select Emacs bindings # Use emacs keys to edit the command prompt
bindkey -k up history-search-backward # Use up and down arrow to search
bindkey -k down history-search-forward
setenv CLICOLOR 1                     # Use colors (if possible)
setenv LSCOLORS ExGxFxdxCxDxDxBxBxExEx

The emacs mode enables to use the emacs keys shortcuts to modify the command prompt line. This is extremely useful (not only for emacs users). The most used commands are:

    *   C-a       Move cursor to beginning of line

    *   C-e       Move cursor to end of line

    *   M-b       Move cursor back one word

    *   M-f       Move cursor forward one word

    *   M-d       Cut the next word

    *   C-w       Cut the last word

    *   C-u       Cut everything before the cursor

    *   C-k       Cut everything after the cursor (rest of the line)

    *   C-y       Paste the last thing to be cut (simply paste)

    *   C-_       Undo

Note: C- = hold control, M- = hold meta (which is usually the alt or escape key).

Scripting

Basics | Script example | awk | sed | Regular Expressions | useful commands

The Bourne shell (/bin/sh) is present on all Unix installations and scripts written in this language are (quite) portable; man 1 sh is a good reference.

Basics

Variables and arguments

Assign with variable=value and get content with $variable

MESSAGE="Hello World"                        # Assign a string
PI=3.1415                                    # Assign a decimal number
N=8
TWON=`expr $N * 2`                           # Arithmetic expression (only integers)
TWON=$(($N * 2))                             # Other syntax
TWOPI=`echo "$PI * 2" | bc -l`               # Use bc for floating point operations
ZERO=`echo "c($PI/4)-sqrt(2)/2" | bc -l`

The command line arguments are

$0, $1, $2, ...                              # $0 is the command itself
$#                                           # The number of arguments
$*                                           # All arguments (also $@)

Special Variables

$$                                           # The current process ID
$?                                           # exit status of last command
command
if [ $? != 0 ]; then
    echo "command failed"
fi
mypath=`pwd`
mypath=${mypath}/file.txt
echo ${mypath##*/}                           # Display the filename only
echo ${mypath%%.*}                           # Full path without extention
var2=${var:=string}                          # Use var if set, otherwise use string
                                             # assign string to var and then to var2.

Constructs

for file in `ls`
do
    echo $file
done

count=0
while [ $count -lt 5 ]; do
    echo $count
    sleep 1
    count=$(($count + 1))
done

myfunction() {
    find . -type f -name "*.$1" -print       # $1 is first argument of the function
}
myfunction "txt"

Generate a file

MYHOME=/home/colin
cat > testhome.sh << _EOF
# All of this goes into the file testhome.sh
if [ -d "$MYHOME" ] ; then
    echo $MYHOME exists
else
    echo $MYHOME does not exist
fi
_EOF
sh testhome.sh

Bourne script example

As a small example, the script used to create a PDF booklet from this xhtml document:

#!/bin/sh
# This script creates a book in pdf format ready to print on a duplex printer
if [ $# -ne 1 ]; then                        # Check the argument
echo 1>&2 "Usage: $0 HtmlFile"
exit 1                                     # non zero exit if error
fi

file=$1                                      # Assign the filename
fname=${file%.*}                             # Get the name of the file only
fext=${file#*.}                              # Get the extension of the file

prince $file -o $fname.pdf                   # from www.princexml.com
pdftops -paper A4 -noshrink $fname.pdf $fname.ps # create postscript booklet
cat $fname.ps |psbook|psnup -Pa4 -2 |pstops -b "2:0,1U(21cm,29.7cm)" > $fname.book.ps

ps2pdf13 -sPAPERSIZE=a4 -sAutoRotatePages=None $fname.book.ps $fname.book.pdf
                                             # use #a4 and #None on Windows!
exit 0                                       # exit 0 means successful

Some awk commands

Awk is useful for field stripping, like cut in a more powerful way. Search this document for other examples. See for example gnulamp.com and one-liners for awk for some nice examples.

awk '{ print $2, $1 }' file                  # Print and inverse first two columns
awk '{printf("%5d : %s\n", NR,$0)}' file     # Add line number left aligned
awk '{print FNR "\t" $0}' files              # Add line number right aligned
awk NF test.txt                              # remove blank lines (same as grep '.')
awk 'length > 80'                            # print line longer than 80 char)

Some sed commands

Here is the one liner gold minehttp://student.northpark.edu/pemente/sed/sed1line.txt. And a good introduction and tutorial to sedhttp://www.grymoire.com/Unix/Sed.html.

sed 's/string1/string2/g'                    # Replace string1 with string2
sed -i 's/wroong/wrong/g' *.txt              # Replace a recurring word with g
sed 's/$.*$1/\12/g'                        # Modify anystring1 to anystring2
sed '/<p>/,/<\/p>/d' t.xhtml                 # Delete lines that start with <p>
                                             # and end with </p>
sed '/ *#/d; /^ *$/d'                        # Remove comments and blank lines
sed 's/[ \t]*$//'                            # Remove trailing spaces (use tab as \t)
sed 's/^[ \t]*//;s/[ \t]*$//'                # Remove leading and trailing spaces
sed 's/[^*]/[&]/'                            # Enclose first char with [] top->[t]op
sed = file | sed 'N;s/\n/\t/' > file.num     # Number lines on a file

Regular Expressions

Some basic regular expression useful for sed too. See Basic Regex Syntaxhttp://www.regular-expressions.info/reference.html for a good primer.

[\^$.|?*+()                          # special characters any other will match themselves
\                                    # escapes special characters and treat as literal
*                                    # repeat the previous item zero or more times
.                                    # single character except line break characters
.*                                   # match zero or more characters
^                                    # match at the start of a line/string
$                                    # match at the end of a line/string
.$                                   # match a single character at the end of line/string
^ $                                  # match line with a single space
[^A-Z]                               # match any line beginning with any char from A to Z

Some useful commands

The following commands are useful to include in a script or as one liners.

sort -t. -k1,1n -k2,2n -k3,3n -k4,4n         # Sort IPv4 ip addresses
echo 'Test' | tr '[:lower:]' '[:upper:]'     # Case conversion
echo foo.bar | cut -d . -f 1                 # Returns foo
PID=$(ps | grep script.sh | grep bin | awk '{print $1}')    # PID of a running script
PID=$(ps axww | grep [p]ing | awk '{print $1}')             # PID of ping (w/o grep pid)
IP=$(ifconfig $INTERFACE | sed '/.*inet addr:/!d;s///;s/ .*//')   # Linux
IP=$(ifconfig $INTERFACE | sed '/.*inet /!d;s///;s/ .*//')        # FreeBSD
if [ `diff file1 file2 | wc -l` != 0 ]; then [...] fi       # File changed?
cat /etc/master.passwd | grep -v root | grep -v \*: | awk -F":" \ # Create http passwd
'{ printf("%s:%s\n", $1, $2) }' > /usr/local/etc/apache2/passwd

testuser=$(cat /usr/local/etc/apache2/passwd | grep -v \    # Check user in passwd
root | grep -v \*: | awk -F":" '{ printf("%s\n", $1) }' | grep ^user$)
:(){ :|:& };:                                # bash fork bomb. Will kill your machine
tail +2 file > file2                         # remove the first line from file

I use this little trick to change the file extension for many files at once. For example from .cxx to .cpp. Test it first without the | sh at the end. You can also do this with the command rename if installed. Or with bash builtins.

# ls *.cxx | awk -F. '{print "mv "$0" "$1".cpp"}' | sh
# ls *.c | sed "s/.*/cp & &.$(date "+%Y%m%d")/" | sh # e.g. copy *.c to *.c.20080401
# rename .cxx .cpp *.cxx                             # Rename all .cxx to cpp
# for i in *.cxx; do mv $i ${i%%.cxx}.cpp; done      # with bash builtins

Programming

C basics

strcpy(newstr,str)                        /* copy str to newstr */
expr1 ? expr2 : expr3                     /* if (expr1) expr2 else expr3 */
x = (y > z) ? y : z;                      /* if (y > z) x = y; else x = z; */
int a[]={0,1,2};                          /* Initialized array (or a[3]={0,1,2}; */
int a[2][3]={{1,2,3},{4,5,6}};            /* Array of array of ints */
int i = 12345;                            /* Convert in i to char str */
char str[10];
sprintf(str, "%d", i);

C example

A minimal c program simple.c:

#include <stdio.h>
main() {
    int number=42;
    printf("The answer is %i\n", number);
}

Compile with:

# gcc simple.c -o simple
# ./simple
The answer is 42

C++ basics

*pointer                                  // Object pointed to by pointer
&obj                                      // Address of object obj
obj.x                                     // Member x of class obj (object obj)
pobj->x                                   // Member x of class pointed to by pobj
                                          // (*pobj).x and pobj->x are the same

C++ example

As a slightly more realistic program in C++: a class in its own header (IPv4.h) and implementation (IPv4.cpp) and a program which uses the class functionality. The class converts an IP address in integer format to the known quad format.
IPv4 class

IPv4.h:

#ifndef IPV4_H
#define IPV4_H
#include <string>

namespace GenericUtils {                          // create a namespace
class IPv4 {                                      // class definition
public:
    IPv4(); ~IPv4();
    std::string IPint_to_IPquad(unsigned long ip);// member interface
};
} //namespace GenericUtils
#endif // IPV4_H

IPv4.cpp:

#include "IPv4.h"
#include <string>
#include <sstream>
using namespace std;                              // use the namespaces
using namespace GenericUtils;

IPv4::IPv4() {}                                   // default constructor/destructor
IPv4::~IPv4() {}
string IPv4::IPint_to_IPquad(unsigned long ip) { // member implementation
    ostringstream ipstr;                          // use a stringstream
    ipstr << ((ip &0xff000000) >> 24)             // Bitwise right shift
          << "." << ((ip &0x00ff0000) >> 16)
          << "." << ((ip &0x0000ff00) >> 8)
          << "." << ((ip &0x000000ff));
    return ipstr.str();
}

The program simplecpp.cpp

#include "IPv4.h"
#include <iostream>
#include <string>
using namespace std;
int main (int argc, char* argv[]) {
    string ipstr;                                 // define variables
    unsigned long ipint = 1347861486;             // The IP in integer form
    GenericUtils::IPv4 iputils;                   // create an object of the class
    ipstr = iputils.IPint_to_IPquad(ipint);       // call the class member
    cout << ipint << " = " << ipstr << endl;      // print the result

    return 0;
}

Compile and execute with:

# g++ -c IPv4.cpp simplecpp.cpp                # Compile in objects
# g++ IPv4.o simplecpp.o -o simplecpp.exe      # Link the objects to final executable
# ./simplecpp.exe
1347861486 = 80.86.187.238

Use ldd to check which libraries are used by the executable and where they are located. Also used to check if a shared library is missing or if the executable is static.

# ldd /sbin/ifconfig                           # list dynamic object dependencies
# ar rcs staticlib.a *.o                       # create static archive
# ar t staticlib.a                             # print the objects list from the archive
# ar x /usr/lib/libc.a version.o               # extract an object file from the archive
# nm version.o                                 # show function members provided by object

Simple Makefile

The minimal Makefile for the multi-source program is shown below. The lines with instructions must begin with a tab! The back slash "\" can be used to cut long lines.

CC = g++
CFLAGS = -O
OBJS = IPv4.o simplecpp.o

simplecpp: ${OBJS}
   ${CC} -o simplecpp ${CFLAGS} ${OBJS}
clean:
   rm -f ${TARGET} ${OBJS}

Online Help

Documentation

Linux Documentation    en.tldp.org

Linux Man Pages    www.linuxmanpages.com

Linux commands directory    www.oreillynet.com/linux/cmd

Linux doc man howtos    linux.die.net

FreeBSD Handbook    www.freebsd.org/handbook

FreeBSD Man Pages    www.freebsd.org/cgi/man.cgi

FreeBSD user wiki    www.freebsdwiki.net

Solaris Man Pages    docs.sun.com/app/docs/coll/40.10

Other Unix/Linux references

Rosetta Stone for Unix    bhami.com/rosetta.html (a Unix command translator)

Unix guide cross reference    unixguide.net/unixguide.shtml

Linux commands line list    www.linuxcmd.org

Short Linux reference    www.pixelbeat.org/cmdline.html

Little command line goodies    www.shell-fu.org

http://cb.vu/unixtoolbox.xhtml#network

squid hardware

emaknyus@gmail.com (tamvan & verani) — Thu, 8 Sep 2011 08:42:00 -0700

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What I am describing below are changes to the default parameters that we needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different. For web pages there are lots and lots of rather small objects. In our applications there are relatively few objects, but they are much, much bigger. You should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below). Warning: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid might be feeding other squids then we highly recommend using our version as we have patches that have not yet (as of September 2010 at least) been incorporated by the squid project. UNDER NO CIRCUMSTANCES use version squid-3.X as it is missing features and bug fixes essential to Frontier.

So for our application:
Basically, you have:
1) Essential changes for squid-2.6.X or earlier are:

hierarchy stoplist cgi-bin
acl QUERY urlpath_regex cgi-bin

These two changes allow queries with ? in the URL to be cached. By default squid does not cache dynamic web pages. Since the Frontier information is generated by TOMCAT, technically they are dynamic web pages. Without these two changes, Frontier won't work. (The default values of these two lines are:

hierarchy stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?

so it's not much.)
Things have changed a little starting from squid-2.7.X. The "hierarchy stoplist" line still needs to be changed, but instead of the

acl QUERY urlpath_regex cgi-bin \?

line there is a line:

refresh_pattern -i (/cgi-bin/|\?) 0    0%      0

that needs to be changed to:

refresh_pattern -i /cgi-bin/    0       0%      0

2) Hardware dependent changes

cache_mem
cache_dir

You have to define a parameter cache_dir which tells the squid where to keep the information on disk and how large it should be. This should be at least 20000 megabytes but probably no more than 70% of the partition size (to allow room for log files and other uses). The other hardware parameter is cache_mem. The default cache_mem is only 8 MB and is probably a very old default and should be increased some, but we have found that squid performs better for large objects out of the disk cache than the memory cache. We recommend at most 1/8 of the physical RAM and no more than 128 MB, leaving a lot of memory for disk buffering.
3) Tuning changes

maximum_object_size 1048576 KB              !or whatever is necessary
maximum_object_size_in_memory 128 KB

icp_port 0                              !this just disables icp

The first item lets us cache objects up to 1 GB in size (which is a lot more than most web pages - the default is only 4 MB). To date, we have often cached tarballs up to 300 MB in size, and more might be needed. The second item lets us use the cache_mem we gave it above (the default is only 8 KB). icp is used to have peer caches communicate. If you have just one squid, it is safer just to turn it off.
An important option for squid-2.6 and later is:

collapsed_forwarding on

This option combines requests agressively so that a file is retrieved only once from the origin server. This is a very good idea for computer farms so make sure it is on. Finally, if your squid might possibly feed other squids then set this:

ignore_ims_on_miss on

The default for that option prevents caching when an upstream squid sends an If-Modified-Since request and the object isn't already cached. 4) Log file changes

strip_query_terms off
cache_store_log none

we do this for Frontier, but not essential
Since the log files can get very big, we run a cron job every night to rotate the log files, keeping 10 days worth. The cron job runs a script that looks like:

#!/bin/bash 

SQUID_DIR=/nthome/bjb/frontier/frontier-cache/squid
FNCRON_DIR=/nthome/bjb/frontier/frontier-cache/utils/cron

$SQUID_DIR/sbin/squid -k rotate 2>&1 >> $FNCRON_DIR/daily.log

You could also use a logrotate.d script.
Daily might not be enough, however. On heavily used squids we run another cron hourly that checks to see if access.log is greater than a chosen size and if so does an extra rotation.

That's basically it, except for one thing. For Frontier we monitor our squids remotely using SNMP. This is not turned on by default in squid, so to use it you have to turn it on at compilation time. If you want to use SNMP there are a few more settings needed in the squid.conf, especially the ACL access for whatever machines are allowed to read the SNMP information. Scientific Linux/Redhat Enterprise Linux squid rpms should already have SNMP enabled at compilation time. Therefore, it should be possible to enable monitoring by adding something like the following to your squid.conf:

snmp_port 3401

acl HOST_MONITOR src 131.225.240.232/32 127.0.0.1/32 frontier.cern.ch
acl HOST_MONITOR_NAME srcdomain cmsdbsfrontier.cern.ch
acl snmppublic snmp_community public

snmp_access allow snmppublic HOST_MONITOR
snmp_access allow snmppublic HOST_MONITOR_NAME
snmp_access deny all

all in the appropriate places in squid.conf. You may also need to open firewall and/or iptables holes for the addresses on the HOST_MONITOR line above. The 131.* address is for the monitor at Fermilab and the cern.ch names are for the monitor at CERN.
One thing you can do, is make a dummy installation of our tarball. It can be installed anywhere by any user. Then do a diff of your squid.conf with our squid.conf. For startup and shutdown procedures, you are on your own.
The compilation options we currently use are (Squid-2.6STABLE18 or later):
--disable-wccp --enable-snmp --disable-ident-lookups --with-large-files
(If you have a 64-bit OS, don't use --with-large-files)

Hardware

The first step is to decide what hardware you want to run the squid cache server on. These are some FAQ's
1) Do I need to dedicate a node to squid and only squid?
This is up to you. It is a good idea. It depends on how many jobs try to access the squid simultaneously and what else the machine is used for (see question 2). Large sites may need more than one squid(see question 4). The node needs to have network access to the internet, and be visible to the worker nodes.
2) What hardware specs (CPU, memory, diskcache)?
For most purposes 2GHZ, 2GB, 100 GB should be adequate. This excludes the space needed for log files which is determined by how heavily the system is used and what the clean up schedule is. We assume something like rotate the logs every day and remove after 10 days. From what we have seen, the most critical resource is the memory. If the machine serves other purposes, make sure the other tasks don't use up all the memory. We see 5-10 % performance gains when using a 64-bit architecture. Scientific Linux 4 should have better IO than Scientific Linux 3. Squid runs as a single thread, so if that is the only use of the machine, having more than 2 cores is a waste. You should also avoid AFS, NFS and RAID for the cache_dir.
Here is a description of squid memory usage: If you have a decent amount of spare memory, the kernel will use that as page cache, so it's a good chance that frequenty-requested items will, in fact, be served from RAM (via the page cache) even if it's not squid's RAM. There's also a design bottleneck in squid that limits cpu efficiency of large cache_mem objects, so resist the urge to give squid all your available memory. Let cache_mem handle your small objects and the kernel handle the larger ones.
3) What network specs (Gigabit if you have it)?
The latencies will be lower to the worker nodes if you have a large bandwidth. The network is always the bottleneck for this system, so "Gb if you got it" is the motto, but not an absolute requirement. If you have many job slots, 2 bonded gigabit network connections is even better, and squid on one core of a modern CPU can pretty much keep up with 2 gigabits. Squid is single-threaded so if you're able to supply more than 2 gigabits, multiple squid processes need to be used (consult with CMS frontier support for details if you want to try).
4) How many squids do I need?
Sites with over 500 job slots should have at least 2 squids for reliability. We currently estimate that sites should have one gigabit on a squid per 500-1000 CMS job slots. A lot depends on how quickly jobs start; an empty batch queue that suddenly fills up will need more squids. If you don't have gigabit ethernet you will be able to handle fewer job slots. The number of job slots that can be safely handled per gigabit increases as the number of slots increase because the chances that they all start at once tends to go down.
5) How should squids be load-balanced?
There are many ways to configure multiple squids: round-robin DNS, load-balancing networking hardware, LVS, etc. The simplest thing to do is just set up two or more squids independently and let Frontier handle it by making a small addition to site-local-config.xml (see below under Multiple Squid Servers). If there are many thousands of job slots, hardware-based load balancers can be easily overloaded so DNS-based or client-based load balancing will probably be called for.

Responsible: BarryBlumenfeld

linux network tune

emaknyus@gmail.com (tamvan & verani) — Thu, 8 Sep 2011 08:07:00 -0700

Don't try this yourself, you might break something. These recommendations were from Pete Vogel, who runs a bigger site than you do (can you do 85Mb/s on just one server? .. and he has many!!). Don't change these params unless you understand them. I am working on adding explainations. Amongst other things (like breaking your network connectivity), you might actually make your machine slower if you just blindly use them. In fact, I only set a couple of them. And always best to set one at a time to see if you notice the improvement you are looking for.

echo 65534 > /proc/sys/fs/file-max
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 0 > /proc/sys/net/ipv4/tcp_sack
echo 0 > /proc/sys/net/ipv4/tcp_ecn
echo 2 > /proc/sys/net/ipv4/tcp_orphan_retries
echo 1 > /proc/sys/net/ipv4/tcp_syncookies

/proc/sys/fs/file-max: This is basically the number of file descriptors available in the kernel. Which also effects the number of fd's a process can have open. For large sites you will definately need to upgrade this, and for some os'es you will need to use ulimit to increase the number of fd's available for that process. For red-hat (and other linux's) you can use:
```
echo 32767 > /proc/sys/fs/file-max
```
which I put into /etc/rc.d/rc.local (at/near the end). You can check your current usage using:
```
[cameron@jack cameron]$ cat /proc/sys/fs/file-nr
5941	2844	32767
[cameron@jack cameron]$ 
```
and what this means is that since the kernel has been running the max number of fd's in use was 5941, the current number is 2844, and the max available is 32767.
/proc/sys/net/ipv4/tcp_tw_recycle:
```
BOOLEAN
	Enable fast recycling TIME-WAIT sockets. Default value is 1.
	It should not be changed without advice/request of technical
	experts.
```
The definition of TIME_WAIT in "man netstat" is
```
       TIME_WAIT
              The socket is waiting after close to handle packets
              still in the network.
```
On Red Hat, the default is 0, so I checked on one of the loaded machines to see 2500 sockets in the TIME_WAIT state
```
[cameron@jack cameron]$ netstat -n | grep TIME_WAIT | wc
   2446   14676  198126
[cameron@jack cameron]$ netstat -n | wc
   2847   17104  230009
```
As you can see, for me, this takes up a lot of the network resources. By turning this on, I saw a slight drop in the number of connections (about 200), and a definate drop in the number of TIME_WAIT connections. I figured that this was good. I see that machine handling slightly more traffic, but my measuring mechanism is "poor". This is the next day after I made the change (at a busier time of the day).
```
[cameron@jack httpd-2.0.35]$ netstat -n | grep TIME_WAIT | wc ; netstat -n | wc
   2291   13746  185571
   2905   17452  234696
[cameron@jack httpd-2.0.35]$ 
```
So by turning it on, you should see a reduction in the number of FD's used in the kernel, but you might get some extra junk traffic??
/proc/sys/net/ipv4/tcp_window_scaling:
```
BOOLEN
	Enable window scaling as defined in RFC1323
```
These are recommendations to make TCP/IP work over very high speeds. I left the default at 1.

/proc/sys/net/ipv4/tcp_timestamps: Todo

BOOLEAN
	Enable timestamps as defined in RFC1323

an except:

      On the other hand, a Timestamps option may appear in any data or
      ACK segment, adding 12 bytes to the 20-byte TCP header.  We
      believe that the bandwidth saved by reducing unnecessary
      retransmissions will more than pay for the extra header bandwidth.

/proc/sys/net/ipv4/tcp_sack:

BOOLEAN
	Enable select acknowledgments (SACKS).

/proc/sys/net/ipv4/tcp_ecn:

BOOLEAN
	Enable Explicit Congestion Notification in TCP.

/proc/sys/net/ipv4/tcp_orphan_retries:

INTEGER
	How may times to retry before killing TCP connection, closed
	by our side. Default value 7 corresponds to ~50sec-16min
	depending on RTO. If you machine is loaded WEB server,
	you should think about lowering this value, such sockets
	may consume significant resources. Cf. tcp_max_orphans.

/proc/sys/net/ipv4/tcp_syncookies:

BOOLEAN
	Only valid when the kernel was compiled with CONFIG_SYNCOOKIES
	Send out syncookies when the syn backlog queue of a socket 
	overflows. This is to prevent against the common 'syn flood attack'
	Default: FALSE

	Note, that syncookies is fallback facility.
	It MUST NOT be used to help highly loaded servers to stand
	against legal connection rate. If you see synflood warnings
	in your logs, but investigation	shows that they occur
	because of overload with legal connections, you should tune
	another parameters until this warning disappear.
	See: tcp_max_syn_backlog, tcp_synack_retries, tcp_abort_on_overflow.

	syncookies seriously violate TCP protocol, do not allow
	to use TCP extensions, can result in serious degradation
	of some services (f.e. SMTP relaying), visible not by you,
	but your clients and relays, contacting you. While you see
	synflood warnings in logs not being really flooded, your server
	is seriously misconfigured.

MTU on DSL lines
It is said that DSL wraps packets in it's own packet. So for some DSL connections decreasing your MTU (Maximum Transfer Unit), from 1500 to 1490 is advantagous. You can do this using:

 ifconfig eth0 mtu 1490

Which I then added to /etc/rc.d/rc.local (for red hat 7.2) Links:

http://hr.uoregon.edu/davidrl/Documentation/networking/ip-sysctl.txt - local copy

Last Change: Sunday, 06-Dec-2009 20:07:29 EST
http://www.bloke.com/linux/kernel/network.html

mikrotik hardcoded

emaknyus@gmail.com (tamvan & verani) — Wed, 7 Sep 2011 18:38:00 -0700

1. Lan card ada 2
2. Alokasi IP address pada Mikrotik Hotspot Gateway :
- to_hotspot ip public
- to_hotspot 10.10.0.1/16

/ip firewall filter add chain=block-forward action=accept protocol=icmp icmp-options=8:0 limit=1,0 comment=”Allow ICMP”

/ip firewall filter add chain=block-forward action=log tcp-flags=syn protocol=tcp limit=5/1m,0 log-prefix=”Firewalled packet:” comment=”# Log Forward”

/ip firewall filter add chain=block-forward action=log protocol=icmp icmp-options=8:0 limit=10/1h,0 log-prefix=”PING of Death ???” comment=”# Log PING of Death”

/ip firewall filter add chain=block-forward action=drop src-address=169.254.0.0/16 comment=”# Block bad IP address”

/ip firewall filter add chain=block-forward action=drop src-address=224.0.0.0/3

/ip firewall filter add chain=block-forward action=drop src-address=224.0.0.0/4

/ip firewall filter add chain=block-forward action=drop src-address=240.0.0.0/5

/ip firewall filter add chain=block-forward action=drop src-address=248.0.0.0/5

/ip firewall filter add chain=block-forward action=drop src-address=255.255.255.255

/ip firewall filter add chain=block-forward action=drop connection-state=invalid comment=”# Drop invalid connections”

/ip firewall filter add chain=block-input action=accept protocol=icmp icmp-options=8:0 limit=5,0 comment=”# Allow ICMP”

/ip firewall filter add chain=block-input action=log protocol=icmp icmp-options=8:0 limit=10/1h,0 log-prefix=”PING of Death ???” comment=”# Log PING of Death”

/ip firewall filter add chain=block-input action=log tcp-flags=syn protocol=tcp limit=5/1m,0 log-prefix=”Firewalled packet:” comment=”# Log input”

/ip firewall filter add chain=block-input action=drop protocol=icmp comment=”# Block PING of Death”

/ip firewall filter add chain=block-input action=drop in-interface=to_internet src-address=10.0.0.0/8 comment=”# Interface facing Public Internet Inbound Section RFC 1918″

/ip firewall filter add chain=block-input action=drop in-interface=to_internet src-address=202.91.11.6

/ip firewall filter add chain=block-input action=drop in-interface=to_internet src-address=192.168.0.0/16

/ip firewall filter add chain=block-input action=drop in-interface=to_internet src-address=202.91.8.160/29

/ip firewall filter add chain=block-input action=drop src-address=0.0.0.0/8

/ip firewall filter add chain=input action=jump jump-target=block-input comment=”# Jump to block-input”

/ip firewall filter add chain=forward action=jump jump-target=block-forward comment=”# Jump to block-forward”

/ip firewall filter add chain=forward action=jump jump-target=block-ddos protocol=udp comment=”# Jump to block-ddos”

/ip firewall filter add chain=forward action=jump jump-target=block-service comment=”# Jump to block-service”

/ip firewall filter add chain=block-service action=drop p2p=all-p2p comment=”# Block ports you don’t want it insert here ..:: drop p2p ::..”

/ip firewall filter add chain=block-service action=drop out-interface=to_internet dst-port=5050 protocol=tcp comment=”..:: drop yahoo messenger ::..”

/ip firewall filter add chain=block-service action=drop out-interface=to_internet dst-port=6666-7000 protocol=tcp comment=”..:: drop irc ::..”

/ip firewall filter add chain=block-input action=drop src-address=127.0.0.0/8

/ip firewall filter add chain=block-input action=drop src-address=169.254.0.0/16

/ip firewall filter add chain=block-input action=drop src-address=172.16.0.0/12

/ip firewall filter add chain=block-input action=drop src-address=192.0.2.0/24

/ip firewall filter add chain=block-input action=drop src-address=204.152.64.0/23

/ip firewall filter add chain=block-input action=drop src-address=224.0.0.0/3

/ip firewall filter add chain=block-input action=drop src-address=224.0.0.0/4

/ip firewall filter add chain=block-input action=drop src-address=240.0.0.0/5

/ip firewall filter add chain=block-input action=drop src-address=248.0.0.0/5

/ip firewall filter add chain=block-input action=drop src-address=255.255.255.255

/ip firewall filter add chain=block-input action=drop in-interface=to_internet src-address=202.91.8.167 comment=”# Block Smurf Attack on all interface”

/ip firewall filter add chain=block-input action=drop in-interface=to_internet src-address=202.91.11.7

/ip firewall filter add chain=block-input action=drop in-interface=to_hotspot src-address=10.10.255.255

/ip firewall filter add chain=block-input action=drop in-interface=to_proxy src-address=192.168.100.255

/ip firewall filter add chain=block-input action=drop dst-port=113 protocol=tcp comment=”# Block Ident”

/ip firewall filter add chain=block-input action=drop fragment=yes comment=”# Block IP Frags”

/ip firewall filter add chain=block-input action=drop ipv4-options=loose-source-routing comment=”# Blocked source routed packets lsrr and ssrr”

/ip firewall filter add chain=block-input action=drop ipv4-options=strict-source-routing

/ip firewall filter add chain=block-input action=reject reject-with=tcp-reset in-interface=to_internet dst-port=80 protocol=tcp comment=”# Reject connection from internet to port 80”

/ip firewall filter add chain=block-forward action=reject reject-with=tcp-reset tcp-flags=syn,ack connection-state=new protocol=tcp comment=”# Reject Bad TCP”

/ip firewall filter add chain=block-forward action=drop protocol=icmp comment=”# Block PING of Death”

/ip firewall filter add chain=block-input action=drop in-interface=to_internet dst-port=23 protocol=tcp comment=”# Block Telnet come from Internet”

/ip firewall filter add chain=block-input action=drop in-interface=to_hotspot dst-port=23 protocol=tcp comment=”# Block Telnet come from Hotspot”

/ip firewall filter add chain=block-forward action=drop dst-port=137-139 protocol=tcp comment=”# Block Netbios Sessions tcp”

/ip firewall filter add chain=block-input action=drop dst-port=137-139 protocol=tcp comment=”# Block Netbios Sessions tcp”

/ip firewall filter add chain=block-input action=drop dst-port=137-139 protocol=udp comment=”# Block Netbios Sessions udp”

/ip firewall filter add chain=block-forward action=drop dst-port=135-139 protocol=udp comment=”# Block Netbios Sessions udp”

/ip firewall filter add chain=block-ddos action=return limit=16,32

/ip firewall filter add chain=block-ddos action=log log-prefix=”DDOS ATTACK:”

/ip firewall filter add chain=block-ddos action=drop limit=16,32

/ip firewall filter add chain=input action=jump jump-target=block-ddos protocol=udp comment=”# Jump to block-ddos”

/ip firewall filter add chain=tcp-flags action=jump jump-target=bad-flags tcp-flags=fin,!ack protocol=tcp comment=”# tcp-flags jumpt to bad-flags”

/ip firewall filter add chain=tcp-flags action=jump jump-target=bad-flags tcp-flags=psh,!ack protocol=tcp

/ip firewall filter add chain=tcp-flags action=jump jump-target=bad-flags tcp-flags=urg,!ack protocol=tcp

/ip firewall filter add chain=tcp-flags action=jump jump-target=bad-flags tcp-flags=fin,rst protocol=tcp

/ip firewall filter add chain=tcp-flags action=jump jump-target=bad-flags tcp-flags=fin,syn protocol=tcp

/ip firewall filter add chain=tcp-flags action=jump jump-target=bad-flags tcp-flags=syn,rst protocol=tcp

/ip firewall filter add chain=tcp-flags action=jump jump-target=bad-flags tcp-flags=fin,syn,rst,psh,ack,urg,ece,cwr protocol=tcp

/ip firewall filter add chain=tcp-flags protocol=tcp action=jump jump-target=bad-flags tcp-flags=!fin,!syn,!rst,!psh,!ack,!urg,!ece,!cwr

/ip firewall filter add chain=tcp-flags protocol=tcp action=jump jump-target=bad-flags tcp-flags=fin,psh,urg,!syn,!rst,!ack,!ece,!cwr

/ip firewall filter add chain=tcp-flags action=jump jump-target=bad-flags tcp-flags=fin,syn,psh,urg,!rst,!ack,!ece,!cwr protocol=tcp

/ip firewall filter add chain=tcp-flags action=jump jump-target=bad-flags tcp-flags=fin,syn,rst,ack,urg,!psh,!ece,!cwr protocol=tcp

/ip firewall filter add chain=input action=jump jump-target=tcp-flags protocol=tcp comment=”# Jump to tcp-flags”

/ip firewall filter add chain=bad-flags action=log log-prefix=”TCP BAD FLAGS:”

/ip firewall filter add chain=bad-flags action=drop

/ip firewall filter add chain=syn-flood action=return tcp-flags=syn,!fin,!rst,!ack protocol=tcp limit=5,10

/ip firewall filter add chain=syn-flood action=return protocol=!tcp

/ip firewall filter add chain=syn-flood action=return tcp-flags=!,syn,!fin,!rst,!ack protocol=tcp

/ip firewall filter add chain=syn-flood action=log log-prefix=”SYN FLOOD:”

/ip firewall filter add chain=syn-flood action=drop

/ip firewall filter add chain=input action=jump jump-target=syn-flood tcp-flags=syn,!fin,!rst,!ack protocol=tcp comment=”# Jump to syn-flood”

/ip firewall filter add chain=block-forward action=drop dst-port=445 protocol=tcp comment=”# Block CIFS tcp”

/ip firewall filter add chain=block-forward action=drop dst-port=135 protocol=tcp comment=”# Block RPC Portmapper”

/ip firewall filter add chain=block-forward action=drop dst-port=135 protocol=udp

/ip firewall filter add chain=block-forward action=drop dst-port=111 protocol=tcp

/ip firewall filter add chain=block-input action=drop dst-port=135 protocol=tcp comment=”# Block RPC Portmapper”

/ip firewall filter add chain=block-input action=drop dst-port=135 protocol=udp

/ip firewall filter add chain=block-input action=drop dst-port=111 protocol=tcp

/ip firewall filter add chain=block-input action=drop dst-port=111 protocol=udp

/ip firewall filter add chain=block-input action=drop dst-port=445 protocol=tcp comment=”# Block CIFS tcp”

/ip firewall filter add chain=block-forward action=drop dst-port=111 protocol=udp

/ip firewall filter add chain=block-forward action=drop dst-port=69 protocol=tcp comment=”# Block TFTP”

/ip firewall filter add chain=block-input action=drop dst-port=69 protocol=tcp comment=”# Block TFTP”

/ip firewall filter add chain=block-input action=drop dst-port=69 protocol=udp

/ip firewall filter add chain=block-input action=drop dst-port=20034 protocol=tcp comment=”# Block Netbus”

/ip firewall filter add chain=block-input action=drop dst-port=20034 protocol=udp

/ip firewall filter add chain=block-input action=drop dst-port=3133 protocol=tcp comment=”# Block Back0riffice”

/ip firewall filter add chain=block-input action=drop dst-port=3133 protocol=udp

/ip firewall filter add chain=block-forward action=drop dst-port=69 protocol=udp

/ip firewall filter add chain=block-forward action=drop dst-port=20034 protocol=tcp comment=”# Block Netbus”

/ip firewall filter add chain=block-forward action=drop dst-port=20034 protocol=udp

/ip firewall filter add chain=block-forward action=drop dst-port=3133 protocol=tcp comment=”# Block Back0riffice”

/ip firewall filter add chain=block-forward action=drop dst-port=3133 protocol=udp

/ip firewall filter add chain=block-input action=drop dst-port=2049 protocol=udp comment=”# Block NFS”

/ip firewall filter add chain=block-input action=drop dst-port=2049 protocol=tcp

/ip firewall filter add chain=block-forward action=drop dst-port=2049 protocol=udp comment=”# Block NFS”

/ip firewall filter add chain=block-forward action=drop dst-port=2049 protocol=tcp

/ip firewall filter add chain=block-service action=drop out-interface=to_internet dst-port=1818 protocol=tcp comment=”# Block Game online”

/ip firewall filter add chain=block-service action=drop out-interface=to_internet dst-port=6112 protocol=tcp

Stop Denial of Service attacks

emaknyus@gmail.com (tamvan & verani) — Wed, 7 Sep 2011 18:37:00 -0700

Taking advantage of a known problem with the Operating System or any running services on the target, a good programmer can build an application that sends some data that causes the targeted system to crash.

The worst case scenario is not when a hacker crashes a service, but when it finds a way to maximize the CPU usage causing a total malfunction on the system.

Of course most of the attacks are not initiated by these kind of programmers, but by "script kiddies" who only read about it, find the program's source over the web and just use it without knowing the mechanisms it uses.

Usually, DoS attacks are closely related to brute force attacks. The brute force attack uses all combinations of possible characters or dictionary word lists to try find out any passwords on the system (eg root accounts). When a root account in a network is found out, any DoS attacks can be done easily over the specific network.

Here are some of the most known DoS attacks:

1. SYN Floods

You should know that when a client and a server want to transmit data over the TCP protocol, a three-way handshake occurs:

The client asks for a connection with a SYN (synchronize) package
The server replies to the client with a SYN-ACK (syn-acknowledgments)
The client sends a third packages as a ACK and the transmission of the data starts.

The SYN flood works by sending SYN packets from false IP addresses (IP spoofing). The server replies to that false IP address with an SYN-ACK and then waits for ACK. Doing this many times will cause the server to end up in the impossibility of opening new connection, creating a network congestion.

Another SYN flood attack involves sending a packet to the server, spoofed with the server's address (let's say the server's IP is 192.168.1.20 then you send a SYN packet from 192.168.1.20 to 192.168.1.20). Repeating this many times will make the server sending SYN-ACK and ACK to itself, blocking it.

Patches to this kind of attack used a connection number limit from the same source/timeframe. SYN cookies also hold down the handling of the packets until the sender's IP address is verified.

2. SMURF attacks

In this kind of attacks a massive amount of ping traffic (ICMP echos) is sent to the broadcast address of the network. The source IP address is spoofed to look like the target's. If this traffic is forwarded to the network, all hosts will reply with an echo to the target, believing that they receive an echo request (PING) from it. In a large networks, a targeted server for example can be flooded by hundreds of replies at once. By sending the spoofed packet several times, the server will be flooded until it crashes from the overload.

This kind of attacks were mostly patched by making the routers not forwarding broadcast directed traffic to the network.

3. LAND attacks

LAND attacks take advantage of opened network services on the target. By using a port sniffer, opened ports and services are found out. Then spoofed packages are sent with IP address source the same as IP address destination (server's address) to make it reply to itself. Let's say for example that it uses SNMP (simple network management protocol - service used to report network and system's usage). By making a SNMP service to reply to itself continuously it finally crashes.

4. Ping of death

This type of DoS attack takes advantage of a known issue with Windows 9x and older NT stations, as well as Linux prior to 2.0.32. Many routers and printers older then 1998 are vulnerable to this too.
It works by sending a malformed format of a ping packet. Usually, ping packets are small-sized (like 32bytes or 64bytes by default). Older Operating Systems and other devices could not handle ping larger than the maximum IP packet size of 65535 bytes (defined by RFC 791). By sending a large packet or a malformed one, any system that doesn't know how to handle it crashes (eg. in Windows 9x a blue screen of death was generated).
Patches are available on the web for any old operating systems or devices.

5. Ping flooding

This is probably the simplest DoS attack that exists. It is also the most used. It works by overwhelming the target with echo requests (pings) having large packets. The target has it's bandwidth occupied by these requests already and floods itself by starting to reply back. Of course, the attacker must have a larger bandwidth than the target (for example flooding a dial-up user from a 1Mbps connection).
With the increase of the servers' bandwidth, this type of attacks became useless for an ADSL user for instance.
The "problem" was solved by using multiple hosts, creating the first DDoS attacks (distributed denial of service).
DDoS attacks work by owning let's say 50 boxes each with 1Mbps bandwidth. Then the attacker uses all of them to ping flood the target, creating a great amount of traffic on the host.
Stacheldraht for example is a console that connects to owned boxes running Stacheldraht server. It then coordinates the attacks from a single point.
The solution to this type of attacks is the firewall, which filters any echo replies from being sent. Of course, firewalls can be crashed as well.

6. Fraggle attacks

A fraggle attack takes place when an attacker send massive amount of UDP echo data to network broadcast addresses, using a the target's IP as the packet's source. All hosts reply to the target, flooding it. It usually uses UDP PORT 7 (echo). This code was written by the same person who written the smurf attack.

7. Teardrop attacks

This attack involves packets sent by the attacker to the target with oversized payloads. This exploits a bug in the TCP/IP protocol stack, crashing the system. Only Windows 3.11, 95 and Linux prior to 2.0.32 were vulnerable to this kind of attack.

8. Other type of attacks

Other type of attacks involve application flooding, like IRC bot raw line which usually crash Windows boxes running mIRC or any other client. These attacks are based on a greater number of raw socket transactions than a computer can handle.

http://www.tutorial5.com/content/view/80/79/

Drop Akses Client Untuk Traceroute

emaknyus@gmail.com (tamvan & verani) — Wed, 7 Sep 2011 18:34:00 -0700

Mungkin karena adanya suatu alasan penting sehingga client dalam suatu jaringan dilarang untuk melakukan traceroute keluar. Jika memang ini harus diperlukan maka, kita harus mempertimbangkannya kembali, mengingat tidak ada yang harus dikhawatirkan terhadap traceroute ini. Kembali kepada permasalahan awal, mungkin dikarenakan adanya suatu alasan yang penting sekali sehingga akses untuk traceroute pada client pun harus di block ~_~

Rule dibawah ini akan melakukan drop kepada client yang mencoba untuk melakukan traceroute keluar (forward):

/ip firewall filter

add chain=forward protocol=icmp \

icmp-options=11:0 action=drop

add chain=forward protocol=icmp \

icmp-options=3:3 action=drop

Dengan rule diatas, semua akses keluar dari client akan di drop ketika melakukan traceroute. Jika menginginkan ip tertentu saja yang hendak didrop cukup menambahkan src-address=ip yang dimaksud. Sebagai contoh pada ip client 192.168.10.20 akses untuk traceroute di block, maka rulenya menjadi:

/ip firewall filter

add chain=forward src-address=192.168.10.20 \

protocol=icmp icmp-options=11:0 action=drop

add chain=forward src-address=192.168.10.20 \

protocol=icmp icmp-options=3:3 action=drop

Jika ip tertentu lebih dari satu, maka gunakan src-address-list=nama_list_address_nya dengan catatan list-address yang dimaksud telah ditambahkan ip tertentu nya.

Salam Codet.

http://feeds.feedburner.com/blogspot/KLBuX

Block TCP Flag rst

emaknyus@gmail.com (tamvan & verani) — Wed, 7 Sep 2011 18:33:00 -0700

Mengapa TCP Flag rst diblock ? Untuk penjelasan lanjutan tentang TCP Flag rst, silahkan dibuka kembali halaman Transmission Control Protocol Flag, jika dirasa tidak diperlukan, maka sebaiknya di block saja karena dikhawatirkan hanya akan menjadi sebab terjadinya kesalahan konfigurasi atau kesalahan-kesalahan lainnya mengingat sifat dasar dari tcp flag rst ini adalah respon atas permintaan pembuatan koneksi.

Cukup ketikkan rule dibawah ini pada MikroTik, maka akses tcp dengan flag rst telah dapat diblock.

/ip firewall filter

add chain=input protocol=tcp tcp-flags=rst \

action=drop comment="block tcp rst"

Dan letakkan diatas input dengan action drop lainnya.

Mencegah User Agar Tidak Bisa Membuka LAN Setting Pada Window Dengan Hide Tool

emaknyus@gmail.com (tamvan & verani) — Wed, 7 Sep 2011 18:31:00 -0700

Terkadang ada diantara beberapa user yang sekedar iseng mengganti ip-address nya sendiri, entah ingin tahu atau ada maksud lain. Tetapi pada sisi admin dan system hal tersebut sedikit menjengkelkan karena dengan bergantinya ip address tersebut ada beberapa konfigurasi yang tidak berjalan dengan semestinya.

Dengan Aplikasi Hide Tool ini, itu bisa diamankankan dengan menutup dialog box Internet Protocol (TCP/IP) pada LAN Connection Properties.

Bagi anda yang belum mengetahui tentang Hide Tool, silahkan membuka artikel pada blog ini. Download dan install Aplikasi tersebut.

Buka dialog Box TCP/IP Properties, Lihat gambar dan catatlah header nya (yang dilingkari dengan warna merah).

Kemudian buka Hide Tool, pada menu "Block Windows" tambahkan (add) tulisan header yang dicatat tersebut ke dalam list block windows. Close, dan jangan lupa untuk mengaktifkannya.

Ketika user membuka TCP/IP Properties, maka akan tertutup secara otomatis, sehingga tidak bisa mengganti Ip address. Cara yang sama bisa dilakukan terhadap Dos-Prompt untuk menghindari pergantian IP melalui dos-prompt tersebut, dan berlaku pula cara yang sama terhadap aplikasi lainnya didalam windows yang memiliki header (judul) pada dialog box nya.

Tipe-Tipe Serangan Denial Of Service Berikut Penangkalnya Pada MikroTik

emaknyus@gmail.com (tamvan & verani) — Wed, 7 Sep 2011 18:30:00 -0700

Denial of service yang biasa disingkat dan disebut DOS (bukan dos-prompt microsoft) merupakan salah satu tipe serangan attacker kearah jaringan yang bertujuan untuk menghentikan sementara atau selamanya fungsi operasi dari sebuah system.

Secara umum ada 4 macam jenis serangan DOS, walaupun tidak menutup kemungkinan terjadinya flood atau jenis serangan DOS lainnya. Sekaligus disini juga akan dituliskan rule penangkalnya pada MikroTik Router dengan asumsi MikroTik RouterOS telah difungsikan sebagai router sebelumnya serta jalur koneksi telah benar (invalid, stabilished, related dan sebagainya). 4 type serangan DOS terdiri dari:

1. Ping Of Death.

Attackers mengirimkan serangkaian paket data ke target yang tidak sesuai ketentuan aturan jaringan. Jika secara terus menerus bisa mengakibatkan jalur koneksi penuh dan berakibat drop nya server karena tidak bisa menampung kebutuhan tersebut.

Penangkalnya, buat rule dengan accept icmp yang ter-filter:

/ip firewall filter

add chain=input protocol=icmp action=jump jump-target=icmp

add chain=icmp protocol=icmp icmp-options=0:0-255 limit=5,5 action=accept comment="Limited Ping Flood" disabled=no

add chain=icmp protocol=icmp icmp-options=3:3 limit=5,5 action=accept comment="" disabled=no

add chain=icmp protocol=icmp icmp-options=3:4 limit=5,5 action=accept comment="" disabled=no

add chain=icmp protocol=icmp icmp-options=8:0-255 limit=5,5 action=accept comment="" disabled=no

add chain=icmp protocol=icmp icmp-options=11:0-255 limit=5,5 action=accept comment="" disabled=no

add chain=icmp protocol=icmp action=drop comment="" disabled=no

2. SYN Flood.

Attackers mengirimkan paket TCP SYN ke objek target, sehingga berdampak objek target mengalami hang karena harus menyelesaikan proses koneksi dalam jumlah yang tak terbatas dan berakibat resources yang dimiliki mengalami drop.

Penangkalnya, buat rule dengan accept TCP-SYN yang ter-filter:

/ip firewall filter

add chain=input protocol=tcp tcp-flags=syn connection-state=new action=jump jump-target=SYN-Protect comment="Flood protect" disabled=no

add chain=SYN-Protect protocol=tcp tcp-flags=syn limit=400,5 connection-state=new action=accept comment="" disabled=no

add chain=SYN-Protect protocol=tcp tcp-flags=syn connection-state=new action=drop comment="" disabled=no

3. Land/Lattiera.

Serangan yang dilakukan oleh Attackers dengan mengirimkan paket SYN, yang memiliki alamat port/source yang sama dengan objek target.

Penangkalnya, lakukan hal yang sama pada cara menangkal SYN-FLOOD, hanya saja disini ditambahkan "chain forward" akses SYN, yang mana dibelokkan (jump) ke "chain SYN-Protect". Sehingga rule-nya menjadi seperti berikut:

/ip firewall filter

add chain=forward protocol=tcp tcp-flags=syn connection-state=new action=jump jump-target=SYN-Protect comment="Flood protect" disabled=no

add chain=input protocol=tcp tcp-flags=syn connection-state=new action=jump jump-target=SYN-Protect comment="" disabled=no

add chain=SYN-Protect protocol=tcp tcp-flags=syn limit=400,5 connection-state=new action=accept comment="" disabled=no

add chain=SYN-Protect protocol=tcp tcp-flags=syn connection-state=new action=drop comment="" disabled=no

4. WinNuke.

Pengiriman data OOB/ORG pada koneksi dengan menggunakan TCP ke port 139 (NetBios Session/SMB) yang berakibat OS (dalam hal ini adalah windows) mengalami hang.

Penangkalnya, tutup protocol tcp yang menggunakan port 139, begitu juga protocol udp port 139 untuk menghindari hal yang tidak diinginkan:

/ip web-proxy direct

add dst-port=139 action=deny comment="" disabled=no

/ip firewall mangle

add chain=prerouting protocol=tcp dst-port=139 action=mark-connection new-connection-mark=drop-port

add chain=prerouting protocol=udp dst-port=139 action=mark-connection new-connection-mark=drop-port

/ip firewall filter

add chain=input connection-mark=drop-port action=drop comment="drop port" disabled=no

add chain=forward connection-mark=drop-port action=drop comment="" disabled=no

Sengaja ditambahkan rule "forward" yang mana artinya koneksi tersebut berasal dari jaringan dalam, untuk menghindari hal yang tidak diinginkan, terkadang Attacker juga bisa berasal dari dalam jaringan kita sendiri.

Salam Codet

http://feeds.feedburner.com/blogspot/KLBuX

https://www.facebook.com/pages/GreenBerret/206210029408584

Mikrotik rogue dhcp detection

emaknyus@gmail.com (tamvan & verani) — Tue, 6 Sep 2011 18:55:00 -0700

Mikrotik actually has a rogue detection service you can configure, but as we have found running apartment complexes, it can give you false positives. To combat this, I’ve come up with a new method.
First configure syslog exporting with the DHCP service dumping. Next enable the dhcp client to run on all your inside facing interfaces. I’ve got my Cacti syslog server set to match “dhcp,info,debug dhcp-client%got ip address”, which is the message sent when the dhcp client receives an IP address. Once the syslog server processes the message it sends us an alert.

Add dhcp-client to each interface, be sure to disable default route, peer dns and peer ntp.

1
2
3

/ ip dhcp-client 
add interface=vlan10 comment="" disabled=no 
add interface=ether2 comment="" disabled=no

This script will need to be scheduled to run around every hour or so. It will release the dhcp reservation on your interfaces. Otherwise it won’t attempt to pull a new address until the old allocation has expired, which can be up to a year. It loops through releasing from all your interfaces.

:log info ("dhcp detect release")
:for e from=0 to=40 do={
/ip dhcp-client release ($e)
}

Once you get an alert from your syslog server, you log into the Mikrotik and issue the:

1	/ip dhcp-client print detail

You’ll get the following:

Flags: X – disabled, I – invalid
0 interface=vlan10 status=bound address=192.168.1.100/24
gateway=192.168.1.1 dhcp-server=192.168.1.1
primary-dns=209.189.224.45 secondary-dns=209.189.224.40
expires-after=21h13m2s

Take the dhcp-server address and use it below:

1	/ip arp print where address="192.168.1.1"

You will get the following result:

Flags: X – disabled, I – invalid, H – DHCP, D – dynamic
# ADDRESS MAC-ADDRESS INTERFACE
0 D 192.168.0.253 00:08:74:4B:7F:BC ether1

Now you track this guy down and shut him off at the switch port, or if you are using mac-track in cacti, you simply look him up, connect to the proper switch and kill him. You could also use this in conjunction with the standard rogue detection service to more quickly find the MAC address

http://gregsowell.com/?p=349

Mikrotik port knocking with ICMP Ping

emaknyus@gmail.com (tamvan & verani) — Tue, 6 Sep 2011 18:54:00 -0700

Port knocking is a very easy and simple way to secure your network services. There are many ways to use port knocking: using a series of UDP packes towards different ports with some content, or just sending a series of SYN packets on different ports.
These methods usually requires a program to do the task.
Ping, however, is usually available from any device.

We now want to open for SSH in to our Mikrotik by pinging the WAN adresse with a packet size of 5000 :

Type in your favorite number / code + 28
i.e: my favorite number is 5000, but because of some overhead we have to add 28 to the number
5000 + 28 = 5028

Then you have to add you src address to an address list:

Now make a new rule:

Add the src address list :

Testing:

Perfect!
http://www.datapels.com/?p=162

implementation prerouting and postrouting

emaknyus@gmail.com (tamvan & verani) — Mon, 5 Sep 2011 23:50:00 -0700

The DNAT target is used to do Destination Network Address Translation, which means that it is used to rewrite the Destination IP address of a packet. If a packet is matched, and this is the target of the rule, the packet, and all subsequent packets in the same stream will be translated, and then routed on to the correct device, host or network. This target can be extremely useful, for example,when you have a host running your web server inside a LAN, but no real IP to give it that will work on the Internet. You could then tell the firewall to forward all packets going to its own HTTP port, on to the real web server within the LAN. We may also specify a whole range of destination IP addresses, and the DNAT mechanism will choose the destination IP address at random for each stream. Hence, we will be able to deal with a kind of load balancing by doing this.

Note that the DNAT target is only available within the PREROUTING and OUTPUT chains in the nat table, and any of the chains called upon from any of those listed chains. Note that chains containing DNAT targets may not be used from any other chains, such as the POSTROUTING chain.

Table 11-5. DNAT target options

Option	--to-destination
Example	iptables -t nat -A PREROUTING -p tcp -d 15.45.23.67 --dport 80 -j DNAT --to-destination 192.168.1.1-192.168.1.10
Explanation	The --to-destination option tells the DNAT mechanism which Destination IP to set in the IP header, and where to send packets that are matched. The above example would send on all packets destined for IP address 15.45.23.67 to a range of LAN IP's, namely 192.168.1.1 through 10. Note, as described previously, that a single stream will always use the same host, and that each stream will randomly be given an IP address that it will always be Destined for, within that stream. We could also have specified only one IP address, in which case we would always be connected to the same host. Also note that we may add a port or port range to which the traffic would be redirected to. This is done by adding, for example, an :80 statement to the IP addresses to which we want to DNAT the packets. A rule could then look like --to-destination 192.168.1.1:80 for example, or like --to-destination 192.168.1.1:80-100 if we wanted to specify a port range. As you can see, the syntax is pretty much the same for the DNAT target, as for the SNAT target even though they do two totally different things. Do note that port specifications are only valid for rules that specify the TCP or UDP protocols with the --protocol option.

Since DNAT requires quite a lot of work to work properly, I have decided to add a larger explanation on how to work with it. Let's take a brief example on how things would be done normally. We want to publish our website via our Internet connection. We only have one IP address, and the HTTP server is located on our internal network. Our firewall has the external IP address $INET_IP, and our HTTP server has the internal IP address $HTTP_IP and finally the firewall has the internal IP address $LAN_IP. The first thing to do is to add the following simple rule to the PREROUTING chain in the nat table:

iptables -t nat -A PREROUTING --dst $INET_IP -p tcp --dport 80 -j DNAT \
--to-destination $HTTP_IP

Now, all packets from the Internet going to port 80 on our firewall are redirected (or DNAT'ed) to our internal HTTP server. If you test this from the Internet, everything should work just perfect. So, what happens if you try connecting from a host on the same local network as the HTTP server? It will simply not work. This is a problem with routing really. We start out by dissecting what happens in a normal case. The external box has IP address $EXT_BOX, to maintain readability.

Packet leaves the connecting host going to $INET_IP and source $EXT_BOX.
Packet reaches the firewall.
Firewall DNAT's the packet and runs the packet through all different chains etcetera.
Packet leaves the firewall and travels to the $HTTP_IP.
Packet reaches the HTTP server, and the HTTP box replies back through the firewall, if that is the box that the routing database has entered as the gateway for $EXT_BOX. Normally, this would be the default gateway of the HTTP server.
Firewall Un-DNAT's the packet again, so the packet looks as if it was replied to from the firewall itself.
Reply packet travels as usual back to the client $EXT_BOX.

Now, we will consider what happens if the packet was instead generated by a client on the same network as the HTTP server itself. The client has the IP address $LAN_BOX, while the rest of the machines maintain the same settings.

Packet leaves $LAN_BOX to $INET_IP.
The packet reaches the firewall.
The packet gets DNAT'ed, and all other required actions are taken, however, the packet is not SNAT'ed, so the same source IP address is used on the packet.
The packet leaves the firewall and reaches the HTTP server.
The HTTP server tries to respond to the packet, and sees in the routing databases that the packet came from a local box on the same network, and hence tries to send the packet directly to the original source IP address (which now becomes the destination IP address).
The packet reaches the client, and the client gets confused since the return packet does not come from the host that it sent the original request to. Hence, the client drops the reply packet, and waits for the "real" reply.

The simple solution to this problem is to SNAT all packets entering the firewall and leaving for a host or IP that we know we do DNAT to. For example, consider the above rule. We SNAT the packets entering our firewall that are destined for $HTTP_IP port 80 so that they look as if they came from $LAN_IP. This will force the HTTP server to send the packets back to our firewall, which Un-DNAT's the packets and sends them on to the client. The rule would look something like this:

iptables -t nat -A POSTROUTING -p tcp --dst $HTTP_IP --dport 80 -j SNAT \
--to-source $LAN_IP

Remember that the POSTROUTING chain is processed last of the chains, and hence the packet will already be DNAT'ed once it reaches that specific chain. This is the reason that we match the packets based on the internal address.

This last rule will seriously harm your logging, so it is really advisable not to use this method, but the whole example is still a valid one. What will happen is this, packet comes from the Internet, gets SNAT'ed and DNAT'ed, and finally hits the HTTP server (for example). The HTTP server now only sees the request as if it was coming from the firewall, and hence logs all requests from the internet as if they came from the firewall.
This can also have even more severe implications. Take an SMTP server on the LAN, that allows requests from the internal network, and you have your firewall set up to forward SMTP traffic to it. You have now effectively created an open relay SMTP server, with horrenduously bad logging!
One solution to this problem is to simply make the SNAT rule even more specific in the match part, and to only work on packets that come in from our LAN interface. In other words, add a --src $LAN_IP_RANGE to the whole command as well. This will make the rule only work on streams that come in from the LAN, and hence will not affect the Source IP, so the logs will look correct, except for streams coming from our LAN.
You will, in other words, be better off solving these problems by either setting up a separate DNS server for your LAN, or to actually set up a separate DMZ, the latter being preferred if you have the money.

You think this should be enough by now, and it really is, unless considering one final aspect to this whole scenario. What if the firewall itself tries to access the HTTP server, where will it go? As it looks now, it will unfortunately try to get to its own HTTP server, and not the server residing on $HTTP_IP. To get around this, we need to add a DNAT rule in the OUTPUT chain as well. Following the above example, this should look something like the following:

iptables -t nat -A OUTPUT --dst $INET_IP -p tcp --dport 80 -j DNAT \
--to-destination $HTTP_IP

Adding this final rule should get everything up and running. All separate networks that do not sit on the same net as the HTTP server will run smoothly, all hosts on the same network as the HTTP server will be able to connect and finally, the firewall will be able to do proper connections as well. Now everything works and no problems should arise.

Everyone should realize that these rules only affect how the packet is DNAT'ed and SNAT'ed properly. In addition to these rules, you may also need extra rules in the filter table (FORWARD chain) to allow the packets to traverse through those chains as well. Don't forget that all packets have already gone through the PREROUTING chain, and should hence have their destination addresses rewritten already by DNAT.

Works under Linux kernel 2.3, 2.4, 2.5 and 2.6.

http://www.iptables.info/en/iptables-targets-and-jumps.html#DNATTARGET

mikrotik nat

emaknyus@gmail.com (tamvan & verani) — Mon, 5 Sep 2011 22:18:00 -0700

Source NAT

If you want to "hide" the private LAN 192.168.0.0/24 "behind" one address 10.5.8.109 given to you by the ISP, you should use the source network address translation (masquerading) feature of the MikroTik router. The masquerading will change the source IP address and port of the packets originated from the network 192.168.0.0/24 to the address 10.5.8.109 of the router when the packet is routed through it.
To use masquerading, a source NAT rule with action 'masquerade' should be added to the firewall configuration:

/ip firewall nat add chain=srcnat action=masquerade out-interface=Public

All outgoing connections from the network 192.168.0.0/24 will have source address 10.5.8.109 of the router and source port above 1024. No access from the Internet will be possible to the Local addresses. If you want to allow connections to the server on the local network, you should use destination Network Address Translation (NAT).

Destination NAT

If you want to link Public IP 10.5.8.200 address to Local one 192.168.0.109, you should use destination address translation feature of the MikroTik router. Also if you want allow Local server to talk with outside with given Public IP you should use source address translation, too.
Add Public IP to Public interface:

/ip address add address=10.5.8.200/32 interface=Public

Add rule allowing access to the internal server from external networks:

/ip firewall nat add chain=dstnat dst-address=10.5.8.200 action=dst-nat \
	to-addresses=192.168.0.109

Add rule allowing the internal server to talk to the outer networks having its source address translated to 10.5.8.200:

/ip firewall nat add chain=srcnat src-address=192.168.0.109 action=src-nat \
	to-addresses=10.5.8.200

1:1 mapping

If you want to link Public IP subnet 11.11.11.0/24 to local one 2.2.2.0/24, you should use destination address translation and source address translation features with action=netmap.

/ip firewall nat add chain=dstnat dst-address=11.11.11.1-11.11.11.254 \
	action=netmap to-addresses=2.2.2.1-2.2.2.254

/ip firewall nat add chain=srcnat src-address=2.2.2.1-2.2.2.254 \
	action=netmap to-addresses=11.11.11.1-11.11.11.254

Port mapping

If you would like to direct requests for a certain port to an internal machine (sometimes called opening a port, port mapping), you can do it like this:

/ip firewall nat add chain=dstnat dst-port=1234 action=dst-nat protocol=tcp to-address=192.168.1.1 to-port=1234

This rule translates to: when an incoming connection requests TCP port 1234, use the DST-NAT action and redirect it to local address 192.168.1.1 and the port 1234

Saying How To Mangle The Packets

emaknyus@gmail.com (tamvan & verani) — Mon, 5 Sep 2011 22:13:00 -0700

So now we know how to select the packets we want to mangle. To complete our rule, we need to tell the kernel exactly what we want it to do to the packets.

6.1 Source NAT

You want to do Source NAT; change the source address of connections to something different. This is done in the POSTROUTING chain, just before it is finally sent out; this is an important detail, since it means that anything else on the Linux box itself (routing, packet filtering) will see the packet unchanged. It also means that the `-o' (outgoing interface) option can be used.
Source NAT is specified using `-j SNAT', and the `--to-source' option specifies an IP address, a range of IP addresses, and an optional port or range of ports (for UDP and TCP protocols only).

## Change source addresses to 1.2.3.4.
# iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to 1.2.3.4

## Change source addresses to 1.2.3.4, 1.2.3.5 or 1.2.3.6
# iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to 1.2.3.4-1.2.3.6

## Change source addresses to 1.2.3.4, ports 1-1023
# iptables -t nat -A POSTROUTING -p tcp -o eth0 -j SNAT --to 1.2.3.4:1-1023

Masquerading

There is a specialized case of Source NAT called masquerading: it should only be used for dynamically-assigned IP addresses, such as standard dialups (for static IP addresses, use SNAT above).
You don't need to put in the source address explicitly with masquerading: it will use the source address of the interface the packet is going out from. But more importantly, if the link goes down, the connections (which are now lost anyway) are forgotten, meaning fewer glitches when connection comes back up with a new IP address.

## Masquerade everything out ppp0.
# iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE

6.2 Destination NAT

This is done in the PREROUTING chain, just as the packet comes in; this means that anything else on the Linux box itself (routing, packet filtering) will see the packet going to its `real' destination. It also means that the `-i' (incoming interface) option can be used.
Destination NAT is specified using `-j DNAT', and the `--to-destination' option specifies an IP address, a range of IP addresses, and an optional port or range of ports (for UDP and TCP protocols only).

## Change destination addresses to 5.6.7.8
# iptables -t nat -A PREROUTING -i eth0 -j DNAT --to 5.6.7.8

## Change destination addresses to 5.6.7.8, 5.6.7.9 or 5.6.7.10.
# iptables -t nat -A PREROUTING -i eth0 -j DNAT --to 5.6.7.8-5.6.7.10

## Change destination addresses of web traffic to 5.6.7.8, port 8080.
# iptables -t nat -A PREROUTING -p tcp --dport 80 -i eth0 \
        -j DNAT --to 5.6.7.8:8080

Redirection

There is a specialized case of Destination NAT called redirection: it is a simple convenience which is exactly equivalent to doing DNAT to the address of the incoming interface.

## Send incoming port-80 web traffic to our squid (transparent) proxy
# iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 \
        -j REDIRECT --to-port 3128

Note that squid needs to be configured to know it's a transparent proxy!

6.3 Mappings In Depth

There are some subtleties to NAT which most people will never have to deal with. They are documented here for the curious.

Selection Of Multiple Addresses in a Range

If a range of IP addresses is given, the IP address to use is chosen based on the least currently used IP for connections the machine knows about. This gives primitive load-balancing.

Creating Null NAT Mappings

You can use the `-j ACCEPT' target to let a connection through without any NAT taking place.

Standard NAT Behavior

The default behavior is to alter the connection as little as possible, within the constraints of the rule given by the user. This means we won't remap ports unless we have to.

Implicit Source Port Mapping

Even when no NAT is requested for a connection, source port translation may occur implicitly, if another connection has been mapped over the new one. Consider the case of masquerading, which is rather common:

A web connection is established by a box 192.1.1.1 from port 1024 to www.netscape.com port 80.
This is masqueraded by the masquerading box to use its source IP address (1.2.3.4).
The masquerading box tries to make a web connection to www.netscape.com port 80 from 1.2.3.4 (its external interface address) port 1024.
The NAT code will alter the source port of the second connection to 1025, so that the two don't clash.

When this implicit source mapping occurs, ports are divided into three classes:

Ports below 512
Ports between 512 and 1023
Ports 1024 and above.

A port will never be implicitly mapped into a different class.

What Happens When NAT Fails

If there is no way to uniquely map a connection as the user requests, it will be dropped. This also applies to packets which could not be classified as part of any connection, because they are malformed, or the box is out of memory, etc.

Multiple Mappings, Overlap and Clashes

You can have NAT rules which map packets onto the same range; the NAT code is clever enough to avoid clashes. Hence having two rules which map the source address 192.168.1.1 and 192.168.1.2 respectively onto 1.2.3.4 is fine.
Furthermore, you can map over real, used IP addresses, as long as those addresses pass through the mapping box as well. So if you have an assigned network (1.2.3.0/24), but have one internal network using those addresses and one using the Private Internet Addresses 192.168.1.0/24, you can simply NAT the 192.168.1.0/24 source addresses onto the 1.2.3.0 network, without fear of clashing:

# iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth1 \
        -j SNAT --to 1.2.3.0/24

The same logic applies to addresses used by the NAT box itself: this is how masquerading works (by sharing the interface address between masqueraded packets and `real' packets coming from the box itself).
Moreover, you can map the same packets onto many different targets, and they will be shared. For example, if you don't want to map anything over 1.2.3.5, you could do:

# iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth1 \
        -j SNAT --to 1.2.3.0-1.2.3.4 --to 1.2.3.6-1.2.3.254

Altering the Destination of Locally-Generated Connections

The NAT code allows you to insert DNAT rules in the OUTPUT chain, but this is not fully supported in 2.4 (it can be, but it requires a new configuration option, some testing, and a fair bit of coding, so unless someone contracts Rusty to write it, I wouldn't expect it soon).
The current limitation is that you can only change the destination to the local machine (e.g. `j DNAT --to 127.0.0.1'), not to any other machine, otherwise the replies won't be translated correctly.

http://www.netfilter.org/documentation/HOWTO/NAT-HOWTO-6.html

When AND Why Proxy-ARP

emaknyus@gmail.com (tamvan & verani) — Mon, 5 Sep 2011 22:11:00 -0700

In my training classes, I go to great lengths to teach when and why proxy arp is necessary on an interface. I have seen (and answered) numerous questions on my mailing lists regarding proxy arp and it seems it is time to create an article that I can just point people to. I will focus this article on Mikrotik RouterOS (as many of my articles are), but it should be noted that proxy arp is not unique to Mikrotik.
If you do not already understand what ARP is, then you will need to understand that before this article will make much sense. ARP, which stands for “Address Resloution Protocol”, in simple terms is a mechanism that makes it possible for devices that are on an Ethernet-like network communicate. Let’s take a very simple example where we have 2 devices plugged into an Ethernet switch using IP addresses 192.168.1.1 for device A and 192.168.1.2 for device B and subnet mask /24 (255.255.255.0). These devices can communicate at layer 3 directly, because they are in the same subnet. If Device A needs to send an IP packet to Device B, it will need to send that packet via the Ethernet switch. This means that the IP packet will need to be framed in an Ethernet frame. Device A already knows the IP address it needs to communicate with, but it needs a method to determine which MAC (Ethernet address) is associated with that IP address. This is where ARP comes in.

In the above circumstance, Device A will know that Device B is plugged into the same Ethernet segment because it’s subnet is the same as it’s own. In order to determine which MAC address is associated with the IP 192.168.1.2, Device A will send a packet onto the Ethernet segment that is destined to the Ethernet Broadcast address. This address is one that every device plugged into the switch will “hear” and “inspect”. This packet includes a request asking “Who has 192.168.1.2?”. Every device on the segment will “hear” this request, but only one device will “own” that IP and respond. The response will be something like “192.168.1.2 is at 00:12:34:56:78:9A”. Now Device A knows the MAC address that is associated with the IP address 192.168.1.2 and it can frame the IP packet and send it out the Ethernet interface.
The above description is very much shortened, but should give you some understanding of the process. It is important to note that both Device A and Device B MUST be connected to the same LAN segment. The ARP process is needed only for LAYER 2 addressing purposes.
So, let’s take a look at a situation where proxy arp would be necessary. Consider this network diagram:

In the above network, we have a client device that could be a Windows PC, Laptop or another router. For the purpose of this explanation, we need to assume it is not a router. The reason for this assumption is that the behaviour of a router may (or may not) be different than I will describe. Let’s assume for a moment that the router (on the right) assigns this client device with an IP of 192.168.2.1. When the client sends IP traffic to the server at 192.168.1.2, that packet will have a source address of 192.168.2.1 (the tunnel IP). The server “sees” that packet and responds to the IP 192.168.2.1, which is not on it’s local subnet. The server would send the packet through it’s “gateway” at 192.168.1.1.
Let’s consider another scenario where the router assigns the pptp client with the IP address 192.168.1.10. Again, when the client sends IP traffic to the server, the source IP address will be the tunnel IP of 192.168.1.10. The server will need to respond to the IP address 192.168.1.10. The problem here is that although the server will ASSUME that the device with the IP address 192.168.1.10 exists on the local LAN, it does not. When the server sends it’s ARP request to the LAN asking “Who has 192.168.1.10″, there is NO device on the LAN with that IP and therefore the communication will fail. If, however, on the router’s LAN interface, we configured it to answer ARP requests “by proxy” for the tunnels it knows about, the ROUTER will answer the server’s arp request. In this way, the server would “deliver” the Ethernet frame (with the IP packet inside) to the router, which in turn will deliver the IP packet to the PPtP client over the tunnel. I am certain that you’ve already figured out the rest, but I’ll say it anyway…the configuration we need to do on the router is to turn on proxy arp on the LAN interface.
In the Mikrotik, this is very simple to do. If the interface is named “LAN”, then the following command would do it:
/interface ethernet set LAN arp=proxy-arp
That’s IT! In MOST circumstances, I recommend using IP space that is NOT in the LAN subnet for tunnels. If it is necessary, due to limited IP space or for any other reason (Active Directory issues are one situation), setting proxy arp will be necessary and will allow this to function.
As always, I hope this article has been helpful to you. If you find it of use, by all means “Digg it” above to let others know about it.

http://blog.butchevans.com/2010/06/when-and-why-proxy-arp/

Breaking a network in two with proxy ARP

emaknyus@gmail.com (tamvan & verani) — Mon, 5 Sep 2011 22:11:00 -0700

Proxy ARP is a technique for splitting an IP network into two separate segments. Hosts on one segment can only reach hosts in the other segment through the router performing proxy ARP. If a router sits between two parts of an IP network and is not running bridging software, then routes to hosts in each segment and proxy ARP are required on the router to allow each half of the network to communicate with the other half.
Occasionally, this technique is incorrectly called proxy ARP bridging. An Ethernet bridge operates on frames and a router operates on packets. The proxy ARP router should have routes to all hosts on both segments. Once the router can reach all locally connected destinations via the correct interfaces, you can begin to configure the proxy ARP functionality.

Although proxy ARP complicates a network, a great advantage of proxy ARP technique is the greater control over IP connections between hosts.
There are two primary proxy ARP techniques. With the 2.4 kernel, it is possible to use the sysctl net/ipv4/conf/all/proxy_arp to perform proxy ARP. Alternatively, manual population of the ARP table reaches the same end.
The key part of the correct functioning of proxy ARP in a network is that the host breaking a network into two parts has correct routes for all destinations in both halves of the network. If the host which has interfaces in both networks does not have an accurate routing table, IP packets will get dropped on the routing device.
One common method of breaking a network in two involves making a very small stub subnet at one end or the other of the IP range. This small subnet (maybe as small as a /30 network, with two usable IPs) makes an excellent sequestered location for a host which requires more protection or even, a generally untrusted host which shouldn't have complete access to the Ethernet to which the other machines connect.
For a practical example of this, see the relationship between the service-router, masq-gw and isolde in the network map. isolde and service-router share the same IP network, 192.168.100.0/24. If either has a packet for the other, it will generate an ARP request which should be answered by masq-gw. Naturally, masq-gw has its routes configured in such a way that both hosts are reachable from it. Thus, the packet will successfully pass through masq-gw.
Let's examine what the sequence of events is by which the packet will reach service-router from isolde. In this example, isolde will send an echo request packet to service-router. Please also refer to Section B.1, “arp” for examples and command lines to create a proxy ARP configuration.

the admin on isolde creates an echo request packet for 192.168.100.1 with ping
isolde sends an ARP request for the owner of 192.168.100.1
masq-gw replies that isolde should send packets for 192.168.100.1 to its Ethernet address, 00:80:c8:f8:5c:71
masq-gw receives the packet, unwraps it and selects eth3 as the output interface
masq-gw sends an ARP request for the owner of 192.168.100.1
service-router replies that masq-gw should send packets for 192.168.100.1 to its Ethernet address, 00:c0:7b:7d:00:c8
service-router receives the packet unwraps it and hands it up the IP stack, which generates an echo reply bound for the source address, 192.168.100.17 (isolde's IP)
service-router sends an ARP request for the owner of 192.168.100.17
masq-gw replies that service-router should send packets for 192.168.100.17 to its Ethernet address, 00:80:c8:f8:5c:74
masq-gw receives the packet, unwraps it and selects eth0 as the output interface
masq-gw sends an ARP request for the owner of 192.168.100.17
isolde replies that masq-gw should send packets for 192.168.100.17 to its Ethernet address, 00:80:c8:e8:4b:8e
isolde receives the reply, unwraps it and hands it up the IP stack to the awaiting ping command

Where possible, a simplified network is easier to maintain, but occasionally, this sort of trickery is necessary. This is an excellent way to insert a firewall into the middle of a network. The firewall, naturally, has to have its routes set properly, and proxy ARP entries will be required for routers.
Now, here's a short script and configuration file which can be run as a SysVInit style script. This script provides a great deal of control over the ARP table directly so may be preferable in some cases to an alternate solution outlined below. This proxy-arp script reads the following configuration file. Each is commented heavily so it should be clear how to use them.
This chapter discussed how to break a network in twain with proxy ARP techniques. For another explanation of the same concepts, read the Proxy ARP Subnet mini-HOWTO. Available in most (all?) 2.4 kernels is built-in capability for Proxy ARP. This is documented in deeper detail above. Consider familiarizing yourself with the methods of suppressing and controling ARP through Julian Anastasov's work.

http://linux-ip.net/html/adv-proxy-arp.html

e-maknyus

Lecture #21 - Asynchronous Transfer Mode (ATM)

ATM Background

ATM Features

ATM Speed

ATM - Circuit Switching or Packet Switching

Advantages

Disadvantages

ATM Virtual Channels

Permanent and Switched Virtual Circuits

ATM - connection-oriented protocol

ATM Virtual Channel Setup

ATM VPI/VCI

ATM and ADSL

High Level View of an ADSL / ATM Network

ATM Switching

ATM Switches

ATM Architecture

ATM Cell Format

The HEC field

ATM Traffic Control and QoS

CBR - Constant bit rate

VBR - Variable bit rate

ABR - Available bit rate

UBR Unspecified bit rate:

ATM Reference Model

ATM Layer

ATM Adaptation Layers (AAL)

AAL1 to AAL5

More on ATM AAL

ATM is all things to all people

ATM AAL5 and Encapsulation of Internet Data

LANs over ATM

CLIP - Classical IP over ATM

LANE - LAN Emulation

NHRP - Next Hop Resolution Protocol

MPOA - Multi-Protocol over ATM

IP Switching

Multi-Protocol Label Switching (MPLS)

Three (Four?) Letter Acronyms (TLAs)

ADSL Information

What is Clonezilla?

Perbedaan redirect to proxy memakai dst-nat dan mark-routing

td 8117 exploit

ultimate.sh

fw_function

Traffic Control

Introduction

Incoming and Outgoing Traffic

Recommended Resources

Recommended Approaches

State of the Code

Inspect Your Rules

Matching

Sample Traffic Control Code

tc code walkthrough

iptables code walkthrough

Further optimizations

SSH

ACKs

Other Links of Interest

IPTables U32 Match Tutorial

Introduction

Check the value of a 2 byte field

Check the value of a 1 byte field

Looking at 4 bytes at once

Inspecting early bytes in the header

Inspecting individual bits

Combining tests

Moving on to the TCP header

Checking for values in the ICMP header

Checking for values in the UDP payload

Checking for values in the TCP payload

Wrapup

Tests

unix toolbox

squid hardware

Hardware

linux network tune

mikrotik hardcoded

`tc` code walkthrough

`iptables` code walkthrough