Simple life, Complicated mind

Drupal - Symfony - a PHP Web Development Framework.

2012-02-13T09:45:00.001-08:00

Drupal - Symfony - a PHP Web Development Framework.

http://symfony.com/

Using vim + xdebug to debug php On FreeBSD

2012-02-01T15:35:00.000-08:00

Using vim + xdebug to debug php On FreeBSD

Here is the scenario: You have multiple developers logged into a Linux server which is running Apache and PHP using Vim to write PHP code. They’re using error_log and echo statements to debug their code. It takes forever, it’s tedious and can result in bugs from forgotten debug statements. You stare enviously at .NET programmers with fancy debuggers (while you snicker knowing that you edit 10x faster with Vim anyways). But still, you know there has to be a better way.

There is.

Here’s how it works. You’re coding away in vim. You hit F5; Vim waits for a connection from the PHP server. You refresh the PHP page you’re working on. It attempts to contact Vim — connection successful. You are launched into a debugging session right inside Vim. You can step into, over, and out of statements, eval statements, get all variables in context, get and set properties, remove and set breakpoints, all on the fly. Finally, some real programming tools.

Install VIM

# cd /usr/ports/editors/vim
# make WITH_PYTHON=yes install
# cp /usr/local/share/vim/vim72/vimrc_example.vim ~/.vimrc

Make sure your VIM supported Python:
# vim --version | grep -E 'python|sign'

If you see "+python" and "+sign" you are good to go.
If you see "-python" and "-sign", please reinstall your vim with WITH_PYTHON=yes option

Install Xdebug client
Now that vim is ready, download the DBGp client script. Extract the two files (debugger.vim and debugger.py) to your vim plugin directory or your .vim home directory

In Vim 7.2 compiled on FreeBSD, the default load-for-everyone plugin location is /usr/local/share/vim/vim72/plugin/, so you would put both files in that directory.

Pick one of clients from:
DBGp client script: http://www.vim.org/scripts/script.php?script_id=2508
DBGp client script: http://www.vim.org/scripts/script.php?script_id=1929

Copy debugger.vim and debugger.py to either one of following directories:

/usr/local/share/vim/vim72/plugin
or
~/.vim/plugin

Try to run vim — if you get no errors, everything should be good. If you get an error, double check :version to make sure +python and +signs are there. If they are, post a comment here with the vi error you get. If they aren’t, the vim compilation/installation didn’t work — go back and try it again.

Install xdebug (the server [engine])

# cd /usr/ports/devel/php-xdebug
# make install

# vim /usr/local/etc/php.ini
[Zend]
zend_extension = /usr/local/lib/php/20060613/xdebug.so

; This switch controls whether Xdebug should try to contact a debug client which is listening on the host and port as set with the settings xdebug.remote_host and xdebug.remote_port.
xdebug.remote_enable = 0
xdebug.remote_port = 9000
xdebug.remote_host = localhost

; When this setting is set to on, the tracing of function calls will be enabled just before the script is run. This makes it possible to trace code in the auto_prepend_file.
xdebug.auto_trace = 1
xdebug.trace_output_dir = "/tmp/xdebug/log"
xdebug.collect_params = 4

;xdebug.var_display_max_children = 128
;xdebug.var_display_max_data = 512
;xdebug.var_display_max_depth = 3

; Enables Xdebug's profiler which creates files in the profile output directory. Those files can be read by KCacheGrind to visualize your data.
xdebug.profiler_enable = 1
xdebug.profiler_output_dir = "/tmp/xdebug/log"

; Controls the protection mechanism for infinite recursion protection. The value of this setting is the maximum level of nested functions that are allowed before the script will be aborted.
xdebug.max_nesting_level = 100

; shows a human readable / computer readable trace file.
xdebug.trace_format = 0

; This setting tells Xdebug to gather information about which variables are used in a certain scope. This analysis can be quite slow as Xdebug has to reverse engineer PHP's opcode arrays. This setting will not record which values the different variables have, for that use xdebug.collect_params. This setting needs to be enabled only if you wish to use xdebug_get_declared_vars().
xdebug.collect_vars = 0

; When set to '1' the trace files will be appended to, instead of being overwritten in subsequent requests.
; Note: this option can be useful if you could not find your function calls anywhere.
xdebug.trace_options = 1

Make sure xdebug is loaded by PHP

<?php
echo phpinfo();
?>

Now, with your site being example.com, go to http://example.com/index.php?XDEBUG_SESSION_START=1 (Add XDEBUG_SESSION_START=1 after your script name). This will set a cookie in your browser which expires in 1 hour which tells the PHP XDebug module to try to make a connection every time a page loads to a debugging client which is listening on port 9000. The cool thing is that if it can’t make a connection, it just keeps loading the page, so there’s no issue just leaving the cookie on.

Now go back to vim and press F5. You should see a message like “waiting for a new connection on port 9000 for 5 seconds…” at the bottom of the screen. You have five seconds to now refresh the PHP page. This will create a connection between the debugger and client. Now you’re debugging. Follow the instructions in the Help window to step into, over and out of code. Press F5 to run the code until a breakpoint (which you can set using :Bp).

When I pressed F1, F2, or F3 keys, it toggles the character to either upper or lower case.

To solve this problem, I changed following line:
map <F2> :python debugger_command('step_into')<cr>

To:
map ^[[12~ :python debugger_command('step_into')<cr>

Note: press ctrl-v F2 to generate ^[[12~.

But what if I have multiple developers on the same machine?

No problem. Simply set g:debuggerPort in each developer’s .vimrc to get the client listening on a different port. So if you wanted one developer to connect on 9001 instead of the standard 9000, you would add this line to their .vimrc:

let g:debuggerPort = 9001

Getting the server to connect on a different port is a little trickier. You need to set a custom php.ini value (xdebug.remote_port) for each user. It works best if you’re using VirtualHost’s in Apache. Just add the following line to the VirtualHost section of your httpd.conf:

php_value xdebug.remote_port 9001

Now restart Apache and if you use that VirtualHost and that vi user, then they should connect successfully.
That’s about it

Please post any questions or suggestions you may have. I hope this helps a few of you out there who want debugging tools but don’t want to give up Vim editing. Also be sure to post any alternate methods, or any patches or improvements to the remote PHP debugger vim script, and I’ll be sure to incorporate them.

[Note: This is the first in a series of posts we'll be doing along the lines of tutorials, tools we've developed, tech commentary, and so on. Please feel free to subscribe, as well as leave any comments, thoughts or suggestions below so we can be sure to improve with each article! Thanks!]

use XDebugToolkit to convert the XDebug cachegrind (= “xdebug.log“) to a dot graph
http://code.google.com/p/xdebugtoolkit/

# svn co http://xdebugtoolkit.googlecode.com/svn/tags/0.1.3/xdebugtoolkit/ xdebugtoolkit
# ./cg2dot.py cachegrind.out.23328 > cachegrind.out.23328.dot

# cd /usr/ports/graphics/graphviz ; make install ; rehash
# dot -Tpng -otest.png

Combine to one command:
# ./cg2dot.py cachegrind.out.23328 | dot -Tpng -otest.png

this might be useful /usr/ports/graphics/py-pydot ??

Windows version of Graphviz is also available at:
http://www.graphviz.org/Download..php

[] XDot - Interactive viewer for Graphviz dot files

[] CacheGrind

[] WinCacheGrind (for Windows)

[] CacheGrindvisualizer

[] Graphviz - Graph Visualization Software

[] ZGRViewer - a GraphViz/DOT Viewer

[] Dot Language

[] Scalable Vector Graphics (SVG)

Reference:
How to Debug PHP with Vim and XDebug on Linux

use ctags and vim on freebsd

2012-02-01T15:14:00.000-08:00

use ctags on freebsd

There are two ctags on freebsd :

ctags from base system : /usr/bin/ctags
exuberant ctags from ports : /usr/local/bin/exctags

If you are linux programmer (like me), then exctags is what you are looking for.

Install ctags from ports:

cd /usr/ports/devel/ctags
make install clean

Generate tags from current directory:
# cd /src/dir/
# /usr/local/bin/exctags -R

Note: a new tags file will be generated under the current directory.

Generate tags from specified directory and store the tag file to the specified path:
# /usr/local/bin/exctags -R -f /tmp/tags /src/dir/

Let exctags to read *.module file as a php script for Drupal:
/usr/local/bin/exctags -R -f /tmp/tags --langmap=php:+.module.inc /src/dir/

Edit your ~/.vimrc:
" To display the tags for only the current active buffer:
let Tlist_Show_One_File=1

" toggle the taglist window.
" Note: press ctrl-v F6 to generate ^[[17~
nnoremap ^[[17~ :TlistToggle<CR>
"nnoremap <F6> :TlistToggle<CR>

Run this command in your vim:
:set tags=/tmp/tags

Add this line to your ~/.vimrc:
set tags=/tmp/tags

Reference:
http://www.codernotes.com/2011/202/use-ctags-on-freebsd/

GNU GLOBAL 作為分析基礎

2012-02-01T12:16:00.000-08:00

GNU GLOBAL 作為分析基礎
by thinker
2 Columns
關鍵字:
coding
最近對程式碼分析 (reverse) 有一些想法，打算做一些小工具。如果從頭打造，大概得花個幾個月以上的時間，而且有許多現成的工具可以幫忙完成部分工作。其中最麻煩的工作大概就是 parse 程式碼， doxygen 已經做了我要的東西，應該直接利用就好了，但 doxygen 卻沒有輸出 raw data 。

本來打算看 doxygen 的程式碼，從中攔截，並輸出我要的資料，但意外中發現 GNU GLOBAL 。 GNU GLOBAL 是類似 ctags 的工具，但提供許多 command line 的命令，可以查詢 symbol 的出處，被 reference 的位置等等，正是我所要的資訊，又是 command line 工具，更易於整合。

雖然和 ctags 相似，但 ctags 只產生 DB ，卻沒有讀取 DB 的 command line 工具，另外 ctags 也沒有產生 reference 的資訊。各方面而言， GNU GLOBAL 都優於 ctags 。

GNU GLOBAL 本身，也提供了 bash 、 tcsh 、 nvi 、 vim 、 less ．．．的整合。另外 GNU GLOBAL 也提供將程式碼輸出成 HTML 的功能，並產生 hyperlink 。更詳細的功能，請查閱

Reference:
http://www.codemud.net/~thinker/GinGin_CGI.py/show_id_doc/247
http://www.gnu.org/software/global/global.html

閱讀 C 和 C++ 原始碼的好幫手

2012-02-01T12:15:00.000-08:00

閱讀 C 和 C++ 原始碼的好幫手

最近有需求讀 C/C++ 的東西, 試了 ctags, cscope 覺得不理想。問了一下收到許多回應 (G+ 、plurk ), 真是太感謝大家了, 減少入門摸索的時間。

試用的感想如下:

grep

優點: 好上手
缺點: 陽春
安裝: 內建於 Linux

gtags

優點: 可找 caller 和 callee
缺點: 因為索引檔是由 ctags 來的, 會漏東西; 執行方式也有些不便
安裝: 程式很久沒人更新了, 要做一些修正才裝得起來
參照官網指示
make 時看少了什麼 header, 手動補一下 header
然後 make 還是會失敗, 將 gas.py 的 "import as" 改為 "import asm", 下面用到的模組名也要跟著改, as.py 也要改為 asm.py。python 2.6 後 as 是 keyword
編好後將幾個用到的 python scripts 第一行由 python2.4 改為 python

GNU global
GNU GLOBAL - 程式碼分析 - cross reference 的分析工具
http://gala4th.blogspot.com/2009/10/gnu-global-cross-reference.html

ack

優點: 比 grep 容易使用, 省得配合一些 command 過濾檔案, 見官網的《Top 10 reasons to use ack instead of grep.》。而且還有彩色的輸出!!
缺點: 因為沒建 index 的關係, 速度較 id-utils 慢, 我的測試情境要 3s, 而 id-utils 只要 0.006s
安裝: curl http://betterthangrep.com/ack-standalone > ~/bin/ack && chmod 0755 !#:3

id-utils

優點: 速度快, 和測 ack 同樣的情況, 建索引 5.3s, 之後搜尋瞬殺
缺點: 介面沒有 ack 直覺易用, 我寫了個小程式 gj 以 id-utils 為底, 提供彩色輸出和進一步過濾檔名的功能。
安裝: Ubuntu 超容易, aptitude install id-utils

Eclipse CDT

優點: 方便開新視窗看 caller、callee
缺點: 不方便搭 vim 使用 (對 vim 重度使用者才有差); 建 index 有點久, 我的測試情境要數分鐘到十分鐘吧
安裝: 結果這個是我試最久的, 因為不知怎麼建 index。參考官網 FAQ, 建索引前要先設 include dir path。我一直找不到 context menu, 結果它就是左側的那個專案清單。另外 Eclipse CDT 也會漏一些東西, C++ 特別嚴重。
結論

用 Eclipes CDT 方便平時快速跳到定義
輔以 id-utils + gj 確保不會漏東西。之後用一用再視需求來更新 gj 功能。
2011-12-16 Update

Eclipse CDT 的問題有一部份是我設錯, 詳細設法見用 Eclipse CDT 讀 C/C++ 原始碼

張貼者： fcamel 於下午11:25
標籤： C, C++
10 意見:

Scott TsaiDec 12, 2011 09:30 AM
以需要能找出全部有用到某函式的地方為前提：
1. 純粹 C 而不是 C++ 的 code base, cscope + VIM plugin 很好用
2. Eclipse 與 VIM 見 Vrapper：http://itrs.tw/wiki/IDEs_with_VIM_Emulation

要在 VIM 中方便跳到 search results，我有個奇怪用法：
1. 將 search result 轉成『grep -n 格式』存在檔案 l 中
2. vim
3. :set grepprg=cat\ l
4. :grep
5. :copen
回覆

fcamelDec 12, 2011 10:05 AM
之前讀 python code 時有用過 pycscope + VIM plugin, 相當不錯, 原以為 cscope + VIM plugin 也可適用, 可惜 C++ 的情況滿多不適用的。

http://vrapper.sourceforge.net/features/
沒想到 vrapper 有支援 macro, 這樣應該可用, 明天來試試。很久以前用過類似的東西, 但沒有 macro 用起來不順手

最後的例子沒看懂, 是指要先在 cmd line 用 grep -n 將輸出導到檔名 "|" 中, 再做後面的操作嗎??
回覆

Scott TsaiDec 12, 2011 10:11 AM
Re: 最後的例子沒看懂

若搜群工具可以輸出 "FILE:LINE:...SNIPPET.." 格式，將結果存於檔案中, 再用以上設定在 VIM 中打開就可以像在 VIM 中用內建 grep 一樣，跳到各搜尋結果。

搜尋工具格式不同，則寫 script 或手動轉。
VIM 也有 "grepformat" 參數，但我自己是用上述方法。
回覆

Scott TsaiDec 12, 2011 10:20 AM
為了主題完整性，跟之後讀者分享 grep 顏色輸出設定：

alias egrep='egrep --color=tty -d skip'
alias egrpe='egrep --color=tty -d skip'
alias fgrep='fgrep --color=tty -d skip'
alias fgrpe='fgrep --color=tty -d skip'
alias grep='grep --color=tty -d skip'
alias grpe='grep --color=tty -d skip'

以上可安全的加在 .bashrc 中，
grep 發現 stdout 不是 tty 時就會改回無顏色控制碼的輸出。
來源 glibc maintainer Ulrich Drepper:
http://udrepper.livejournal.com/17109.html
回覆

小鄭Dec 12, 2011 04:16 PM
跟ack一樣的功能，不過用Python寫成，程式碼也蠻易讀。
http://eli.thegreenplace.net/2011/10/14/announcing-pss-a-tool-for-searching-inside-source-code/
回覆

小迪克/MarkDec 12, 2011 06:55 PM
Eclipse漏東西是什麼情況? 試試看把indexer的cache limit調大一點...
By the way, gj還不錯用!! Good Job~XD
回覆

scwDec 28, 2011 02:32 AM
還有一個東西叫 lxr,
http://lxr.sourceforge.net/
我覺得也滿好用的溜
回覆

fcamelDec 28, 2011 07:31 AM
@scw 謝啦, 看起來頗強大的, 明天來試試, 好像是類似 doxygen 的工具?
回覆

Hello WaylingJan 5, 2012 05:30 PM
lxr沒顏色~有點難看
回覆

Scott TsaiJan 5, 2012 05:53 PM
較新的 lxr 替代品是 dxr:
http://dxr.mozilla.org/mozilla/index.html
是以 Clang 為基礎的：
https://github.com/mozilla/dxr

@fcamel: "lxr 好像是類似 doxygen 的工具?"
lxr, Linux Cross Reference, 原本是為了研究 Linux Kernel source code 而寫出來的。至今討論 Linux Kernel 內部 API 時還是很常用。
回覆

===
大家都怎麼追 C++ 程式碼啊? 試了 gtags, 不過來源就是 ctags, 會漏掉很多東西, cscope 也會, 但單用 grep 太慢, 沒好工具的話, 想說自己硬幹一個基於 grep 的 index 小工具

doxygen or Source Insight

Sid666 說 cscope

Dec 09, 2011 - 04:46PM
av. visual assist (誤)

York 說這得視程式碼規模及你打算投入多少時間來決定

York 說當然，你追這份 code 的原因也要納入考慮

fcamel 說 YorkJong: 我原本指用什麼工具, 的確也如同解決其它問題一樣, 得視情況調整做法

小迪克/SDK 說 eclipse...

Thinker gtags 可以用 regex, 可以 reverse, 可以 grep

Thinker 不是 ctags 可以比擬的..

Thinker 而 cscope 的功能就和 gtags 差不多..

Thinker 如果你只是想加快 grep，建議你用 idutils, 能為所有的 token 建立 index

Thinker 我都是 idutils 和 gtags 合在一起用.. 一些 gtags 沒辨法分析的東西，可以用 idutils 快速改到..

Thinker 我都是 idutils 和 gtags 合在一起用.. 一些 gtags 沒辨法分析的東西，可以用 idutils 快速找到..

Thinker idutils 裡面包括一個叫 gid 的工具，相當於 grep。但 gid 只能針對單一個 token 做 match，而不是一整行。因此，我都是先用 gid 縮小目標，然後把輸出丟給 grep 再做過濾

fcamel 說 Thinker: 謝啦, 之後來試試, 另外在 G+ 有看到 kcwu 推 ack, 看起來也不錯

google-gtags
Server-based tags serving for large codebases. Clients in python and for emacs and vim
- Comment - Hang out - Share
+3
15 comments

Shaka Huang - 好久沒看 C++ 程式碼了.. Orz
Dec 9, 2011

Kuang-che Wu - 只跟 grep 比的話, +ack 不錯. 不過跟你的需求還是差很遠.
Dec 9, 2011 +2

Chia Hao Lo - +Kuang-che Wu 喔喔, 這東西好用, 該有的不會漏比較重要, 謝啦

btw, 今天才在試 find 的語法, 想少搜一堆不相干的檔案, 但沒找到適合的語法。結果在 ack 的說明裡就看到要怎麼用 find + grep 達到只搜 .cpp 和 .h 的寫法, 原來是 -or 啊
Dec 9, 2011

Yu-Teh Shen - cscope 的確有時會漏... 我也來試試看ack~
Dec 9, 2011 (edited)

Scott Tsai - For large C++ code bases I use Eclipse CDT.
I expect that In a few years, IDEs using libclang based indexers should be pretty sweet as well.
Dec 11, 2011

Yu-Teh Shen - 話說我同事用CDT, 建立index的時候, memory吃到快2g...
Dec 11, 2011

Yu-Teh Shen - 話說 +Scott Tsai 你平常debug是用vimgdb還是用甚饃方式使用gdb? 不會是直接用gdb吧...
Dec 11, 2011

Chia Hao Lo - +Yu-Teh Shen 現在記憶體超便宜的, 吃 2G ram 不算是問題了
Dec 11, 2011 (edited)

Chia Hao Lo - +Scott Tsai 之後都來試看看, 那一個用起來最順手 Eclipse CDT, ack, ID Utils
Dec 11, 2011

Scott Tsai - @Yu Teh Shen:
1. 最常用的方法真的是『直接用 gdb 』，最有用的指令還是 backtrace
2. 我後來把 gdb script 語法和 gdb 的 Python API 學了一下。比較複雜的動作寫到 "gdb -x SCRIPT" 裡面
3. 有時直接在程式碼中加入 break point: https://github.com/scottt/debugbreak/blob/master/debugbreak.h
4. 有時切到 gdb-tui: http://davis.lbl.gov/Manuals/GDB/gdb_21.html
Dec 11, 2011

Scott Tsai - @Yu Teh Shen: re: Eclipse CDT Indexer 用很多記憶體:
我會改 eclipse.ini 中的 -X ，讓 JVM 一啟動就 alloc 4 G

公司配給你的電腦太爛的話，一邊帶自己的去，一邊『提醒』主管記憶體大一點、有 SSD 的話你每天可以省 15 ~ 60 min 。
Dec 11, 2011 +1

Yu-Teh Shen - +Scott Tsai 剛剛試了一下gdb -tui, 如果你要編trace code一邊debug, 這樣似乎你可能還是要另外開一個terminal 看code (因為要跳來跳去)?
Dec 12, 2011 (edited)

Scott Tsai - +Yu Ten Shen: 基本上對。詳細一點說：
我開發 C/C++ 程式的方法，刻意避開了『很依賴 debugger』的風格。

多數會先擬好除錯計畫，在 code base 中插入好檢查、pretty print 資料結構的程式才開始。在除錯時，不只是另外開 editor 看 code，同時還有除錯計畫的筆記，紀錄如『我懷疑哪些資料有壞掉』，『已經作過哪些實驗』、『目前看到的 corruption 在 stack 還是 heap 上 address 某某附近』等。

除錯到一半，覺得問題不容易解時，我也會先 trace code 把局部的 data flow 與 control flow 寫在筆記裡面才繼續開始除錯，避免要『跳來跳去』看 code。

在處理複雜的 code base 或長時間執行的系統軟體時，上述風格頗有道理 -- 最重要的是逼自己先分析問題，而不是先開始單步執行然後見樹不見林。如concurrency 的問題不適合用 debugger 觀察、最佳化編譯的 release build 中，你要看的變數原本已經沒用到的話，暫存器已被拿來存別的值了。

我也同意有經驗的 developer 看較簡單 code base 時，直接在 debugger 邊跑邊讀 code 可能比較快。最極端的如大學『C 程式語言入門』助教改作業。但我在工研社教了多屆 C 入門，也是除了分析 core dump 以外，不用 debugger 的。因為學生人數少，看他們的 code 可以教的更深入。

也有遇過很需要 debugger 的時候：編譯或連結時間很長的 C++ code base 要改 code 追特定問題成本太高 (header only style 的 C++ 配上 binutils 中舊的、非 gold 的那版 linker 就很慘了）。我會盡量將 break point, watch point, 檢查與dump 資料結構的動作寫成 gdb script 或從 Python 用 gdb API，避免在 debugger 中重複同樣操作。

結論：
我同意直接用命令列的 gdb 比起用整合在 IDE 中的除錯界面，開發者要多記住很多資訊，有時感覺起來像多餘負擔。最後，整合 gdb 的 IDE 中，Eclipse CDT 與 QtCreator 是支援 remote debugging （包含 cross debugging）且有好的團隊在維護的；但我平常還是覺得從命令列用 gdb 比較快。
Dec 12, 2011 (edited) +6

Chia Hao Lo - 可惜 +1 只能按一次啊~~
Dec 12, 2011

Yu-Teh Shen - +Scott Tsai 感謝你的分享, 我來我的wiki上面開個scott 專區好了~

Reference:
http://fcamel-life.blogspot.com/2011/12/cc.html
https://plus.google.com/111353793049965752735/posts/GAVtr4HXXc4
http://www.plurk.com/p/eyqy5v

GNU GLOBAL - 程式碼分析 - cross reference 的分析工具

2012-02-01T12:09:00.000-08:00

GNU GLOBAL - 程式碼分析 - cross reference 的分析工具
by thinker
2 Columns
關鍵字:
程式碼分析
GNU GLOBAL 是類似於 ctags 的工具，用以建立 source code 符號定義位置的 database 。除了符號定義之外，global 還建立符號 xref (cross reference) 的 database ，用以快速查詢 xref 的位置，是個很方便的設計。

查詢 xref 位置的功能，讓使用者能很快的得知何處使用某個指定的符號，可用以確定符號名稱更改時，是否有漏網之魚。或者，也可透過 xref ，以了解符號在系統中的作用、角色。

除了建立 tag 和 xref database 之外， global 提供將程式碼轉成 HTML ，並依 xref 產生 link 的能力。然而，這方面的功能， doxygen 應該更為成熟，輸出的結果也更為美觀。

除此， global 還提供 vim 、 nvi 、 emacs 甚至是 bash 和 less 的查詢介面。透過 editor 查詢的功能就不用說了，和 bash 整合，提供簡短的指令，能快速的查詢、瀏灠程式碼。例如:

[/usr/src/sys] x main
> 1 main 70 alpha/alpha/gensetdefs.c main(in
2 main 1500 alpha/alpha/ieee_float.c main(i
3 main 227 boot/alpha/boot1/boot1.c main()
[/usr/src/sys] show 3
(Load editor and show boot/alpha/boot1/boot1.c at line 227.)

這個範例是在 bash 下，查詢有哪些位置定義了 main function ，查詢的結果列出了三個 main function 。接著 show 第三個 main function 的內容，也就是執行 editor (vi 或其它) 顯示該段程式碼。除此之外，還提供 bash 下的 bookmark 等等的功能。

至於 less 的整合，也提供了一些有趣的功能。例如:

$ less -t main
main(int argc, char **argv)
{
int i;
.....
[xxx/main.c (tag 1 of 55)]

執行 less 以顯示 main function 的內容。

我之所以介紹 GNU GLOBAL ，是因為我在最近的程式碼分析研究當中，以 global 為基礎，對程式碼的 cross reference 進行分析。分析的結果，能幫助我們了解 module 在系統中的作用和關聯性。這對經常在看別人程式碼的 open source 的 programmer 非常的重要，在不確定 module 的關聯性時， programmer 往往是處於傍偟和猶豫當中，遲遲不敢動手。 cross reference 的分析工具，讓 programmer 快速掌握 module ，減少猶豫的尷尬階段。

Reference:
http://heaven.branda.to/~thinker/GinGin_CGI.py/show_id_doc/258

Use grep to search a string in files recursively and store the grep results in a quickfix window

2012-01-31T11:47:00.000-08:00

Use grep to search a string in files recursively and store the grep results in a quickfix window

:grep -r 'pattern' . --include '*.c'

:copen             " open the quickfix window
:cw                " open the quickfix window if there are entries (so if your grep has no results, it won't appear).

<ENTER>            " Open the line of a file in a new buffer.
Ctrl-W <ENTER>     " Open the file/line in a new window.
:.cc               " Open the line of a file in a new buffer.

:cfile debug.txt   " read the content in debug.txt to a quickfix window.
                   " Note: follow this format: "filename:line number:error message"

:cgetfile debug.txt " Just like "cfile" but don't jump to the first entry.

:cclose            " close the quickfix window.
:quit              " close the quickfix window.

Edit ~/.vimrc:
" open quickfix window automatically after running grep command.
command! -nargs=+ Mygrep execute "silent grep! <args>" | copen

:Mygrep -r 'pattern' . --include '*.c'

Performing a customized grep search on the word under the cursor:
map <F4> :execute " grep -srnw --binary-files=without-match --exclude-dir=.git --exclude-from=exclude.list . -e " . expand("<cword>") . " " <bar> cwindow<CR>

:help grep
:help copen
:help quickfix

Reference
http://stackoverflow.com/questions/1234394/selecting-resulting-files-from-grep-in-vim
http://stackoverflow.com/questions/6373293/vim-how-to-store-the-grep-results-in-a-buffer
http://vim.wikia.com/wiki/Find_in_files_within_Vim

grep searchs a string in filename wildcards recursively

2012-01-31T09:51:00.000-08:00

To search all files under /dir for "blah", run the following:

# grep -r blah /dir

You can not use wildards directly in this manner, for example, the following will not search all text files:

# grep -r blah *.txt

This doesn't work because the wildcard is expanded by the shell before grep is called. Instead, search the current directory (or whichever one you want) and pass the --include option:

# grep -r blah . --include "*.txt"

Reference:
http://mindspill.net/computing/linux-notes/recursive-grep-and-filename-wildcards/

Debugging load library path issues

2012-01-31T09:50:00.001-08:00

Debugging load library path issues
If you get a problem with a library that can't find another, you should check the library's dependencies and make sure that the directory in which the missing library resides is part of the load library path.

One example of this problem, that I experienced recently, occurred when the Magick.so library was unable to open the libMagick.so.10 library. The error message was as follows:

Can't load '/usr/lib/perl5/site_perl/5.8.7/i686-linux/auto/Image/Magick/Magick.so' for module Image::Magick: libMagick.so.10: cannot open shared object file: No such file or directory at /usr/lib/perl5/5.8.7/i686-linux/DynaLoader.pm line 230. at (eval 113) line 1 Compilation failed in require at (eval 113) line 1. BEGIN failed--compilation aborted at (eval 113) line 1.
Find out library dependencies with ldd

The ldd command will check the dependencies of a library.

ldd /the/library/path/libWhatever.so
Using the above example problem:

username@localhost [~]# ldd /usr/lib/perl5/site_perl/5.8.7/i686-linux/auto/Image/Magick/Magick.so
libMagick.so.10 => not found
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x0099a000)
libz.so.1 => /usr/lib/libz.so.1 (0x00111000)
libtiff.so.3 => /usr/lib/libtiff.so.3 (0x00121000)
libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0x00d54000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x001f4000)
libSM.so.6 => /usr/X11R6/lib/libSM.so.6 (0x0016e000)
libICE.so.6 => /usr/X11R6/lib/libICE.so.6 (0x00910000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x0039d000)
libXt.so.6 => /usr/X11R6/lib/libXt.so.6 (0x005bc000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0x004cc000)
libm.so.6 => /lib/tls/libm.so.6 (0x00216000)
libc.so.6 => /lib/tls/libc.so.6 (0x00bda000)
libdl.so.2 => /lib/libdl.so.2 (0x00177000)
/lib/ld-linux.so.2 (0x002c2000)
You can see in that the libMagick.so.10 library was not found.

Locate the library with find or locate or slocate or by any other means

Find the location of the library that couldn't be found by ldd. The chances are that the library will be in a subdirectory of /usr. Continuing with the above example...

username@localhost [~]# find /usr -name "libMagick.so.10"
/usr/local/lib/libMagick.so.10
Add the library to the load library path with ldconfig

In the words of the man page, ldconfig will configure dynamic linker run time bindings. Add your missing library to the bindings with the following command:

ldconfig /path/to/add
For our libMagick example, this would be:

username@localhost [~]# ldconfig /usr/local/lib
You can see a list of librarys that are being used with the -p flag.

username@localhost [~]# ldconfig -p
907 libs found in cache `/etc/ld.so.cache'
libzvt.so.2 (libc6) => /usr/lib/libzvt.so.2
libzvt.so (libc6) => /usr/lib/libzvt.so
libzip.so (libc6, hwcap: 0x0001000000000000) => /opt/blackdown-jdk-1.4.2.03/jre/lib/i386/libzip.so
libz.so.1 (libc6) => /lib/libz.so.1
libz.so (libc6) => /lib/libz.so
libxslt.so.1 (libc6) => /usr/lib/libxslt.so.1
libxslt.so (libc6) => /usr/lib/libxslt.so
[...cut...]
If you want to make make the changes more permanent, so that you don't have to remember to specify the directory next time you run the command, you can add it to the /etc/ld.so.conf file, though different linux distributions manage this file in different ways.

Reference:
http://mindspill.net/computing/linux-notes/debugging-load-library-path-issues/

FreeBSD - Backup and restore FreeBSD using Fixit CD

2012-01-30T12:33:00.000-08:00

Since FreeSBIE have not been updated since year 2007, its kernel recognizing new hardware starts to worry me as it is based on FreeBSD 6.2 and it is going to reach EoL (end of life) by end of this year, 2010. Is time to experiment on new way restoring backup.

The FreeBSD installation process does mentioned about "Fixit" CD booting. After meddling around with it, it's actually referring to another bootable CD which its label name consits of "livefs" (Live File System). It's also known as "disk 2" prior to FreeBSD 7.
e.g.
Installation disc -- > 8.0-RELEASE-i386-disc1.iso
Live File systems disc --> 8.0-RELEASE-i386-livefs.iso

Do take note that the dump/restore instruction from the previous post "Freebsd – Backup & restore for disaster recovery" is still valid. This post is served as a "update" as it won't be using FreeSBIE, rather, it will using the livefs to start the restoration of the partition(s).

Without further ado, here's the instruction :

A quick note, after experimenting with a few backup compress method, below are the summary of my findings :
dump and compress using bzip2 (not so good as bzip is the slowest)
dump -0auLf - /dev/ad0s1a | bzip2 > root_partition.bzip2
dump and compress using gzip (good choice as gzip perform faster than bzip2 and the compression is fairly good)
dump -0auLf - /dev/ad0s1a | gzip > root_partition.gzip
plain dump without compression (fast if target disk can write fast enough)
dump -0auLf root_partition.dump /dev/ad0s1a

My preferred compress method will be using gzip as it's compression ratio is good with fair compressing time taken. In this case, the backup of the partition(s) is done with the command :
dump -0auLf - /dev/ad0s1a | gzip > root_partition.gzip

This is the updated version of how to use Live File System CD to start the restoration process :
Partition & Label the disks
Boot the Installation disc (disc 1)
At the "sysinstall Main Menu", go to "Configure --> Fdisk" and perform the following :
Partition the disk as desire.
Next, type "w" to write changes to disk.
BEWARE:this step will wipe all the contents of the hard disk.
Choose "Label" to label the partitions created in the previous step. Perform the following (similar to previous steps) :
Label the partition as desire.
After labeling the partition, note down the partition label. e.g.
ad0s1a --> "/" or root partition

ad0s2b --> swap partition
Move the up/down keys to highlight the "Disk:" and NOT the partition ("ad0s1a").
Next, type "w" to write changes to disk.
Start the Fixit CD
At the main menu, go to "Fixit --> CDROM/DVD"
At the message "Please insert a FreeBSD live filesystem CD/DVD and press return" message, change the installation disc in the drive to Livefs disc (aka disc2).
Then press enter to continue.
Restore the backup
Use the command "mount" to confirm the partitions are mounted accordingly. "/" or root partition should be mounted as /mnt
Create the external mount point & temporary directory. In this scenario, the backup file is stored in a FAT32 formatted USB external hard disk. The temporary directory is for later use with gzip command.
mkdir /tmp/usb /mnt/writable_tmp
Mount the external USB hard disk :
mount_msdosfs /dev/da0s1 /tmp/usb

*** using the usual command to mount USB hard disk "mount -t msdosfs /dev/da0s1 /tmp/usb" will get an error. See below caveats.
The temporary directory environment variable originally points to a read only directory. This result in "restore" command complaining about having not enough disk space to restore.
Re-point it to a writable disk, use the below command :
export TMPDIR=/tmp/writable_tmp/
Start the restoration process :
cd /mnt
gzcat /tmp/usb/root_partition.gzip | restore -rvf -
Repeat the previous step for the rest of the partitions until all partitions are restored.

CAVEATS :
When executing the command :
mount -t msdosfs /dev/da0s1 /tmp/usb

An error return :
mount: exec mount_msdosfs not found in /sbin:/usr/sbin: No such file or directory

For some reason, mount_msdosfs is in /mnt2/sbin/ but "mount" cannot find it.
Solution 1 :
Replace the "mount" command with equivalent "mount_msdosfs". Execute it without the need to specify parameter "-t msdosfs"
e.g.
mount_msdosfs /dev/da0s1 /tmp/usb

Solution 2
Soft link the command "mount_msdosfs" to the same directory where mount resides :
cd /sbin;ln -s /mnt2/sbin/mount_msdosfs mount_msdosfs

then mount it again with the usual command.

Nemaste !!!

Reference:
http://scratching.psybermonkey.net/2010/01/freebsd-backup-and-restore-freebsd.html

Install Nginx, PHP-FPM and Varnish on FreeBSD 8.2

2012-01-30T10:59:00.001-08:00

Install Nginx, PHP-FPM and Varnish on FreeBSD 8.2

# cd /usr/ports/www/nginx

# make install clean

Note: make sure you select important option such as:
HTTP_REWRITE_MODULE=on
HTTP_SSL_MODULE=on

and others as per your requirements.

# echo '### Nginx' << /etc/rc.conf
# echo 'nginx_enable="YES"' << /etc/rc.conf

# vim /usr/local/etc/nginx/nginx.conf

gzip on;
    gzip_min_length 1000;
    gzip_proxied expired no-cache no-store private auth;
    gzip_disable "MSIE [1-6]\.";
    gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;

    server {
            server_name mydomain1.com mydomain2.com mydomain3.com;
            root /usr/local/www/nginx/drupal7; ## <-- Your only path reference.

            client_max_body_size 20M; # Maximum allowed size for uploaded files

            location = /favicon.ico {
                    log_not_found off;
                    access_log off;
            }

            location = /robots.txt {
                    allow all;
                    log_not_found off;
                    access_log off;
            }

            # This matters if you use drush
            location = /backup {
                    deny all;
            }

            # Very rarely should these ever be accessed outside of your lan
            location ~* \.(txt|log)$ {
                    allow 192.168.0.0/16;
                    deny all;
            }

            location ~ \..*/.*\.php$ {
                    return 403;
            }

            location / {
                    # This is cool because no php is touched for static content
                    try_files $uri @rewrite;
            }

            location @rewrite {
                    # Some modules enforce no slash (/) at the end of the URL
                    # Else this rewrite block wouldn't be needed (GlobalRedirect)
                    rewrite ^/(.*)$ /index.php?q=$1;
            }

            location ~ \.php$ {
                    fastcgi_split_path_info ^(.+\.php)(/.+)$;
                    ### NOTE: You should have "cgi.fix_pathinfo = 0" in php.ini
                    include fastcgi_params;
                    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                    fastcgi_intercept_errors on;
      #fastcgi_pass 127.0.0.1:9000;  #pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
                    fastcgi_pass unix:/tmp/php-fpm.sock;
            }

            ### disable PHP execution on upload attachement directory
            location /sites/default/files/ {
              location ~ .*\.(php)?$
              {
                 deny all;
              }
            }

            # Fighting with ImageCache? This little gem is amazing.
            location ~ ^/sites/.*/files/imagecache/ {
                    try_files $uri @rewrite;
            }
            # Catch image styles for D7 too.
            location ~ ^/sites/.*/files/styles/ {
                    try_files $uri @rewrite;
            }

            location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
                    expires max;
                    log_not_found off;
            }

            # nginx status
            location /NginxStatus {
              stub_status on;
              access_log on;
              auth_basic NginxStatus;
              auth_basic_user_file conf/htpasswd;
            }
    }

# /usr/local/etc/rc.d/nginx start

Change this line for nginx:

# vim /usr/local/etc/php.ini
cgi.fix_pathinfo = 0

Install php and php-fpm:

// For PHP5.3
# cd /usr/ports/lang/php5 ; make install

// For PHP5.2
# cd /usr/ports/lang/php52 ; make install

===< The following configuration options are available for php5-5.3.8:
CLI=on "Build CLI version"
CGI=on "Build CGI version"
FPM=on "Build FPM version (experimental)"
APACHE=off "Build Apache module"
AP2FILTER=off " Use Apache 2.x filter interface (experimental)"
DEBUG=off "Enable debug"
SUHOSIN=on "Enable Suhosin protection system"
MULTIBYTE=on "Enable zend multibyte support"
IPV6=off "Enable ipv6 support"
MAILHEAD=on "Enable mail header patch"
LINKTHR=on "Link thread lib (for threaded extensions)"
===< Use 'make config' to modify these settings

// For PHP5.3
# cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini

// For PHP5.2
# cp /usr/local/etc/php.ini-recommended /usr/local/etc/php.ini

Install PHP Extensions
// For PHP5.3
# cd /usr/ports/lang/php5-extensions ; make install

// For PHP5.2
# cd /usr/ports/lang/php52-extensions ; make install

Edit php-fpm.conf:
# vim /usr/local/etc/php-fpm.conf
; comment out following line and add the line below.
;listen = 127.0.0.1:9000
listen = /tmp/php-fpm.sock

request_terminate_timeout=30s

Edit rc.conf
# vim /etc/rc.conf
### php-fpm
php_fpm_enable="YES"

Change this line for nginx

# vim /usr/local/etc/php.ini

date.timezone = America/Vancouver
magic_quotes_gpc = Off
max_execution_time = 120
max_input_time = 60
memory_limit = 256M
upload_max_filesize = 20M
post_max_size = 30M
cgi.fix_pathinfo = 0
log_errors = On
error_log = /var/log/php_err.log

# touch /var/log/php_err.log
# chown www:www /var/log/php_err.log
# chmod 660 /var/log/php_err.log

# /usr/local/etc/rc.d/php-fpm start
# /usr/local/etc/rc.d/nginx restart

# tail /var/log/nginx-error.log

# tail /var/log/php-fpm.log

# tail /var/log/messages

Rotate Nginx and PHP log files:
# vim /etc/newsyslog.conf
/var/log/nginx-access.log 600 7 100000 * JC /var/run/nginx.pid
/var/log/nginx-error.log 600 7 100000 * JC /var/run/nginx.pid
/var/log/php-fpm.log 600 7 100000 * JC /var/run/php-fpm.pid
/var/log/php_errors.log 600 7 100000 * JC

# /etc/rc.d/newsyslog restart

Install Varnish

# cd /usr/ports/www/varnish ; make install clean

Reconfigure your web server to listen on localhost:8080

# echo 'varnishd_enable="YES"' << /etc/rc.conf
# echo 'varnishd_flags="-s malloc,1G -a 127.0.0.1:80 -b 127.0.0.1:8080"' << /etc/rc.conf
# echo 'varnishlog_enable="YES"' << /etc/rc.conf

# /usr/local/etc/rc.d/varnishd start

# /usr/local/etc/rc.d/varnishncsa onestart

# tail /var/log/varnishncsa.log

# /usr/local/bin/varnishtest

# /usr/local/bin/varnishstat

# ls -lh /usr/local/varnish/`hostname`

Install webbench
# cd /usr/ports/benchmarks/webbench ; make install clean

# webbench -c 3000 -t 60 http://test.local/index.php

Reference:
Install Nginx, PHP-FPM and Varnish on FreeBSD 8.2
http://gala4th.blogspot.com/2012/01/install-nginx-php-fpm-and-varnish-on.html

Apache MPM Worker + mod_fastcgi + PHP-FPM
http://gala4th.blogspot.com/2011/07/apache-mpm-worker-modfastcgi-php-fpm.html

Web Performance Tuning Tips Solutions for Drupal Sites
http://gala4th.blogspot.com/2010/12/web-performance-tuning-tips-solutions.html

Running Drupal with Nginx
http://wiki.nginx.org/Drupal

high performance caching reverse proxy: Varnish (安裝架設篇)
http://gala4th.blogspot.com/2010/12/high-performance-caching-reverse-proxy.html

[FreeBSD & Linux]網站分流：簡易架設 HAProxy 伺服器
http://blog.wu-boy.com/2008/06/freebsd-linux%E7%B6%B2%E7%AB%99%E5%88%86%E6%B5%81%EF%BC%9A%E7%B0%A1%E6%98%93%E6%9E%B6%E8%A8%AD-haproxy-%E4%BC%BA%E6%9C%8D%E5%99%A8/

Scaling Rails Site：Reading Material # 1
http://wp.xdite.net/?p=1597

rsync - synchronizing two file trees strcuture

2012-01-30T10:59:00.000-08:00

rsync - synchronizing two file trees strcuture

rsync is an amazing and powerful tool for moving files around. I know of people that use it for file transfers, keeping dns server records up-to-date, and along with sshd to remote restart the services when rsync reports a file change (how they do that, I don't know, I'm just told they do it).

This article describes how you can use rsync to synchronize file trees. In this case, I'm using two websites to make sure one is a backup of the other. As an example, I'll be making sure that one box contains the same files as the other box in case I need to put the backup box into production, should a failure occur.
Overview

rsync can be used in six different ways, as documented in man rsync:

1. for copying local files. This is invoked when neither source nor destination path contains a : separator
2. for copying from the local machine to a remote machine using a remote shell program as the transport (such as rsh or ssh). This is invoked when the destination path contains a single : separator.
3. for copying from a remote machine to the local machine using a remote shell program. This is invoked when the source contains a : separator.
4. for copying from a remote rsync server to the local machine. This is invoked when the source path contains a :: separator or a rsync:// URL.
5. for copying from the local machine to a remote rsync server. This is invoked when the destination path contains a :: separator.
6. for listing files on a remote machine. This is done the same way as rsync transfers except that you leave off the local destination.

I'll only be looking at copying from a remote rsync server (4) to a local machine and when using a remote shell program (2).

Installing

This was an easy port to install (aren't they all, for the most part?). Remember, I have the entire ports tree, so I did this:

# cd /usr/ports/net/rsync
# make config-recursive
# make install clean distclean

===> The following configuration options are available for rsync-3.0.9:
     POPT_PORT=off "Use popt from devel/popt instead of bundled one"
     SSH=on "Use SSH instead of RSH"
     FLAGS=off "File system flags support patch, adds --fileflags"
     ATIMES=off "Preserve access times, adds --atimes"
     ACL=off "Add backward-compatibility for the --acls option"
     ICONV=on "Add iconv support"
     TIMELIMIT=on "Time limit patch"
===> Use 'make config' to modify these settings

If you don't have the ports tree installed, you have a bit more work to do.... As far as I know, you need rsync installed on both client and server, although you do not need to be running rsyncd unless you are connecting via method 4.

Setting up the server

Edit /etc/rc.conf
# vi /etc/rc.conf
### enable rsyncd, using IPv4 instead of the default IPv6.
rsyncd_enable="YES"
rsyncd_flags="-4"

You might run rsyncd manually. If your server only uses IPv4, then, make sure you add the "-4" argument to the command
# vi /usr/local/etc/rc.d/rsyncd

Change:
command_args="--daemon"

To:
command_args="-4 --daemon"

Edit /usr/local/etc/rsyncd.conf

In this example, we're going to be using a remote rsync server (4). On the production web server, I created the /usr/local/etc/rsyncd.conf file. The contents is based on man rsyncd.conf.

# vi /usr/local/etc/rsyncd.conf
address = 192.168.100.78

charset=utf-8

uid = www
gid = www

use chroot = no

max connections = 20

syslog facility = local5

log file = /var/log/rsyncd.log

#pid file=/var/run/rsyncd.pid
#lock file=/var/run/rsyncd.lock

max verbosity = 2

transfer logging = yes

[web]
hosts deny = 0.0.0.0/0.0.0.0
hosts allow = 192.168.1.2, 192.168.1.3

auth users = rsync_bot1, rsync_bot2
secrets file = /usr/local/etc/rsyncd.secrets

path = /www/rsync_tmp
comment = whoe www (approx 10gb)

read only = no

[home_ftp]
uid = root
gid = ftp
hosts deny = 0.0.0.0/0.0.0.0
hosts allow = 192.168.1.2

path = /home/ftp
comment = ftp files

auth users = home_ftp_user0
secrets file = /usr/local/etc/rsyncd.secrets

read only = no

Note: you can choose to put setting on global level or module (section) level.

Adding any local-net entries to your /etc/hosts file so that rsync's name lookup uses that information.
# vi /etc/hosts
192.168.1.2 test2
192.168.1.3 test3

Creating Log file
# touch /var/log/rsyncd.log

automatically rotate logs
# vi /etc/newsyslog.conf
### rsync
/var/log/rsyncd.log 600 9 100000 * Z

Create the Secret File
# touch /usr/local/etc/rsyncd.secrets

# vi /usr/local/etc/rsyncd.secrets
rsync_bot1:mypass
rsync_bot2:mypass
home_ftp_user0:mypass

Make /usr/local/etc/rsyncd.conf non-world readable:
# chmod 440 /usr/local/etc/rsyncd.secrets
# chown root:wheel /usr/local/etc/rsyncd.secrets

You'll note that I'm running rsync as www:www (or rsync:rsync depends on your situation).

# cat /etc/master.passwd | grep www
www:*:80:80::0:0:World Wide Web Owner:/nonexistent:/usr/sbin/nologin

# cat /etc/group | grep www
www:*:80:

Then I started the rsync daemon and verified it was running by doing this:
# /usr/local/etc/rc.d/rsyncd start

Monitor rsync log file
# tail -F /var/log/rsyncd.log
rsyncd version 3.0.6 starting, listening on port 873

# ps auxww | grep rsync
root 737 0.0 0.1 3128 1348 ?? Is 10:50AM 0:00.00 /usr/local/bin/rsync -4 --daemon

# sockstat | grep rsync
root rsync 1763 4 tcp4 *:873 *:*

Then I verified that I could connect to the daemon by doing this:

# telnet localhost 873
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
@RSYNCD: 21

I determined the port 873 by looking at man rsyncd.conf.

Open rsync port in firewall.
# vi /usr/local/etc/ipfw.rules
### rsync
$IPF 260 allow tcp from 192.168.1.2 to any 873 in

Rotate rsync log file:
# vim /etc/newsyslog.conf
/var/log/rsyncd.log 600 7 100000 * JC /var/run/rsyncd.pid

# /etc/rc.d/newsyslog restart

Setting up the client

You may have to install rsync on the client as well.. There wasn't much to set up on the client. I merely issued the following command. The rsync server in question is 192.168.1.1.

Create the password file first
# echo "mypass" > /usr/local/etc/rsyncd.passwd_rsync_bot1

Note: put password ONLY. Do NOT put username!

Perform a dry run (pseudo):
# rsync -navu --stats --safe-links --password-file=/usr/local/etc/rsyncd.passwd_rsync_bot1 rsync_bot1@192.168.1.1::web /www/rsync_tmp/

Note: -n parameter makes rsync perform a trial run preview that doesn't make any changes (and produces mostly the same output as a real run).

Pulling remote files from the remote rsync daemon:
# rsync -avu --stats --safe-links --password-file=/usr/local/etc/rsyncd.passwd_rsync_bot1 rsync_bot1@192.168.1.1::web /www/rsync_tmp/

Note: -a parameter turns on archive mode. Bascially this causes rsync to recurse the directory copying all the files and directories and perserving things like case, permissions, and ownership on the target. (Note: Ownership may not be preserved if you are not logged in as the root user.)

Note: -v parameter turns on verbose mode.

Note: the reason why I added the --safe-links parameter is because without it, the symbolic link files will be messed up.

Pushing local files to the remote rsync daemon:
# rsync -avu --stats --safe-links --password-file=/usr/local/etc/rsyncd.passwd_rsync_bot1 /www/rsync_tmp/ rsync_bot1@192.168.1.1::web

Note: do not forget the trailing slash of the directory path.

Rsync between two local directories:
# rsync -avu --stats --safe-links --iconv=CP950,utf-8 --exclude='*.svn' --exclude='*.log' /source/path/ /destination/path/

Note: if the filenames on the source server contain traditional chinese characters, make sure you do include the --iconv option.

To mount a remote Microsoft shared samba SMB / CIFS directories folders:
# mkdir /path/to/local/mnt
# mount_smbfs -f 400 -d 500 -I 1.2.3.4 //Username@NetBIOS-Server-Name/SharedFolder /path/to/local/mnt

-I 1.2.3.4 // Do not use NetBIOS name resolver and connect directly to host, which can be either a valid DNS name or an IP address.

Avoid password prompt:

Use smbutil to generate encrypted password:
# smbutil crypt MyPassword
$$14144762c293a0314e6e1

You need to create a ~/.nsmbrc file as follows:
# vim ~/.nsmbrc

Set username and password as follows:

[NetBIOS-Server-Name:Username]
password=$$14144762c293a0314e6e1

Now mount the directory as follows:
# mount_smbfs -f 400 -d 500 -N -I 10.1.2.3 //Username@NetBIOS-Server-Name/SharedFolder /path/to/local/mnt

The -N option forces to read a password from ~/.nsmbrc file. At run time, mount_smbfs reads the ~/.nsmbrc file for additional configuration parameters and a password. If no password is found, mount_smbfs prompts for it. You need to use the -N option while writing a shell script.

Note: ~/.nsmbrc Keeps static parameters for connections and other information. See /usr/share/examples/smbfs/dot.nsmbrc for details.

Note: man mount_smbfs

Mount the shared folder on system startup

mount_smbfs does not make the mount permanent. If the FreeBSD system is rebooted, you will have to mount the share again. To make the mount occur each time you start the FreeBSD system, you can put an entry in your /etc/fstab file. An example file would look like this:

//myUser@serverName/mySharedFolder /mnt/mySharedFolder smbfs rw,-N,-I192.168.1.1 0 0

If the share is password protected, don't forget to create ~/.nsmbrc with your usename and password.

Example:

From:
/some_path/test_link/

To:
/rsyncd-munged//some_path/test_link/

Other resource
# man rsync

# man rsyncd.conf

Connecting to remote rsync server via SSH

ServerA 192.168.1.1 // the server that is running rsync server.

ServerB 192.168.1.2 // the rsync client that is used for pulling and pushing files from the rsync server.

Pulling remote files from the remote rsync server:
ServerB # rsync -e ssh -avu --stats my_account@192.168.1.1:/www/rsync_tmp/ /www/rsync_tmp/

Pushing local files to the remote rsync server:
ServerB # rsync -e ssh -avu --stats /www/rsync_tmp/ my_account@192.168.1.1:/www/rsync_tmp/

Note: do not forget the trailing slash.

ServerB # ssh-keygen -t dsa

ServerB # scp ~/.ssh/id_dsa.pub my_account@192.168.1.1:/home/some_account

ServerA # cat id_dsa.pub >> ~/some_account/.ssh/authorized_keys2

ServerB # ssh my_account@192.168.1.1

=====================================================================

FAQ

Problem: name lookup failed for 192.168.100.157: hostname nor servname provided, or not known

Solution:
> Is there a way to prevent rsyncd from doing reverse IP lookups on
> connecting clients? I didn't find any config option for this.

There is no such option in the rsync code. I'd suggest adding any
local-net entries to your /etc/hosts file so that rsync's name lookup
uses that information.

Problem: secrets file must be owned by root when running as root (see strict modes)

Solution: Set both server side and client's secret file owned by root XD

Problem: ERROR: module is read only

Solution: add following line to the /usr/local/etc/rsyncd.conf file.
read only = no
=====================================================================

> > @ERROR: access denied to home from localhost (127.0.0.1)
>
> This is the important bit. This means that you got through
> to the rsync daemon and it rejected your access. The log
> file for the daemon will have more explicit information
> (which is hidden from the client on purpose), but I'd imagine
> that you need to add localhost to the list of acceptable IPs
> that are authorized to connect.

Rsync.conf
-------------------------------------------------
use chroot = no
strict modes = yes
auth users = backup
secrets file = /etc/rsyncd.secrets
hosts allow = *, localhost, 127.0.0.1, 192.168.180.53
log file = /var/log/rsyncd.log
max verbosity = 2
transfer logging = yes

[home]
path = /cygdrive/d/home/
exclude = .ssh/ .ssh/** TEST/ TEST/**
read only = no
timeout = 600
--------cut-other-modul-configuration---
========================================================================
- > I notice that the performance is pretty slow ranges between 2 and 6
> MB/s.
>
> The command that I use from a remote machine to this, the archiving
> host, is:
>
> rsync -avuz -e ssh ./dir archsrv:/archive/DATA
>
>
> An investigation with top shows that the system is cpu bound rather than
> IO bound and that the sshd process is consuming 75% of the CPU compared
> to the rsync process which uses about 25%.
>
> Why is ssh using so much CPU? It seems wrong to me. I would expect rsync
> to be using most as it has to do compression.

ssh has to do encryption, which is pretty CPU-intensive stuff.
You can tell ssh to use an encryption method that is less CPU-intensive,
such as arcfour; your command would look like this:

rsync -avuz -e 'ssh -c arcfour' ./dir archsrv:/archive/DATA

Alternatively, if security permits, use an rsync daemon.
======================================

Reference:
http://gala4th.blogspot.com/2010/02/rsync-synchronizing-two-file-trees_19.html
不使用密碼的SSH連線 - ssh-keygen
http://www.cyberciti.biz/faq/mounting-a-nas-with-freebsd-mount_smbfs/
http://blog.up-link.ro/freebsd-how-to-mount-smb-cifs-shares-under-freebsd/
http://www.freebsddiary.org/rsync.php
http://www.freebsddiary.org/secure-file-copy.php
http://www.freebsddiary.org/ssh-authorized-keys.php
http://blog.weithenn.org/2009/05/freebsdrsync.html
http://www.sanitarium.net/golug/rsync_backups_2010.html
http://slv922.pixnet.net/blog/post/26419814
http://lists.samba.org/archive/rsync/2005-October/013649.html

Setup a Pure-FTPd FTP server with virtual users

2012-01-30T10:58:00.001-08:00

Setup a Pure-FTPd FTP server with virtual users

Pure-FTPd is a free (BSD), secure, production-quality and standard-conformant FTP server.

This guide provides instructions for using the virtual user system to manage and control users. By using virtual users, FTP accounts can be administrated without affecting system accounts.

Let's initiate Pure-FTPd's installation by entering the following commands:

% su
# portsnap update
# cd /usr/ports/ftp/pure-ftpd
# make config-recursive

A menu containing Pure-FTPd options will pop-up. In my case, I've opted to leave these options at their defaults.

# cat /var/db/ports/pure-ftpd/options
WITH_UTF8=true
WITH_LARGEFILE=true

# make install clean distclean
# rehash

Having finished the installation process we now move into the configuration stage. We'll start by copying the sample configuration file and set the configuration options:

# cd /usr/local/etc
# cp pure-ftpd.conf.sample pure-ftpd.conf
# chmod 444 pure-ftpd.conf

The chmod command was run to be able to edit the file (default permissions are set to -r--r--r--).

# vim pure-ftpd.conf
# Port range for passive connections replies. - for firewalling.
PassivePortRange 30000 30900

# IP address/port to listen to (default=all IP and port 21).
Bind 192.168.100.1,21

# If you want to log all client commands, set this to "yes".
# This directive can be duplicated to also log server responses.
VerboseLog yes

# PureDB user database (see README.Virtual-Users)
PureDB /usr/local/etc/pureftpd.pdb

# Create an additional log file with transfers logged in the standard W3C
# format (compatible with most commercial log analyzers)
AltLog w3c:/var/log/pureftpd.log

# Automatically create home directories if they are missing
CreateHomeDir yes

# Disallow anonymous connections. Only allow authenticated users.
NoAnonymous yes

# Disallow anonymous users to upload new files (no = upload is allowed)
AnonymousCantUpload yes

The CreateHomeDir option makes adding virtual users more easy by creating a user's home directory upon login (if it doesn't already exist).

We can either import users with system-level accounts (defined in /etc/master.passwd) at once or create new users manually. To import users that already exist on your system into the virtual user database, enter these commands:

# pure-pwconvert >> /usr/local/etc/pureftpd.passwd
# chmod 600 /usr/local/etc/pureftpd.passwd
# pure-pw mkdb

It should be noted that pure-pwconvert only imports accounts that have shell access. Accounts with the shell set to nologin have to be added manually.

To add users to the Pure-FTPd virtual user database manually, we need to create a system-level account that will be associated with virtual users. Create a new user named ftp like this:

# pw useradd -n ftp -s /sbin/nologin -w no -d /home/ftp -c "FTP user" -m

Note: When you install pureftp, an ftp group is created, but no ftp user; this results in the error "mail pure-ftpd:(?:?) [ERROR] Unable to find the 'ftp' account". So we need to manually create the ftp user.

Having done this we can now add users to the virtual users database using the commands below:

# pure-pw useradd ftp_test -u ftp -g ftp -d /home/ftp/ftp_test
# pure-pw mkdb

Replace user with the desired username. With -d flag, the user will be chrooted. If you want to give user access to the whole filesystem, use -D instead of -d.

If you want to add additional users, just repeat the commands above with a different user.

Allow clients' IP addresse:
# pure-pw usermod user_name -r 209.17.11.12
# pure-pw mkdb

To reset a user's password:
# pure-pw passwd user_name
# pure-pw mkdb

To view info about one user:
# pure-pw show user_name

To view a list of users:
# pure-pw list

To remove a user:
# pure-pw userdel user_name
# pure-pw mkdb

Now to start Pure-FTPd:

# /usr/local/etc/rc.d/pure-ftpd onestart

Initiate a FTP connection to test the server:

% ftp localhost

    Trying 127.0.0.1...
    Connected to localhost.
    220---------- Welcome to Pure-FTPd [TLS] ----------
    220-You are user number 2 of 50 allowed.
    220-Local time is now 13:39. Server port: 21.
    220-IPv6 connections are also welcome on this server.
    220 You will be disconnected after 15 minutes of inactivity.
    Name (localhost:username):

Now log in with a user account created as explained above. Commands such as ls, cp, pwd and less work just like in tcsh and bash shells. To quit the FTP session type exit.

To configure Pure-FTPd to start at boot time:

# echo 'pureftpd_enable="YES"' >> /etc/rc.conf

To restart Pure-FTPd and determine if it is running:

# /usr/local/etc/rc.d/pure-ftpd restart
# /usr/local/etc/rc.d/pure-ftpd status

# sockstat | grep :21

Rotate pure-ftpd log file:
# vim /etc/newsyslog.conf
/var/log/pureftpd.log 600 7 100000 * JC /var/run/pure-ftpd.pid

# /etc/rc.d/newsyslog restart

Pure-FTPd provides useful features for personal users as well as hosting providers. I've only touched the tip of the iceberg so do take a look at the project's website for the excellent documentation that is available.

PF supports FTP servers and FTP clients behind NAT

WAN0 IP: 11.11.11.11
WAN1 IP: 22.22.22.22

FTP0 server (behind NAT) IP: 192.168.100.1
FTP1 server (behind NAT) IP: 192.168.100.2

# vim /etc/rc.local
### ftp-proxy for outside world FTP clients accessing internal FTP server behand NAT.
/usr/sbin/setfib 0 /usr/sbin/ftp-proxy -R 192.168.100.1 -p 21 -b 11.11.11.11
/usr/sbin/setfib 1 /usr/sbin/ftp-proxy -R 192.168.100.2 -p 21 -b 22.22.22.22

### Note:
### -b // bind to the external IP.
### -R // redirect to internal FTP server IP.

### ftp-proxy for internal FTP clients behand NAT accessing outside world FTP servers.
/usr/sbin/ftp-proxy -p 8021 -b 127.0.0.1

# vim /etc/pf.conf
### [FTP] RDR DMZ to Outside (through ftp-proxy)
### Note: make sure you have this line "ftpproxy_enable="YES"" in your /etc/rc.conf.
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr pass proto tcp from $dmz_if/24 to any port 21 -> 127.0.0.1 port 8021

### enabloe log for debugging purpose.
block log all

### [FTP] from ROUTER to ANY
pass out quick log on $wan_if0 proto tcp from $wan_if0 to any port 21 rtable 0
pass out quick log on $wan_if0 proto tcp from $wan_if0 to any port > 1023 rtable 0

pass out quick log on $wan_if1 proto tcp from $wan_if1 to any port 21 rtable 1
pass out quick log on $wan_if1 proto tcp from $wan_if1 to any port > 1023 rtable 1

### [FTP] from ANY to $dmz_ftp0_ip0 $dmz_ftp1_ip0
pass in quick log on $wan_if0 proto tcp from any to $wan_if0 port 21 rtable 0
pass in quick log on $wan_if0 proto tcp from any to $dmz_ftp0_ip0 port 30000:30900 rtable 0
pass in quick log on $wan_if1 proto tcp from any to $wan_if1 port 21 rtable 1
pass in quick log on $wan_if1 proto tcp from any to $dmz_ftp1_ip0 port 30000:30900 rtable 1

pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp0_ip0 port 21 rtable 0
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp0_ip0 port 30000:30900 rtable 0
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp1_ip0 port 21 rtable 1
pass out quick log on $dmz_if proto tcp from $dmz_if to $dmz_ftp1_ip0 port 30000:30900 rtable 1

### [FTP] We need to have an anchor for ftp-proxy.
anchor "ftp-proxy/*"

### [FTP] from DMZ to ANY
pass in quick log on $dmz_if proto tcp from $dmz_if/24 to any port 21 rtable 0

Flush the pf.conf setting:
# pfctl -f /etc/pf.conf

Try to connect to ftp:
# ftp ftp.freebsd.org
Name: anonymous
Password: anonymous

Reference:
Setup a Pure-FTPd FTP server with virtual users
http://gala4th.blogspot.com/2012/01/setup-pure-ftpd-ftp-server-with-virtual.html

http://gala4th.blogspot.com/2012/01/pf-supports-ftp-servers-and-ftp-clients.html
http://forums.freebsd.org/showthread.php?t=591

Set up OpenVPN server on FreeBSD running PF packet filter firewall and run VPN client on Windows 7 and FreeBSD 8.2

2012-01-30T10:58:00.000-08:00

Set up OpenVPN server on FreeBSD running PF packet filter firewall and run VPN client on Windows 7 and FreeBSD 8.2

OpenVPN is a free and open source software application that implements virtual private network (VPN) techniques for creating secure point-to-point or site-to-site connections in routed or bridged configurations and remote access facilities. It uses a custom security protocol[2] that utilizes SSL/TLS for key exchange. It is capable of traversing network address translators (NATs) and firewalls. It was written by James Yonan and is published under the GNU General Public License (GPL)

My server has two WAN connections from two different ISP, so the setting in this tutorial allows the VPN clients to connect to the VPN server with my two public IP addresses. However, you can still use this tutorial to set up the OpenVPN server with a single WAN interface.

If you need help on how to set up multiple WAN interfaces with two different ISP:
http://gala4th.blogspot.com/2011/10/multiple-default-routes-gateways-with.html

Running OpenVPN in different scenarios:
1. Single OpenVPN instance on single interface.
2. Multiple OpenVPN instances on single interface. Ex: you want different rules for each set of VPN clients.
3. Single OpenVPN instance on each of multiple interfaces.
4. Multiple OpenVPN instance on each of multiple interfaces.

We will set up OpenVPN in "Single OpenVPN instance on each of multiple interfaces" scenario.

Install OpenVPN:
# cd /usr/ports/security/openvpn
# make config-recursive
# make config-recursive
# make config-recursive

Note: run "make config-recursive" few times. You may enable a dependency the first time through that has its own configuration options.

# make install

# mkdir /usr/local/etc/openvpn

This /usr/local/etc/rc.d/openvpn script supports running multiple instances of openvpn. To run additional instances link this script to something like:

For example:
# ln -s openvpn openvpn_foo

and define additional openvpn_foo_* variables in one of:
/etc/rc.conf, /etc/rc.conf.local or /etc/rc.conf.d/openvpn_foo

A list of avaiable interface on my server:

# ifconfig -l | tr ' ' '\n'
wan0 // IP 11.11.11.11. First WAN interface for my first ISP.
wan1 // IP 22.22.22.22. Second WAN interface for my second ISP.
dmz0 // IP 192.168.100.1 and 192.168.100.2. DMZ interface.
lan0 // IP 192.168.200.1 and 192.168.200.2. LAN interface.
tun0 // IP 10.8.0.1. First VPN interface for VPN clients connecting to first WAN interface.
tun1 // IP 10.9.0.1. Second VPN interface for VPN clients connecting to second WAN interface.

# vim /etc/rc.conf
ifconfig_wan0="inet 11.11.11.11 netmask 255.255.255.0"
ifconfig_wan1="inet 22.22.22.22 netmask 255.255.255.0"

ifconfig_dmz0="inet 192.168.100.1 netmask 255.255.255.0"
ifconfig_dmz0_alias0="inet 192.168.100.2 netmask 255.255.255.255"

ifconfig_lan0="inet 192.168.200.1 netmask 255.255.255.0"
ifconfig_lan0_alias0="inet 192.168.200.2 netmask 255.255.255.255"

// Note: do this if you want to run one instance of OpenVPN on each of two interfaces.
# ln -s /usr/local/etc/rc.d/openvpn /usr/local/etc/rc.d/openvpn_wan0
# ln -s /usr/local/etc/rc.d/openvpn /usr/local/etc/rc.d/openvpn_wan1

Sample configuration files can be found in:
# ls -l /usr/local/share/doc/openvpn/sample-config-files

Copy the OpenVPN config file
# cp /usr/local/share/doc/openvpn/sample-config-files/server.conf /usr/local/etc/openvpn/openvpn_wan0.conf

# cp /usr/local/share/doc/openvpn/sample-config-files/server.conf /usr/local/etc/openvpn/openvpn_wan1.conf

Note: The reason why we need to rename the config filename from server.conf to openvpn_foo.conf is because the /usr/local/etc/rc.d/openvpn script supports running multiple instances of openvpn. The openvpn script will try to read the config from the SCRIPT_NAME.conf. For example:

If you are startup script is called /usr/local/etc/rc.d/openvpn_wan0, it will read config file from /usr/local/etc/openvpn/openvpn_wan0.conf.

Edit OpenVPN config file:
# vim /usr/local/etc/openvpn/openvpn_wan0.conf
### Specify device
dev tun

### Which local IP address should OpenVPN
### listen on? (optional)
local 11.11.11.11

### Certificates for VPN Authentication
ca /usr/local/etc/openvpn/keys/ca.crt
cert /usr/local/etc/openvpn/keys/server.crt
key /usr/local/etc/openvpn/keys/server.key # This file should be kept secret

### Diffie hellman parameters
dh /usr/local/etc/openvpn/keys/dh1024.pem

### Server and client IP and Pool
### The server will take 10.8.0.1 for itself,
### the rest will be made available to clients.
server 10.8.0.0 255.255.255.0

# Maintain a record of client <-> virtual IP address
# associations in this file. If OpenVPN goes down or
# is restarted, reconnecting clients can be assigned
# the same virtual IP address from the pool that was
# previously assigned.
ifconfig-pool-persist ipp.txt

### Routes to push to the client.
### The specified network will be added automatically on the VPN client.
push "route 192.168.0.0 255.255.255.0"

### Enable compression on the VPN link.
### If you enable it here, you must also
### enable it in the client config file.
comp-lzo

### Make the link more resistant to connection failures
keepalive 10 120
persist-tun
persist-key

### optional ?
#ping-timer-rem

# It's a good idea to reduce the OpenVPN
# daemon's privileges after initialization.
#
# You can uncomment this out on non-Windows systems.
# By specifying 'nobody' OpenVPN service privileges are
# dropped to nobody. This increases system security.
user nobody
group nobody

# By default, log messages will go to the syslog (or
# on Windows, if running as a service, they will go to
# the "\Program Files\OpenVPN\log" directory).
# Use log or log-append to override this default.
# "log" will truncate the log file on OpenVPN startup,
# while "log-append" will append to it. Use one
# or the other (but not both).
;log openvpn.log
log-append /var/log/openvpn_wan0.log

# Set the appropriate level of log
# file verbosity.
#
# 0 is silent, except for fatal errors
# 4 is reasonable for general usage
# 5 and 6 can help to debug connection problems
# 9 is extremely verbose
verb 3

For openvpn_wan1.conf, most settings are similar to the previous openvpn_wan0.conf file, except:

# vim /usr/local/etc/openvpn/openvpn_wan1.conf
### Which local IP address should OpenVPN
### listen on? (optional)
local 22.22.22.22

### Server and client IP and Pool
### The server will take 10.9.0.1 for itself,
### the rest will be made available to clients.
server 10.9.0.0 255.255.255.0

log-append /var/log/openvpn_wan1.log

Review the difference between openvpn_wan0.conf and openvpn_wan1.conf:
# diff /usr/local/etc/openvpn/openvpn_wan0.conf /usr/local/etc/openvpn/openvpn_wan1.conf

Logging
# touch /var/log/openvpn_wan0.log
# touch /var/log/openvpn_wan1.log

Rotate OpenVPN log files:
# vim /etc/newsyslog.conf
/var/log/openvpn_wan0.log 600 9 100000 * JC /var/run/openvpn_wan0.pid
/var/log/openvpn_wan1.log 600 9 100000 * JC /var/run/openvpn_wan1.pid

# /etc/rc.d/newsyslog restart

Copy easy-rsa script to the home directory
# cp -r /usr/local/share/doc/openvpn/easy-rsa ~
# find ~/easy-rsa/ -type d -print0 | xargs -0 -I @ chmod 700 @
# find ~/easy-rsa/ -type f -print0 | xargs -0 -I @ chmod 400 @

Make sure these scripts are executable:
# find ~/easy-rsa/2.0/ -type f -print0 | xargs -0 -I @ chmod u+x @

Change follow lines in vars file to reflect your setting:
# cd ~/easy-rsa/2.0/
# vim ~/easy-rsa/2.0/vars
### These are the default values for fields
### which will be placed in the certificate.
### Don't leave any of these fields blank.
export KEY_COUNTRY="CA"
export KEY_PROVINCE="BC"
export KEY_CITY="Vancouver"
export KEY_ORG="Fort-Funston Ltd"
export KEY_EMAIL="me@myhost.mydomain"
export KEY_CN="CommonName_Can_Be_YourName_or_HostName"
export KEY_NAME="YourName"
export KEY_OU="UnitName_or_DepartmentName"
export PKCS11_MODULE_PATH="dummy"
export PKCS11_PIN="A_RANDOM_PASSWORD"

Very Important Step for FreeBSD/TCSH users. FreeBSD ships with tcsh as its native shell, at the time of writing the following scripts do not work. To get around this you must drop down to a bourne shell. To do this just type the following at a prompt:

# sh
# cd ~/easy-rsa/2.0/

Now you can carry on with building the certificates, once you have built them you can exit back out to tcsh.

Sourcing the vars file and other scripts:
# . vars
# ./clean-all
# ./build-ca

You will have to answer a few questions on the last step, once this has been done your CA certs will be created in the keys subdirectory.

Generate certificate & key for server:
# ./build-key-server server
A challenge password []: LEAVE IT BLANK AND PRESS ENTER
An optional company name []: LEAVE IT BLANK AND PRESS ENTER

Again answer the questions and the certs will be placed in the keys subdirectory.

Generate certificates & keys for 3 clients (each client will require their own certificates, if multiple clients log in with the same certs then they will be assigned the same ips and will kick each other off the network):

Generating client certificates is very similar to the previous step. You need to ensure that all your details are the same as for the CA, apart from the common name, which needs to be different for each client. For the sake of clarity this should relate to person who is assigned this vpn certificate. All of these details can be found in keys/server.crt for the server and keys/client*.crt for the client details.

# ./build-key client1
# ./build-key client2
# ./build-key client3

Generate Diffie Hellman parameters

Diffie Hellman parameters must be generated for the OpenVPN server:

# ./build-dh

Exit the sh shell:
# exit

Now move the whole keys directory to /usr/local/etc/openvpn:

# mv ~/easy-rsa/2.0/keys /usr/local/etc/openvpn/

Protect our certificate keys directory and config files for only root can see it.
# find /usr/local/etc/openvpn/keys/ -type d -print0 | xargs -0 -I @ sh -c 'chmod 700 @ ; chown root:wheel @'
# find /usr/local/etc/openvpn/keys/ -type f -print0 | xargs -0 -I @ sh -c 'chmod 400 @ ; chown root:wheel @'
# find /usr/local/etc/openvpn/ -type f -name '*.conf' -print0 | xargs -0 -I @ sh -c 'chmod 400 @ ; chown root:wheel @'

You need to edit /etc/rc.conf if you set up single instance of OpenVPN with single interface:

Note: I have commented out following lines because I set to start OpenVPN daemon in /etc/rc.local later.

# vim /etc/rc.conf
#openvpn_wan0_enable="YES"
#openvpn_wan0_configfile="/usr/local/etc/openvpn/openvpn_wan0.conf"

#openvpn_wan1_enable="YES"
#openvpn_wan1_configfile="/usr/local/etc/openvpn/openvpn_wan1.conf"

We will be using the setfib command to run our utilities with an different routing table:

Note: You do not need the setfib command if you are hosting your OpenVPN server with single WAN interface.

# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable0
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable1

Edit sshd_config.* config file:
# vim /etc/ssh/sshd_config.rtable0
### IP address of first WAN interface
ListenAddress 11.11.11.11
### IP address of DMZ interface
ListenAddress 192.168.100.1
### IP address of LAN interface
ListenAddress 192.168.200.1

# vim /etc/ssh/sshd_config.rtable1
### IP address of second WAN interface
ListenAddress 22.22.22.22
### IP address of DMZ interface
ListenAddress 192.168.100.2
### IP address of LAN interface
ListenAddress 192.168.200.2

Note: Multiple ListenAddress options are permitted.

Note: you can let sshd to listen to the IP address of the tunnel interface. However, I do not recommend you do so. Because if OpenVPN did not start properly, sshd will have problem to start as well (because it could not listen to the specified IP address).

Edit rc.local:
# vim /etc/rc.local
### add first routing table for first WAN interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 11.11.11.11

### add second routing table for second WAN interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 22.22.22.22

### start OpenVPN on the first WAN interface.
/usr/sbin/setfib 0 /usr/local/etc/rc.d/openvpn_wan0 onestart

### start OpenVPN on the second WAN interface.
/usr/sbin/setfib 1 /usr/local/etc/rc.d/openvpn_wan1 onestart

### Use different routing table for each sshd daemon.
### Note: we need to start SSH after OpenVPN started
### since the IP address of tunnel interface (for VPN) has been listened by sshd,
### which is defined in the sshd_config.rtable0 and sshd_config.rtable1 config file.
### Note: make sure the line 'sshd_enable="YES"' has bee commented out
### or removed from /etc/rc.conf file. We will start sshd in /etc/rc.local file.
/usr/sbin/setfib 0 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable0
/usr/sbin/setfib 1 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable1

### Flush PF rules after OpenVPN and SSH daemons started
### Note: the reason is because there are some settings related to these VPN
### interface tun0 and tun1. So we need to flush the PF rules after OpenVPN loaded.
/sbin/pfctl -f /etc/pf.conf

Comment out or remove sshd_enable="YES" line in /etc/rc.conf:
# vim /etc/rc.conf
### Note: commented out for multiple WAN routes setup. SSH daemon will be started in /etc/rc.local file.
#sshd_enable="YES"

Let's start OpenVPN server manually now:

# setfib 0 /usr/local/etc/rc.d/openvpn_wan0 onestart
Starting openvpn_wan0.
add net 10.8.0.0: gateway 10.8.0.2

# setfib 1 /usr/local/etc/rc.d/openvpn_wan1 onestart
Starting openvpn_wan1.
add net 10.9.0.0: gateway 10.9.0.2

Make sure the network and the gateway have been correctly added:

# setfib 0 netstat -rn | egrep '10.[8|9].0'
10.8.0.0/24 10.8.0.2 UGS 0 0 tun0
10.8.0.1 link#6 UHS 0 0 lo0
10.8.0.2 link#6 UH 0 0 tun0
10.9.0.1 link#7 UHS 0 0 lo0
10.9.0.2 link#7 UH 0 0 tun1

# setfib 1 netstat -rn | egrep '10.[8|9].0'
10.8.0.2 link#6 UH 0 0 tun0
10.9.0.0/24 10.9.0.2 UGS 0 0 tun1
10.9.0.2 link#7 UH 0 0 tun1

In some cases, the network did not be added successfully, you will need to add the network maually:
# setfib 0 route add 10.8.0.0/24 10.8.0.2
# setfib 1 route add 10.9.0.0/24 10.9.0.2

You should have seen two different IP addresses are listening port 1194:

# netstat -an | grep 1194
udp4 0 0 11.11.11.11.1194 *.*
udp4 0 0 22.22.22.22.1194 *.*

You should have seen two different process IDs are running:

# ls -l /var/run/openvpn*.pid
-rw-r--r-- 1 root wheel 6 Dec 5 15:49 /var/run/openvpn_wan1.pid
-rw-r--r-- 1 root wheel 6 Dec 5 15:44 /var/run/openvpn_wan0.pid

# ifconfig | grep tun
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
tun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500

Let's start ssh daemon now:
# /usr/sbin/setfib 0 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable0
# /usr/sbin/setfib 1 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable1

See port 22 is binding to which IP addresses:
# netstat -an | grep 22
tcp4 0 0 11.11.11.11.22 *.* LISTEN
tcp4 0 0 192.168.100.1.22 *.* LISTEN
tcp4 0 0 192.168.200.1.22 *.* LISTEN
tcp4 0 0 10.8.0.1.22 *.* LISTEN
tcp4 0 0 22.22.22.22.22 *.* LISTEN
tcp4 0 0 192.168.100.2.22 *.* LISTEN
tcp4 0 0 192.168.200.2.22 *.* LISTEN
tcp4 0 0 10.9.0.1.22 *.* LISTEN
tcp4 0 0 *.22 *.* LISTEN

To show listening/opening TCP port:
# sockstat -Ptcp

To show listening/opening UDP port:
# sockstat -Pudp

Note: man sockstat

Restart the machine and re-check everything is fine after reboot:
# sync; sync; sync; reboot

Edit pf.conf PF firewall setting:
# vim /etc/pf.conf
wan_if0 = "wan0"
wan_if1 = "wan1"

dmz_if = "dmz0"
lan_if = "lan0"

### Note: the reason why we specified VPN network below is because we want to avoid
### PF parse host specification error. Since PF runs before OpenVPN starts,
### PF could not figure it out the VPN network until OpenVPN started.
vpn_if0 = "tun0"
vpn_net0 = '"10.8.0.0/24"'
vpn_if1 = "tun1"
vpn_net1 = '"10.9.0.0/24"'

icmp_types = "{echoreq echorep unreach}"

### [NAT] the VPN connections (for access to the remote secure networks)
nat on $wan_if0 from $vpn_net0to any -> $wan_if0
nat on $wan_if1 from $vpn_net1 to any -> $wan_if1

### [VPN] Inbound from Outside
### Establish VPN connections on each WAN interface.
pass in quick on $wan_if0 proto udp from any to $wan_if0 port 1194 keep state rtable 0
pass in quick on $wan_if1 proto udp from any to $wan_if1 port 1194 keep state rtable 1

### [SSH] Inbound from VPN to ANY
pass in quick log on $vpn_if0 proto tcp from $vpn_net0to any port 22 flags S/SA keep state rtable 0
pass in quick log on $vpn_if1 proto tcp from $vpn_net1 to any port 22 flags S/SA keep state rtable 1

### [SSH] Outbound to LAN
pass out quick log on $lan_if proto tcp from {$dmz_if/24 $lan_if/24 $vpn_if0/24} to $lan_if/24 port 22 flags S/SA keep state rtable 0
pass out quick log on $lan_if proto tcp from {$vpn_net1} to $lan_if/24 port 22 flags S/SA keep state rtable 1

### [SSH] Outbound to DMZ
pass out quick log on $dmz_if proto tcp from {$dmz_if/24 $lan_if/24 $vpn_if0/24} to $dmz_if/24 port 22 flags S/SA keep state rtable 0
pass out quick log on $dmz_if proto tcp from {$vpn_net1} to $dmz_if/24 port 22 flags S/SA keep state rtable 1

### [ICMP] Inbound from VPN
pass in quick on $vpn_if0 proto icmp from $vpn_net0to any icmp-type $icmp_types rtable 0
pass in quick on $vpn_if1 proto icmp from $vpn_net1 to any icmp-type $icmp_types rtable 1

### [ICMP] Inbound from Outside
pass in quick log on $wan_if0 proto icmp from any to $wan_if0 icmp-type $icmp_types rtable 0
pass in quick log on $wan_if1 proto icmp from any to $wan_if1 icmp-type $icmp_types rtable 1

### [ICMP] Outbound to Outside
pass out quick on $wan_if0 proto icmp from $wan_if0 to any icmp-type $icmp_types rtable 0
pass out quick on $wan_if1 proto icmp from $wan_if1 to any icmp-type $icmp_types rtable 1

### [ICMP] Outbound to LAN
pass out quick log on $lan_if proto icmp from {$lan_if $dmz_if/24 $vpn_if0/24} to $lan_if/24 icmp-type $icmp_types rtable 0
pass out quick log on $lan_if proto icmp from {$vpn_net1} to $lan_if/24 icmp-type $icmp_types rtable 1

### [ICMP] Outbound to DMZ
pass out quick log on $dmz_if proto icmp from {$dmz_if $lan_if/24 $vpn_if0/24} to $dmz_if/24 icmp-type $icmp_types rtable 0
pass out quick log on $dmz_if proto icmp from {$vpn_net1} to $dmz_if/24 icmp-type $icmp_types rtable 1

Watch the pflog logged packets in real time:
# tcpdump -n -e -ttt -i pflog0

Watch the icmp packect of an interface:
# tcpdump -ni wan0 icmp
# tcpdump -ni wan1 icmp
# tcpdump -ni dmz0 icmp
# tcpdump -ni tun0 icmp
# tcpdump -ni tun1 icmp

# tcpdump -tt -i tun0

# tail /usr/local/etc/openvpn/openvpn-status.log

# tail /var/log/openvpn_wan0.log

# tail /var/log/openvpn_wan1.log

# tail /var/log/messages

Download OpenVPN 2.2.1 client for Windows:
http://openvpn.net/index.php/open-source/downloads.html

Once this is installed you will need to copy the following files from your server /usr/local/etc/openvpn/keys directory to the Windows PC C:\Program Files\Openvpn\config directory (this should be done in as secure a manner as possible, i.e. USB Stick or floppy rather than email!!!):

ca.crt
client1.crt
client1.key

Note: For the next client you will need to copy the client2.crt and client2.key files to prevent issues later.

Create VPN config file:

create a myvpn.ovpn file in C:\Program Files\Openvpn\config and insert the following:

client
remote nic1.myserver.com 1194
#remote nic2.myserver.com 1194
dev tun
comp-lzo

ca ca.crt
cert client1.crt
key client1.key

# Set log file verbosity.
verb 3

Turn off the firewall for the new Interface:

On Windows XP, the firewall can be accessed by:
Control Panel -> Security Center -> Windows Firewall -> Advanced. In the Network Connection Settings control, uncheck the box corresponding to the TAP-Win32 adapter.

Now right-click the OpenVPN Icon in your Taskbar and click "connect". Once connected try pinging the remote interface and check (using tracert) that the remote network is available. Use tcpdump on the server to check traffic too:

On Windows 7, you must run OpenVPN as an administrator by:

right click on openvpn-gui-1.0.3.exe > Property > Shortcut > Run this program as an administrator

or

right click on openvpn-gui-1.0.3.exe > Property > Compatibility > Run this program as an administrator

cmd> tracert

cmd> ping 10.8.0.1
cmd> ping 10.9.0.1

Use Firewall's VPN IP address to connecto the firewall:
cmd> putty.exe username@10.8.0.1
cmd> putty.exe username@10.9.0.1

Note: If you want to ssh to the IP address of the firewall's LAN/DMZ interface, make sure you do set up your sshd to listen to these IP addresses with proper routing table.

Use FreeBSD as a VPN client:

# cd /usr/ports/security/openvpn
# make config-recursive
# make config-recursive
# make config-recursive

Note: run "make config-recursive" few times. You may enable a dependency the first time through that has its own configuration options.

# make install

Link the openvpn startup script:
# ln -s /usr/local/etc/rc.d/openvpn /usr/local/etc/rc.d/openvpn_client

Create OpenVPN directory:
# mkdir -p /usr/local/etc/openvpn/keys

Put following three files to /usr/local/etc/openvpn/keys:
ca.crt
client1.crt
client1.key

Protect our certificate keys directory and config files for only root can see it.
# find /usr/local/etc/openvpn/keys/ -type d -print0 | xargs -0 -I @ sh -c 'chmod 700 @ ; chown root:wheel @'
# find /usr/local/etc/openvpn/keys/ -type f -print0 | xargs -0 -I @ sh -c 'chmod 400 @ ; chown root:wheel @'
# find /usr/local/etc/openvpn/ -type f -name '*.conf' -print0 | xargs -0 -I @ sh -c 'chmod 400 @ ; chown root:wheel @'

You will need to copy the client.conf instead of server.conf from /usr/local/share/doc/openvpn/sample-config-files directory:
# cp /usr/local/share/doc/openvpn/sample-config-files/client.conf /usr/local/etc/openvpn/openvpn_client.conf

Edit openvpn_client.conf:
# vim /usr/local/etc/openvpn/openvpn_client.conf
client
dev tun
proto udp

# The hostname/IP and port of the server.
# You can have multiple remote entries
# to load balance between the servers.
remote 11.11.11.11 1194
;remote my-server-2 1194

# Downgrade privileges after initialization (non-Windows only)
user nobody
group nobody

# Try to preserve some state across restarts.
persist-key
persist-tun

# Certificates for VPN Authentication
ca /usr/local/etc/openvpn/keys/ca.crt
cert /usr/local/etc/openvpn/keys/client1.crt
key /usr/local/etc/openvpn/keys/client1.key

# Verify server certificate by checking
# that the certicate has the nsCertType
# field set to "server". This is an
# important precaution to protect against
# a potential attack discussed here:
# http://openvpn.net/howto.html#mitm
ns-cert-type server

# vim /etc/rc.conf
openvpn_client_enable="YES"
openvpn_client_configfile="/usr/local/etc/openvpn/openvpn_client.conf"

Start OpenVPN client:
# /usr/local/etc/rc.d/openvpn_client start

# tail -F /var/log/messages

# ifconfig
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet 10.8.0.6 --> 10.8.0.5 netmask 0xffffffff
Opened by PID 98483

Display the full output line of ps process status command:
# ps -auxww | grep openvpn
nobody 1015 /usr/local/sbin/openvpn --cd /usr/local/etc/openvpn --daemon openvpn_wan0 --config /usr/local/etc/openvpn/openvpn_wan0.conf --writepid /var/run/openvpn_wan0.pid
nobody 1035 /usr/local/sbin/openvpn --cd /usr/local/etc/openvpn --daemon openvpn_wan1 --config /usr/local/etc/openvpn/openvpn_wan1.conf --writepid /var/run/openvpn_wan1.pid

# ls -l /var/run/openvpn*
-rw-r--r-- 1 root wheel 6 Dec 21 12:53 /var/run/openvpn_client.pid

Difference Between TUN and TAP

In computer networking, TUN and TAP are virtual network kernel devices. They are network devices that are supported entirely in software, which is different from ordinary network devices that are backed up by hardware network adapters.

TAP (as in network tap) simulates an Ethernet device and it operates with layer 2 packets such as Ethernet frames. TUN (as in network TUNnel) simulates a network layer device and it operates with layer 3 packets such as IP packets. TAP is used to create a network bridge, while TUN is used with routing.

Packets sent by an operating system via a TUN/TAP device are delivered to a user-space program that attaches itself to the device. A user-space program may also pass packets into a TUN/TAP device. In this case TUN/TAP device delivers (or "injects") these packets to the operating system network stack thus emulating their reception from an external source.
===
As far as I could see, tap allows netbios broadcast to propagate across the
VPN so Windows clients can find network shares by PC name even if no WINS
server is being used; with tun, either you find network shares in some other
way (by IP or DNS, for example) or you need to have WINS running (which often
amounts to "wins proxy = yes" in smb.conf).

On the other hand, I do not feel comfortable in having VPN clients appear as
if physically attached to the LAN (this is what bridging with TAP amounts to)
and prefer routing with tun as it makes easier to write appropriate firewall
rules for VPN clients.
===
if it's ok to create vpn on layer 3 [ one more hop between subnets ] - go for tun.

if you need to bridge two ethernet segments in two different locations - then use tap. in such setup you can have computers in the same ip subnet [ eg 10.0.0.0/24 ] on both ends of vpn, and they'll be able to 'talk' to each other directly without any changes in their routing tables. vpn will act like ethernet switch. this might sound cool and is useful in some cases but i would advice not to go for it unless you really need it. if you choose such layer 2 bridging setup - there will be a bit of 'garbage' [ that is broadcast packets ] going across your vpn.
===
I always set up tun. Tap is used by ethernet bridging in OpenVPN and introduces an unprecendented level of complexity that is simply not worth bothering with. Usually when a VPN needs to be installed, its needed now, and complex deployments don't come fast.
===
I chose "tap" when setting up a VPN for a friend who owned a small business because his office uses a tangle of Windows machines, commercial printers, and a Samba file server. Some of them use pure TCP/IP, some seem to only use NetBIOS (and thus need Ethernet broadcast packets) to communicate, and some I'm not even sure of.

If I had chosen "tun", I would probably have faced lots of broken services — lots of things that worked while you are in the office physically, but then would break when you went off-site and your laptop couldn't "see" the devices on the Ethernet subnet anymore.

But by choosing "tap", I tell the VPN to make remote machines feel exactly like they're on the LAN, with broadcast Ethernet packets and raw Ethernet protocols available for communicating with printers and file servers and for powering their Network Neighborhood display. It works great, and I never get reports of things that don't work offsite!
===
I started out using tun, but switched to tap since I didn't like the use of a /30 subnet for each PC (I need to support Windows). I found that to be wasteful and confusing.

Then I discovered the "topology subnet" option on the server. Works with the 2.1 RCs (not 2.0), but it gives me all the advantages of tun (no bridging, performance, routing, etc) with the convenience of one (sequential) IP address per (windows) machine.
===
What All Ports Are Rrequired By Domain Controllers And Client Computers?

Active Directory communication takes place using several ports. These ports are required by both client computers and Domain Controllers. As an example, when a client computer tries to find a domain controller it always sends a DNS Query over Port 53 to find the name of the domain controller in the domain.

The following is the list of services and their ports used for Active Directory communication:

UDP Port 88 for Kerberos authentication
UDP and TCP Port 135 for domain controllers-to-domain controller and client to domain controller operations.
TCP Port 139 and UDP 138 for File Replication Service between domain controllers.
TCP and UDP Port 389 for LDAP to handle normal queries from client computers to the domain controllers.
TCP and UDP Port 445 for File Replication Service
TCP and UDP Port 464 for Kerberos Password Change
TCP Port 3268 and 3269 for Global Catalog from client to domain controller.
TCP and UDP Port 53 for DNS from client to domain controller and domain controller to domain controller.
Opening above ports in Firewall between client computers and domain controllers, or between domain controllers, will enable Active Directory to function properly.

http://www.windowsnetworking.com/kbase/WindowsTips/Windows2000/AdminTips/ActiveDirectory/WhatAllPortsAreRrequiredByDomainControllersAndClientComputers.html
===
Reference:
http://gala4th.blogspot.com/2011/12/set-up-openvpn-server-on-freebsd.html
http://kuapp.com/2010/07/01/openvpn-with-freebsd-pf-and-windows-xp.html
http://www.secure-computing.net/wiki/index.php/FreeBSD_OpenVPN_Server/Routed
http://en.wikipedia.org/wiki/TUN/TAP
http://openvpn.net/archive/openvpn-users/2007-02/msg00184.html
http://serverfault.com/questions/21157/should-i-use-tap-or-tun-for-openvpn
http://forums.freebsd.org/showthread.php?t=15213
http://en.wikipedia.org/wiki/OpenVPN

Resolutions for 2012

2012-01-28T03:25:00.000-08:00

[] C

[] gcc, gdb, automake, autoconf

[] LLVM, clang, LLDB, OpenCL, KLEE

[] Git

[] Python

[] PostgreSQL

[] Dtrace

[] ZFS

[] RAID

[] English

[] NFS

remove old svn revisons history

2012-01-26T10:17:00.001-08:00

Say you have 1000 revisions and you want to shrink to only have the revisions from r950-r1000. You can do the following:

svnadmin dump /path/to/current/repo -r950:1000 > small_svn.dump
svnadmin create /path/to/new/repo
svnadmin load /path/to/new/repo < small_svn.dump Reference:
http://stackoverflow.com/questions/4350145/how-to-remove-old-svn-revisons

mysqldump read password from file for crontab

2012-01-26T09:56:00.001-08:00

mysqldump read password from file for crontab

# vim ~/.my.cnf

[client]
user = root
password = mypassword

[mysqldump]
user = root
password = mypassword

Note:
[client] section is for "mysql" command.
[mysqldump] section is for "mysqldump" command.

Make sure no other people can read .my.cnf file:
# chmod 400 ~/.my.cnf

Following two commands work:
# mysql --defaults-file=/root/.my.cnf
# mysqldump --defaults-file=/root/.my.cnf db_name > db_name.sql

or simply:
# mysql
# mysqldump db_name > db_name.sql

disable PHP script execution in upload attachment directory

2012-01-25T12:03:00.000-08:00

disable PHP script execution in upload attachment directory

// Apache

<Directory /website/attachments>
  php_flag engine off 
</Directory>

// Nginx

location /sites/default/files/ { 
  location ~ .*\.(php)?$
  { 
    deny all; 
  }
}

Multiple default routes gateways with two different ISP ipfw PF setfib load balancing

2012-01-24T11:28:00.000-08:00

Multiple default routes gateways with two different ISP ipfw PF setfib load balancing

Thanks to phoenix help I was able to setup multiple default routes, or a default route per network/interface to be precise, in Debian/Linux it is as simple as that:

# cat /etc/network/interfaces

iface eth0 inet static
    address 10.0.0.2
    netmask 255.255.255.252
    gateway 10.0.0.1

iface eth1 inet static
    address 20.0.0.2
    netmask 255.255.255.252
    gateway 20.0.0.1

That would be example topology (but more than 2 interfaces is also possible).

ISP NETWORK 0                   ISP NETWORK 1
         \                          /
          \                        /
           \                      /
          ROUTER 0            ROUTER 1
          10.0.0.1/30         20.0.0.1/30
              \                  /
        +------\----------------/------+
        |       \              /       |
        |       wan0          wan1     |
        |    10.0.0.2     20.0.0.2     |
        |                              |
        |   FREEBSD (Firewall Router)  |
        |                              |
        |             dmz0             |
        |       192.168.0.1/24         |
        |       192.168.0.2/24         |
        +------------------------------+
                       |
                   +--------+
                   | switch +
                   +--------+
                       |
                      /
               ______/
              /
    +--------/-----+
    |       /      |
    |      lan0    |
    |  192.168.0.3 |
    |  192.168.0.4 |
    |              |
    | Web Server 0 |
    +--------------+

Now, You can not use the 'casual' defaultrouter="X" cause it will be only for one network.

We will have to use setfib(1) to create two (or more) separete routing tables per network/interface.

Note: FIB (Forward Information Base, synonym for a routing table here)

Add these lines to /boot/loader.conf file:

# vim /boot/loader.conf
### select one of following firewalls to enable.
#ipfw_load="YES" # Firewall
pf_load="YES" # packet filter
pflog_load="YES" # packet filter log

### set number of routing tables to support.
net.fibs=2

Note: to view a list of options: cat /boot/defaults/loader.conf.

It will unfortunately require kernel recompile, but its not as that hard:

# cd /usr/src/sys/`uname -m`/conf
# cp GENERIC MYKERNEL8.2

# vim MYKERNEL8.2
options ROUTETABLES=2 # max 16. 1 is back compatible.

Note: to view a list of available options:
# cat /usr/src/sys/conf/NOTES | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.

# cat /usr/src/sys/`uname -m`/conf/DEFAULTS

# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL8.2
# make installkernel KERNCONF=MYKERNEL8.2

# sync ; reboot

Note: if the system could not boot properly, press any keys other than enter key when you see the count down number. and type following at boot: prompt:
boot: unload
boot: /kernel.old
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/boot-blocks.html

Remove the files generated during recompiling the kernel.
# rm -rf /usr/obj/usr/src/sys/MYKERNEL8.2

After the kernel has been built and rebooted the different routing tables can be accessed as shown in the setfib(1) man page by issuing command setfib 0 netstat -rn. 0 is the default routing table.

After this one has to create the second routing table by prepending every route add command with setfib 1 route add… e.g:

# setfib 0 /sbin/route add -net default 10.0.0.1
# setfib 1 /sbin/route add -net default 20.0.0.1

Note: with packet filter one can control how the routing table is selected by using rtable option but it should be noted that this selection can only be done on the input of the packets as the routing decision is done at the input not at the output.

# vi /etc/rc.conf
### WAN interfaces. You do not need to set defaultrouter="XXX" here.
ifconfig_wan0="inet 10.0.0.2/32"
ifconfig_wan1="inet 20.0.0.2/32"

### Note: commented out for multiple WAN routes setup. Routes will be added in /etc/rc.local file.
#defaultrouter="10.0.0.1"

### LAN static route
#static_routes="lan"
#route_lan="-net 192.168.50.0/24 192.168.0.11"

### PF (Packet Filter Firewall)
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"
pflog_flags=""

### Enable as LAN gateway
gateway_enable="YES"

All the rest configuration resides in /etc/rc.local file:

# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 10.0.0.1

### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 20.0.0.1

### assing route tables to interfaces
### Note: uncomment following lines if you are using ipfw. Leave it if you are using PF.
###
#ipfw -f flush
#ipfw add allow ip from any to any via lo0
#ipfw add setfib 0 ip from any to any via wan0
#ipfw add setfib 1 ip from any to any via wan1
#ipfw add allow ip from any to any

These would be handy for restarting:
# /etc/rc.d/netif restart && /etc/rc.d/routing restart
# /etc/rc.d/local restart
# pfctl -f /etc/pf.conf

View the fib:
# sysctl -a | grep fib
net.my_fibnum: 0
net.add_addr_allfibs: 1
net.fibs: 2

View the default route of each routing table:

# setfib 0 netstat -rn
# setfib 1 netstat -rn

Try to ping:
# setfib 0 ping google.com
# setfib 1 ping google.com

===============================================================
pf.conf

Looking for something like this?

# man 1 setfib
# man 2 setfib

And pf.conf(5) has the route-to and reply-to directives, of course.

# man 5 pf.conf | less +/rtable
# man 5 pf.conf | less +/route-to
# man 5 pf.conf | less +/reply-to

# vim /etc/pf.conf
### [VARIABLES]
wan_if0 = "wan0"
wan_if1 = "wan1"
dmz_if0 = "dmz0"
www0_ip0 = "192.168.0.3"
www0_ip1 = "192.168.0.4"

##########################
##### TABLES - A structure used to hold lists of IP addresses.
##########################
table <blocked_ip> persist file "/etc/pf-blocked_ip"

### [NAT]
nat on $wan_if0 from $dmz_if0/24 to any -> $wan_if0
nat on $wan_if1 from $dmz_if0/24 to any -> $wan_if1

### [HTTP] RDR Outside to DMZ
rdr on $wan_if0 proto tcp from any to $wan_if0 port 80 -> $www0_ip0
rdr on $wan_if1 proto tcp from any to $wan_if1 port 80 -> $www0_ip1

### [HTTP] outside to router
pass in quick on $wan_if0 proto {tcp} from any to $wan_if0 port 80 rtable 0
pass in quick on $wan_if1 proto {tcp} from any to $wan_if1 port 80 rtable 1

### [HTTP] router to dmz
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip0 port 80 rtable 0
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip1 port 80 rtable 1

Troubleshooting
dump traffic on a network, see packets that match a certain port
# tcpdump -ni re0 port 53

You may also want to limit tcpdump to just show icmp:
# tcpdump -ni re0 icmp

Or just to/from a certain host:
# tcpdump -ni re0 icmp host 172.16.70.12

Reading a PF (packet filter) log file

The log file written by pflogd is in binary format and cannot be read using a text editor. Tcpdump must be used to view the log.

To view the log file:

# tcpdump -n -e -ttt -r /var/log/pf.log

Note: that using tcpdump(8) to watch the pflog file does not give a real time display. A real time display of logged packets is achieved by using the pflog0 interface:

# ifconfig | grep pflog
# tcpdump -n -e -ttt -i pflog0

Filtering Log Output

Because pflogd logs in tcpdump binary format, the full range of tcpdump features can be used when reviewing the logs. For example, to only see packets that match a certain port:

# tcpdump -n -e -ttt -r /var/log/pf.log port 80

set up PF (packet filter) firewall on FreeBSD 8.1

Other Solution:
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-aggregation.html
http://www.freebsd.org/cgi/man.cgi?query=lagg&sektion=4
http://scratching.psybermonkey.net/2009/06/freebsd-combine-2-or-more-nic-using.html
===============================================================
Just a quick note, if doing this on a router with a single internal interface.

Traffic originating on the local network will go out the default route of FIB 0.

Only traffic coming in on the second public interface will go out the same interface.

IOW, the setup above is really only useful for incoming traffic, to make sure that it goes back out the correct interface.

However, a few more IPFW rules can be added to classify traffic on the internal NIC.

It all depends on what you want to accomplish.

Yes, its mainly for that purpose, to assign proper gateways to networks/interfaces (which is a lot more easier @ Debian in /etc/network/interfaces)

We may even use FIB# 1 and 2 for these networks, and use FIB# 0 with other NIC as a default interface, posibilities are of course big, but I wanted to point posibility of having a gateway per network.

http://www.daemonforums.org/showthread.php?t=4610
===============================================================
setfib selects the routing table for locally originated
outgoing packets. Besides locally originated packets, there
are packets arriving from the network and need to be forwarded.
These packets can be classified in a specific routing table
with the aid of ipfw. That's all there is. I can't think
of something else that needs to be thought with regard to
multiple routing tables.

HTH, Nikos

http://lists.freebsd.org/pipermail/freebsd-questions/2009-July/202462.html
===============================================================
Something has been bugging me for several years now. In that time I have usually had access to multiple WAN connections, owing to my participation in the telecom industry. However, I've never been able to get SSHD to behave the way I wanted it to. I wanted to be able to connect to the SSH daemon on my (FreBSD) router from whichever WAN connection I wanted. Unfortunately, SSHD is stuborn about always routing its response to the default gateway of the router, which breaks an SSH connection coming in from the secondary WAN connection.

I have finally, at long last, found the solution.

The Problem: When describing this problem to other FreeBSD users, they pretty universally assumed that I was mistaken, and that the SSH daemon would by default route its responses out the interface that the initial request had been received on. Evidently, it is uncommon to run SSHD on more than one WAN interface. At its root, this problem just boils down to the routing table. SSHD doesn't route it's responses to the interface that they were received on, it uses the routing table to determine where to send it's responses. If the request came from a local network, then it responds to the local network. If it originated from a non-local network, then it uses the default gateway. It's really that simple. There is no logic for having multiple paths to non-local networks.

Frequently Offered Solution: Since I use pf for my firewall, I'm frequently told to use pf's route-to and reply-to functionality to solve this problem. I have at times used route-to and reply-to extensively in my pf.conf. But route-to and reply-to do not trump the default routing table for traffic the originates or terminates on the router itself. They are useful only for traffic passing through the router. pf can only make routing decisions when a packet passes through an interface. It can try and set the reply-to interface to be the second WAN connection when an inbound SSH connection is made, but neither the SSH daemon nor the routing table on the host know or care about the routing preferences of pf.

The Real Solution: FreeBSD has support for multiple routing tables. It's little known, and even less documented, but it does exist. Basically, you need to recompile your kernel with multiple routing table support ("options ROUTETABLES=2"), and then use the setfib program to set which routing table to use when starting another program. The syntax is similar to nice: setfib 1 route add default 192.168.1.1 would add a default route of 192.168.1.1 to the second routing table on the host. If not specified, the default routing table is 0. On FreeBSD, pf also has support for multiple routing tables with the little discussed rtable option.

So here are the steps to solving this problem:

Step 1: Rebuild your kernel with the ROUTETABLES option set to a non-zero integer. This is how many routing tables your host will support.

# cat /usr/src/sys/`uname -m`/conf/MYKERNEL8.2 | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.

Step 2: Disable the SSH daemon in your rc.conf:

# cat /etc/rc.conf | grep ssh
#sshd_enable="YES" # This is now handled by /etc/rc.local

Step 3: Create /etc/rc.local file to start multiple SSH daemons. To do this, copy the /etc/ssh/sshd_config file to several alternates, one per interface you want SSHD to listen to, and set the ListenAddress for each file to only the IP for that interface.

# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable0
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable1

# vim /etc/ssh/sshd_config.rtable0
ListenAddress 20.0.0.2
ListenAddress 192.168.0.1

# vim /etc/ssh/sshd_config.rtable1
ListenAddress 30.0.0.2
ListenAddress 192.168.0.2

Note: Multiple ListenAddress options are permitted.

# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 20.0.0.1

### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 30.0.0.1

### Start SSH daemons for each interface
/usr/sbin/setfib 0 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable0
/usr/sbin/setfib 1 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable1

Step 4: Add rtable awareness to your pf.conf file:

[root@router]~ # cat /etc/pf.conf | grep rtable
pass in log on tun0 inet proto icmp from any to (tun0) icmp-type rtable 0
pass in log on tun1 inet proto icmp from any to (tun1) icmp-type rtable 1
pass in log on tun0 inet proto tcp from any to (tun0) port ssh rtable 0
pass in log on tun1 inet proto tcp from any to (tun1) port ssh rtable 1
pass in log on wan0 inet proto tcp from wan0:network to (wan0) port 22 rtable 0

Conclusion: Because the SSH daemon listening on tun1 is using a routing table that features the tun1 interface as the default gateway, the response will go out tun1. An inbound connection to tun0 will hit the SSH daemon listening on tun0 (which is an entirely separate process from the one listening on tun1) and uses the routing table associated with tun0, which features tun0 as the default gateway.

In my above config it's worth pointing out that it doesn't actually matter which routing table the SSH daemon listening on the LAN interface uses, because both routing tables see the LAN network as a local one. By default on FreeBSD with multiple routing tables enabled, all local networks will still appear in all the routing tables. There is a sysctl option to disable this behavior.

High Availability HA cluster solution for FreeBSD

HAST + CARP + ZFS

HAST (Highly Available Storage) - allows to transparently store data on two physically separated machines connected over the TCP/IP network.

CARP (Common Address Redundancy Protocol) - allows multiple hosts to share the same IP address. In some configurations, this may be used for availability or load balancing. Hosts may use separate IP addresses as well, as in the example provided here.

# man carp
# man ng_one2many
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.

http://www.freebsd.org/doc/handbook/carp.html
http://forums.freebsd.org/showthread.php?t=17133
http://blather.michaelwlucas.com/archives/241
===============================================================

Reference:
http://gala4th.blogspot.com/2011/10/multiple-default-routes-gateways-with.html
High Availability HA cluster solution for freebsd
http://www.daemonforums.org/showthread.php?t=4610
http://freebsd-forums.liquidneon.com/showthread.php?t=20166
http://www.mmacleod.ca/blog/2011/06/source-based-routing-with-freebsd-using-multiple-routing-table/
http://www.melen.org/u/jan/wp/?p=102
http://gala4th.blogspot.com/2010/12/set-up-pf-packet-filter-on-freebsd.html
Running PF & IPFW Together
Prevent default gateway route and resolv.conf being overwritten by DHCP

kill sendmail processes and clean up mail queue mailq

2012-01-24T09:21:00.000-08:00

kill sendmail processes and clean up mail queue mailq

# ps auxww | grep 'sendmail: ./' | awk '{print $2}' | xargs kill
# rm /var/spool/mqueue/*

Check if a file is opened in use or locked by another process

2012-01-23T15:30:00.000-08:00

The PHP flock() function is good when the file locking method is used the same way by all programs that will lock the file.

However, you can't check if any process in the system is actually using the file or not. If you for example are monitoring an incoming ftp folder, you don't want to start processing a file until the file is completely tranferred (i.e. wait until the ftp daemon is no longer having the file open).

The following code illustrates how the command lsof (Linux) or fstat (FreeBSD) can be used for the purpose:

lsof on Linux:

<?php
$filename = "locktest.txt";
$directory = ".";
while (1) {
        $result = exec("lsof +D $directory | grep -c -i $filename");
        if ($result == "0") {
                echo "$directory/$filename is NOT open ($result)\n";
        } else {
                echo "$directory/$filename IS OPEN ($result)\n";
        }
        sleep(1);
};
?>

fstat on FreeBSD:

<?php
$out = exec('fstat test.pdf | wc -l');
$out = trim($out);

if ($out == 1) {
  echo 'not in use' . PHP_EOL;
}
else {
  echo 'in use' . PHP_EOL;
}
?>

test.sh:

#!/bin/sh

dir="/home/backup/"

find "${dir}" -type f | while read file
do
  isFileInUse=`fstat "${file}" | grep -c "${file}"`

  #echo "${isFileInUse}"

  if [ "${isFileInUse}"  = "0" ]; then
    #echo ${isFileInUse}
    echo "File is Not in use. ${file}"
  else
    echo "File is in use. ${file}"
  fi
done

Reference:
http://nerdia.net/2011/01/17/check-if-a-file-is-open-by-another-process-in-php-linuxunix/

Install Samba on FreeBSD 8.2

2012-01-23T13:03:00.000-08:00

Install Samba on FreeBSD 8.2

root@bmw0 [/root] # cd /usr/ports/net/samba36; make install clean

Edit smb.conf

root@bmw0 [/root] # vi /usr/local/etc/smb.conf
security = user
hosts allow = 192.168.100. 192.168.200.168 127.
load printers = no

[syncdb]
comment = hmm
path = /usr/local/www/apache22/data/syncdb
valid users = smb_user0
public = no
writable = yes
printable = no
create mask = 0600
directory mask = 0700
; write list = @staff

Add a user called "smb_user0"
root@bmw0 [/root] # pw useradd -n smb_user0 -s /sbin/nologin -w no -d /home/smb_user0 -c "Samaba User0" -m

Adding users to samba
root@bmw0 [/root] # /usr/local/bin/pdbedit -a smb_user0
or
root@bmw0 [/root] # smbpasswd -a smb_user0

List user
root@bmw0 [/root] # /usr/local/bin/pdbedit -L

Edit rc.conf
root@bmw0 [/root] # vi /etc/rc.conf
### Samba
samba_enable="YES"
#nmbd_enable="YES"
#smbd_enable="YES"
winbindd_enable="YES"

Start Samba
root@bmw0 [/root] # /usr/local/etc/rc.d/samba start
root 56804 0.2 0.7 39668 14484 ?? Ss 5:15PM 0:00.04 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
root 56807 0.2 0.7 39752 14564 ?? S 5:15PM 0:00.00 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
root 56800 0.0 0.4 29696 7464 ?? Ss 5:15PM 0:00.01 /usr/local/sbin/nmbd -D -s /usr/local/etc/smb.conf

Check to see if samba is running
root@bmw0 [/root] # ps -auwxx | egrep '[sn]mbd|winbindd'

Test samba using smbclient
root@bmw0 [/root] # /usr/local/bin/smbclient -U smb_user0 \\\\localhost\\smb_user0
or
root@bmw0 [/root] # /usr/local/bin/smbclient -U smb_user0 //localhost/smb_user0

Edit ipfw.rules
root@bmw0 [/root] # vi /usr/local/etc/ipfw.rules
### samba
$IPF 260 allow tcp from 192.168.100.0/24 to any 139 in

Reference:
http://gala4th.blogspot.com/2012/01/install-samba-on-freebsd-82.html

set up PF (packet filter) firewall on FreeBSD 8.2

2012-01-20T12:07:00.000-08:00

set up PF (packet filter) firewall on FreeBSD 8.2

Network Structure
Outside World <-----> PF on FreeBSD 8.1 as a router <-----> DMZ network (more FreeBSD boxes)

Quick Notes

[] "pfctl -e" or "pfctl -d" can be used to enable and disable PF respectively. Note that this just enables or disables PF, it doesn't actually load a ruleset. The ruleset must be loaded separately, either before or after PF is enabled. To load the pf.conf ruleset file, run "pfctl -f /etc/pf.conf".

[] Filter rules are evaluated in sequential order, first to last. Unless the packet matches a rule containing the quick keyword, the packet will be evaluated against all filter rules before the final action is taken.

[] The last rule to match is the "winner" and will dictate what action to take on the packet.

[] There is an implicit pass all at the beginning of a filtering ruleset meaning that if a packet does not match any filter rule the resulting action will be pass.

[] The keyword any meaning all addresses.

[] The keyword all which is short for from any to any.

Edit /boot/loader.conf:
# vim /boot/loader.conf
pf_load="YES" # packet filter
pflog_load="YES" # packet filter log

Edit /etc/rc.conf
### LAN
ifconfig_xl0="inet 192.168.3.1 netmask 255.255.255.0"

### WAN
ifconfig_re0="inet 219.17.112.243 netmask 255.255.255.192"

### WAN - extra IP addresses
### Note: the netmask of extra IP addresses must be 255.255.255.255
ifconfig_re0_alias0="inet 219.17.112.244 netmask 255.255.255.255"
ifconfig_re0_alias1="inet 219.17.112.245 netmask 255.255.255.255"

### Default Route
defaultrouter="219.17.112.193"

### LAN static route
static_routes="lan"
route_lan="-net 192.168.5.0/24 192.168.100.9"

### enable PF (Packet Filter Firewall)
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"
pflog_flags=""

### enable as LAN gateway.
### You must enable this if you want to do NAT (network address translation).
gateway_enable="YES"

### FTP-Proxy.
### Note: add these two lines to allow people to connect to your FTP server (192.168.100.8) behind NAT.
ftpproxy_enable="YES"
ftpproxy_flags="-R 192.168.100.8"

Note: Since NAT is almost always used on routers and network gateways, it will probably be necessary to enable IP forwarding so that packets can travel between network interfaces on the OpenBSD machine. IP forwarding is enabled using the sysctl(3) mechanism

On FreeBSD, setting gateway_enable="YES" in /etc/rc.conf will automatically turn on net.inet.ip.forwarding=1. You can check this by running:

# sysctl net.inet.ip.forwarding
net.inet.ip.forwarding: 1

If you are using OpenBSD, you can run follow command:

# sysctl net.inet.ip.forwarding=1
# sysctl net.inet6.ip6.forwarding=1 (if using IPv6)

To make this change permanent, the following lines should be added to /etc/sysctl.conf (on OpenBSD. No need on FreeBSD):

net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

These lines are present but commented out (prefixed with a #) in the default install. Remove the # and save the file. IP forwarding will be enabled when the machine is rebooted.

Create a file for later use:
# touch /etc/pf-blocked_ip

Edit /etc/pf.conf
root@fw1 [/root] # cat /etc/pf.conf

##########################
##### MACROS - User-defined variables that can hold IP addresses, interface names, etc.
##########################
wan_if="re0"
dmz_if="xl0"
#lan_if=""

icmp_types = "{echoreq echorep unreach}"
#icmp_types = "{echoreq}"

tcp_services = "{ssh smtp domain http https ntp nicname}"

udp_services = "{domain ntp}"

allow_ssh_ip = "{219.17.152.199 224.1.1.1}"

dmz_www_ip = "192.168.100.5"

dmz_sql_ip = "192.168.100.6"

dmz_svn_ip = "192.168.100.7"

dmz_ftp0_ip0 = "192.168.100.8"

cn_srv_ip = "222.144.87.188"

ca_pub_ip = "219.17.152.196"

allow_sql_ip = "{" $cn_srv_ip $ca_pub_ip "}"

##########################
##### TABLES - A structure used to hold lists of IP addresses.
##########################
table <blocked_ip> persist file "/etc/pf-blocked_ip"

##########################
##### OPTIONS - Various options to control how PF works.
##########################

### Sets the default behavior for filter rules that specify the block action.
### Return error codes for ports that are blocked. Allows faster error recovery.
#set block-policy return

### Set pf's debugging level.
#set debug misc

### Skip all PF processing on specified interface. This can be useful on loopback interfaces where filtering, normalization, queueing, etc, are not required.
set skip on lo0

##########################
##### NORMALIZATION
##########################
### Traffic normalization for incoming packets - scrub provides a measure of protection against certain kinds of attacks based on incorrect handling of packet fragments.
### One reason not to scrub on an interface is if one is passing NFS through PF. Some non-OpenBSD platforms send (and expect) strange packets -- fragmented packets with the "do not fragment" bit set, which are (properly) rejected by scrub. This can be resolved by use of the no-df option. Another reason is some multi-player games have connection problems passing through PF with scrub enabled. Other than these somewhat unusual cases, scrubbing all packets is a highly recommended practice.
### The scrub directive syntax is very similar to the filtering syntax which makes it easy to selectively scrub certain packets and not others. The no keyword can be used in front of scrub to specify packets that will not be scrubbed. Just as with nat rules, the first matching rule wins.
scrub in all

##########################
##### QUEUEING - Provides bandwidth control and packet prioritization.
##########################

##########################
##### TRANSLATION
##########################
### NAT
nat on $wan_if from $dmz_if/24 to any -> $wan_if

### [HTTP] RDR Outside to DMZ
rdr on $wan_if proto tcp from any to $wan_if port 80 -> $dmz_www_ip

### [MYSQL] RDR Outside to DMZ
rdr on $wan_if proto tcp from $allow_sql_ip to $wan_if port 3306 -> $dmz_sql_ip

### [RSYNCD] RDR Outside to DMZ
rdr on $wan_if proto tcp from $cn_srv_ip to $wan_if port 873 -> $dmz_www_ip

### [MAIL] RDR Outside to DMZ
#rdr on $wan_if proto tcp from any to $wan_if/32 port {25 110} -> 192.168.10.2

### [FTP] RDR Outside to DMZ
#rdr on $wan_if proto tcp from $allow_ftp_ip to $wan_if port 21 -> 192.168.10.8 port 21

### [FTP] RDR DMZ to Outside (through ftp-proxy)
### Note: make sure you have this line "ftpproxy_enable="YES"" in your /etc/rc.conf.
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr pass proto tcp from any to $wan_if port 21 -> 127.0.0.1 port 8021

### [RAID-3WARE]
#rdr on $wan_if proto tcp from $wan_if/26 to $wan_if port 888 -> $dmz_sql_ip

##########################
##### FILTERING RULES - Allows the selective filtering or blocking of packets as they pass through any of the interfaces.
##### Filter rules can be given parameters to specify network address translation (NAT) and packet redirection.
##########################
### setup a default deny policy - block everything (inbound and outbound on all interfaces).
### Note: if you want to see the log file to debug your filter rules, change following line to "block log all", otherwise use "block all" instead..
block log all

block in quick on $wan_if from <blocked_ip> to any

### activate spoofing protection for all interfaces.
block in quick from urpf-failed

### [DNS] Outside to Router
pass in quick on $wan_if proto {tcp udp} from any to $wan_if port 53

### [DNS] DMZ to Router
pass in quick on $dmz_if proto {tcp udp} from $dmz_if/24 to any port 53

### [DNS] Router to Outside
pass out quick on $wan_if proto {tcp udp} from $wan_if to any port 53

### [DNS] Router to DMZ
pass out quick on $dmz_if proto {tcp udp} from $dmz_if to $dmz_if/24 port 53

### [HTTP] Outside to DMZ
pass in quick on $wan_if proto tcp from any to $dmz_www_ip port 80
pass out quick on $dmz_if proto tcp from any to $dmz_www_ip port 80

### [HTTP] DMZ to Router
### Note: FreeBSD's portsnap command uses port 80 to grab latest FreeBSD ports/patches.
pass in quick on $dmz_if proto tcp from $dmz_if/24 to any port 80

### [HTTP] Router to Outside
pass out quick on $wan_if proto tcp from $wan_if to any port 80

### [MYSQL] DMZ to Outside
pass in quick on $dmz_if proto tcp from $dmz_sql_ip to $cn_srv_ip port 3306
pass out quick on $wan_if proto tcp from $wan_if to $cn_srv_ip port 3306

### [MYSQL] Outside to DMZ
pass in quick on $wan_if proto tcp from $allow_sql_ip to $dmz_sql_ip port 3306
pass out quick on $dmz_if proto tcp from $allow_sql_ip to $dmz_sql_ip port 3306

### [SMTP] DMZ to Outside
pass in quick on $dmz_if proto tcp from $dmz_www_ip to any port 25
pass out quick on $wan_if proto tcp from $wan_if to any port 25

### [RSYNCD] Outside to DMZ
pass in quick on $wan_if proto tcp from $cn_srv_ip to $dmz_www_ip port 873
pass out quick on $dmz_if proto tcp from $cn_srv_ip to $dmz_www_ip port 873

### [SSH] Outside to Router
pass in quick on $wan_if proto tcp from $allow_ssh_ip to $wan_if port 22 flags S/SA keep state

### [SSH] Router to DMZ
pass out quick on $dmz_if proto tcp from $dmz_if to $dmz_if/24 port 22 flags S/SA keep state

### [ICMP] DMZ/LAN to ANY
pass in quick on $dmz_if proto icmp from {$dmz_if/24 $lan_net} to any icmp-type $icmp_types

### [ICMP] Router to Outside
pass out quick on $wan_if proto icmp from $wan_if to any icmp-type $icmp_types

### [ICMP] DMZ to DMZ/LAN
pass out quick on $dmz_if proto icmp from $dmz_if/24 to {$dmz_if/24 $lan_net} icmp-type $icmp_types

### [NTP] DMZ to Router
pass in quick on $dmz_if proto {tcp udp} from $dmz_if/24 to any port 123

### [NTP] Router to Outside
pass out quick on $wan_if proto {tcp udp} from $wan_if to any port 123

### [FTP] We need to have an anchor for ftp-proxy.
anchor "ftp-proxy/*"

### [FTP] from DMZ to ANY
pass in quick log on $dmz_if proto {tcp} from $dmz_if/24 to any port 21
pass in quick log on $dmz_if proto {tcp} from $dmz_if/24 to any port > 1023

### [FTP] from ROUTER to ANY
pass out quick log on $wan_if proto {tcp} from $wan_if to any port 21
pass out quick log on $wan_if proto {tcp} from $wan_if to any port > 1023

### [FTP] from ANY to $dmz_ftp0_ip
pass out quick on $dmz_if proto {tcp} from $dmz_if to $dmz_ftp0_ip0 port 21

### [NICNAME] DMZ to Router
pass in quick on $dmz_if proto tcp from $dmz_if/24 to any port 43

### [NICNAME] Router to Outside
pass out quick on $wan_if proto tcp from $wan_if to any port 43

### [SVN] Router to DMZ
pass out quick on $dmz_if proto tcp from $dmz_if to $dmz_svn_ip port 3690

### [RAID-3WARE]
#pass in quick on $wan_if proto tcp from $wan_if/26 to $dmz_sql_ip port 888
#pass out quick on $dmz_if proto tcp from $wan_if/26 to $dmz_sql_ip port 888

### [TRACEROUTE] Allow outgoing Trace route
#pass out on $ext_if inet proto udp from any to any port 33433 >< 33626 keep state

Solution to No ALTQ support in kernel ALTQ related functions disabled

Recompile your kernel:
# cd /usr/src/sys/`uname -m`/conf
# cp GENERIC MYKERNEL8.2

# vim MYKERNEL8.2

### PF Packet Filter Related settings
device pf
device pflog
device pfsync

options         ALTQ
options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
options         ALTQ_RED        # Random Early Detection (RED)
options         ALTQ_RIO        # RED In/Out
options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
options         ALTQ_NOPCC      # Required for SMP build

Note: to view a list of available options:
# cat /usr/src/sys/conf/NOTES
# cat /usr/src/sys/`uname -m`/conf/DEFAULTS

# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL8.2
# make installkernel KERNCONF=MYKERNEL8.2

# sync ; reboot

Note: if the system could not boot properly, press any keys other than enter key when you see the count down number. and type following at boot: prompt:
boot: unload
boot: /kernel.old
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/boot-blocks.html

Useful Commands

Make sure the pf kernel module has been loaded:
# kldstat | grep pf
2 2 0xc0f8c000 32e98 pf.ko
3 1 0xc61db000 3000 pflog.ko

Diable the packet filter
# pfctl -d

Enable the packet filter
# pfctl -e

Flush all (nat, filter, queue, state, info, table) rules and reload from the file /etc/pf.conf
# pfctl -F all -f /etc/pf.conf

If you are setting up pf remotely, and you are not confident enough with your PF rules, you might be blocked out by the wrong rules. Following command allows you to try the rules, and disable PF after 20 seconds:
# pfctl -f /etc/pf.conf; sleep 20; pfctl -d

This does not actually load any rules, but allows you to check for syntax errors in the file before you do load the ruleset. This is obviously good for testing.
# pfctl -vnf /etc/pf.conf

Report on the currently loaded filter ruleset.
# pfctl -s rules

Report on the currently loaded nat ruleset.
# pfctl -s nat

Report on the currently running state table (very useful).
# pfctl -s state

Tables can be manipulated on the fly by using pfctl(8). For instance, to add entries to the <blocked_ip> table:
# pfctl -t blocked_ip -T add 218.70.0.0/16

This will also create the <blocked_ip> table if it doesn't already exist.

To list the addresses in a table:
# pfctl -t blocked_ip -T show

Note: The -v argument can also be used with -T show to display statistics for each table entry.

To remove addresses from a table:
# pfctl -t blocked_ip -T delete 218.70.0.0/16

Troubleshooting
dump traffic on a network, see packets that match a certain port
# tcpdump -ni re0 port 53

You may also want to limit tcpdump to just show icmp:
# tcpdump -ni re0 icmp

Or just to/from a certain host:
# tcpdump -ni re0 icmp and host 172.16.70.12

Reading a PF (packet filter) log file

The log file written by pflogd is in binary format and cannot be read using a text editor. Tcpdump must be used to view the log.

To view the log file:

# tcpdump -n -e -ttt -r /var/log/pf.log

Note: that using tcpdump(8) to watch the pflog file does not give a real time display. A real time display of logged packets is achieved by using the pflog0 interface:

# ifconfig | grep pflog
# tcpdump -n -e -ttt -i pflog0

Filtering Log Output

Because pflogd logs in tcpdump binary format, the full range of tcpdump features can be used when reviewing the logs. For example, to only see packets that match a certain port:

# tcpdump -n -e -ttt -r /var/log/pf.log port 80

Reference:
set up PF (packet filter) firewall on FreeBSD 8.2
http://gala4th.blogspot.com/2010/12/set-up-pf-packet-filter-on-freebsd.html

PF supports FTP servers and FTP clients behind NAT

http://www.openbsd.org/faq/pf/index.html

http://www.openbsd.org/faq/pf/nat.html

http://www.weithenn.org/cgi-bin/wiki.pl?PF-%E5%88%A9%E7%94%A8_PF_%E8%BC%95%E9%AC%86%E9%81%94%E6%88%90_NAT

http://www.thedeepsky.com/howto/newbie_pf_guide.php

http://home.nuug.no/~peter/pf/en/ftpproblem.html

Active FTP vs. Passive FTP, a Definitive Explanation

Active FTP vs. Passive FTP, a Definitive Explanation

Setting up a Secure Subversion Server

2012-01-20T12:04:00.000-08:00

Setting up a Secure Subversion Server

Backing up your scripts.

# tar czvf - /usr/local/etc/www/data | ssh dru@192.168.2.2 "cat > www.tar.gz"

Preparing the System

In my scenario, it was important that only the members of the development team have access to the repository. We also chose to have the repository on a system separate from the actual web server and left it up to the web administrator to copy over files from the repository to the web server as he saw fit.

To accomplish this, start by creating a backup of the existing directory structure you wish to put under revision control, and send it securely to the repository server. In my case, I backed up the www data on the web server to an internal server at 192.168.2.2.

Add a user called svn.

# adduser

Next, on the repository system, create a new group called svn and add to it any existing user accounts that need access to the repository. For example, I added my existing web administrator as I created the group by adding this line to /etc/group:

# vi /etc/group
svn:*:3690:webadmin

Then, create a new user called svn and, if necessary, any missing user accounts that need access to the repository. Make sure each account is a member of the svn group and has a password and a valid shell. I used sysinstall to create user accounts for the new web developers. When I finished, I double-checked the membership of the svn group. It looked something like this:

# grep svn /etc/group
svn:*:3690:webadmin,devel1,devel2

Dealing with umask

Before installing Subversion, take a close look at the existing umask for the svn user. On my FreeBSD system it was:

# su - svn
% umask
022

In Unix, the umask value determines the default permissions of a newly created directory or file. It does this by defining which permissions to disable. If you remember:

r = 4
w = 2
x = 1

you'll see that this umask doesn't turn off any (0) permissions for the user (svn); it turns off write (2) for the group (svn); and it turns off write (2) for world.

Because the members of the svn group should be able to write to the repository, change that group 2 to a 0. If you don't want nongroup members even to be aware of the existence of the repository, also change the world 2 to a 7.

The easy part is changing the umask for the svn user's shell. If it uses csh:

# su - svn
svn # vi ~svn/.cshrc
# A righteous umask
umask 027

Note: the meaning of each umask:
umask 002 // File permission 644. Owner can read/write. Group and Others can only read.
umask 007 // File permission 660. Owner and Group can read/write. Others can not read or write.
umask 027 // File permission 640. Owner can read/write. Group can read. Others can not read or write.

Note: I personally prefer to set umask 027. There is a security reason behind the thought. In order to prevent bad scripts trying to create new scripts or modify existing scripts on your server, you can have "svn update" running automatically in crontab. That will take care of source code update part. Then, you would want to make "svn" user to be the only user that has the write permission. www group users will only have read permission.

then find the existing umask line and change it to either 002, 007 or 027.

If your svn user has a shell other than csh, make your edit in your chosen shell's configuration file.

Once you've saved your changes to ~svn/.cshrc (or wherever), don't forget to tell the shell:

svn # source ~svn/.cshrc

Repeat the umask command to verify that your changes have taken place.
Installing Subversion with the correct umask

If you chose a umask of 002, you can compile a wrapper into Subversion when you build it from the ports collection. If you chose a umask of 007 or 027, or prefer to install the precompiled version of Subversion, create a wrapper script to ensure that the Subversion binaries use your umask value.

To compile in a wrapper that sets a umask of 007 or 027:

# cd /usr/ports/devel/subversion
# make config-recursive
# make install clean

To compile in a wrapper that sets a umask of 002:
# cd /usr/ports/devel/subversion
# make -DWITH_SVNSERVE_WRAPPER install clean

Note: you do NOT need -DWITH_SVNSERVE_WRAPPER option if you decided to use umask of 007 or 027.

Make sure you DO uncheck this option:
[] BDB=off "db4 repository backend"

Because some poeple said:

- I never used a BDB repos myself, exactly because the Subversion manual warns strongly against it. So second this. – jfs Sep 23 at 8:30

- I also second this - we used to have Subversion BDB repository problems. Switching to the FSFS repository type helped. – Phill Sacre Sep 23 at 8:38

Alternatively, to install the precompiled binary:

# pkg_add -r subversion

Note: before installing by either method, finish reading the article. You may find some additional compile options that interest you.

If you didn't compile in your wrapper (that means you use a umask of 007 or 027), move your existing binary and create your own wrapper script:

# mv /usr/local/bin/svn /usr/local/bin/svn.orig

# vi /usr/local/bin/svn

#!/bin/sh

### initialize
svnarg=""

### use encoding utf-8 as default if run "svn ci" or "svn commit".
if [ "$1" != "help" ]; then
  for myarg in "$@"; do
    if [ "${myarg}" = "commit" ] || [ "${myarg}" = "ci" ]; then
      svnarg="--encoding utf-8"
      break
    fi
  done
fi

### wrapper script to set umask to 027 on subversion binaries
### Note: the meaning of each umask:
### umask 002 // File permission 644. Owner can read/write. Group and Others can only read.
### umask 007 // File permission 660. Owner and Group can read/write. Others can not read or write.
### umask 027 // File permission 640. Owner can read/write. Group can read. Others can not read or write.
umask 027

### svn command
/usr/local/bin/svn.orig ${svnarg} "$@"

Set your umask to either 002, 007 or 027 so that it is the same as the umask for your svn user.

Don't forget to make your wrapper script executable:

# chmod +x /usr/local/bin/svn

Repeat the same steps for proj2 files.

Creating the Repository
Create a central place to store all repositories.
# mkdir /usr/local/repositories

# chown svn:svn /usr/local/repositories

Now that your environment is set up properly, you're ready to create the repository itself.

Log in as the user svn to ensure that both the svn user and the svn group own the files you create in the repository.
# su - svn

svn # cd /usr/local/repositories
svn # svnadmin create proj1
svn # svnadmin create proj2

In this example, I've called my repository "proj1" and "proj2" two separate repositories. You can choose any name that is useful to you.

svnadmin create simply creates the directory infrastructure required by the Subversion tools:

svn # ls -F proj1 proj2
proj1:
README.txt conf/ db/ format hooks/ locks/

proj2:
README.txt conf/ db/ format hooks/ locks/ db/ hooks/

Notice that db directory? By default, Subversion uses databases to track changes to the files that you place under revision control. This means that you must import your data into those databases.

Edit svnserve.conf & passwd file:
svn # vi /usr/local/repositories/proj1/conf/svnserve.conf
[general]
anon-access = none
auth-access = write
password-db = passwd

svn # vi /usr/local/repositories/proj1/conf/passwd
[users]
danny = mypassword

Start SVN server as a stand-alone daemon
# /usr/local/bin/svnserve -d --listen-port=3690 --listen-host=0.0.0.0 -r /usr/local/repositories

Preparing Files to be imported
At that point, I untarred my backup so that I had some data to import. If you do this, don't restore directly into the /usr/local/repositories/proj1 directory. (It's a database, remember?) Instead, I first made a new directory structure:
# mkdir /usr/local/www/apache22/data/proj1
# cd /usr/local/www/apache22/data/proj1
# mkdir branches tags trunk
# cd trunk
# tar xzvf /full/path/to/www.tar.gz .

Importing the Data
Next, it's time to import the information from /usr/local/www/apache22/data/proj1 into the Subversion databases. To do so, use the svn import command:
# su - svn
svn # cd /usr/local/www/apache22/data
svn # svn import proj1 file:///usr/local/repositories/proj1 -m "initial import"
svn # svn import proj2 file:///usr/local/repositories/proj2 -m "initial import"

svn import is one of many svn commands available to users. Type svn help to see the names of all the available commands. If you insert one of those commands between svn and help, as in svn import help, you'll receive help on the syntax for that specified command.

After svn import, specify the name of the directory containing the data to import (proj1 or proj2). Your data doesn't have to be in the same directory; simply specify the full path to the data, but ensure that your svn user has permission to access the data you wish to import. Note: once you've successfully imported your data, you don't have to keep an original copy on disk. In my case, I issued the command rm -Rf www.

Next, notice the syntax I used when specifying the full path to the repository. Subversion supports multiple URL schemas or "repository access" RA modules. Verify which schemas your svn supports with:

# svn --version
svn, version 1.1.3 (r12730)
compiled Mar 20 2005, 11:04:16

Copyright (C) 2000-2004 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).

The following repository access (RA) modules are available:

* ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol.
- handles 'http' schema
- handles 'https' schema
* ra_local : Module for accessing a repository on local disk.
- handles 'file' schema
* ra_svn : Module for accessing a repository using the svn network protocol.
- handles svn schema

Because I wished to access the repository on the local disk, I used the file:/// schema. I also appended www at the very end of the URL, as I wish that particular part of the repository to be available by that name. Yes, you can import multiple directory structures into the same Subversion repository, so give each one a name that is easy for you and your users to remember.

Finally, I used the -m message switch to append the comment "initial import" to the repository log. If I hadn't included this switch, svn would have opened the log for me in the user's default editor (vi) and asked me to add a comment before continuing.

This is a very important point. The whole reason to install a revision control system is to allow multiple users to modify files, possibly even simultaneously. It's up to each user to log clearly which changes they made to which files. It's your job to make your users aware of the importance of adding useful comments whenever an svn command prompts them to do so.

Edit /etc/rc.conf:
Just went through this (thank you) however I came across the issue where my FreeBSD box was just listening on tcp6. I'm using this internally on my network but without a tcp6 router this of course doesn't help. To make it work, I just modified my rc.conf to listen on host 0.0.0.0 (telling it to use tcp4). Also for anyone who wants it to start easily on boot, add this to your /etc/rc.conf (replacing the data dir, user, and group as necessary)

svnserve_enable="YES"
svnserve_flags="-d --listen-port=3690 --listen-host=0.0.0.0"
svnserve_data="/usr/local/repositories"
svnserve_user="svn"
svnserve_group="svn"

Edit Firewall Rules:
# vi /usr/local/etc/ipfw.rules
### SVN
$IPF 250 allow tcp from 192.168.500.0/24 to any 3690 in

Refresh Firewall Rules:
# sh /usr/local/etc/ipfw.rules

Backup Repositories
There are at least four ways to backup repositories:
- SVN hotcopy
- SVN dump
- tar entire directory.
- svnsync (also use svnsync + hook script is great. Sync at each commit)

SVN Hotcopy backup
# svnadmin hotcopy /usr/local/repositories/proj1 /home/bot/repositories/proj1
Note: the target (destination) directory must be a empty directory.
More: http://gala4th.blogspot.com/2009/08/svn.html

SVN Restore from hotcopy
Hotcopy should produce a usable file-level repository. You should be
able to use it as-is if the ownership and permissions are suitable. If
you are running a server, you may have to copy back to the location the
server expects or adjust the configuration to use the new location.

You must read carefully the following sections:

http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.reposadmin.maint.migrate
http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.reposadmin.maint.backup
http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.ref.svnadmin.c.dump
http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.ref.svnadmin.c.load
http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.ref.svnadmin.c.hotcopy

And you will be golden and all set.

As you read those links, then you must realize by now that restoring a
hotcopy which is basically a copy of all your repository its easy just
copy to your Subversion scope, you may need to change owner
permissions and change your hook-scripts (only if you were using
hook-scripts) after this you will be ready to go.

SVN Client - TortoiseSVN
On Windows, create folders:
C:\svn_repositories\proj1
C:\svn_repositories\proj2

Right click on each folder (proj1 & proj2) > check out > enter respectively:
svn://192.168.100.156/proj1
svn://192.168.100.156/proj2

or try command line:
# svn checkout svn://192.168.100.156/proj1 /usr/local/www/apache22/data/proj1
# svn update /usr/local/www/apache22/data/proj1
# svn update
# svn status /usr/local/www/apache22/data/proj1
# svn status -u
# svn add /usr/local/www/apache22/data/proj1/test111.php
# svn commit -m "LogMessage"

Deciding Upon a URL Schema

Congratulations! You now have a working repository. Now's the best time to take a closer look at the various URL schemas and choose the access method that best suits your needs.

Chapter 6 of the freely available e-book Version Control with Subversion gives details about the possible configurations. You can choose to install the book when you compile the FreeBSD port by adding -DWITH_BOOK to your make command.

If all of your users log in to the system either locally or through ssh, use the file:/// schema. Because users are "local" to the repository, this scenario doesn't open a TCP/IP port to listen for Subversion connections. However, it does require an active shell account for each user and assumes that your users are comfortable logging in to a Unix server. As with any shell account, your security depends upon your users choosing good passwords and you setting up repository permissions and group memberships correctly. Having users ssh to the system does ensure that they have encrypted sessions.

Another possibility is to integrate Subversion into an existing Apache server. By default, the FreeBSD port of Subversion compiles in SSL support, meaning your users can have the ability to access your repository securely from their browsers using the https:// schema. However, if you're running Apache 2.x instead of Apache 1.x, remember to pass the -DWITH_MOD_DAV_SVN option to make when you compile your FreeBSD port.

If you're considering giving browser access to your users, read carefully through the Apache httpd configuration section of the Subversion book first. You'll have to go through a fair bit of configuration; fortunately, the documentation is complete.

A third approach is to use svnserve to listen for network connections. The book suggests running this process either through inetd or as a stand-alone daemon. Both of these approaches allow either anonymous access or access once the system has authorized a user using CRAM-MD5. Clients connect to svnserve using the svn:// schema.

Anonymous access wasn't appropriate in my scenario, so I followed the configuration options for CRAM-MD5. However, I quickly discovered that CRAM-MD5 wasn't on my FreeBSD system. When a Google search failed to find a technique for integrating CRAM-MD5 with my Subversion binary, I decided to try the last option.

This was to invoke svnserve in tunnel mode, which allows user authentication through the normal SSH mechanism as well as any restrictions you have placed in your /etc/ssh/sshd_config file. For example, I could use the AllowUsers keyword to control which users can authenticate to the system. Note that this schema uses svn+ssh://.

The appeal of this method is that I could use an existing authentication scheme without forcing the user to actually be "on" the repository system. However, this network connection is unencrypted; the use of SSH is only to authenticate. If your data is sensitive, either have your users use file:// after sshing in or use https:// after you've properly configured Apache.

If you decide to use the svnserve server and you compiled in the wrapper, it created a binary called svnserve.bin. Users won't be able to access the repository until:

# cp /usr/local/bin/svnserve.bin /usr/local/bin/svnserve

That's it for this installment. In the next column, I'll show how to start accessing the repository as a client.

Dru Lavigne is a network and systems administrator, IT instructor, author and international speaker. She has over a decade of experience administering and teaching Netware, Microsoft, Cisco, Checkpoint, SCO, Solaris, Linux, and BSD systems. A prolific author, she pens the popular FreeBSD Basics column for O'Reilly and is author of BSD Hacks and The Best of FreeBSD Basics.

Reference:
http://gala4th.blogspot.com/2010/01/setting-up-secure-subversion-server.html
http://onlamp.com/pub/a/bsd/2005/05/12/FreeBSD_Basics.html
http://blog.jostudio.net/2007/06/svn.html

Install Jails on FreeBSD 8.2

2012-01-20T11:54:00.000-08:00

Install Jails on FreeBSD 8.2

Jails, sometimes referred to as an enhanced replacement of chroot environments, are a very powerful tool for system administrators, but their basic usage can also be useful for advanced users.

Setting up the Host Environment

First, you will want to set up your real system's environment to be "jail-friendly".

For consistency, we will refer to the parent box as the "host environment", and to the jailed virtual machine as the "jail environment". Since jail is implemented using IP aliases, one of the first things to do is to disable IP services on the host system that listen on all local IP addresses for a service. If a network service is present in the host environment that binds all available IP addresses rather than specific IP addresses, it may service requests sent to jail IP addresses if the jail did not bind the port. This means changing inetd(8) to only listen on the appropriate IP address, and so forth. Add the following to /etc/rc.conf in the host environment:

sendmail_enable="NO"
inetd_flags="-wW -a 192.168.0.1"
rpcbind_enable="NO"

man inetd for more information.

192.168.0.1 is the native IP address for the host system, in this example. Daemons that run out of inetd(8) can be easily set to use only the specified host IP address. Other daemons will need to be manually configured for some this is possible through the rc.conf(5) flags entries; for others it is necessary to modify per-application configuration files, or to recompile the applications. The following frequently deployed services must have their individual configuration files modified to limit the application to listening to a specific IP address:

To configure sshd(8), it is necessary to modify /etc/ssh/sshd_config.

To configure sendmail(8), it is necessary to modify

/etc/mail/sendmail.cf.

For named(8), it is necessary to modify /etc/namedb/named.conf.

In addition, a number of services must be recompiled in order to run them in the host environment. This includes most applications providing services using rpc(3), such as rpcbind(8), nfsd(8), and mountd(8). In general, applications for which it is not possible to specify which IP address to bind should not be run in the host environment unless they should also service requests sent to jail IP addresses. Attempting to serve NFS from the host environment may also cause confusion, and cannot be easily reconfigured to use only specific IPs, as some NFS services are hosted directly from the kernel. Any third-party network software running in the host environment should also be checked and configured so that it does not bind all IP addresses, which would result in those services' also appearing to be offered by the jail environments.

Once these daemons have been disabled or fixed in the host environment, it is best to reboot so that all daemons are in a known state, to reduce the potential for confusion later (such as finding that when you send mail to a jail, and its sendmail is down, the mail is delivered to the host, etc.).

Getting services to not listen to *

First off, we should make sure we get the system so that we have nothing listening on *, to check what what we need to modify issue this command

# sockstat | grep "\*:[0-9]"

This should give you a synopsys of all the processes and ports you need to trim down. Here are some hints with your ipv4 addr being 10.0.0.1 and your ipv6 addr being 2002::7ea9

# vi /etc/rc.conf
sendmail_enable="NO"
inetd_flags="-wW -a 192.168.0.1"
rpcbind_enable="NO"

sshd:
# vim /etc/ssh/sshd_config

change ListenAddress derivative

ListenAddress 192.168.0.1
ListenAddress 192.168.0.111
ListenAddress 2002::7ea9

syslogd:

# vim /etc/rc.conf
### enable syslogd, no network socket will be opened for syslogd
syslogd_enable="YES"
syslogd_flags="-s -s"

rsyncd:
# vim /usr/local/etc/rsyncd.conf
address = 192.168.0.1

MySQL:

# vim /etc/my.cnf
bind-address=10.0.0.1

Samba: (this will get you most of the way there)
edit /usr/local/etc/smb.conf
change the following:
interfaces = 10.0.0.242/24 127.0.0.1
socket address = 10.0.0.242

bind interfaces only = yes
note: if you don't need wins lookups and netbios name translation
you can safely disable nmbd. There doesn't seem to be a way
for nmb to not listen to *:138 anyhow.

To disable nmb go to /etc/rc.conf and replace samba_enable="YES" with smbd_enable="YES"

open ntpd (xntpd listens on all and cannot be changed)

# vim /usr/local/etc/ntpd.conf
listen on 10.0.0.1
listen on 2002::7ea9

bind
edit your named.conf (may be in /var/named/etc/named.conf)
In the options section:
listen-on { 10.0.0.242; };
listen-on-v6 port 53 { 2002:d8fe:10f1:6:202:b3ff:fea9:7ea9; };
query-source address 10.0.0.242 port *;
query-source-v6 address 2002:d8fe:10f1:6:202:b3ff:fea9:7ea9 port *;

The file system layout is described in the following list:

- Each jail will be mounted under the /home/jail/THE_JAIL_NAME directory.

- /home/jail/read-only/root # is the template for each jail and the read-only partition for all of the jails.

- A blank directory will be created for each jail under the /home/jail directory.

- Each jail shall have its own read-write system that is based upon /home/jail/read-only/skel.

- Each jailspace (read-write portion of each jail) shall be created in /home/jail/read-write/THE_JAIL_NAME.

- Each jail will have a /s directory, that will be linked to the read-write portion of the system.

Creating the Template
This section will describe the steps needed to create the master template that will be the read-only portion for the jails to use.

It is always a good idea to update the FreeBSD system to the latest -RELEASE branch. Check the corresponding Handbook Chapter to accomplish this task. In the case the update is not feasible, the buildworld will be required in order to be able to proceed. Additionally, the sysutils/cpdup package will be required. We will use the portsnap(8) utility to download the FreeBSD Ports Collection. The Handbook Portsnap Chapter is always good reading for newcomers.

Install cpdup

# cd /usr/ports/sysutils/cpdup ; make install clean

Create directories

# mkdir /home/jail
# mkdir /home/jail/read-only
# mkdir /home/jail/read-write

Create a directory structure for the read-only file system which will contain the FreeBSD binaries for our jails:

# mkdir /home/jail/read-only/root

Change directory to the FreeBSD source tree and install the read-only file system to the jail template:

# cd /usr/src

If you have already rebuilt your userland using make world or make buildworld, you can skip following step and install your existing userland into the new jail.

# make buildworld

This command will populate the directory subtree chosen as jail's physical location on the file system with the necessary binaries, libraries, manual pages and so on.

# make installworld DESTDIR=/home/jail/read-only/root

The distribution target for make installs every needed configuration file. In simple words, it installs every installable file of /usr/src/etc/ to the /etc directory of the jail environment: /home/jail/THE_JAIL_NAME/etc/.

# make distribution DESTDIR=/home/jail/read-only/root

Next, prepare a FreeBSD Ports Collection for the jails as well as a FreeBSD source tree, which is required for mergemaster:

# mkdir /home/jail/read-only/root/usr/ports
# portsnap -p /home/jail/read-only/root/usr/ports fetch extract
# portsnap -p /home/jail/read-only/root/usr/ports fetch update
# cpdup /usr/src /home/jail/read-only/root/usr/src

Setup for Drupal:

# cd /home/jail/read-only/root
# mkdir www
# cd www
# fetch http://ftp.drupal.org/files/projects/drupal-6.22.tar.gz
# tar zxvf drupal-6.22.tar.gz
# mv drupal-6.22 drupal6
# chown -R root:wheel drupal6

Create a skeleton for the read-write portion of the system:

# mkdir /home/jail/read-only/skel
# mkdir /home/jail/read-only/skel/home
# mkdir /home/jail/read-only/skel/usr-X11R6
# mkdir /home/jail/read-only/skel/distfiles

??? # mkdir /home/jail/read-only/skel/drupal-sites
??? # mkdir /home/jail/read-only/skel/drupal-files

# cd /home/jail/read-only/root
# mv etc /home/jail/read-only/skel
# mv usr/local /home/jail/read-only/skel/usr-local
# mv tmp /home/jail/read-only/skel
# mv var /home/jail/read-only/skel
# mv root /home/jail/read-only/skel

# mv www/drupal6/sites /home/jail/read-only/skel/drupal-sites

Use mergemaster to install missing configuration files. Then get rid of the extra directories that mergemaster creates:

# mergemaster -t /home/jail/read-only/skel/var/tmp/temproot -D /home/jail/read-only/skel -i
How should I handle ./.cshrc? [Leave it to install later] enter
How should I handle ./.profile? [Leave it to install later] enter
...
keep pressing space until you see end : q

*** You installed a login.conf file, so make sure that you run
'/usr/bin/cap_mkdb /home/jail/read-only/skel/etc/login.conf'
to rebuild your login.conf database

Would you like to run it now? y or n [n] y

*** You installed a services file, so make sure that you run
'/usr/sbin/services_mkdb -q -o /home/jail/read-only/skel/var/db/services.db /home/jail/read-only/skel/etc/services'
to rebuild your services database

Would you like to run it now? y or n [n] y

*** You installed a new master.passwd file, so make sure that you run
'/usr/sbin/pwd_mkdb -d /home/jail/read-only/skel/etc -p /home/jail/read-only/skel/etc/master.passwd'
to rebuild your password files

Would you like to run it now? y or n [n] y

# cd /home/jail/read-only/skel
# rm -R bin boot lib libexec mnt proc rescue sbin sys usr dev

Now, symlink the read-write file system to the read-only file system. Please make sure that the symlinks are created in the correct s/ locations. Real directories or the creation of directories in the wrong locations will cause the installation to fail.

# cd /home/jail/read-only/root
# mkdir s
# ln -s s/etc etc
# ln -s s/home home
# ln -s s/root root
# ln -s ../s/usr-local usr/local
# ln -s ../s/usr-X11R6 usr/X11R6
# ln -s ../../s/distfiles usr/ports/distfiles
# ln -s s/tmp tmp
# ln -s s/var var

# ln -s s/home/cwn_www/nginx www
# ln -s s/home/cwn_www/scripts scripts

??? # ln -s ../../s/drupal-sites www/drupal6/sites
??? # ln -s ../../../../s/drupal-files www/drupal6/sites/default/files

??? Changing the directory permission
??? # find /home/jail -type d -print0 | xargs -0 -I @ sh -c 'chmod 700 @ ; chown root:wheel @'

As a last step, create a generic /home/jail/read-only/skel/etc/make.conf with its contents as shown below:

# echo 'WRKDIRPREFIX?= /s/portbuild' >> /home/jail/read-only/skel/etc/make.conf

Having WRKDIRPREFIX set up this way will make it possible to compile FreeBSD ports inside each jail. Remember that the ports directory is part of the read-only system. The custom path for WRKDIRPREFIX allows builds to be done in the read-write portion of every jail.

Creating Jails:

Now that we have a complete FreeBSD jail template, we can setup and configure the jails in /etc/rc.conf. This example demonstrates the creation of 3 jails: "NS", "MAIL" and "WWW".

Put the following lines into the /etc/fstab file, so that the read-only template for the jails and the read-write space will be available in the respective jails:

# vim /etc/fstab
### read-only partitions
/home/jail/read-only/root /home/jail/read-only/ns nullfs ro 0 0
/home/jail/read-only/root /home/jail/read-only/mail nullfs ro 0 0
/home/jail/read-only/root /home/jail/read-only/www nullfs ro 0 0

### read-write partitions
/home/jail/read-write/ns /home/jail/read-only/ns/s nullfs rw 0 0
/home/jail/read-write/mail /home/jail/read-only/mail/s nullfs rw 0 0
/home/jail/read-write/www /home/jail/read-only/www/s nullfs rw 0 0

Note: Partitions marked with a 0 pass number are not checked by fsck(8) during boot, and partitions marked with a 0 dump number are not backed up by dump(8). We do not want fsck to check nullfs mounts or dump to back up the read-only nullfs mounts of the jails. This is why they are marked with "0 0" in the last two columns of each fstab entry above.

# vim /etc/rc.conf

### for jail host
ifconfig_re0="inet 192.168.0.1  netmask 255.255.255.0"

### for jail guests
ifconfig_re0_alias0="inet 192.168.0.2 netmask 255.255.255.255"
ifconfig_re0_alias1="inet 192.168.0.3 netmask 255.255.255.255"
ifconfig_re0_alias2="inet 192.168.0.4 netmask 255.255.255.255"

### jail settings
jail_enable="YES"                   # Set to NO to disable starting of any jails
jail_set_hostname_allow="NO"        # Allow root user in a jail to change its hostname
#jail_socket_unixiproute_only="YES" # Route only TCP/IP within a jail
jail_list="ns mail www"             # Space separated list of names of jails
                                    # Note: Jail names in jail_list should contain alphanumeric characters only.

### j-ns
jail_ns_hostname="j-ns.local"                  # jail's hostname
jail_ns_ip="192.168.0.2"                 # jail's IP address
jail_ns_rootdir="/home/jail/read-only/ns" # jail's root directory
jail_ns_devfs_enable="YES"               # mount devfs in the jail
jail_ns_devfs_ruleset="ns_ruleset"       # devfs ruleset to apply to jail

### j-mail
jail_mail_hostname="j-mail.local"                  # jail's hostname
jail_mail_ip="192.168.0.3"                   # jail's IP address
jail_mail_rootdir="/home/jail/read-only/mail" # jail's root directory
jail_mail_devfs_enable="YES"                 # mount devfs in the jail
jail_mail_devfs_ruleset="mail_ruleset"       # devfs ruleset to apply to jail

### j-www
jail_www_hostname="j-www.local"                  # jail's hostname
jail_www_ip="192.168.0.4"                  # jail's IP address
jail_www_rootdir="/home/jail/read-only/www" # jail's root directory
jail_www_devfs_enable="YES"                # mount devfs in the jail
jail_www_devfs_ruleset="www_ruleset"       # devfs ruleset to apply to jail

Note: The jail_name_rootdir variable must not be set to a path which includes a symbolic link, otherwise the jails will refuse to start. Use the realpath(1) utility to determine a value which should be set to this variable. Please see the FreeBSD-SA-07:01.jail Security Advisory for more information.

Note: for devfs ruleset, cat /etc/defaults/devfs.rules.

Make sure your jail host has all the IP addresses for jail guests:
# /etc/rc.d/netif restart && /etc/rc.d/routing restart
# ifconfig

Create the required mount points for the read-only file system of each jail:

# cd /home/jail/read-only
# mkdir ns mail www
??? # chmod 700 ns mail www
??? # chown root:wheel ns mail www

Install the read-write template into each jail. Note the use of sysutils/cpdup, which helps to ensure that a correct copy is done of each directory:

# cpdup /home/jail/read-only/skel /home/jail/read-write/ns
# cpdup /home/jail/read-only/skel /home/jail/read-write/mail
# cpdup /home/jail/read-only/skel /home/jail/read-write/www

In this phase, the jails are built and prepared to run. First, mount the required file systems for each jail, and then start them using the /etc/rc.d/jail script:

# mount -a
# df -h

Start a specific jail:
# /etc/rc.d/jail start ns
# /etc/rc.d/jail start mail
# /etc/rc.d/jail start www

Start all jails:
# /etc/rc.d/jail start

Stop a specific jail:
# /etc/rc.d/jail stop www

A clean way to shut down a jail(8) is not available at the moment. This is because commands normally used to accomplish a clean system shutdown cannot be used inside a jail. The best way to shut down a jail is to run the following command from within the jail itself or using the jexec(8) utility from outside the jail:

j-www # sh /etc/rc.shutdown
or
# jexec 2 shutdown -p now

The jails should be running now. To check if they have started correctly, use the jls(8) command. Its output should be similar to the following:

# jls
JID IP Address Hostname Path
7 192.168.0.2 j-ns /home/jail/read-only/ns
8 192.168.0.3 j-mail /home/jail/read-only/mail
9 192.168.0.4 j-www /home/jail/read-only/www

At this point, it should be possible to log onto each jail, add new users or configure daemons. The JID column indicates the jail identification number of each running jail. Use the following command in order to perform administrative tasks in the jail whose JID is 9:

# jexec 9 tcsh

Use google's public DNS server as a DNS resolver:

j-www # vi /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4

Modify /etc/rc.conf:
j-www # vi /etc/rc.conf
### enable syslogd, no network socket will be opened for syslogd
syslogd_enable="YES"
syslogd_flags="-s -s"

### SendMail
sendmail_enable="NO"

Modify /etc/hosts:
j-www # vim /etc/hosts
::1 localhost localhost.j-www.local
127.0.0.1 localhost localhost.j-www.local

192.168.100.34 j-www.local j-www

To fix this error "hash map "Alias0": missing map file /etc/mail/aliases.db in FreeBSD jail":
# sendmail -bi
# ls -l /etc/mail/aliases.db

Note: you must edit /etc/hosts before running sendmail -bi command.

Comment out adjkerntz in crontab:
j-www # vim /etc/crontab
#1,31 0-5 * * * root adjkerntz -a

SVN server setup
# cd /usr/ports/devel/subversion
# make install clean

Make sure you DO uncheck this option:
[] BDB=off "db4 repository backend"

# mv /usr/local/bin/svn /usr/local/bin/svn.orig

# vi /usr/local/bin/svn


#!/bin/sh

### initialize
svnarg=""

### use encoding utf-8 as default if run "svn ci" or "svn commit".
if [ "$1" != "help" ]; then
  for myarg in "$@"; do
    if [ "${myarg}" = "commit" ] || [ "${myarg}" = "ci" ]; then
      svnarg="--encoding utf-8"
      break
    fi
  done
fi

### wrapper script to set umask to 027 on subversion binaries
### Note: the meaning of each umask:
### umask 002 // File permission 644. Owner can read/write. Group and Others can only read.
### umask 007 // File permission 660. Owner and Group can read/write. Others can not read or write.
### umask 027 // File permission 640. Owner can read/write. Group can read. Others can not read or write.
umask 027

### svn command
/usr/local/bin/svn.orig ${svnarg} "$@"

Don't forget to make your wrapper script executable:

# chmod +x /usr/local/bin/svn

# mkdir /home/jail/svn-rep
# cd /home/jail/svn-rep
# svnadmin create ns
# svnadmin create mail
# svnadmin create www

# vi www/conf/svnserve.conf
[general]
anon-access = none
auth-access = write
password-db = passwd

# vi www/conf/passwd
[users]
danny = mypassword

Start SVN server as a stand-alone daemon
# /usr/local/bin/svnserve -d --listen-port=3690 --listen-host=0.0.0.0 -r /home/jail/svn-rep

# vi /etc/rc.conf
### SVN
svnserve_enable="YES"
svnserve_flags="-d --listen-port=3690 --listen-host=0.0.0.0"
svnserve_data="/home/jail/svn-rep"
svnserve_user="root"
svnserve_group="wheel"

# svn import /home/jail/read-only/skel/drupal-sites/all file:///home/jail/svn-rep/www -m "initial import"

SVN client setup
# rm -r /www/drupal6/sites/all/*
# svn checkout svn://192.168.0.1/www /www/drupal6/sites/all

sending mail through sendmail inside a jail:
# cat mail.php
<?php
mail('test@example.com', 'My Subject', 'my msg');
?>

# php mail.php
collect: Cannot write ./dfpA4NVE5H097195 (bfcommit, uid=25, gid=25): Permission denied
queueup: cannot create queue file ./qfpA4NVE5H097195, euid=25, fd=-1, fp=0x0: Permission denied

# chown smmsp:smmsp /var/spool/clientmqueue
# chmod 770 /var/spool/clientmqueue

Fine Tuning and Administration
There are several options which can be set for any jail, and various ways of combining a host FreeBSD system with jails, to produce higher level applications. This section presents:

Some of the options available for tuning the behavior and security restrictions implemented by a jail installation.

Some of the high-level applications for jail management, which are available through the FreeBSD Ports Collection, and can be used to implement overall jail-based solutions.

Fine tuning of a jail's configuration is mostly done by setting sysctl(8) variables. A special subtree of sysctl exists as a basis for organizing all the relevant options: the security.jail.* hierarchy of FreeBSD kernel options. Here is a list of the main jail-related sysctls, complete with their default value. Names should be self-explanatory, but for more information about them, please refer to the jail(8) and sysctl(8) manual pages.

security.jail.set_hostname_allowed: 1

security.jail.socket_unixiproute_only: 1

security.jail.sysvipc_allowed: 0

security.jail.enforce_statfs: 2

security.jail.allow_raw_sockets: 0

security.jail.chflags_allowed: 0

security.jail.jailed: 0

These variables can be used by the system administrator of the host system to add or remove some of the limitations imposed by default on the root user. Note that there are some limitations which cannot be removed. The root user is not allowed to mount or unmount file systems from within a jail(8). The root inside a jail may not load or unload devfs(8) rulesets, set firewall rules, or do many other administrative tasks which require modifications of in-kernel data, such as setting the securelevel of the kernel.

Ping from jail not permitted error

ICMP is disallowed by defaut for jails, see the sysctl :

# echo security.jail.allow_raw_sockets=1 >> /etc/sysctl.conf

Now restart your jails and the problem should be fixed.

There are good reasons for this default, so if you test remember to set it back when you are done.

Also, on a point of style, jails in their current form (see VIMAGE) do not get a network stack of their own so they don't have a firewall but share the hosts' network and firewall, etc.

Cannot remove files inside a jail directory
override r-sr-xr-x root/wheel schg for j/mroot/bin/rcp? y
rm: j/mroot/bin/rcp: Operation not permitted
rm: j/mroot/bin: Directory not empty
override r--r--r-- root/wheel schg for j/mroot/lib/libc.so.7? y
rm: j/mroot/lib/libc.so.7: Operation not permitted

# chflags -R noschg /home/jail/read-write/ns
# rm -r /home/jail/read-write/ns

High-level administrative tools in FreeBSD Ports Collection

# cd /usr/ports/sysutils/jailutils ; make install clean

# cd /usr/ports/sysutils/ezjail ; make install clean

Reference:
http://gala4th.blogspot.com/2012/01/install-jails-on-freebsd-82.html

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html

VIMAGE - Better virtualization in FreeBSD 8
http://gala4th.blogspot.com/2011/10/vimage-better-virtualization-in-freebsd.html

FreeBSD 8 VIMAGE + epair howto
http://gala4th.blogspot.com/2011/10/freebsd-8-vimage-epair-howto.html

Creating a FreeBSD Jail
http://www.section6.net/wiki/index.php/Creating_a_FreeBSD_Jail