Technical Journeys

my vagrant best practices

2015-06-23T01:47:00.002-07:00

I tried to create vagrant machines recently and found that there is no best practice advices in internet. Here I wrote down some tricks I used during my usage of vagant

Use provision command to test shell scripts or provision scripts changes.

NTP setup

2015-06-02T02:07:00.000-07:00

NTP Troubleshooting

check if NTP port is opened on ntp server
nmap -p123 -sU -P0 --reason 0.rhel.pool.ntp.org
check if response is dropped : firewall/router drop returned packets
ntpdate -dv 0.rhel.pool.ntp.org
ntpq

reachable, reject
non-reachable, reject
non-reachable, reject, mobilize

Juniper terminal client can't launch at IE11 and Java

2015-04-30T03:12:00.001-07:00

Recently juniper terminal clients are failed to launch after some windows updates.

My env:

IE11

Java 8.0.45

Windows 7 64bit

I tried the following ways, however they don't help.

Uninstall juniper softwares

Clear IE cache

Applet testing from www.java.com can pass.

After adding the vpn site into ie compatibility view and allow it in pop-up blocker, juniper terminal client can launch.

ETL: awk vs other script languages vs ETL tools

2015-03-20T10:06:00.001-07:00

I did a lot of data migration (ETL) last year. I didn't want to write program to do those things and want to try something new. I knew that AWK is very good script language to precessing text file. so I decided to give it a try.

The whole experience using AWK is quite good. I think the main advantage is that the mental model of using AWK processing text files is more nature than other languages.

Now I think, the more easy way is possible using Pentaho data integration tools to do the same things.

Compile rlwrap in Redhat/Oracle Enterprise Linux 5.4

2014-05-29T01:35:00.001-07:00

Rlwrap(0.41) depends on readline library and readline depends on libtermcap.

Download source code of rlwrap from http://utopia.knoware.nl/~hlub/uck/rlwrap/
Get dependencies of rlwrap from http://public-yum.oracle.com/repo/EnterpriseLinux/EL5/4/base/x86_64/

libtermcap-devel-2.0.8-46.1.x86_64.rpm
readline-5.1-3.el5.x86_64.rpm
readline-devel-5.1-3.el5.x86_64.rpm

build and install....

Edit:
maybe it's more easy to install the package directly from Extra Packages for Enterprise Linux instead of building manually.

Implementing oracle Data guard via SSH and autossh

2014-04-14T22:10:00.001-07:00

Oracle 9i and 10g gave instructions on how to configure SSH port forwarding with data guard. However it can't find in MOS now. I recently configured a 11.2 physical standby configuration. The articles I used as reference is as following:
http://www.scribd.com/doc/68622653/Implementing-SSH-Port-Forwarding-With-Data-Guard

http://ermanarslan.blogspot.sg/2013/07/database-redo-transport-compression.html
http://linuxaria.com/howto/permanent-ssh-tunnels-with-autossh?lang=en

I don't want to manually start ssh when machine is restarted or some network issues. I search other options to mitigate this issues. Autossh seems a sound solution to this issue. http://linuxaria.com/howto/permanent-ssh-tunnels-with-autossh?lang=en give the configuration. I followed the configuration and found that autossh can't start after reboot when put the following command into /etc/rc.local

su -s /bin/sh autossh -c 'autossh -M 0 -q -f -N -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -L 3306:localhost:3306 autossh@remotemachine'

After specify the full path of autossh, it works.

su -s /bin/sh autossh -c '/usr/local/bin/autossh -M 0 -q -f -N -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -L 3306:localhost:3306 autossh@remotemachine'

Other configurations are similar as in the above links, I don't repeat those here.

Don't debug procedure in standby database

2014-03-12T22:48:00.001-07:00

Always got the following error when debugging a procedure in SQL Developer even the privileges are granted.

Executing PL/SQL: CALL DBMS_DEBUG_JDWP.CONNECT_TCP( '10.20.30.125', '3703' )
ORA-01031: insufficient privileges
ORA-06512: at "SYS.DBMS_DEBUG_JDWP", line 68
ORA-06512: at line 1
This session requires DEBUG CONNECT SESSION and DEBUG ANY PROCEDURE user privileges.

I found that the connected database is logical standby after several testing and troubleshooting. Can do the debugging after changing the connection to primary database.

AWS EC2--can't get windows password without keypair assigned.

2014-02-11T22:24:00.002-08:00

I setup a windows EC2 instances without keypair. I thought I can use password to login window machine via RDP. However I always can't get the password. I stopped and startted the EC2 instances. It still don't work. I tried to launched a new instance, the same issue. Don't find solution from stackoverflow as well. At last, I noticed that "Unable to retrieve public keys to generate Administrator password. Please ensure that you have launched with a keypair assigned" in system log. I launched a new instance with keypair and I can get the windows password now.

Mysql server very slow with low cpu,io usage and little data

2014-02-09T23:20:00.001-08:00

Recently, one mysql database in my company is very slow. The solution:

skip-name-resolve
or add host names into host file
or add hostnames and their ip addresses to name server.

Ref:

http://allurcode.com/2010/04/07/mysql-very-slow-without-a-reason/
http://dev.mysql.com/doc/refman/5.1/en/host-cache.html

Build WebKit on windows 7 with visual studio 2013 express

2014-01-03T00:00:00.001-08:00

cygwin

Install cygwin by cygwin-downloader.exe, after cygwin installation, do the following:

Install gcc

downgrade python from 2.7 to 2.6.8

MathML fonts

download MathML fonts from https://developer.mozilla.org/en-US/docs/Mozilla/MathML_Project/Fonts

https://developer.mozilla.org/@api/deki/files/6388/=MathML-fonts.msi

update-webkit script

the script must be runned. It will change environment variables.

Error:

LocalizedStrings.cpp(248): error C2001: newline in constant

https://bugreports.qt-project.org/browse/QTBUG-23590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Source:

http://www.webkit.org/building/tools.html

http://www.webkit.org/building/checkout.html

My hosting plan

2013-12-26T19:11:00.001-08:00

I have figured out several startup ideas and want to start my own opportunities. I need to setup website, mobile app etc . To give my customer best user experience in my budget, I compare cloud hosting, web hosting, VPS hosting etc. Cloud hosting is too expensive, Webhosting is too many restrictions. Finally I choose VPS hosting. Linode and digitalocean are in my list. In the end, I choose https://www.digitalocean.com/ after comparing the price plans between Linode and digitalocean.

Missed opportunity

2013-12-04T01:04:00.000-08:00

Half years ago, I have a plan to make this kind of mobile app for Singapore. What a pity! don't implement it.

Scrapy to crawl dynamic contents generated from javascript

2013-11-25T22:31:00.001-08:00

Some websites adopt anti-scrapping technologies. One of such technologies is to use javascript to generate content (http://blog.databigbang.com/scraping-vs-antiscraping/). Javascript enabled crawlers are not feature complete as scrapy. Scrapy and selenium to scrape this kind of sites after researching and several trial.

Selenium env setup:

Download selenium server from http://www.seleniumhq.org/download/, run the download jar file: java -jar selenium-server-standalone-2.37.0.jar
Install Selenium IDE from Firefox (http://www.seleniumhq.org/download/)
Install Selenium Client & WebDriver Language Bindings for Python: pip install selenium

Code changes of Scrapy CrawlSpider:

Code from http://snipplr.com/view/66998/ is almost enough. only the time.sleep(2.5) statement cann't be parsed. I changed that statements as "sel.wait_for_page_to_load(10000) " according to http://blog.fangruhua.name/post-59.html

Install Generic Collectible Card Game on 64bit Kubuntu

2013-11-12T19:32:00.001-08:00

gccg.sourceforge.net only mentioned how to install on 32bit system. That don't work on my 64 bit kubuntu virtual machine.

The solution to install 64bit is given at http://sourceforge.net/p/gccg/discussion/248070/thread/01f14149/#f36a

After installed SDL, SDL_image, SDL_net, SDL_ttf and SDL_mixer, I sill got the following error when I run : sh ./Metw

./ccg_client: error while loading shared libraries: libjpeg.so.62: cannot open shared object file: No such file or directory

As my vm is 64bit, I run to install libjpeg62.

apt-get install libjpeg62

change server.name following the instruction at http://sourceforge.net/p/gccg/discussion/248070/thread/01f14149/#f36a

Oracle RAC 11g Database on Linux Using VirtualBox

2013-10-23T06:16:00.000-07:00

I installed RAC following the instruction at http://www.oracle-base.com/articles/11g/oracle-db-11gr2-rac-installation-on-oracle-linux-6-using-virtualbox.php#vm_setup. I failed to completed the installation as our office subnet is not as the subnet mentioned at the article. Search via google and found another article http://www.lab128.com/rac_installation_using_vb/article_text.html. The article at lab128 sounds good for our office env. The following are some specical issues during the installation following the article at lab128.

In the software type installation screen, I choose Desktop as I am not comfortable to configure many things in text mode.

Run Automatic Setup by installing 'oracle-rdbms-server-11gR2-preinstall' package

I got error: Error: failure: repodata/primary.xml.gz from ol6_latest: [Errno 256] No more mirrors to try. when running "yum install oracle-rdbms-server-11gR2-preinstall". Following the correct answer at https://forums.oracle.com/thread/2550364, I added the following one line into /etc/yum.conf and cleaned repository metadata.

http_caching=none

Install ASMLib

I got the following error. [Errno 14] PYCURL ERROR 56 - "Failure when receiving data from the peer.

Solution: use download manager to download primary.xml.gz from http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/repodata/primary.xml.gz. replace primary.xml.gz at /var/cache/yum/x86_64/6Server/ol6_latest with the downloaded primary.xml.gz

Clone:

Refresh MAC address for the cloned machine on adapter 1 and adapter 2 of network, change the corresponding ifcfg-eth0, ifcfg-eth1, ifcfg-eth2 file at /etc/sysconfig/network-scripting

changes: delete UUID, change HWADDR

Recoverable scripts in streams setup

2013-10-11T03:44:00.003-07:00

The following errors were encountered during setting up streams.

dbms_recoverable_script.execute_block: exception, sql code = -44001, error message: ORA-44001: invalid schema
ORA-06512: at "SYS.DBMS_RECO_SCRIPT_INVOK", line 139
ORA-06512: at "SYS.DBMS_STREAMS_RPC", line 485

I searched the internet and find a very useful article talking about recoverable scripts at http://www.juliandyke.com/Blog/?m=201011 . However my case is different with Julian Dyke's case, my case is stopped at step 7.

After comparing the block scripts with my saved setup script, I found that the user name -strmadmin is in lower case. There is no documentation about dbms_recoverable_script API. After several trials, I used the following scripts to change the forward script in step 7. I run dbms_recoverable_script.run to run my scripts. It passed step 7 successfully.

declare
v_block clob := '-- Set up queue "strmadmin"."PRIMARY$APPQ"' || CHR(10);
begin
dbms_lob.append(v_block,'BEGIN' || CHR(10));
dbms_lob.append(v_block,'dbms_streams_adm.set_up_queue(' || CHR(10));
dbms_lob.append(v_block,'queue_table => ''"STRMADMIN"."PRIMARY$APPQT"'',' || CHR(10));
dbms_lob.append(v_block,'storage_clause => NULL,' || CHR(10));
dbms_lob.append(v_block,'queue_name => ''"STRMADMIN"."PRIMARY$APPQ"'',' || CHR(10));
dbms_lob.append(v_block,'queue_user => ''"STRMADMIN"'');'|| CHR(10));
dbms_lob.append(v_block,'END;' || CHR(10));

dbms_recoverable_script.MODIFY_FORWARD_BLOCK(
SCRIPT_ID =>'E8741630FB012C11E043291E140A7C6F',
BLOCK_NUM =>7,
FORWARD_BLOCK=>v_block,
DBLINK_FORWARD =>'DALIAN',
ACTION =>'REPLACE');
end;

Should change undo_block for step 7 as well.

Resolve ORA-26786 in logical standby configuration

2013-09-26T01:14:00.001-07:00

Recently I got ORA-26786 errors several times in our logical standby database 11.2.0.3. Search support.oracle.com and found Bug 10302680 - Logical Standby skips some rows inserted after instantiate table and may produce ORA-26786 ORA-26787 (Doc ID 10302680.8) . After using 11.2.04 as logical standby, I still got ORA-26786 on the logical standby database. I was thinking what's happened. After read the events in DBA_LOGSTDBY_EVENTS and compared the schema between primary db and the logical standby, I found that errors is caused by non published bug 14305552 ORA-26786 / ORA-1403 After New Column Added to Table with Default Value (Doc ID 1529239.1). However the workaround solution in the Doc ID 1529239.1 don't work on logical standby database.

I don't want to re-create the logical standby database as the database is big and it will take several hours to recreate the logical standby database. The good thing is that I noticed How To Reinstantiate a Single Table in a Streams Environment (Doc ID 335502.1) in the search results during my searching ORA-26786 in support.oracle.com. I wonder if I can do the same thing on logical standby database.

Luckily, there is indeed a document about this in MOS. that is Synchronizing tables in a Logical Standby Database (Doc ID 271455.1) . I followed the way to handle this as Example A in (Doc ID 271455.1) .

I created database link and run DBMS_LOGSTDBY.INSTANTIATE_TABLE. Unluckily, I got another error: ORA-16276. Google internet and found another trick thing at https://forums.oracle.com/thread/1009899 . The database link need to be an public database link.

Still unluckily I got another error ORA-31631 when I nun DBMS_LOGSTDBY.INSTANTIATE_TABLE again. Fortunately get the solution from https://forums.oracle.com/thread/841693 . I did grant EXP_FULL_DATABASE to the database link user at primary database and IMP_FULL_DATABASE to the database link user at standby database.

Finally I got the error resolved after the so long trouble shooting journey.

A final notes, I am not sure if granting SELECT_CATALOG_ROLE to the database link users also contribute to the success running of DBMS_LOGSTDBY.INSTANTIATE_TABLE. I did that grant when I got the error ORA-16276. This solution is mentioned at http://www.linuxidc.com/Linux/2012-05/61129.htm

Update: The workaround solution work in logical standby database. It should be applied on primary database.

ORA-01341 errors in oracle logical standby database

2013-09-17T08:33:00.003-07:00

All solutions from forums and blogs don't help me. I couldn't find capture name in logical standby database. I tried to run the following statement to adjust max SGA size for sql apply to use after I read the Oracle® Database PL/SQL Packages and Types Reference (http://docs.oracle.com/cd/B28359_01/appdev.111/b28419/d_lsbydb.htm#BDCIEGEI)

execute DBMS_LOGSTDBY.APPLY_SET('MAX_SGA', 1024);

After the setting was changed, ORA-01341 disappeared.

startup oracle database with ORA-19815: WARNING: db_recovery_file_dest_size of xxx bytes is 100.00% used, without RMAN configuration

2013-08-19T01:58:00.001-07:00

startup mount;

alter system set DB_RECOVERY_FILE_DEST='' scope=both sid='*';

alter system set log_archive_dest_1='location=/backup/sid/archivelog' scope=both sid='*'

alter database open;

Koreader can't open a deleted and copied-back pdf file.

2013-07-17T19:36:00.001-07:00

I deleted a file which previously can be read from koreader. However koreader can't open the file again after I copied back the file into kindle. I connected kindle to PC usign usb cable, and deleted the corresponding file in koreader/history folder. Koreader can open the file again.

Feedly migration from google reader

2013-07-09T06:39:00.001-07:00

After migration from google reader to feedly cloud, the same resources are marked as read if the resources were read in other devices. The reading synchronisation saves readers a lot of time during reading across many devices. I like this feature very much.

What's the point to define an interface implemented by only one class?

2013-05-27T20:29:00.001-07:00

I didn't see the point to define an interface which is implemented by only one class. I thought it was code smell. I had some doubts about how to fit domain service into DDD+CQRS and researched forums, groups and blogs. I found Jimmy Bograd at this link http://lostechies.com/jimmybogard/2008/08/21/services-in-domain-driven-design/ give his opinion on this.

Services are always exposed as an interface, not for “swappability”, testability or the like, but to expose a set of cohesive operations in the form of a contract. On a sidenote, it always bothered me when people say that an interface with one implementation is a design smell. No, an interface is used to expose a contract. Interfaces communicate design intent, far better than a class might.

How to understand the internal implementation of c# syntax sugars.

2013-05-14T00:15:00.001-07:00

IL Dasm is mentioned many times in the book "LINQ unleashed for C#", It is quite useful to understand the magics the c# compile do in the compiling stage for many syntax sugars.

Loading GeoNames Data Into SQL Server 2008

2013-05-09T21:28:00.001-07:00

I use linux to prepare the allcountries file for MSSQL to load data as EditPad Pro is not free. The steps I took:

#convert line ending

unix2dos allcountries.txt

iconv -f utf-8 -t UTF-16LE allcountries.txt -o allcountries-utf16.txt

#add UTF-16LE BOM

echo -ne '\xFF\xFE' > BOM.txt

#as I convert allcountries.txt from utf8 to utf16 at first, so I concatenate the two files instead of using the method mentioned in stackoverflow to save my time.

cat BOM.txt allcountries-utf16.txt > allcountries-utf16-BOM.txt

table creation and bulk inserting are same as Spatial Ed's blog: population of one country is too big and it will cause overflow error during bulk inserting. I changed the data type of population from int to numeric.

CREATE TABLE GeoNames(
      geonameid int NOT NULL,
      name nvarchar(200) NULL,
      asciiname nvarchar(200) NULL,
      alternatenames nvarchar(max) NULL,
      latitude float NULL,
      longitude float NULL,
      feature_class char(2) NULL,
      feature_code nvarchar(10) NULL,
      country_code char(3) NULL,
      cc2 char(60) NULL,
      admin1_code nvarchar(20) NULL,
      admin2_code nvarchar(80) NULL,
      admin3_code nvarchar(20) NULL,
      admin4_code nvarchar(20) NULL,
      population numeric(18,0) NULL,
      elevation int NULL,
      gtopo30 int NULL,
      timezone char(31) NULL,
      modification_date date NULL
)
GO

Reference:

http://blogs.msdn.com/b/edkatibah/archive/2009/01/13/loading-geonames-data-into-sql-server-2008-yet-another-way.aspx

http://stackoverflow.com/questions/4364156/iconv-converting-from-windows-ansi-to-utf-8-with-bom

http://en.wikipedia.org/wiki/Byte_order_mark

http://download.geonames.org/export/dump/

http://forum.geonames.org/gforum/posts/list/847.page

Startup ideas from non technic point of view

2013-04-11T03:58:00.001-07:00

From average joe's point of view, instead of personal technic, maybe one can find a lot of opportunities to start up businesses. One example is unroll.me. A lot of non-IT savvy persons don't know how to unsubscribe or move those unwanted mails to junk. Think as dumb person!