Code Tips

Battling Base64 Uploads on #GAE

2011-05-01T10:52:00.000-07:00

I can't believe it, I seem to be battle multiple fronts of erros in my code, trying to get base64 uploaded to #GAE.

My eclipse Jetty dev posts aren't pulling out of any parameters from the post. This is really firing me up. I am having to divert to my virtual box windows install to try to distinguish if its eclipse or my code. It appears something is broken on the jetty stuff b/c I can see the content come or the payload.

Oh, how I wish I could just get my parameters back with a click of a key. Something went wrong and I have no idea what happened.

Couple Hours Later:
ahhhh I got it working after messing with the classpath setup. I'm not sure what it was yet.

Another Hour goes by:
Maybe its a fluke. I updated the mac jvm and still the same thing, no parameters.

Encoding URL delimiters (for GWT)

2011-01-26T06:23:00.000-08:00

This is what I do to encode a parameter for my querystring:

public static String encodeUrlDelimiters(String s) {

if (s == null) {

return null;

}

s = s.replaceAll(";", "%2F");

s = s.replaceAll("/", "%2F");

s = s.replaceAll(":", "%3A");

s = s.replaceAll("\\?", "%3F");

s = s.replaceAll("&", "%26");

s = s.replaceAll("\\=", "%3D");

s = s.replaceAll("\\+", "%2B");

s = s.replaceAll("\\$", "%24");

s = s.replaceAll(",", "%2C");

s = s.replaceAll("#", "%23");

return s;

}

This is how I use it:

private String getGoogleLoginUrl() {

if (cp.getAccessToken() == null && cp.getAccessToken().getThirdPartyLoginData() == null) {

return null;

}

String domain = GWT.getHostPageBaseURL();

String path = Window.Location.getPath();

String qs = Window.Location.getQueryString();

String historyToken = History.getToken();

if (qs == null) {

qs = "";

}

if (domain.matches(".*/") == true) {

domain = domain.substring(0,domain.length()-1);

}

String url = domain + path + qs + "#" + historyToken;

url = URL.encode(url);

url = Global_String.encodeUrlDelimiters(url);

String baseUrl = cp.getAccessToken().getThirdPartyLoginData().getGoogleLoginUrl();

baseUrl = baseUrl.replaceAll("=%2F.*", "=" + url);

return baseUrl;

}

javax-usb deployment

2011-01-26T06:20:00.000-08:00

Control USB using Java. Observe and Listen to USB Events with Java.

1. download http://sourceforge.net/projects/javax-usb/files/ ri-linux

2. $ export JAVA_HOME=/usr/lib/jvm/java-6-sun ;echo $JAVA_HOME

3. /usr/lib/jvm/java-6-sun/include/linux$ sudo ln -s ../jni.h

4. compile the c interface $ ./javax-usb-ri-linux/jni/make

Reference

http://javax-usb.org/ - main site

http://javax-usb.cvs.sourceforge.net/javax-usb/javax-usb-ri-linux/ - source

This Advertising Combination Made Me Laugh!!!

2010-08-15T19:44:00.000-07:00

Eclipse Donation

2010-05-22T07:47:00.000-07:00

Eclipse is the best IDE I have ever used. So I donated when they asked :).

http://www.eclipse.org/helios/friends.php

Daughter crawled behind the server

2010-05-16T23:29:00.000-07:00

While I was sitting at my desk, my daughter crawled under the server and stood up behind it. She thought she was pretty cool because it got my attention. I didn't have my camera, so I used photobooth on my mac book pro.

Pushing Forward

2010-05-08T21:37:00.001-07:00

Saturday, my GoneVertical.com development day. I set out to write more code and move the project forward at every possible chance I get. I keep finding it takes a huge amount of initiative and will power when I sit down during these sessions to write code. Today, my right arm, pointer and index finger muscles are burning from dragging on the mouse track pad all week. I am also suffering some kind of back injury from working out side a month ago and this is creating some tingling in the same arm and leg. Even though I suffer from aches, pains and tiredness, I strive to code on and try my best to move forward. Yet, second guessing and questioning occurs twice as much when the labor seems 10 times as hard and pushing though this is a challenge but not impossible. Small incremental task achievements are critical for me during theses stages to push through.

Today I did make progress despite all my pain in the arm and back. I'm making progress with GoneVertical integration into my core-engine. Someday I hope to look back and say, it was good to keep pushing on despite everything.

GWT + GAE + Gadget Timestamp Converter Application

2010-05-05T23:20:00.000-07:00

I built a Google Web Toolkit application which can be quickly compiled for the Google App Engine and IGoogle. I made a source project then included it into a GAE and Gadget project so I didn't have to change compiling techniques depending on which I wanted to build it for. This actually ended up to be very easy to deploy the application this way.

http://www.google.com/ig/directory?url=demogwtdatetimegadget.appspot.com/demogwtdatetime_gadget/org.gonevertical.demo.client.DemoGwtDateTime_Gadget.gadget.xml - IGoogle Gadget

http://demogwtdatetimegae.appspot.com - GAE Example

http://gwt-examples.googlecode.com - Find the source in the DateTime Converter

http://code.google.com/p/gwt-examples/wiki/project_Gadget - gadget examples

Made a new GWT Demo - DateTime converter

2010-05-05T09:08:00.000-07:00

Last night I made a new GWT date timestamp converter.

demo: http://demogwtdatetimegae.appspot.com/

This will convert an epoch javascript or unix time stamp.

Intel X-25e SSD For MySQL Database

2010-05-03T21:06:00.000-07:00

I setup two servers a master and a slave with 3 Intel X-25e SSD 64GB hard drives each in a Raid 0 configuration on Dell Poweredge Servers. I use them at Verizon for one of the sales metrics applications.

I use a Ubuntu, Apache Tomcat with MySQL and the Google Web Toolkit to serve up the app. With Java's concurrency I can multi-thread queries concurrently data stored on these drives very quickly. I can easily hit 3000+ concurrent SQL seeks load balanced which is around 6000+ seeks a second to both servers at the same time. The web server and 2 database servers are AMD 64bit 8 core, can't remember all the stats.

I think the only thing that I can bog down the server are range queries like SELECT UserId FROM employee WHERE UserId IN (SELECT UserId FROM employee_data WHERE value='b'). Changing the sql to a join query fixes this. Seeking an entire record set is no problem and even better is seeking a multi-milion indexed record set. Talk about a kick in the pants.

Another worthy note is how fast it takes to dump a database on these. A 10GB takes around a minute, and maybe less than 30 minutes to load. Where as, on 15k rpms raid 5, I noticed it takes around an hour or more.

Moving files around on these bad boys is ridiculously fast.

These hard drives overall are impressive and I am not sure I have hit there limit yet to what they can do. There lot of fun to work with.

DBCP Pool Broken Pipe

2010-05-03T00:08:00.000-07:00

What may look like a broken pipe may actually be a SQL Statement failure. I have found in the cases where SQL statements are incorrect will cause a Broken Pipe to throw. It can cause anything trying to connect using the connection ont that pool to throw an exception too.

Add Log4J and watch your SQL Statements. First check file descriptors before you get carried away with this, although this is pretty easy to do. Add logging to your exceptions so you can easily identify your errors.

What to check:
1. check OS Max file descriptors, this is usually the case
2. if you raised the file descriptors, you may have a bad SQL statement causing the error. Seeing the stack trace isn't always clear to determine which sql statement threw the event.

Delete .svn Recursively - When copying projects

2010-04-25T18:05:00.000-07:00

Did you copy a eclipse project, and need to delete all the .svn directories. When svn cleanup ./directory/path doesn't work.

Use in Terminal:
sudo rm -rf `find . -type d -name .svn`

FYI:
pwd # see what directory your in
ls -la # see all the files including the hidden

GWT Horizontal Events Panel

2010-04-07T16:09:00.000-07:00

Have you wanted to add event observation/Listening handlers to a panel? This is and easy way to observe events in a GWT HorizontalPanel. Gwt 1.6, 2.0, 2.01, 2.02, 2.03+...

// easy extend and observe mouse events on a gwt horizontal panel widget

public class HorizontalEventPanel extends HorizontalPanel implements HasMouseOverHandlers, HasMouseOutHandlers {

public HorizontalEventPanel() {

sinkEvents(Event.MOUSEEVENTS);

}

public HandlerRegistration addMouseOverHandler(MouseOverHandler handler) {

return addDomHandler(handler, MouseOverEvent.getType());

}

public HandlerRegistration addMouseOutHandler(MouseOutHandler handler) {

return addDomHandler(handler, MouseOutEvent.getType());

}

MySql Connection Really Slow?

2010-04-05T16:12:00.001-07:00

If your MySql connection is really slow? Do you think its JDBC or Bandwidth sucking the speed? Are you going crazy checking wireshark for packet problems? Did you see Malformed Packet? Has the server been on for months? Did you get an update?

Before you check all that, make sure your Reverse DNS is setup, so MySql can authenticate you fast. On linux, add your workstation to the list in /etc/hosts 192.168.1.70 workstationHostName.

***** Check Reverse DNS ***** before you get into the deep debugging.

Make DBCP Connection Pool Using Tomcat Resource Context

2010-02-19T20:45:00.000-08:00

Setup JDBC DBCP (data base connection pooling) for use with any Java class.

Reference:

http://code.google.com/p/parsecsv2sql/source/browse/trunk/Csv2Sql/src/org/gonevertical/dts/lib/pooling/SetupInitialContext.java

How I set it up:

public DatabaseConnection(boolean pool) {

File executionlocation = null;

try {

executionlocation = new File(DatabaseConnection.class.getProtectionDomain().getCodeSource().getLocation().toURI());

} catch (URISyntaxException e) {

e.printStackTrace();

}

String execPath = executionlocation.getParent();

execPath = StringUtil.getValue("(.*?)/war", execPath);

String path = execPath + "/war/META-INF/context.xml";

String contextXmlPath = path;

String tmpPath = "/home/branflake2267/tmp";

// make my own context

SetupInitialContext ic = new SetupInitialContext(tmpPath);

ic.setContextXmlFileLocation(contextXmlPath);

ic.run();

context = ic.getInitialContext();

}

Rough Java DBCP Connection Pool Context Example Setup

2010-02-18T21:47:00.000-08:00

Setting up initial context for DBCP connection pool.

public static void test() {

System.setProperty(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.fscontext.RefFSContextFactory");

System.setProperty(Context.PROVIDER_URL, "file:///Users/branflake2267/tmp");

InitialContext ic = null;

try {

ic = new InitialContext();

} catch (NamingException e) {

e.printStackTrace();

}

// Construct BasicDataSource reference

Reference ref = new Reference("javax.sql.DataSource", "org.apache.commons.dbcp.BasicDataSourceFactory", null);

ref.add(new StringRefAddr("driverClassName", "com.mysql.jdbc.Driver"));

ref.add(new StringRefAddr("type", "javax.sql.DataSource"));

ref.add(new StringRefAddr("url", "jdbc:mysql://ark/system?autoReconnect=true"));

ref.add(new StringRefAddr("username", "Web"));

ref.add(new StringRefAddr("password", "pass*7"));

ref.add(new StringRefAddr("removeAbandoned", "true"));

ref.add(new StringRefAddr("removeAbandonedTimeout", "90"));

ref.add(new StringRefAddr("logAbandoned", "true"));

ref.add(new StringRefAddr("maxActive", "1000"));

ref.add(new StringRefAddr("maxIdle", "30"));

ref.add(new StringRefAddr("maxWait", "900"));

try {

ic.rebind("jdbc/basic", ref);

} catch (NamingException e) {

e.printStackTrace();

}

InitialContext ic2 = null;

try {

ic2 = new InitialContext();

} catch (NamingException e) {

e.printStackTrace();

}

DataSource ds = null;

try {

ds = (DataSource) ic2.lookup("jdbc/basic");

} catch (NamingException e) {

e.printStackTrace();

}

Connection conn = null;

try {

conn = ds.getConnection();

} catch (SQLException e) {

e.printStackTrace();

}

try {

conn.close();

} catch (SQLException e) {

e.printStackTrace();

}

MySql Communication Failure and Aborted Connections

2010-02-18T21:10:00.000-08:00

Have you got an error like this using MySql?

1. Communications link failure

2. Aborted connection

3. Last packet sent to the server was 0 ms ago.

4. Too many connections

Things you can do to figure out whats going on with mysql connections.

1. See how many connections are currently connected with this query: SHOW PROCESSLIST;

2. Check /etc/mysql/my.cnf max connection limit: max_connections = 100 #100 by default

3. Are you running out of file descriptors, check by: ulimit -a #by default its a 1000 on Ubuntu

4. Run your mysql queries and check the active sockets: netstat -na | grep 3306

5. set the file descriptor limits in /etc/security/limits.conf

When rapidly connecting and reconnecting to mysql database you may find that you run out of max connections to start with. The next thing you run out of is file descriptors because each time you connect and cleanly close the tcp socket it will wait to close the connection so that stray packets to not wonder in after you closed it. You can see if this is happening by checking ulimit -a and then watching how many are in time_wait with netstat -na while running your queries.

When you run out of file descriptors (sockets) to open up, mysql will crash and abort all the connections which could be caused on the sending and/or receiving server. MySql can handle thousands of connections at one time, although it may not be the most efficient process due to the authorization and setup cost, it gets the job done. I would recommend using connection pooling which saves time with reuse and keeping them ready to go when needed.

If you want to solve the communications failure with out pooling, raise the file descriptors limit to over 24000 or whatever number you determine suite your needs. Remember you will need alot more file descriptors than actual connections because of time_waits after a socket closes.

MySql Open Tables and Max Linux Open Files

2010-02-15T09:05:00.001-08:00

Ubuntu linux default max open files (limit) is 1000 and if your hitting mysql with huge amount of queries at one time, you may hit the open files limit. I also run into this problem with development with eclipse and having to open so many java classes.

Reference to whats happening

http://dev.mysql.com/doc/refman/5.0/en/table-cache.html

Check to see what the limits are:

ulimit -a

Change the open files limit:

sudo pico /etc/security/limits.conf

#add - the file open integers are huge due to my deployment, choose your own

mysql soft nofile 24000

mysql hard nofile 32000

sudo reboot

Two Gateways on Ubuntu, Split access.

2010-02-06T08:04:00.000-08:00

I set my Ubuntu server with to private gateways, which had access too two public ip addresses. I wanted to receive requests to the web tomcat and apache servers from both networks. This is how I configured my split access on my server to have traffic come in through both private subnets and sent out through the ip that it came in on.

#!/bin/sh

# split access

# http://lartc.org/howto/lartc.rpdb.multiple-links.html#AEN268

# http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.rpdb.multiple-links.html

# GoneVertical.org

# 1. One creates two additional routing tables, say T1 and T2. These are added in /etc/iproute2/rt_tables. Then you set up routing in these tables as follows:

# echo 1 T1 >> /etc/iproute2/rt_tables

# echo 2 T2 >> /etc/iproute2/rt_tables

# interface

IF0=lo

IF1=eth0

IF2=eth1

# ips

IP1=192.168.12.100

IP2=192.168.10.100

# gateways

P1=192.168.12.1

P2=192.168.10.1

# ip network

P0_NET=0.0.0.0

P1_NET=192.168.12.0

P2_NET=192.168.10.0

#echo $IF0 $IF1 $IF2

#echo $IP1 $IP2

#echo $P0_NET $P1_NET $P2_NET

# create routing tables

ip route add $P1_NET dev $IF1 src $IP1 table T1

ip route add default via $P1 table T1

ip route add $P2_NET dev $IF2 src $IP2 table T2

ip route add default via $P2 table T2

# create routing for local requests

# not sure if i need this

#ip route add $P0_NET dev $IF0 table T1

#ip route add $P2_NET dev $IF2 table T1

#ip route add 127.0.0.0/8 dev lo table T1

#ip route add $P0_NET dev $IF0 table T2

#ip route add $P1_NET dev $IF1 table T2

#ip route add 127.0.0.0/8 dev lo table T2

# main routing table

ip route add $P1_NET dev $IF1 src $IP1

ip route add $P2_NET dev $IF2 src $IP2

# default route preference

ip route add default via $P1

# routing rules

ip rule add from $IP1 table T1

ip rule add from $IP2 table T2

Add more JVM memory in tomcat on ubuntu 9.10

2010-01-15T14:31:00.000-08:00

I tried to change CATALINA_OPTS AND JAVA_OPTS with no success in increasing memory for my tomcat6 on ubuntu 9.10. I tried in /etc/enviroment, I tried export CATALINA_OPTS AND JAVA_OPTS="-xMX1024M" with no success. But I found you can edit the init.d/tomcat6 and works great.

- permgen size is the application size, or how much memory you need for the application

- xmx max memory size that will be needed for operation of the application

#change the attributes in: /etc/init.d/tomcat6

if [ -z "$JAVA_OPTS" ]; then

JAVA_OPTS="-Djava.awt.headless=true -Xmx1024m -XX:MaxPermSize=256m"

Share Libraries with All Servlet Application in Tomcat6

2009-11-27T18:09:00.000-08:00

When including jar libraries with Tomcat. Don't double include and do it in the right directory.

Every time I want to share a library with all Tomcat6 applications on Ubuntu 8.10+ stick the library in : /usr/share/tomcat6/lib

For instance I am using this library: /usr/share/tomcat6/lib/RXTXcomm.jar -> /usr/share/java/RXTXcomm.jar

For some reason, when I stick the library in /var/lib/tomcat6/lib has no include value. And if you stick the libraries in both it can create a problem, especially with RXTXcomm. So whats the deal, do both directories include, and what order, and does that matter?

I have been frustrated with the fact there are two locations to stick libraries for tomcat. I'll have to find why later. Not enough time to discover why now.

Teach your Kids to Code

2009-06-28T21:54:00.001-07:00

Now this is really cool and it works great on Ubuntu 9.04.

Scratch - http://scratch.mit.edu/ Teach your kids to program.

Check it out, my kids really like doing stuff in this program.

Run Java via Shell Using GWT Classes

2009-06-27T13:33:00.000-07:00

When I want to cron a process in the background using my gwt application classes I do something like this:


#!/bin/sh

BASE="/home/design/workspace/GV/war/WEB-INF";

# classes
MYCP="$BASE/classes:"

# jars
MYCP="$MYCP$BASE/lib:"
MYCP="$MYCP$BASE/lib/commons-codec-1.3.jar:"
MYCP="$MYCP$BASE/lib/commons-lang-2.4.jar:"
MYCP="$MYCP$BASE/lib/gwt-servlet.jar:"
MYCP="$MYCP$BASE/lib/log4j-1.2.15.jar:"
MYCP="$MYCP$BASE/lib/mail.jar:"
MYCP="$MYCP$BASE/lib/mysql-connector-java-5.1.5-bin.jar"

# debug output
echo "debug classes: "
echo $MYCP
echo ""

# run the java
#java -cp $MYCP com.gonevertical.test.Run_Test_DB_Conn;
java -cp $MYCP com.gonevertical.test.Run_Test_Email;
exit;

Change Ubuntu TimeZone

2009-06-27T12:23:00.000-07:00

I had a problem with Eclipse timezone looking for /usr/lib/jvm/ia32-java-6-sun-1.6.0.13/jre/lib/zi/Canada/Pacific. I selected Vancouver instead of Las Angeles.

I changed the time zone for my system by doing this:


# change ubuntu timezone
dpkg-reconfigure tzdata

MySql 6.11 Compile on Ubuntu 9.04

2009-06-24T20:54:00.000-07:00

MySql install notes on Ubuntu 9.04
If you use this, you may have to read in between the lines here and there. I wasn't able to get the icc already compiled version to work b/c of a syntax error with my_print_defaults. But I compiled it from source and its working perfect. The sub-query system is so much faster, its unbelievable. I recommend using the mysql6-falcon alpha linux source install.

Compile Mysql6-falcon for ubuntu 9.04.


# http://www.howtoforge.com/mysql5_debian_sarge
# http://mapopa.blogspot.com/2009/03/compiling-mysql6.html

# get mysql source

# copy source - compile from this - do this incase you mess up
cp -R mysql-6.0.10-alpha mysql6

cd ./mysql6

sudo apt-get install build-essential
sudo apt-get install lib64ncurses5 libncurses5 libncurses5-dbg libncurses-dev
sudo apt-get install autoconf libtool

# auto configure
autoreconf -vif

# configure options
#./configure --help

#it was weird it stuck some stuff in the data directory that probably shouldn't be there but is fixable afterwords. I moved it to basedir/share/mysql
./configure --prefix=/srv/mysql6 --datadir=/mnt/ssd/mysql
#./configure --prefix=/srv/mysql6_11 #upgrading to .11

make

sudo make install



# copied my.cnf to /etc/mysql/my.cnf and linked /etc/my.cnf
# set [mysqld] datadir=/srv/mysql6 basedir=/srv/ssd/mysql

# add user and group
# set perimissions where needed

# setup the default system
./bin/mysql_install_db --user=mysql --basedir=/srv/mysql6 --builddir=/home/branflake2267/downloads/mysql6b --datadir=/mnt/ssd/mysql
#./bin/mysql_install_db --user=mysql --basedir=/srv/mysql6_11 --builddir=/home/branflake2267/downloads/mysql-6.0.11-alpha

# add hash next to skip-federated - notate it

#add ./share/mysql/mysql.server to /etc/init.d/mysql.server

# change basedir and datadir in mysql.server

# link ./bin/mysql to /usr/bin/mysql

# start server
sudo /etc/init.d/mysql.server start

# mysql - no password exists for root localhost user yet
./bin/mysql -uroot -p


# add user
CREATE USER 'Branflake2267'@'localhost' IDENTIFIED BY 'password*7';
GRANT ALL PRIVILEGES ON *.* TO 'Branflake2267'@'localhost' WITH GRANT OPTION;
flush privileges;



#mysqls help./b
shell> groupadd mysql
shell> useradd -g mysql mysql
shell> gunzip <> cd mysql-VERSION
shell> ./configure --prefix=/usr/local/mysql
shell> make
shell> make install
shell> cp support-files/my-medium.cnf /etc/my.cnf
shell> cd /usr/local/mysql
shell> chown -R mysql .
shell> chgrp -R mysql .
shell> bin/mysql_install_db --user=mysql
shell> chown -R root .
shell> chown -R mysql var
shell> bin/mysqld_safe --user=mysql &

My data project wiki notes - http://code.google.com/p/parsecsv2sql/w/edit/mysql6_falcon_install