Development Days

Synology DS413j NAS

noreply@blogger.com (rdev02) — Wed, 28 Aug 2013 23:25:00 +0000

In my previous post I was rambling about Seagate BlackArmor and my sad experience with it. Apart from the sadness it also has my initial thoughts and requirements, so I won't duplicate them in this one. But, to keep consistent with the previous review I'll go with the same headings/experience areas. That being said - here we go!

Half a year has passed and I've been using Synology NAS for all this time. I ordered it from Amazon and off it came right in the delivery timeframes, ready for my experiments and scrutiny.

Packaging and build quality.

I couldn't be impressed more. Apple usually is a master in packaging their products, but these guys(synology) can stand up the competition. Everything in the box was perfectly packed a secured in a very little amount of space yet preserving a pleasant unpacking experience. One of the nice things is - when I was moving I managed to pack it with the box I saved just as if it was never unpackaged. Somehow they thought out the packaging in a way you use all the parts, perfectly seal the device and accessories and you have no junk left:) - KUDOS!

Build is a step up from the aforementioned NAS. All the parts that were plastic in seagate are metallic in synology. The drive bay has 2 solid fans and the drives themselves are secured inside the bay with bolts. Accessing the drives is also not that easy - you have to unscrew 4 bolts at the back, lower the back panel with fans and then get to the drives. This is not as convenient as seagate's open-dorr-click-and-take-away system, but how often do you actually want to replace drives inside your NAS? ;)

Other then that Synology is about the same size and noise, although heavier(due to metal parts). Move on to usage:)

UI and Software.

It comes with DSM 4.2. Usually, when you log into the web interface you would expect some semi-lame web app that'll link you to the features of your device. Not the case! Not even close! DSM web interface if one of the most(in not just the most) advanced web interfaces I've ever seen. It's all ajax and feels like a fine-tuned linux window manager. you literally have a web-like sort-of OS, working on your NAS giving you full control over it. Pages can be written about all the possibilities you have with the built in apps and there's even more with thirds party packages. Literally, everything I might have thought/dreamed about that I could combine/use NAS for - is already there. With a nice, multitasking, rich intuitive and responsive interface. I won't go into much details about the apps - there's just too many of them to cover and each and every of them deserves good comments. Those who want to learn more - I welcome you to use link I mentioned a couple lines up.

In 6 months I've never seen any app crash or misbehave. Neither did I see any hardware faults or reboots. Stable. Reliable.

The worx.

I did the same tests, that I had done for the previous NAS. They all showed good consistent results, so I went on started configuring and placing my data on it. For some reason new apps did not want to install to second partition, so I had to use the first one for all the apps and the second one for all other stuff which was not my initial plan. Whatever. I checked the read/write speeds and they were almost at what the specs were saying. Later I tested out 2 simultanious clients accessing 1080p video from the device, while one other device was listing pictures from it. They all showed decent performance and that is probably the biggest load I'll ever put on this device. So in general - it worx:).

The problems.

I did not have them:) There was one problem I had not being able to set up Glacier backup from my NAS after I moved. But luckily they do have support forums which helped me understand that my time was not synchronized. After I fixed the time issue everything worked like a charm.

Summary.

pros:

+ build quality

+ realiability

+ software

+ configurability

cons:

- pricey

Bonus.

It costs around 2.5 times more than seagate, but I'd say it's worth every penny I paid. In fact I was so happy with my purchase that I spread the word around and now a couple of my friends also have similar setups. And neither of them is unhappy. These boxes just work, so far.

*all pictures are referenced from www.synology.com

Seagate BlackArmor NAS 400.

noreply@blogger.com (rdev02) — Tue, 05 Feb 2013 07:20:00 +0000

So I decided that I'm sick and tired of plugging in and out usb drives. Be it my tiny 16GB daily thumb drive, or store-it-all 2TB external seagate - this is a pain... And to make sure everything is stored you have to "detach/remove" them safely all the time. And than eventually you need more storage(or will need) and expanding this kind of solution means more devices OR significantly more money. This is long and tedious so for quick and impatient there's a summary at the end.
Believe it or not more money is inevitable so I decided to try and reduce the number of devices - pile everything up in one place - shall it stay safe and robust and accessible at all times. NAS. I had very few requirements:

at least 4 bays to be able to play with RAID 5/6/10.
3.5 drive interface to keep them cheap
"quiet" feature - at this would stay in my room.
robust and reliable.

    I had a couple of Seagate drives before and overall my experience with them was good. Nowadays you can find complaints about pretty much any manufacturer and any product so I did not really pay attention to what others said - was looking for my set of requirements for the right price. And there it shines - perfect solution for all my needs and even a little bit more! Remote management, encryption, DDNS, configurable backups, all-you-can-imagine media sharing, internal downloader and more - sweet!
    So I went to Amazon and ordered one. It was just before new years so I received my order in a month - but that was not urgent: storage is not smth. that one should make haste with IMO.

Packaging and build quality.
    I was actually impressed - the thing was of manageable size, pretty compact and fitted into my nightstand lower deck. Loading this thing up with drives was super easy - it had a smart clip drive holding system. No installation - you can install some discovery program from CD but really - you do not need it. The only thing it does - it finds the IP of your NAS and forwards you to the browser.
Thus overall I was pleased with build quality: yes it was not top notch aluminum\nickel but mostly plastic. Yes it was a bit rough on the edges. Yes the green screen was smth both enjoying and amusing, but for the price it was absolutely great and matched my needs in full. I was not going to use this in hazardous environments and I do not have little kids so that I would really worry about the build and plastic. The chip and software is what mattered....

UI and software

    UI was slow. As a developer I'm used to wait times. I'm patient. But even for me it was tad slow. It was intuitive though. I quickly found my way around without the manual but then the limitations kicked in
- you have have a usb stick plugged in to the first usb slot(and only 1st, which front btw) if you want encryption
- you have to have entire volume encrypted
- you have to wait all surface scan and zero-out if you change anything about the volumes(and it lasts about 8 hours on my 2TB drives)
- you cannot have a read only share(bummer!)
- you have to do 1 thing at a time or else everything you configured before may get messed up and only hard reset followed by total zero-out\check of drives(8 hours again) will save you
- and probably many other I did not discover due to I did not plan to use that functionality at all.
BUT I went through all this, patiently, believing that once I get this linux set up and working - it'll be rock solid and serve me well for many years. That is how it is usually done with linux - you get it for cheep, but you pay with your time. Your configuration is a pain, but after all it worx (period)

The worx.
    So I wanted to make sure that I can rely on this thing once I configured it. So I wrote a little program, that would generate a set of files, get their md5's, measure speed and have some brain to verify that files later. After all, the only way to understand if the drive would work for you - is to fill it up with data and see how it performs on parallel reads/writes. I'll post the code soon to github in case anyone would ever need it. I will.
    I started it up on 2 computers: one writing to encrypted partition, other - to public one(both raid 10) and a few days later I had my NAS populated with dummy, but recorded data. Thus:

write speed turned out to be much slower for the encrypted volume than for the public one - around 2-5 Mb/s VS 5-7 for the unencrypted.
read speed was good for both 10Mb/s for encrypted and 15-20 for unencrypted volume.

That was more than satisfying for me. I do not really need fast write, and that yields enough speed for one HD movie stream. Next were the problems tests that I was planning to execute - like remove a drive and see how NAS tackles it. But I did not have to do this problems arrived by themselves...

The problems.
Another morning I woke up and found out that NAS reports one drive to be failed. No big deal - take it away and put a fresh new one. But I wanted to test out the speed and file integrity before doing this. So I launched my program again in test mode and an hour later my second drive was reported to be failing. Now I was in doubt, because the filesystem was still available. 2 hard drives from one branch of raid 10 fail and FS worx...how? Well. I restarted the NAS and drives showed up as good but both volumes showed degraded. Then 2 drives rendered failed and public volume remained degraded. Let me remind you that both volumes were based on same drives with same raid setting. Next the unpredictable happened - all data from public volume disappeared. Volume was mountable, however displaying 90% full and empty at the same time. So I decided to take both of them and inspect in the external enclosure. The drives indeed turned out to have bad sectors on them. Fedora immediately told me that I want to replace those drives. BUT they were readable and fine. a couple of bad sectors could have been easily remapped afaik....
So I decided to give this NAS the last try to restore the disks with the spare drive I bought just for this purpose. Actually I decided to give NAS a chance to do what it was essentially bought for - tackle the hardware problem. But after I inserted the disks back(nothing was done to them - I just ran disk utils and gparted, I did not repair or remap anything. and yes - I labeled all drives before putting them into nas). - NAS told me that "Problem....Data is recoverable. Please contact support". and pay money, probably... That was it for me - filed a return on amazon and the next morning it was gone.

SUMMARY
pros:
+ relatively good pricing(~200$)
+ nice form factor(small)
+ nice set of standard features out of the box(ddns, media service, downloader, raids)
+ enough speed for small office/home experiences.
cons:
- not reliable(easy to corrupt internal FS, unable to restore/maintain data)
- clunky, slow and strange software(not only UI). IMO
- apparently has problems with concurrent usage. AFAIU
- not able to address basic hdd hardware problems(remap or even test for bad sectors - only does a S.M.A.R.T test and even that unreliably).
- virtual support(The forum I visited looks like gives answers only to satisfactory questions or those you can find out by yourself.)

Next stop: Synology. To be continued...

*all pictures are referenced from seagate.com

Javascript performance tests

noreply@blogger.com (rdev02) — Thu, 02 Aug 2012 22:35:00 +0000

I'm dealing a lot with javascript nowadays and obviously I have to take performance into account. One way of doing this is writing a bunch of test code and record the time for your scripts, but what if you wanted to quickly test out some approaches and do not really want to invest a lot into them? Today I was pointed to a solution: http://jsperf.com/. This awesome tool does not just give you the framework to easily test out javascript but also provides and already created library of tests other people created when investigating their performance issues. Sweet! I finally have answer to this question. And many more :)
It also quite clear that when dealing with math caching arrays is a must.

WP7 Mango dive

noreply@blogger.com (rdev02) — Sun, 26 Feb 2012 15:15:00 +0000

It has been a few month since I started mangling with WP7 platform. The more I learnt about it from the inside the more I wanted to buy new real WP7 phone. Things changed slightly after I have done my very first app(didn't make it to the app store as I found plenty of similar ones already there and mine was not adding much :() but still as of now I would really be interested in posessing a WP7 phone and giving it a solid try. But let me start from the beginning...

Learning

As my primary learning resource I used 2 jumpstart sessions : WP7 jumpstart and Mango jumpstart. If you have the time to invest into watching those I highly recommend them as these are very good sources of information to study the platform fast. Especially keeping in mind that all the sessions there are accompanied by code examples which are also easily found on codeplex. The code especially nice due to the availability of the before-after versions of demo applications so that it is easy to see the transition from incomplete/incorrect state to the solid code.

Environment

Windows phone SDK 7.1 integrates seamlessly into VS2010 and makes a very pleasant environment to work with. The emulator gives you a lot of power when debugging your app. Still I missed the camera support(it is there but I kinda didn't like it) and the FS support. It's not so easy to look at things located on the Isolated storage as well as on the SD card. Otherwise there's everything you'd want from the emulator, including recording gyroscope and accelerometer changes and playing them afterwards again and again.

Development

Developing WP7 is tricky yet interesting. You're using almost all of what .NET can offer with certain limitations(e.g. AsyncCallback is almost useless) seamlessly and easily. There's a lot you can customise in the standard controls collection. Still some common things which I would expect to be just working are in fact not. As an example I cannot grasp why would one want the Popup class if it still has to be injected into the visual tree manually... Why not use the panel with visibility instead? It's a lot more natural to me than injecting the Popup somewhere at runtime... Another interesting discovery for me was the fact that IsolatedStorage is not available in the OnNavigatedTo event. I guess MS guys give us a hint here that we should probably use only the StateStorage as it is much faster, and then use progress indicators to load heavy stuff after the page is loaded... but then why would I bother using StateStorage at all? Just put the "loading..." progress on top in the first place and get all the stuff from the IsolatedStorage where it is grouped together nicely and not scattered between 2 storage places(not to mention the fact that when storing to StateStorage you are highly recommended to store things to the IsolatedStorage as well as the state might not be available if the application was killed in the meantime..)
Nevertheless there's a lot to admire here as well... Among other things I liked the controls customisation ability and overlaying concept best. WYSIWYG designing is really helping out here cutting redeployment and testing time. Debugging the real phone is easy too.

Bottom line

WP7 development surely has its own intricacies one might not like, but for me this is what makes mobile development thrilling. You can discover completely new ways of using the very old concepts/controls and rethink your strategies towards certain solutions. Mobile changes the way you treat data and UI and Metro-powered mobile development really broadens ones vision on how the data can be processed and delivered to utilise the customer needs.

MySql performance. 5.1 VS 5.5 VS 5.6 (dev)

noreply@blogger.com (rdev02) — Wed, 14 Sep 2011 06:50:00 +0000

Recently had to do an investigation about MySql server performance. So what I basically did was: created a winXP wmWare image, created a script for generating populating sql; installed db server version on that machine and kept track of times while executing sql statements; reverted to the original virgin machine state; start again. The versions I tested on were MySql community server 5.1, 5.5 and the development verion of 5.6. Here is the chart that represents my findings[the smaller the better]:

As a quick summary - interestingly enough I do not see any speed improvement since 5.1:) there is one between 5.5 and 5.6 but it's nowhere near the 5.1.
Now just a quick overview of what I used to actually get this chart.
The testTable:
The queries:

Q1	select count(*) from test.testtable;
Q2	select count(*) from testtable;
Q3	SELECT * FROM testtable tst WHERE tst.id>1000 AND tst.id<10000;
Q4	SELECT COUNT(*) FROM testtable tst WHERE tst.id>1000 AND tst.id<10000;
Q5	SELECT * FROM testtable tst WHERE tst.testInt between 10000 AND 100000 ;
Q6	SELECT COUNT(*) FROM testtable tst WHERE tst.testInt between 10000 AND 100000 ;
Q7	SELECT * FROM testtable tst WHERE tst.testDouble between 0.1 AND 0.5 ;
Q8	SELECT COUNT(*) FROM testtable tst WHERE tst.testDouble between 0.1 AND 0.5 ;
Q9	SELECT * FROM testtable tst WHERE tst.testDouble <= 0.1 AND tst.testDouble > 0.5 ;
Q10	SELECT COUNT(*) FROM testtable tst WHERE tst.testDouble >= 0.1 AND tst.testDouble < 0.5 ;
Q11	SELECT * FROM testtable tst WHERE tst.testDate > '2005/08/04 07:00';
Q12	SELECT COUNT(*) FROM testtable tst WHERE tst.testDate > '2005/08/04 07:00';
Q13	SELECT * FROM testtable tst WHERE tst.testVarchar LIKE '%AAAAAA';
Q14	SELECT * FROM testtable tst WHERE tst.testVarchar LIKE '%AAAAAA' OR tst.testVarchar LIKE 'BBBBBB%';
Q15	SELECT * FROM testtable tst WHERE tst.testInt between 10000 AND 100000 AND tst.testDouble between 0.1 AND 0.5 AND tst.testDate > '2005/08/04 07:00' OR tst.id> 9000000
Q16	SELECT COUNT(*) FROM testtable tst WHERE tst.testInt between 10000 AND 100000 AND tst.testDouble between 0.1 AND 0.5 AND tst.testDate > '2005/08/04 07:00' OR tst.id> 9000000

Absolute times and machine spec can be found here.
Of course this is the developer machine setup and ran on one virtual machine, but still I thought that would be interesting for other to know. Ireally looking forward to making some more production-like tests now...
One more thing in case interested - the code snippet I used for generating the data:

public class CreateTestTableSql {

    private static final int QUERY_ROWS = 10 * 1000 * 1000;
    private static final int BATCH_SIZE = 10 * 1000;
    private static final boolean USE_BATCH = true;

    static Random r = new Random();
    static SimpleDateFormat sdf = new SimpleDateFormat("yyyy/MM/dd HH:mm");

    public static void main(String[] args) throws Exception {

        StringBuilder spotTradeSQLRow = new StringBuilder();


        File f = new File(USE_BATCH ? "batchmysql.sql" : "mysql.sql");
        if (f.exists()) {
            f.delete();
        }

        assert f.createNewFile();

        BufferedWriter out = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(f)));

        System.out.println("Start");
        int batchCounter = 0;
        for (int i = 0; i < QUERY_ROWS; i++) {
            if (USE_BATCH) {
                if (batchCounter == BATCH_SIZE) {
                    spotTradeSQLRow.append(";\n");
                    batchCounter = 0;
                }
                if (batchCounter == 0) {
                    spotTradeSQLRow.append("INSERT INTO test.testtable VALUES(");
                } else {
                    spotTradeSQLRow.append(",(");
                }
            } else {
                spotTradeSQLRow.append("INSERT INTO test.testtable VALUES(");
            }
            spotTradeSQLRow.append(i + ","); //id
            spotTradeSQLRow.append((r.nextBoolean() ?( "'" +getRandomString() + "'"): "NULL") + ","); //testVarchar
            spotTradeSQLRow.append((r.nextBoolean() ? r.nextInt() : "NULL") + ","); //testInt
            spotTradeSQLRow.append((r.nextBoolean() ? r.nextDouble() : "NULL") + ","); //testInt
            spotTradeSQLRow.append((r.nextBoolean() ? ("'" + getRandomDate() + "'") : "NULL") +
              ")"); //testInt
            if (!USE_BATCH) {
                spotTradeSQLRow.append(";");
            }
            out.write(spotTradeSQLRow.toString());
            out.write("\n");
            spotTradeSQLRow.delete(0, spotTradeSQLRow.length());
            batchCounter++;

        }
        if (USE_BATCH) {
           out.write(";");
        }

        out.close();
        System.out.println("end");

    }

    private static String getRandomDate() {
        int rand = r.nextInt(10);
        rand *= (r.nextBoolean() ? -1 : 1);
        final Calendar cal = Calendar.getInstance();
        cal.add(Calendar.YEAR, rand);
        cal.add(Calendar.DAY_OF_YEAR, rand);
        cal.add(Calendar.HOUR_OF_DAY, rand);

        return sdf.format(cal.getTime());

    }

    private static String getRandomString() {
        final int rand = r.nextInt(10);
        if (rand < 3) {
            return "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
        } else if (rand >= 3 && rand < 6) {
            return "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB";
        } else {
            return "CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC";
        }
    }
}

java.net.MalformedURLException: no protocol: and]

noreply@blogger.com (rdev02) — Tue, 06 Sep 2011 08:06:00 +0000

Had a problem with injecting JMSFactory for JBoss MQ using spring(for Camel :)). The problem had the next StackTrace:

Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.net.MalformedURLException: no protocol: and]
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:780)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at org.springframework.jndi.JndiTemplate$1.doInContext(JndiTemplate.java:154)

I was sure I configured it right as I have

<jee:jndi-lookup id="JBossMQ" jndi-name="XAConnectionFactory" proxy-interface="javax.jms.ConnectionFactory"
                     environment-ref="jndiProps" resource-ref="true"/>
    <util:properties id="jndiProps" location="META-INF/spring/jndi.properties"/>

And with those jndi.properties I was sure I have to get a refference as I could do that using the InitialContext by hand". Turned out that it was maven that made things go wrong. moving the repository to a path with no spaces solved the issue(e.g. c:\maven\repo).
Hope this helps someone.

Plymouth fix for Ubuntu 11.04

noreply@blogger.com (rdev02) — Tue, 28 Jun 2011 20:22:00 +0000

If you're in the same trouble as I was with black screen during boot(plymouth has the theme installed and even displays it during shutdown) then there is a small thing that has to be edited in order to get rid of this annoyance. Go to

/etc/grub.d/10_linux

and remove the handoff=7 parameter. Inspired by this and also this(pretty nice tool for configuring plymouth for dummies).

Hackathon in Lviv

noreply@blogger.com (rdev02) — Mon, 20 Jun 2011 08:34:00 +0000

Great news! Best Hackathon traditions to invade Lviv. DOU is planning a hackathon @ June 9 — 10. Location is not set yet, but people are welcome to join now, so that we know there'll be enough people comming. More details here. If anyone interested to create a team - reach me via comments here or on IMs...

IT ramblings

noreply@blogger.com (rdev02) — Sun, 19 Jun 2011 21:16:00 +0000

There is going to be an interesting event soon in Lviv both for those who are in IT and those who love to use and improve their english. It's called IT Ramblings. Follow the link and check it out. I'm definitely coming, join in ;)

New quest from Microsoft

noreply@blogger.com (rdev02) — Sat, 04 Jun 2011 07:54:00 +0000

33+k UAH for those lucky to win the lottery among the ones most skilled in MS puzzle solving:) More here

j2ee sdk install hanging fix.

noreply@blogger.com (rdev02) — Wed, 01 Jun 2011 18:41:00 +0000

Spent about 3 hours trying to install the latest java ee sdk on my natty narwhal with no luck. It usually hang with 45% and 5seconds left forever. Might be I'm a bad googler but turned out that there is a bug in the installer which prevents the installation from continuing. The workaround is to disable the network connections before the installation begins(disabling while it is hanging did not work for me). You won't have the Update Tool Bootstrap tool configured but still you'll have the sdk installed and ready for your tests/work.

Avahi and "network service discovery disabled"

noreply@blogger.com (rdev02) — Fri, 20 May 2011 21:07:00 +0000

Had a real problem with my wifi after moving to Ubuntu from win -> my download speed was dead slow with ~2Mb/s. Tried various things including this this and finally this. The last one did the trick. It was Avahi which spoiled everything. The thing that led me to Avahi was this irritating message popping out each time I connected:

Having this changed in

AVAHI_DAEMON_DETECT_LOCAL=1 to AVAHI_DAEMON_DETECT_LOCAL=0 in
/etc/default/avahi-daemon

Got me up to 22Mb/s after reboot. We'll see if I could squeeze more from my cheap router flashing it with some different firmware, but for now I'm quite happy to gain a 10x speed improvement :)

Tabkit => Vertical tabs

noreply@blogger.com (rdev02) — Thu, 12 May 2011 12:34:00 +0000

A post for those who, once tried, cannot live without vertical tabs in FF.
I used to have TabKit installed but this unfortunately does not support FF 4. So I even was very reluctant to switch to the new version as having my browsing (or should I say research ) organized, is one of the crucial parts of my productivity. But turns out there is Vertical Tabs which does almost the same thing. It's not that polished as TabKit but it "does the thing" and so far I'm happy with it.

UPD: Found the VertTabBar addon also while writing this post... might suite someone more.

Lift + ajax. The "proper" way of doing presentations.

noreply@blogger.com (rdev02) — Sun, 24 Apr 2011 10:43:00 +0000

24-th seconds rules...

This mostly ru-zone compliant :)

Java to BlazeDS NAN in double values.

noreply@blogger.com (rdev02) — Fri, 22 Apr 2011 09:07:00 +0000

Internet is basically full of subj but vice-versa(BlazeDS to java). So the problem I had was that when the object is "translated" from java via BlazeDS to flex application it had NAN's in the double fields. The transfer object was a simple POJO I passed via the interface. First thing I checked were the getters - all present and valid. I took quite a lot of time till we figured out that it was the missing setters which made the code malfunction. The problem was that on some objects I declared an immutable interface and implemented it in a DTO object. I did not add setters where I never planned to use them. This turned out to be the crucial mistake: Always be sure to have both getters and setters in your DTO. It doesn't matter if you pass it via immutable interface or not - the underlying object has to have both methods on the field otherwise we get NAN.

maven-compiler-plugin causing eclipse projects to have j2se 1.4 instead of configured one

noreply@blogger.com (rdev02) — Thu, 21 Apr 2011 10:52:00 +0000

Today was investigating the warnings around some of my eclipse maven projects and noticed that I have a few

There are no JREs installed in the workspace that are strictly compatible with this environment

messages. It took some time to figure out that it was actually the maven-compiler-plugin causing the the J2SE version to be picked up correctly. If I changed the packaging of the maven project to jar everything was generated fine. After some searching it turned out there is a bug with the generation tool m2eclipse I use. As described here the bug was fixed TODAY by removing the java nature from the project, which in turn removes the .classpath necessity which in turns removes the warnings. Few errors less - great work m2eclipse :).

String.split() VS String.startsWith() VS Regex simple performance comparison.

noreply@blogger.com (rdev02) — Tue, 19 Apr 2011 14:45:00 +0000

Now really, what would be faster/more reliable for a typical not so complicated and not over-simple case? Yes the one we tend to deal with in our day-to-day work? For those impatient benchmarks are here.Reading a 100/1000-line file is not fun and I really think you shouldn't bother choosing if you're dealing with such volumes of data. But if you have to parse 320k-500k records lines, which, in fact, is not that large - 320k of CSV data is ~ 12MB performance is considerable. So let's start with data definition. Let's assume we have data like

1812332,"String1","String2","String3",16,
1812332,"String1","String2","String3",16,
1812332,"String1","String2, extra string","String3",16,

Also we are going to start with the abstract "blank" for csv reading like:

public abstract class CsvStreamReader implements Iterator<CsvData> {
    protected BufferedReader reader;

    /**
     * creates an new {@code CsvStreamReader}. Caller is responsible for stream management.
     *
     */
    public CsvStreamReader() {
    }

    public abstract CsvData next();

    public boolean hasNext() {
        try {
            return reader.ready();
        } catch (IOException e) {
            e.printStackTrace();
        }
        return false;
    }

    @Deprecated
    public void remove() {
        // not used
    }

    public void setInputStream(InputStream inputStream){
        reader = new BufferedReader(new InputStreamReader(inputStream));
    }
}

And a simple data-holder like:

public class CsvData {
    private String[] data;

    public CsvData(String... data) {
        this.data = new String[data.length];
        System.arraycopy(data, 0, this.data, 0, data.length);
    }

    public String getColumn(int index) {
        return data[index];
    }

    public int getColumnCount() {
        return data.length;
    }

    @Override
    public String toString() {
        return "CsvData{" +
          "data=" + (data == null ? null : Arrays.asList(data)) +
          '}';
    }
}

Now the first and obvious way to read and parse the CSV would be smth like:

public class SplitBasedReader extends CsvStreamReader {

    public CsvData next() {
        try {
            String line = reader.readLine();
            return new CsvData(line.split(","));
        } catch (IOException e) {
            e.printStackTrace();
        }
        return null;
    }
}

This looks fairly good with one lil' problem: it doesn't work good. If we take the last line with "String2, extra string" we see that the split would play a bad trick with us breaking the data consistency. A quick fix would suggest smth like:

public class StringBasedReader extends CsvStreamReader {
    private static final String QUOTE = "\"";

    @Override
    public CsvData next() {
        try {
            List<String> data = new ArrayList<String>();
            final String line = reader.readLine();
            int commaStart = 0;
            int commaEnd = 0;
            StringBuffer possibleToken = new StringBuffer();

            while (commaEnd >= 0) {
                commaEnd = line.indexOf(',', commaStart + 1);
                if (commaEnd < 0) {
                    break;
                }
                final String possibleColumn = line.substring(commaStart, commaEnd);
                possibleToken.append(possibleColumn);
                commaStart = commaEnd + 1;

                final int length = possibleToken.length();

                boolean columnIsOpen = (possibleColumn.startsWith(QUOTE)) && (!(possibleColumn.endsWith(QUOTE)));
                if (!columnIsOpen) {
                    data.add(possibleToken.toString());
                    possibleToken.delete(0, length);
                }
            }

            if (possibleToken.length() != 0) {
                data.add(possibleToken.toString());
            }

            return new CsvData(data.toArray(new String[]{}));
        } catch (IOException ioex) {
            ioex.printStackTrace();
        }
        return null;
    }
}

Well yeah, but having string.startsWith checking on String all this takes int account variable length which in fact we do not care about... sounds wasteful. Let's improve it.

public class StringBasedReaderImproved extends CsvStreamReader {
    private static final char QUOTE = '"';

    @Override
    public CsvData next() {
        try {
            List<String> data = new ArrayList<String>();
            final String line = reader.readLine();
            int commaStart = 0;
            int commaEnd = 0;
            StringBuffer possibleToken = new StringBuffer();

            while (commaEnd >= 0) {
                commaEnd = line.indexOf(',', commaStart + 1);
                if (commaEnd < 0) {
                    break;
                }
                possibleToken.append(line.substring(commaStart, commaEnd));
                commaStart = commaEnd + 1;

                final int length = possibleToken.length();
                boolean columnIsOpen = (possibleToken.charAt(0) == QUOTE) && (!(possibleToken.charAt(length - 1) == QUOTE));
                if (!columnIsOpen) {
                    data.add(possibleToken.toString());
                    possibleToken.delete(0, length);
                }
            }

            if (possibleToken.length() != 0) {
                data.add(possibleToken.toString());
            }

            return new CsvData(data.toArray(new String[]{}));
        } catch (IOException ioex) {
            ioex.printStackTrace();
        }
        return null;
    }
}

OKAY! now it a char and StringBuffer. Looks like we've done a great job! Awesome.... although a bit magical. It's not rocket science but still it's not pleasant to read. What else might we suggest? Regex.

public class RegexBasedReader extends CsvStreamReader {
    private static final String RE_ANY_INSIDE_QUOTES = "\"([^\"]*)\"";
    private static final String RE_ANY_WITHOUT_COMMAS = "([^,]*)";
    private static final String RE_ANY = "(" + RE_ANY_INSIDE_QUOTES + "|" + RE_ANY_WITHOUT_COMMAS + ")";
    private static final String RE_6_DIGIT_GROUP = "(\\d{6})";

    private static final String CSV_REGEX = "^" + RE_6_DIGIT_GROUP + "," + RE_ANY + "," + RE_ANY + "," + RE_ANY + "," + RE_ANY + ",$";


    protected final Pattern pattern;

    /**
     * {@inheritDoc}
     */
    public RegexBasedReader() {
        pattern = Pattern.compile(CSV_REGEX);
    }

    public CsvData next() {
        try {
            List<String> data = new ArrayList<String>();
            final String line = reader.readLine();
            final Matcher matcher = pattern.matcher(line);
            matchDataToColumns(data, matcher);

            return new CsvData(data.toArray(new String[]{}));
        } catch (IOException e) {
            e.printStackTrace();
        }
        return null;
    }

    /**
     * @param data
     * @param matcher
     */
    protected void matchDataToColumns(List<String> data, Matcher matcher) {
        if (matcher.matches()) {
            final int groupCount = matcher.groupCount();
            assert groupCount == 13 : "Matched an invalid string - regex should be revised";

            final String data1 = matcher.group(1);
            assert data1 != null : "data1 should always be present";
            data.add(data1);

            String data2 = matcher.group(3);
            if (data2 == null) {
                data2 = matcher.group(4);
            }
            assert data2 != null : "data2 should somehow be present";
            data.add(data2);

            String data3 = matcher.group(6);
            if (data3 == null) {
                data3 = matcher.group(7);
            }
            assert data3 != null : "data3 should somehow be present";
            data.add(data3);

            String data4 = matcher.group(9);
            if (data4 == null) {
                data4 = matcher.group(10);
            }
            assert data4 != null : "data4 should somehow be present";
            data.add(data4);

            String data5 = matcher.group(12);
            if (data5 == null) {
                data5 = matcher.group(13);
            }
            assert data5 != null : "data5 should somehow be present";
            data.add(data5);
        }
    }
}

That looks like Chuck Norris would be proud. Strong and reliable checking, precompiled patern, groups of data, many cases handled same way... room for extension etc.

Now lets look at the benchmarks reading over 500k lines:

public class Test2 {
    static List times = new ArrayList(4);

    public static void main(String[] args) throws IOException {

        read320KRecords(new SplitBasedReader());
        read320KRecords(new StringBasedReader());
        read320KRecords(new StringBasedReaderImproved());
        read320KRecords(new RegexBasedReader());
}

    private static void read320KRecords(CsvStreamReader reader) throws IOException {
        InputStream inputStream = Test2.class.getResourceAsStream("/CardInfo.csv");
        reader.setInputStream(inputStream);
        long timeStart = System.currentTimeMillis();
        while (reader.hasNext()) {
            CsvData csvData = reader.next();
        }
        long timeEnd = System.currentTimeMillis();
        times.add(reader.toString() + "  time: " + (new DecimalFormat("#.####")).format(new Double(
          (timeEnd - timeStart) / 1000.0)) + " sec.");
        inputStream.close();
    }
}

Output run from IntellyJ Idea:

	SplitBasedReader	StringBasedReader	StringBasedReaderImproved	RegexBasedReader
Run 1	1.468	0.968	0.984	1.593
Run 2	1.562	0.953	0.985	1.594
Run 3	1.531	1.031	1.047	1.625

In a chart:

But wait - there's more :)
Now let's see if we can get even more results out of this. In the previous benchmark runs we did nothing with the CsvData we obtained, but that is not the case in real world right? now consider we add a single

System.out.println("csvData = " + csvData);

and this would always result in "csvData = myCSvData". Running this in IDEA gives next results:

SplitBasedReader time: 14.634 sec.
StringBasedReader time: 13.086 sec
StringBasedReaderImproved time: 12.118 sec
RegexBasedReader time: 15.244 sec

SAME but in console:

SplitBasedReader time: 138.025 sec.
StringBasedReader time: 136.337 sec.
StringBasedReaderImproved time: 130.975 sec.
RegexBasedReader time: 131.146 sec.

So we see that the buffers and IO impact is immense. And one more test. Lets assume out toString is smth. a little meaningful like

"CsvData{" + data[0] + data[1] + data[2];

Now the times(idea env execution) are:

SplitBasedReader time: 21.655 sec.
StringBasedReader time: 9.772 sec.
StringBasedReaderImproved time: 9.256 sec.
RegexBasedReader time: 12.461 sec.

Bottom line:

While robust and effective Regex is not the fastest thing in the world;
working with relatively not-so-complex data is faster with "smaller" API's;
; IO is a bottleneck, not the parsing, usually.

Nokia OVI suite & sms extract.

noreply@blogger.com (rdev02) — Sun, 26 Dec 2010 23:00:00 +0000

There was a nice program for syncing phone and PC called Nokia PC Suite. While still available on some dusty hard drives it is now replaced with something called Nokia Ovi Suite. As I mentioned before - nice piece of software if we forget about all the "store" thing. But it has one important for me thing missing - sms extracting mechanism. You can't just take all of your sms and store to the file you want. It will either go into backup file or will be stored on you PC. I was really disappointed to learn that so I tried to find where exactly is that info stored on my PC. First search revealed that they use the SQLite db for storing all data located in
"c:\Users\[username]\AppData\Local\Nokia\Nokia Data Store\DataBase\MDataStore.db3"
The DB is not in any way protected or encrypted so I felt free to dig into my own data. Downloaded a splendid SQLiteManager plugin and started seeking my sms's. Didn't take long to find. Whereas the DB structure might be a subject of argue I'll post some code I used to get the info I needed out of the DB:

 class Program
    {
        static void Main(string[] args)
        {
            if (args == null || args.Length < 1)
            {
                Console.Out.WriteLine("Please specify db3 file path as an argument");
                return;
            }
            string baseName = args[0];
            StreamWriter writer = null;
            try
            {
                SQLiteFactory factory = (SQLiteFactory)DbProviderFactories.GetFactory("System.Data.SQLite");
                using (SQLiteConnection connection = (SQLiteConnection)factory.CreateConnection())
                {
                    connection.ConnectionString = "Data Source = " + baseName;
                    connection.Open();
                    Console.WriteLine("Connection opened..");
                    using (SQLiteCommand command = new SQLiteCommand(connection))
                    {
                        command.CommandText = @"SELECT MessageEx.Sender, MessageEx.Text, MessageEx.SentReceivedTimestamp, Item.CreationDate FROM MessageEx INNER JOIN Item ON MessageEx.GUID = Item.GUID ORDER BY MessageEx.SentReceivedTimestamp;";
                        var reader = command.ExecuteReader();
                        FileInfo outF = new FileInfo("result" + DateTime.Now.Date.ToString("dd_mm_yyyy") + ".txt");
                        writer = new StreamWriter(outF.Open(FileMode.Create));
                        writer.WriteLine("Sender, Text, Timestamp, Created");
                        while (reader.Read())
                        {
                            writer.WriteLine(String.Format("{0}, {1}, {2}, {3}", reader.GetString(0), reader.GetString(1), reader.GetDouble(2), reader.GetDouble(3)));
                            Console.Write(".");
                        }
                        Console.WriteLine("Done! Press eny Key to continue.");
                    }
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine("OOPS:" + ex.Message);
            }
            finally {
                if (writer != null)
                    writer.Close();
            }

        }
    }

This is pretty much all needed to get a well-formed CSV. There is still some work left to get the date from Real to date/time, but I thought it can wait until I really need that info. CSV is easily processed:). Some of this sourcecode is borrowed from here. Personally wouldn't ever dig into such software peripherals, but they literally forced me to, not providing the info I actually poses.
Hope that helps someone who needs data synchronized from the phone - the DB is a pretty much comprehensible structure and tables and with the mentioned above plugin and a lil' SQL it's quite easy to get your data even without programming.

.NET WMI. Or how do I get all that info?

noreply@blogger.com (rdev02) — Wed, 13 Oct 2010 15:06:00 +0000

There's a ton of Win properties that your program might want to access/possess/transform and there's a single place to find them: WMI queries. Though some of the properties might be scattered through different classes like Environment and *Info and even *Stream and more(and I believe accessing them through that places is significantly cheaper and faster) there's one sigle silver bullet to get them all via

Select * From [win_category]

The problem is there are tons of "win_category]' and * values. Luckily there is a WMICodeCreator. Actually the tool is pretty self explanatory - just a few tips:

instead of using Select * you can use Select [ActualPropertyNameYouWant] which I believe is better
instead of tedious browsing through combobox-ui use the text search on the code file supplied in the package

All Win properties nicely, fast and reliably right into your BL.
do-not-forget-to-use-post

.NET tab control.

noreply@blogger.com (rdev02) — Fri, 08 Oct 2010 19:39:00 +0000

Consider some UI that you would like to present as tabs. And you also know that all these tabs would have a considerate amount of code in common. There are some thoughts here but those are more about visual representation. Also creating usercontrols I believe is not an option due to it's not expected to have more tah one tab of a kind. The solution has to both utilize the VS designer(otherwise tab creation would take considerably more time) and keep the code separate from the tabcontrol - holder form. So that you have all functionality neatly separated.
Haven't found any except here which looks kinda OK but lots of custom code. Would like to have just tabs used just in normal tabcontrol...
Will try to find/develop one over weeks or find a good reason not to do it.(know WPF solves it with templates. requirements - plain old WinFroms) Any comments are welcome.

org.springframework.oxm.jaxb.Jaxb2Marshaller

noreply@blogger.com (rdev02) — Wed, 29 Sep 2010 15:45:00 +0000

There is a nice tutorial here on how to create spring web services using jax-b. Works just fine but it's hell lots of issues you would run into while making it actually work. The most frustrating for me as of today was the absence of Jaxb2Marshaller. in the std "spring-ws-core" artifact package. It's just not there and noone would tell you. Only Jaxb1Marshaller is. After an hour googling found this (BTW very simple and straightforward) post where spotted the "spring-oxm-tiger" artifact. This is your saver if you you're tearing your hair out seeking for the beloved interface. Maybe it's just late today on the other hand...

MCPD acquired.

noreply@blogger.com (rdev02) — Sun, 28 Mar 2010 20:06:00 +0000

   Finally. Now frankly - preparing for MS exams while doing die-hard java is not a funny thing to do. NEVER practice such a combination...
   About the exam - it was better then the previous 2: it obeyed no rules:). What I mean here is that I could have skipped the preparation kit completely! Yes there were a few questions related to the book's content, BUT reading it is completely optional IMHO. Even more - you'll be fine if you just skim through the chapter headings and introductions.
   About the questions - common sense required. They are all about common sense. If you developed the apps and know how things are working there - answers are obvious. Of course there are a few "do it the MS way" questions, but most of 'em - common sense and logic job. So my advise - schedule for Tuesday morning.
   And the last one - free certificates are over - you'll have to order one now if you want. Looks like taking those exams has to be lightning fast, but when would you do work then? So that's where I'm quitting with MS certification until I get .NET related stuff to do at work again. Heading to SCJP now:) Wish me luck...

907: Invalid cod and other BlackBerry OTA problems.

noreply@blogger.com (rdev02) — Wed, 10 Mar 2010 09:37:00 +0000

Most of us, who developed any BlackBerry app that had to be shipped OTA had seen this "descriptive" error. It might be anything from invalid jad to signing problems. The one I'm going to describe here is when the size of the cod file is more than 64K.
Finding out if you have this problem is fairly easy: just look into the final cod and see if there are actually 2 cod files packaged in one cod. Can be easily done via renaming to .zip and extracting. Other things that would show you smth. has gone wrong:

-1.cod files are being generated
-1.debug files. Cod might be ok, so may be the jad file. But still will get 907.
*-1.cod references in ad files.
nested cod files as already mentioned.

The general approach to fix this issue is creating application libraries and shipping them linked along with application. This is more easily said then done, because decoupling your app when it's already working might be a tedious job to do. Moreover when you seem to have completed what is needed you might see that you're 2-5 Kb over limit and that really sucks. But that is not all of the "presents" you've got with RIM.

Let us say you've mastered your app in a way that it is shipped with few libraries and it work OK. Now requirements have come to use a newer JDE/add some features. First thing to check would be just to compile using the new JDE. Although you won't probably have any issues with compiling it is VERY likely that you'll get larger cod files as result. I tested with JDE's from 4.0.2 till 4.7. And every new version compiles more code then the previous one. One exception with 4.6 - it compiles the larges binaries whatsoever :). So you're back to your previous job - refactoring the source code:)

In order for these things to happen less often use these approaches when developing with RIM:

Avoid nested classes. Each Nested class would enlarge the original class file and produce and extension worth at least 1.1K
Avoid public static or just public/protected methods/properties. Generates huge amount of header bytecode.
Avoid anonymous classes. Each anonymous class would result in $N.class where N is the number of your anonymous class:) Each worth at lest 0.9K. If you structure your code to use existing classes, you'll reduce the output bytecode significantly.
If the nested class has to be used - try to use anonymous instead. You'll gain about 0.5 K on each.
RIM doesn't like generics. Full-stop
Try to avoid using interfaces. If, for example, the base class implements and interface and the child one also states that - you'll get double same bytecode. From my experience with RIM I'd better use classes(if applicable) and only then interfaces. The overhead is tremendous.

Happy development days with RIM ;)

Idea ant hanging

noreply@blogger.com (rdev02) — Thu, 11 Feb 2010 09:18:00 +0000

I had a very 'uncomfortable' issue with Idea when using it for ant builds - it hang in certain places like jar update task, rename task or similar. All my struggle can be found on Idea track. Fortunately Carl Abramsson found a workaround for that issue using

-noinput

switch to ant launch. Now everything works like a charm. Decided to create this post in case some one will be desperately googling for a solution in the wrong place :)

Switching to container based datasource...

noreply@blogger.com (rdev02) — Mon, 11 Jan 2010 11:08:00 +0000

Quite a time ago had to do this but had no time for posting a complete solution. Read a few articles on that so that I got the idea. Although it looks quite simple you might run into several configuration issues firing weird and long stacktraces. To get rid of those I'll describe the way it worked for me with some additional comments.
First of all lets assume we use JBoss app server. Different versions of JBoss have their own configuration "specials" so I'll stick to 5.0.1 jdk 6. First of all we need to register the desired datasource withing the container obtaining a JNDI name. Example configurtion files for registration can be found in [JBoss install dir]/docs\examples\jca\. The resulting config file which has to be pout under server/[configuration_name]/deploy would look like

<local-tx-datasource>
    <jndi-name>MySqlDS</jndi-name>
    <connection-url>jdbc:mysql://localhost:3306/mydatasource</connection-url>
    <driver-class>com.mysql.jdbc.Driver</driver-class>
    <user-name>user</user-name>
    <password>pwd</password>
    <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.MySQLExceptionSorter</exception-sorter-class-name>
    <metadata>
       <type-mapping>mySQL</type-mapping>
    </metadata>
 </local-tx-datasource>

Then in hibernate properties

###### JNDI std JBOSS connection ####################
hibernate.connection.datasource = java:MySqlDS
hibernate.transaction.factory_class = org.hibernate.transaction.JTATransactionFactory
hibernate.transaction.manager_lookup_class = org.hibernate.transaction.JBossTransactionManagerLookup

Deploying your app, especially if you deployed with Tomcat compatibility you might see the next exception on startup:

ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/mydb]] (main) Exception starting filter ServiceFilter

java.lang.ClassCastException: com.arjuna.ats.jbossatx.jta.TransactionManagerDelegate cannot be cast to javax.transaction.TransactionManager

at org.hibernate.transaction.JNDITransactionManagerLookup.getTransactionManager(JNDITransactionManagerLookup.java:23)

at org.hibernate.impl.SessionFactoryImpl.(SessionFactoryImpl.java:264)

at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1055)

In short this means that you have a classloader issue having loaded the wrong manager first. Most likely you have the javax-transaction.jar in your bin. Remove it and everything will work fine :).

If you need to set up the very same thing on Tomcat, then you have to keep in mind the next thing

Tomcat is a hybrid sort of environment (neither fish nor fowl) from the standpoint of Hibernate. Tomcat provides a JNDI and DataSource, unlike a standalone application. The DataSource facility should facilitate portability among application servers. But like a standalone environment, Tomcat provides no Transaction Manager. Hence your code must use the Hibernate Transaction Manager, as it would in a standalone application environment

With this in mind you'll have to configure the server.xml with

<Resource name="jdbc/testdb" auth="Container" 
type="javax.sql.DataSource"
maxActive="100" maxIdle="30" maxWait="10000"
username="user" password="pwd" 
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/mydatasource?autoReconnect=true"/>

And web.xml with

<resource-ref>
<description>My datasource</description>
<res-ref-name>jdbc/testdb</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
</resource-ref>

Then your hibernate.properties configuration would look like:
hibernate.dialect=org.hibernate.dialect.MySQLInnoDBDialect
hibernate.connection.datasource = java:comp/env/jdbc/testdb
#hibernate.transaction.factory_class = org.hibernate.transaction.JTATransactionFactory