Technology, Software, Predictive Analytics and Big Data

Reading resource files in Spark using Scala

2018-09-02T18:49:00.001+05:30

Often while coding up unit tests in Scala, I need to read from a file which is available in the resources folder. There are a few variations to how this can be done, specifically if I am using the contents of the file as DataFrame in Spark. Here are some examples what I want to keep for myself as notes.

All these examples work with JDK 1.8u144, Scala 2.11.8 and Spark 2.3.1. Test them out if your version of software differ substantially from these versions.

Get the contents of the text file as Iterator:

// In the path below, "/" refers to the root of the resources folder
val fileStream = scala.io.Source.fromInputStream(getClass.getResourceAsStream("/path/to/file.txt"))
val iterator: Iterator[String] = fileStream.getLines()

// Print the lines 
fileStream.getLines().foreach(println)

// To get the complete contents of the file in a string
val contents = fileStream.getLines().mkString("\n")

Get a text file as Spark RDD[String], individual lines as rows:

// In the path below, "/" refers to the root of the resources folder
val session: SparkSession = SparkSession.builder().config(conf).getOrCreate()
val fileStream = getClass.getResourceAsStream("/path/to/file.txt")
val input = ss.sparkContext.makeRDD(scala.io.Source.fromInputStream(fileStream).getLines().toList)

// Another shorter alternative:
val input = session.sparkContext.textFile(getClass.getResource("/path/to/file.txt").getPath)

Get a text file as a [n x 1] Spark DataFrame with individual lines as rows:

// In the path below, "/" refers to the root of the resources folder
val session: SparkSession = SparkSession.builder().config(conf).getOrCreate()
val input: DataFrame = session.sparkContext.textFile(getClass.getResource("/path/to/file.txt").getPath).toDF("text")

Read a line-delimited JSON file into a Spark DataFrame:

// In the path below, "/" refers to the root of the resources folder
val session: SparkSession = SparkSession.builder().config(conf).getOrCreate()
// The below line works when the file is in the resource folder and the program is run
// through IDE or through command line. However, doesn't work when packaged in a jar.
// Results in org.apache.spark.sql.AnalysisException: Path does not exist
val input: DataFrame = session.read.json(getClass.getResource("/path/to/json/file").getPath).select("text")

// The following works when the resource file is packaged in a jar
import session.implicits._
val ds = session.createDataset[String](Source.fromInputStream(getClass.getResourceAsStream("/path/to/json/file")).getLines().toSeq)
val input = session.read.json(ds).select("text")

The final one is a niche use case where we have a bunch of events in an avro file, and would like to read the events. In this case, we use an iterator style, i.e. we stream the file lazily. For more notes on this, see here.

// In the path below, "/" refers to the root of the resources folder
val resourceURI: URI = getClass.getResource("/path/to/events.avro").toURI
val file: File = Paths.get(resourceURI).toFile
val datumReader = new SpecificDatumReader[Event]()
val dataFileReader = new DataFileReader[Event](new File(file.getAbsolutePath), datumReader)

// DataFileReader behaves like an iterator, i.e., a stream of events.
// E.g., if we wanted filter a few events to a separate collection:
def eventFilter(event: Event) = ???
val specialSet = dataFileReader.withFilter(eventFilter).foldLeft(...)

Update (Sep 16, 2018): Added way to read JSON file in Spark through read.json
Update (Jan 05, 2019): Updated title

DisplayLink again: sun-dried adapter, anyone?

2017-12-06T20:26:00.002+05:30

I have written about DisplayLink technology before (the USB3.0 adapter I use for projecting my laptop to dual screens), and my tricks on how I get the device working when it doesn't want to work. As it turns out, I was not even on the right track in the other post, and it is pretty hilarious how the discovery went.

Some days back, the Targus DisplayLink adapter (model ACP70US) completely stopped working: however many times I power-cycled, the monitors were not getting detected (however the mouse and keyboard was being detected every time, obviously). Initially I thought that it was because of an Win 10 update that recently got pushed. I tried switching driver versions, but that didn't help either. Desperate, I wrote in the DisplayLink support forum and the helpful admin took me through a few debugging steps.

It turns out that, my docking station has an issue in the USB 3.0 hub. Specifically, the temperature tolerance of the USB3.0 hub used in the docking station is at issue here. When using this docking station from cold condition (e.g., when it has not been used for a few hours etc), or when the ambient temperature is cold, it has (or, is having) trouble in getting started since the temperature is not within tolerance. It seems, this issue was corrected by the manufacturer (Targus) in subsequent revision or batch. Now, this product was originally purchased in US, however I am now located in Bangalore, India. For what its worth, it was actually raining here for last couple of days and definitely a few degrees colder (almost 3-4 degC colder).

Indeed, it was sunny today here in Bangalore, and I baked the device in sun for sometime. Lo and behold - it started working!!

This has to be the coolest trick I have ever encountered with computer peripherals. Oh well!!

Using HTCondor for grid computing

2017-03-05T23:48:00.001+05:30

For a project I am experimenting with, I wanted to run multiple instances of a program parallely, each instance dealing with different input data. An example to demonstrate the use case: Let’s say I have a folder full of movies in wmv format, all of which I want to convert to mp4 format, using the ffmpeg program. I would like to run as many as these conversion jobs parallely, possibly on multiple computers.

Given my current exposure to big data solutions like Hadoop/Spark, I initially thought that, for e.g., we can use yarn scheduler and run these jobs on a spark cluster. But my research indicated that these use cases are not really served by yarn/spark. These tasks are not cloud computing tasks, but they are grid computing tasks.

Now, given my graduate days in University of Wisconsin, Madison, for me, grid computing means condor (currently called HTCondor). I have used condor for my Ph.D. research, and it was awesome. So, I took a look, and gave it a try. Since I never had to set up a cluster before, here are some notes for setting up a cluster aka grid for these jobs.

Note that, this is not really a Getting Started guide; this doesn’t show, for example, sample jobs or sample submission files. For that purpose, follow the manual.

Spark vs HTCondor

Let's get started with a small discussion on when it is appropriate to use a system like Spark vs a system like HTCondor. As mentioned before, Spark is an engine for cloud computing and HTCondor is a system for grid computing. In my mind, the difference comes from the following:

HTCondor would let me run multiple instances of the same program on (single or) multiple nodes, each running on different inputs. Spark would let me parallelize and run one computation/algorithm on multiple nodes, from one input.
HTCondor nodes would be appropriate when I need to run the same program on many (somewhat small) input files, possibly being accessed from a shared drive. Spark would be appropriate when the input is a massive dataset, and we can use the map-reduce paradigm for computation.

Having that out of our way, let's proceed with HTCondor.

Machines Setup

Currently using a cluster of 3 VM, each running CentOS 6. Using v8.6.0 of HTCondor, and the manual is available here.

HTCondor Setup

Here are the commands that would install HTCondor on each of these CentOS 6 systems. To be run as root user. The original instruction can be found here.

cd /etc/yum.repos.d/

wget https://research.cs.wisc.edu/htcondor/yum/repo.d/htcondor-stable-rhel6.repo

wget http://research.cs.wisc.edu/htcondor/yum/RPM-GPG-KEY-HTCondor

rpm --import RPM-GPG-KEY-HTCondor

yum install condor-all

Notes:

CentOS <version> is equivalent to RHEL <version>
You might have to clear yum cache by using ‘yum clear all’
Once the installation is successful, the binaries goes into /usr/bin (/usr becomes the “installation folder”), and the config files go into /etc/condor
As part of the installation process, user & group ‘condor’ is created.

Here are the corresponding commands that would install HTCondor on debian systems (I ended up setting up on my Virtualbox VM for experimenting while on the go, and I run a debian distro there: Bunsen Labs Linux). Again, to be run as root user. The original instruction can be found here.

echo "deb http://research.cs.wisc.edu/htcondor/debian/stable/ jessie contrib"

                                                     >> /etc/apt/sources.list

wget -qO - http://research.cs.wisc.edu/htcondor/debian/HTCondor-Release.gpg.key

                                                     | sudo apt-key add -

apt-get update
apt-get install -t jessie condor

HTCondor Configuration

Configurations are available at condor_config in /etc/condor
One of the most important gotchas of configuring HTCondor is if the reverse DNS lookup works properly on your hosts, meaning an ‘nslookup <your ip address>’ returns your full host name. If this works, lots of trouble would be saved. Otherwise you have to add additional configs as required.
Changed the following configurations (first two because of the above reason):

For all boxes in the grid, including CM (Central Manager):

condor_config: ALLOW_WRITE = *

For all boxes in the grid, including CM (Central Manager), if you want to use it for computations:

condor_config.local: TRUST_UID_DOMAIN=true

For boxes other than the CM, point to the correct CM IP address, or CM host name if reverse DNS is working.

condor_config: CONDOR_HOST = x.x.x.x

For Debian executor boxes, blank out the BASE_CGROUP variable. More notes here.

condor_config.local: BASE_CGROUP =

If you do not want the CM box to run jobs, do not run the STARTD daemon. More information here.

condor_config.local: DAEMON_LIST = COLLECTOR, MASTER, NEGOTIATOR, SCHEDD

Running HTCondor on a Single Node

We will start with running HTCondor on a single node, which will eventually become our central manager.

Start the condor daemons:

On CentOS 6: /sbin/service condor start
On Debian jessi: /etc/init.d/service condor start OR /etc/init.d/condor start

Check if the correct processes are running.

[root@... condor]# ps -elf | egrep condor_

5 S condor 4200 1 0 80 0 - 11653 poll_s 14:53 ? 00:00:00 condor_master -pidfile /var/run/condor/condor_master.pid

4 S root 4241 4200 0 80 0 - 6379 poll_s 14:53 ? 00:00:00 condor_procd -A /var/run/condor/procd_pipe -L /var/log/condor/ProcLog -R 1000000 -S 60 -C 498

4 S condor 4242 4200 0 80 0 - 11513 poll_s 14:53 ? 00:00:00 condor_shared_port -f

4 S condor 4243 4200 0 80 0 - 16380 poll_s 14:53 ? 00:00:00 condor_collector -f

4 S condor 4244 4200 0 80 0 - 11639 poll_s 14:53 ? 00:00:00 condor_negotiator -f

4 S condor 4245 4200 0 80 0 - 16646 poll_s 14:53 ? 00:00:00 condor_schedd -f

4 S condor 4246 4200 0 80 0 - 11760 poll_s 14:53 ? 00:00:00 condor_startd -f

0 S root 4297 4098 0 80 0 - 25254 pipe_w 14:54 pts/0 00:00:00 egrep condor_

Check if condor_status returns correctly, with an output similar to this.

[root@... condor]# condor_status

Name OpSys Arch State Activity LoadAv Mem ActvtyTim

slot1@... LINUX X86_64 Unclaimed Idle 0.080 7975 0+00:00:0

slot2@... LINUX X86_64 Unclaimed Idle 0.000 7975 0+00:00:2

Machines Owner Claimed Unclaimed Matched Preempting Drain

X86_64/LINUX 2 0 0 2 0 0 0

Total 2 0 0 2 0 0 0

Notes:

If the condor_status does not return anything, it is probably because the DNS reverse lookup is not working in the system. Please follow the details in this thread, and start with setting the ALLOW_WRITE variable in the condor_config file to *.
My CentOS6 VM is dual core with 16GB of RAM, so HTCondor automatically breaks this up into 2 slots, each having 1 core/8GB RAM.

I directly experimented with a java universe job, but a vanilla universe job can be tried as well. Once the daemons are running and condor_status returns correctly, get back to the user login and submit the job, using ‘condor_submit <job props file>’. If it takes a bit of time to run the job, condor_status would show it.

[root@akka01-samik condor]# condor_status

Name OpSys Arch State Activity LoadAv Mem ActvtyTim

slot1@... LINUX X86_64 Claimed Busy 0.000 7975 0+00:00:0

slot2@... LINUX X86_64 Unclaimed Idle 0.000 7975 0+00:00:2

Machines Owner Claimed Unclaimed Matched Preempting Drain

X86_64/LINUX 2 0 1 1 0 0 0

Total 2 0 1 1 0 0 0

The command condor_q would show it too.

[root@... condor]# condor_q

-- Schedd: ... : <x.x.x.x:42646> @ 02/24/17 14:23:43

OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS

samik.r CMD: java 2/24 07:38 _ 1 _ 1 2.0

1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended

Troubleshooting/Notes:

Because of the reverse DNS problem, initially when I submitted the job, it was put on hold, and the condor_status and condor_q was showing as much. Here are the steps I followed for debugging:

I followed the instructions available here, specifically using the ‘condor_q -analyze <job id>’ command.
Looked at the log file StarterLog.slot1

Another time, I got a ClassNotFoundException, since the jar file was not available at proper place. This error showed up in the output folder of the job, in the error file (mentioned in the condor submission file).

Running HTCondor on multiple nodes

Once things work for a single work, it is easy to add more nodes to the cluster. Just install the HTCondor package, change the configurations as described, and start the service.

Notes for running jobs

Some points that were not very obvious to me even after reading the instruction manual:

In vanilla universe jobs, if the executable reads from standard input, use the input command, else use the arguments command. Consider the following snippet:

executable = testprog

input = input_file

For this snippet, condor tries to execute something like: “testprog < input_file”. If this is how testprog is supposed to run, great. Otherwise, this will not work. On the other hand, if what you really want is “testprog input_file”, then the following snippet should work.

executable = testprog

should_transfer_files = yes

# Define input file

my_input_file = input_file

transfer_input_files = $(my_input_file)

args = "<other args if required> $(my_input_file)"

Note that this possibly works a little bit differently in the java universe, where the input file name shows up as an argument to the executable (java).

For executables that are supposed to be available at the exact same path on all the nodes in the cluster, it is recommended to mention the path and switch off transferring executable.

executable = /path/to/executable

transfer_executable = false

Update (March 9, 2017): Added Debian Jessie related commands, and formatting changes.
Update (April 15, 2017): Added more Debian notes.
Update (June 3, 2018): Added notes about CM.
Update (Dec 05, 2019): Added comparison between HTCondor and Spark.

Using ssh tunnels for launching hadoop or spark jobs

2016-12-31T10:31:00.002+05:30

This is a rather niche topic, so if you are here, you probably have weighed your options, talked with your colleagues, and have enough reason to do just this. Hence, without further ado, we will get right in the topic.

As mentioned in another post, I use an ssh double tunnel, and a socks proxy to launch hadoop commands. That works quite well. The commands I use to set up the ssh double tunnel and socks host are:

ssh -4 -A -f -N -i "</path/to/keyfile>" username@intermediate-host -L5522:<final_host>:22
ssh -f -ND 8020 -i "</path/to/keyfile>" samikr@localhost -p 5522

Next I can run commands like:

hadoop fs -ls hdfs:///user/samik

Note that, in order to do this, I will have to have the following:

Hadoop distribution needs to be available locally
The version of local hadoop distribution needs to exactly match the version server-side
Appropriate configuration files

I initially thought that I will extend this method by having the spark distribution locally and launch the spark jobs on the cluster using this. I progressed quite a bit, however hit an insurmountable roadblock in the form of a bug in the spark library. The bug text explains the issue in detail. However, if your remote machine (where you are trying to execute the spark-submit from) can resolve the hadoop namenodes and yarn master, check out this page and try out - it might work for you.

Instead I had to revert to the painful process of uploading jar every time I was changing something in the program, which is what I wanted to avoid in the first place. But the method is quite clean - here are the steps to achieve this.

Create a single jar of your spark code

One caveat here: if your dependencies include one or more signed jar, you will have to either manually edit the jar to remove the signature files, or use maven settings to exclude the signature files. Otherwise you will get the following error when you run the code on cluster: Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes. Follow this StackOverflow thread for more details. Recommended process here is to use a maven file using the shade plugin.

Push the jar to your hadoop user folder

This is one way to make the jar available to the YARN executer.

ssh -i "<key file>" samikr@localhost -p 5522 "/usr/bin/hadoop fs -put - hdfs:///user/samikr/<jarname>.jar" < /path/to/jar/file

Might have to delete existing jar file.

ssh -i "<key file>" samikr@localhost -p 5522 "/usr/bin/hadoop fs -rm -skipTrash hdfs:///user/samikr/<jarname>.jar"

Push the spark-hadoop jar to the hadoop user folder as well. Even better, run a sample spark application (e.g., this page shows running SparkPi in HDP) using spark-submit on the remote host, and note down the path to the spark-hadoop jar that is being used. It should typically already be available somewhere in HDFS. Agin, the reason is to avoid copying this jar every time the job is launched.
Finally, launch the job:

Command:
ssh -i "<key file>" samikr@localhost -p 5522 "/usr/bin/spark-submit --class <complete.classname.for.ExecutableClass> --master yarn-cluster --driver-memory 4g --executor-memory 2g --executor-cores 1 --conf spark.yarn.jar=hdfs://production1/hdp/apps/x.x.x/spark/spark-hdp-assembly.jar hdfs:///user/samikr/<jarname>.jar"
In the above command, “--conf spark.yarn.jar=hdfs://production1/hdp/apps/x.x.x/spark/spark-hdp-assembly.jar” specifies the location of the spark-hadoop jar in HDFS. This part should not be required when the command is being run through an ssh tunnel - check out.

To launch hadoop jobs (or yarn jobs for that matter), it is a bit straightforward. In this case, I had to just copy the fat jar in the remote node (not in HDFS) and launch the job. Commands were as follows.

scp -P5522 -i "<key file>" /path/to/fat.jar samikr@localhost:.
ssh -i "<key file>" samikr@localhost -p 5522 "/usr/bin/yarn jar fat.jar <complete.classname.for.ExecutableClass>"

More notes:

In order to avoid the loop of compile-upload-test, it would be better to create a local hadoop-spark node, test out the code there, and then use the above procedure to run the job on the cluster. There are some interesting notes here - possibly a topic for a later post.
Using ssh config files are recommended to shorten some of the large ssh commands above.

Update (Sep 5, 2018): Added few relevant links.

Using ssh: multiple security algorithms and keys

2016-12-22T22:40:00.000+05:30

While using ssh to connect to hosts, I recently faced this interesting issue. I usually use a double ssh tunnel to connect to varied internal hosts which are behind firewall. Typically the way I set the double tunnel is using the command:

ssh -4 -A -f -N -i "</path/to/keyfile>" username@intermediate-host -L5522:<final_host>:22

You will notice that the above command sets up a tunnel which forwards the ssh port of the final host to a local port (5522), so that I can run commands. This works pretty well, and I have used this tunnel to run hadoop commands or submit spark jobs.

I was recently trying to set up a tunnel to a new host to submit spark jobs. The tunnel setup went well, but when I tried to run a hadoop command over the tunnel, I got a message regarding the key type being used for ssh handshake.

$ ssh -i "</path/to/keyfile>" samikr@localhost -p 5522 "/usr/bin/hadoop fs -put - hdfs:/

//user/samikr/datapipes.jar" < datapipes.jar

Unable to negotiate with 127.0.0.1: no matching host key type found. Their offer: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519

I checked the config file and the contents were as follows.

$ cat config

Host *

HostkeyAlgorithms ssh-dss

Clearly I needed to add one of the accepted key types for this server, but I faced some trouble specifying multiple keys in the same line. After some searching, this is what worked (note no space after comma).

$ cat config

Host *

HostkeyAlgorithms ssh-dss,ssh-rsa

Now the command seemed to be going through, but I got another error message.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that a host key has just been changed.

The fingerprint for the RSA key sent by the remote host is

SHA256:xxxxxxxxxxxx

Please contact your system administrator.

Add correct host key in /home/<user>/.ssh/known_hosts to get rid of this message.

Offending DSA key in /home/<user>/.ssh/known_hosts:5

RSA host key for [localhost]:5522 has changed and you have requested strict checking.

Host key verification failed.

There seems to be already an entry for localhost/5522 in the known_hosts, but for ssh-dss algorithm. I was hoping that another line with the new algorithm would get added in the known_hosts file for localhost, but apparently, with strict checking, only one entry per host is allowed. I had to get rid of that line, and then things went through for the command.

My DisplayLink troubleshooting guide

2016-01-17T15:47:00.000+05:30

I use Targus DisplayLink adapter for using my Win 10 ASUS laptop with multiple monitors. When things work, they work really well. But often they don't. And the problems are weird. All of a sudden, the adapter just stop working. How many times I try, nothing works, except when it starts working again.

Here are some of my notes from my troubleshooting experience.

The adapter needs correct voltage to get detected the first time I am inserting the USB 3.0. Most of the time the voltage is there, and things work. But when it doesn't work, I have seen things kick in if I do a few of the below:

Let Laptop hard drive activity subside. That means the HDD LED is not active.
Power cycle through the DisplayLink adapter.

This sometimes still doesn't work because the power adapter that powers the DisplayLink adapter also powers the external monitors.

Just detach the mini plug from the power box of the DisplayLink adapter, and put that back in.

I have recently discovered this, and this seems to work well so far.

Once I did the last one, and the monitors started misbehaving. Specifically, the two external monitors constantly started switching between #1 and #2. I have a Dell 23' monitor connected through HDMI, and a Dell 19' connected through DVI. The switching was rapid (e.g., once every second or so) and very weird, haven't seen this before. I had to detach my 2nd monitor and reattach it back in order to get things working again.
[Update - March 4, 2016] New issue: when I inserted the USB cable, I started getting the following error: "USB Device Malfunctioning", and details had something like "Device Descriptor request failed". I tried a bunch of stuff, including the usual uninstall-reinstall etc., but finally what worked is a power cycling of the display adapter.
[Update - Nov 6, 2017] New Issue: after my corp laptop was upgraded to Win 10 Creators Update (Version 1607, build 14393.1770), the DisplayLink adapter stopped working again. Repeated power cycling did not help much either. Note that, how DisplayLink adapter works has also changed a bit (read more here).

I tried using version 8.3M1, but that didn't seem to work too well. I am now using version 8.4 Alpha, and the following points are corresponding to this.

In 'Settings -> Devices -> Connected Devices' an 'Unknown Device' or something similar was showing up along with 'DisplayLink USB Graphics Device'. Similarly, something similar like 'Unknown Device' was showing up in the 'Devices and Printers' window as well.
What finally helped was uninstalling the unknown device from the 'Connected Devices' screen, multiple times, and then power cycling. After doing this a few times, at one point the device got finally recognized as 'DisplayLink USB Graphics Device'.
Still looks like it is a hit-miss situation.

Black Screen of Death on Windows 10

2015-10-29T18:23:00.000+05:30

I have so far liked Win 10, but was recently hit by the dreaded black screen of death (looks like, this is the new blue screen of death), where I am stuck with a black screen with a mouse pointer. There are numerous posts on the Internet, but it was actually not that easy to get through most of them.

Let me state the issue clearly: I have a few months old ASUS Q302L model, which was working great. This is how it started:

I had the brilliant idea of connecting the ASUS Q302L laptop to Chromecast, so that I can project videos (mostly Youtube) on TV.
Installed Chromecast for PC, but realized that it doesn't do much.
Installed Chromecast plugin for Chrome browser, and that seemed to do the trick.
Uninstalled Chromecast for PC. [I think, this is the root cause for all my woes! Ok, Google!!]

After a day or so of the above, I noticed that my Youtube videos are not working on the notebook any more. The flash based player doesn't work, the HTML5 based player doesn't work. In fact, to my horror, I found out, that videos doesn't play at all on the notebook, even on standalone software like WMP. Bummer!!

Decided to restore the PC to a recent restore point.

The most recent restore point errored out, saying something like "Can't change registry settings" etc., and booted back to Win 10 with no video.
Used an earlier restore point.

That seemed to work, until I was stuck at the black screen with mouse pointer.

Then started the long process of google searches and tryouts. Here are the steps in the ordeal:

Realized soon that I need to get to the 'System Repair' option by using Win 10 installation media.

Used another Win 10 PC to create an USB Win 10 installation media.

Inserted the USB media, but it was not used for boot up. Realized that I have to change the boot order.

REMINDER #1: When I get a new laptop, always change the boot order to have the USB drive before the hard drive. (More caveats to this later)

Realized that unlike lot of other systems, the nice ASUS logo screen when powering up by Q302L model, doesn't show any details of how to get into BIOS menu. More searching followed, including taking a peek at the manual for ASUS Q302L model. Seemed like either F2 or F8 might help. Tried both options, but no dice.

Eventually F2 worked, seemed like it was just a matter of timing. Now I actually appreciate having a separate button to boot into BIOS menu, like Lenevo models.

I tried a bunch of stuff:

Other restore points: they errored out.
I had no System Image set up.

REMINDER #2: Probably a System Image would have been an easier option to restore, when the restore points failed. This is time consuming however, so can possibly be done once a month.

I also tried out booting into safe mode, safe mode with networking, with low-res graphics etc., however none of these worked. Every time I got stuck with the same black screen - it was really stubborn.
To my chagrin, I found out that every time I tried booting without the Win 10 USB drive to check if things work, the boot option went back to 'Hard Drive', so I had to start at F2 again.
Finally decided to reinstall Win 10.

Reinstalled Win 10. Things went reasonably smooth, and I got back the display.
Got stuck at activating Win 10 though - the process errored out with an error code: 0xC004C003

As per this page, this means: The MAK is blocked on the activation server.
My Win 10 copy was initially upgraded using Windows Update from Win 8.1, so as such there shouldn't be any problem regarding a fresh installation at a later point, specifically when there is no hardware change.
Contacted Customer Support over chat, and finally got things resolved.

Looks like, I have not set up my Country. This can be set up by bringing in the dialog by executing 'slui 4' command from Run. This has to be the same country from where you activated Win 10 the first time.
That may not have been enough. After that, the CSR needed my Win 8.1 product key. This can be obtained by running the following command in an Admin command prompt: wmic path SoftwareLicensingService get OA3xOriginalProductKey
Then the CSR did something in the background, got the Win 10 product key, and used that to activate.
My suggestion is to get this done using CSR help.

Long ordeal. I still have to reinstall all my apps, but I can use the laptop - which is more than enough of what I can ask :-)

Some links that were good but not helpful to me (may be they would be helpful to you):
- Youtube link
- MS link

Setting up Orientdb - a graph database

2015-09-03T10:24:00.000+05:30

I recently found myself dabbling to graph databases for some interesting analytics work. One of the software that we wanted to test out is a document-graph database called Orientdb. The software is interesting because it covers functionality from both document databases (e.g., MongoDB) and pure graph databases (e.g., Neo4j). While trying to set up the server however, faced a few gaps in the documentation, which is why this post. I am using Orientdb Community Edition v2.1.1.

Using Custom Port

By default, Orientdb binary server runs at port 2424 and web server runs at port 2480. To use different ports with Orientdb, one has to change the settings in config/orientdb-server-config.xml. There you would find the port ranges provided for each server operation (i.e., binary and web servers). Honestly, I have not seen port ranges for other servers before, presumably this is for the purpose that if a particular port is occupied, the software will run on next available port - kinda curious. Anyways, you will have to change the starting port numbers for your desired port in those settings.

Connecting from Orientdb console

The console example provided in the documentation does not provide examples when you are running the servers on custom port. Here is how you can connect to the server, or to a database.

orientdb> connect remote:localhost:3306 root root

Connecting to remote Server instance [remote:localhost:3306] with user 'root'...OK
orientdb {server=remote:localhost:3306/}>

orientdb {server=remote:localhost:3306/}> list databases

Found 1 databases:

* GratefulDeadConcerts (plocal)

You can also directly connect to a database, while mentioning the port, as follows:

orientdb> connect remote:localhost:3306/GratefulDeadConcerts root root

Connecting to database [remote:localhost:3306/GratefulDeadConcerts] with user 'root'...OK
orientdb {db=GratefulDeadConcerts}> classes

CLASSES
----------------------------------------------+------------------------------------+------------+----------------+
NAME                                         | SUPERCLASS                         | CLUSTERS   | RECORDS        |
----------------------------------------------+------------------------------------+------------+----------------+
E                                            |                                    | 10         |              0 |
followed_by                                  | [E]                                | 11         |           7047 |

Will update this post with more notes.

Using Apache Crunch with own avro files

2014-08-05T18:32:00.000+05:30

I have been working in big data and machine learning for a while now after moving on from Oracle Crystal Ball. Recently I started using/learning Apache Crunch for constructing and analyzing our data pipelines. If you are in big data, but have not heard about Crunch before - check it out. If you are familiar with Cascading, this is a very similar project, with certain differences.

We store our data (which are different type of events) in HDFS using avro format. My 'Hello World' task to teach myself was to read data from HDFS consisting of a bunch of avro files, and print out the count of events in those files. I started out following the example in the code mentioned in this blog post from Cloudera. I faced couple of stumbling blocks while getting to my objective, and this post details the way I resolved them.

Note: This post is written from the perspective of CDH3u6 dist.

One of the aims I had was that I wanted to run my code directly, not by passing the jar to hadoop (as in, hadoop jar MyProgram.jar ...). The reason for this was that I want to, at some point, throw in a UI for easy use. We will see if I face additional issues with that, but that is the motivation anyways.

First off, our HDFS namenode can only be connected through a double ssh tunnel form my machine. Agreed, this has nothing to do with Apache Crunch, but I had to embed the configurations in the program itself. The double ssh tunnel essentially transfers the namenode to a local port. If you want to know the details of how to do this, follow this link. Here is how the configuration part of my program looks like, to use the tunnel for running the Crunch job.

    public static void main(String[] args) throws Exception
    {
        // Create a configuration object to load specific configurations.
        Configuration conf = new Configuration();
        conf.set("hadoop.socks.server", "localhost:8020");
    conf.set("hadoop.rpc.socket.factory.class.default", "org.apache.hadoop.net.SocksSocketFactory");
        conf.set("fs.default.name", "hdfs://namenode.xxx.com:8020");
        conf.set("hadoop.tmp.dir", "/var/tmp/hadoop");
        // Run the tool.        ToolRunner.run(conf, new DataLoader(), args);
    }

Moving on to the next bummer: the first time I tried to compile and run my program, I got the following exception:

Exception in thread "main" org.apache.hadoop.ipc.RPC$VersionMismatch: Protocol org.apache.hadoop.hdfs.protocol.ClientProtocol version mismatch. (client = 61, server = 63)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:401)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)

There were a few more lines before the above exception, showing another exception: but this exception was listed as the root cause. Searching the Internet with the error statement yielded the following reason: Hadoop server version that I am using is newer than the Hadoop client libraries that one or some of the dependencies are using. A simple solution is suggested in both this link and this link: specify the hadoop libraries in the program classpath and move them at the top of the classpath stack. I had a local hadoop client set up which I was using for tests, and it was easy to set up the classpath to do this. Unfortunately, In IntelliJ, the editor I use, the procedure changes almost every version!! Here is a link to do this for version 13.

Even after doing this, I got another exception: [Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/thirdparty/guava/common/collect/LinkedListMultimap]. This was resolved by following this link and adding the missing jar file in the same library group as above.

That was that, got the expected output count. Hello Crunch!!

Update [8th Sept, 2014]
Before I could do any more cool stuff, our cluster got upgraded to CDH5 and I had to set up my stuff all over again. This time it was much easier though. Here are the notes:

Had to upgrade the pom.xml for pulling in the new maven repository. Note that the hadoop website doesn't contain any information about the maven repositories, but a search yielded the following website which contains all the information: http://mvnrepository.com/artifact/org.apache.hadoop

This version of hadoop doesn't contain the hadoop-core artifact that I was using for CDH3 (which was hadoop v0.20.2). I made the reference to artifact "hadoop-client", with version 2.5.0 (the latest as of this writing).
Also, the Crunch version had to be upgraded to work with hadoop 2.x: used "crunch-core" version 0.10.0-hadoop2.

Changed the namenode information in java code.
Removed the classpath reference from the local hadoop client (which I have also upgraded to use CDH5). No RPC error this time.

Initially I forgot to remove the reference to local hadoop jars (as mentioned in the original post), which was still referencing hadoop 0.20.2 libraries. I was getting the following error then: "Server IPC version 9 cannot communicate with client version 4"

Adding android module to an imported android project in IntelliJ

2014-05-17T13:35:00.000+05:30

I started using IntelliJ recently - and was mighty pleased using it. For my Android apps, I used to use Eclipse before - but I decided to migrate those projects to IntelliJ as well. Hit some roadblocks - so here are some notes for myself (and others reading this post).

I am using IntelliJ v13.1.2, which is fairly recent. My app was meant to be compiled using Android SDK for v3.2 (Gingerbread).

First off, I imported the project in IntelliJ using the 'File -> Import Project' menu item. I had to just point to the Eclipse .project file, and IntelliJ did the rest to import the project, and opened in a new window. However, when I tried rebuilding the project, I got the following message:

Error: Android Source Generator : [] Android SDK is not specified

Screenshot 1: Error Message

Looks like, it was not able to get the Android SDK information. I have already installed the Android SDK in my Windows 7 laptop, so obviously, I am missing some connection setting somewhere.

Here is how I fixed it, after doing some web searches.

Go to File -> Project Structure. Here, click on the 'Modules' link on the left hand side. Now on the right hand side, in the 'Dependencies' tab, you would see that the Android SDK is not mentioned. Hit 'New..' button by the side of the drop down and select 'Android SDK'. From there, it should be pretty obvious what needs to be done to add your required android SDK to your project.

Some screenshots along the way:

Screenshot 2: Adding Android SDK

Screenshot 3: After adding Android SDK

Hope this helps me the next time.

New version of Oracle Crystal Ball (Release 11.1.2.3.500) is out

2014-05-02T19:13:00.000+05:30

Here is an excerpt from the official text:

Crystal Ball Users,

Release 11.1.2.3.500 is now Generally Available on the Oracle Software Delivery Cloud (eDelivery) and on the Oracle Technology Network. This release is available to new customers and customers under support.

The following is a summary of what’s new in Crystal Ball 11.1.2.3.500:
· Correlation Definition Enhancements
· Bulk Constraint Handling for OptQuest
· Distribution Parameter Enhancements

More information can be found in the online New Features Guide:
http://docs.oracle.com/cd/E40248_01/epm.1112/cb_new_features_11123500/cb_new_features.html

In addition, I would like to remind everyone of the following resources:

Oracle Crystal Ball home page:
http://www.oracle.com/us/products/applications/crystalball/overview/index.html

Oracle Technology Network:
http://www.oracle.com/technetwork/middleware/crystalball/overview/index.html

CB Users LinkedIn Group:
http://www.linkedin.com/groups/Oracle-Crystal-Ball-120531?home=&gid=120531

CB Solution Factory: http://launch.oracle.com (Access URL/Page Token: cb4me)

We appreciate your engagement and use of Crystal Ball in your businesses and organizations.

Enjoy!!

Parsing JSON strings in MVEL interactive shell

2014-03-27T14:25:00.000+05:30

This is a pretty niche application - so I will get started without any introduction. Assumption here is that if you reach here, you are probably looking for this exact same thing in google or some other search engine.

I wanted to parse the following JSON string in MVEL interactive shell:

{"f":{"v":{"dd":"Windows-7~D~windows","tz":0,"br":"c~33.0.1750.154"},"p":{"cp":"http://247-demo.net/"}},"bsid":"6fb4363f-5e06-4f10-b1ba-463a1f5e961f-1","it":0,"vi":"6fb4363f-5e06- 4f10-b1ba-463a1f5e961f","tseq":1,"tp":"1","vg":"4","vt":"1.3","as":"1","idm":{"lkp":"email","lkv":"myemail@email.com"},"vp":"6.0","ec":"10022","ti":"t1","up":1395809903783,"v":"default","tnt":"DEMO_C2C","rt":30.753,"vc":"v0.1","st":1395809906804,"rv":69.864,"bd":"Mozilla%2F5.0%20(Windows%20NT%206.1%3B%20WOW64)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F33.0.1750.154%20Safari%2F537.36"}

I wanted to use the excellent Gson library from google. Here is a link to the JavaDoc for the library. Of course, you can also use any other JSON parser library of your choice. I wanted an easy way to get to a value in a nested JSON string as shown above, e.g., "f.v.br", and the JsonParser class in the Gson library is a great choice for that.

First, in order to use Gson interactively from MVEL shell, we have to load both jars. As most of us MVEL users are aware, MVEL is notoriously poor on documentation - you have to scramble to do anything. Anyways, rants aside, here is how to load two jars and get the MVEL prompt:

$ java -cp "gson-2.2.2.jar;mvel2-2.1.3.Final.jar" org.mvel2.sh.Main

MVEL-Shell (MVELSH)
Copyright (C) 2010, Christopher Brock, The Codehaus
Version 2.1.0

Starting session...
No config file found. Loading default config.
[01:28PM] mvel2$ import com.google.gson.JsonParser;
class com.google.gson.JsonParser

This, of course, assumes that both the jars are located in the same folder from where the above command is executed. Once you are at the shell, it is reasonably straightforward. We will have to use JsonParser to parse the JSON string in a JsonObject, and then use get() calls to reach to the value. Here is the demo session. The output from the shell is in grey.

[01:29PM] mvel2$ a='{ "f": { "v": { "dd": "Windows-7~D~windows", "tz": 0, "br": "c~33.0.1750.154" }, "p": { "cp": "http://247-demo.net/" } }, "bsid": "6fb4363f-5e06-4f10-b1ba-463a1f5e961f-1", "it": 0, "vi": "6fb4363f-5e06- 4f10-b1ba-463a1f5e961f", "tseq": 1, "tp": "1", "vg": "4", "vt": "1.3", "as": "1", "idm": { "lkp": "email", "lkv": "myemail@email.com" }, "vp": "6.0", "ec": "10022", "ti": "t1", "up": 1395809903783, "v": "default", "tnt": "DEMO_C2C", "rt": 30.753, "vc": "v0.1", "st": 1395809906804, "rv": 69.864, "bd": "Mozilla%2F5.0%20(Windows%20NT%206.1%3B%20WOW64)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F33.0.1750.154%20Safari%2F537.36" }'
{ "f": { "v": { "dd": "Windows-7~D~windows", "tz": 0, "br": "c~33.0.1750.154" }, "p": { "cp": "http://247-demo.net/" } }, "bsid": "6fb4363f-5e06-4f10-b1ba-463a1f5e961f-1", "it": 0, "vi": "6fb4363f-5e06- 4f10-b1ba-463a1f5e961f", "tseq": 1, "tp": "1", "vg": "4", "vt": "1.3", "as": "1", "idm": { "lkp": "email", "lkv": "myemail@email.com" }, "vp": "6.0", "ec": "10022", "ti": "t1", "up": 1395809903783, "v": "default", "tnt": "DEMO_C2C", "rt": 30.753, "vc": "v0.1", "st": 1395809906804, "rv": 69.864, "bd": "Mozilla%2F5.0%20(Windows%20NT%206.1%3B%20WOW64)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F33.0.1750.154%20Safari%2F537.36" }
[01:40PM] mvel2$ p = new JsonParser();
com.google.gson.JsonParser@133e8c9
[01:40PM] mvel2$ e = p.parse(a);
{"f":{"v":{"dd":"Windows-7~D~windows","tz":0,"br":"c~33.0.1750.154"},"p":{"cp":"http://247-demo.net/"}},"bsid":"6fb4363f-5e06-4f10-b1ba-463a1f5e961f-1","it":0,"vi":"6fb4363f-5e06- 4f10-b1ba-463a1f5e961f","tseq":1,"tp":"1","vg":"4","vt":"1.3","as":"1","idm":{"lkp":"email","lkv":"myemail@email.com"},"vp":"6.0","ec":"10022","ti":"t1","up":1395809903783,"v":"default","tnt":"DEMO_C2C","rt":30.753,"vc":"v0.1","st":1395809906804,"rv":69.864,"bd":"Mozilla%2F5.0%20(Windows%20NT%206.1%3B%20WOW64)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F33.0.1750.154%20Safari%2F537.36"}
[01:47PM] mvel2$ e.getClass()
class com.google.gson.JsonObject
[01:49PM] mvel2$ e.get("f").get("v").get("br")
"c~33.0.1750.154"

That is pretty much it - parsing a nested JSON string using MVEL shell.

Moving on from Oracle Crystal Ball development team

2013-08-23T12:12:00.000+05:30

As of August 16th, 2013, I am not part of the Oracle Crystal Ball development team any more. For questions related to Oracle Crystal Ball, please post in either our LinkedIn group, or our Yahoo group (but not both) - and the Crystal Ball community (of which I will continue to be a part) and the organization should be able to help you out.

Personally, I intend to continue my journey in Monte Carlo simulation, stochastic optimization and time-series forecasting, and hope to keep posting on some topics related to Crystal Ball as well, as time permits.

I will miss being part of the Oracle Crystal Ball development team.

In search of a good code editor

2013-07-29T09:28:00.000+05:30

Every coder worth his/her salt is always looking for a better code editor. Why? Because, there is always something in a code editor you are currently using (or, are forced to use) that ... you don't like, or doesn't work, or is plain broken. But, why stick with it then? Well, might be for variety of reasons:

Your organization mandates using it, and everyone in the team uses it
It is still the editor with least amount of nuisance
The editor has something special about certain software project you are working on

Happens with everyone. Happens with me. Here are my set of editors:

Visual Studio

C#, .NET
Javascript, HTML5 (mobile coding)

Eclipse

Java
Perl (with the EPIC plugin)

PSPad

SQL
Text files
Everything else.

The features I value in a code editor, in no specific preference, are color coding/syntax highlighting, auto-completion of symbol names, method names and properties, code folding (in regions), line wrapping and split windows. I think Visual Studio does all this things perfectly, and would have been pathway to coding nirvana, had it not for the two problems:

The latest and greatest of it costs money (I know, there is an express version, and I think that is really really good)
It is not general purpose for all languages that I use.

The last one is big, so even if I have VS open almost all the time, I sill have to open bunch of other editors as well to get stuff done. Eclipse, on the other hand, doesn't give me that feel. It is very powerful and free, but always feels a little sluggish, weird, and what not. Eclipse also does not have auto-completion of symbol names (without doing CTRL+SPACE), which I find very helpful. But I stick with it, because it has special tooling for android development.

PSPad, on the lower end, has been reasonably good to me. It doesn't have any auto-completion or code folding, but the files I work with in PSPad, I mostly don't need those features. I needed those features for perl, which is why moved perl from PSPad to Eclipse.

Courting with Geany

A few months back, I uninstalled PSPad after using it for more than five or so years, because I discovered another editor called Geany. I read about it, installed it and it looked good. Not only was it small and fast, but it seemed to have all the features I need. I was even thinking I would be able to bring back perl, and even java [*shiver*] (with android). If I can replace Eclipse and PSPad with Geany - that would itself be pretty cool.

Unfortunately, the dating period wasn't too long. Here are a few reasons:

The editor is based on GTK+. That means, in order to install it, you will have to install GTK+ first. I don't like to install too many things, and this didn't go well. Note: this wasn't a deal-breaker in itself, but accompanied with other ones, it was.
The autocomplete feature of Perl or java wasn't a true autocomplete - if you press CTRL+SPACE after typing one or ore letters, the editor would show any token in the file starting with those characters. It works as autocomplete, and is better than nothing, but isn't the same as I understand from VS.
Finally, interface was a bit old style ... for my taste anyways.

Next in Line: Notepad++
So, when I got my new laptop, I decided to look for another editor yet again. This time it was the turn for Notepad++. The feature set looked good (yes, it also supports autocomplete without having to press any extra keys), the interface looks good, so let's see how this goes.

Let me finish this post with some links to a few posts from coding horror by Jeff Atwood. If you are into programming, read his blog.

Update (3rd Sep, 2015): After almost two years of publishing this post, I am still using Notepad++, and I have to say that I am very happy using it. If you have not tried this editor (on Windows) yet, check it out.

Simulation Modeling for football (soccer) games

2013-05-20T23:17:00.000+05:30

In the last few weeks, I have come across two posts and discussions on modeling football (soccer) games with Monte Carlo simulation. Thought it might be interesting for our readers. Yes, I am more used to calling this game football, but let's just skip that part, shall we?

The first one is a blog post detailing a simple model which tries to simulate the number of goals scored by each competing team in a match. The blog discusses quite a bit on the inner details of finding out distribution parameters using optimization, which may not be that relevant, but the post is interesting. Find the blog post here. The post models the world cup final of 2011.

The second model is from our own Dave Hammal, who has posted the model in our LinkedIn group. He models the ongoing English Premier league, and predicts the match-by-match results. To access the post and the model, use this direct link.

Both models are similar, in that, they try to use discrete distributions to model the number of goals that are going to be scored in the game, and they find out the parameters of the discrete distribution from the past data for that team. The first post also discusses a few interesting improvements to the model. Note that, the first post is a static analysis of an old event (something similar to what I mentioned about the World Cup Cricket final in this post), whereas the model in the second post is dynamic, in that the model predictions are updated with the new data as the league progresses.

Finally, here are two websites (I am sure there are more of these), which claim to contain stats related to soccer games: Squawka [link to TechCrunch coverage] and FTBPro [link to TechCrunch coverage]. None of these websites contain predictive modeling though, just stats and user generated contents.

New version of Oracle Crystal Ball (Release 11.1.2.3.000) is out

2013-05-02T04:03:00.000+05:30

Here is an excerpt from the official text:

Crystal Ball Users,

Crystal Ball 11.1.2.3 is released! The following is a summary of what’s new in Crystal Ball 11.1.2.3:
Redesigned Tornado Analysis Tool
Integrated Correlation Matrix Tool
Addition to Crystal Ball Developer Kit
Expanded OptQuest Efficient Frontier Results View
Support for Microsoft Excel 2013
Support for Microsoft Windows 8
Localization into Additional Languages
Documentation Now Certified on Apple Mobile Devices
More information can be found in the online New Features Guide: http://docs.oracle.com/cd/E40248_01/epm.1112/cb_new_features/cb_new_features.html

A trial version is downloadable from the Oracle Technology Network: http://www.oracle.com/technetwork/middleware/crystalball/downloads/index.html

If you have not already done so, I also recommend visiting the Crystal Ball Solution Factory. Here you will find useful information about new releases in addition to presentations, discussions, and resources that cover different applications of Crystal Ball.

Crystal Ball Solution Factory (http://launch.oracle.com Access URL/Page Token: cb4me)

We appreciate your engagement and use of Crystal Ball in your business.

If there are questions about the availability of the software or any other aspect, please get in touch.

A clarification/update: The link I posted for downloading the new version of CB is not yet active - sorry for the inconvenience. Please use the following sites:

- If you want to download the trial version, please use: http://www.oracle.com/technetwork/middleware/epm/downloads/crystal-ball-1112x-1939011.html

- Upgrades are only available for customers on support. In those cases customers should use the Oracle Software Delivery Cloud (https://edelivery.oracle.com/) to download.

Update (05/02/2013): Updated with more download links.

Demo video for Oracle Crystal Ball

2013-04-16T22:01:00.000+05:30

We have a new demo video out, and it looks fantastic. This would be a great way to introduce someone to the methodology of "thinking in ranges" espoused by Oracle Crystal Ball. Available from this link, it needs a quick registration. Enjoy !!

Monte Carlo simulation example in transportation

2013-04-16T08:58:00.000+05:30

I used to work on transportation services procurement auction in my doctoral days. Since graduation, I have moved away from that sector. But, recently I had a chance to work on a technical research presentation in the transportation sector.

In my dissertation, I mostly used mixed integer programming formulations for solving bidding problems. Although there was a little bit of predictive analytics in the model, I did not really explore much of the Monte Carlo simulation aspect of it, except generating random numbers from normal distributions. This time though, I was able to use MC simulation quite a bit. The problem was on determining the safety stocks of tractors/trailers/drivers for a small trucking company which wishes to maintain certain service level.

Although, network is an important part of transportation, I sidestepped that aspect by aggregating on the lanes in the transportation network. Agreed, that makes the model somewhat weak, but in my opinion, the model is still useful enough to get some mileage out of it.

Following is the title and abstract of the presentation. Please get in touch using the comments section if you are working on something similar.

Title: Determining Safety Stock for a Small Trucking Operation
Abstract: In this presentation, we will determine the safety stock of trucks required by a small trucking operation to maintain a certain service level. We will start off with a simple model and use Monte Carlo simulation and stochastic optimization to determine the safety stock. Next we will extend the model to include more realistic situations and show that our approach still provides a fast, efficient and reliable way to calculate the variables and statistics of interest in the model.
Presentation: Available here.

My Ph.D. dissertation is also available on this page, just scroll to the 2006 entry.

Displaying statistic in assumption cells

2013-03-13T04:54:00.000+05:30

This question often comes up in group discussions and support questions: although the assumptions and forecast cells in an Oracle Crystal Ball spreadsheet model change values while running a simulation, they revert back to the original values at the end of simulation. This often confuses new users. In this post, I will discuss the reasoning and some options to change this behavior (partially). For an example of such discussion, check this question in our LinkedIn group.

First off, this is the default behavior of the software, and there is an explanation for this behavior. We can think about it this way: a simulation is a series of automated what-if analysis. We are trying out various input values in the assumption cells and then recording the output values from the forecast cell(s). The objective of the simulation is to get some statistical understanding of the output values (your forecasts) for a range of input values (your assumptions).

At the end of simulation, we are done with the series of what-if analysis, and we just return back to the original value that was there in the cells before we started the analysis. We consider these original/starting values as base-case scenario. Since the modeler (you, in this case) have started off with these values, that is where we return to.

But one can choose to see a different statistic in the assumption cells, extracted from the simulation results, after the simulation is complete. To do so, open the "Cell Preferences" dialog from the ribbon (it is the last one in the column left to the "Start" button, see Figure 1 below). There, choose your preferred statistic you wish to see after a simulation run for the assumptions in your model and apply this setting to your model using the 'Apply To..' button. Since the forecast cells are linked to one or more assumption cells by a formula, after the simulation they will automatically reflect the calculated values for the chosen statistic displayed in the assumption cells.

Figure 1: Cell Preferences

For example, if you have chosen to show the mean statistic in the assumption cells after the simulation, the forecast cells will show the calculated value using the assumption means. Note that, depending on the formula, this final value shown in the forecast cell may or may not match with the corresponding statistic shown in the assumption cell (mean in this case). If the forecast is non-linearly related to the assumption cells (i.e., the formula used in the forecast cell has nonlinear components like square, square root, log etc.), then the forecast mean would be different from the displayed value.

Note, it is not possible to show certain statistic for a few assumption cells and certain other statistic or none for a few other ones, as that would lead to display problems. The same display preference has to be applied to all assumptions in the model.

Update (3/15/2013): Corrected statement.

A praise on C#

2013-02-06T23:35:00.000+05:30

I am sure that there is no dearth of praises for the C# programming language, and you can say that I am a fan on the outset, but a paragraph I read today seem to summarize aspects of C# quite nicely. It was an article on testing C# (and Python) code appearing on drdobbs.com by Gigi Sayfan. The pragraph is quoted below:

C# is a wonderful programming language. It started out as a better Java, and in each revision, it has added more and more good stuff (generics, anonymous functions, attributes, LINQ, async support) with great balance. For some reason (great language designers?), the power and richness of C# never gave me a feeling of bloat like other languages do. When you throw in the .NET framework and Visual Studio, you get a superb development and runtime environment.

The only thing the author is missing here is mentioning the Mono engine, which makes C# cross-platform as well in the same sense as java is cross-platform, and that makes it really appealing. I hope to continue coding in C# for years to come.

Try Oracle Crystal Ball Free for 30 Days

2013-01-30T23:23:00.000+05:30

Breaking my long-time silence (have been really busy developing the next release) to bring a nice news. We have changed our software trial policy - our evaluation license now allows you to try out Oracle Crystal Ball for thirty days. We definitely hope that you are going to come back for more after the end of your month-long trial :-)

Here is the direct link for our download page: [http://www.oracle.com/technetwork/middleware/crystalball/downloads/index.html]

Enjoy.

Embedding a google drive form in your website

2012-06-04T05:08:00.000+05:30

If you do not want to embed the complete google form iframe in your webpage, you can bypass that and have some simple code instead. The following lines show a way where I create the input field myself and set up everything, but the response gets stored in a google drive spreadsheet.

<form method="post" action="https://docs.google.com/spreadsheet/formResponse?formkey=xxxx">
<span style="font: 16px calibri">Enter email: </span>
<input type="text" name="entry.0.single" value="" style="font: 16px calibri" />
<input type="submit" value="Submit Email" style="font: 16px calibri" />
</form>

Check it out - can be useful in certain circumstances.

Here is a slightly longer version, with a form having four fields, where I take care of the form submission myself, using jQuery and jQuery Mobile (slightly dated version of both - for reference only). The jQuery .ajax(..) call is used here - works great. Note: do not use dataType: "jsonp" while POST-ing, since the call returns a complete HTML page which we have no use and can't process directly. While developing, use a tool like Firebug, so you are sure what is getting in and what is getting out.

<!DOCTYPE HTML>
<html>
<head>
    <title>ConferenceToGo</title>
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta charset="utf-8" />

    
    <link rel="stylesheet" href="http://ajax.aspnetcdn.com/ajax/jquery.mobile/1.2.0/jquery.mobile-1.2.0.min.css" />
    <script type="text/javascript" charset="utf-8" src="http://ajax.aspnetcdn.com/ajax/jQuery/jquery-1.8.2.min.js"></script>
    <script type="text/javascript" charset="utf-8" src="http://ajax.aspnetcdn.com/ajax/jquery.mobile/1.2.0/jquery.mobile-1.2.0.min.js"></script>
</head>
<body>
<div id="app-request" data-role="page" class="backdrop">
    <script type="text/javascript" charset="utf-8">
        function formSubmit()
        {
            console.log("Submitting form");
            // Collect form data
            var formData =
            {
                "entry.0.single" : $("#entry_0", $.mobile.activePage).val(),
                "entry.4.group" : $("#req_form", $.mobile.activePage).find("input[type='radio'][name='entry.4.group']:checked").val(),
                "pageNumber" : 0,
                "backupCache" : ""
            };
            // Show the loading image.
            $("#req_form", $.mobile.activePage).html("<img src='../common/images/progressbar.gif' />");
            $.ajax(
            {
                type: "POST",
                url: "https://docs.google.com/spreadsheet/formResponse?formkey=xxxx",
                data: formData,
                success: function (data)
                {
                    $("#req_form").html(
                        "Thanks for your interest!! We will be in touch shortly.<br />" +
                        "<br />" +
                        "Regards,<br />" +
                        "ConferenceToGo Team"
                    );
                },
                failure: function (data)
                {
                    $("#req_form").html(
                        "Sorry !! Couldn't send your request.<br />" +
                        "Can you please try again?<br />" +
                        "<br />" +
                        "Regards,<br />" +
                        "ConferenceToGo Team"
                    );
                }
            });
        }
    </script>
    <div data-role="header" style="height: 45px">
        <a href="#" class="ui-btn-left top-btn" data-icon="back" data-rel="back">Back</a>
    </div>

    <div data-role="content" style="padding-top:5px;">
        <h2>ConferenceToGo for your Conference</h2>
        <div id="req_form">
            <p>
            Thanks for your interest in the ConferenceToGo app. Please fill out this short
            form and one of our team members will get back to you shortly.<br />
            <span style="color:Red">* Required</span>
            </p>
            <form data-ajax="false" onsubmit="formSubmit(); return false;">
                <input type="text" name="entry.0.single" placeholder="Your Name (*)" id="entry_0" value="" required />

<fieldset data-role="controlgroup">
                    <legend>Typical number of attendees:</legend>
                    <label for="group_4_1">Less than 500</label>
                    <input type="radio" name="entry.4.group" value="Less than 500" id="group_4_1" />
                    <label for="group_4_2">500 to 1000</label>
                    <input type="radio" name="entry.4.group" value="500 to 1000" id="group_4_2" />
                    <label for="group_4_3">1000 to 5000</label>
                    <input type="radio" name="entry.4.group" value="1000 to 5000" id="group_4_3" />
                    <label for="group_4_4">More than 5000</label>
                    <input type="radio" name="entry.4.group" value="More than 5000" id="group_4_4" />
                </fieldset>
                <input type="submit" name="submit" value="Submit" data-inline="true" />
            </form>
        </div>
        <br />
        <br />
        <span style="font-size:small">Powered by <a href="http://docs.google.com" target="_new">Google Docs</a></span>
    </div>
</div>

</body>
</html>

Update (3/15/2013): Added a longer example.

New version of Oracle Crystal Ball (Release 11.1.2.2.000) is out

2012-05-24T06:42:00.000+05:30

Here is an excerpt from the official text:

Crystal Ball Users,

Crystal Ball 11.1.2.2 is released! The following is a summary of what’s new in Crystal Ball 11.1.2.2:
Grouped Assumptions in Sensitivity Charts
Data Filtering When Fitting Distributions
Parameter Edits When Fitting Distributions
Expanded Distribution Parameters
Predictor Parallelization
Localization into Additional Languages (Japanese, Spanish, French, German, Portuguese)

More information can be found in the online New Features Guide: http://download.oracle.com/docs/cd/E17236_01/epm.1112/cb_new_features/cb_new_features.html

A trial version is downloadable from the Oracle Technology Network: http://www.oracle.com/technetwork/middleware/crystalball/downloads/index.html

If you have not already done so, I also recommend visiting the Crystal Ball Solution Factory. Here you will find useful information about new releases in addition to presentations, discussions, and resources that cover different applications of Crystal Ball.

Crystal Ball Solution Factory (http://launch.oracle.com Access URL/Page Token: cb4me)

We appreciate your engagement and use of Crystal Ball in your business.

I will have a series of posts on some of the features in the days ahead. If there are questions about the availability of the software or any other aspect, please get in touch.

Restoring between Installatron and Softaculous

2012-04-29T03:26:00.000+05:30

Recently I changed my webhosts. On the positive side, both hosts use DirectAdmin based control panel, and have the same version of DirectAdmin software. But apparently, that was not enough. On my old hosting account, I had an installation of Mantis (a bug tracking software), which was installed through an automated installation system called Installatron. The new host doesn't support Installatron, they only support another software installation system called Softaculous.

Softaculous has a page which talks about restoring the backup of an installation from a downloaded backup file. Unfortunately, these instructions do not work when the backup is made through Installatron installation system. But, irrespective of which installation system is used to install Mantis, the database structure of Mantis remains the same (if both installation systems install same version of Mantis). And since Mantis is completely database driven, it is possible to restore a Mantis installation by restoring the database. This is what I resorted to at the end.

Using the DirectAdmin control panel of the old hosting account, I downloaded the database backup corresponding to the Mantis installation. The database backup download link can be found at DirectAdmin control panel -> Your Account -> MySQL management. The screenshot below shows the links to backup the database downloads.

Screenshot for downloading and uploading database backups

After I saved the downloaded backup in my computer, I used the DirectAdmin control panel of the new hosting account to upload the backup to the database corresponding to the Mantis installation there. Note: You have to have a new installation of Mantis in the new hosting account in order for this to work.

Once the database backup was uploaded and the data was restored, I got back all my data with the new hosting provider. Hope this helps someone.

Text mining with perl: Resources

2012-03-23T04:41:00.000+05:30

One of the exciting area of the analytics movement is text mining, where you mine relevant information from unstructured text. This is, hence, different from data mining, where you mine data for patterns. It is generally accepted that text mining is considerably harder than data mining, mostly because the source is unstructured. but the benefits are often substantial - for example, there is an investment fund which uses analysis of twitter feeds for predicting stock movement. This question on a quant Q&A site discusses a few similar applications. This is just one example application. Another example is IBM's Watson, who famously won the jeopardy in February of 2011.

Text mining and natural language processing

Text mining has close association with natural language processing (NLP). Since you have to search for information in unstructured text which are mostly text in natural languages, you end up using quite a few of the algorithms used in NLP. Recently, I have been moving a bit deeper into text mining and NLP, for a side project of mine (maybe a later blog post, but let me tell you, it is not for investment purposes :-). Text mining and NLP is a good fit for me (I am a developer of a Monte Carlo simulation software called Oracle Crystal Ball in my day job) - they use a lot of probabilities and modeling, similar to what I do for Monte Carlo simulation. There isn't much optimization and O.R. yet, but it is analytics all the way.

To that end, I am learning the tricks of the game, and a suitable language to program.

Perl for data extraction

Although I mostly work in the .NET world, often times, for some processing jobs, I revert to perl. I have used perl for many years, and love the language. Initially, I learned perl to do some simple web-CGI programming when I was in IIT Kharagpur, in my undergraduate years. Then I used perl for heavy database scripting at a job I did immediately following my undergraduate degree. I am by no means an expert in perl, but I know decent amount to get by. Since then, I have used perl in a variety of capacity: in my current work, in hobby web-CGI programming and so on. Although, lot of people have moved on to php for web-CGI and python for non-CGI scripting, I never could do so. There is nothing wrong with perl anyway, why bother?

I also think perl is a great language for people who deal with data. As a software developer for an analytic tools, I deal with data all the time, and lots of them. The strength of perl comes from its rich feature set suitable for data extraction (indeed, perl is, unofficially, an acronym for Practical Extraction and Reporting Language). I have used perl (along with unix shell e.g., bash, sed and awk) scripts for extracting data and formatting the way I need with great success.

Perl for text mining
When you think about the strengths of perl, it naturally becomes a contender for the language of choice for NLP. Perl has strong but easy to use regular expressions, and various modules for extracting anything from anywhere. Given that, and my familiarity with perl, I decided to take a look at the resources available for this purpose. Not so surprisingly, there are quite a few.

Resources for text mining
Here is my compiled list of resources, with an obvious bias towards using perl.

Books: Let's start with books. There are no dearth of books on text mining and NLP, but I will list a few ones I thought are good.

Manning and Schutze. Foundations of Statistical Natural Language Processing. The MIT Press. 1st Ed. 1999. This book is a gem, sorta bible for modern NLP. It has got nothing to do with perl, but since text mining has lot of NLP stuff, this book is a great reference.
Manu Konchady. Text Mining Application Programming. Charles River Media. 1st Ed. 2006. This is probably the best book out there to learn text mining using perl. Note that the book is not that heavy on perl code though, but stresses on developing the concepts and the models that are used in text mining. The book covers text mining by talking about various problems that appear in (and can be solved using) text mining. I liked that approach. It also helps that the author maintains an open source library for text mining written in perl.
Roger Bilisoly. Practical Text Mining with Perl. Wiley. 1st Ed. 2008. This is probably the only book that directly deals with both text mining and perl. This is also probably a pretty good book, but unfortunately it did not meet my exact needs. I found the topic covered somewhat lacking when compared to Konchady's book.
Weiss et. al. Text Mining: Predictive Methods for Analyzing Unstructured Information. Springer. 1st Ed. 2005 (Softcover - 2010). The contents of this book seemed interesting, similar to Konchady's book but the treatment more rigorous. I have put this in my "to check out" list.

Software: As with books, there are no dearth of software for text mining either. Since I was looking for perl related solutions, those are the only ones I list here.

Text mine: The open source perl library I mentioned earlier.
Text::Mining module on CPAN: This module seems comprehensive, but is rather poorly documented. It is not very clear from the documentation available in the module about how to use various sub-modules effectively. Hope the authors work on that.

Video learning: There are quite a few lectures available on Youtube. But the best one has to be an online course from Stanford, which is going on right now. The course is from Prof. Manning, the author of the first book in my list. The online course instructs the assignments to be submitted in java or python, and I understand their logic - those two being at the top of the heap of popular languages, but I have decided to use perl for the purpose of doing the assignments. I won't be graded then, but hey, I did not want to be graded in the first place ;-). The reason for choosing perl: my familiarity and knowledge of the strengths of perl in this area (although, I am familiar with java too, at almost the same level), the lure of being able to use the same code in my side project, and possibly reuse some of that stuff in a web-CGI context at some point.

I will update this post if I come across more relevant resources.