varun pant

how to install the latest python 3 on mac

2019-09-09T00:00:00z

Here is a quick guide on installing Python3 on a mac

Installation

brew update
brew upgrade
sudo mkdir -p /usr/local/Frameworks
sudo chown -R $(whoami) /usr/local/* 
brew install python3
brew link python3
brew doctor

Aliasing

echo "alias python=/usr/local/bin/python3.7" >> ~/.zshrc

echo "alias python=/usr/local/bin/python3.7" >> ~/.bashrc

You could probably have just done this as well

alias python='python3'
alias pip='pip3'

I hope this is useful

how to play apple music on alexa in uk

2019-04-10T00:00:00z

You can now use Alexa to play Apple Music on Amazon Echo speakers.

It's just available for Echo devices and now customers in UK can also connect their apple music account with Alexa.

1. Open the Alexa app then sign into your Amazon account and make sure that your Echo device is connected.

2. Open Menu (top left icon), then select settings.

3. Select Music.

4. Select "Link New Service"

5. Tap, Apple music Icon

6. You will be prompted to enable Apple music skill and asked to sign into Apple account and provide access.

7. Finally choose apple music as default music service.

download file using webdriver firefox and python selenium

2019-02-24T00:00:00z

Selenium is one of my favourite tool for automation.

In this post, I will demonstrate some basic code to download a file from a website in a headless mode , and also provide a docker file to make things simpler.

Python Code

Here is some basic code which will make an attempt to download a 7zip exe.

from pyvirtualdisplay import Display
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import os
import time

print("******************************** STARTING ********************************")

display = Display(visible=0, size=(1024, 768))
display.start()

# To prevent download dialog
profile = webdriver.FirefoxProfile()
profile.set_preference('browser.download.folderList', 2) # custom location
profile.set_preference('browser.download.manager.showWhenStarting', False)
profile.set_preference('browser.download.dir', '/srv/download')
profile.set_preference("browser.download.manager.alertOnEXEOpen", False)
profile.set_preference("browser.download.manager.closeWhenDone", False)
profile.set_preference("browser.download.manager.focusWhenStarting", False)
#application/octet-stream,application/vnd.ms-excel 
profile.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/x-msdownload,application/octet-stream')
try:
    browser = webdriver.Firefox(profile)
    browser.get('https://www.7-zip.org/')
    download_button = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'td.Item a')))
    download_button.click()
    print("clicked...")
    time.sleep(10) 
    print (os.listdir("/srv/download"))
except Exception as ex:
    print(ex)
 
browser.close()
display.stop()


print("******************************** FINITO ********************************")

The code is fairly simple , we need

pyvirtualdisplay, to support the webdriver .
FirefoxProfile, to auto accept the firefox download model screen.
The code sets /srv/download as the download location, more details in Makefile shown in the end of this post.
Finally, we set 'application/x-msdownload,application/octet-stream' as our allowed file mime type.

Python dependencies

selenium
pyvirtualdisplay

pip install selenium pyvirtualdisplay

DockerFile

FROM python:2.7-stretch


RUN apt-get update && apt-get install -yq \
    firefox-esr \
    chromium \
    git-core \
    xvfb \
    xsel \
    unzip \
    python-pytest \
    libgconf2-4 \
    libncurses5 \
    libxml2-dev \
    libxslt-dev \
    libz-dev \
    xclip=0.12+svn84-4+b1


# GeckoDriver v0.19.1
RUN wget -q "https://github.com/mozilla/geckodriver/releases/download/v0.19.1/geckodriver-v0.19.1-linux64.tar.gz" -O /tmp/geckodriver.tgz \
    && tar zxf /tmp/geckodriver.tgz -C /usr/bin/ \
    && rm /tmp/geckodriver.tgz


# chromeDriver v2.35
RUN wget -q "https://chromedriver.storage.googleapis.com/2.35/chromedriver_linux64.zip" -O /tmp/chromedriver.zip \
    && unzip /tmp/chromedriver.zip -d /usr/bin/ \
    && rm /tmp/chromedriver.zip


# create symlinks to chromedriver and geckodriver (to the PATH)
RUN ln -s /usr/bin/geckodriver /usr/bin/chromium-browser \
    && chmod 777 /usr/bin/geckodriver \
    && chmod 777 /usr/bin/chromium-browser


RUN pip install selenium pyvirtualdisplay

Running python code inside Docker container

A simple command as shown below will execute the script in docker, which will create a folder locally called download and download the 7zip.exe into it.

#Build image from Dockerfile
docker build -t docker-selenium
#Run image after mounting the folder at "${PWD}:/srv" and mapping it with /srv in docker container.
docker run -v "${PWD}:/srv"  -w /srv --name img -d docker-selenium tail -f /dev/null
# I have used a make file locally to execute python command python run main.py
docker exec img make run
docker stop img
docker rm img

Makefile

Here is the make file which will makes things easier.

img_name := docker-selenium


deps:
	pip install selenium pyvirtualdisplay


run:
	python main.py


docker-run:
	docker run -v "${PWD}:/srv"  -w /srv --name img -d $(img_name) tail -f /dev/null
	docker exec img make run
	docker stop img
	docker rm img


build:
	docker build -t $(img_name) .

Chorme in UI Mode

from selenium import webdriver
import os
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


browser = webdriver.Chrome()
browser.get('https://www.7-zip.org/')
download_button = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'td.Item a')))
download_button.click()
time.sleep(10)

The Code above is just a snippet if someone needs to quickly download any file from web url using selenium.

Hope this helps.

web crawling or scraping using scrapy in python

2018-12-02T00:00:00z

Scrapy is a very popular web scraping/crawling framework, I have been using it for quite some time now.

In this post, I will demonstrate creating a very basic web crawler.

Install Scrapy

Installation is via pip pip install scrapy

Minimalistic Code

A very simple scraper is created like this

import scrapy
class MySpider(scrapy.Spider):
    name = "MySpider"
    start_urls = ['https://varunpant.com'] 
    def parse(self, response):
        # Get page title using xpath.
        page_title = response.xpath('//title/text()').extract_first()
        print(page_title)

To Run , simply type scrapy runspider scraper.py

Running, above code will output something like below

2018-12-02 14:01:18 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: scrapybot)
2018-12-02 14:01:18 [scrapy.utils.log] INFO: Versions: lxml 4.2.5.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.1, w3lib 1.19.0, Twisted 18.9.0, Python 2.7.15 (default, Oct  2 2018, 11:42:04) - [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.2)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0j  20 Nov 2018), cryptography 2.4.2, Platform Darwin-18.2.0-x86_64-i386-64bit
2018-12-02 14:01:18 [scrapy.crawler] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True}
2018-12-02 14:01:18 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2018-12-02 14:01:18 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-12-02 14:01:18 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-12-02 14:01:18 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-12-02 14:01:18 [scrapy.core.engine] INFO: Spider opened
2018-12-02 14:01:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-12-02 14:01:18 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-12-02 14:01:18 [scrapy.core.engine] DEBUG: Crawled (200) <get https:="" varunpant.com=""> (referer: None)
2018-12-02 14:01:18 [scrapy.core.engine] INFO: Closing spider (finished)
2018-12-02 14:01:18 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 212,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 12403,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2018, 12, 2, 14, 1, 18, 850998),
 'log_count/DEBUG': 2,
 'log_count/INFO': 7,
 'memusage/max': 50581504,
 'memusage/startup': 50577408,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2018, 12, 2, 14, 1, 18, 570736)}
2018-12-02 14:01:18 [scrapy.core.engine] INFO: Spider closed (finished)
Varun Pant Blog | Index

In the last line, one can see that the page title Varun Pant Blog | Index was printed.

The principle of a web scraper is generally quite simple,

Start from a seed url and extract all hyperlinks.
Then crawl each of them, further extracting links until all of them have been visited.

In Scrapy start_urls is the list of seed urls to begin scraping.

One can also provide another varibale to add domain constraint. allowed_domains = ["varunpant.com"] so that links which lead out of the principal domain are rejected.

Scrappy takes care of ensuring that links which have already been visited are not visited again.

Extracting Links

In the simple example above, the code does not extract hyperlinks from the response body, lets modify it to do so.

import scrapy
class MySpider(scrapy.Spider):
    name = "MySpider"
    start_urls = ['https://varunpant.com'] 
    def parse(self, response):
        all_links = response.xpath('*//a/@href').extract()
        for link in all_links:
            print(link)

Line all_links = response.xpath('*//a/@href').extract() , uses xpath and extracts all hyperlinks in the page.

/
/contact
/archives
/feed
/posts/integration-testing-with-apache-beam-using-pubsub-and-bigtable-emulators-and-direct-runner
http://varunpant.com/posts/integration-testing-with-apache-beam-using-pubsub-and-bigtable-emulators-and-direct-runner#disqus_thread
/posts/gdal-2-on-mac-with-homebrew
http://varunpant.com/posts/gdal-2-on-mac-with-homebrew#disqus_thread
/posts/how-to-print-bar-chart-in-chrome-browser-console
http://varunpant.com/posts/how-to-print-bar-chart-in-chrome-browser-console#disqus_thread
/posts/how-to-make-https-requests-with-python-httplib2-ssl
http://varunpant.com/posts/how-to-make-https-requests-with-python-httplib2-ssl#disqus_thread
/posts/how-to-read-json-from-web-http-request-of-url-via-python-urllib
http://varunpant.com/posts/how-to-read-json-from-web-http-request-of-url-via-python-urllib#disqus_thread
/posts/minimum-insertions-to-form-a-palindrome
http://varunpant.com/posts/minimum-insertions-to-form-a-palindrome#disqus_thread
/posts/inverse-weighted-distance-interpolation-in-golang
http://varunpant.com/posts/inverse-weighted-distance-interpolation-in-golang#disqus_thread
/posts/basic-sorting-algorithms-implemented-in-golang
http://varunpant.com/posts/basic-sorting-algorithms-implemented-in-golang#disqus_thread
/posts/reading-and-writing-binary-files-in-go-lang
http://varunpant.com/posts/reading-and-writing-binary-files-in-go-lang#disqus_thread
/posts/create-linear-color-gradient-in-go
http://varunpant.com/posts/create-linear-color-gradient-in-go#disqus_thread
/?page=2
http://stackoverflow.com/users/95967
https://www.linkedin.com/in/varunpant
mailto:varun@varunpant.com
https://github.com/varunpant

The final step is to make the parse generator method yield response like so

import scrapy

class MySpider(scrapy.Spider):
    name = "MySpider"
    start_urls = ['https://varunpant.com'] 
    allowed_domains = ["varunpant.com"]
    def parse(self, response):
        page_title = response.xpath('//title/text()').extract_first()
        print(page_title)
        all_links = response.xpath('*//a/@href').extract()
        for link in all_links:
            yield scrapy.Request(
                    response.urljoin(link),
                    callback=self.parse
                )

This should crawl all hyperlinks found with each page.

Save all crawlled pages to link

Here is a minimalistic code snippet, which saves all crawlled pages to the disk.

import scrapy
import unicodedata
import re
from pprint import pprint

class MySpider(scrapy.Spider):
    name = "MySpider"
    allowed_domains = ["varunpant.com"]
    start_urls = ['https://varunpant.com']

    def slugify(self,value): 
        value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore')
        value = unicode(re.sub('[^\w\s-]', '', value).strip().lower())
        value = unicode(re.sub('[-\s]+', '-', value))
        return value

    def save(self,name,content):
         #save to html in the pages folder
        p = "pages/%s.html"
        fn = self.slugify(name)
        with open(p%name,"w") as f:
            f.write(content.encode("utf-8"))

    def parse(self, response):
        
        page_title = response.xpath('//title/text()').extract_first()
        page_body = response.body.decode("utf-8")
        self.save(page_title,page_body)
        all_links = response.xpath('*//a/@href').extract()
        for link in all_links:
            yield scrapy.Request(
                    response.urljoin(link),
                    callback=self.parse
                )

integration testing with apache beam using pubsub and bigtable emulators and direct runner

2018-08-03T00:00:00z

Summary

Recently I have been looking into ways to test my Apache Beam pipelines at work. Most common use cases of Beam generally involves either batch reading data from GCS and writing to analytical platforms such as Big Query or stream reading data from Pubsub and writing to perhaps Bigtable.

A pipelines consists of transforms and its generally easy to test them in isolation as a independent unit test per stage, however I am personally a big fan of "end-to-end" testing or “Integration testing” and this is where things can sometimes get tricky.

Apache beam has various runners, the" one of interest" for testing purposes is the Direct runner.

From the doc:

The Direct Runner executes pipelines on your machine and is designed to validate that pipelines adhere to the Apache Beam model as closely as possible. Instead of focusing on efficient pipeline execution, the Direct Runner performs additional checks to ensure that users do not rely on semantics that are not guaranteed by the model. Some of these checks include:

enforcing immutability of elements.

enforcing encodability of elements.

elements are processed in an arbitrary order at all points.

serialization of user functions (DoFn, CombineFn, etc.)

It’s also important to realise the memory considerations as mentioned below.

Local execution is limited by the memory available in your local environment. It is highly recommended that you run your pipeline with data sets small enough to fit in local memory. You can create a small in-memory data set using a Create transform, or you can use a Read transform to work with small local or remote files.

I am going to show an example where I have written a basic integration test which listens for some payload from pubsub emulator, transforms the data to Mutation and then writes it to BigTable emulator, there are no aggregations performed hence every thing works in global window and triggering of a write is immediate. This example can also serve as a good way to front your data warehouse with apache beam for dynamically scalable writing,i.e as the pubsub message load would increase, beam would add more workers and as the load would decrease beam would reduce the workers.

SetUp PubSub Emulator

Guide is here.

Code

gcloud components install pubsub-emulator
gcloud components update
gcloud beta emulators pubsub start [options]

optional step

$(gcloud beta emulators pubsub env-init)

You don’t need to run the optional step above as we would supply the url in the beam options.

SetUp BigTable Emulator

Guide is here.

Code

gcloud components update beta
gcloud beta emulators bigtable start

This step is required if you want to use cbt to point to BigTable to browse the data.

$(gcloud beta emulators bigtable env-init)

Code

Here are the steps involved in the pipeline.

  		Pipeline p = Pipeline.create(options);

        p.apply("ReadPubsubMessages",
               
               PubsubIO.readMessages().fromSubscription(options.getSubscription()))

               .apply("ConvertMessageToBTMutation", new PubsubMessageToBigTableMutation())


                .apply("WriteToBigTable",

                        CloudBigtableIO.writeToTable(

                                getConfigurationForTable(options).withTableId(BIGTABLE_TABLE_ID).build()
                ));

        p.run().waitUntilFinish();

The pipeline is triggered via main method which is extended to include a testing variable

public interface EventListenerOptions extends StreamingOptions, PubsubOptions {
  
        @Description("Pub/Sub subscription to read from")
        @Validation.Required
        @Default.String("projects/test-project/subscriptions/evtsToBigTbl")
        String getSubscription();

        void setSubscription(String value);

        @Description("The Google Cloud project ID for the Cloud Bigtable instance.")
        @Validation.Required
        @Default.String("BT-PROD-PROJECT")
        String getBigtableProjectId();

        void setBigtableProjectId(String bigtableProjectId);

        @Description("The Google Cloud Bigtable instance ID .")
        @Validation.Required
        @Default.String("BT-PROD-INSTANCE")
        String getBigtableInstanceId();

        void setBigtableInstanceId(String bigtableInstanceId);


        @Description("For integration test.")
        @Validation.Required
        @Default.Boolean(true)
        Boolean getTesting();

        void setTesting(Boolean testing);


    }

Insure that the project and instance id for bigtable matches those in the cofiguration file ~/.cbtrc for bigtable, you can check this by typing in cbt in the console.

 EventListenerOptions options = PipelineOptionsFactory.fromArgs(args).withValidation()
                .as(EventListenerOptions.class);
        options.setProject("PubSubToBigTable");

        if (options.getTesting()) {

            options.setPubsubRootUrl(PUBSUB_EMULATOR_HOST);//must point to emulator http://localhost:8085
            options.setBigtableProjectId(BIGTABLE_PROJECT_ID);//must be the same as in ~/.cbtrc
            options.setBigtableInstanceId(BIGTABLE_INSTANCE_ID);//must be the same as in ~/.cbtrc

        }

        RunPipeLine(options);

The intreseting bits are as follows

Override PubsubOptions url in Beam

This was easy, just make sure your options extend org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions. This includes a method called options.setPubsubRootUrl, whcih then can be pointed to emulator.

Point beam to BigTable Emulator.

This took me ages to figure out and eventually after digging in the code I was able to locate a bunch of properties which must be overridden.

private static CloudBigtableTableConfiguration.Builder getConfigurationForTable(EventListenerOptions options) {

       CloudBigtableTableConfiguration.Builder config = new CloudBigtableTableConfiguration.Builder()
                .withProjectId(options.getBigtableProjectId())
                .withInstanceId(options.getBigtableInstanceId());
                
        if (options.getTesting()) {
            config.withConfiguration("google.bigtable.instance.admin.endpoint.host", "localhost")
                    .withConfiguration("google.bigtable.admin.endpoint.host", "localhost")
                    .withConfiguration("google.bigtable.endpoint.host", "localhost")
                    .withConfiguration("google.bigtable.endpoint.port", "8086")
                    .withConfiguration("google.bigtable.use.plaintext.negotiation", "true")
                    .withConfiguration("google.bigtable.grpc.read.partial.row.timeout.ms", "5000");
        }

        return config;


    }

and thats it, calling StarterPipeline.main(new String[]{"--testing=true", "--runner=DirectRunner"}); should do the trick.

I will soon include full code.

gdal 2 on mac with homebrew

2018-02-13T00:00:00z

GDAL is one of the most frequently used utility in my toolkit. I am writing this post to make it easier for others to install it from scratch on their macs.

Setting up GDAL

The traditional way has always been to visit the dear old kyngchaos.com, and install “GDAL Complete” Framework vi deb installer. Do make sure that GDAL Framework is in your path otherwise something like this always helps

echo 'export PATH=/Library/Frameworks/GDAL.framework/Programs:$PATH' >> ~/.bash_profile source ~/.bash_profile

To test your installation, run the Terminal command gdalinfo --version.

VIA Homebrew

Below is the recipe to upgrade via homebrew to latest GDAL 2.

Update

gdal2 is now a proper formulae brew install gdal2 should work.

If you have already installed GDAL via gdal formula brew install gdal, then you need to do the following to upgrade.

brew unlink gdal
brew tap osgeo/osgeo4mac && brew tap --repair
brew install jasper netcdf # gdal dependencies
brew install gdal2 --with-armadillo \
--with-complete --with-libkml --with-unsupported
brew link --force gdal2

Finally Verify

$> gdal-config --version
2.1.1
$> gdal-config --libs
-L/usr/local/Cellar/gdal2/2.1.1/lib -lgdal
$> gdal-config --cflags
-I/usr/local/Cellar/gdal2/2.1.1/include

how to print bar chart in chrome browser console

2018-02-05T00:00:00z

This post doesn't really have anything valuable to contribute, just some cool console trick.

Have you ever wanted to plot out a chart, very quickly ? Did you ever had an urge to visualise a bunch of numbers without having to use a charting api or copy pasting the data in a spreadsheet ? If you did then you might even learn something today :)

Here is a simple , yet neat way to plot out bunch of numbers in chrome console as a horizontal bar chart.

for(i = 5;i<20;i++){console.log(`%c ${Array(Math.round(i*2)).join('█')}`, 'color: crimson');}

Hope you like it as much as I do as well :)

how to make https requests with python httplib2 ssl

2018-02-02T00:00:00z

Here are few snippets to make secure http requests using various python libraries.

httplib2

import httplib2

link = "https://example.com
h = httplib2.Http(".cache")
r, content = h.request(link, "GET")

another exmaple

import httplib2

h = httplib2.Http(".cache")
h.add_credentials('user', 'pass')
r, content = h.request("https://api.github.com", "GET")

print r['status']
print r['content-type']

Urllib2

Here is a simmilar example using urlib2 for comparison and lines of code.

import urllib2

gh_url = 'https://example.com'

auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password(None, gh_url, 'user', 'password')

opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
handler = urllib2.urlopen(gh_url)

print handler.getcode()
print handler.headers.getheader('content-type')

Requests

The easiest, has always been requests.

import requests

r = requests.get('https://example.com')
#r = requests.get('https://example.com', auth=('user', 'pass'))
print r.status_code
print r.headers['content-type']

Hope this helps.

how to read json from web http request of url via python urllib

2018-01-30T00:00:00z

I generally am looking for a quick snippet to issue an http request using urllib2 lib.

Here is a quick snippet to do so

url = "https://api.waqi.info/mapq/bounds/?bounds=%20-180.0,-85.0,180.0,85.0"

import urllib2
import json
from pprint import pprint

req = urllib2.Request(url)
opener = urllib2.build_opener()
f = opener.open(req)

#data varibale recieves the parsed json
data = json.loads(f.read())


print len(data)
for r in data:
    pprint(r)
    print "-"*50

Hope this helps

minimum insertions to form a palindrome

2017-10-15T00:00:00z

Brute-force approach

Here I present a few approaches to deduce "minimum insertions" required to convert a string into a palindrome.

The basic brute force approach is quite simple, given a string with length L, start comparing, the first character from left and the last character while scanning inwards.

Here is a basic test for a palindrome.

def ispalindrome(s):
    L = len(s) 
    for i in range(L):
        if s[i] != s[L - i - 1]:
            return False,i,L-i -1
    return True,0,0

The above code returns True if the string is a palindrome or returns False with mismatching indices.

result,left,right = ispalindrome("aba") will return True,0,0 while result,left,right = ispalindrome("abc") will return False,0,2 indicating the characters at 0^th position, a, does not match with character at 2^nd position, c

Once the indices are found a basic recursive solution can be written to reflect the missing strings from the mismatching indices.

def to_palendrome_left_reflect(s):
    p,l,r = ispalindrome(s)
    if not p: 
        if l == 0:
            s = s[r] + s
        else:
            s = s[0:l]+ s[r] + s[l:]
        return to_palendrome_left_reflect(s)
    else: 
        return s
        
def to_Palendrome_rigth_reflect(s):
    p,l,r = ispalindrome(s)
    if not p: 
        if l == 0:
            s =  s + s[l]
        else:
            s = s[0:r+1] + s[l] + s[r+1:]
        return to_Palendrome_rigth_reflect(s)
    else: 
        return s

Some tests are as follows.

print to_palendrome_left_reflect("ab") #bab
print to_palendrome_left_reflect("abc")#cbabc

print to_Palendrome_rigth_reflect("ab") #aba
print to_Palendrome_rigth_reflect("abc")#abcba

To find which is smaller we write a quick function as shown below.

def find_Minimum_Insertions(s):
    L = len(s)
    left_insertion = to_palendrome_left_reflect(s)
    right_insertion = to_Palendrome_rigth_reflect(s)
    delta_l = len(left_insertion) - L
    delta_r = len(right_insertion) - L

    print "LEFT:",left_insertion
    print "RIGHT:",right_insertion
    if delta_l < delta_r:
        print "MIN-LEFT",delta_l
    else:
        print "MIN-RIGHT",delta_r

Its important to consider that above mentioned techniques ignores deletion or susbtitution.

Another approach which only calculates count, while considering all possible operations.

def findMinInsertions(s,start,end):
    
    if start == end:
        return 0
    if start == end -1:
        if s[start] == s[end]:
            return 0
        return 1
    if s[start] == s[end]:
        return findMinInsertions(s,start+1,end - 1)
    else:
        a = findMinInsertions(s,start,end - 1)
        b = findMinInsertions(s,start+1,end) 
        return min(a,b)+1

Dynamic Programming approach (fastest)

def findMin_Insertions_count_Dynamicly(s):
    n = len(s)
    table = [[0 for i in range(n)]  for j in range(n)]
    start = 0
    end = 0 
    for y in range(0,n-1):
        a = 0
        for  x in range(n):
            if x>y-1:
                table[y][x] = a
                a+=1
                 
    return table[0][n-1]

This, by far is the fastest of all methods, but only works for string which are not already apalindrome , the trick is to have a n x n matrices and then fill in a particual order from left to right, increasing the gaps per row as shown below, for string abcde

a b c d e
----------
0 1 2 3 4
0 0 1 2 3 
0 0 0 1 2 
0 0 0 0 1 
0 0 0 0 0

The M[0][n-1] is 3

Hope this helps.

Inverse weighted distance interpolation in golang

2017-09-07T00:00:00z

Inverse distance weighting (IDW) is a type of deterministic method for multivariate interpolation with a known scattered set of points. The assigned values to unknown points are calculated with a weighted average of the values available at the known points.

This technique, explicitly makes the assumption that things that are close to one another are more alike than those that are farther apart. To predict a value for any unmeasured location, IDW uses the measured values surrounding the prediction location. The measured values closest to the prediction location have more influence on the predicted value than those farther away. IDW assumes that each measured point has a local influence that diminishes with distance. It gives greater weights to points closest to the prediction location, and the weights diminish as a function of distance, hence the name inverse distance weighted.

Here is an implementation in golang. The code reads from shapefile and then interpolates a 100 x 100 grid.

package main

import (
	"fmt"

	"github.com/lukeroth/gdal"
	"math"
	"image"
	"image/color"
	"os"
	"image/png"
)

type point struct {
	x      int
	y      int
	weight float64
}

func readShapeFile(path string, rows int, cols int) ([]point, [6]float64) {

	dataset := gdal.OpenDataSource(path, 0)
	defer dataset.Release()

	layer := dataset.LayerByIndex(0)

	extent, _ := layer.Extent(true)
	geotransform := [6]float64{}
	geotransform[0] = extent.MinX()
	geotransform[1] = (extent.MaxX() - extent.MinX()) / float64(cols)
	geotransform[2] = 0
	geotransform[3] = extent.MaxY()
	geotransform[4] = 0
	geotransform[5] = (extent.MinY() - extent.MaxY()) / float64(rows)

	fc, _ := layer.FeatureCount(true)

	var points []point = make([]point, fc)
	for i := 0; i < fc; i++ {
		feature := layer.Feature(i)
		x, y, _ := feature.Geometry().Point(0)
		X := int((x - geotransform[0]) / geotransform[1])
		Y := int((y - geotransform[3]) / geotransform[5])
		points[i] = point{X, Y, feature.FieldAsFloat64(1)}

	}
	return points, geotransform

}

func iwdistance(x int, y int, points []point, power float64, smoothing int) float64 {
	L := len(points)
	nominator := 0.0
	denominator := 0.0
	for i := 0; i < L; i++ {
		pt := points[i]
		dist := math.Sqrt(float64((x-pt.x)*(x-pt.x) + (y-pt.y)*(y-pt.y) + smoothing*smoothing));
		if dist < 0.0000000001 {
			return pt.weight
		}
		nominator = nominator + (pt.weight / math.Pow(dist, power))
		denominator = denominator + (1 / math.Pow(dist, power))
	}

	value := 0.0
	if denominator > 0 {
		value = nominator / denominator
	} else {
		value = -9999
	}
	return value
}

func drawGeoTiff(points []float32, cols int, rows int, geotransform [6]float64) {

	driver, err := gdal.GetDriverByName("GTiff")
	if err != nil {
		fmt.Println(err.Error())
		return
	}

	dataset := driver.Create("idw.tif", cols, rows, 1, gdal.Float32, nil)
	defer dataset.Close()

	spatialRef := gdal.CreateSpatialReference("")
	spatialRef.FromEPSG(4326)
	srString, err := spatialRef.ToWKT()
	dataset.SetProjection(srString)
	dataset.SetGeoTransform(geotransform)
	raster := dataset.RasterBand(1)
	raster.SetNoDataValue(-1)
	raster.IO(gdal.Write, 0, 0, cols, rows, points, cols, rows, 0, 0)

}

func HSVtoRGB(h float64, s float64, v float64) (uint8, uint8, uint8) {

	i := math.Floor(h * 6);
	f := h*6 - i;
	p := v * (1 - s);
	q := v * (1 - f*s);
	t := v * (1 - (1-f)*s);
	r := 0.0
	g := 0.0
	b := 0.0
	switch int(i) % 6 {
	case 0:
		{
			r = v
			g = t
			b = p
		}
	case 1:
		{
			r = q
			g = v
			b = p
		}
	case 2:
		{
			r = p
			g = v
			b = t
		}
	case 3:
		{
			r = p
			g = q
			b = v
		}
	case 4:
		{
			r = t
			g = p
			b = v
		}
	case 5:
		{
			r = v
			g = p
			b = q
		}
	}
	return uint8(math.Floor(r * 255)), uint8(math.Floor(g * 255)), uint8(math.Floor(b * 255))
}

func getRGB(value float64) (uint8, uint8, uint8) {

	h := 10 - value/9;
	return HSVtoRGB(h, 1, 1)
}
func drawRaster(points []float32, cols int, rows int, name string) {
	dst := image.NewRGBA(image.Rect(0, 0, cols, rows))

	for x := 0; x < cols; x++ {
		for y := 0; y < rows; y++ {
			pw := float64(points[x+y*cols])
			r, g, b := getRGB(pw)

			c := color.RGBA{

				r,
				g,
				b,
				255,
			}

			dst.Set(x, y, c)
		}
	}

	img, _ := os.Create(name)
	defer img.Close()
	png.Encode(img, dst)
}

func main() {

	w, h := 100, 100
	points, geotransform := readShapeFile("shp/points.shp", w, h)

	var power float64 = 2
	var smoothing int = 0

	buffer := make([]float32, w*h)
	for x := 0; x < w; x++ {
		for y := 0; y < h; y++ {
			pw := iwdistance(x, y, points, power, smoothing)
			buffer[x+y*w] = float32(pw)
		}
	}

	drawGeoTiff(buffer, w, h, geotransform)
	drawRaster(buffer, w, h, "idw.png")

}

The results looks like this

Geotiff

coloured png

basic sorting algorithms implemented in golang

2017-09-04T00:00:00z

This post includes go-lang based implementation of some of the classic sorting algorithms.

This article primarily, has been written, as an academic exercise, to not forget the basic principles of sorting.

Bubble Sort

wiki

Bubble sort is perhaps the simplest of all sorting algorithms, the basic principle is to bubble up the largest (or the smallest) and then the second largest and then the third largest and so on. Each bubble takes the full sweep of the list, hence best-case, average-case as well as the worst-case performance are all O(n²)

package main

func BubbleSort( items []int)  {

   L:=len(items)

   for  i:=0;i<L;i++{

      for j:=0;j< (L-1-i);j++{
         if items[j] > items[j+1]{
            items[j], items[j+1] = items[j+1], items[j]
         }
      }
   }

}

Insertion Sort

wiki

Insertion sort takes each element through the list and inserts it at the right place. At each iteration, insertion sort removes one element from the input data, finds the location it belongs within the sorted list, and inserts it there. It repeats until no input elements remain. Worst-case and average-case performance is O(n2), however the best case is O(n).

package main

func InsertionSort(items []int) {

   L := len(items)

   for i := 1; i < L; i++ {

      j := i
      for j > 0 && items[j] < items[j-1] {
         items[j], items[j-1] = items[j-1], items[j]
         j -= 1
      }

   }

}

Merge Sort

wiki

Mergesort operates by diving the list into two sublist recursively till each sublist contains only one element, it then merges the adjacent list back while sorting each element in the merged list. Best-case, average-case as well as the worst-case eprformance are all O(nlogn)

package main

const NADA int = -1

func DeepCopy(vals []int) []int {
   tmp := make([]int, len(vals))
   copy(tmp, vals)
   return tmp
}

func MergeSort(items []int) {

   if len(items) > 1 {
      mid := len(items) / 2
      left := DeepCopy(items[0:mid])
      right := DeepCopy(items[mid:])

      MergeSort(left)
      MergeSort(right)

      l := 0
      r := 0

      for i := 0; i < len(items); i++ {

         lval := NADA
         rval := NADA

         if l < len(left) {
            lval = left[l]
         }

         if r < len(right) {
            rval = right[r]
         }

         if (lval != NADA && rval != NADA && lval < rval) || rval == NADA {
            items[i] = lval
            l += 1
         } else if (lval != NADA && rval != NADA && lval >= rval) || lval == NADA {
            items[i] = rval
            r += 1
         }

      }
   }

}

Quicksort Sort

wiki

Quick sort works by first selecting a pivot element from the list. It then creates two lists, one containing elements less than the pivot and the other containing elements higher than the pivot. It then sorts the two lists and join them with the pivot in between. Just like the Merge sort, when the lists are subdivided to lists of size 1, they are considered as already sorted. Best-case and avergae-case is O(n log n) however the worstcase(very rare) is O(n²)

func QuickSort(items []int) {

   if len(items) > 1 {
      pivot_index := len(items) / 2
      var smaller_items = []int{}
      var larger_items = []int{}

      for i := range items {
         val := items[i]
         if i != pivot_index {
            if val < items[pivot_index] {
               smaller_items = append(smaller_items, val)
            } else {
               larger_items = append(larger_items, val)
            }
         }
      }

      QuickSort(smaller_items)
      QuickSort(larger_items)

      var merged []int = append(append(append([]int{}, smaller_items...), []int{items[pivot_index]}...), larger_items...)
      //merged := MergeLists(smaller_items, items[pivot_index], larger_items)

      for j := 0; j < len(items); j++ {
         items[j] = merged[j]
      }

   }

}

Heap Sort

wiki

Heapsort can be thought of as an improved selection sort: like that algorithm, it divides its input into a sorted and an unsorted region, and it iteratively shrinks the unsorted region by extracting the largest element and moving that to the sorted region. The improvement consists of the use of a heap data structure rather than a linear-time search to find the maximum,best-case, average-case as well as the worst-caseeprformance are all O(nlogn)

package main


func heapify(items []int, idx int, size int) {
   l := 2*idx + 1 // left = 2*i + 1
   r := 2*idx + 2 // right = 2*i + 2

   var largest int;
   if l <= size && items[l] > items[idx] {
      largest = l
   } else {
      largest = idx
   }

   if r <= size && items[r] > items[largest] {
      largest = r
   }

   if largest != idx {
      t := items[idx]
      items[idx] = items[largest]
      items[largest] = t
      heapify(items, largest, size)
   }

}

func HeapSort(items []int) {

   L := len(items) //heap size
   m := int(L / 2) //middle

   for i := m; i >= 0; i-- {

      heapify(items, i, L-1)
   }

   F := L - 1
   for j := F; j >= 0; j-- {
      t := items[0]
      items[0] = items[j]
      items[j] = t
      F -= 1
      heapify(items, 0, F)
   }

}

Finally to run these algos I have written a quick main program

import (
  "fmt"
  "time"
  "math/rand"
)

func main() {

  fmt.Println("Preparing!")
  var randomVals = []int{}
  s1 := rand.NewSource(time.Now().UnixNano())
  r1 := rand.New(s1)
  for i := 0; i < 11; i++ {
    randomVals = append(randomVals, r1.Intn(99)+1)
  }

  fmt.Println("Unsorted!")
  fmt.Println(randomVals)
  //BubbleSortrandomVals()
  //InsertionSort(randomVals)
  //MergeSort(randomVals)
  //QuickSort(randomVals)
  HeapSort(randomVals)

  fmt.Println("sorted!")
  fmt.Println(randomVals)
}

finally some nice visuals here

reading and writing binary files in go lang

2017-08-29T00:00:00z

Reading and writing binary files can often be a fast and very efficient alternatives to csv. They obviously have their challenges, however in this post I intent to present a very basic example of saving models (struct) into binary file and later reading it.

package main

import (
	"log"
	"os"
	"encoding/binary"
	"bytes"
	"fmt"
	"math/rand"
	"time"
)

//this type represnts a record with three fields
type payload struct {
	One float32
	Two float64
	Three uint32
}

func main() {

	writeFile()
	readFile()

}

func readFile() {

	file, err := os.Open("test.bin")
	defer file.Close()
	if err != nil {
		log.Fatal(err)
	}

	m := payload{}
	for i := 0; i < 10; i++ {
		data := readNextBytes(file, 16)
		buffer := bytes.NewBuffer(data)
		err = binary.Read(buffer, binary.BigEndian, &m)
		if err != nil {
			log.Fatal("binary.Read failed", err)
		}

		fmt.Println(m)
	}

}

func readNextBytes(file *os.File, number int) []byte {
	bytes := make([]byte, number)

	_, err := file.Read(bytes)
	if err != nil {
		log.Fatal(err)
	}

	return bytes
}

func writeFile() {
	file, err := os.Create("test.bin")
	defer file.Close()
	if err != nil {
		log.Fatal(err)
	}

	r := rand.New(rand.NewSource(time.Now().UnixNano()))

	for i := 0; i < 10; i++ {


		s := &payload{
			r.Float32(),
			r.Float64(),
			r.Uint32(),
		}
		var bin_buf bytes.Buffer
		binary.Write(&bin_buf, binary.BigEndian, s)
		//b :=bin_buf.Bytes()
		//l := len(b)
		//fmt.Println(l)
		writeNextBytes(file, bin_buf.Bytes())

	}
}
func writeNextBytes(file *os.File, bytes []byte) {

	_, err := file.Write(bytes)

	if err != nil {
		log.Fatal(err)
	}

}

The code presented above will randomly populate 10 records of type payload and save it to a binary file, then it will read those 10 records as well.

Performance

To see how long it takes to create and then read 10000000 records try changeing the 10 to 10000000 and then building an executable by running go build filename.go, once built execute by typing time ./filename

on my macbook pro it took 0m35.433s to generate and then read all records (without priniting them on shell)

real    0m35.433s
user    0m16.713s
sys     0m19.031s

hope this helps.

Create linear color gradient in go

2017-08-28T00:00:00z

Go doesn't have any builtin gradient functions to paint/fill the background of a raster, however it has all the required primitives one needs to create a very basic linear gradient.

Here is a basic code which defines a 256 x 256 image, then calculates the value of each pixel in the raster based on a linear interpolation formula.

package main

import (
	"image"
	"image/color"
	"image/png"
	"log"
	"os"
	"os/exec"

)
var(
	colorB = [3]float64{248, 54, 0}
	colorA = [3]float64{254, 140, 0}
)

var(
	width = 256
	height = 256
	max = float64(width)
)

func linearGradient(x, y float64) (uint8, uint8, uint8) {
	d := x/max
	r := colorA[0] + d * (colorB[0] - colorA[0])
	g :=  colorA[1] + d * (colorB[1] - colorA[1])
	b :=  colorA[2] + d * (colorB[2] - colorA[2])
	return uint8(r), uint8(g), uint8(b)
}

func main() {

	var w, h int = width, height
	dst := image.NewRGBA(image.Rect(0, 0, w, h)) //*NRGBA (image.Image interface)

	for x := 0; x < w; x++ {
		for y := 0; y < h; y++ {
			r, g, b := linearGradient(float64(x), float64(y))
			c := color.RGBA{

				r,
				g,
				b,
				255,
			}
			dst.Set(x, y, c)
		}
	}

	img, _ := os.Create("new.png")
	defer img.Close()
	png.Encode(img, dst) //Encode writes the Image m to w in PNG format.

	Show(img.Name())

}

// show  a specified file by Preview.app for OS X(darwin)
func Show(name string) {
	command := "open"
	arg1 := "-a"
	arg2 := "/Applications/Preview.app"
	cmd := exec.Command(command, arg1, arg2, name)
	err := cmd.Run()
	if err != nil {
		log.Fatal(err)
	}
}

Here is how the generated image looks like

Function linearGradient is where all the magic happens, this method takes in x and y pixel locations and then calculates the appropriate gradient value based on the maximum width (256) to produce a horizontal gradient, now if one was to change the the d := x/max to d := y/max, this now becomes a top down-gradient instead of left-right (our image is 256x256 hence max of both width and height is 256) like so

or swap colors arrays to get a differrent affect.

Circular Gradient

To get something like a circular gradient replace linearGradient function with circularRadient

func circularGradient(x, y float64) (uint8, uint8, uint8) {
	var dx, dy float64 = cenX - x, cenY - y
	d:= math.Sqrt(dx*dx+dy*dy)
	p := d/255
	 
		r := colorA[0] + p*(colorB[0]-colorA[0])
		g := colorA[1] + p*(colorB[1]-colorA[1])
		b := colorA[2] + p*(colorB[2]-colorA[2])
		return uint8(r), uint8(g), uint8(b)
	 
}

This should result in an image like below

or if you want to confine it to the circle try this

func circularGradient(x, y float64) (uint8, uint8, uint8) {
	var dx, dy float64 = cenX - x, cenY - y
	d:= math.Sqrt(dx*dx+dy*dy)
	p := d/255
	if d < 128 {
		r := colorA[0] + p*(colorB[0]-colorA[0])
		g := colorA[1] + p*(colorB[1]-colorA[1])
		b := colorA[2] + p*(colorB[2]-colorA[2])
		return uint8(r), uint8(g), uint8(b)
	}

	return 0,0,0

}

Hope this helps.

merge sort in python

2017-07-23T00:00:00z

Many useful algorithms are recursive in structure: to solve a given problem, they call themselves recursively one or more times to deal with closely related subproblems.

These algorithms typically follow a divide-and-conquer approach: they break the problem into several subproblems that are similar to the original problem

but smaller in size, solve the subproblems recursively, and then combine these solutions to create a solution to the original problem.

The divide-and-conquer paradigm involves three steps at each level of the recursion:

Divide the problem into a number of subproblems that are smaller instances of the same problem.
Conquer the subproblems by solving them recursively. If the subproblem sizes are small enough, however, just solve the subproblems in a straightforward manner.
Combine the solutions to the subproblems into the solution for the original problem.

The merge sort algorithm closely follows the divide-and-conquer paradigm. Intuitively,

it operates as follows.

Divide: Divide the n-element sequence to be sorted into two subsequences of n=2 elements each.
Conquer: Sort the two subsequences recursively using merge sort.
Combine: Merge the two sorted subsequences to produce the sorted answer.

here is a basic code in python

from __future__ import division
import sys
SENTINEL = sys.maxint

def merge(input_array,first,middle,last):
    n1 = middle - first + 1
    n2 = last - middle

    L = [None for t in range(n1+1)]
    R = [None for t in range(n2+1)]

    
    for i in range(n1):
        L[i] = input_array[first + i - 1]
        
    for j in range(n2):
        R[j] =input_array[middle + j]
         

    L[n1] = SENTINEL
     
    R[n2] = SENTINEL
     

    i = 0
    j = 0 
    for k in range(first-1,last): 
        if L[i] <= R[j]:
            input_array[k] = L[i]
            i = i + 1
        else:
            input_array[k] = R[j]
            j = j + 1



def mergeSort(input_array,first,last):
    if first < last:
        middle =  int((first + last)/2) 
        mergeSort(input_array,first,middle)
        mergeSort(input_array,middle + 1,last)
        
        merge(input_array,first,middle,last)


arr = [5,2,4,7,1,3,2,6]
 
mergeSort(arr,1,len(arr))
print arr

Hope this helps ;)

Using Geotiff as datasource via gdal

2016-12-21T00:00:00z

Recently, I have been working on algorithms which needs elevation data as well as Land Cover data, with world coverage. Google has an excellent elevation API however free usage comes with a limit.

While searching, I came across a dataset in geotiff format for landcover as well as a processed version of world elevation. Elevation data comes in various resolution (250m,500m,1km), landcover is 500m .

So how do we read it ? In python its quite easy to use osgo/gdal library and read all bands. Geotiff is a rster file and values can be packed in every band (which is basically a 2d array).

landUsageLookup = [
    "Water",  # 1
    "Evergreen Needle leaf Forest",  # 2
    "Evergreen Broadleaf Forest",  # 3
    "Deciduous Needle leaf Forest",  # 4
    "Deciduous Broadleaf Forest",  # 5
    "Mixed Forests",  # 6
    "Closed Shrublands",  # 7
    "Open Shrublands",  # 8
    "Woody Savannas",  # 9
    "Savannas",  # 10
    "Grasslands",  # 11
    "Permanent Wetland",  # 12
    "Croplands",  # 13
    "Urban and Built-Up",  # 14
    "Cropland/Natural Vegetation Mosaic",  # 15
    "Snow and Ice",  # 16
    "Barren or Sparsely Vegetated"  # 17
]

def getLandUsage(latitude, longitude):
    ds = gdal.Open("data/LCType.tif") 
    gt = ds.GetGeoTransform()
     
    band = ds.GetRasterBand(1)
    px = int((longitude - gt[0]) / gt[1])  # x pixel
    py = int((latitude - gt[3]) / gt[5])  # y pixel
    structval = band.ReadRaster(px, py, 1, 1, buf_type=gdal.GDT_UInt16)
    lc = struct.unpack('h', structval)[0]
    return lc, landUsageLookup[lc]

The code sample above takes in Lat/Lng in wgs84 and convert them into 2d index.

Hope this helps :)

How to create fishnets or geospatial grids

2016-12-11T00:00:00z

There are many use cases in GIS world, where the information has to be aggregated, an easy way to achieve this is via gridding or binning, where the area of interest is divided into small sections called grids or bins.

These sections are mostly of rectangular form (which can be easily converted into geotiffs), but in some cases even circles or hexagons are also used.

You can read a good tutorial from mapbox using Qgis with a mmqgis plugin here.

While MMQGIS is a good enough for most use cases, I was looking for a way to create fishnets or grids by giving the bbox of the region and width/height in meters, which would then make it easy to script the process for any dynamic use.

Luckily there is a "recipe" available which takes input in EPSG:3857 projections and hence takes the width/height of grid cell in meter, here.

The code is pretty easy to comprehend and creates simple polygons for each computed cell in the layer.

import os, sys
import ogr
from math import ceil


def main(outputGridfn,xmin,xmax,ymin,ymax,gridHeight,gridWidth):

    # convert sys.argv to float
    xmin = float(xmin)
    xmax = float(xmax)
    ymin = float(ymin)
    ymax = float(ymax)
    gridWidth = float(gridWidth)
    gridHeight = float(gridHeight)

    # get rows
    rows = ceil((ymax-ymin)/gridHeight)
    # get columns
    cols = ceil((xmax-xmin)/gridWidth)

    # start grid cell envelope
    ringXleftOrigin = xmin
    ringXrightOrigin = xmin + gridWidth
    ringYtopOrigin = ymax
    ringYbottomOrigin = ymax-gridHeight

    # create output file
    outDriver = ogr.GetDriverByName('ESRI Shapefile')
    if os.path.exists(outputGridfn):
        os.remove(outputGridfn)
    outDataSource = outDriver.CreateDataSource(outputGridfn)
    outLayer = outDataSource.CreateLayer(outputGridfn,geom_type=ogr.wkbPolygon )
    featureDefn = outLayer.GetLayerDefn()

    # create grid cells
    countcols = 0
    while countcols < cols:
        countcols += 1

        # reset envelope for rows
        ringYtop = ringYtopOrigin
        ringYbottom =ringYbottomOrigin
        countrows = 0

        while countrows < rows:
            countrows += 1
            ring = ogr.Geometry(ogr.wkbLinearRing)
            ring.AddPoint(ringXleftOrigin, ringYtop)
            ring.AddPoint(ringXrightOrigin, ringYtop)
            ring.AddPoint(ringXrightOrigin, ringYbottom)
            ring.AddPoint(ringXleftOrigin, ringYbottom)
            ring.AddPoint(ringXleftOrigin, ringYtop)
            poly = ogr.Geometry(ogr.wkbPolygon)
            poly.AddGeometry(ring)

            # add new geom to layer
            outFeature = ogr.Feature(featureDefn)
            outFeature.SetGeometry(poly)
            outLayer.CreateFeature(outFeature)
            outFeature.Destroy

            # new envelope for next poly
            ringYtop = ringYtop - gridHeight
            ringYbottom = ringYbottom - gridHeight

        # new envelope for next poly
        ringXleftOrigin = ringXleftOrigin + gridWidth
        ringXrightOrigin = ringXrightOrigin + gridWidth

    # Close DataSources
    outDataSource.Destroy()


if __name__ == "__main__":

    #
    # example run : $ python grid.py <full-path><output-shapefile-name>.shp xmin xmax ymin ymax gridHeight gridWidth
    #

    if len( sys.argv ) != 8:
        print "[ ERROR ] you must supply seven arguments: output-shapefile-name.shp xmin xmax ymin ymax gridHeight gridWidth"
        sys.exit( 1 )

    main( sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5], sys.argv[6], sys.argv[7] )

Here is an example usage: python grid.py grid.shp 992325.66 1484723.41 494849.32 781786.14 10000 10000

Here is an example of a 10km/10 km grid

Python scripts are generally usable in most use cases, how ever incase one needs a Postgis alternative, i.e a function which runs in Postgis database and creates a fishnet in a table, then It might be useful to use a query which I found here.

CREATE OR REPLACE FUNCTION public.makegrid_2d (
  bound_polygon public.geometry,
  grid_step integer,
  metric_srid integer = 3857 --metric SRID  
)
RETURNS public.geometry AS
$body$
DECLARE
  BoundM public.geometry; --Bound polygon transformed to the metric projection (with metric_srid SRID)
  Xmin DOUBLE PRECISION;
  Xmax DOUBLE PRECISION;
  Ymax DOUBLE PRECISION;
  X DOUBLE PRECISION;
  Y DOUBLE PRECISION;
  sectors public.geometry[];
  i INTEGER;
BEGIN
  BoundM := ST_Transform($1, $3); --From WGS84 (SRID 4326) to the metric projection, to operate with step in meters
  Xmin := ST_XMin(BoundM);
  Xmax := ST_XMax(BoundM);
  Ymax := ST_YMax(BoundM);

  Y := ST_YMin(BoundM); --current sector's corner coordinate
  i := -1;
  <>
  LOOP
    IF (Y > Ymax) THEN  --Better if generating polygons exceeds the bound for one step. You always can crop the result. But if not you may get not quite correct data for outbound polygons (e.g. if you calculate frequency per sector)
        EXIT;
    END IF;

    X := Xmin;
    <>
    LOOP
      IF (X > Xmax) THEN
          EXIT;
      END IF;

      i := i + 1;
      sectors[i] := ST_GeomFromText('POLYGON(('||X||' '||Y||', '||(X+$2)||' '||Y||', '||(X+$2)||' '||(Y+$2)||', '||X||' '||(Y+$2)||', '||X||' '||Y||'))', $3);

      X := X + $2;
    END LOOP xloop;
    Y := Y + $2;
  END LOOP yloop;

  RETURN ST_Transform(ST_Collect(sectors), ST_SRID($1));
END;
$body$
LANGUAGE 'plpgsql';

I hope this helps.

How to transform projections between Spherical Mercator and EPSG 4326

2016-12-10T00:00:00z

Projections in GIS are commonly referred to by their “EPSG” codes, these are identifiers managed by the European Petroleum Survey Group.

One common identifier is “EPSG:4326”, which describes maps where latitude and longitude are treated as X/Y values.

Spherical Mercator has an official designation of EPSG:3857. However, before this was established, a large amount of software used the identifier EPSG:900913. This is an unofficial code, but is still the commonly usedin many GIS systems.

Any time you see the string “EPSG:4326”, you can assume it describes latitude/longitude coordinates.

Any time you see the string “EPSG:900913”, it will be describing coordinates in meters in x/y.

In python its quite easy to transform coordinates using OGR and OSR tools with their python wrapper.

def transform(pointX, pointY):
    # Spatial Reference System
    inputEPSG = 4326
    outputEPSG = 3857

    # create a geometry from coordinates
    point = ogr.Geometry(ogr.wkbPoint)
    point.AddPoint(pointX, pointY)

    # create coordinate transformation
    inSpatialRef = osr.SpatialReference()
    inSpatialRef.ImportFromEPSG(inputEPSG)

    outSpatialRef = osr.SpatialReference()
    outSpatialRef.ImportFromEPSG(outputEPSG)

    coordTransform = osr.CoordinateTransformation(inSpatialRef, outSpatialRef)

    # transform point
    point.Transform(coordTransform)

    # return point in EPSG 3857
    return point.GetX(), point.GetY()

Basically you can use coordTransform to transform any geometry (point,polygon,line e.t.c) between any osr supported projections.

Hope this helps

How to Query a Shape file for Point inside a polygon using ogr python

2016-09-01T00:00:00z

Recently I was trying to build a quick geo lookup service in python, which could be used like an "info tool" in QGIS. This task is trivial in almost all geospatial databases, however I wasn't able to find much online around querying a shape file.

In this post I will demonstrate a simple python code to query a shape file which contains world countries. The file can be downloaded from here.

Here is the code

You will need python-gdal bindings. sudo apt-get install python-gdal

from osgeo import ogr
ogr.UseExceptions()
from pprint import pprint

 

path = "TM_WORLD_BORDERS_SIMPL-0.3.shp"
drv = ogr.GetDriverByName("ESRI Shapefile")
ds = drv.Open(path, 0) 

 

result = ds.ExecuteSQL("select name from 'TM_WORLD_BORDERS_SIMPL-0.3' where ST_Intersects('TM_WORLD_BORDERS_SIMPL-0.3'.geometry,makePoint(-0.2225382,51.5253007) )",dialect="SQLITE")

c = result.GetFeatureCount()
for i in range(c):
    print result.GetFeature(i).GetField(0)
 

ds.ReleaseResultSet(result)

Hope this helps.

How to read a csv file in go

2016-08-30T00:00:00z

One of the most common task for a programmer is either to read or write a csv file

Read a csv file

package main

import (
    "bufio"
    "encoding/csv"
    "os"
    "fmt"
    "io"
)

func main() {
    // Load a csv file.
    f, _ := os.Open("/path/to/my/csv/file.csv")

    // Create a new reader.
    r := csv.NewReader(bufio.NewReader(f))
    //define seperator
    r.Comma = ','
    for {
        record, err := r.Read()
        // Stop at EOF.
        if err == io.EOF {
            break
        }
        // Display record.
        fmt.Println(record)
        // ... Display record length.
        fmt.Println(len(record))
        // ... Display all individual elements of the slice.
        for value := range record {
            fmt.Printf("  %v\n", record[value])
        }
    }
}

How to integrate Google container engine with Stackdriver with kubernaties

2016-08-04T00:00:00z

Most folks have been using Grafana or something similar to monitor Kubernaties cluster.

If you are using Google cloud container engine , then the standard unified place to monitor all your application and services is Slackdriver.

According to googles documentation, when you create a new container cluster using GUI or gcloud, Slackdriver is automatically configured with a sink, however It did not work out of the box for me hence this post :)

Well, the easiest way to check if Slackdriver is configured is to issue a command

gcloud container clusters describe <cluster-name>

If the response contains something like monitoringService: monitoring.googleapis.com then yer all good and you can simply go and visit https://app.google.stackdriver.com/gke link which should point you to you cluster.

other wise issue this command to register sinks with monitoring.googleapis.com

gcloud alpha container clusters update <clustername> --monitoring-service=monitoring.googleapis.com

Hope this helps

Binary Search Tree in python

2016-08-01T00:00:00z

BST data structure supports many dynamic-set operations including

Search
Minimum
Maximum
Predecessor
Successor
Insert
Delete

These basic operations allow us to treat this data structure both as a dictionary and as a priority queue.

Basic operations on a binary tree takes time proportional to the height of the tree, O(lg n) [worst case] and even O(n) if the tree is a linear chain.

If you want to learn more about practical application of these trees check this post out.

Here I present a very basic version of binary tree written in python

class Node:
    def __init__(self,left=None,right=None,parent=None,key=None):
        self.left = left
        self.right = right
        self.parent = parent
        self.key = key
    
class BinaryTree:
    
    def __init__(self): 
        self.root = None

    def inorderWalk(self,node):
        if node is not None:
            self.inorderWalk(node.left)
            print node.key
            self.inorderWalk(node.right)

    def treeSearch(self,node,key):
        if node is None or key == node.key:
            return node
        if key < node.key:
            return self.treeSearch(node.left,key)
        else:
            return self.treeSearch(node.right,key)

    def iterativeTreeSearch(self,node,key):
        while node is not None and key != node.key:
            if key < node.key:
                node = node.left
            else:
                node = node.right

        return node

    def treeMinimum(self,node):
        while node.left is not None:
            node = node.left

        return node

    def treeMaximum(self,node):
        while node.right is not None:
            node = node.right

        return node

    def treeSuccessor(self,node):
        if node.right is not None:
            return self.treeMinimum(node.right)
        y = node.parent
        while y is not None and node == y.right:
            node = y
            y = y.parent
        return y

    def treePredecessor(self,node):
        if node.left is not None:
            return self.treeMaximum(node.left)
        y = node.parent
        while y is not None and node == y.left:
            node = y
            y = y.parent
        return y

    def insert(self,node):
        y = None
        x = self.root
        while x is not None:
            y = x
            if node.key < x.key:
                x = x.left
            else:
                x = x.right
        node.parent = y
        if y == None:
            self.root = node #Tree was empty
        elif node.key < y.key:
            y.left = node
        else:
            y.right = node

    def transplant(self,u,v):
        if u.parent is None:
            self.root = v
        elif u == u.parent.left:
            u.parent.left = v
        else:
            u.parent.right = v
        if v is not None:
            v.parent = u.parent

    def delete(self,node):
        if node.left is None:
            self.transplant(node,node.right)
        elif node.right is None:
            self.transplant(node,node.left)
        else:
            y = self.treeMinimum(node.right)
            if y.parent is not node:
                self.transplant(y,y.right)
                y.right = node.right
                y.right.parent = y
            self.transplant(node,y)
            y.left = node.left
            y.left.parent = y



A = [12,5,2,9,18,15,19,13,17]

T = BinaryTree()

for _key in A:
    N = Node(key = _key)
    T.insert(N)

T.inorderWalk(T.root)

print "Max: " + str(T.treeMaximum(T.root).key)
print "Min: " + str(T.treeMinimum(T.root).key)

sn = T.treeSearch(T.root,17)

print T.treeSuccessor(sn).key
print T.treePredecessor(sn).key

# for k in A:
#     print "-"*10
#     f = T.treeSearch(T.root,k)
#     print "deleting", f.key
#     T.delete(f) 
#     T.inorderWalk(T.root)

Heap Sort in python

2016-07-25T00:00:00z

The (binary) heap data structure is an array object that we can view as a nearly complete binary tree.

Each node of the tree corresponds to an element of the array.

The tree is completely filled on all levels except possibly the lowest, which is filled from the left up to a point.

An array A that represents a heap is an object with two attributes:

length, which (as usual) gives the number of elements in the array.

heap-size, which represents how many elements in the heap are stored within array A.

Here is the implementation in python.

//not used
def parent(i):
    return int(i/2)

def left(i):
    return 2 * i

def right(i):
    return left(i) + 1

def maxHeapify(A,i,sz):
    l = left(i)
    r = right(i)
    largest = None
    if l <= sz and A[l] > A[i]:
        largest = l
    else:
        largest = i

    if r <= sz and A[r] > A[largest]:
        largest = r

    if largest != i:
        temp = A[i]
        A[i] = A[largest]
        A[largest] = temp 
        maxHeapify(A,largest,sz)

def heapsort(A):
    heap_size = len(A)
    middle = int(heap_size/2)
    for i in xrange(middle,0,-1):
        maxHeapify(A,i,heap_size - 1)

    m = heap_size - 1
    for i in xrange(m,-1,-1): 
        temp = A[0]
        A[0] = A[i]
        A[i] = temp 
        m = m-1 
        maxHeapify(A,0,m)




A = [16,4,10,14,8,7,9,3,2,4,1]

print "Unsorted: ", A 
heapsort(A)
print "Heap Sorted: ", A

Insertion sort in python

2016-07-22T00:00:00z

Insertion sort, is an efficient algorithm for sorting a small number of elements.

Insertion sort works the way many people sort a hand of playing cards. We start with an empty left hand and the cards face down on the table. We then remove one card at a time from the table and insert it into the correct position in the left hand.

To find the correct position for a card, we compare it with each of the cards already in the hand, from right to left.

Here is the pseudocode for it.

Here is the python version of it

def insertionSort(a):
    for j in range(1, len(a)):  
        key = a[j]
        i = j - 1
        while (i > -1 and a[i] > key): 
            a[i + 1] = a[i]
            i = i - 1
        a[i + 1] = key
       

    return a


a = [6, 5, 3, 1, 8, 7, 2, 4] 
print insertionSort(a)

Hope this helps :)

How to install Ubuntu bash on windows

2016-04-07T00:00:00z

Microsoft announced at Build 2016 , about the new feature to run native Bash and GNU/Linux command-line tools

Gabriel Aul has just announced the release of the Windows 10 build #14316 to the Windows Insiders Fast-Ring

This build contains the first public release of Bash on Ubuntu on Windows and the underlying Windows Subsystem for Linux.

Turn on Developer Mode: Settings | Update & Security | For Developers | Check the Developer Mode radio button:
From the start menu, open “Turn Windows Features on or off”:
Scroll down and check the “Windows Subsystem for Linux (Beta)” feature:
Hit okay and reboot (required step)
Once rebooted, open a PowerShell/command prompt and run “Bash” and follow the simple prompts to accept Canonical’s license and kick-off the download of the Ubuntu image:
After download has completed, you’ll be able to start “Bash on Ubuntu on Windows” from the Start menu:

How to use Google Appengine Remote API for Java

2016-03-22T00:00:00z

Google App Engine Remote API enables programmers to access Google DataStore remotely from any python script or a Java Program.

Remote api essentially is a generic web service, which allows datastore to be accessed from outside the app engine's environment therefore using this mechanism, users can either bulk insert data or modify existing data from their servers.

With the help of remote api, data from one datastore can be copied to data on another app engine instance as well.

Configuring Remote api on server - Java

Configuring remote api on an app engine - java web application simply requires addition of a servlet in the web.xml.

To install the Remote API servlet, add the following to your web.xml

<servlet>
<display-name>Remote API Servlet</display-name>
<servlet-name>RemoteApiServlet</servlet-name>
<servlet-class>com.google.apphosting.utils.remoteapi.RemoteApiServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>RemoteApiServlet</servlet-name>
<url-pattern>/remote_api</url-pattern>
</servlet-mapping>

Class com.google.apphosting.utils.remoteapi.RemoteApiServlet is already bundled with the GAE SDK so thats all which needs to be done to get it up and running, also the security is already handled and only the user with admin privilege can access this handler .

For you client Java Program (which is not you regular app engine project), you would need to include these two dependencies in your pom.xml

<dependency>
<groupId>com.google.appengine.tools</groupId>
<artifactId>appengine-gcs-client</artifactId>
<version>0.5</version>
</dependency>
<dependency>
<groupId>com.google.appengine</groupId>
<artifactId>appengine-remote-api</artifactId>
<version>1.9.32</version>
</dependency>

Then install remote api in your standalone(non appengine) project

RemoteApiOptions options_local = new RemoteApiOptions()
.server("localhost", 8080)
.useDevelopmentServerCredential();
or
RemoteApiOptions options_remote = new RemoteApiOptions()
.server("your_app_id.appspot.com", 443)
.useApplicationDefaultCredential();RemoteApiInstaller installer = new RemoteApiInstaller();
installer.install(options_local);

The Remote API client will rely on Application Default Credentials that use OAuth 2.0.

Finally create an instance of datastore factory after installing the API

final DatastoreService ds = DatastoreServiceFactory.getDatastoreService();

For example if one needs to delete all entities in the local or remote datastore via code, then

String[] entities = "Entity1,Entity2,Entity".split(",");
for (String _e : entities) {
Query query = new Query(_e);
for (Entity entity : ds.prepare(query).asIterable()) {
ds.delete(entity.getKey());
}
}

Or here is another example where we create some entity on a remote server

Entity e = new Entity("SomeEntity");
e.setProperty("prop1","blah");
e.setProperty("prop2","blah blah ?");
e.setProperty("prop3","blah blah blah ??");

Basically , as it should'd have been clear by now,after the step involving installation of Remote API, all datastore operations occur in the remote server.

Finally , while performing operations on remote server via remote api, always prefer to do operations on batch, read more here about Java DataStore API Batch Operations.

Hope this helps.

Read GAE Admin Backups fromLevelDB format and export GAE Entities using bulkloader

2016-01-11T00:00:00z

Google datastore is pretty awesome when one needs a quick no-sql data storage. However recently I have experienced a problem in exporting my GAE Datastore as csv and in certain cases as a line delimited Json file. Its not very hard to do so and perhaps the easiest way to handle such thing is to write an export handler in you web app, however, there are alternative ways which I have highlighted below.

One of the easiest alternative is by using Datastore Admin . This tool will easily let you backup you GAE DataStore Entities to google cloud storage in the same project which can later be downloaded locally, by using cloud console or gsutil like this gsutil cp -R /gs/your_bucket_name/your_path /local_target
Then there is Remote API for python and Java, which perhaps was created to modify the datastore directly via code from your local machine
Finally there is a python utility called bulkloader.py which is coupled with remote_apii , this utility does require python sdk to be installed and added to your system path.

Using the Bulk Loader Tool

To quickly verify if bulkloader is correctly installed simply type bulkloader.py

If you have a Python application, you can use the appcfg.py upload_data and appcfg.py download_data commands to do data uploads and downloads. These are equivalent to bulkloader.py, but can also derive the application ID and the remote API URL from your app.yaml file. (Future releases may make more bulkloader features available via appcfg.py; check the App Engine website for the latest.)

f you have a Java application, you just run the bulkloader.py tool, specifying the application ID and remote API URL as command-line arguments: bulkloader.py --appid=app-id --url=http://app-id.appspot.com/remoteapi The loader tool keeps track of its progress during an upload or download in a data file and can begin from where it last left incase the process is interrupted, this feature requires sqlite3 installed though which can be verified by typing squlite3 in the shell. One of the easiest features of the bulk loader to use is the backup and restore feature. The bulk loader can download every entity of a given kind to a data file, and can upload that data file back to the app to restore those entities.

To Download all entities


bulkloader.py --dump
--appid=app-id
--url=http://app-id.appspot.com/remoteapi
--kind=kind
--filename=backup.dat

To Upload all entitites


bulkloader.py --restore
--appid=app-id
--url=http://app-id.appspot.com/remoteapi
--kind=kind
--filename=backup.dat

When using --dump and --restore, entities are not validated using the app’s model classes. They are dumped and restored directly as they appear in the datastore.

Entity keys are preserved, including system-assigned numeric IDs. Properties with key values will remain valid as long as the entities they are referring to are in the app, or are part of the dump being restored. Restoring an entity with the same key as an existing entity will replace the existing entity. There are clever ways to read csv and upload data into appengine datastore using bulkloader which I will explain in another post, however for those who have been taking dumps using the admin tool, here is quick python code to read entities from the backup files which are in Googles level DB format

import sys,datetime
sys.path.append('/usr/local/google_appengine/')
sys.path.append('/usr/local/google_appengine/lib/yaml/lib/')
if 'google' in sys.modules:
    del sys.modules['google']

from google.appengine.api.files import records
from google.appengine.datastore import entity_pb
from google.appengine.api import datastore
import unicodecsv as csv
from pprint import pprint

toCSV = []

def getKeyName(r):
    entity_proto = entity_pb.EntityProto(contents=r)
    for element in entity_proto.key().path().element_list():
        if element.has_name():
            return element.name()

def getProto(r):
    entity_proto = entity_pb.EntityProto(contents=r)
    return datastore.Entity.FromPb(entity_proto)

fls = ['output-0','output-1','output-2','output-3','output-4']
for f in fls:
    raw = open(f, 'r')
    reader = records.RecordsReader(raw)

    for record in reader:
            entity = getProto(record)
            entity[u"name"] = getKeyName(record)
            # pprint(entity)

            toCSV.append(entity)
            print "-"*50

            # pprint(entity)

            # for key in entity.keys():
            #     print key + " - " + str(type(entity[key]))

keys = toCSV[0].keys()
with open('settings.csv', 'wb') as output_file:
    dict_writer = csv.DictWriter(output_file, keys,encoding='utf-8')
    dict_writer.writeheader()
    dict_writer.writerows(toCSV)

Serving raster layers on Google Cloud Platform

2015-09-21T00:00:00z

In this blog post, I will write about using Google cloud storage as a Raster Tile Server for static imagery.

In the GIS domain, various techniques are used to add custom raster overlay on top of a base maps, or to even use custom imagery, as a base map itself.

This approach is also useful if you have a large quantity of satellite or aerial imagery, that you need to serve at scale onto a Google map or any other GIS tool.

There are various clients out there for web as well as android or IOS, which provide an ability to add overlays while pointing to an external service for the image source.

Google Cloud Storage serves as a perfect online, static imagery severing service.

The general idea is to save map tiles, which are usually 256 x 256 sized images in the cloud and then point the client to the base url .

Map tiles

Map tiles can have various naming standards and can vary based on map projection.

This article focuses on map tiles in a z/x/y.png naming standard, where Z is the current zoom level, and X is the X coordinate and Y is the Y coordinate, measured from the top left of the map for each zoom level.

Google Maps, ESRI, Bing, Open Street Map, and others use this top left “standard.” However, the OGC standard, called TMS, starts from the bottom left.

You can see the differences in the two examples below, which show a typical tiles file structure (Z/X/Y.png), where, in the 4th X column of zoom level 4, there are two images for the Y coordinates.

Note that their names are different based on whether counting began at the top left or bottom left.

The file structure in which your software exported the tiles will determine the logic you need to use in your application when you want to load a specific tile.

However, the file structure doesn’t impact the actual upload process.

Generating Map Tiles

There are various open source tools , which can directly generate map tiles from a vector or raster data source. There is a great example here from CartoDB, which shows how to generate raster tiles from geotiff or shape files (Any OGC data Source supported by GDAL tools)

In this article , I will present an alternative way to do so, by using a python library called Tilestache .

Tilestache is a map tiles server which seamlessly integrates with Google maps tiling scheme. It also provides a seeding script which can generate tiles on the disk and has plugins to directly add those tiles in the google cloud.

I prefer Tilestache for various reasons, but most importantly because it is based on Mapnik, therefore, one can use open source tools like Tilemill by mapbox to style the maps and later export the mapnik xml directly, which in-turn, can later be configured in Tilestache for generating highly styled images.

Here is a quick example of styling states of America, the style used in TileMill is below

#ne110madmin1statespr {
  line-color:#fff;
  line-width:1;
  polygon-opacity:0.5;
  text-name: "[name]";
  text-face-name: "Arial Bold";
}
#ne110madmin1statespr { polygon-fill: #FFFFCC; }
#ne110madmin1statespr[name_len >= 4 ] { polygon-fill: #D9F0A3; }
#ne110madmin1statespr[name_len >= 7] { polygon-fill: #ADDD8E; }
#ne110madmin1statespr[name_len >= 10] { polygon-fill: #78C679; }
#ne110madmin1statespr[name_len >= 13] { polygon-fill: #41AB5D; }
#ne110madmin1statespr[name_len >= 15] { polygon-fill: #238443; }
#ne110madmin1statespr[name_len >= 20] { polygon-fill: #005A32; }

Here is how it looks.

Now we can easily extract the mapnik xml from the tilemill.

Next step is to configure a tilestache config file which is essentially a json document which is used by rendering engine to draw tiles. Here is a sample configuration

{
  "cache":
  {
    "name": "Disk",
    "path": "tiles",
    "dirs": "portable",
    "umask": "0000"
  },
  "layers":
  {
    "NE":
    {
        "provider": {"name": "mapnik", "mapfile": "C:\\Users\\Varun Pant\\Desktop\\NE\\NE.xml"},
        "projection": "spherical mercator"
    }
  }
}

Few things to note in the above configuration is the attribute cache and attribute layers. Cache instructs the library to save the files onto the disk, the dirs attribute instructs the file structure to be of portable type which essentially means to save is X/Y/Z.png format on the disk.

Now to start the webserver, one can simply type in tilestache-server.py -c ne.cfg

I have also added a quick javascript below, which shows ,how to add a custom overlay on google maps and point it to our raster service.

function CustomOverlay(tileSize) {
    this.tileSize = tileSize;
}
CustomOverlay.prototype.getTile = function(coord, zoom, ownerDocument) {
    var img = ownerDocument.createElement('img');
    img.src = "http://localhost:8080/NE/" + zoom + "/" + coord.x + "/" + coord.y + ".png";
    img.style.width = this.tileSize.width + 'px';
    img.style.height = this.tileSize.height + 'px';
    return img;
};

var map;

function initialize() {
    map = new google.maps.Map(
        document.getElementById("map-canvas"), {
            center: new google.maps.LatLng(42.660851192127, -96.5624457296875),
            zoom: 4
        });

    map.overlayMapTypes.insertAt(0, new CustomOverlay(new google.maps.Size(256, 256)));
}

google.maps.event.addDomListener(window, 'load', initialize);

If all goes well, then this is how the end result will look like.

Here is a gist with complete html mark-up.

As stated above, tilestache also allows to just create static images on the disk instead of serving them, which is very usefull if one wants to serve these tiles fronted by a proxy or load it to cloud like Google or amazon data storage.

The Command to do so is tilestache-seed.py –c <config name>.cfg –l <layername> –b <bounding box> -e png <list of zoom levels to render>

In my case,I will use the bounding box of my shapefile and the command goes like this tilestache-seed.py -c ne.cfg -l NE -b 18.9161900000001420 -171.7911106028911700 71.3577635769417500 -66.9646599999999810 -e png 1 2 3 4 5 6 7 8

This is how the folders are created in my cache directory, which is configured in the ne.cfg shown above.

Uploading data to the cloud

So now that we have our tiles generated, the next task is to upload them to the cloud.

1. Sign in to the Google Developers Console and click Create Project.

2.Enable billing.

3. Create a Google Cloud storage bucket, choose the appropriate location and storage class.

And that's it you are all set. Finally click the bucket name and you will have the following options

The easiest way to upload folders and tiles here is to click upload folder button and point to the folders on the disk. This is by far the easiest method, however in our case we have only generated tiles for first 8 zoom levels, so they are not much in number and using a web UI upload mechanism will work.

If we choose to generate large number of tiles and cover more than few countries we are possibly looking at millions to tiles specially around the street level (zoom 19 to 22), to cover such a scenario, I will recommend using another option in tilestache ,which directly allows saving the tiles in Google cloud storage. Tilestache, has a plugin which can be used instead of a disk based cache. here is how to configure it

{
  "cache":
  {
    "class": "TileStache.Goodies.Caches.GoogleCloud:Cache",
    "kwargs": {
      "bucket":  "daa-transfers-static",
      "access": "accesstoken",
      "secret": "clientsecret""
    }
  },
  "layers":
  {
    "NE":
    {
        "provider": {"name": "mapnik", "mapfile": "C:\\Users\\Varun Pant\\Desktop\\NE\\NE.xml"},
        "projection": "spherical mercator"
    }
  }
}

You will notice that to work with this plugin,one needs to have access token and client secret. This can be easily generated using the developer console.

To setup, follow these steps

GS Security Credentials:

Select the project for Google cloud storage [Google APIs Console]
Select Google Cloud Storage
Select Make this my default project for interoperable storage access.
Select Interoperable Access.
You can find gs_access_key_id ,gs_secret_access_key for your project.

Once configured run the tile seed command again and now, the tiles would be directly stored in the bucket, to access them use the following format http://storage.googleapis.com/<bucket_name>/<path>/<tile.png>.

In our case we can access the tiles like this.

http://storage.googleapis.com/states-demo/NE/8/68/103.png , in the Html code given above, one can easily change the url from img.src = "http://localhost:8080/NE/" + zoom + "/" + coord.x + "/" + coord.y + ".png"; to img.src = "

http://storage.googleapis.com/states-demo/NE/" + zoom + "/" + coord.x + "/" + coord.y + ".png";

Security and ACL

Google cloud also allows you to configure the permissions by setting up ACL against your bucket. It is possible to make the bucket and all its content public, or authenticate them via domain ,SSO, user groups and many more ways. Read this article to know more.

GSUtil

Finally there is another way to upload your cache by using GSUtil.

gsutil cp -a public-read -r C:\tiles\NE gs://states-demo

I have hosted a link here

Hope this helps.

Install all Go project dependencies in one command

2015-08-25T00:00:00z

So if you are looking to compile you go project on your server, you would probably need to worry about any third party dependencies.

Here is quick recursive way to do that

go get ./...

Hope this helps

Bubble-sort in python

2015-07-23T00:00:00z

Bubble sort is a sorting algo that repeatedly steps through the list to be sorted, compares each pair of adjacent items and swaps them if they are in the wrong order.

It is a popular, but inefficient sorting algo, it has worst-case and average complexity both О(n²), where n is the number of items being sorted.

Here is an implementation in python

def bubbleSort(B):
    count = len(B)
    for i in xrange(count): 
        for j in xrange(count-1,i,-1): 
            #print "compairing %i with %i"%(B[j],B[j-1])
            if B[j] < B[j - 1]:
                temp = B[j]
                B[j] = B[j - 1]
                B[j - 1] = temp
                #print "Swapping  %i with %i"%(B[j],B[j-1])
        #print "Transformed Array: " , B
            

B = [6,5,3,1,8,7,2,4]
print "Unsorted: ", B 
bubbleSort(B)
print "Bubble Sorted: ", B

Hope this helps

ultimate developer and power users tool list for mac and linux

2015-02-07T00:00:00z

This is a collection of tools and utilities which I have found indispensable as an engineer. Each has a distinct purpose, and I probably touch each at least a few times a week. These tools save me time and I hope that you will find some of them useful as well.

Productivity

s-tui s-tui is a terminal UI for monitoring your computer. s-tui allows to monitor CPU temperature, frequency, power and utilization in a graphical way from the terminal.
Cockpit is a server manager that makes it easy to administer your GNU/Linux servers via a web browser.

Security

https://www.tinc-vpn.org/ Virtual Private Network (VPN) daemon that uses tunnelling and encryption to create a secure private network between hosts on the Internet.

Fun

https://github.com/abishekvashok/cmatrix Matrix digital rain with CMatrix on a Linux or Unix terminal .

Automation

FunnelScripts Facebook Adverts.

Creating Isochrones and Isodistances using Googles Direction Search API

2014-12-29T00:00:00z

Introduction

Isochrones are curves of equal travel time from a certain point of origin, another way of saying this would be that an Isochrone is an isoline for travel time, if the weighting factor is changed form time to distance, then the resulting curve us called an Isodistance.

In this post I will present a rough way to create an Isochrone and an Isodistance using google's directions API.

Algorithm

The actual algorithm to calculate Isocrone or Isodistance is perhpas a bit complicated than the simplistic approach being described here, if you are intrested in having a deeper look then I would recommend visiting this link by OpenStreetMap folks. Here is a link to my version on github. Open Isochrone.html to take a look at the functionality or visit this hosted link.

Details

Since we do not have direct access to road network (to crawl minimum spanning tree), we fist start the process by drawing a circle of x meters for Isodistance or make a fair assumption for Isochrones. Following code describes the process.

function getCirclePoints(center, radius) {

	var circlePoints = [];
	var searchPoints = [];
	with(Math) {
		var rLat = (radius / 6378.135) * (180 / PI);
		var rLng = rLat / cos(center.lat() * (PI / 180));
		for (var a = 0; a < 361; a++) {
			var aRad = a * (PI / 180);
			var x = center.lng() + (rLng * cos(aRad));
			var y = center.lat() + (rLat * sin(aRad));
			var point = new google.maps.LatLng(parseFloat(y), parseFloat(x));
			circlePoints.push(point);
			if (a % pointInterval == 0) {
				searchPoints.push(point)
			}
		}
	}
	searchPolygon = new google.maps.Polygon({
		paths: circlePoints,
		strokeColor: '#000000',
		strokeOpacity: 1,
		strokeWeight: 1,
		fillColor: '#ffffff',
		geodesic: true,
		fillOpacity: 0.5,
		clickable: false
	});
	searchPolygon.setMap(map);
	map.fitBounds(searchPolygon.getBounds());
	return searchPoints
}

getCirclePoints method uses a point interval of 5 and returns around 72 points (360/5), all around the circumference of the circle. Next, we use the center of the circle as our starting point, and call direction services for each of the point in the circumference.

    var from = startpoint.lat() + ' ' + startpoint.lng();
	var to = searchPoints[0].lat() + ' ' + searchPoints[0].lng();

	//Removed processed Point.
	searchPoints.shift()

	var request = {
		origin: from,
		destination: to,
		travelMode: google.maps.TravelMode[selectedMode],
		avoidHighways: false
	};

	directionsService.route(request, directionsearch)

Steps

For each of the requested responses, code traverse through the steps in the legs, then depending on the weighing factor we compute the desired distance or time as shown below

for (var n = 0; n < steps.length; n++) {

		if(ISOCHRONE)
			unit += steps[n].duration.value;
		else
			unit += steps[n].distance.value;

		if (unit < comparator) {
			temp_Points.push(steps[n].end_location)
		}
		 else {
			break;
		}
	}

The whole process can be observed by directions debug lines.

Tilestache on google maps

2014-12-26T00:00:00z

Introduction

In this post I will demonstrate how to add custom TileStache map tiles on google maps, using google maps JavaScript API V3. TileStache has emerged as one of the easiest way to render your own map .There are many examples out there on creating basic tiles and then adding them on top of Leaftlet or Openlayers. In this post I will add them as an overlay on Google Maps.

Overlays

In Google’s terminology, adding a map layer basically means adding an overlay. Here is an example on adding a Ground Overlay. Here is another one called Custom overlay. Basically there is a concept of map types and is documented here in great detail, just skip past the Standard map types and scroll to Custom Map Types. This example, of a Basic Map Type proved to be most helpful. In case you are after adding your own Base Map, instead of an Overlay, then take a look at this example.

TileMill and TileStache

Ok, so now lets quickly create a map server side. In my Map I have a Uk District polygon layer which have been computed to get a count score with UK Crime DataSet November 2014, I simply used Point inside a polygon feature in Qgis to get a score. Then I styled the data using Carto Css in TileMill to a get a Heat Map Like effect.

Here is the Carto Css if you are wondering.

/* Color variables */
@r1:#28ee00;
@r2:#4fee00;
@r3:#77ee00;
@r4:#9fee00;
@r5:#c6ee00;
@r6:#eec600;
@r7:#ee9f00;
@r8:#ee7700;
@r9:#ee4f00;
@r10:#ee2800;
/* Layers */
#sectors {
  [zoom<8]
  {
    line-width:0.2;
    polygon-opacity:0.9;
  }
  [zoom>=8]
  {
  line-width:0.8;
   polygon-opacity:0.7;
  }

  [zoom>=12]
  {
  line-width:1;
   polygon-opacity:0.6;
  }
  [zoom>20]
  {
    line-width:1.2;
    polygon-opacity:0.4;
  }

  line-opacity: 1;
  [PNTCNT=0]{polygon-opacity:0;line-opacity:0;}
  [PNTCNT > 0][PNTCNT <= 24]{polygon-fill:@r1;line-color:#FFF;}
  [PNTCNT > 24][PNTCNT <= 48]{polygon-fill:@r2;line-color:#FFF;}
  [PNTCNT > 48][PNTCNT <= 72]{polygon-fill:@r3;line-color:#FFF;}
  [PNTCNT > 72][PNTCNT <= 96]{polygon-fill:@r4;line-color:#FFF;}
  [PNTCNT > 96][PNTCNT <= 120]{polygon-fill:@r5;line-color:#FFF;}
  [PNTCNT > 120][PNTCNT <= 144]{polygon-fill:@r6;line-color:#FFF;}
  [PNTCNT > 144][PNTCNT <= 168]{polygon-fill:@r7;line-color:#FFF;}
  [PNTCNT > 168][PNTCNT <= 192]{polygon-fill:@r8;line-color:#FFF;}
  [PNTCNT > 192][PNTCNT <= 216]{polygon-fill:@r9;line-color:#FFF;}
  [PNTCNT > 216]{polygon-fill:@r10;line-color:#FFF;}
}

Its very easy to export Mapnik Configuration xml of your TileMill Project.

This is my Map configuration for TileStache.

{
  "cache":
  {
    "name": "Test",
    "path": "/tmp/stache",
    "umask": "0000"
  },
  "layers":
  {
     "sectors":
     {
         "provider": {"name": "mapnik", "mapfile": "crime.xml"},
         "projection": "spherical mercator"
     }
  }
}

Finally start the server . TileStache-master/scripts/tilestache-server.py -c /path/to/my/configuration/tiles.cfg

Creating a Custom Overlay for Google Maps

Its quite easy to add a TileStache Overlay on top of google maps. TileStache URLs are already based on a Google Maps-like scheme: /{layer name}/{zoom}/{column}/{row}.{extension} So we create a CoordMapType overlay.

 /** @constructor */
  function CoordMapType(tileSize) {
    this.tileSize = tileSize;
  }
  CoordMapType.prototype.getTile = function(coord, zoom, ownerDocument) {
    var img = ownerDocument.createElement('img');
    img.src="http://localhost:8080/sectors/"+ zoom+"/"+coord.x+"/"+ coord.y+".png";
    img.style.width = this.tileSize.width + 'px';
    img.style.height = this.tileSize.height + 'px';
    return img;
  };

and then we simply add this overlay to map to overlayMapTypes .

map = new google.maps.Map(document.getElementById('map-canvas'),
      mapOptions);
    map.overlayMapTypes.insertAt(
      0, new CoordMapType(new google.maps.Size(256, 256)));

and thats it, we are done. Here is a screenshot of how the map looks.

I have added a working sample here , incase you wanna check out the full source code. Hope this helps.

Merge multiple Shapefiles using OGR2OGR in windows Linux or OSX

2014-12-22T00:00:00z

Introduction

This post is aimed at providing assistance to folks who are trying to merge many shapefiles into one single file in either windows or linux environment. Stuff like this may seem obvious to power users of GDAL/OGR, but not to those that are just new to using these great free tools.

Merging Shape (*.shp) Files on Windows

Essentially, these commands will grab every shapefile in a directory, and merge them into a single file called “merged.shp”.

for %f in (*.shp) do (
if not exist merged.shp (
ogr2ogr -f “esri shapefile” merged.shp %f) else (
ogr2ogr -f “esri shapefile” -update -append merged.shp %f -nln Merged )
)

Merging Shape (*.shp) Files on OSX or Linux

In Linux like environment it's even easier than one would imagine. Infact I am quite sure it should be a one liner for windows enviroment as well, will have to try one day :)

for f in *.shp; do ogr2ogr -update -append merged.shp $f -f "ESRI Shapefile"; done;

Merging Shape (*.shp) Files using QGIS

Quite frankley I found this method to be the most slow of all, however this has GUI and should be fine for moderate size files.

Open Qgis and click Vector on menu bar, then click on Data Management Tools and finally on Merge shapefiles to one

This will open a dialog, which will ask for the input source folder and the destination file name , and thats it, click OKto begin the process.

Creating Index using shapeindex

After merging shapefiles , its most likely that one would like to create a spatial index. Here is how to do it. Migrate to the folder and execute shapeindex myShapeFileName.shp

Copying Ordinance Survey Vector Map shape files into singular folders

Incase you are looking at a help full script to copy all similar named shape files from grid folders into dedicated layer named folders using python then I have a goodie for you

import fnmatch
import os
import shutil
import glob
from collections import defaultdict

groups = defaultdict(list)

basedir ='/Users/varunpant/Desktop/O vectormap'
mergeddir = os.path.join(basedir, 'merged')

def copyFiles(src,desti):

    files = glob.glob(src.replace('shp','*'))
    for f in files:
        shutil.copy(f, desti)


for root, dirnames, filenames in os.walk(basedir):
  for filename in fnmatch.filter(filenames, '*.shp'):
      basename, extension = os.path.splitext(filename)
      name = basename.split('_')[1]
      groups[name].append(os.path.join(root, filename))

for group in groups:
    g = groups[group]

    directory = os.path.join(mergeddir, group)
    if not os.path.exists(directory):
       os.makedirs(directory)

    for n in g:
        copyFiles(n,directory)
        #print n

HTML Geolocation and Altitude

2014-12-21T00:00:00z

HTML5 Geolocation is a feature which allows the browser on a computer or a mobile phone, to acquire the position from the wifi,2g/3g/4g network or GPS.

The HTML Geolocation API is used to get the geographical position of a user. Since this can compromise user privacy, the position is not available unless the user approves it. The simplest way to use it through navigator.geolocation object

if (navigator.geolocation) {
        navigator.geolocation.getCurrentPosition(showPosition);

function showPosition(position) {
    alert(position.coords.latitude +  " , "  + position.coords.longitude);
}

The latitude and longitude are available to JavaScript on the page, which in turn can send it back to the remote web server and do fancy location-aware things like finding local businesses or showing your location on a map, geofencing and a lot more.

The position object contains few more intresting parameters, but are generally available only when in use in smart phones as those devices are equipped with supporting hardware. Here is a table with detailed information.

Position Object
Property	Type	Notes
`coords.latitude`	double	decimal degrees
`coords.longitude`	double	decimal degrees
`coords.altitude`	double or `null`	meters above the reference ellipsoid
`coords.accuracy`	double	meters
`coords.altitudeAccuracy`	double or `null`	meters
`coords.heading`	double or `null`	degrees clockwise from true north
`coords.speed`	double or `null`	meters/second
`timestamp`	DOMTimeStamp	like a `Date()` object

Handling Errors and Rejections

The second parameter of the getCurrentPosition() method is used to handle errors. It specifies a function to run if it fails to get the user's location.

function showError(error) {
    switch(error.code) {
        case error.PERMISSION_DENIED:
            x.innerHTML = "User denied the request for Geolocation."
            break;
        case error.POSITION_UNAVAILABLE:
            x.innerHTML = "Location information is unavailable."
            break;
        case error.TIMEOUT:
            x.innerHTML = "The request to get user location timed out."
            break;
        case error.UNKNOWN_ERROR:
            x.innerHTML = "An unknown error occurred."
            break;
    }
}

Some popular mobile devices — like the iPhone and Android phones — support two methods of figuring out where you are. The first method triangulates your position based on your relative proximity to different cellular towers operated by your phone carrier. This method is fast and doesn’t require any dedicated GPS hardware, but it only gives you a rough idea of where you are. Depending on how many cell towers are in your area, “a rough idea” could be as little as one city block or as much as a kilometer in every direction.

The second method actually uses dedicated GPS hardware on your device to talk to dedicated GPS positioning satellites that are orbiting the Earth. GPS can usually pinpoint your location within a few meters. The downside is that the dedicated GPS chip on your device draws a lot of power, so phones and other general purpose mobile devices usually turn off the chip until it’s needed. That means there will be a startup delay while the chip is initializing its connection with the GPS satellites in the sky. If you’ve ever used Google Maps on an iPhone or other smartphone, you’ve seen both methods in action. First you see a large circle that approximates your position (finding the nearest cell tower), then a smaller circle (triangulating with other cell towers), then a single dot with an exaction position (given by GPS satellites).

The getCurrentPosition() function has an optional third argument, a PositionOptions object. There are three properties you can set in a PositionOptions object. All the properties are optional. You can set any or all or none of them.


	navigator.geolocation.getCurrentPosition(showPosition,{enableHighAccuracy:true, maximumAge:30000, timeout:27000});

Generally one would constantly need to acquire the current position of the device, for such a usecase there is another built in method called watchPosition The sinature will remind you of setTimeout and clearTimeout built in methods.

		//This will start calling showPosition in a timedout manner where variable wpid acts as a reference.
		var wpid=navigator.geolocation.watchPosition(showPosition, geo_error, {enableHighAccuracy:true, maximumAge:30000, timeout:27000});

		//If you wanna stop the loop.
		//navigator.geolocation.clearWatch(wpid);
	    // wpid=false;

Finally there is a full working example here.

Classes and Inheritance in EcmaScript 6 (ES6 Harmony)

2014-11-30T00:00:00z

So if you are a JavaScript geek and haven't yet heard,(if every one is listening to ECMA announcements, who's gonna drink beer :) ) ECMA 6 feature set has been drafted,its feature set is frozen, it is mostly being refined now. You can already program in it and compile it to current JavaScript . The JavaScript frameworks AngularJS and Ember.js will be based on it (with ways to opt out). Again, via cross-compilation.

As you may already know JavaScript is being used both client side i.e. in the browser and also on server side eg:Node.js. To check which features are currently supported in your browser visit here

For me , the most exciting feature has been Classes. We all know that JavaScript needs this soo badly, even though JavaScript, has always been referred to as an OOP language, and arguments have been made by many that as long as you can implement a class like encapsulation and inheritance like functionality, the language must be regarded as an OOP language(I know, there is much more to this as well), now I being, a humble developer, in most cases just agreed to all that mumbo-jumbo, but there has always been a secret fire burning all along. The desire to actually have a Class keyword and extend keyword, so that my brain finds symmetry in the world

Well somebody must have been listening up there, I mean in the ECMA world, and eventually these babies are here, hence this amazing posts to share the joy :).

So before these keywords were present, a class like construct was generally created like the snippet below

Class

function Employee( name,age, salary ) { //approximate a class/constructor
   this.name = name;
   this.age= age;
   this.salary = salary;

   this.printEmployeeDetails = function(){ //expose a function
          console.log(this.name + ' is ' + this.age + ' years old, and earns ' + this.salary);
    }
}

// Instantiate a new Employee
var smith = new Employee( "Smith",35,100000);
smith.printEmployeeDetails(); //Smith is 35 years old, and earns 100000

In ECMA6 this can be done as follows

class Employee {
    constructor(name,age, salary ) { //constructors!
      this.name = name;
      this.age= age;
      this.salary = salary;
    }

    printEmployeeDetails(){
          console.log(this.name + ' is ' + this.age + ' years old, and earns ' + this.salary);
    }
}

Inheritance

In current JavaScript, Inheritance is prototypical, which to laymen is a keyword and also lets you override the constructer. Inheritance can be defined like this ChildClassName.prototype = new ParentClass(); To Illustrate, consider this example


function Janitor(name) {
 this.name = name;

//These values are not passed in by constructor and are local to class Janitor
 this.age = 66;
 this.salary = 5000;
 }
Janitor.prototype = new Employee();// Here's where the inheritance occurs
Janitor.prototype.constructor=Janitor;// Override the Employee (super) constructor, real confusing, I know

//Define new Janitor Instance
var bob = new Janitor('Bob');

//note: bob has printEmployeeDetails method inherited from Employee class
bob.printEmployeeDetails() // Bob is 66 years old, and earns 5000

In ECMA6 this can be done as follows

class Janitor extends Employee { //inheritance
    constructor(name) {
        super(name); //call the parent constructor with super
        this.age = 66;
        this.salary = 5000;
    }
}

let bob = new Janitor('Bob');
bob.printEmployeeDetails() // Bob is 66 years old, and earns 5000

Unfortunately Chrome, which is my browser of choice does not support classes, so you can't run this code in console, but I am sure, I will strike out this line soon :)

B.T.W check these projects out too Microsoft’s TypeScript, Facebook’s Flow and Google’s AtScript.

longest palindrome in a string

2014-10-15T00:00:00z

  def isPailenDrome(s):
        if len(s)==1:
                return False
        L = len(s)
        for i in range(L):
            if s[i] != s[L-1-i]:
                return False
        return True
                

def longestPalindrome(s):
        L = len(s)
        if len(s)<=1:
                return s
        if  isPailenDrome(s):
                return s
        p = {}
        for u in range(L):
                for v in range(u+1,L+1):  
                        k = s[u:v]
                        # print k,isPailenDrome(k)
                        if isPailenDrome(k):
                                p[len(k)] = k
        
        if len(p) > 0:
                return p[sorted(p,reverse = True)[0]]
        return s[0]

print two-dimensional array in spiral order

2014-10-15T00:00:00z

So I saw this problem in a book today about printing a 2d matrix in spiral order

Here are two solutions to it

Solution one.

def printSpiralTL(m,x1,y1,x2,y2):

for i in range(x1,x2):

print m[y1][i]

for j in range(y1+1,y2+1):

print m[j][x2-1]

if x2-x1 > 0:

printSpiralBL(m, x1, y1 + 1, x2-1, y2)

def printSpiralBL(m,x1,y1,x2,y2):

for i in range(x2-1,x1-1,-1):

print m[y2][i]

for j in range(y2-1,y1-1,-1):

print m[j][x1]

if x2-x1 > 0:

printSpiralTL(m, x1+1, y1, x2, y2-1)

m = [

[1, 2, 3, 4],

[5, 6, 7, 8],

[9, 0, 1, 2],

[3, 4, 5, 6],

[7, 8, 9, 1]

]

printSpiralTL(m,0,0,4,4)

Output:

Here is another way

Solution two (print - pop -Transpose).

m = [
    
    [1, 2, 3, 4], 
    [5, 6, 7, 8],
    [9, 0, 1, 2],   
    [3, 4, 5, 6], 
    [7, 8, 9, 1]
        
    ]

def spiral(arr):
    while arr:
        for a in arr[0]:
            print a
        arr = list(reversed(zip(*arr[1:])))

spiral(m)

Hope this helps.

merge two sorted arrays in python

2014-10-11T00:00:00z

Here is a quick code snippet to merge two sorted arrays in python

def merge(a,b):
        merged = []
        l = 0
        r = 0
        for i in range(len(a)+len(b) ):
                lval = None
                rval = None

                if l < len(a):
                        lval = a[l]
                if r < len(b):
                        rval = b[r]
                
                if (lval < rval and rval and lval)  or rval == None:
                        merged.append(lval)
                        l+=1
                elif (lval >= rval and rval and lval)or lval == None:
                        merged.append(rval)
                        r+=1

        return merged

print merge([3,5],[2,4])

Update the locate DB on OS X

2014-09-10T00:00:00z

Apparently out OS X does not have an updatedb command like linux has, so I was stumped on how to update the locate database.

I have quickly added this command here, incase I need to use it again.

sudo /usr/libexec/locate.updatedb

How to print the Fibonacci Sequence in Python

2014-08-09T00:00:00z

The Fibonacci numbers are the sequence of numbers defined by the linear recurrence equation

with and

So in simple terms a method to compute fibonacci number for nth place looks like this

 def fibonacci(n):  
     if n == 0 or n ==1:
         return n
     else:
        return fibonacci(n-1)+fibonacci(n-2)

here is quick test

for i in range(10):
    print i," : ",fibonacci(i)


#This will print
0  :  0
1  :  1
2  :  1
3  :  2
4  :  3
5  :  5
6  :  8
7  :  13
8  :  21
9  :  34

or an inline method like this print [fibonacci(i) for i in range(11)] will result

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

Running time of fibonacci series is quite interesting. The first version using recursion has a very bad polynomial time (one can try seeing the performance degrade by passing in larger numbers like 30) so

def fab(n):  
    if n == 0 or n ==1:
        return n
    else:
        return fab(n-1)+fab(n-2)

#This would take polynomial time, (3.4)seconds on my macbook.
fab(35)

There is a better way of doing this in linear time

def fab2(n):
    if n == 0:
     return 0
    f = [0 for j in range(n+1)]
    f[0] = 0
    f[1] = 1
    for i in range(2,n+1):
        f[i] = f[i-1]+f[i-2]
    return f

#this will print the sequence in linear time less than 0.1 seconds
fab2(35)

Few days ago I found another interesting way to code this using enumerate and yield in python

def fib():
    a, b = 0, 1
    while True:            # First iteration:
        yield a            # yield 0 to start with and then
        a, b = b, a + b    # a will now be 1, and b will also be 1, (0 + 1)

and usage

for index, fibonacci_number in enumerate(fib()):
     print('{i:3}: {f:3}'.format(i=index, f=fibonacci_number))
     if index == 10:
         break

prints

Here are some interesting facts about the Leonardo Fibonacii.

Using ElasticSearch for Spatial Search tutorial

2014-07-20T00:00:00z

In this post I will demonstrate using ElasticSearch to spatially query records and filter them by attributes.

ElasticSearch is built on top of Lucence which in version 4.0 supports Spatial query features, for those interested here is an example and link to javadocs.

You will need to install ElasticSearch, read about installing it here, also install marvel plugin for configuration and testing.

For this demonstration I have used data from Geofabrik. I have used point.shp file from Great Britan but you can use data from any other country as well.

To extract the data,I wrote a quick python script using pyshp library

One can easily install it by typing sudo easy_install pyshp in shell

Mapping type setup

Before uploading the records we need to create an Index in elasticsearch and apply schema mapping where we will instruct elasticsearch to create a geo index on location attribute.

We can do this easily by issuing a curl request

curl -XPUT http://localhost:9200/places -d '
{
    "mappings": {
        "place": {
            "properties": {
                "id": {"type": "double"},
                "name": {"type": "string"},
                "type": {"type": "string"},
                "location": {"type": "geo_point"}
            }
        }
    }
}

Once the schema for index places has been set up, its now time to add records to the index.

In the python script below modify the name of the shape file path and execute.

import shapefile
import urllib2
import json
sf = shapefile.Reader("points")
sr = sf.shapeRecords()

for r in sr:
    try :
        if r.record[2].strip() and r.record[3].strip():
            req = urllib2.Request('http://localhost:9200/places/place/')
            req.add_header('Content-Type', 'application/json')
            data = {'id': r.record[0].strip(),'name':r.record[2].strip(),'type':r.record[3].strip(),'location':{'lat':r.shape.points[0][1],'lon':r.shape.points[0][0]}}
            response = urllib2.urlopen(req, json.dumps(data))
            print r.record[2]
    except Exception,e:
        print e
        #print "ERROR ",r.record[0],r.record[2],r.record[3] , r.shape.points[0][0], r.shape.points[0][1]
        pass

The script inserts all records which have a valid name and type column into index.

Verify

Once complete we can quickly verify the number of records in the index by issuing.

GET places/_count from marvel or from a browser http://localhost:9200/places/_count

Spatial Search

In Marvel/Sense

GET places/_search
{
  "sort" : [
      {
          "_geo_distance" : {
              "location" : {
                    "lat": 51.5286416,
                "lon": -0.10159870000006777
              },
              "order" : "asc",
              "unit" : "km"
          }
      }
  ],
  "query": {
    "filtered" : {
        "query" : {
            "match_all" : {}
        },
        "filter" : {
            "geo_distance" : {
                "distance" : "20km",
                "location" : {
                   "lat": 51.5286416,
                   "lon": -0.10159870000006777
                }
            }
        }
    }
  }
}

or by curl

curl -XGET 'http://localhost:9200/places/place/_search?pretty=true' -d '
{
  "sort" : [
      {
          "_geo_distance" : {
              "location" : {
                    "lat": 51.5286416,
                   "lon": -0.10159870000006777
              },
              "order" : "asc",
              "unit" : "km"
          }
      }
  ],
  "query": {
    "filtered" : {
        "query" : {
            "match_all" : {}
        },
        "filter" : {
            "geo_distance" : {
                "distance" : "20km",
                "location" : {
                    "lat": 51.5286416,
                   "lon": -0.10159870000006777
                }
            }
        }
    }
  }
}'

To search by Geodistance as well as a term filter, modify the query to.

GET places / _search ? size = 100 & from = 0 {
    "sort": [{
        "_geo_distance": {
            "location": {
                "lat": 51.5286416,
                "lon": -0.10159870000006777
            },
            "order": "asc",
            "unit": "km"
        }
    }],
    "query": {
        "filtered": {
            "query": {
                "bool": {
                    "should": [{
                        "term": {
                            "type": "pub"
                        }
                    }]
                }
            },
            "filter": {
                "geo_distance": {
                    "distance": "1km",
                    "location": {
                        "lat": 51.5286416,
                        "lon": -0.10159870000006777
                    }
                }
            }
        }

    }
}

For better visualisation I have created a nice webapp here.

Hope this helps

CCS3 only Loading Icon

2014-07-13T00:00:00z

I generally have been using loading gifs in my work most of the time to inform the user that the resources are being fetched asynchronously from the server.

Ajaxload website is perhaps one of the most used site to download a suitable gif.

In this post I will demonstrate a css3 only way to create a nice loading simulation

Basic Code

We will use nested divs to create the loading animation control

<div class="wrap">
   <div class='circle'>
      <div class="clockNeedle"></div>
    </div>
</div>

and css looks like this

body{
  background-color: #fff;
  text-align: center;
  height: 100%;
}

html {
  height: 100%;
  overflow: hidden;
}

.wrap{
 position: absolute;
  top:0; left:0; bottom:0; right:0;
  margin: auto;
  width: 150px;
  height: 150px;
  border: 1px solid transparent;
}

.circle{
  width: 80%;
  height: 80%;
  border: 5px solid black;
  border-radius: 50%;
  position: absolute;
  bottom:0;
  margin-left: auto; margin-right: auto; left:0; right:0;
}


.clockNeedle{
  width: 50px;
  height: 5px;
  background: black;
  position: absolute;
  top: 50%;
  left: 10px;
  transform-origin:100% 50%;
  -webkit-transform-origin:100% 50%;
  -webkit-animation: needleAnimation 3s linear 0s infinite;
  animation: needleAnimation 3s linear 0s infinite;
}

/* Chrome, Safari, Opera */
@-webkit-keyframes needleAnimation {
    from {   }
    to {
    	-ms-transform:rotate(360deg);
    	-moz-transform:rotate(360deg);
    	-webkit-transform:rotate(360deg);
	}
}

@keyframes needleAnimation {
   from { }
    to {
    	-ms-transform:rotate(360deg);
    	-moz-transform:rotate(360deg);
    	-webkit-transform:rotate(360deg);
	}
}

Explanation

Basically to form the outer circle we simply use a div and make it circular by applying border-radius property to 50% border-radius: 50%;

Needle div is styled to have the same background as the wrap, then the width is made 50% and height is 5 px so it looks like a line. Later the a transform origin is applied to make it look like a clock needle and position it adjacent to the circle centre.

Css line animation: needleAnimation 3s linear 0s infinite; makes it rotate for infinite time.(it takes 3s for the animation to finish and repeat it self)

Hope you find this useful.

Live

See the Pen bwnaL by Varun (@varunpant) on CodePen.

Here is a link to the actual demonstration

Edit

I also found another traditonal icon done in css3 here

See the Pen Clean simple Ajax Spinner by emgerold (@emgerold) on CodePen.

How to install and configure grunt on mac Osx

2014-07-12T00:00:00z

This post is intended to assist folks who are trying to install and work with grunt on mac osx

Install `Node.js` and npm with Homebrew

First, install Homebrew by typing in the following command ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"

Then, type in brew update to ensure your Homebrew is up to date.

brew update

Run brew doctor to make sure your system is all good.

Follow any other recommendations from brew doctor.

Next, add the Homebrew location to your $PATH and source your bash profile

export PATH="/usr/local/bin:$PATH"

install Node with NPM

brew install node

To test out your Node and npm install, try installing Grunt (you might be asked to run with sudo)

sudo npm install -g grunt-cli

Add the path from the grunt installer to your bash profile by typing in export PATH=/usr/local/lib/node_modules/grunt/bin:$PATH

Made the profile an executable source ~/.bash_profile

this should do the trick

Normalizing Ranges of Numbers

2014-07-09T00:00:00z

Range Normalization is a normalization technique that allows you to map a number to a specific range.

Lets say that we have a data set where the values are in a range of 1 to 10, however we wish to normalise it to a range between 0 and 5

Mathematically speaking the equation comes down to

translated to Python

from __future__ import division
class Normaliser:

    def __init__(self,dH,dL,nH,nL):
        self.dH = dH
        self.dL = dL
        self.nH = nH
        self.nL = nL

    def normalize(self,x):
        return ((x - self.dL) / (self.dH - self.dL))  * (self.nH - self.nL) + self.nL

    def denormalize(self,x):
        return ((self.dL - self.dH) * x - self.nH * self.dL + self.dH * self.nL) / (self.nL - self.nH)

if __name__ == "__main__":
    norm = Normaliser(10,1,5,0);

    for a in range(1,11):
        x = norm.normalize(a);
        y = norm.denormalize(x);
        print str(a) + " : " + str(x) + " : " + str(y)

The results

1 : 0.0 : 1.0
2 : 0.555555555556 : 2.0
3 : 1.11111111111 : 3.0
4 : 1.66666666667 : 4.0
5 : 2.22222222222 : 5.0
6 : 2.77777777778 : 6.0
7 : 3.33333333333 : 7.0
8 : 3.88888888889 : 8.0
9 : 4.44444444444 : 9.0
10 : 5.0 : 10.0

Hope this helps.

Install tile-stache on OsX Mavericks

2014-07-08T00:00:00z

This is quick tutorial about installing Tilestache library in OSX Mavericks.

Step 1

Make sure Developer tools are installed xcode-select --install

Then add these flags to tell xcode to use python like rest of the world does

export CFLAGS=-Qunused-arguments

export CPPFLAGS=-Qunused-arguments

Finally type in sudo easy_install tilestache

Thats it all done, you might see some warnings but thats fine ignore them .

Read the introduction here to get started

Hope it helps.

how to make a web crawler in python

2014-02-01T00:00:00z

Here is a very simple web crawler in python

import sys, thread, Queue, re, urllib, urlparse, time, os, sys
dupcheck = set()  
q = Queue.Queue(100) 
q.put("http://www.varunpant.com") 
def queueURLs(html, origLink): 
    for url in re.findall(''']+href=["'](.[^"']+)["']''', html, re.I): 
        link = url.split("#", 1)[0] if url.startswith("http") else '{uri.scheme}://{uri.netloc}'.format(uri=urlparse.urlparse(origLink)) + url.split("#", 1)[0] 
        if link in dupcheck:
            continue
        dupcheck.add(link)
        if len(dupcheck) > 99999: 
            dupcheck.clear()
        q.put(link) 
def getHTML(link): 
    try:
        html = urllib.urlopen(link).read() 
        print link
        # open(str(time.time()) + ".html", "w").write("" % link  + "\n" + html) 
        queueURLs(html, link) 
    except (KeyboardInterrupt, SystemExit): 
        raise
    except Exception:
        pass
while True:
    thread.start_new_thread( getHTML, (q.get(),)) 
    time.sleep(0.5)

Ternary search tree

2014-01-26T00:00:00z

In computer science, a ternary search tree is a type of prefix tree where nodes are arranged as a binary search tree. Like other prefix trees, a ternary search tree can be used as an associative map structure with the ability for incremental string search. However, ternary search trees are more space efficient compared to standard prefix trees, at the cost of speed. Common applications for ternary search trees include spell-checking and auto-completion.

As with other trie data structures, each node in a ternary search tree represents a prefix of the stored strings. All strings in the middle subtree of a node start with that prefix.

In this blog post I present a c# version of Ternary Search Tree.

The interface

interface ITernaryTree<T>
    {
        void Add(string key, T value);
        bool Contains(string key);
        System.Collections.Generic.IEnumerable<string> Keys { get; }
        int Length { get; }
        System.Collections.Generic.IEnumerable<T> NearSearch(string query, int distance);
        System.Collections.Generic.IEnumerable<string> PrefixMatch(string prefix);
        System.Collections.Generic.IEnumerable<T> Search(string prefix);
        T this[string key] { get; }
        System.Collections.Generic.IEnumerable<string> WildcardMatch(string pat);
    }

The interface above defines the API for the data structure. Lets take a deeper look.

void	Add(string key, T value)	Add a key value pair in the tree.
bool	Contains(string key)	Check whether a key is in tree.
IEnumerable<string>	<string> Keys	Return all keys in the tree.
int	Length	Returns the length of the tree.
IEnumerable<T>	NearSearch(string query, int distance)	Returns all values for keys in the dictionary that are within a given Hamming distance of a query.
IEnumerable<string>	PrefixMatch(string prefix)	Returns all keys starting with a given prefix.
IEnumerable<T>	Search(string prefix)	Searches all values of keys starting with given prefix.
T	this[string key]	Gets the node value with the specified key
IEnumerable<string>	WildcardMatch(string pat)	Returns all keys matching given wilcard pattern.

The Trie Node

class Node
        {
            /// <summary>
            /// character
            /// </summary>
            internal char c;

            /// <summary>
            /// The  left, middle, and right subtries.
            /// </summary>
            internal Node left, mid, right;

            /// <summary>
            /// The value associated .
            /// </summary>
            internal T value;
        }

The Class is mostly self explanatory, but the key points to note is that it contains three internal node(sub-tries) unlike a traditional trie node.

Adding node in the data structure

        /// <summary>
        /// Adds the specified key.
        /// </summary>
        /// <param name="key">The key.</param>
        /// <param name="value">The value.</param>
        public void Add(string key, T value)
        {
            if (string.IsNullOrEmpty(key)) { throw new InvalidOperationException("Keys cannot be null or empty."); }
            if (!Contains(key)) N++;
            root = Add(root, key, value, 0);
        }

        /// <summary>
        /// Adds the specified node in the tree.
        /// </summary>
        /// <param name="node">The Node.</param>
        /// <param name="key">The key.</param>
        /// <param name="value">The val.</param>
        /// <param name="charIndex">The d.</param>
        /// <returns></returns>
        Node Add(Node node, string key, T value, int charIndex)
        {
            char charAtIndex = key[charIndex];
            if (node == null) { node = new Node(); node.c = charAtIndex; }
            if (charAtIndex < node.c) node.left = Add(node.left, key, value, charIndex);
            else if (charAtIndex > node.c) node.right = Add(node.right, key, value, charIndex);
            else if (charIndex < key.Length - 1)
                node.mid = Add(node.mid, key, value, charIndex + 1);
            else node.value = value;
            return node;
        }

Get

        /// <summary>
        /// Gets the specified x.
        /// </summary>
        /// <param name="node">The x.</param>
        /// <param name="key">The key.</param>
        /// <param name="charIndex">The d.</param>
        /// <returns></returns>
        Node Get(Node node, string key, int charIndex)
        {
            if (node == null) return null;
            char c = key[charIndex];
            if (c < node.c) return Get(node.left, key, charIndex);
            else if (c > node.c) return Get(node.right, key, charIndex);
            else if (charIndex < key.Length - 1)
                return Get(node.mid, key, charIndex + 1);
            else return node;
        }

and finally

PrefixMatch

        /// <summary>
        /// Returns all keys starting with a given prefix.
        /// </summary>
        /// <param name="prefix">The prefix.</param>
        /// <returns></returns>
        public IEnumerable<string> PrefixMatch(string prefix)
        {
            Queue<string> queue = new Queue<string>();
            Node node = Get(root, prefix, 0);
            if (node == null) return queue;
            if (node.value != null) queue.Enqueue(prefix);
            Collect(node.mid, prefix, queue);
            return queue;
        }

To see a complete working example check out the Github page here.

Full repo can be found here. There is a test web application demonstrating auto suggest included in the project, I will write more about it in a later post.

Gheat Java – Heat maps

2014-01-24T00:00:00z

A heat map is a graphical representation of data where the individual values contained in a matrix are represented as colors.

This article will attempt to explain the process or creating and using GHEAT-JAVA, which is a port of famous aspen based gheat and took great inspiration from Gheat.net

Writing a service which would serve heat map tiles is a bit tricky,there are three major components involved

The client part i.e. some kind of mapping library which has a concept of layer, I chose Google maps.
The data source part, i.e. a spatially aware database or an in memory data structure, I have used postgres ,an in memory quad tree and a flat file as data sources.
The renderer part or basically the code which excepts requests , parses tile bounds, fetches data and then renders gradients on the tile and later colorizes them.

The Tiling layer (Client part)

Google maps allows developers to add a custom layer , the code looks like this

function getHeatMapObject(colorScheme) {
            var heatMapOptions = {
                getTileUrl:function (tile, zoom) {

                    return 'http://localhost:8080?colorScheme=' + colorScheme + '&zoom=' +
                            zoom +
                            '&x=' + tile.x +
                            '&y=' + tile.y; //+
                    //remove this bit if browser caching is desired
                    '&rand=' + Math.random();
                },
                tileSize:new google.maps.Size(256, 256),
                isPng:true,
                releaseTile:function (tile, zoom) {
                    //Called when a tile is out of view
                },
                name:colorScheme + "Heat Map"
            };
            return new google.maps.ImageMapType(heatMapOptions);
        }

I have left the bit which deals with different color schemes out as its trivial.

The Data Source

The service can query a spatial database, I have included a data provider for postgres. Here is the sample query which the application executes

SELECT ST_X(geom) as longitude,ST_Y(geom) as latitude, weight as weight from mySpatialTable where geom @ ST_MakeEnvelope(?,?,?,?,4326)"

As its evident that a table is needed with x, y and weight columns, where x and y are the coordinates of datum in EPSG:4326 projection and weight is an integer between1 and 5

I have also included an in-memory spatial data structure called Quadtree which is pretty ideal for fast spatial look up, look in App.java to see how to populate this data source.

Finally there is a file data source which is pretty slow for large data lookup but is included anyways ;/

The Renderer

The rendering algorithm is as follows

Parse the request and get tile bounds.
Use the bounds to query data source for points.
if the data source returns empty, return empty tile,
Otherwise
1. Expand the height and width of tile by one dot width.(Apply padding)
2. Generate a blank image of desired dimensions (256 x 256)
3. Add gradient depending on the zoom level and weight for each point on the blank image.
4. To the gray scale gradient map we add color pixel by pixel by mapping them to the color scheme which is 255 pixels high.
5. Finally the tile is trimmed due to it having to be larger to accommodate points on the edges of a tile.
Return heat map tile.

Raphael-js-tutorial-part-II

2013-12-25T00:00:00z

This is second part of the tutorial about getting started with Raphael.js. To see this first part go here.

Writing Text with Raphael

Some code first ;)

var canvas = Raphael(0, 0, 250, 250 );

var t = canvas.text(50, 50, "Some text Goes here");

var t2 = canvas.text(60,90,"First line \n Second Line \n Third line");

var t3 = canvas.text(214, 140, "1079").attr({"text-anchor":"middle",'font-size':16,"font-family":"arial","fill":"#fff"});

In the above examples canvas.text will create a simple text element at 50,50. Strangely VML /SVG do not have Line breaks therefore the programmer is responsible for creating new lines,in most cases appending '\n' does the trick.

While styling your text ,attribute text-anchor is quite important, it defaults to 'middle' but can be changed to start and end as well.

Default Attributes Styling and adding Custom Attributes

Raphael honors SVG Attribute Specification and lets the user customize them by attr() method.

canvas = Raphael(0, 0, 250, 250 );
// Creates circle at x = 125, y = 125, with radius 50
var circle = canvas.circle(125, 125, 50);
// Sets the fill attribute of the circle to red (#f00)
circle.attr("fill", "blue");
// Sets the stroke attribute of the circle to white
circle.attr("stroke", "#fff");

Animation

The library comes with an animate function which is perhaps one of my favorite implementation, its simple easy and packs a big punch.

Animation is a basically a process of fiddling with the attributes of a vector in a timed routine, so lets say that we have a circle i.e

var circle = canvas.circle(125,125,50)

i.e. a circle with its center at 125,125 and radius 50 , so I want to animate the radius of the circle to say 25, I will simply do this


var anim = Raphael.animation({r:25}, 1000);

circle.animate(anim);

There are some helper methods built in like

circle.animate(anim.delay(500)); // run the given animation after 500 ms

or circle.animate(anim.repeat(Infinity)); One can also pass in a easing functions, like this

var anim = Raphael.animation({r:25,fill:"red"}, 1000,"bounce");

Check out more easing functions here

Finally I will add a custom widget which I animated from my last example in the first part.

Turn Off Autocomplete for Input

2013-12-01T00:00:00z

The autocomplete attribute specifies whether or not an input field should have autocomplete enabled.

Autocomplete allows the browser to predict the value from locally saved history/cache. When a user starts to type in a field, the browser should display options to fill in the field, based on earlier typed values.

Note: The autocomplete attribute works with the following <input> types: text, search, url, tel, email, password, datepickers, range, and color.

example usage:

<input name="q" autocomplete="off" />

Embedding Flash,Quicktime,Windows Media,Silverlight or HTML5 Video

2013-10-01T00:00:00z

Flash Object

This is quick Snippet which demonstrate how to embed flash object in a web page.

<object type="application/x-shockwave-flash"
  data="path-to-my-flash-file.swf"
  width="0" height="0">
  <param name="movie" value="your-flash-file.swf" />
  <param name="quality" value="high"/>
</object>

Quicktime

This is quick Snippet which demonstrate how to embed quicktime object in a web page

<object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B"
       codebase="http://www.apple.com/qtactivex/qtplugin.cab"
       width="200" height="16">
 <param name="src" value="pathToMyMovie.mov" />
 <param name="autoplay" value="true" />
 <param name="pluginspage" value="http://www.apple.com/quicktime/download/" />
 <param name="controller" value="true" />
 <!--[if !IE]> <-->
   <object data="movie.mov" width="200" height="16" type="video/quicktime">
     <param name="pluginurl" value="http://www.apple.com/quicktime/download/" />
     <param name="controller" value="true" />
   </object>
 <!--> <![endif]-->
</object>

Windows Media

This is quick Snippet which demonstrate how to embed Windows Media object in a web page.

<object type="video/x-ms-wmv"
  data="pathToMyMovie.wmv"
  width="320" height="260">
  <param name="src"
    value="pathToMyMovie.wmv" />
  <param name="autostart" value="true" />
  <param name="controller" value="true" />
</object>

Silver Light

This is quick Snippet which demonstrate how to embed Silverlight object in a web page.

<object width="300" height="300"
    data="data:application/x-silverlight-2,"
    type="application/x-silverlight-2" >
    <param name="source" value="SilverlightApplication1.xap"/>
</object>

HTML5 Video

<video width="300" height="300" controls>
  <source src="pathToMyMovie.mp4" type="video/mp4">
  <source src="pathToMyMovie.ogg" type="video/ogg">
Your browser does not support the video tag.
</video>

Supported Videoformats

Browser	MP4	WebM	Ogg
Internet Explorer	YES	NO	NO
Chrome	YES	YES	YES
Firefox	YES	YES	YES
Safari	YES	NO	NO
Opera	NO	YES	YES

Mod Proxy equivalent in IIS using ARR and URL Rewrite Module

2013-08-25T00:00:00z

IIS7 is quite modular, it is shipped with lots of goodies as separate modules and together it is now one of the most powerful and flexible web server.

In this post I intent to cover how we can easily configure ARR and URL Rewrite Module to get a similar functionality as of Mod Proxy in Apache.

Here I will demonstrate configuration of a reverse proxy which according to the definition is

Reverse proxy is a type of proxy server that retrieves resources on behalf of a client from one or more servers.

A general use case to configure such proxies could be to handle cross domain Ajax requests, so basically if I would like to call a service hosted as http://anotherdomain.com/service from http://localdomain/service I would need to configure a proxy which will request resources from anotherdomain.com/service on behalf of localdomain

Configuration

STEP 1: To set up a rule click on the web application from the sites tree on the left and then click on the URL Rewrite module, then form the command pallete on the right hand click on Add Rule as shown in image below.

STEP 2: From the add rule dialog choose reverse proxy template

Configure Inbound Rules window and provide server name as shown in screen shot below

Finally configure the last form, this is where one can specify cache parameters and override server header variables, such as basic authentication (http_authorisation) etc.

In the patterns text box entering something like ^/service(.*) will trap all requests going to http://localdomain/service and they will be proxyed as if the requests were made like http://anotherdomain.com/service , The query strings if present are also carried forward, all these settings and modification are preserved in web.config of the web application, under tag. The rules after configuration look like

<rewrite>
    <rules>
        <rule stopprocessing="true" name="ReverseProxyInboundRule1">
            <match url="^service/(.*)" />
            <action url="http://anotherdomain.com/service{R:1}" type="Rewrite" />
            <servervariables />
        </rule>
    </rules>
</rewrite>

If desired server variables and cache params can also be included.

Apart form using the reverse proxy template one can directly use url re-write module and even import settings from apache . Read here for more information.

I hope you will find this post use full. Happy coding ;)

Signing android release apk from the command line (via ant)

2013-08-25T00:00:00z

This post is probably out too late ,as almost all IDE's these days (which support android development) ,probably have an Integrated release signing or debug signing utility/wizard built in, but if somebody is using an old IDE or perhaps want to integrate release in a CI(continues integration) environment , this tutorial might come in handy.

Basically one has to sign an apk either with a debug key (which is generally present in your .android folder) or a create a fresh application specific keystore , to know more go here

Assuming you have setup android SDK properly in a Windows /Mac or Linux environment and all paths etc. are configured properly, lets begin by opening a command shell and navigating to the root of the android project,If the project is setup properly then a basic build command using ant ant clean debug should trigger a build

In order to sign an application we need to make a keystore that we use for the lifetime of this application. It's important to note that this keystore should be backed up in a secure place, losing it means you cannot install any updates on devices that have this app already installed. We are going to use the "keytool" program to create a keystore.

keytool -genkey -v -keystore release.keystore -alias mykey -keyalg RSA -keysize 2048 -validity 10000

This command will invoke a console wizard which will eventually create a keystore in the "release.keystore" file in the local directory with a key in it called mykey that will have 2048 size to it as well as be valid for 10,000 days. I generally include this file in my project version control and keep it in the same level as my ant.properties file

The next step is to modify your ant properties file and add these (I have store and alisa password same) key.store=release.keystore key.alias=mykey key.store.password=secret key.alias.password=secret and thats it, finall step is to issue a command ant clean release

If everything was successful it will say "BUILD SUCCESSFUL" and put an apk named something like "MyProject-release.apk" in the /bin directory.

Importing CSV File Into SQL SERVER Database using BULK INSERT

2013-03-11T00:00:00z

Importing data into a database from a delimited file is perhaps one of the most common tasks that one might have to perform. SQL server gives us an import utility which supports various data sources and has an intutive interface as well,however there is another way which can be utilized to quickly get the job done, its called BULK INSERT

BULK INSERT is a SQL command which can be typed in management console,takes various parameters to control the import and is very simple to use.

A typical use would look like this:

GO
BULK INSERT  from 'C:\sourceFile.csv' WITH
(
FIELDTERMINATOR =',',
ROWTERMINATOR ='\n'
)

and that's it, the command will automatically read csv file, try to do any default type conversion(determined by the TargetTables colum types) and will fail if there are any errors.

There are many other properties which can be set to make import easy ti handle, the ones I typically use are

GO
BULK INSERT  from 'C:\sourceFile.csv' WITH
(
FIELDTERMINATOR =',',
ROWTERMINATOR ='\n',

CODEPAGE='RAW',--Helps in Determining the source/destination Encoding (UTF8 etc)
MaxErrors = 999999-- This will allow 999999 errors to be ignored before the whole import fails. This is a workaround as its not possible to tell the importer to ignore errors completely.
)

Here is a full syntax and supported properties, for their explanation, please visit < href="http://msdn.microsoft.com/en-us/library/aa225968(v=sql.80).aspx" >this page.


BULK INSERT [ [ 'database_name'.] [ 'owner' ].] { 'table_name' FROM 'data_file' }
    [ WITH
        (
            [ BATCHSIZE [ = batch_size ] ]
            [ [ , ] CHECK_CONSTRAINTS ]
            [ [ , ] CODEPAGE [ = 'ACP' | 'OEM' | 'RAW' | 'code_page' ] ]
            [ [ , ] DATAFILETYPE [ =
                { 'char' | 'native'| 'widechar' | 'widenative' } ] ]
            [ [ , ] FIELDTERMINATOR [ = 'field_terminator' ] ]
            [ [ , ] FIRSTROW [ = first_row ] ]
            [ [ , ] FIRE_TRIGGERS ]
            [ [ , ] FORMATFILE = 'format_file_path' ]
            [ [ , ] KEEPIDENTITY ]
            [ [ , ] KEEPNULLS ]
            [ [ , ] KILOBYTES_PER_BATCH [ = kilobytes_per_batch ] ]
            [ [ , ] LASTROW [ = last_row ] ]
            [ [ , ] MAXERRORS [ = max_errors ] ]
            [ [ , ] ORDER ( { column [ ASC | DESC ] } [ ,...n ] ) ]
            [ [ , ] ROWS_PER_BATCH [ = rows_per_batch ] ]
            [ [ , ] ROWTERMINATOR [ = 'row_terminator' ] ]
            [ [ , ] TABLOCK ]
        )
    ]

Arguments 'database_name'

Importing Exporting CSV Files in PostgreSQL Databases via COPY

2013-03-11T00:00:00z

This is going to be a short article which will illustrate importing and exporting a table from or to a csv file using PostgreSQL COPY command.

Importing a table from CSV

Assuming you already have a table in place with the right columns, the command is as follows

COPY <TargetTableName> FROM '/path/to/csv/SourceCSVFile.csv' DELIMITERS ',' CSV;

Exporting a CSV from a table.

COPY <SourceTableName> TO '/path/to/csv/TargetCSVFile' DELIMITERS ',' CSV;

Its important to mention here that generally if your data is in unicode or need strict Encoding, then Always set client_encoding before running any of the above mentioned commands.

To set CLIENT_ENCODING parameter in PostgreSQL

set client_encoding to 'UTF8'

set client_encoding to 'latin1'

Another thing to guard against is nulls, while exporting , if some fields are null then PostgreSQL will add '/N' to represent a null field, this is fine but may cause issues if you are trying to import that data in say SQL server.

A quick fix is modify the export command by specifying what would you prefer as a null placeholder in exported CSV

COPY <SourceTableName> TO '/path/to/csv/TargetCSVFile' DELIMITERS ',' NULL as E'';

Another common requirement is import or export with the header.

Import CSV to table with Header for columns present in first row of csv file.

COPY <TargetTableName> FROM '/path/to/csv/SourceCSVFile.csv' DELIMITERS ',' CSV HEADER

Export a table to CSV with Headers present in the first row.

COPY <SourceTableName> TO '/path/to/csv/TargetCSVFile' DELIMITERS ',' CSV HEADER;

A complete list of all supported settings can be found here

COPY table_name [ ( column [, ...] ) ]
    FROM { 'filename' | STDIN }
    [ [ WITH ] ( option [, ...] ) ]

COPY { table_name [ ( column [, ...] ) ] | ( query ) }
    TO { 'filename' | STDOUT }
    [ [ WITH ] ( option [, ...] ) ]

where option can be one of:

    FORMAT format_name
    OIDS [ boolean ]
    DELIMITER 'delimiter_character'
    NULL 'null_string'
    HEADER [ boolean ]
    QUOTE 'quote_character'
    ESCAPE 'escape_character'
    FORCE_QUOTE { ( column [, ...] ) | * }
    FORCE_NOT_NULL ( column [, ...] ) |
    ENCODING 'encoding_name'

Raphael JS Tutorial - Part I

2013-03-10T00:00:00z

Raphael.JS is a SVG/VML library which helps you to create quick vector shapes and images. Its very small and compact (only 86kb),easy to use,well documented and works in all browsers and yes even IE. Here is a list of all major browsers this library works on:

Chrome 5.0+ Firefox 3.0+, Safari 3.0+, Opera 9.5+ and Internet Explorer 6.0+

,finally its distributed under MIT license.

In this tutorial ,I intend to introduce basics of this library and help you get started.

Lets begin with some basic concepts. Raphael.JS uses the SVG W3C Recommendation and VML as a base for creating graphics, it hides quite a lot of technology specific nitty-gritties and instead present a unified api model . Here is a quick code sample

// Creates canvas 250 × 250 at 0, 0
var canvas = Raphael(0, 0, 250, 250 );
// Creates circle at x = 125, y = 125, with radius 50
var circle = canvas .circle(125, 125, 50);
// Sets the fill attribute of the circle to red (#f00)
circle.attr("fill", "#f00");
// Sets the stroke attribute of the circle to white
circle.attr("stroke", "#fff");

The above code will create a circle in a middle of a canvas, here a canvas represent an HTML canvas element which is used as a drawing board with an id canvas. There is another way of binding your desired area with Raphael and that is by simply passing in the container element itself by using plain old document.getElementById("element").

var canvas = new Raphael(x, y, width, height); //first option
var canvas = new Raphael(document.getElementById('element'), width, height); //second option

Circle is one of the shape which can be drawn by using Raphael, others included in the library are ellipse,rect ,path and finally image.

Here is a quick sample of drawing circle,ellipse and rectangle

// Creates canvas 250 × 250 at 0, 0
canvas = Raphael(0, 0, 250, 250 );
// Creates circle at x = 125, y = 125, with radius 50
var circle = canvas.circle(125, 125, 50);// Sets the fill attribute of the circle to red (#f00)
circle.attr("fill", "blue");
// Sets the stroke attribute of the circle to white
circle.attr("stroke", "#fff");
// Creates rectangle at x = 5, y = 5, with width and height = 50
var rectangle = canvas.rect(5, 5, 50, 50);
rectangle.attr("fill", "green");
// Creates ellipse at x and y = 180 ,horizontal-axis = 20 and vertical-axis = 30
var ellipse = canvas.ellipse(180, 180, 20, 30);
ellipse.attr("fill", "red");

Even though above mentioned are the most basic shapes which one might need, however it's the path element which actually gives us a true flexibility

We start a path by calling canvas.path("<instructions") method, where instructions are a sequence of directions we can give a path element along with the coordinates. The list of instructions can be found here.

Command	Name	Parameters
M	moveto	(x y)+
Z	closepath	(none)
L	lineto	(x y)+
H	horizontal lineto	x+
V	vertical lineto	y+
C	curveto	(x1 y1 x2 y2 x y)+
S	smooth curveto	(x2 y2 x y)+
Q	quadratic Bézier curveto	(x1 y1 x y)+
T	smooth quadratic Bézier curveto	(x y)+
A	elliptical arc	(rx ry x-axis-rotation large-arc-flag sweep-flag x y)+
R	Catmull-Rom curveto*	x1 y1 (x y)+

Here a simple diagonal line is created using path

// Creates canvas 250 × 250 at 0, 0
canvas = Raphael(0, 0, 250, 250 );
// draw a diagonal line:
// move to 10,10, line to 90,90
var path = canvas.path("M10 10 L 190 190");
// Sets the stroke attribute of the path to red
path.attr("stroke", "red");
// Sets the stroke-width attribute of the path to 3.
path.attr("stroke-width", "3");

Path is extremly versatile and one could create other known shapes by using it. A simple circle can be created by using path for example

// Creates canvas 250 × 250 at 0, 0
canvas = Raphael(0, 0, 250, 250 );
// Creates circle shape using path at x = 125, y = 125, with radius 50
var circleDrawnasPath = canvas.path(getCircletoPath(125, 125, 50));
circleDrawnasPath.attr("stroke", "green");
circleDrawnasPath.attr("stroke-width", "3");

//Helper method to take x,y and r and return a path instruction string.
// x and y are center and r is the radius
function getCircletoPath(x , y, r)  {  return "M"+ x + "," + (y - r) + "A" + r + "," + r + ",0,1,1," + (x - 0.1) + "," + (y - r) +" z"; }

Paths can be used to create other intesting controls as well

			var canvas;
			window.onload = function () {
				// Creates canvas 250 × 300 at 0, 0
				canvas = Raphael(0, 0, 250, 270 );
				DrawBars(
				{
					"56%":56,
					"28%":28,
					"15%":15,
					"01%":1,
				} );
			};
			var DrawBars = function(data)
			{
				var vals = [];
				var labels = [];
				for(var key in data)
				{
					labels.push(key);
					vals.push(Number(data[key]));
				}
				var startingY = 55;
				var totalHeight = 60;
				var verticalGap = 8;
				var Height1 = Math.round(vals[0]/100*totalHeight);
				var Height2 = Math.round(vals[1]/100*totalHeight);
				var Height3 = Math.round(vals[2]/100*totalHeight);
				var Height4 = Math.round(vals[3]/100*totalHeight);

				//Cube 4
				drawCube((startingY + Height1 + Height2 + Height3 + (verticalGap*3)),Height4,canvas,"#BBE6FB","#89D7f8","#39C8F4",labels[3]);
				//Cube 3
				drawCube((startingY + Height1 + Height2 + (verticalGap*2)),Height3,canvas,"#3FC8F4","#0CB5EB","#0094DA",labels[2]);
				//Cube 2
				drawCube((startingY + Height1 + (verticalGap*1)),Height2,canvas,"#2A80B8","#0269A6","#003F84","28% Android",labels[1]);
				//Cube 1
				drawCube(startingY,Height1,canvas,"#7F8184","#575759","#221E1F",labels[0]);

			}

			var drawCube = function(y,height,canvas,f1,f2,f3,label)
			{
				var rect1 = canvas.path("M 0 " + y + " 57 " + (y + 22) +" 188 " + y +" 127 " + (y - 16) + " 0 " + y);
				rect1.attr({fill: f1, stroke: f1, 'stroke-width': 0});

				var rect2 = canvas.path("M 0 " + y + " 0 " +(y + height) +" 57 " + (y + height + 22) + " 57 " + (y + 22) + " 0 " + y);
				rect2.attr({fill: f2, stroke: f2, 'stroke-width': 0});

				var rect3 = canvas.path("M 57 " + (y + 22) +" 57 " + (y + height + 22) +" 188 " + (y + height)+" 188 " + y +" 57 " + (y + 22));
				rect3.attr({fill: f3, stroke: f3, 'stroke-width': 0});
				var labely = y+height/2;
				var label1 = canvas.path("M 188 " + labely + " 226 " + labely)
				label1.attr({stroke: "#fff", 'stroke-width': 1});
				var text = canvas.text(226, labely, label);
				text.attr({"text-anchor":"start",'font-size':12,"font-family":"arial","fill":"#fff"});


			}

Rapael.js is a very versatile library, it suports many other feature like drawing images , text and animation. I intend to cover all that in future posts. Hope this helps.

How to configure Apache mod_wsgi

2013-02-26T00:00:00z

I am a big fan and user of python. one of the most popular ways to create quick web app in python is via using mod wsgi.

The aim of mod_wsgi is to implement a simple to use Apache module which can host any Python application which supports the Python WSGI interface.

The module would be suitable for use in hosting high performance production web sites, as well as your average self managed personal sites running on web hosting services.

There are many frameworks in python, I like web.py so here is a quick example.

First lets configure modwsgi in Ubuntu

sudo apt-get install libapache2-mod-wsgi

sudo a2enmod wsgi

sudo service apache2 restart

configuration in /etc/apache2/sites-available/defaults

  WSGIScriptAlias /appname /var/www/webpy-app/code.py/

  Alias /appname/static /var/www/webpy-app/static/
  AddType text/html .py

  <Directory /var/www/webpy-app/>
      Order deny,allow
      Allow from all
  </Directory>

There is generally a handler present in most web applications for wsgi calls, here is one from web.py

 import web

  urls = (
      '/.*', 'hello',
      )

  class hello:
      def GET(self):
          return "Hello, world."

  application = web.application(urls, globals()).wsgifunc()

and that's it.Hope it helps

Serve the contents of any directory with Python's SimpleHTTPServer

2013-02-08T00:00:00z

Generally, when I am in a middle of prototyping a concept or in a need of quickly executing Ajax requests or using browser features which would need the page to be hosted on a web server, I use Python's SimpleHTTPServer module.

Python's SimpleHTTPServer is a great way of serve the contents of the current directory,all one needs to do is change directory and execute a command which will expose all contents as if they were hosted in a web page.

cd myfolder
python -m SimpleHTTPServer

you don't need a line of code or a even a python file anywhere inside the directory for this to work, the module works all on its own and just needs a port number.

If there is a file called index.html inside the root, then its served when the user types localhost:port.

SimpleHTTPServer is a great module for quick start, however its not a production ready tool and must be used only for lightweight use.

Another use of this module is a quick utility Proxy. This does need a few lines of code though :!

import SocketServer
import SimpleHTTPServer
import urllib

PORT = 3333

class Proxy(SimpleHTTPServer.SimpleHTTPRequestHandler):
    def do_GET(self):
        self.copyfile(urllib.urlopen(self.path), self.wfile)

httpd = SocketServer.ForkingTCPServer(('', PORT), Proxy)
print "serving at port", PORT
httpd.serve_forever()

Hope it helps.

Avoid console errors in browsers that lack a console

2013-01-30T00:00:00z

I love JavaScript and code a lot in it .Since I code, I also encounter problems and bugs from time to time and therefore need a Debugger and a debug mechanism to view whats going on.

In good old days when I learned this language, I used JavaScript's inbuilt alert mechanism to view the value of a variable or a property of an object. This was a terrible mechanism, calling alert was blocking, one had to press OK to continue and if you had to take a look inside an array then well it was emotionally an overwhelming task is all I would say here.

There were of course workarounds and techniques(like printing in a text area) but I guess people soon realized the need of console like mechanism which is present in other languages.

I had my first encounter with console in FF and right there I had a Nerdgasm. It was nice , easy to use and simply put beautiful, and then chrome came in and did it even better. Today id you print an xml object then good capable consoles will actually pretty print the markup and objects will be printed in a tree structure thus a JavaScript programmer can breathe easy, well almost.... we still have IE to haunt us for ever.

IE will as always screw with your happiness and will find a way to make you do something extra and different . To enable console in IE, Click the "Script" tab, then click "Console" on the right, however your JavaScript will throw error if you simply use console.log without enabling it, which we can safely assume our users will not.

Here is a workaround

HTML5 Boilerplate guys present us with this technique

(function() {
    var method;
    var noop = function () {};
    var methods = [
        'assert', 'clear', 'count', 'debug', 'dir', 'dirxml', 'error',
        'exception', 'group', 'groupCollapsed', 'groupEnd', 'info', 'log',
        'markTimeline', 'profile', 'profileEnd', 'table', 'time', 'timeEnd',
        'timeStamp', 'trace', 'warn'
    ];
    var length = methods.length;
    var console = (window.console = window.console || {});

    while (length--) {
        method = methods[length];

        // Only stub undefined methods.
        if (!console[method]) {
            console[method] = noop;
        }
    }
}());

my personal fav is this

var log = function (text) {
    // if (!window.console) window.console = {};
    // if (!window.console.log) window.console.log = function () { };
    "use strict";

    if (typeof (console) !== "undefined" && console.log !== undefined) {
        try {
            console.log.apply(console, arguments);
        } catch (e) {
            var log = Function.prototype.bind.call(console.log, console);
            log.apply(console, arguments);
        }
    }
}

use which ever you prefer. Hope this helps.

sqlite3 Bulk Import CSV from command line

2013-01-29T00:00:00z

I absolutely love SQLite, be it windows, linux or mac, this light weight database never stops to amaze me.

Few days ago I wanted to import a chunk of CSV data, which was 4 GB in size into a table in SQLite.

I generally use Firefox 's sqlite-manager plug-in for usual tasks with SQLite database ,hence decided to use the import interface .

I have had a very good experience with this GUI tool and it has rarely failed me, but this time I was disappointed.

The import process started normally, but my RAM started increasing rapidly till it was 60% and after 20 minutes I was presented with some kind of a stack overflow/out of memory exception,luckily SQLite website is also one of the best documented ones, a quick search brought me to the command line utility page, which has a .import command which I was looking for.

Before importing a CSV, you may want to have a well defined database structure,although this is not compulsory but I would highly recommend that you do that, once done then simply open the SQLite shell by typing sqlite3 followed by the database file name, if you are in windows then you can download the command line utility packaged as an exe and open that via shell by typing in sqlite3 <filename> , the default separator is ',' , to change use .separator command in shell sqlite> .separator '|'

The next step is to let the utility know that you intend to use a CSV, by typing in sqlite> .mode csv and finally the import command

sqlite> .import <filename> <tablename>

That should do it, there are more commands to configure the delimiter and the escape characters which can be found here.

The entire import of 28122486 was finished in less than 6 minutes and surprisingly the SQLite file database took less space than the original CSV, the original file was 4.2 GB where as the SQLite file database file was only 3.13 GB, that's a whopping gig difference.

I hope you find this useful and use SQLite more than you already do.

Install Apache Cassandra

2013-01-27T00:00:00z

Cassandra is an Open Source Distributed Data Persistence system which is designed for storing and managing large amounts of data across servers.

To quote elevator pitch by Eben Hewitt

Apache Cassandra is an open source, distributed, decentralized, elastically scalable, highly available, fault-tolerant, tuneably consistent, column-oriented database that bases its distribution design on Amazon’s Dynamo and its data model on Google’s Bigtable. Created at Facebook, it is now used at some of the most popular sites on the Web.

Installing Cassandra

Cassandra can be installed in most popular operating systems Windows (vista/XP/7/8), Mac OSX, and Linux variants such as Ubuntu, Red Hat, and CentOS.

Ideally all platforms which have JVM 1.6 or higher should be just fine. There is a Debian Packaging or a third party RPM distribution by DataStax.

There is a more popular way of downloading the tar distribution , follow this article by downloading and then un packing binary distribution from Apache website.

I am demonstrating this in windows 7.

Once un packed the folder structure should look like this.

I generally create an environment variable JAVA_HOME which points to java jdk and CASSANDRA_HOME which points to the root directory of Cassandra as shown above in the screen shot.

Quick details about the directory structure under Cassandra root

bin
As the convention has it, this directory contains executable(batch and shell) to run Cassandra, along with the startup scripts and the nodetool utility. It also has scripts for converting SSTables (the datafiles) to JSON and back.
conf
This folder contains configurations for Cassandra. Storage-conf.xml file allows to create data store by configuring keyspace and column families,cassandra.yaml and SH file are for configuring cassandra and the environemnt, log4j files are to configure logging levels.
interface
This folder contains an RPC Description file defining Cassandra's interface in cassandra.thrift
javadoc
Contains Standard Javadoc API documentation for Cassandra.Cassandra is a wonderful project, but the code contains precious few comments, so you might find the JavaDoc’s usefulness limited. It may be more fruitful to simply read the class files directly if you’re familiar with Java.
lib
This folder contains all the dependecies and libraries which Cassandra needs, there are json parsers and google's collection libraries.

There is a git repo with source available git clone git://git.apache.org/cassandra.git

Before starting Cassandra

By default, Cassandra uses the following directories for data and commitlog storage (this is linux filesystem):

/var/lib/cassandra
/var/log/cassandra

In the past (version <= 0.6), Cassandra use to have a file called storage-conf.xml, however from 0.7 all log related stuff goes in log4j-tools.properties,in windows I recommend changing them appropriately.

If in Linux/Unix machine, you may have to give rights to the folder which Cassandra creates.

 cd apache-cassandra-$VERSION

 sudo mkdir -p /var/log/cassandra

 sudo chown -R `whoami` /var/log/cassandra

 sudo mkdir -p /var/lib/cassandra

 sudo chown -R `whoami` /var/lib/cassandra

If you like to configure other variables DataStax has a nice description of what goes in the cassandra.yaml

Start Cassandra in windows

open command prompt, and type

cd %CASSANDRA_HOME%/bin
cassandra.bat -f

Start cassandra in *nix

cd $CASSANDRA_HOME
sh bin/cassandra -f/

You should see something like this (Window screen shot)

Hope this helps

Position div in the center of the page using css

2013-01-27T00:00:00z

When I first started web programming, creating a center aligned div was one of the most common task that I saw myself doing.

since div is a block element, i.e. it takes up the full width available, and has a line break before and after it, it can easily be centered using relative styling

.center
{
 margin-left:auto;
 margin-right:auto;
 width:70%;
 background-color:#b0e0e6;
}

However lately I have preferred using absolute positioning technique over relative, this works for all browsers and is best for login screens or alert messages over a translucent shim.

.center{
    position: absolute;
    top: 50%;
    left: 50%;
    height: 200px;
    width: 200px;
    margin-left: -100px
    margin-top: -100px;
}

In case yer wondering what is happening above, then here is a quick explanation.

Absolute positioning works by treating the browser view port as a drawing board and top left and bottom right as axis-es. In the code above any element with center class will have top and left corners aligned to the center of the page, however since I need to align the center of the element with the page, I push them halfway to the right and half way to the top by applying negative margins.

Here is how the div would look inside view port without margins

This is how it looks with them (Properly aligned to the center)

A guide to hosting in Red Hats OpenShift PaaS

2013-01-25T00:00:00z

In this post,I am going to write about OpenShift, which is Red hat's free, auto-scaling Platform as a Service (PaaS) for applications.

OpenShift to me is very similar to Google's app engine in some ways,but openshift's offerings are quite diverse, for instance support for ruby, php, Perl, node.js, etc..

Openshift is not like a traditional VPS hosting , although once registered with them, user does get a shell access,but its quite limited , instead OpenShift has a concept of gears and cartridges.

A Cartridge is a like a capability which an application can use, for eg: mysql,mongodb,phpmyadmin etc. There are some officially supported ones available for quick inclusion otherwise users can create their own as well,even though thats a tricky task.

I also like the pricing structure of OpenShift (Developers Preview), which may be what many developers are looking for,however the web site says right at the top that it may change in future.

So basically to use,Redhat's PaaS hosting, one needs to
1. Quickly sign up for an account.

2.Use web interface to create an application (good for rapid kick start),or use a ruby command line client tool(good for coding/debugging/app-management) or even use tools integrated in IDE (Eclipse)

3.OpenShift follows a code check-in model,where a git repository is created and associated with the application created in above step, every check in in that folder structure will cause a rebuild of environment and will reboot the web server.

To download the repo simply type git clone ssh://...rhcloud.com/~/git/myapp.git/

When changes have been made then type

$ cd myapp
$ vim php/index.php
(Make a change...  :wq)
$ git commit -a -m "My first change"
$ git push

To get you up and running quickly OpenShift has a bunch of quick starts here, there is also a getting started guide.

OpenShift also has a well maintained blog , a FAQ and even videos.

I must say that the companion rhc command line utility is very helpful, it can be installed in all popular operating systems.

I also like their github page where they have many examples (see the bottom of list).

There is a concept of gear in openshift, which to me is very similar to what a compute instance is in Amazon.

This is what guys at OpenShift say about it

A gear is a resource constrained container that runs one or more user-specified software stacks, also known as cartridges. Each gear has a limited amount of RAM and disk space. If an application needs more resources, it can use multiple gears. Gears come in multiple sizes to suit the needs of various software stacks.

If you would like to read more about their architecture, here is a link, the best part is that every bit is open sourced.

Basic authentication in web.py via attribute

2013-01-23T00:00:00z

Here I demonstrate the process of Basic Authentication in web.py python web framework.

There is a proof of concept article provided in the main site,however I just thought doing the same via an attribute might be a cleaner solution.

HTTP Basic authentication implementation is one of the easiest ways to secure web pages because it doesn't require cookies, session handling, or the development of login pages. Rather, HTTP Basic authentication uses static headers which means that no handshakes have to be done in anticipation,however the n the credentials are passed as plain-text and could be intercepted.

The header looks something like this

WWW-Authenticate: Basic realm="insert realm"

To construct the header

Username and password are concatenated into a string "username:password" and this sting is encoded using Base64 and then a constant string is prepended with a space Authorization: Basic . So the final value looks like Authorization: Basic dXNlcm5tYWU6cGFzc3dvcmQ=

To decode the string on server side, the reverse of above mentioned process is performed.

###############################################################################
# BASIC AUTH      #############################################################
###############################################################################

def check_auth(username, password):
    return username == 'username' and password == 'password'


def requires_auth(f):
    @wraps(f)
    def decorated(*args, **kwargs):
        auth = web.ctx.env['HTTP_AUTHORIZATION'] if 'HTTP_AUTHORIZATION' in  web.ctx.env else None
        if auth:
            auth = re.sub('^Basic ', '', auth)
            username, password = base64.decodestring(auth).split(':')
        if not auth or not check_auth(username, password):
            web.header('WWW-Authenticate', 'Basic realm="admin"')
            web.ctx.status = '401 Unauthorized'
            return Unauthorized()

        return f(*args, **kwargs)

    return decorated

class Unauthorized():
    def GET(self):
        return "401 Unauthorized"

    def POST(self):
        return "401 Unauthorized"

So basically if a route handler needs authentication, all it needs is an attribute

@requires_auth
class Index:
    def GET(self):
	pass;

Hope this helps.

Cross browser drop shadow with CSS3

2013-01-23T00:00:00z

When css3 arrived, one of the goodies that I found was, the inclusion of Drop Shadow effect, namely Box Shadow and Text Shadow.

Here I present a cross browser utility css class which can be used for a drop shadow effect, without any image.

.shadow {
       /* For IE 8 */
	-ms-filter: "progid:DXImageTransform.Microsoft.Shadow(Strength=4, Direction=135, Color='#000000')";
	/* For IE 5.5 - 7 */
	filter: progid:DXImageTransform.Microsoft.Shadow(Strength=4, Direction=135, Color='#000000');
	-moz-box-shadow: 3px 3px 4px #000;
	-webkit-box-shadow: 3px 3px 4px #000;
	box-shadow: 3px 3px 4px #000;
}

For cross browser text shadow effect ,I found a nice article here.

* default setup that everything sees */
.shadow {
  /* needed for Internet explorer */
  height: 1em;
  filter: Shadow(Color=#666666,
                 Direction=135,
                 Strength=5);

  /* Needed for Gecko */
  line-height: 2em;
  white-space: nowrap;
}

/*
 * used by browsers which know about
 * :before to create the shadow
 */
.shadow:before {
  display: block;
  margin: 0 0 -2.12em 0.15em;
  padding: 0;
  color: #666666;
}

#shadow_1:before {
  content: 'In shadow';
}
#second_2:before {
  content: 'Happy Shadowing!';
}

/*\*/
html*.shadow {

  [color:red;/* required by Safari
              * so that [] is correctly
              * begun. associated with
              * the property, yet hiding
              * it. Seen by IE6 */

    /*
     * seen by IE6 and Safari, but hidden
     * from Gecko
     */
    text-shadow: #666666 5px 5px 5px;

  ]color:auto; /* resets color for IE6 */
}/**/

/*
 * end hack using dummy attribute selector
 * for IE5 mac
 */
.dummyend[id]{clear: both;}

/*\*/
html*.shadow:before {

  [color:red;/* required by Safari.
                seen by IE6 */

    /*
     * seen by IE6 and Safari, but hidden
     * from Gecko
     */
    display: none;

  ]color:auto; /* resets color for IE6 */
}/**/

/*
 * end hack using dummy attribute selector
 * for IE5 mac
 */
.dummyend[id]{clear: both;}

Hope this helps.

Cross browser CSS3 border-radius (rounded corners)

2013-01-22T00:00:00z

When I started programming and had to begin designing my first user interface in HTML, I was always puzzled as to why are all visual elements in HTML rectangular. I soon learned that in those days if one would want a circle to be drawn on a page, then the only way possible was using an image, the same was true for drawing elements with rounded corners.

Thankfully HTML5/CSS3 guys were listening and the latest css3 standard supports defining the corner radius of a circle, this can be used to make an element look like a circle too if you want.

Here I present a cross browser css3 class which will introduce a curve on otherwise rectangular elements.

Border-radius for modern browsers including IE9

.curve{
  -webkit-border-radius: 10px; /* Safari, Chrome */

  -moz-border-radius: 10px; /* Firefox */

  border-radius: 10px; /* CSS3 */
}

Border-radius for IE (version < 9)

Download the border-radius.htc file,and put it on the website.

.curve{
  behavior: url(http://mywebsite.com/border-radius.htc);
}

If you would like to convert your rectangular element to circle the trick is to make your corner radius half the width/height of your shape and make sure that the shape is originally a square,i.e. height and width of the shape should be same.

#circular {
height: 100px;
width: 100px;
-webkit-border-radius: 50px;
-moz-border-radius: 50px;
border-radius: 50px;
}

In above example, a div of id circular with height and width of 100px will look circular as the border radius is 100 / 2 = 50px . Hope this helps ;)

Cross Browser Opacity or Transparency

2013-01-22T00:00:00z

Css has a very useful property called Opacity, which basically is a measure of transparency of an element.

All modern browsers Yes yes even IE has support for it.

In most browsers the value is between 0 minimum [transparent] i.e not visible to 1 maximum [opaque] i.e. completely visible

In IE the value is between 0 and 100 ,don't ask why they just like to make us unhappy.

Here I present a helper css class which will work in all browsers.

.transparent {
 /* IE 5-7 */
  filter: alpha(opacity=50);

  /* IE 8 */
  -ms-filter: "progid:DXImageTransform.Microsoft.Alpha(Opacity=50)";

  /* FF/Netscape */
  -moz-opacity: 0.5;

  /* Safari 1.x */
  -khtml-opacity: 0.5;

  /* Good browsers  (aka chrome ;) ) */
  opacity: 0.5;
}

I hope you find this use full

Installing Redis on Ubuntu

2013-01-20T00:00:00z

Redis is an open source, advanced key-value store. Installing it on Linux debian platforms is pretty easy.

There are two ways of getting this done, one is perhaps an easy and less verbose method and it involves using an alternative repository Dotdeb.

A while ago I had posted a tutorial about installing redis from the googlecode repo, but things have changed since then and here I post a fairly latest way of installing it.

Install using aptitude and Dotdeb repo

My experience with Dotdeb has been pretty nice,it has a bunch of nice packages, always up to date with the very latest version.

Begin by adding Dotdeb repositories to your APT sources. Create a new list file in /etc/apt/sources.list.d/ and fill it with the following content.

# /etc/apt/sources.list.d/dotdeb.org.list
deb http://packages.dotdeb.org squeeze all
deb-src http://packages.dotdeb.org squeeze all

Then you need to authenticate these repositories using their public key. wget -q -O - http://www.dotdeb.org/dotdeb.gpg | sudo apt-key add - Finally update the apt-cache and install

$ sudo aptitude update
$ sudo aptitude install redis-server

Manual Install

The other way is downloading the source, building it and then installing it

Download redis from its git repo. git clone https://github.com/antirez/redis.git Change directory to the redis folder and type make, for 32 bits typemake 32bit and then test make test To run redis with default configuration change directory to src cd src and then type ./redis-server If you want to provide your custom conf, then simply pass it in as an arg ./redis-server /path/to/redis.conf To install Redis here /usr/local/bin type make install , this however simply copies the binaries at the said location. Redis provides a proper script to install it properly in Ubuntu and Debian systems. change directory to Utils cd utils and then execute./install_server use commands below to start stop the server

$ sudo start redis-server
$ sudo restart redis-server
$ sudo stop redis-server

gzip compression in apache using mod_deflate

2013-01-19T00:00:00z

HTTP Compression is a very simple and effective way to save bandwidth and improve web applications performance over network.

Output compression is basically a process of compressing web servers response by using a loss-less compression algorithm called gzip.

This technique is fairly modern and almost all modern browsers honor it, however if a page is requested from a browser which does not send a header Accept-Encoding: gzip,deflate then the response comes back uncompressed.

In Apache under Ubuntu OS this is fairly simple.

Enable mod_deflate in Apache2 sudo a2enmod deflate

configure /etc/apache2/httpd.conf mod_deflate by adding content mentioned below.

# compress text, html, javascript, css, xml:
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
# Or, compress certain file types by extension:
<files *.html>
SetOutputFilter DEFLATE
</files>

Restart apache to anable the changes sudo apachectl graceful

There is another powerfull apache module called mod_gzip which can precompress output as well.

Hope this helps.

Start Stop Restart Apache 2 Web Server

2013-01-19T00:00:00z

I have been using Linux environment a lot lately, particularly for hosting my web experiments and other application. One of the biggest and most searched question for newbies have been

How do I restart Apache 2 Web Server under Debian / Ubuntu Linux or UNIX operating systems?

In this post I simply attempt to lay this down for my and their benifit.

Start Apache2 # sudo /etc/init.d/apache2 start

Stop Apache2 # sudo /etc/init.d/apache2 stop

Restart Apache2 # sudo /etc/init.d/apache2 restart

Graceful # sudo /etc/init.d/apache2 graceful or # sudo /etc/init.d/apache2 reload

There are other less verbose ways as well. If using apache2ctl (If its not on your path generally apche2ctl can be found here /usr/sbin/apache2ctl

Restart Apache2 gracefully sudo apache2ctl graceful Stop Apache2 gracefully sudo apache2ctl graceful-stop

Factorial in python

2012-08-14T00:00:00z

Mathematically, the formula for the factorial is as follows. If n is an integer greater than or equal to 1, then n ! = n ( n - 1)( n - 2)( n - 3) ... (3)(2)(1) If p = 0, then p ! = 1 by convention.

def factorial(n):
    if n == 0:
        return 1
    else:
        return n * factorial(n-1)

Hope this helps

Auto Complete with Redis (Python)

2012-05-06T00:00:00z

This is a quick port of auto complete implementation in ruby using Redis by antirez.

I like python and had little time to burn today, so here goes:

The file used for input is female-names.txt

Hope it helps :)

How To Install and use Python Web.py framework on Windows

2012-05-06T00:00:00z

Web.py has been one of my favorite web frameworks as its pretty easy to get cracking on it.

It's super quick to install and one can come up with a prototype and rapid web services in matter of minutes.

Install on windows

If you haven't configured easy_install on windows, then read this article.

Once easy_install has been configured believe it or not, all you have to do is open a command prompt and type

easy_install web.py

and that’s it, you are all set , to test if things work properly, create a test.py file and paste this code from Web.py web site

  import web

  urls = (
      '/(.*)', 'hello'
  )
  app = web.application(urls, globals())

  class hello:
      def GET(self, name):
          if not name:
              name = 'World'
          return 'Hello, ' + name + '!'

  if __name__ == "__main__":
      app.run()

run the code by typing python test.py 3333, then open the web browser and go to

http://localhost:3333/Einstein, on the browser you should see Hello,Einstein.

Hope this helps .

How to setup easy_install on Windows

2012-05-06T00:00:00z

If one has been using python, then installing various libraries and modules is basically a breeze using easy_install utility, however for folks using windows, easy_install utility has to be setup properly before using it.

First lets make sure that python is properly installed and PYTHON_HOME environment variable is configured:

Install Python on Windows

If not already installed download python installer from here.

After it's done downloading, double click to run the installer, and select default options (unless you have other custom needs of course ).

Once done lets quickly setup a PYTHON_HOME environment variable which points to the python directory which contains python exe, in my case its c:/Python27, to set up environment variable right click "My Computer select properties Advanced Tab Environment Variables Button , add new environment variable and call it “PYTHON_HOME” and set path to whatever is in your case, to make sure every thing is proper ,open NEW dos prompt and type echo %PYTHON_HOME% and you should see the path you have entered.

Next open you environment variables GUI again as instructed above and this time locate your path variable, once found append %PTHON_HOME% at the end by separating it with a semicolon. To check everything is setup properly open NEW command line and enterecho %PATH%, and you should see your entry at the end of the string.

Setup Easy_Install

First we'll need to download setuptools.Download the exe and run it.

The directory that contains easy_install is by defaultc:\<pythonfolder>\Scripts

All you need to do now is append %PYTHON_HOME%\Scripts , separated by a semicolon ; at the end of path variable in your environment settings as instructed above, as usual enterecho %PATH% to confirm change, then open command promt and type easy_install and you should see a warning (its ok we havent passed in a package name or url )

error: No urls, filenames, or requirements specified (see --help)

And that’s it folks. Happy easy_installing.

Find Longitude and Latitude in Google Maps

2012-04-15T00:00:00z

This post is basically a way of quick appreciation to the Google maps Team, they are obviously a bunch of caring, loving people ,who make useful (although surreptitious) tools for the betterment of Geek Kind.

So as most of you may have accidently clicked a mysterious looking link at the bottom of your Google maps page ( on the left hand panel)

This awesome looking link opens up a popup which lets you choose wonderful goodies which makes your Google maps page more useful than it already is:

Now one can choose things like Smart Zoom but this post is about LatLng marker and tooltip ,

which are very handy,especially if one works as a GIS application developer.

Ones you are done selecting, simply click Save changes and that’s it.(You may have to refresh your page though ).

After these features have been enabled, a user can now right click anywhere on the map, and in the options window , a new link “Drop latLng Marker” will appear.

Select that and the obvious will happen.

and there you have it , an easy, helpful, Hidden(not anymore), but never the less extremely useful utility to get latitude and longitude from Google maps.

How to set up Redis in Ubuntu Linux

2012-04-14T00:00:00z

Redis is an open source, advanced key-value store, and is quickly picking up momentum in real-time software development , it is now a well known and trusted product and can be used in various problem scenarios as a No SQL implementation.

Redis is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

Here is quick guide to Install Redis in Ubuntu 10.04, and it should be similar for higher versions and similar deb environment.

Prepare System for Redis

Issue the following commands to update your system's package repositories and ensure that all installed packages are up to date:

apt-get update
apt-get upgrade

You may also need to prepare a vanilla system to get some installation tools. apt-get install build-essential

Begin the installation process by issuing the following sequence of commands to download the software and prepare it for use:

cd /tmp
wget http://redis.googlecode.com/files/redis-2.2.4.tar.gz
tar -zxf redis-2.2.4.tar.gz
cd redis-2.2.4
make
sudo make install

Your Redis binaries should now be located in /usr/local/bin.

note:Check the Redis upstream project source to ensure that you are downloading the most up to date version of Redis

All Redis configuration options can be specified in the redis.conf file located at /tmp/redis/redis.conf.

Sample redis conf

daemonize yes
pidfile /var/run/redis.pid
logfile /var/log/redis.log
port 6379
bind 127.0.0.1
timeout 300
loglevel notice
##Default configuration options
databases 16
save 900 1
save 300 10
save 60 10000
rdbcompression yes
dbfilename dump.rdb
dir /tmp/redis/
appendonly no
glueoutputbuf yes

Most of the values in this configuration mirror the default Redis configuration. However, this configuration configures Redis to run in a daemon mode bound only to the local network interface. You may want to change these values depending on the needs of your application.

Before you can fire up the Redis server for the first time, you’ll need add a Redis user and prep a data and logging folder.

sudo mkdir -p /var/lib/redis
sudo mkdir -p /var/log/redis
sudo useradd --system --home-dir /var/lib/redis redis
sudo chown redis.redis /var/lib/redis
sudo chown redis.redis /var/log/redis

Also, you need to activate your Redis services init script by adding it to your system’s run-level configuration. That way the service will startup during the boot sequence and stop nicely during the OS’ shutdown procedure.

sudo update-rc.d redis-server defaults

Finally Start

sudo /etc/init.d/redis-server start

Installing Ubuntu Linux in VMware and VMwareTools

2012-01-09T00:00:00z

I have been using Ubuntu for last two years as a deployment environment for my pet projects, however recently, after having installed “Natty Narwhal” on one of my old ‘HP’ laptop I moved into using Ubuntu as a full time development environment.

There are times when I need more than one deployment units and since I use Linodeas my primary deployment VPS host, I sometime need to test the perf of my software locally thus there came a need of replicating an exact or close to exact “Deployment Environment”

Enter VmWare, as most of you folks know, there is nothing like a quick and dirty VM Image boot, and VmWareis one of the best in the market although “ Oracle’s Virtual box” and “Parallels for mac” are pretty awesome too, there also is a repository for pre-installed vanilla images for most of the popular flavours of Linux here.

The cheapest version of Linode is 512MB RAM and 20GB HD and this is exactly what you get when you download the image from above link, although you can tweak and configure it to you likings at all times.

Another important step to remember is installing VMware Tools .

With the VMware Tools SVGA driver installed, Workstation supports significantly faster graphics performance. The VMware Tools package provides support required for shared folders and for drag and drop operations. Other tools in the package support synchronization of time in the guest operating system with time on the host, automatic grabbing and releasing of the mouse cursor, copying and pasting between guest and host, and improved mouse performance in some guest operating systems. To install choose VM > Install VMware Tools from the VMware Workstation menu, although you will be automatically prompted when you boot you Image for the first time.

If this is your first time dealing with this stuff then you may need a little help installing the script written in pearl, If GNOME has been your baby for a while, then the installed location and booting your image is not so obvious for the command line server interface.


  #install kernel headers so modules will work
  #needed this on a 10.04 guest running in a Fusion 3 host
  apt-get install linux-headers-virtual

  # install kernel modules
  apt-get install --no-install-recommends open-vm-dkms

  # EITHER: install tools for an xorg install
  apt-get install open-vm-tools
  # OR: a headless install
  apt-get install --no-install-recommends open-vm-tools

Start up a terminal window and do the following to ensure that you have the required packages for building VMware Tools or your kernel.


	sudo apt-get install build-essential linux-headers-`uname -r` psmisc

If the cdrom was not automatically mounted, mount the cdrom (in your guest OS) by doing


	# make a mount point if needed :
	sudo mkdir /media/cdrom

	# Mount the CD
	sudo mount /dev/cdrom /media/cdrom

	# Copy and extract VMWareTools

	sudo cp /media/cdrom/VMwareTools*.tar.gz /tmp

	# You can extract with archive manager, right click on the archive and extract ... or

	cd tmp

	tar xvf VMwareTools*.tar.gz

	cd vmware-tools-distrib
	sudo ./vmware-install.pl

	#You can configure the tools as root
	sudo vmware-toolbox

Once you are done installing you can simply issue sudo shutdown –P or sudo init 0 and shutdown the VM safely. Have fun Vm-innnng :)

LIMBO Game Review

2011-10-19T00:00:00z

LIMBO is a horrifically beautiful, monochrome or rather a gray-scale purgatory painted perhaps while watching Tim burton movie on the side. It was published initially in Jul 2010 by Microsoft Game Studios(XBLA) and Playdead(PSN, Windows), and then later released for Windows platform by Steam.

The primary character of the game is a boy, who wakes up right at the beginning of the game i.e while you are trying to figure out WTF is happening or wonder if you graphics driver (windows platform) screwed up or something, but there as you are fiddling with the movement buttons in keyboard you will see pair of bright white eyes flickering, and that’s when you start falling in love with the game and the care of detail added in a game which is totally unlike other in its league.

LIMBO game essentially is a 2D puzzle game, the boy with a twee outline and messy hair ideally is in search of her sister, the games puzzle are master minded by Dino Patti and lead designer of PlayDead Jeppe Carlsen,according to Wikipedia, Dino Patti had conceived this idea around 2004, and had even tried coding it in visual basic,he later made a video trailer to recruit a developer. They had grants from Nordic game program and Danish Government.

Although Patti helped in the first few months with programming, he realised that the project was much larger than what the two of them could handle, and Patti developed the business around the game's expanded development

Limbo was awarded both the "Technical Excellence" and "Excellence in Visual Art" titles at the Independent Games Festivalduring the 2010 Game Developers Conference, at the 2010 Electronic Entertainment Expo—about a month before its release—Limbo won GameSpot's "Best Downloadable Game",and was nominated for several other "Best of Show" awards, including "Best Platformer" by IGN, “Most Original Game" by G4 TVand "Best Puzzle Game" by GameSpot The game was nominated as one of 32 finalists at the 2010 IndieCadefestival for independent developers, ultimately winning the "Sound" award.

Limbo Game Playfrom varun panton Vimeo.

VPS Hosting Past and Present

2011-09-27T07:26:00z

I had been hosting this blog in the past for an extremely competitive (read cheap) price here at Arvixe . I was using ASP.net based blogging software called BlogEngine. over and all I will describe my experience as pleasant. Arvixe is a great Hosting Provider, has good up time and has plans which are easy on the pocket, and does almost all the common stuff like emails and cpanel ,blah blah that other hosting provider do, but kinda does it on a low price. I was a happy blogger for an year although I could only manage close to 8 post and and few demos in that time.

Enter Linode.

Linode is a different beast than Arvixe, this baby gives you way more control than any conventional hosting provider(shared) or a VPS provider generally gives you , and this is done by a custom made dash board panel which has a plethora of features, now Arvixe also has other plans which will end up giving you a dedicated hosting (VPS) and I bet they would be better than most others in the similar category, but Linode (a different league) is like Amazon Elastic,but only for Linux operating system like Ubuntu (debian), Fedora(centos) etc or rather dedicated for them, Linode means Linux Node anyways ;)

If I quote them verbatim:

We're a VPS hosting company built upon one simple premise: provide the best possible tools and services to those that know what they need — better hosting.

and this is pretty true , they have many features which can be found here. They also do a Getting Started video here, if you still have questions look here and here, they also have an Irc channel and other discussions and bulletin boards which can be reached through this page.

Linode has six data centers around the world , so one can switch between them either while setting up an account or even migrate like a hot swap later on, depending on what speed they would like to get for dedicated regional customers, they do a blog as well.

Linode does a lotta stuff and does is good, but the bang on the buck is that they have a kinda “do it yourself library”, which is pretty vast, quality and covers every thing you need to know, or at least the important bits. I have lately spend hours just browsing through it, this library will have “getting started” information and stuff which you need to do to get cracking!!.

By the way did I mention that since I had purchased a good(slightly expensive than earlier )plan from them I also decided to write my own blog,I have used python and SQLite so that it stays light weight and have used WSGI . I will soon be open sourcing it at git hub. I used Bottle.py, micro web-framework to build it so I will be calling the software Message, “Message in the bottle…..”, rings any silly bell anywhere ??

I wish my very best to Linode team and hope some of you linux ninjas would like to give it a try someday.

Web Sockets in HTML 5

2011-01-10T00:00:00z

HTML5 WebSockets, defines a full-duplex communication channel that operates through a single socket over the web. WebSocket is not just another incremental enhancement to conventional HTTP communications, it represents a large advance, especially for real-time, event driven web applications.

Normally when a browser visits a web page, an HTTP request is sent to the web server that hosts that page. The web server acknowledges this request and sends back the response. In many cases—for example, for stock prices, news reports, ticket sales, traffic patterns, medical device readings, and so on—the response could be stale by the time the browser renders the page.

Current attempts to provide real-time web applications largely revolve around polling and other server-side push technologies, the most notable of which is "Comet", which delays the completion of an HTTP response to deliver messages to the client.

There are many open source implementation of Http Push, one of my personal favorite is APE i.e. Ajax Push Engine.

Update: Microsoft has added a small demo of HTML5 support , it comes as an extension to WCF (Windows communication foundation) .The links can be found here, here and here.

There is still no news of Microsoft adding this feature in IE 9, at the time of writing this post, only Firefox 4, Chrome 4, Safari 5 and Opera 11 and above have support for this feature.

Update: Due to security flaw Firefox and Opera have disabled their support for Web Sockets, more info can be found here.

For those interested here is the draft , official website, the API and a nice tutorial.

To establish a WebSocket connection, the client and server upgrade from the HTTP protocol to the WebSocket protocol during their initial handshake, as shown in below, note that this connection description represents draft 76 of the protocol, the connection handshake may change during future specification revisions.

From client to server:
GET /demo HTTP/1.1
Host: example.com
Connection: Upgrade
Sec-WebSocket-Key2: 12998 5 Y3 1 .P00
Sec-WebSocket-Protocol: sample
Upgrade: WebSocket
Sec-WebSocket-Key1: 4@1 46546xW%0l 1 5
Origin: http://example.com
[8-byte security key]

From server to client:
HTTP/1.1 101 WebSocket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
WebSocket-Origin: http://example.com
WebSocket-Location: ws://example.com/demo
WebSocket-Protocol: sample
[16-byte hash response]

I was trying to find an alternative which is comfortable enough to be used in my pet projects in Microsoft Tech stack, at the time of writing I couldn’t any so I here it is my very own HTTP Web Socket server written in .Net via C#, this implementation does not use WCF and is basically written using HTTP Async Sockets, the client API is pretty straight forward

var url = "ws://localhost:999&rdquo"
var sock = new WebSocket(url);
sock.onopen = function () {
    log("open");
    sock.send("Message….");
}
sock.onmessage = function (e) {
    log(e.data);
}
sock.onclose = function (e) {
    log(" closed");
}

and that's it!!! I plan to add some more examples and perhaps come with something more concrete later, till then I hope you enjoy using this code sample and enhancing it as you desire.

A complete example in c# can be downloaded from here

Water Simulation

2010-09-25T00:00:00z

Its been a while since my last post, had been busy with office work…sigh!!!

Anyways, I was poking around web to come up with interesting water simulation techniques, as you could see in many games these days, so I did find a reference to an old C-based technique, after little refinement, what I have is a water simulation which uses a very old algorithm based on two heightmaps.

As always it’s a grid based animation which divides the screen into a mesh(more slices better this simulation looks), I use a jagged array in c# to do this (load it with a texture of 640 X 480 dimension ), to animate I use another array of same dimensions, now to make every thing move I kinda do something like this

               for (int y = 1; y < 479; y++) {
                    for (int x = 1; x < 639; x++) {
                        this.currentState[x, y] = ((this.oldState[(x - 1), y] + this.oldState[(x + 1), y] + this.oldState[x, (y + 1)] + this.oldState[x, (y - 1)] >> 1) - this.currentState[x, y]);
                        this.currentState[x, y] -= (this.currentState[x, y] >> 7);
                        int data = (short)(1024 - this.currentState[x, y]);
                        int a = (x - 640) * data / 1024 + 640;
                        int b = (y - 480) * data / 1024 + 480;
                        if (a >= 640) a = 639;
                        if (a < 0) a = 0;
                        if (b >= 480) b = 479;
                        if (b < 0) b = 0;
                        int textOffset = x + y * 640;
                        bmp.Pixels[textOffset] = texture[(a + b * 640)];
                    }
                }

The choice of display entity(acting as viewport and display canvas) is WritableBitmap, and rest is pretty much fun to play.

Click here to watch demo

Liquid Particles

2010-07-03T00:00:00z

After ages of constantly being one of the most used browser as well as a serious pain in Butt!!! IE is finally about to mature into something good and perhaps a useful software , with its support to most of the standards being used today , IE will join back the league of web 3.0 browsers, hopefully very soon

So I was going through various experiments done with Canvas and came across a great example on Liquid Particle physics simulation, which amazingly is written in JavaScript and Canvas element, I was so mesmerized by the application that I had to port it to SilverLight, using WritableBitmap.

Click Here to watch Demo.

Image Carousel

2010-06-29T00:00:00z

Here is another of many JavaScript based Image Carousel, I had seen many similar to these made in SilverLight and Flash, I came across a wonderful carousel made in SilverLight sometimes back which was very fluid in behavior and could easily be customized and configured to take various shape and form.

I have managed to port it to JavaScript, it works best on chrome or FF, hope this would be fun to play with.

Enjoy!!!

Click here to watch it in action

I-Phone Scroll

2010-06-18T00:00:00z

One of the amazing features of i-phone user interface is smooth panning/scrolling animation.

I-phone “List Scrolling” perhaps gives a very funky and usable List –View to its users and allows them to scroll up and down the interface without any need of a scroll bar.

This interface is sensitive to the push speed and also shows traces of resilience.

I have been experimenting with various techniques to bring out similar effect using JavaScript and HTML, I am not quite sure if this would be useful in its present form but with some modifications may be this sample could prove its worthiness somewhere.

The code involved is also very little.

    var scroll = {
    down: false,
    verticalOffset: -5,
    ym: 0,
    decay: .93,
    mdDecay: 0.27,
    speedOffset: 0.4,
    sprinessOffset: 0.08,
    _velocity: 0,
    _mouseDownY: 0,
    $: function (id) {
        return document.getElementById(id);
    },
    /* --- events --- */
    addEvent: function (o, e, f) {
        if (window.addEventListener) o.addEventListener(e, f, false);
        else if (window.attachEvent) r = o.attachEvent('on' + e, f);
    },
    init: function () {
        setInterval(scroll.compositiontarget_rendering, 30);

        scroll.verticalOffset = scroll.$('scrollMe').offsetTop;

        scroll.addEvent(scroll.$('scrollMe'), 'mousedown', function (e) {
            scroll.$('scrollMe').style.cursor = 'move';
            if (!scroll.down) {
                scroll.down = true;
                scroll.ym = e.clientY;
                scroll._mouseDownY = scroll.$('scrollMe').offsetTop;
            }

            if (e.preventDefault) e.preventDefault();
        });

        scroll.addEvent(scroll.$('scrollMe'), 'mouseup', function () {
            scroll.down = false;
            scroll.$('scrollMe').style.cursor = 'default';

        });


        scroll.addEvent(scroll.$('scrollMe'), 'mousemove', function (e) {
            if (scroll.down) {
                if (scroll.canScrollFurther()) scroll.$('scrollMe').style.top = (scroll._mouseDownY + (e.clientY - scroll.ym)) + 'px';
                // update the velocity
                scroll._velocity += ((e.clientY - scroll.ym) * scroll.speedOffset);
            }

        });

        document.onselectstart = function () {
            return false;
        }
    },
    compositiontarget_rendering: function () {
        // decay the velocity
        if (scroll.down) {
            scroll._velocity *= scroll.mdDecay;
        } else {
            scroll._velocity *= scroll.decay;

            var screenHeight = scroll.$('screen').offsetHeight;
            var scrolRegionHeight = scroll.$('scrollMe').offsetHeight;

            var y = scroll.$('scrollMe').offsetTop - scroll.verticalOffset;
            var spriness = 0;

            // calculate a spriness when the text moves over the canvas size
            if (y > 0) {
                spriness = -y * scroll.sprinessOffset;
            } else if (y + scrolRegionHeight < screenHeight) {
                spriness = ((screenHeight - scrolRegionHeight - y) * scroll.sprinessOffset);
            }
            if (!scroll.canScrollFurther()) {
                scroll._velocity = -(scroll._velocity / 8)
            }
            scroll.$('scrollMe').style.top = (y + scroll._velocity + spriness) + 'px';
        }


    },
    canScrollFurther: function () {
        var _top = scroll.$('scrollMe').offsetTop;
        var bottom = _top + scroll.$('scrollMe').offsetHeight;
        var center = (scroll.$('screen').offsetTop + scroll.$('screen').offsetHeight) / 2;
        return (_top < center && bottom > center);
    }
};

Click here to watch demo

Data URI and IE

2010-06-15T00:00:00z

Few days ago I was experimenting with a nifty technique of embedding images in web pages by base64 encoding them first and then using a standard known as Data URI Scheme, which basically defines a method of assigning a ‘src’ of an image tag as a base64 serialized string, like this:


background - image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABkAAAAUBAMAAACKWYuOAAAAMFBMVEX///92dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnYvD4PNAAAAD3RSTlMAACTkfhvbh3iEewTtxBIFliR3AAAAUklEQVQY02NgIBMwijgKCgrAef5fkHnz/y9E4kn+/4XEE6z/34jEE///A4knev7zAwQv7L8RQk40/7MiggeUQpjJff+zIpINykbIbhFSROIRDQAWUhW2oXLWAQAAAABJRU5ErkJggg==");

Although this technique could save some server round trips, however Microsoft ver. (IE < 8). does not support this scheme even though its a standard, which then makes this technique unfavorable to use as its not cross browser.

I was trying to find a work around and came across MHTML hackwhich is not an elegant approach especially for dynamic solutions and is also not supported by Microsoft any more.

I found an interesting articleby Dean Edwards, which suggested in IE <8 scenarios a request could be send back with base64 encoded data as a query string to server, where a server module would then de-serialize the image and write it back in response:

Client Side:

function urlEncode(str) {
    str = escape(str);
    str = str.replace(new RegExp('\\+', 'g'), '%2B');

    return str.replace(new RegExp('%20', 'g'), '+');
}

function assign() {
    document.getElementById("img1").src = "Handler.ashx?q=" + urlEncode("iVBORw0KGgoAAAANSUhEUgAAABkAAAAUBAMAAACKWYuOAAAAMFBMVEX///92dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2dnYvD4PNAAAAD3RSTlMAACTkfhvbh3iEewTtxBIFliR3AAAAUklEQVQY02NgIBMwijgKCgrAef5fkHnz/y9E4kn+/4XEE6z/34jEE///A4knev7zAwQv7L8RQk40/7MiggeUQpjJff+zIpINykbIbhFSROIRDQAWUhW2oXLWAQAAAABJRU5ErkJggg==");
}

and on the server side one can use a generic http Handler like this

public class Handler: IHttpHandler { public void ProcessRequest(HttpContext
    context) { context.Response.ContentType = "image/gif"; try { string data
    = context.Request.QueryString["q"]; byte[] todecode_byte = Convert.FromBase64String(data);
    context.Response.BinaryWrite(todecode_byte); } catch (Exception e) { //Log
    } } }

The only thing to note here is to URLEncode the string before sending it to avoid FormatException .

This is not by any means an elegant solution and will work only for small images, as bigger images will serialize into large string chunks and browsers limits urls to 2083 characters, but I just thought writing a c# version of this approach would be interesting. :)

Notepad++ with Python

2010-06-15T00:00:00z

After reading an excellent article by Kazi Manzur Rashidon setting up a development environment for Iron Ruby using Notepad++, I was immediately struck with an idea of using the same excellent tool with Python 2.6.

Now don’t get me wrong here, theoretically there is nothing wrong with IDLE, but having a light weight IDE for those who don’t want to use Pydevplugin for Aptanaor Eclipse, I think Notepad++ is indeed a nice little dev tool.

One of the amazing capabilities in Notepad++ is that one can assign shortcuts like(Cntrl + F5) and debug/interpret python script with python interpreter.

In order to Run python script from Run menu in Notepad++ you must Open Run > Run (or simply press F5), in the window that opens


           cmd /K "$(FULL_CURRENT_PATH)"

this command assumes that Python path i.e.


           c:/PythonXX/Python.exe


          c:/PythonXX/Pythonw.exe

(If you don’t like console) is added in your Environment variables , if not you couldeither go to:


          My Computer -> Properties -> Advanced -> Environment Variables -> System Variables -> PATH –> Edit

and add this path there or use


         cmd /k C:\Python26\python.exe

then press save after putting in a friendly name and assigning a key board shortcut for example CNTRL + F5.

You could now see your entry in Run drop down menu at the last, incase you want to edit or remove this go to Settings>Shortcut mapper and press Run Commands tab and then right click to choose options from context menu and your all set.

Hope this helps :)

Hello World!

2010-06-12T18:00:00z

After long period of wait, here it is my very first and personal Blogging Space.

I have been hoping to go online for about an year now, but then there always were events happening around to prevent this immense desire of mine.

I intend to blog about .net in general , JavaScript, Silverlight, Animation and maybe some other cool stuff .

Hope to update this space again.

Cheers!!!

-- Varun