Hotline for IT engineers/model developers

Handout Series - Big Data - Spark - Performance Tuning

2022-03-14T00:25:00.007-04:00

Background:
In this handout, I tried to cover some key aspects which we need to keep in mind while trying to tune a spark application. This covers tools I frequently used to get more detail info to do better debugging.

Tools and techniques to debug performance issues:
Tools/Techniques
Things to focus on
Spark UI
Stage and task completion time
25 percentile, 50 percentile, 75 percentile of execution time
Memory usage by executor
In and out data for tasks
DAG – helps to find out job/stage optimization
Garbage collection time
Spark Log
Partition size – evenly distributed or not
Yarn Manager (if yarn is used as resource manager), UI or yarn command
Number of executors
Job/task execution time for each executor
Memory usage for each executor
Data Storage (HDFS, object storage (S3), etc)
File size - helps to identify skew in data
DataFrame API
Number of partitions
Number of records per partitions
SparkLens
Tool that analyzes the application and suggests possible configuration parameters with values to get optimal performance that could be head start before further tuning

Primary areas in Spark to tune performance:
Areas
Note
Memory
Amount of memory for per executor might increase garbage collection
Spark parameters
spark.driver.memory
spark.executor.memory
Processing
Increase in cores gives parallelism but might impact scheduling and additional shuffle
Spark parameters
spark.driver.cores
spark.executor.cores
spark.cores.max
spark.executor.instances
Garbage Collection
Type of GC to use (ParallelGC, G1GC etc.)
Configure each type of GC to get better performance
Specify high level GC parameters like parallel threads, concurrent threads, heap occupancy percentage etc.

Note:
We need to be cognizant about all the areas when tuning performance as change in one area has direct impact to others.
It has to be balancing act, need to fine the combination of parameters across all areas which fit better for the problem in hand

Performance Improvement technique- resolve skewness in data
What is skew

Most common problem, mostly happens during initial phase
Extreme imbalance of work in the cluster
Tasks within a stage take uneven amounts of time to finish
How to identify

Stage/task wait time and execution time – spark UI
Memory usage – Spark UI, yarn
Check partition size – spark UI, yarn
Hearbeat is missing by executor – spark log
Partition size and records processed per partition – DataFrame API
How it could happen

Data ingestion has skew in it where data is not partitioned properly. This will be propagated to other parts of the application
How to handle skew data

Use a partition column in the source data (if possible) and use it to read – gives best performance as partitions don’t need to be created in processing time. If partition column cannot be added in source, create the partition column to dataset immediately after reading data
Make sure the partition column has even distribution of data
Partition column has to be numeric. Can be a composite column
Spark parameters:
partitionColumn
lower/upperBound – used to determine stride
numPartitions – max # of partitions
Make sure stride is not skew (specially for JBDC load)
Use hash + Mod function to have evenly distributed stride if some strides have uneven data volume
Partition data at dataframe level – great for ‘narrow transformation’
To increase, use partition function
To reduce, use coalesce for less data shuffling

Performance Improvement technique – use cache/persist
What is cache/persist

Persist – saves data in storage
Cache – in memory persist
Reuses DataFrame with transformations rather than building it from scratch
Spark uses columnar format (allows scanning on required columns), compression to minimize memory usage and less pressure on GC
How to use cache/persist

Select appropriate storage type as each type has it’s own performance implication
Un-persist when done and don’t over persist
Has impact on memory performance and possible slowdown
Puts pressure on GC
Without cache, DataFrame will be built from scratch if the same one is used in processing, saves read time, serialization time etc.
Spark properties:
spark.sql.inMemoryColumnarStorage.compressed
spark.sql.inMemoryColumnarStorage.batchSize

Performance Improvement technique – use seq.par.foreach
Allows items in a sequence (seq in scala) to be processed in parallel, hence improving on processing time. Though we need to be careful that logic doesn’t yield non-deterministic results or else will create a race condition.
Example: if we have a foreach loop to process (read and/or transformation) a file on a seq where file names are stored, replacing with seq.par.foreach will allow parallel processing of files and give performance benefit

Performance Improvement technique – use proper joining strategies
Use broadcast join

Broadcast puts a DataFrame to every executor before joining. Idea is to broadcast smaller dataset to all executors and use local copy of DataFrame to join with bigger dataset
Spark 3.x has auto-broadcast join threshold to 10M, datasets smaller than that will be automatically broadcasted if not done explicitly in the code
In broadcasting, spark uses broadcast hash join instead of sort merge join which triggers data shuffle and exchanges
Code snippet: largeDF.join(broadcast(smallDF), largeDF.col(“col_name”) === smallDF.col(“col_name”))
If DataFrame is big: apply filters to keep the dataset as small possible
Dynamic partition pruning

Out of the box functionality in spark 3.x
Eliminates partition at the read time
Introduce salting to reduce skew

Performance Improvement technique – use right serializer
Types of serializers

Java serializer
Kryo serializer
Default for most types
Can work on any class
More flexible
Gives slower performance
Default for shuffling RDDs and simple types (integer, string)
Significantly faster (4x) and more compact (10x)

Serialization strategies

Use kryo for serialization boosting
Register classes to use as serializer in kryo – doesn’t save class name as part of object, saves 6 bytes for each character in class name

Few simple handy tricks
Keep larger dataset at left in the joining – spark implicitly try to shuffle right dataset first
Apply appropriate partitioning –
very less partitions = less parallelism
too many partitions = scheduling issues, more data shuffling
Try not to use UDFs
UDF execution flow - Deserializes every row to an object > apply lamda function > reserializes the row
Above flow puts pressure on garbage collection by generating lot of garbage
Avoid using UDFs specially if not using scala, as everything needs to be translated to JVM code
Apply all the filters on dataset as early as possible in the processing

Few advance configuration
JVM parameters for memory -
Reduce long Garbage collection time
Use ‘Spark UI’ to check time spent by task vs garbage collection
First step of GC tuning is enable GC logging
Analyze log to look for
How frequent GC is occurring
How much memory is cleaned up
Algorithm specific (G1GC, ParallelGC) stage information like minor GC, full GC, major GC
ParallelGC (spark’s default)
G1GC
Heap space is divided into young and old generation
Breaks heap into thousands of region

If minorGC occurs frequently: increase Eden and Survivor space
Recommended if heap size are large (>8GB) GC times are long
If majorGC occurs frequently: increase young and old space, decrease spark.memory.fraction
Lower latency traded with higher cpu usage for GC bookkeeping
If fullGC occurs before task finishes: increase memory, try cleaning young space by GC triggers

For large scale applications, setting below parameters help a lot
-XX:ParallelGCThread=n
-XX:ConcGCThreads=[n, 2n]
-XX:InitialHeapOccupancyPercent=35

Unix - Shell Script - Connect to SFTP from script

2020-10-07T18:50:00.002-04:00

Description:

This post cover various ways to connect to SFTP from shell script. Passing the password in the command prompt is easier and secure as user is explicitly typing it down when prompted. But in shell script, password needs to be passed automatically and securely.

Approach-01: using sshpass command

Resource-01 covers detail about the command and how to install in various version on unix.

SFTP_USER=<user>
SSHPASS=<password>

TARGET_SERVER=<host>
sshpass -e sftp -oBatchMode=no -b - $SFTP_USER@$TARGET_SERVER <<EOF
    ls -lrt
    cd <target path>
    put <source path/file name>
    bye
EOF

Approach-02: using lftp command

SFTP_USER=<user>

LFTP_PASSWORD=<PASSWORD>

TARGET_SERVER=<host>

lftp --env-password sftp://$SFTP_USER@$TARGET_SERVER <<EOF
    ls -lrt
    cd <target path>
    put <source path/file name>
    bye
EOF

Resources:

Detail about 'sshpass' command. How to install the command, various ways to pass password to connect to SFTP etc.

https://www.cyberciti.biz/faq/noninteractive-shell-script-ssh-password-provider/

Selenium - Learning

2020-08-25T14:35:00.002-04:00

Description:
Intention of this post is to track high level concepts, components needed to understand Selenium-2 or Web driver. This can be considered as hand out for quick reference. Examples or demo of using it in real code will be covered in subsequent posts.

What is Selenium-2 or Web Driver:

Development stacks:

Java - 1.8
Eclipse IDE - neon
Windows - 2012 R2

How to setup:

Download Selenium web driver. I used Selenium 3.8.1 for Java. This will download a zip file which has 'client-combined-3.8.1-sources.jar' and 'client-combined-3.8.1.jar' jars at root level. Copy these two jars along with all the jars from lib folder. Check Link-1 below.
Download the browser driver intent to use for testing. In my case, I have downloaded web driver for IE, chrome and firefox as my application has to support these browser and phantomJS to run testing on headless browser to support deployment/testing automation. Check Link-2 below.

High level overview:

WebDriver commands

Fetching a page

Different ways

driver.get("http://www.google.com");
driver.navigate().to("http://www.google.com");

Activity on browser

Closes the current browser associated with the driver. driver.close();
Quit the driver and close all associated windows. driver.quit();
Refresh the current page. driver.refresh();

Locate UI element named as WebElement. 'Find Element' and Find Elements' method available in webdrvier or WebElement which takes locator or query object called 'By'

Different 'By' strategies are

By ID
By Name
By Class
By tag name
By name
By link text
By xpath
Using javascript
and more

User interactions - check Link-5 for examples

Textbox
RadioButton
CheckBox
Drop down item selection
Keyboard action
Drag and drop
Mouse action
Multi selection
Find all links

Moving between windows and frames

Examples

driver.switchTo().window("windowName");
driver.switchTo().frame("frameName");

Popup dialogs

Examples

Alert alert = driver.switchTo().alert();

Navigation: History and Location

Examples

driver.navigate().to("http://www.google.com");
driver.navigate().forward();
driver.navigate().back();

Manage cookies. Add/delete/list all cookies

Synchronization

Old fashioned java way using 'Thread.sleep(xxx);' This will halt the execution for specified milliseconds.

Implicit wait - the web driver will wait for specified time and will not try to find the element before the time is elapsed. After the time is crossed, web driver will try to find the element, execution will continue on success, throws exception on failure. This wait works at global level for all web element search.

Example:

WebDriver driver = new FirefoxDriver();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);

Fluent wait - defines maximum amount of time to wait before a condition take place as well as the frequency to check the condition

Example:

Wait wait = new FluentWait(driver).withTimeout(60, SECONDS).pollingEvery(10, SECONDS).ignoring(NoSuchElementException.class);
WebElement dynamicelement = wait.until(new Function <webdriver,webElement>() {
public WebElement apply(WebDriver driver) { return driver.findElement(By.id("dynamicelement")); } });

Explicit wait - set a certain condition to occur before move to next step. The condition will be periodically checked (by default every 500ms) until the specified timeout is reached which is defined during explicit wait creation. Common use is to perform an action based on element is visible. WebDriverWait extends FluentWait.

Example:

WebDriver driver = new FirefoxDriver(); driver.get("http://somedomain/url_that_delays_loading"); WebElement myDynamicElement = (new WebDriverWait(driver, 10)) .until(ExpectedConditions.presenceOfElementLocated(By.id("myDynamicElement")));

Links:

Selenium download page - http://www.seleniumhq.org/download/
Download web driver - check 'Third Party Drivers, Bindings, and Plugins' section of Link-1
http://www.seleniumhq.org/docs/01_introducing_selenium.jsp
https://www.tutorialspoint.com/selenium/
https://www.tutorialspoint.com/selenium/selenium_user_interactions.htm

Unix - Shell Script - Find files and execute a command on matches

2020-08-25T14:25:00.021-04:00

Description:

Often time we need to perform additional tasks on file/folder found in a certain path. Instead of fetching the list and looping through the items, it's easier to combine the search and additional task in to one command, more like chaining commands. "find" unix command allows executing additional command to be executed on matched items. This article provides few examples to demonstrate how to do so with explanation.

Example-01: Delete all the log files older than 14 days

find . -maxdepth 1 -type f -mtime +14 -exec rm -fr {} \;

Explanation:

Scan current folder path for only files (find .)
Skip scanning sub folder(s) (-maxdepth 1)
Consider only files, exclude directory (-type f)
Match files which are older than 14 days (-mtime +14)
Perform delete on matched items (-exec). -exec allows to execute a command on items
Forcefully delete any match found without any confirmation message (rm -fr)
Use {} (curly braces) as a placeholder for each file that the find command locates
Need to add command terminator (\;). This is a must if -exec attribute is used in find command

Example-02: Copy all the matching files

find . -maxdepth 1 -type f -exec cp {} ~/ \;

Explanation:

Copy all matched items in home directory (-exec cp {} ~/).
Other parts are explained in example-01

Example-03: Change permission of matched files

find . -maxdepth 1 -type f -exec chmod 744 {} \;

Explanation:

Change the file permission to 744 (-exec chmod 744).
Other parts are explained in example-01

Example-04: Execute multiple commands. Change permission and move all files to a folder

find . -maxdepth 1 -type f -exec chmod 744 {} \; -exec mv {} ~/test/ \;

Explanation:

Chains multiple commands using (-exec) multiple times. Each command must be ended with (\;)
First command changes the permission and second command moves the files to test folder under home path.
Other parts are explained in example-01

References:

Detail about 'find' command - https://kb.iu.edu/d/admm

Batch scripting - SQL Server - Execute all sql files from a folder

2018-02-09T15:30:00.001-05:00

Description:
In my project code base, I have a structure for all database objects (stored procedure, functions, trigger etc.) as well as some upgrade scripts. Need a nice way to deploy all the database scripts rather manually execute one by one.

Environment information:
1. Windows Batch script
2. SQL Server 2014

Script:

@ECHO OFF
REM Database connection info
SET databaseName=db_name
SET databaseServer=db_server
SET databaseUserName=db_user
SET databasePassword=db_password

REM Location of scripts. Multiple folders can be specified by giving space in between and enclosed with quotation.
REM This script will go through only one level of sub folders and execute all .sql files
SET scriptLocation="location1" "location2"



REM -----------------------------------------------------------
REM - Logic 
REM -----------------------------------------------------------

SET initialLocation=%cd%

FOR %%e IN (%scriptLocation%) DO (
 @ECHO ------------------------
 @ECHO %%e
 @ECHO ------------------------
 IF EXIST %%e (
  CALL :executeScripts %%e
 
  FOR /D %%i IN (%%e\*) DO (
   @ECHO ------------------------
   @ECHO %%i
   @ECHO ------------------------
   CALL :executeScripts "%%i"
   
  )
 ) 
)

CD %initialLocation%
PAUSE


:: This function execute all the scripts from given path
:executeScripts
FOR %%G IN (%1\*.sql) DO (
 @ECHO %%~nxG  
 sqlcmd /S %databaseServer%  /d %databaseName%  -U %databaseUserName%  -P %databasePassword% -i"%%G"
)
EXIT /B 0

Java - JDBC - use parameterized query with IN clause having dynamic list

2017-12-09T21:38:00.002-05:00

Background: I needed to execute query with IN clause where the list is dynamic in nature. The database is sql server 2012 and the driver doesn't support java.sql.Statement.setArray(1, java.sql.Array) yet. Had to come up with the below trick to make it work. In the example, tried it on both SELECT and DELETE. Code is written in Java and tried on SQL server 2012.

Script to create table and populate data:

CREATE TABLE employee (id INT, name VARCHAR(1000));

INSERT INTO employee (id, name) VALUES(1, 'First');
INSERT INTO employee (id, name) VALUES(2, 'Second');
INSERT INTO employee (id, name) VALUES(3, 'Third');
INSERT INTO employee (id, name) VALUES(4, 'Fourth');

Java Class:

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;

public class TestParameterizedIN {

 // JDBC driver name and database URL
   static final String JDBC_DRIVER = "com.microsoft.sqlserver.jdbc.SQLServerDriver";
   static final String DB_URL = "jdbc:sqlserver://localhost:1433;databaseName=testingGround";

   // Database credentials
   static final String USER = "***";
   static final String PASS = "***";

   public static void main(String[] args) {
     Connection conn = null;
     PreparedStatement stmt = null;
     try {
       // Register JDBC driver
       Class.forName(JDBC_DRIVER);

       // Open a connection
       System.out.println("Connecting to database...");
       conn = DriverManager.getConnection(DB_URL, USER, PASS);

       //Fetch records
       System.out.println("Fetching data...");
       int[] ids = new int[]{1,2,3};//IDs of records to select data
       String sql = "";
       for(int each=0; each<ids.length; each++){
        if(each == 0)
         sql = "SELECT id, name FROM employee WHERE ID IN (?";
        else
         sql += " ,?";
       }
       sql += ")";
       
       System.out.println("--------------------");
       stmt = conn.prepareStatement(sql);
       for(int each=0; each<ids.length; each++){
        stmt.setInt(each+1, ids[each]);
       }
       try(ResultSet rs = stmt.executeQuery()){
      // Extract data from result set
        while (rs.next()) {
          // Retrieve by column name
          int id = rs.getInt("id");
          String name = rs.getString("name");

          System.out.print("ID: " + id);
          System.out.print(", Name: " + name);
          System.out.println("");
        }
       }
       System.out.println("--------------------");
       stmt.close();
       
       //Delete records
       System.out.println("Delete records...");
       sql = "";
       for(int each=0; each<ids.length; each++){
        if(each == 0)
         sql = "DELETE FROM employee WHERE ID IN (?";
        else
         sql += " ,?";
       }
       sql += ")";
       
       stmt = conn.prepareStatement(sql);
       for(int each=0; each<ids.length; each++){
        stmt.setInt(each+1, ids[each]);
       }
       System.out.println(stmt.executeUpdate()+" records are deleted");
       
     } catch (SQLException se) {
       se.printStackTrace();
     } catch (Exception e) {
       e.printStackTrace();
     } finally {
       // finally block used to close resources
       try {
         if (stmt != null)
           stmt.close();
       } catch (SQLException se2) {
       }
       try {
         if (conn != null)
           conn.close();
       } catch (SQLException se) {
         se.printStackTrace();
       }
     }
   }
}

Output of the class execution:

Connecting to database...
Fetching data...
--------------------
ID: 1, Name: First
ID: 2, Name: Second
ID: 3, Name: Third
--------------------
Delete records...
3 records are deleted

Note:

Used mssql-jdbc-6.2.2.jre8.jar for SQL Server driver.
Will work on another blog to try out java.sql.Statement.setArray(1, java.sql.Array), may be on HSQLDB.

MS SQL Server - Query - Concatenate data from multiple records in a GROUP BY query

2017-11-20T13:43:00.000-05:00

Problem: Need to find out the employees reporting to a manager and create a comma separated list of names. Data will look like table-01.

Expected output would look like:
Table-01:

Solution-01: using FOR XML PATH

-- Create table
CREATE TABLE
#employeeManagerAssociation (managerID INT, employeeID INT, firstName VARCHAR(100), lastName VARCHAR(100))
;

-- Insert data
INSERT INTO #employeeManagerAssociation (managerID,employeeID,firstName,lastName) VALUES (1, 101,'E1_First', 'E1_Last');
INSERT INTO #employeeManagerAssociation (managerID,employeeID,firstName,lastName) VALUES (1,102,'E2_First', 'E2_Last');
INSERT INTO #employeeManagerAssociation (managerID,employeeID,firstName,lastName) VALUES (2,103,'E3_First', 'E3_Last');
INSERT INTO #employeeManagerAssociation (managerID,employeeID,firstName,lastName) VALUES (2,104,'E4_First', 'E4_Last');
INSERT INTO #employeeManagerAssociation (managerID,employeeID,firstName,lastName) VALUES (3,105,'E5_First', 'E5_Last');

SELECT
managerID,
STUFF((
SELECT ', ' + firstName + ' ' + lastName
FROM #employeeManagerAssociation
WHERE managerID = main.managerID
FOR XML PATH('')
)--,TYPE).value('(./text())[1]','VARCHAR(MAX)')
,1,2,'') AS NameValues
FROM #employeeManagerAssociation main
GROUP BY managerID

-- Delete temporary table

DROP TABLE #employeeManagerAssociation

Eclipse - Setup - How to set java for during eclipse start up

2017-04-07T12:46:00.001-04:00

Description:
I faced starting up Eclipse IDE which complaining about old java version. The java version in my environment was 1.6 which is needed to run old programs. I installed Java 8 did below thing to make it work.

Environment:
Eclipse version - neno.3
Old java version - 1.6.0_29
New Java version - 1.8.u111

Steps to follow:

Find and write down the Java 8 installation location (for me, it was in e:\java\)
Go to location where eclipse is installed
Open file 'eclipse.ini' with any file editor
Type the below lines - (change the java path)
-vm
e:\Java\jdk1.8_u111\bin\javaw.exe
Save the change
Execute the eclipse.exe

Java - Coding - Java 8 new features - Currying with Java 8

2016-12-09T16:01:00.000-05:00

Description:
This was my testing sample to understand how currying can be done in Java 8. I used both lambda and function reference to demonstrate currying behavior which would help great deal to understand behind the scene caricatures.

Code:

import java.util.function.*;
import static java.lang.System.out;

public class CurryApplication {
 public static void main(String[] args) {
  IntBinaryOperator simpleAddWithLambda = (a, b) -> a + b;
  out.println("Simple Add (2params)= " + simpleAddWithLambda.applyAsInt(10, 20));

  IntFunction<IntBinaryOperator> simpleMultipleWithFunctionalReference = new IntFunction<IntBinaryOperator>() {
   @Override
   public IntBinaryOperator apply(int value) {
    return new IntBinaryOperator() {
     @Override
     public int applyAsInt(int left, int right) {
      return value * left * right;
     }
    };
    
   }
  };
  out.println("Simple Multiple (3 params)= " + simpleMultipleWithFunctionalReference.apply(10).applyAsInt(20, 30));

  IntFunction<IntUnaryOperator> curriedAdd = a -> b -> a + b;
  out.println("Curried Add (2 params)= " + curriedAdd.apply(4).applyAsInt(5));
  
  IntFunction<IntFunction<IntUnaryOperator>> curriedMultiple = a -> b -> c -> c*a*b;
  out.println("Curried multiple (3 params)= "+curriedMultiple.apply(3).apply(4).applyAsInt(5));
 }
}

Explanation:
Functional reference implementation has to have (n-1) level function interface reference in the type declaration - IntFunction>. Here IntFunction is Java 8's feature where provided type is the type of the result of the function. Curried invocation of multiple's implementation has combination of apply and appyAsInt is used in curried fashion.

Resources:
- http://stackoverflow.com/questions/6134278/does-java-support-currying
- https://dzone.com/articles/whats-wrong-java-8-currying-vs

Raspberry Pi - OpenCV - Setup

2016-11-14T15:21:00.000-05:00

Description:

This post is to cover OpenCV installation in raspberry pi and list down the steps in concise fashion and get things up and running so that anyone can focus on the project has in mind. Majority of the contents can from the links and I found those are workable and tested at my end. Check the Notes sections for any problem faced or explanation on any particular step.

Pre-requisites and version of components:
- Raspberry Pi 3
- Operating System raspbian Jessie
- OpenCV 3.1.0
- Python 3 (3.4.2)

Instruction:

Steps	Description
Install dependencies
sudo apt-get install build-essential cmake pkg-config	CMake to configure OpenCV build process
sudo apt-get install libjpeg-dev libtiff5-dev libjasper-dev libpng12-dev	image I/O packages to load image files
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev	video I/O packages to load video from files and streams
sudo apt-get install libxvidcore-dev libx264-dev
sudo apt-get install libgtk2.0-dev	Sub module highgui to display image on screen
sudo apt-get install libatlas-base-dev gfortran	library to optimize OpenCV
sudo apt-get install python3-dev	Python 3 header file to compile OpenCV
Download OpenCV
> cd ~ > wget -O opencv.zip https://github.com/Itseez/opencv/archive/3.1.0.zip > unzip opencv.zip	OpenCV source code
> wget -O opencv_contrib.zip https://github.com/Itseez/opencv_contrib/archive/3.1.0.zip > unzip opencv_contrib.zip	OpenCV contribute source code
Setup Python3
> wget https://bootstrap.pypa.io/get-pip.py > sudo python get-pip.py	install python package manager (pip)
sudo pip install virtualenv virtualenvwrapper	install virtualenv and virtualwrapper to keep the python dependencies separate
sudo rm -rf ~/.cache/pip	clear pip cache
WORKON_HOME=$HOME/.virtualenvs source /usr/local/bin/virtualenvwrapper.sh	write these two lines in ~/.profile
source ~/.profile	reload profile changes
mkvirtualenv cv -p python3	make python virtual environment
> source ~/.profile > workon cv	all works will be done in cv python virtual environment
pip install numpy	Python package for numeric processing
Install OpenCV
> cd ~/opencv-3.1.0/ > mkdir build > cd build > cmake -D CMAKE_BUILD_TYPE=RELEASE \ -D CMAKE_INSTALL_PREFIX=/usr/local \ -D INSTALL_PYTHON_EXAMPLES=ON \ -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib 3.1.0/modules \ -D BUILD_EXAMPLES=ON ..	Cell 3:2
make -j4	compile OpenCV on four cores
> sudo make install > sudo ldconfig	install OpenCV
> cd /usr/local/lib/python3.4/site-packages/ > sudo mv cv2.cpython-34m.so cv2.so	rename cv file to cv2
> cd ~/.virtualenvs/cv/lib/python3.4/site-packages/ > ln -s /usr/local/lib/python3.4/site-packages/cv2.so cv2.so	create soft link to cv2.so
Test OpenCV installation
> source ~/.profile > workon cv > python >>> import cv2 >>> cv2.__version__	test OpenCV version (3.1.0)
Clean Up
> rm -rf opencv-3.1.0 opencv_contrib-3.1.0	test OpenCV version (3.1.0)

Notes:
1. If NOOBS is used, the SD card is already expanded. In case SD card needs to be expanded, follow below steps -
- Type command 'sudo raspi-config' in terminal
- Select option '1 - Expand Filesystem'
- Restart the system using 'sudo reboot' in terminal

2. OpenCV and OpenCV contribute version has to be same

Resources:
1. http://www.linuxcircle.com/2015/05/18/open-computer-vision-opencv3-with-python-3-on-raspberry-pi-2/
2. http://www.pyimagesearch.com/2016/04/18/install-guide-raspberry-pi-3-raspbian-jessie-opencv-3/
3.

Java - Coding - Java 8 new features - Aggregate Operation

2015-12-24T18:38:00.001-05:00

Concept:

Stream - A stream is a sequence of elements. Unlike a collection, it is not a data structure that stores elements. Instead, a stream carries values from a source through a pipeline
Intermediate operation - an intermediate operation produce a new stream
Terminal operation - produces a non-stream result. Can be a collection, primitive value or no value at all
Pipeline - a sequence of aggregate operations.

Components in a pipeline

A source or stream
Zero or more intermediate operation
A terminal operation

Reduction function - the terminal operations returning one value combining the content of a stream.

Raw reduction functions are provided by JDK to have custom implementation. Example-4 and 5 elaborates that

Parallelism in Stream - stream execution can be done in parallel. the Java runtime partitions the stream into multiple substreams. Aggregate operations iterate over and process these substreams in parallel and then combine the results. Method to use '.parallelStream(...)'.

Example-1: basic operations on a stream.

int[] values = {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20};
System.out.println("Any match of element value 3. "+ (Arrays.stream(values).anyMatch(each -> each==3) ? "FOUND": "NOT FOUND"));
System.out.println("All element between 1 and 15. "+ Arrays.stream(values).allMatch(each -> each > 0 && each <15));
//Show how many elements, average and sum of elements
System.out.print(Arrays.stream(values).count()+" elements, where SUM= "+Arrays.stream(values).sum()+" AVERAGE="+Arrays.stream(values).average().getAsDouble());

Full source code below.

Explanation - Arrays.stream(...) converts any array to a stream than any pipeline operation can be used. anyMatch, allMatch methods are passed lambda expression. count, sum, average, max, min methods don't take any input parameter or expression and are terminal operations. Terminal operations return OptionalInt/OptionalDouble/etc. if returns a value. max method returns OptionalInt and getAsInt method is used to fetch the value. average method returns OptionalDouble and getAsDouble is used to fetch the value.

Example-2: filter out value elements and show the values.

1
2
3

//Filter out odd value elements and print the even values.
System.out.print("Even value elements are= ");
Arrays.stream(values).filter(each -> each %2 == 0).forEach(each -> System.out.print(each+", "));

Full source code below

Explanation - first operation in the pipeline is filter which takes a lambda expression. When the lambda expression is executed, it returns true if element value is a even number and that even value is passed to the next operation which is forEach. forEach operation is passed with another lambda expression which executes a regular statement, in this case it prints the value from stream to console. Filter method returns a stream as input for forEach and the stream type is IntStream.

Example-3: use aggregate operation on collection and map to a value.

Source code

import java.util.function.DoubleConsumer;
import java.util.function.IntConsumer;
import java.lang.reflect.Array;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class TestAggregateOperation {

 private static List<Course> courses = new ArrayList<Course>();
 static{
  courses.add(new Course("Basic math", 89, Course.Type.Math));
  courses.add(new Course("English", 75, Course.Type.Literature));
  courses.add(new Course("Geomatry I", 82, Course.Type.Geomatry));
  courses.add(new Course("Advance math", 92, Course.Type.Math));
  courses.add(new Course("Basic science", 89, Course.Type.Science)); 
  courses.add(new Course("Physics", 78, Course.Type.Science));
  courses.add(new Course("Chemistry", 77, Course.Type.Science));
 }
 public static void main(String[] args) {
  new TestAggregateOperation().example1_2();
  new TestAggregateOperation().example3();
  new TestAggregateOperation().example4();
  new TestAggregateOperation().example5();
 }

 private void example1_2() {
  int[] values = {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20};
  
  System.out.println("Any match of element value 3. "+ (Arrays.stream(values).anyMatch(each -> each==3) ? "FOUND": "NOT FOUND"));
  System.out.println("All element between 1 and 15. "+ Arrays.stream(values).allMatch(each -> each > 0 && each <15));
  //Show how many elements, average and sum of elements
  System.out.print(Arrays.stream(values).count()+" elements, where SUM= "+Arrays.stream(values).sum()+" AVERAGE="+Arrays.stream(values).average().getAsDouble());
  //Show min and max value
  System.out.println(" MAX= "+Arrays.stream(values).max().getAsInt()+" MIN= "+Arrays.stream(values).min().getAsInt());
  //Filter out odd value elements and print the even values.
  System.out.print("Even value elements are= ");
  Arrays.stream(values).filter(each -> each %2 == 0).forEach(each -> System.out.print(each+", "));
  System.out.println("");
 }
 
 public void example3(){
  //Find Math courses and get total and average
  double total = courses
   .stream()
   .filter(each -> each.getCourseType()== Course.Type.Math)
   .mapToDouble(each -> each.getMark())
   .sum();
  double average = courses
   .stream()
   .filter(each -> each.getCourseType()== Course.Type.Math)
   .mapToDouble(each -> each.getMark())
   .average().getAsDouble();
  System.out.println("Total mark ("+total+") and average ("+average+") on math courses");
  
  //Print all courses mark along with name and type
  System.out.println("Courses");
  courses.stream().forEach(each -> System.out.println("Course ("+each.getName()+"), Type ("+each.getCourseType()+"), Mark ("+each.getMark()+")"));
 }
 
 public void example4(){
  System.out.println("Total= "+courses.stream().mapToDouble(each -> each.getMark()).sum());
  System.out.println("Total using reduce= "+courses.stream().mapToDouble(each -> each.getMark()).reduce(0.0, (a,b)-> a + b));
 }
 
 public void example5(){
  CalculateAverage avg = courses.stream().mapToDouble(each -> each.getMark())
    .collect(CalculateAverage::new, CalculateAverage::accept, CalculateAverage::combine);
  System.out.println("Average using collect= "+ avg.average()); 
 }
}

class Course{
 enum Type{Science, Math, Geomatry, Literature};
 private String name;
 private double mark;
 private Type courseType;
 
 public Course(String name, double mark, Type courseType){
  this.name = name;
  this.mark = mark;
  this.courseType = courseType;
 }
 public String getName() {
  return name;
 }
 public void setName(String name) {
  this.name = name;
 }
 public double getMark() {
  return mark;
 }
 public void setMark(double mark) {
  this.mark = mark;
 }
 public Type getCourseType() {
  return courseType;
 }
 public void setCourseType(Type courseType) {
  this.courseType = courseType;
 }
}

class CalculateAverage implements DoubleConsumer{
 int total = 0;
 int count = 0;
 public double average(){
  return count > 0 ? total/count : 0.0;
 }
 @Override
 public void accept(double value){
  total += value;
  count++;  
 }
 public void combine(CalculateAverage other){
  total += other.total;
  count += other.count;
  
 }
}

Explanation - Course class is created to represent a course which has mark and course type attributes. Course type is an enum. In example3 method 5 Course instance are created. To find total and average on Math courses, filter is used with lambda expression to short out Math courses and passes to mapToDouble as stream of Course element. This mapToDouble maps each Course element to a double value which is the mark of the Course instance and passes to sum/average as another stream (DoubleStream). sum/average performs as terminal operation and returns OptionalDouble and getAsDouble method fetches the value.

Usage of Stream.reduce function
This method can be used to work on custom reduction functions which not provided in the API. For example - median.

Arguments of Stream.reduce -

identity - represents both initial value of reduction function and default value if no elements in the stream
accumulator - this is a function which takes two parameters, partial value of reduction and next element of stream. Returns a new partial result

Example-04: Stream.reduce to find total marks obtained.

1
2

System.out.println("Total= "+courses.stream().mapToDouble(each -> each.getMark()).sum());
System.out.println("Total using reduce= "+courses.stream().mapToDouble(each -> each.getMark()).reduce(0.0, (a,b)-> a + b));

Full source code above.

Usage of Stream.collect function
Stream.reduce is not a good choice when dealing with complex object or collection. Because every time accumulator function processes an element in reduce method, will create a new collection including that element. In such situation, it's better to use Stream.collect.

Example-05 - average using new data type.

1
2
3

CalculateAverage avg = courses.stream().mapToDouble(each -> each.getMark())
  .collect(CalculateAverage::new, CalculateAverage::accept, CalculateAverage::combine);
System.out.println("Average using collect= "+ avg.average());

Full source code above.

Explanation -

New data type CalculateAverage is created to store total of elements and count of elements
Stream.collect method is used
Signature of Stream.collect-

supplier - factory function, creates a new instance
accumulator - incorporates stream element to container (CalculateAverage)
combiner - combines two resulting containers and merges

accumulator and combiner don't return any value
'method reference' is passed in supplier/accumulator/combiner which a new feature of JDK 8. Lambda expression could be used

AngularJS - Best practices

2015-11-03T14:05:00.000-05:00

Introduction:
This a quick guide for myself to check/review what pattern/style/standard should be followed during AngularJS application development. Please check the resources section for the links where the gurus have explained elaborately the reason behind each item in this post. Once someone gets the head around the explanations or becomes more advanced developer, this post will serve as a quick cheat sheet for a refresher and reference. Added a facts section which would help going through the practices to understand better.

Some quick facts:

All services (value, factory, service) in AngularJS are Singleton
Service services are instantiated with the new keyword, need to use this for public methods and variables
Factory services expose it's members (variable, function) using return{} defined in it

Practices grouped by component:

Component	Practices to follow
Module	1 - 11
Controller	Module + (12, 13, 15, 16, 17, 18)
Service	Module + (14, 19)
Directive	Module + (20, 21, 22, 23, 24, 25)
Router	Module + (18)

Best practices items:

Create separate file for each controller/service/router/directiveReason: code separation, better testability of code, developers work on independently without being affected by others
File name should follow - ..js
Test file should use spec suffix. ..spec.js
Application Structure should follow LIFT(Locate, Identify, Flat, Try to be DRY) Principle.

Locating our code is easy
Identify code at a glance
Flat structure as long as we can
Try to stay DRY (Don’t Repeat Yourself) or T-DRY

Use unique naming conventions for modules and with separators for sub-modules
Reason: help avoid module name collisions
Put any member variables in any module on top
Reason: helps identify instantly what members are defined and used in the implementation flow
Use named functions instead of passing an anonymous function in as a callback. May ideal to use on major functions
Reason: easy to debug, reduces the amount of nested callback code, increases code readability
Declare any function in any module at the bottom. JavaScript anyway hoists the function definition at the top during execution
Reason: helps separate the implementation detail from the flow and increases readability
Use functional declaration over functional expression (declare and assign to a var)
Reason: removes the concern over using a function before it is defined. Code organization (moving a block of code up or down) won't break anything
Use IIFE scoping. Ideal solution should be creating build task to wrap the concatenated files inside an IIFE. IIFE - Immediately Invoked Function Expression.
Reason:to avoid polluting the global scope with custom function declarations
Consider manual annotation for dependency injection
Reason: to make the code safe of minification
Use the 'controller as' syntax over the classic $scope. Need to follow below steps for that -

Use 'controller as' in view (HTML). Like - 'MyController as ctl'
Refer any members using the controller reference. Like - 'ctl.aValue'
In controller, use a capture variable and associate members to the capture variable. Like - 'var vm = this; vm.aValue = 'A Value';'

Reason: avoids any reference issues, gives way to write code on contextual manner, easier to read

Put bindable variables on top
Reason: helps identify instantly which members of the controller can be bound and used in the View
Put member variables on top
Reason: helps identify instantly which members can be called and must be unit tested
Defer logic in a controller by delegating to services and factoriesReason: Controller would be slimmer where implementation is hidden. Allows logic to be reused by multiple modules (specially by other controllers) and easier for unit testing
Use one controller per view. If any logic needs to be reused, better to move it to a factory/service and use it from each separate controller
Reason: Controller will be slimmer, writing unit test will be simpler
Should use prototype to extend a controller and Object.create to create new object
When a controller must be paired with a view and either component may be re-used by other controllers or views, define controllers along with their routes (check this)
Use this for public methods and variables as services are instantiated with the new keyword
Consider creating a directive for any DOM manipulation. If alternative ways can be used such as using CSS to set styles or the animation services, Angular templating, ngShow or ngHide, then use those instead.
Reason: DOM manipulation can be difficult to test, debug
Use unique and short directive prefix
Reason: helps identify the directive's context and origin
Use restrict E for standalone element represented by the directive
Use restrict A when the directive is enhancing a DOM element's attribute
Avoid ng- as these are reserved for Angular directives
Directives normalization

Prefer using the dash-delimited format (e.g. ng-bind for ngBind). To use an HTML validating tool, data-prefixed version can be used (e.g. data-ng-bind for ngBind).
Prefer using directives via tag name and attributes over comment and class names.
Reason: easier to determine what directives a given element matches

Directive creation

Prefer using the definition object over returning a function
prefix own directive names
Reason: to avoid any future collisions
Unless template is very small, it's typically better to break it apart into its own HTML file and load it with the templateUrl option

Directives should clean up after themselves. Use $element.on('$destroy', ...) or $scope.$on('$destroy', ...) to run a clean-up function when the directive is removed
use controller when want to expose an API to other directives. Otherwise use link.
Use angular wrapper services. Like $document, $window, $timeout, $interval etc. instead of using native.
Reason: easily testable rather mocking native ones by yourself

Few Additional items: these items are more like personal preference. Though the Angular gurus (John Papa, Todd Motto) are more used to doing these or suggesting to do so.

Consider separate line to inject dependencies rather in-line dependencies
Reason: long list of dependencies makes it harder to read. Little confusing as last item in the array is a function rather string value
Use ng-annotate for Gulp or Grunt and comment functions that need automated dependency injection using /* @ngInject */.
Tool usage:

Use Jasmine or Mocha for testing
Use protractor for E2E (End to End) testing
Use karma for test runner
Use Sinon for stubbing and spying
Use headless browser like PhantomJS
Use JSHint as code analyzer

Resources:
1. https://github.com/johnpapa/angular-styleguide
2. https://github.com/toddmotto/angularjs-styleguide
3. http://addyosmani.com/resources/essentialjsdesignpatterns/book/#revealingmodulepatternjavascript
4. http://www.johnpapa.net/angular-function-declarations-function-expressions-and-readable-code/

Tips - Blogger - Useful tips for organize and write post on Blogger

2015-11-01T12:23:00.001-05:00

Introduction:
As I got more involved writing posts frequently, the need of advance features to facilitate it grew which basic blogger hasn't come up with yet. To name such features could be putting a table, inserting programming code etc. Target of this post is to list down such finding/techniques which are used somewhere in my blogs or about to or considering to use. Expect this as working in progress and growing numbers of items over the period.

Features:

1. How to put various type of programming code blocks in a post.
Found this beautiful link which can convert most of the well known programming language code to a formatted one which can be directly used in the Blogger post.

2. Add a JavaScript block on a blog.

Steps to follow:
1. Go to 'https://www.blogger.com/' and login
2. Click on 'More Options' dropdown icon left to 'View blog' button and select 'Layout'
3. Click on 'Add a Gadget' link at right side of the page that will show a popup
4. Click on 'HTML/JavaScript' link
5. On next page, provide a title and write your code in content. I have added a small code snippet to try it out to comply with seeing is believing term.

Script snippet

3. Add a JavaScript block on a post.

Steps to follow:
1. Copy above code snippet
2. Go to Post Edit screen
3. Click on 'HTML' button at top left blogger menu next to Compose button.
4. Paste the code block (better to put at the beginning or at the end)
5. Save/Update the post and view

4. How to add a table in a post.

Table snippet

Steps to follow:
1. Copy above code snippet
2. Go to Post Edit screen
3. Click on 'HTML' button at top left blogger menu next to Compose button.
4. Paste the code block
5. Save/Update the post and view

5. How to make a section collapsible.

Script snippet
1. Copy snippet-01 which is a javascript and follow feature-03 mentioned above. This JavaScript show or hide a element passed to the function as name.
2. Put a DIV around the section needs to be collapsible. '<div id="aID" style="display:block">'. Need to make sure ID value is unique across the section. I used DIV to define a section, any other HTML element can be used as long as ID and style values are provided. Valid values are 'display:block' and 'display:none'.
3. Use an anchor element to generate show/hide action like '<a href="javascript:void(0);" onClick="showHideADiv('aID')"></a>'.
4. Feature 4 & 5 above used exact same technique

Resources:
1. http://hilite.me/

Java - New features in Java 8 - default method

2015-10-24T23:53:00.003-04:00

Introduction:
Java 8 introduces 'default' method declaration in interfaces. The idea behind this to make a change in the interface (adding new method signature) without affecting already implemented classes of the interface. This a nice thing to kept the old implementations intact still adding new features whoever needs it. See Example-01.

Notes:
1. Extend an interface with a default method

Not mention the default method at all, which lets your extended interface inherit the default method
Re-declare the default method, which makes it abstract
Redefine the default method, which overrides it

2. Conflict with multiple interface - if a class uses multiple interfaces having same default method signature, the implementation class should explicitly specify which default method to use or define it's own one. Because the implementation class does not know which default method to use. See Example-02.

Examples:

Example-01: Bird class inherits FlyInterface where the interface has a default implementation which Bird class is happy with rather than providing it's own implementation.

BehaviorInterface -
public interface BehaviorInterface {
public default void fly(){
System.out.println("Generic fly");
}
public default void sing(){
System.out.println("Generic sing");
}
public void swim();
}

Bird:
public class Bird implements BehaviorInterface{

public Bird(){
System.out.println("A bird is created");
}
@Override
public void swim(){
System.out.println("Swim by me");
}
@Override
public void sing(){
System.out.println("My own singing");
}
}

Test above implementation:
public class TestBird {
public static void main(String[] args){
Bird aBird = new Bird();
aBird.fly();
aBird.sing();
aBird.swim();
}

}
Output:
A bird is created
Generic fly
My own singing

Swim by me

Example-02: Two interface with same default method would throw compile time error 'Duplicate default methods named speed with the parameters () and () are inherited from the types SwimInterface and FlyInterface'.

FlyInterface -
public interface FlyInterface {
public void fly();
public default void print(){
System.out.println("Print in FlyInterface");
}

}

SwimInterface -
public interface SwimInterface {
public void swim();
public default void print(){
System.out.println("Print in SwimInterface");
}

}

NewBird -
public class NewBird implements FlyInterface, SwimInterface {
public NewBird(){
System.out.println("A new bird is created");
}
@Override
public void swim() {
System.out.println("My swim implementation");
}
@Override
public void fly() {
System.out.println("My fly implementation");
}
@Override
public void print(){
System.out.println("I am forced to provide specific implementation");
FlyInterface.super.print();
SwimInterface.super.print();
}

}

Test above implementation -
public class TestNewBird {
public static void main(String[] args){
NewBird aBird = new NewBird();
aBird.fly();
aBird.swim();
aBird.print();
}

}

Output:
A new bird is created
My fly implementation
My swim implementation
I am forced to provide specific implementation
Print in FlyInterface

Print in SwimInterface

Resources:
1. https://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html
2. http://www.programcreek.com/2014/12/default-methods-in-java-8-and-multiple-inheritance/

Design pattern - Strategy pattern

2015-10-23T15:13:00.000-04:00

Description:
Define a family of algorithms, encapsulate each one and make it interchangeable. Strategy lets the algorithm vary independently from the client use it.

In short - "Encapsulates an algorithm inside a class"

Problem:
Sometimes new behavior or algorithm or action on related objects are changed or new implementation needs to be provided. Using interface leaves each concrete class to have it's own implementation, no way to reuse/leverage exists implementation. Using class inheritance leaves in situation where all sub classes are forces to provide implementation of the behavior even not appropriate to that sub class.
Strategy pattern encapsulates set of behaviors/algorithms/actions in a separate hierarchy allowing client to use algorithm without knowing implementation and run time change of algorithm. Other hand behavioral implementation changes can be done without affecting client.

How to use Strategy pattern:
1. Concepts involved in Strategy pattern, 'context', 'abstract behavior', 'concrete behavior'

2. Create an interface 'behavior' with abstract method (like execute/do).

3. Create derived concrete class(es) from 'behavior' base class which will provide the implementation of the abstract method(s) of interface. Each concrete class will have alternative implementation detail

4. Need to create separate behavior interface and concrete class creation for each set of behaviors

5. Client will create an instance of concrete behavior and store it in abstract reference. Client class will have a entry point (a method) which will make the abstract method call on abstract reference

Most of the times, client side has a hierarchy where concrete classes derived from an abstract class, the abstract class holds the reference of strategy provided by concrete class and has a method to make a call to abstract strategy reference without knowing the actual concrete class.

Class diagram:

Examples:
1. Duck pond simulation game from Head First Design Pattern. The large variety of duck species can swim but in need of adding behaviors like fly, make quacking sounds on some specific kind of ducks. Fly behavior can be externalized through strategy and ducks would just use composition of fly behavior.

2. Modes of transportation to an airport is an example of a Strategy. Several options exist such as driving one's own car, taking a taxi, an airport shuttle, a city bus, or a limousine service. For some airports, subways and helicopters are also available as a mode of transportation to the airport. Any of these modes of transportation will get a traveler to the airport, and they can be used interchangeably. The traveler must chose the Strategy based on tradeoffs between cost, convenience, and time. (Resources-2)

3. A car objects break behavior can be considered as strategy where regular break or ABS break can be used based on need.

Benefits:
1. Decouples the algorithm/behavior from client who would use it
2. Allows client to pick any implementation of algorithm/behavior
3. Allows algorithms/behaviors to interchange without affecting client
Notes:

1. Strategy pattern uses composition between objects rather than inheritance
2. Follows 'Open/Closed Principle' of OO Design Principle
3. Abstract coupling - client is coupled only to an abstraction and not a particular realization of that abstraction
4. Ideally client should have a mean (mostly using setter method) to change strategy

Things to check back:
1. Coexistence with other patterns

Resources:
1. Head First Design Pattern
2. https://sourcemaking.com/design_patterns
3. https://en.wikipedia.org/wiki/strategy_pattern

Bootstrap - concepts

2015-10-18T17:52:00.002-04:00

About bootstrap:

Uses HTML, CSS and Javascript
front-end framework
Easy and faster to use and develop
Responsive design
Mobile-first approach: In Bootstrap 3, mobile-first styles are part of the core framework
Browser compatible

Bootstrap concepts:

Application setup with bootstrap

Reference to bootstrap

Download - http://getbootstrap.com
CDN -

CSS - http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css
JavaScript - http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js

<!DOCTYPE html> at the beginning of the html page
Mobile first settings - ''

Width is the width application should take to display
initial-scale - initial zoom of the page

Container - bootstrap requires a containing element to wrap site contents. There are two container classes to choose from -

.container - class provides a responsive fixed width container
.container-fluid class provides a full width container

Bootstrap Grid System - grid systems are used for creating page layouts through a series of rows and columns that host application content

Rows must be placed within a .container class
Content should be placed within the columns, and only columns may be the immediate children of rows
Predefined grid classes

.row
.col-xs-<#> .col-sm-<#> .col-md-<#> .col-lg-<#> - # could be 1-12

Multiple column classes can be used to support wide range of devices

Normalize - bootstrap uses Normalize to establish cross browser consistency in the default styling of HTML elements. Normalize.css is a modern, HTML5-ready alternative to CSS resets. CSS reset is to set styling to html elements to null thus avoiding cross-browser differences as much as possible
Text emphasis - text-left text-center text-right text-muted text-primary text-success text-info text-warning text-danger.
List - list-unstyled list-inline.
Bootstrap tables -

base table markup - table
Optional markups - table-striped table-bordered table-hover.(example)
Contextual classes - active success warning danger. (example)
The .table-responsive class creates a responsive table. Put this class in a div which holds a table

Bootstrap form layouts-

Types

Vertical (default) form
In-line form
Horizontal form

Rules to follow

improve accessibility - always use >form role="form"<
optimum spacing - wrap labels and form controls in >div class="form-group"<
class .form-control to all textual input, textarea and select elements
For Horizontal form -

Add class .form-horizontal to the
element
Add class .control-label to all elements
Use grid classes to align labels and groups of form controls

Example - includes all three type of forms

Alert -

Create an alert

Use .alert class in a div
Followed by one of four contextual classes .alert-success, .alert-info, .alert-warning or .alert-danger.

To add close functionality

Add a link or button element in alert div
add class="close" and data-dismiss="alert".
Use × (×) HTML entity for close buttons

Animated alert - add class fade in in alert div.

Dropdown - (example)

To create

Mark a div with class="dropdown".
Add a button or link to open dropdown menu with "class=dropdown-toggle" and data-target="dropdown".
Add class="caret" in a span element to show dropdown arrow
Add class="dropdown-menu".

Dropdown divider - class="divider" in li element
Dropdown header- class="dropdown-header" in li element
Make it accessible to screen readers

Add role="menu" and aria-labelledby="aName" in ul element
Add role="presentation" in li element
Add role="menuitem" in HTML element used inside li

Panel

Create with .panel class in a div
need to define panel-header panel-body panel-footer
Contextual classes - panel-default panel-success panel-info panel-danger panel-warning panel-primary.
To group panels - panel-group.

Collapsible

On button or link

Create a div with collapse class
Create a link or button HTML element with data-toggle="collapse" and data-target="#div_id". This is to show/hide the collapsible div

On a panel - (example)

Add button or link in panel header (see above)
Put panel body and footer in a div with classes panel-collapse collapse.

Accordion - (example)

Panel group div should be ided with a name
Use the data-parent attribute to specify the div id - all collapsible elements under the specified parent will be closed when one of the collapsible item is shown

Tabs

Tabs are created with $gt;ul class="nav nav-tabs"<.
Current page is marked using class="active" in li element
Use dropdown creation structure in a li which needs to be displayed as dropdown and mark that li with class="dropdown". (example)
dynamic tabs

add the data-toggle="tab" attribute to each link
add a .tab-pane class with a unique ID for every tab
wrap them inside a div element with class .tab-content.

Pills - similar concept as tab. Most cases replacing tab with pill will work

Modal - (example)

Use a button or a link with attributes:

data-toggle="modal" opens the modal window
data-target="#id" points to the id of the modal

Defining modal

The parent div of the modal must have an ID, will be used in data-target attribute and .modal class
The attribute role="dialog" to improve accessibility
The .modal-dialog class sets the proper width and margin of the modal.

Defining modal content

A div with class="modal-content"
Add the modal's header .modal-header, body .modal-body, and footer .modal-footer.
To add close feature in header

Add a button/link inside the header with data-dismiss="modal"
Add .close class styles the close button

Add .modal-title class styles the header with a proper line-height.

Images - below classes need to use in img html element

img-rounded img-thumnail img-circle img-responsive.

Note:

1. To learn about bootstrap elements which are not mentioned and know more in detail about the elements explained, see the resources

Resources:

1. http://www.w3schools.com/bootstrap/

2. http://www.tutorialspoint.com/bootstrap/

JQuery - Handout - quick memory refresher

2015-10-17T18:21:00.001-04:00

Introduction:
This post is to provide some quick refreshing facts and best practices for those folks who haven't worked on jQuery for a while. For beginners, it's better to go through the basic/advance tutorials first and check back with this later.

About jQuery:
1. jQuery is a JavaScript Library
2. jQuery greatly simplifies JavaScript programming

jQuery concepts:

Pointing to jQuery

From local reference - ''- assuming jquery.js file is in same folder with the html
Content Delivery Network (CDN) based - 'src="http://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js" type="text/javascript"'

launching code - Option-02 is prepared as that's invoked as soon as the DOM is ready rather waiting the page loaded fully (including images)

Option-01: window.onload = function() { //code};
Option-02: $( document ).ready(function() { //code });
Option-03: is shortcut to option-02. '$(function(){ //code});'

Callbacks and functions - A callback is a function that is passed as argument to another function and will be executed only after the function where it's passed has completed

Callback without Arguments - '$.get('aPage.html', callBackFn);'
Callback with Arguments -

Not possible - '$.get('aPage.html', callBackFn(param1, param2));'. This will execute the callBackFn first and return of it will be passed to get function
Correct way - '$.get( "aPage.html", function() { callBackFn( param1, param2 ); });'

$ vs $()

Methods called on jQuery selections (used $()) are in the $.fn namespace, and automatically receive and return the selection as this
Methods in the $ namespace are generally utility-type methods, and do not work with selections; they are not automatically passed any arguments

Ways to avoid conflict with other libraries

new alias - noConflict returns a reference to jQuery function
'var $j = jQuery.noConflict();'
use self invoking function -
'jQuery.noConflict();(function($){ //code})(jQuery);'
Argument through jQuery(document).ready() function
'jQuery(document).ready(function( $ ) { // code });'

Selecting elements

tag name - '$("aDOM");'
ID - '$("#idToFind");'
class name - '$(".className");'

Pseudo selectors - :visible :enabled :checked :text :button :file (see references for complete list)
comma to use to select multiple DOM objects
chaining - any method call on a selection returning a jQuery object, allows to call jQuery method on the object without pausing for a semicolon.
'("#aDiv").find("p").eq("1").html("Changed text");' - find second p in a dive called aDiv and change content

Methods -

getter/setter - html, text, val - can be both setter (with argument) or getter (no argument)
Move, copy and remove elements
Attributes - .attr() to get/set value to selector's attribute.

Object of key-value pair can be passed to set multiple

Selection traversing

parents - .parent() .parents() .parentUntil() .closest().
children - .children() .find().
Siblings - .prev() .next() .siblings() .nextAll() .prevAll() .nextUnti() .prevUntil()
Styling and dimensions
CSS classes - addClass("aClass") removeClass("aClass").
Dimensions - width, height - getter/setter

Data methods - stores data to the element and manages the memory issues.
'$("#aDiv").data("aKey", {name:"Riaz"});' '$("#aDiv").data("aKey");'
Utility methods -

$.trim() -
$.each() - generic iterator function for looping over object/arrays/ array-like objects
example:
'$.each([1,2,3,4], function(idx, val){ console.log("Element "+idx + " is "+val);});'
output:
Element 0 is 1
Element 1 is 2
Element 2 is 3
Element 3 is 4
$.proxy(function, scopeObj) -
example:
var aFn = function() { console.log( this ); };
var aObj = { name: "Riaz" };
aFn();
var proxyAFn = $.proxy( aFn, aObj );
myProxyFunction();

Output:
Window {external: Object, chrome: Object, document: document, speechSynthesis: SpeechSynthesis, caches: CacheStorage…}
Object {name: "Riaz"}
$.type() - returns the type of passed object
.map() - returns array from matched elements from selector

jQuery events -

Some common events

Mouse Event	Key Event	Form Event	window Event
click	keypress	submit	load
dblclick	keydown	change	unload
mouseenter	keyup	focus	size
mouseleave		blur	scroll

Event call syntax -

Assign an event - $("p").click();
Indicates what should happen after event -
$("p").click(function(){ /*actions*/});

Usage of '.on()'.

All the methods above are shortcut of this
Syntax - '$("p").on("click", function(){/*actions*/});'

Handling event -

Many events single event handler - comma separated events
Multiple event and event handler association -'$( "p" ).on({ "click": function() { console.log( "clicked!" ); },
"mouseover": function() { console.log( "hovered!" ); } });'
Data can be passed to event -
'.on( "click", {name:"Riaz"}, function(event){ console.log("Name:"+event.data.name");});'

Instead of anonymous function, a named function can be passed as event handler
Namespacing event - for complex and plugin application, it's a good idea to namespace application specific events to avoid conflict
tead down events

Unbind all click event - $( "p" ).off( "click" );
Unbind only aHandler from click - $( "p" ).off( "click", aHandler );

Event handler to run only once - using '.one(...)'
Properties and method of eventhanlder

Properties - pageX pageY which type data target namespace timestamp etc.
Methods - preventDefault() stopPropagation() etc

12. Ajax in jQuery

Syntax - '$.ajax(...)'
Data types - type of data you expect to get back from an Ajax request

text html script json jsonp xml

API -
// Using the core $.ajax() method
$.ajax({
// The URL for the request
url: "- page url",
// The data to send (will be converted to a query string)
data: { },
// Whether this is a POST or GET request
type: "GET",
// The type of data we expect back
dataType : "json",
// Code to run if the request succeeds;
success: function( json ) { /* do something */ },
// Code to run if fails; raw request and status codes are passed
error: function( xhr, status, errorThrown ) { /* do something */ },
// Code to run regardless of success or failure
complete: function( xhr, status ) { /* do something */ } });
Convenient functions - below functions are wrapper around core .ajax()

functions - $.get $.post $.getScript $.getJSON
parameters -

Required - url
Optional - data success dataType

Ajax events

Local events - specify within ajax request object

beforeSend success error complete

Global events

ajaxtStart ajaxStop

13. Serializing form data

Using $("#aForm").serialize() - serializes form data into a query string
Using $("#aForm").serializeArray() - serializes to array of objects

Resources:

1. http://api.jquery.com/

2. http://www.w3schools.com/jquery/

3. https://learn.jquery.com/

Design Pattern - Command pattern

2015-10-16T09:59:00.002-04:00

Description:
The command pattern encapsulates a request as an object to perform an operation on an object (receiver). This allows client to parameterize other objects (invoker) with different requests, queue or log requests, and support undoable operations.

In short - "Encapsulate a command request as an object"

Problem:
Need to issue requests to objects without knowing anything about the operation being requested or the receiver of the request.

How to use Command pattern:
1. Concepts involved in Command pattern, 'receiver', 'invoker' and 'command'

2. Create an interface 'command' with abstract method 'execute'

3. Create derived concrete class(es) from 'command' base class which encapsulates a 'receiver', a method to invoke and the arguments to pass

4. 'Receiver' is actually responsible to perform the desired action which is encapsulated in the 'command' as attribute. Mostly passed to command through constructor

5. 'Invoker' gets the command object(s) through method like 'setCommand(Command)' and provides a mean to make a call to 'command' class's 'execute' method

6. Client of Command pattern creates a concrete 'command' object, passes reciever to it and sets it to 'invoker' by making call to 'setCommand(Command)'. Later client asks 'invoker' to execute the command.

Class diagram:

Benefits:
1. decouples the object that invokes the operation from the one that know how to perform it
2. Allows parameterization of clients with different requests
3. Allows invoker to save request in a queue
4. Undo and redo functionality can be implemented as

Notes:

1. Macro Command - represents set of commands to be performed on call to be executed. Macro commands can be implemented as Composite, can also have other operation like undo, redo, forward/backward traversing, logging, queue etc.
2. Undo/redo action - can be performed two ways

Store receiver's previous state in memory
Store set of performed actions on receiver in memory

3. Queuing the requests - as the commands are isolated from invoker, those can be executed asynchronously in background of an application. Usually invoker keeps a set of commands, runs in main thread and action request to receiver can happen in a separate thread. To have faster processing, multi-thread can be considered to run through the set of commands. Scheduler, thread pool, job queues etc. applications can be considered as good example of this scenario.

Things to check back:
1. Coexistence with other patterns
2. Alternates of Command patterns

Resources:
1. Head First Design Pattern
2. https://sourcemaking.com/design_patterns
3. https://en.wikipedia.org/wiki/command_pattern

Design Pattern - Iterator pattern

2015-10-15T23:07:00.001-04:00

Description:
"The Iterator Pattern provides a way to access the elements of an aggregate object without exposing its underlying representation".

This simplifies aggregate interface and implementation by placing traversal responsibility on iterator and this traversal of a collection is promoted to a full object comparing to a logic embedded in the aggregate. In the process helping the client of aggregate object a polymorphic traversal.

In short - "Sequentially access the elements of a collection"

How to use Iterator pattern:
1. Two types of objects needed, one is called 'aggregate', other is called 'iterator'

2. "Aggregate" abstraction should have a method (i.e. createIterator) to create corresponding iterator

3. Each derived concrete class of aggregate base class has to provide implementation of the abstract method (i.e. createIterator)

4. Create abstract "iterator" base which can have method like next, hasNext, remove. See Note-3 and 4 to know more

5. Each derived concrete class of "iterator" base class should provide implementation of methods described in the "iterator".

6. The client will create necessary aggregate object based on collection structure and ask for "iterator" from the aggregate object which will return a concrete iterator as generic type.

Class diagram:

Benefit:
1. Aggregate object can have it's encapsulation intact without exposing how the collection structure is defined
2. Allows the client to traverse different ways
3. Supports multiple simultaneous traversal on a collection
4. Provides uniform interface to traverse heterogeneous collections

Notes:

1. Internal Iterator vs External Iterator - java.util.Iterator is an external iterator as client has to control the iteration. Internal iterator is controlled by the iterator itself and client has to inform the iterator needs to be done as it iterator through.

2. Robust Iterator - the iterator can provide both insertion and deletion during traversal and doesn't make a copy of the aggregate. This iterator makes sure the element(s) not fetched twice or skipped due to insert or delete in the collection.

3. java provides java.util.Iterator which can be used in most of the cases. This interface has three methods (next, hasNext, remove). If derived concrete Iterator doesn't need to support for remove method, java.lang.UnsupportedOperationException can be thrown.

4. An Iterator can be defined to move backwards as well forwards. Java provides such interface java.util.ListIterator with next, hasNext, previous, hasPrevious, remove methods along with other like (nextIndex, previousIndex, set, add).

Things to check back:
1. How to write a thread safe, robust Iterator

Resources:
1. Head First Design Pattern
2. https://sourcemaking.com/design_patterns
3. https://en.wikipedia.org/wiki/Iterator_pattern

Design Pattern - Visitor pattern

2015-10-13T23:09:00.000-04:00

Definition:
From gang of four - "Represent an operation to be performed on the elements of an object structure. Visitor lets you define a new operation without changing the classes of the elements on which it operates".

Head First - "Use the Visitor pattern when you want to add capabilities to a composite of objects and encapsulation is not important".

In short - "Defines a new operation to a class without change"

How to use Visitor pattern:
1. Two types of objects needed, one is called 'element', other is called 'visitor'

2. Create a base visitor class hierarchy with "visit" method for each concrete derived class the composite hierarchy. Each "visit" method accepts a single argument - reference to the original element derived class

3. Each derived concrete class of visitor base class has to provide implementation of each "visit" method

4. Add abstract/virtual "accept" method in base element hierarchy. This takes an argument of abstract base class of visitor hierarchy

5. All sub-element or derived concrete classes has to provide implementation of "accept" method by simply calling "visit" method of passed concrete Visitor class where "this" is passed as reference

6. For any operation needed for client will create and instance of concrete visitor class and pass that as reference to "accept" method in each concrete element class. The client of Visitor pattern is called traverser which knows how to guide the visitor through the composite structure.

Class diagram:

Examples:
1. A restaurant Application might have menu, menu item and ingredient objects with related attributes and operations. If reality changes where customers demand nutritional information on menu items as well as on ingredients, rather making elements (menu, menu item, ingredient etc) level changes visitor pattern can be implemented.

2. A Shopping cart where we can add different type of items (Elements), when we click on checkout button, it calculates the total amount to be paid. Now we can have the calculation logic in item classes or we can move out this logic to another class using visitor pattern

Benefit:
1. Allows to add operations/functionalities to a composite structure without changing the structure itself
2. Adding new operations/functionalities is relatively easy
3. The code for operations performed by the Visitor is centralized

Trade-off:
1. The composite classes' encapsulation is broken when the visitor pattern is used
2. Because the traversal function is involved, changes to composite structure are more difficult

Notes:

1. Double dispatch - the operation executed depends on: the name of the request (method name), and the type of TWO receivers (the type of the Visitor and the type of the element it visits)

From above class diagram double dispatch can be as below -

When the accept method is called in the program, its implementation is chosen based on both:

The dynamic type of the element.
The static type of the visitor.

When the associated visit method is called, its implementation is chosen based on both:

The dynamic type of the visitor.
The static type of the element as known from within the implementation of the accept method, which is the same as the dynamic type of the element. (As a bonus, if the visitor can't handle an argument of the given element's type, then the compiler will catch the error.)

Consequently, the implementation of the visit method is chosen based on both:

The dynamic type of the element.
The dynamic type of the visitor.

2. A more flexible approach to this pattern is to create a wrapper class implementing the interface defining the accept method. The wrapper contains a reference pointing to the 'abstract base element' which could be initialized through the constructor. This approach avoids having to implement an interface on each element. (Resource-04)

Resources:
1. Head First Design Pattern
2. https://sourcemaking.com/design_patterns
3. https://en.wikipedia.org/wiki/Visitor_pattern
4. Java Tip 98: Reflect on the Visitor design pattern.

JavaScript - Karma - Setup - prepare the environment to work with karma

2015-10-01T13:13:00.002-04:00

Introduction:
This article covers how to setup karma and run a sample test to make sure installation worked without any issues. Both local and global installation will be covered. Karma installation depends on some other dependent technologies which are not part of this article.

Brief introduction about the technologies involved:

Node:

Considered as server side JavaScript or browser less JavaScript
JavaScript runtime built on Chrome’s V8 JavaScript engine
Node uses event driven, non-blocking I/O model which makes it lightweight and efficient

npm:

This is a package manager for JavaScript is default for Node.js
Elaboration is 'Node.js Package Manager'

karma:

Test Runner tool for JavaScript invented by AngularJS team
Karma runs on Node.js and is available as an NPM package
Test runner means way to run jasmine/mocha (Javascript unit test tool) without loading explicitly in the browser. To confuse thing, it's testing without the browser in the browser.

Karma setup:
Karma installation needs node and npm installed prior. So those two installation will be covered first.

Environment info:

Operating system - Windows 2008 R2
node version - 4.1.0
npm version - 3.3.3

1. Setup node:

Link to download: https://nodejs.org/en/
Click on download button which will download file ‘node-v4.1.0-x64.msi’
Execute the msi file. Setup wizard is simple enough, just click on Next button
It will install command prompt and node.js, both are shell/command prompt. Node.js is used for node programming.
Test installation’s done successfully. Use command prompt. Run ‘node –v’. returned 4.1.0.

2. Update npm:

Node is bundled with a version of npm. However, npm gets updated more frequently than Node does, so it’s a good idea to upgrade npm. Before upgrading npm, run ‘npm -v’ which returned ‘2.14.3’.
Install npm – execute command ‘npm install npm -g’ in ‘node command prompt’
Check upgraded version. Execute ‘npm -v’ which returned 3.3.3.

3. Install karma:

Karma can be installed per project or in global context accessible for each project.

Local installation:

Go to the folder where karma needs to be installed
Execute command 'npm install karma' from 'node.js command prompt
This will create a folder called 'node_modules'. Actually installing any npm package will create that folder where all the dependent modules will be installed

Global installation:

Execute command 'npm install karma -g' from 'node.js command prompt. -g argument indicates installation in global space.
The 'node_modules' will be created in the user's directory.

4. Install dependencies:

Karma requires to install test tool, browser launcher etc. Below commands will add all the dependencies need to run the karma locally. Just add -g to install globally.

- 'npm install karma-jasmine --save-dev'

- 'npm install karma-requirejs --save-dev'

- 'npm install karma-phantomjs-launcher --save-dev'

- 'npm install karma-chrome-launcher --save-dev'

5. Initialize karma:

Command: 'node node_modules/karma/bin/karma init'. This will ask series of questions.
Testing framework - default jasmine. Use up or down arrow keys to select preferred one
Need requireJS - default no. Up arrow to select yes. I use requireJS and all my posts will be referring it
Select Browser - default Chrome. Can select multiple in next line. To go to next option, needs to hit double Enter
Next needs to specify all source file locations. Double enter for next option
Exclude files.
Bootstrap file for requireJS. Selecting yes will create a test-main.js in root location with RequireJS configured and run the tests. Selected no for the purpose of this post.
Watch file modification. Select yes. This streamlines any changes in source files and testing done immediately
Initialization is done. This will create a file 'karma.conf.js' in the root location where all the above settings can be seen and modified based on need.

6. Run karma:

Command - 'node node_modules/karma/bin/karma start'
This will start karma on default port 9876 (if not changed) which will start Chrome browser for it's purpose.

Notes:

1. Error-01: Not so obvious error during global karma installation

Solution: use command -

- 'npm install --ignore-script -g karma'

2. Error-02: throw error('No provider for "' + name + '"!');

Solution: next line of above should display which provider is missing. Install that provider using 'npm install --save-dev'

Resources:
1. http://karma-runner.github.io/0.13/intro/installation.html
2.

Hand out - Javascript - Regular Expression

2015-09-25T03:13:00.004-04:00

Introduction:
This post is to know regular expression, see usage in sample examples using JavaScript. Plan is to update this post every time a regular expression is solved. Considering this as a quick hand out to refresh memories as well as a list regular expression which are encountered at some point of development life time.

What is regular expression:
A regular expression is a sequence of characters that forms a search pattern to match a combination in strings. In JavaScript regular expression is an object which can be created explicitly or implicitly.

Regular expression creation:

Option-01: implicit way -

var regExp = /[0-9]/;

Option-02: explicit way-

var regExp = new RegExp('[0-9]');

Note: better to use option-01 as option-02 has implication on execution speed.

Various patterns:

1. Direct match - /direct/. This pattern will find a match in string 'Looking for a direct match' but not in 'Misspelled the word diract'.

2. Special characters -

Pattern	Meaning/example	Matching String	Non matching string
\	Escape character
^	matches beginning of input. /^A/	Apple	Banana
$	matches end of input. /a$/	Banana	apple
*	matches preceding expression 0 or many times. */p/**	- Apple - Orange
+	matches preceding expression 1 or many times. /p+/	- Apple - Peach	- Orange, any string not have p in it
?	matches preceding expression 0 or 1 times. /p?/	- Peach - Pear	- Apple -
()	matches and remembers it. Also called capturing parentheses. /(p)/		Table Cell
(?:)	Matches 'x' but does not remember the match
(?=)	Look ahead. /A(?=p)/. Match only if A (capital) followed by p.	- Apple - Apricot	- Avocado - apple
(?!)	negated look ahead. /A(?!p)/. Match only if A (capital) doesn't follow by p.	- Avocado - apple	- Apple - Apricot
\|	matches either one. /A\|a/.	- Apple - apple	- one - Two - Three
{}	matches exactly that many occurrences. /p{2}/. Has to be positive number.	- Apple - Pepper	- Apricot - Pear - Peach
{ , }	matches at least first and at most second. /p{1,2}/.	- Apple - Pepper	- Apricot - Pear - Peach
[]	Character set. /[Pe\|pe]/.	- Pepper - Pear - Peach	- Apricot - Apple
[^ ]	Negated character set. /[^pe]/.	- Pear - Peach - Apple - Apricot	- Pepper
\b	matches word boundary where word character not preceded or followed wit any character. /\bp/.	- pear - peach	-Apple
\d	matches digit character. /\d/. equals to /[0-9]/.	- 2apples - 3 Pears	- Apple - Pear
\D	match non digit character. /\D/. equals to /[^0-9]/.	- Apple - Pear	- 2apples - 3 Pears
\s	matches white space. /\s/.	- 2 Apples - Apple or orange	- AppleOrOrange - PearAndPeach
\S	matches a single character other than white space. /\S/.
\w	matches any alphanumeric words including _. /\w/. equals to /[A-Za-z0-9_]/.	- &^%$A match A
\W	Matches any non-word character. /\W/. equals to /[^A-Za-z0-9_]/.	- Appl# matches #
Note: the list is not complete. Check Resource links for complete list

Regular Expression properties:

Property Name	Description
lastIndex	The index at which to start the next match
ignoreCase	Indicates if the "i" flag was used to ignore case. Returns true or false
global	Indicates if the "g" flag was used for global match. Returns true or false
multiline	Indicates if the "m" flag was used match from multiple lines. Returns true or false
source	Returns the pattern used in regular expression
split	This method is in String. If match found, returns array of substrings.

Methods related to regular expression:

Method Name	Description
exec	that executes a search for a match in a string. It returns an array
re.exec(str) - matches the regular expression (re) against the input parameter string (str). Properties: [0] - The full string of characters matched [1], ...[n ] - matched substrings index - zero based, index of matched string input - the string match happening against
test	tests for a match in a string and returns true or false
re.test(str) - regular expression (re), input parameter string (str).
match	This method is in String. Executes a search and returns an array of information or null on a mismatch
str.match(re) - regular expression (re) pattern to match on string (str).
search	This method is in String. Returns the index if match found, else -1.
str.search(re) - regular expression (re) pattern to search from string (str).
replace	This method is in String. Replaces matched substring with replacement substring.
str.replace(re, newSubStr) - replace substring from string (str) if match found based on regular expression (re) pattern with new sub string (newSubStr). Note: a function can be used instead of newSubStr as replacement
split	This method is in String. If match found, returns array of substrings.
str.split([separator[, limit]]) - Splits the string (str) based on separator. Limit value can be specified which is optional. If used, it splits on every match on separator but truncates the array to meet the limit.

Regular expression with flags:

Syntax: /regular expression/flag;
Flags:

i - case ignore search

g - global search

m - multi line search

Examples:

1. Replace all instances of e or E in a String - /e|E/g.

2.

Resources:
1. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions
2.

Hand out - Javascript - requireJS - part-01

2015-09-18T02:54:00.000-04:00

What is RequireJS?
RequireJS is a module loader for JavaScript which facilitate module based programming where large application is broken down into relatively small and manageable code blocks. That leads to the problem of managing those dependencies between modules and emerges the need of module loader. RequireJS is one of the most popular. It improves script load speed.

Few concepts:

1. Without RequireJS, html file has to have all the dependent scripts loaded using script tag and the correct order has to be maintained. That is treated as synchronous loading and scripts will be loaded one at a time which slows the page view. RequireJS uses AMD (Asynchronous Module Definition) where modules are loaded in parallel and boosts the page load

2. Using require() vs. define()

Both require() and define() to load dependencies. The require() function is used to run immediate functionalities, while define() is used to define modules for use in multiple locations. In the example, in both app.js and main.js, need to run immediately. So, require() was used. Check the code below to get better understanding.

3. In the example, in all places where require() is used, can be replaced with requirejs or vice versa.

4. In main.js, define() could be used instead of require().

Use RequireJS - integrate jQuery/bootstrap and print hello world in console

1. Folder structure - this folder structure is not that important to test RequireJS as long as references are pointed correctly. But have good folder structure helps organizing the files, improves manageability. Folder structure followed -

css - for css files

fonts - for fonts

img - for images

js - any JavaScript

js/app - will have application specific js files

js/vendors - any third party js (like - jQuery, bootstrap etc.)

app.js - contains RequireJS configuration

main.js - contains the web app dependencies and once loaded can start application using any framework

index.html - to load the script

note: app.js and main.js can be combined into one and point from script in html file

2. Download RequireJS and put in js/vendors folder

3. Define RequireJS script in the html

Explanation:

- only one script tag where src refers to RequireJS. data-main attribute tells require.js to load js/app.js after require.js loads. js extension is not needed as RequireJS works only on JavaScript

4. Configure RequireJS to load dependecies (jQuery/bootstrap)

Explanation:

baseUrl - the root path to use for all module lookups

paths - path mappings for module names not found directly under baseUrl. The path settings are assumed to be relative to baseUrl, unless the paths setting starts with a "/" or has a URL protocol in it ("like http:")

shim - Configure the dependencies, exports, and custom initialization for older, traditional "browser globals" scripts that do not use define() to declare the dependencies

End of the file, it makes call to main.js using require() function

5. Load application dependencies

Explanation:
- application code requires jQuery.

- require method - takes two parameters. First parameter takes dependency array. Second parameter defines the function which have application code. In this case just printing a message to screen.

6. Open the index.html file in the browser. Used chrome to view it and developer tool to check console message

Resources:

1. http://requirejs.org/

2. http://www.sitepoint.com/understanding-requirejs-for-effective-javascript-module-loading/

Hand out - JavaScript - using Jasmine

2015-09-16T15:52:00.004-04:00

jasmine what:
Testing framework for javascript code. Very lightweight framework as it is not dependent on any other frameworks or any DOM. This follows BDD (business Driven Development), meaning based on user needs rather than code needs.

Jasmine concepts:

Suite –

A test suite begins with a call to the global Jasmine function describe with two parameters: a string and a function. String is the title of the suite. Function is the block of code executes the suite.

points -

defined using describe jasmine function
describe is a namescope
describe can have multiple describe in it

Specs –

Defined using describe jasmine function are defined by calling the global Jasmine function it, which, like describe takes a string and a function. The string is the title of the spec and the function is the spec, or test.

Points -

it represents actual unit test
it is also a namescope
it can have multiple expectations (expect)
A spec javascript which points to the module to be tested. Naming convention of spec file is ‘Spec.js’

Expectation –

Expectations are built with the function expect which takes a value, called the actual. It is chained with a Matcher function, which takes the expected value.

Points -

expect – to check ‘it’ test passed or not
systax – expect().toBe();

Matcher -

Each matcher implements a boolean comparison between actual value and expected value. It reports to jasmine if expectation is true or false. Jasmine will pass or fail the spec.

How to install/use jasmine:

Download link: https://github.com/jasmine/jasmine/releases

Version tested: 2.3.4

Option-01: using jasmine runner in web

Sample application - this application will test a very simple javascript function which simply adds two variables. testJasmine01.html is considered as jasmine spec runner where test result would be shown.

folder structure followed:

Add jasmine libraries, code to test and jasmine spec definition on code in html:

Code to be tested: main.js file with a very simple function

var sum = function(a, b){

return a * b;

}

Spec code: mainSpec.js

(function(){

describe("main file", function(){

describe("add method", function(){

it("Should add 2 numbers together", function(){

expect(sum(2,3)).toBe(5);

});

})();

Run the test: just open the html page in a browser. Output should look like below where the test failed.

Fix the error: change main.js, * to +

var sum = function(a, b){

return a + b;

}

Run the test again: this time it passes

Modify the spec to add concatenation test:
(function(){
describe("main file", function(){
describe("add method", function(){
it("Should add 2 numbers together", function(){
expect(sum(2,3)).toBe(5);
});

it("Should concatenate two values", function(){
expect(sum('Riaz ', 'Ahmed')).toBe('Riaz Ahmed');
});
});
});

})();

Run the test:

Option-02: using command line jasmine.

This will be covered in a separate post in future.

Resource:

http://jasmine.github.io/2.0/introduction.html

Hand out - Javascript - quick facts and best practices

2015-09-12T09:38:00.000-04:00

Introduction:

This post is to provide some quick refreshing facts and best practices for those folks who haven't worked on JavaScript for a while. For beginners, it's better to go through the basic/advance tutorials first and check back with this.

Some quick facts -

External JavaScript Advantages

Separation of concerns HTML and code
HTML and JavaScript can be maintained and read easily
Cached JavaScript files can speed up page loads

JavaScript Display Possibilities

Writing into an alert box, using window.alert()
Writing into the HTML output using document.write()
Writing into an HTML element, using innerHTML
Writing into the browser console, using console.log()

JavaScript statements are composed of: Values, Operators, Expressions, Keywords, and Comments.
All JavaScript identifiers are case sensitive
JavaScript ignores multiple spaces. Add white space to script to make it more readable
A variable declared without a value will have the value undefined. If re-declare a JavaScript variable, it will not lose its value
JavaScript Objects - JavaScript objects are written with curly braces. Object properties are written as name:value pairs, separated by commas

Example: var employee = {name:”Riaz”, address:”Ashland”};
Accessing object properties:

objectName.propertyName
objectName[propertyName]

Accessing object method:

objectName.methodName();
calling objectName.methodName without () will return function definition

Using an Object Constructor - Sometimes we like to have an "object type" that can be used to create many objects of one type.

Example:
function person(first, address) {this.firstName = first;this.address = address;}
var p1 = new person("Riaz", “Ashland”);
var p2 = new person("Rahat", “Dhaka”);

Object prototype - All JavaScript objects (Date, Array, RegExp, Function, ....) inherit from the Object.prototype.
Adding Properties and Methods to Objects –

Add new properties (or methods) to an existing object

Example: p1.age = 20; This will add age property to p1 object (above example) but p2 will not have it.

Add new properties (or methods) to all existing objects of a given type. This can be done by adding property or method directly in the constructor function.
Add new properties (or methods) to an object prototype - using prototype keyword on an object.

Example: p1.prototype.age = 20; This code will add age property to both p1 and p2 objects, though p1.age will have valid value only.

JavaScript Objects are Mutable, addressed by reference, not by value. var x = y; The object x is not a copy of y. It is y. Both x and y points to the same object. Any changes to y will also change x, because x and y are the same object

In JavaScript, all objects have the valueOf() and toString() methods
JavaScript for/in statement loops through the properties of an object

typeof operator to find the type of a JavaScript variable

typeof "Riaz" // Returns string
typeof 10 // Returns number
typeof false // Returns boolean
typeof [1,2,3,4] // Returns object
typeof {name:”Riaz”, address:”Ashland”} // Returns object

undefined - a variable without a value, has the value undefined. The typeof is also undefined.
Null - is "nothing". It is supposed to be something that doesn't exist. Unfortunately, in JavaScript, the data type of null is an object.

Difference Between Undefined and Null

typeof undefined // undefined
typeof null // object
null === undefined // false
null == undefined // true

JavaScript Function

Function Invocation - the code inside the function will execute when "something" invokes (calls) the function using operator ():

When an event occurs (when a user clicks a button)
When it is invoked (called) from JavaScript code
Automatically (self invoked)

Javascript event

list of some common HTML event

onchange - An HTML element has been changed
onclick - The user clicks an HTML element
onmouseover - The user moves the mouse over an HTML element
onmouseout - The user moves the mouse away from an HTML element
onkeydown - The user pushes a keyboard key
onload - The browser has finished loading the page
etc...

Javascript Array

Create array

var arrayName = [“item1”, “item2”, ….];
var arrayName = new Array(“item1”, “item2”, ….);

Array can Have Different Objects in it
Array Methods

Join() – joins all array elements into a string
Push() – add element at the end
Pop() – removes last element
sort() –
reverse() –
concat() - creates a new array by concatenating two arrays
etc..

Javascript regular expression - (more on regular expression)

A regular expression is a sequence of characters that forms a search pattern.
Syntax

/pattern/modifier

Methods

search() - uses an expression to search for a match, and returns the position of the match.
replace() - returns a modified string where the pattern is replaced.
test() - searches a string for a pattern, and returns true or false, depending on the result

Modifier

i - Perform case-insensitive matching
g - Perform a global match (find all matches rather than stopping after the first match)
m - Perform multiline matching

Patterns

Brackets are used to find a range of characters

example: [abc] [0-9]

Metacharacters are characters with a special meaning

\d - find a digit
\s - find a whitespace character
\b - find a match at the beginning or at the end of a word
etc...

Quantifiers define quantities:

n+ - Matches any string that contains at least one n
n* - Matches any string that contains zero or more occurrences of n
n? - Matches any string that contains zero or one occurrences of n

Best practices:

Using document.write() after an HTML document is fully loaded, will delete all existing HTML
It is a good idea to place scripts at the bottom of the body element. This can improve page load, because HTML display is not blocked by scripts loading.
Avoid String, Number, and Boolean objects. They complicate code and slow down execution speed.
If a value assigned to a variable that has not been declared, it will automatically become a GLOBAL variable.
Avoid Global Variables - Global variables and functions can be overwritten by other scripts.
Always Declare Local Variables - Local variables must be declared with the var keyword, otherwise they will become global variables.
It is a good coding practice to initialize variables when you declare them
Use === Comparison - the == comparison operator always converts (to matching types) before comparison. The === operator forces comparison of values and type.
JavaScript is loosely typed. A variable can contain different data types, and a variable can change its data type. Numbers can accidentally be converted to strings or NaN.
Don't Use new Object()

Use {} instead of new Object()
Use "" instead of new String()
Use 0 instead of new Number()
Use false instead of new Boolean()
Use [] instead of new Array()
Use /()/ instead of new RegExp()
Use function (){} instead of new function()

If a function is called with a missing argument, the value of the missing argument is set to undefined. Undefined values can break code. It is a good habit to assign default values to arguments.
With JavaScript, null is for objects, undefined is for variables, properties, and methods. To be null, an object has to be defined, otherwise it will be undefined
Only modify own prototypes. Never modify the prototypes of standard JavaScript objects
Reduce DOM Access - Accessing the HTML DOM is very slow, compared to other JavaScript statements. If a DOM element needs to access several times, access it once, and use it as a local variable
Global variables (or functions) can overwrite window variables (or functions). Any function, including the window object, can overwrite global variables and functions.

Resources:

http://www.w3schools.com/js
http://peerprogrammer.blogspot.com/2015/09/hand-out-javascript-regular-expression.html

Tools/Techniques	Things to focus on
Spark UI	Stage and task completion time 25 percentile, 50 percentile, 75 percentile of execution time Memory usage by executor In and out data for tasks DAG – helps to find out job/stage optimization Garbage collection time
Spark Log	Partition size – evenly distributed or not
Yarn Manager (if yarn is used as resource manager), UI or yarn command	Number of executors Job/task execution time for each executor Memory usage for each executor
Data Storage (HDFS, object storage (S3), etc)	File size - helps to identify skew in data
DataFrame API	Number of partitions Number of records per partitions
SparkLens	Tool that analyzes the application and suggests possible configuration parameters with values to get optimal performance that could be head start before further tuning

Areas	Note
Memory	Amount of memory for per executor might increase garbage collection Spark parameters spark.driver.memory spark.executor.memory
Processing	Increase in cores gives parallelism but might impact scheduling and additional shuffle Spark parameters spark.driver.cores spark.executor.cores spark.cores.max spark.executor.instances
Garbage Collection	Type of GC to use (ParallelGC, G1GC etc.) Configure each type of GC to get better performance Specify high level GC parameters like parallel threads, concurrent threads, heap occupancy percentage etc.

What is skew
	Most common problem, mostly happens during initial phase Extreme imbalance of work in the cluster Tasks within a stage take uneven amounts of time to finish
How to identify
	Stage/task wait time and execution time – spark UI Memory usage – Spark UI, yarn Check partition size – spark UI, yarn Hearbeat is missing by executor – spark log Partition size and records processed per partition – DataFrame API
How it could happen
	Data ingestion has skew in it where data is not partitioned properly. This will be propagated to other parts of the application
How to handle skew data
	Use a partition column in the source data (if possible) and use it to read – gives best performance as partitions don’t need to be created in processing time. If partition column cannot be added in source, create the partition column to dataset immediately after reading data Make sure the partition column has even distribution of data Partition column has to be numeric. Can be a composite column Spark parameters: partitionColumn lower/upperBound – used to determine stride numPartitions – max # of partitions Make sure stride is not skew (specially for JBDC load) Use hash + Mod function to have evenly distributed stride if some strides have uneven data volume Partition data at dataframe level – great for ‘narrow transformation’ To increase, use partition function To reduce, use coalesce for less data shuffling

What is cache/persist
	Persist – saves data in storage Cache – in memory persist Reuses DataFrame with transformations rather than building it from scratch Spark uses columnar format (allows scanning on required columns), compression to minimize memory usage and less pressure on GC
How to use cache/persist
	Select appropriate storage type as each type has it’s own performance implication Un-persist when done and don’t over persist Has impact on memory performance and possible slowdown Puts pressure on GC Without cache, DataFrame will be built from scratch if the same one is used in processing, saves read time, serialization time etc. Spark properties: spark.sql.inMemoryColumnarStorage.compressed spark.sql.inMemoryColumnarStorage.batchSize

Use broadcast join
	Broadcast puts a DataFrame to every executor before joining. Idea is to broadcast smaller dataset to all executors and use local copy of DataFrame to join with bigger dataset Spark 3.x has auto-broadcast join threshold to 10M, datasets smaller than that will be automatically broadcasted if not done explicitly in the code In broadcasting, spark uses broadcast hash join instead of sort merge join which triggers data shuffle and exchanges Code snippet: largeDF.join(broadcast(smallDF), largeDF.col(“col_name”) === smallDF.col(“col_name”)) If DataFrame is big: apply filters to keep the dataset as small possible
Dynamic partition pruning
	Out of the box functionality in spark 3.x Eliminates partition at the read time
Introduce salting to reduce skew

ParallelGC (spark’s default)	G1GC
Heap space is divided into young and old generation	Breaks heap into thousands of region
If minorGC occurs frequently: increase Eden and Survivor space	Recommended if heap size are large (>8GB) GC times are long
If majorGC occurs frequently: increase young and old space, decrease spark.memory.fraction	Lower latency traded with higher cpu usage for GC bookkeeping
If fullGC occurs before task finishes: increase memory, try cleaning young space by GC triggers
For large scale applications, setting below parameters help a lot -XX:ParallelGCThread=n -XX:ConcGCThreads=[n, 2n] -XX:InitialHeapOccupancyPercent=35