Interview Questions On Java,Java EE

BigData Apache Hadoop HDFS Interview QnA - II

noreply@blogger.com (Unknown) — Sat, 19 Sep 2020 02:03:00 +0000

This is the second post in continuation to my previous post on Apache Hadoop HDFS related interview questions. I have covered almost 16 questions in this post.

Q 1:How data or file is written into Hadoop HDFS?

A: In order to write a file in HDFS:

- A client needs a handle to master i.e. namenode (master).

- The namenode provides the address of the datanodes (slaves) on which client will start writing the data.

- The client directly writes data on the datanodes by creating data write pipeline.

Q 2: What should be the block size in Hadoop, ideally?

A: There is no hard rule in nailing down the block size in Hadoop. It boils down to what size is the input data. If input data is huge then the block size of 128/256 MB is good to have for optimized performance. While dealing with small files, small block sizes are recommended. This is done by using parameter dfs.block.size in order to override the default block size value.

Few points to remember:

- With larger block is being processed and some failure occur more work need to be done

- Fewer blocks if the block size is larger.

- Fewer blocks mean fewer nodes hence reduced throughput for parallel access

- Makes it possible for client to read/write more data without interacting with the Namenode, saving time.

-Larger blocks reduce metadata size of the Namenode, reducing Namenode load.

-Having fewer & larger blocks, also means longer tasks which in turn may not gain maximum parallelism

Q 3:What is Heartbeat in Hadoop?

A: In Hadoop, by using heartbeat, Namenode and datanode communicate. Heartbeat is a signal that is sent by the datanode to the namenode after the regular interval to time to indicate its presence, i.e. to indicate that it is alive.

- The default heartbeat interval is 3 seconds.

- If there is no heartbeat from DataNode to NameNode in ten minutes, then NameNode considers the DataNode to be out of service. This triggers blocks replicas hosted by that DataNode to be made unavailable.

- The NameNode then schedules the creation of new replicas of those blocks on other DataNodes.

Q 4:How often DataNode send heartbeat to NameNode in Hadoop?

A: The default is 3 seconds.

Q 5:How HDFS helps NameNode in scaling in Hadoop?

A: The primary benefit of Hadoop is its Scalability of cluster by adding more nodes.

Two types of Scalability in Hadoop: Vertical and Horizontal

Vertical scalability/Scale up:

- Increase the hardware capacity of the individual machine.

- Add more RAM or CPU to your existing system to make it more robust and powerful.

Horizontal scalability/Scale out

- Addition of more machines or setting up the cluster.

- Add more nodes to existing cluster and most importantly, you can add more machines without stopping the system, so no downtime or green zone, nothing of such sort while scaling out. So at last to meet your requirements you will have more machines working in parallel.

HDFS has two main layers:-

1. Namespace – manages directories, files and blocks. It supports file system operations such as creation, modification, deletion and listing of files and directories.

2. Block Storage – Block storage provides operations like creation, deletion, modification and getting the location of the blocks. It also takes care of replica placement and replication.

In architecture without HDFS Federation, datanode can be scaled both vertically & horizontally. But namenode was scaled only vertically not horizontally. This architecture has multiple datanodes, but it has only one NameNodefor (one namespace) for all datanodes .This limits the number of blocks, files, and directories supported on the file system.

In order to overcome this limitation HDFS Federation is introduced, wherein scaling the namenode horizontally is possible through use of multiple independent Namenodes/namespaces. In HDFS Federation, Namenodes do not require coordination with each other as the namenode is independent. All the datanodes are used as common storage for blocks by all the Namenodes.

Q 6:What is Secondary NameNode in Hadoop HDFS?

A: Secondary NameNode in hadoop is a specially dedicated node in HDFS cluster. It's main function is to take checkpoints of the file system metadata present on namenode. It is not a backup namenode. It just checkpoints namenode’s file system namespace. The Secondary NameNode is a helper to the primary NameNode but not replace for primary namenode.As the NameNode is the single point of failure in HDFS, if NameNode fails entire HDFS file system is lost. So in order to overcome this, Hadoop implemented Secondary NameNode whose main function is to store a copy of FsImage file and edits log file.

Q 7:Ideally what should be the replication factor in Hadoop?

A: The replication factor is the number of times Hadoop framework replicate each and every Data Block, in order to provide Fault Tolerance. The default replication factor is 3. It can be configured as per the requirement; can be increased (more than 3.) or decreased(less than 3) as per need.

Q 8:How one can change Replication factor when Data is already stored in HDFS

A:The replication factor can be set in the HDFS configuration file( hdfs-site.xml).It is used to set global replication factor for the entire cluster and only work on the newly created files but not on the existing files.

e.g.

hadoop fs –setrep –w 3 /file/filename.xml //Change the replication factor on a per-file basis :

hadoop fs –setrep –w 3 -R /directory/dir.xml//change the replication factor for files that already exist in HDFS.-R flag would recursively change the replication factor on all the files

Q 9:Why HDFS performs replication, although it results in data redundancy in Hadoop?

A: This is done in order to improve fault tolerance and downtime. In the process it improves overall data reliability.

Q 10:What is Safemode in Apache Hadoop?

A: A Safemoder in Hadoop is meant for NameNode wherein it does not allow to write to a file. During Safemode, the entire cluster remains in read only mode.

Q 11:What happen when namenode enters in safemode in hadoop?

A: When NameNode enters in SafeMode in Hadoop, no changes are allowed to file system. The file system is loaded from its last saved fsimage to memory and log file is edited.

Q 12:How to remove safemode of namenode forcefully in HDFS?

A: There is one command to leave SafeMode:

hadoop dfsadmin -safemode leave

Q 13:How to create the directory when Name node is in safe mode?

A: One has to come out of SafeMode, only then a directory can be created else error while creating a directory would keep popping up

Q 14:What is difference between a MapReduce InputSplit and HDFS block

A: HDFS Block is the physical representation of data in Hadoop. While MapReduce InputSplit is the logical representation of data present in the block in Hadoop. It is basically used during data processing in MapReduce program or other processing techniques.

Q 15:Explain Small File Problem in Hadoop

A:A HDFS block size is having default value as 64MB. HDFS is not designed to handle small files efficiently : it is primarily designed for streaming access of large files. Reading through small files normally causes lots of seeks and lots of hopping from datanode to datanode to retrieve each small file, all of which is an inefficient data access pattern

Q 16:How to create Users in Hadoop HDFS?

A: There are following steps involved in making users in Hadoop HDFS:

a. Create a user on your Host (client machine) and add it to "Hadoop" group

# useradd pogouser -G Hadoop

b. Create Home directory for this user on HDFS.

# su - hdfs -c "hdfs dfs -mkdir /user/pogouser"
# su - hdfs -c "hdfs dfs -chown newuser1:hadoop /user/pogouser"
# su - hdfs -c "hdfs dfs -chmod 755 /user/pogouser"

and then can work on hdfs like putting a file to HDFS.

# su - pogouser
# hdfs dfs -put /etc/passwd /user/pogouser
# hdfs dfs -cat /user/pogouser/passwd

Big Data - Apache Hadoop HDFS Interview Questions And Answers

noreply@blogger.com (Unknown) — Mon, 13 Jan 2020 16:25:00 +0000

In first post on Apache Hadoop technology, it was briefly touched upon what Apache Hadoop technology is, what is its use, what are its constituents. In this blog, main focus is on HDFS(Hadoop Distributed File Systems) related interview questions which are most frequently asked. HDFS is the main storage for large datasets and forms a key part of Hadoop Architecture. HDFS handles streaming data and running clusters on the commodity hardware. In this first list of questions on HDFS, the focus is to provide QnA in one single post in an effective manner. Please share your queries in comments section and if you like the post, please share it with like minded friends of yours.

1) How is HDFS different from Network Attached Storage(NAS)?
Answer:
NAS is computer level data storage server which requires dedicated hardware. HDFS is a distributed filesystem(in form of data blocks) using clusters of machines having local hard drives. It is cost effective as compared with NAS.

HDFS is designed to work with MapReduce system for analysis as computation are move to data. NAS is unsuitable for MapReduce as data is stored separately from the computations. In NAS, data is stored first and then on top of that analysis/computations are done. It gets really difficult to work on NAS as data grows exponentially.

As HDFS is distributed and clustered, it has high availability and reliability as compared with NAS.

2) What are NameNode and DataNode in HDFS? How are they different from each other?
Answer:

NameNode is the master node which primarily stores metadata information of data blocks like location of the block, name of the block, replication factors etc. It regulates data access for actual file for end clients. It also assigns work to slaves(i.e. DataNode). NameNode is responsible for a faster retrieval of main data and must be stored/hosted on a reliable hardware

DataNode is the slave node/worker node in Hadoop cluster which actually stores business data. As instructed from Master, DataNode performs creation/replication/deletion of data blocks. DataNode requires huge amount of storage for its operation and commodity hardware can be used for hosting it.

3) What is metadata in Hadoop?
Answer:
Metadata means data about data. In Hadoop, both NameNode and DataNode store metadata information. The Master node(NameNode) contains HDFS state namespace(metadata of all files and directories, information about blocks(name, location etc.)) and transaction information(edit logs). Slave nodes contain checksum(.meta file) metadata for each and every particular block on slaves nodes.

4) How NameNode handles DataNode failures in HDFS?
Answer:
HDFS follows master-slave architecture wherein master is NameNode and slave is DataNode. There is just one NameNode containing metadata. The multiple numbers of DataNodes are linked to the NameNode which actually is responsible for storing actual data in HDFS.

Each DataNode sends Heartbeat, every 2 minutes, implying that DataNode is functioning properly. It means all data blocks are getting reported to NameNode.

When NameNode does not receive heartbeat signals from Data node, it assumes that the data node is either dead or non-functional.When DataNode is declared dead/non-functional, all the data blocks it hosts are transferred to the other data nodes with which the blocks are replicated initially.

This is how DataNode failures are handled by NameNode.

5) How much Metadata will be created on NameNode in Hadoop?
Answer:
NameNode(NN) metadata stores:

- file to Block mapping
- locations of blocks on DataNodes
- active data nodes
- a bunch of other metadata

This is all stored in memory on the NN.

The things stored on disk are:
- fsimage
- edit log
- status logs.

a) fsimage – An fsimage file contains the complete state of the file system at a point in time. Every file system modification is assigned a unique, monotonically increasing transaction ID. An fsimage file represents the file system state after all modifications up to a specific transaction ID.

b) edits – An edits file is a log that lists each file system change (file creation, deletion or updation) that was made after the most recent fsimage.

When a file is put into HDFS, it is split into blocks (of configurable size).

Let’s say we have a file called “file.txt” that is 1GB (1000MB) and our block size is 128MB. We will end up with 7 128MB blocks and a 104MB block. The NameNode keeps track of the fact that “file.txt” in HDFS maps to these eight blocks and three replicas of each block. DataNodes store blocks, not files, so the mapping is important to understanding where our data is and what our data is.

Corresponding to a block 150 bytes (roughly) of metadata is created, Since there are 8 blocks with replication factor 3 i.e. 24 blocks. Hence 150×24 = 3600 bytes of metadata will be created.

On disk, the NameNode stores the metadata for the file system. This includes file and directory permissions, ownerships, and assigned blocks in the fsimage and the edit logs. In properly configured setups, it also includes a list of DataNodes that make up the HDFS (dfs.include parameter) and DataNodes that are to be removed from that list (dfs.exclude parameter).

6) What is Safe Mode and when does NameNode enter in Safe Mode?
Answer:
Safemode is a maintenance state of NameNode. In Safemode, a NameNode doesn’t allow any modifications to the file system, HDFS cluster remains in read-only state and doesn’t replicate or delete data blocks.

When a HDFS cluster is unstable, modifications in datablocks are prevented by making HDFS Safemode. So no deletion or replication of data happens over data blocks.

Status of Safemode :
hadoop dfsadmin –safemode get

To Manually enter Safemode:
hadoop dfsadmin –safemode enter

To Exit Safemode:
hadoop dfsadmin -safemode leave

7) How to restart all the daemons in Hadoop HDFS?
Answer:

start-all.sh & stop-all.sh : These commands are used to start and stop Hadoop daemons all at once. These commands are deprecated.

8) What are the modes in which Apache Hadoop run?
Answer:
There are three modes in which Apache Hadoop can run:

a. Standalone(Local) Mode
It is default mode under which Hadoop is configured to run with no distributed mode. Runs as a single Java process. This mode utilizes the local file system. This mode useful for debugging and usually the fastest mode in Hadoop.

b. Pseudo Distributed Mode(Single node)
Runs on a single node in a Pseudo Distributed mode wherein each daemon runs on seperate java process. Custom configuration is required( core-site.xml, hdfs-site.xml, mapred-site.xml ). Here HDFS is utilized for input and ouput. Useful for testing and debugging purposes.

c. Fully Distributed Mode
Production mode of running Hadoop wherein typically one machine in cluster is designated as NameNode and another as Resource Manager exclusively. These are masters. All other nodes act as Data Node and Node Manager which are the slaves. Configuration parameters and environment need to specified for Hadoop Daemons.Key features of this mode are fully distributed computing capability, reliability, fault tolerance and scalability.

9) On what basis name node distribute blocks across the data nodes in HDFS?
Answer:
Hadoop Data Blocks distribution strategy across clusters is having trade offs like data reliability, write and read bandwidth.The basic placement policy places the first block on the client.The second replica is placed on a different rack than the first one (also known as off rack) while third replica is placed on the same rack as the second one but in a different datanode. Further replicas are placed in random nodes avoiding same rack placement.

Hadoop also has a Balancer daemon. It distributes blocks by moving them from over-utilized datanodes to under-utilized datanodes while keeping the basic placement policy in mind.

10) What is a block in HDFS and why it has a size of 64MB?
Answer:
In Hadoop, large sized files are broken down into smaller chunks known as data blocks. HDFS Data blocks are the smallest unit of data. These data blocks are sequentially stored on a physical disk that makes reading and writing a block fast. Lets say you have a 128Mb file. With a 4k block size, you'd have to make 32k requests to get that file (1 request per block). In HDFS, each request has to be processed by the Name Node to figure out where that block can be found. That's a lot of traffic! If you use 64MB blocks, the number of requests goes down to 2.

11) What is Fault Tolerance in Hadoop HDFS?
Answer:
HDFS is highly fault tolerant. It handles faults by the process of replication. firstly data is broken down into blocks and these blocks are distributed across different machines present in the cluster. After this, replica of each block is created on other machines present in the cluster. By default HDFS creates 3 copies of a file on other machines present in the cluster. So due some reason if any machine on the HDFS goes down or fails, then also user can easily access that data from other machines in the cluster in which replica of file is present. Hence HDFS provides faster file read and write mechanism, due to its unique feature of distributed storage.

12) Why is block size set to 128 MB in HDFS?
Answer:
The default size of a block in HDFS is 128 MB (Hadoop 2. x) and 64 MB (Hadoop 1. x). The reason of having block size of 128MB is same as mentioned in Question#12 given above.

13) What happens if the block on Hadoop HDFS is corrupted?
Answer:
Though HDFS is quite robust when it comes to fault tolerance, yet there are cases when HDFS corrupts files then there are two ways to rectify this situation:

a. HDFS fsck : HDFS fsck command(run "hdfs fsck") is an offline process that examines on-disk structures.This command operates only on data(DataNodes in HDFS case) and determines which files contain corrupt blocks, and gives you options about how to fix them.

b. NameNode recovery : HDFS is highly tolerant and available due to replication process of data blocks which are distributed in cluster. If there is "x" replication factor then there are "x-1" replicas of a data block. Usually HDFS does auto correction by identifying the corrupt blocks, replacing them with good blocks and maintains the replication factor.
Sart the NameNode with the -recover flag to activate recovery mode, like:
./bin/hadoop namenode -recover

14)How data or file is read in Hadoop HDFS?
Answer:
Data or file read in HDFS is a sequential process. There are following steps involved in "read" operation:

a. A client initiates read request by calling 'open()' method of FileSystem object; it is an object of type DistributedFileSystem.

b. This object connects to namenode using RPC and gets metadata information such as the locations of the blocks of the file.

c. In response to this metadata request, addresses of the DataNodes having a copy of that block is returned back.

d. Once addresses of DataNodes are received, an object of type FSDataInputStream is returned to the client. FSDataInputStream contains DFSInputStream which takes care of interactions with DataNode and NameNode. A client invokes 'read()' method which causes DFSInputStream to establish a connection with the first DataNode with the first block of a file.

e.Data is read in the form of streams wherein client invokes 'read()' method repeatedly. This process of read() operation continues till it reaches the end of block.

f.Once the end of a block is reached, DFSInputStream closes the connection and moves on to locate the next DataNode for the next block

g.Once a client has done with the reading, it calls a close() method.

More questions will follow in my subsequent post.

What is the difference between AI(Artificial Intelligence) and ML(Machine Learning)?

noreply@blogger.com (Unknown) — Sun, 29 Dec 2019 09:56:00 +0000

Most of the times Artificial Intelligence(AI) and Machine Learning(ML) are used interchangeably but they are two distinct terminologies.

AI is an umbrella term under which ML lies. AI is a broad concept which focuses on building smart/intelligent machines/robots/software systems.

ML is a process by which a machine auto trains itself for achieving a certain objective over datasets. While going through multiple iterations, machines get more and more refined intelligence, in an autonomous fashion. Machines learn and improve its accuracies eventually. ML allows machine to recognize patterns and make predictions.Some real world examples of ML are speech recognition/conversational technologies, language translations, object recognition and other tasks.

There are underlying concepts like Automated Speech Recognition(ASR) and Natural Language Processing(NLP) in AI which overlap with ML.

Deep Learning(DL) is a subset/class of ML algorithms which involves ANN(Artificial Neural Networks). Learning can be supervised, semi-supervised or unsupervised. DL concepts can be applied in image processing, speech recognition, designing drugs, board games, machine translations etc.

The intent of this post is to pick these really complex topics at a very high level rather deep diving each one of them.

What are some of the common mistakes Java developers, especially beginners make?

noreply@blogger.com (Unknown) — Wed, 25 Dec 2019 05:56:00 +0000

Some of the most common mistakes that are committed by Java developers, especially who are newbies, are enumerated as given below:

Not writing thread safe Java program
A code is called a thread safe when a thread is modifying or reading shared data, no other thread can access it in a way that changes the data. So, when code depends upon a certain order of execution for correctness then other synchronization techniques are to be used.

Poor exception handling
While one throws top level exception object, it becomes difficult to trace to exact root cause of any problem in code, at every first instance. A good coder takes care of throwing explicit exception by proper handling of same.

Poor indentation, formatting and documentation of code being done leading to poor maintainability
A well indented code with proper documentation helps in better maintainability.

Hard coding credentials
A very commonly done mistake by beginners. The credentials must be handled in an independent manner from the code, viz. configuration files having them in encrypted format

Failing to modularize code
Do not make your code look like a procedural programming one. A better modularized code implementing Object Oriented programming concepts is the best way.

Lack of code optimization practices
It is understood for a newbie to be oblivious of code optimization techniques but being conscious of same from early days go a long way.

The most prominent example is when excessive garbage allocation is done. While creating many short-lived objects the garbage collector executes continuously, it degrades application performance. An example is of use of String objects which are immutable, each time a different String object is created for storing a new space in memory. The solution to deal it is to use mutable alternatives when instancing new objects.

Proper allocation of heap size: Start with minimum heap size and gradually increase the size as the need multiplies. This mechanism will keep a tab on speed of execution of code.

Better concurrency control techniques help in running multiple programs in parallel efficiently. It is advised to use optimistic locking with detached entities or an

EXTENDEDPersistence
Context

What does it take to be a fullstack Java developer?

noreply@blogger.com (Unknown) — Sat, 21 Dec 2019 07:30:00 +0000

A frequent need in modern professional Java technology world is a full stack Java developer. All technology companies are bullish on full stack developers in order to save big bucks. How can one become a full stack Java developer, a question which lingers with who has just started get his/her feet wet in the technology.

The answer is straightforward : one has to look after all technical building components which are responsible for making an enterprise level solution i.e. presentation layer, application logic layer/server side programming for application logic and data logic layer.

A fair hands-on knowledge of following pieces in order to be a full stack Java developer:

1. Web Front-End: HTML, CSS, JavaScript. Learn about responsive UI and UI frameworks. Learn JavaScript based MVC frameworks like AngularJS.

Another one is Bootstrap Javascript which is a free and open-source CSS framework, containing CSS- and JavaScript-based design templates(optional) for typography, forms, buttons, navigation and other interface components. It is used for responsive, mobile-first front-end web development.

2. Server Side programming: which provides Java Enterprise frameworks for Web development (Servlets, JSP, JSF) and/or other Java frameworks like Spring, EJB.

3. Data Logic Layer End: Frameworks like Java ORM Frameworks (Hibernate, JPA) or create your own database ecosystem using helper classes with the DAO Pattern. JDBC fundamentals and its practical usage must be known.

The idea is to leverage multiple skills from a single individual for efficiency(cost, time, quality) and get work done on lean principles.

It in not necessary that full stack developer skills should be the ultimate goal of one's tech journey rather attaining multiple skills with fair level of competency and flexibility in breadth of jobs and roles available.

How Machine Learning works with Python?

noreply@blogger.com (Unknown) — Sat, 14 Dec 2019 02:07:00 +0000

Machine learning is a subset of Artificial Intelligence wherein a machine is made to learn with a set of data to achieve a specified objective. In this whole process, it is very important the data integrity is kept to the best possible quality level as output of machine is primarily dependent upon that. Machine learning has intersection with Mathematics and Psychology. In programming world there are many languages through which Machine learning objectives can be developed, viz. C, R, Prolog, Java and Python.

Over the years, Python as a programming language has gained lot of popularity due to its high flexibility, simplicity and multi purpose uses.Python contains special libraries for machine learning namely scipy and numpy.

NumPy is a fundamental package for scientific computing with Python. It is a general-purpose array-processing package which provides a high-performance multidimensional array object, and tools for working with these arrays.

The key features contained are:

Useful linear algebra, Fourier transform, and random number capabilities

A powerful N-dimensional array object

Sophisticated (broadcasting) functions
Tools for integrating C/C++ and Fortran code

These are great for linear algebra and getting to know kernel methods of machine learning. The language is great to use when working with machine learning algorithms and has easy syntax relatively. For beginners, this is the best language to use and to start with.

Why DevOps?

noreply@blogger.com (Unknown) — Sun, 15 Sep 2019 06:38:00 +0000

Top five reasons that come to my mind, why one should adopt for DevOps

Faster response to resolve an issue in production affecting customers' businesses as a one team (for development and operations silos perspective)
Cost effective way to build softwares with better quality by eliminating waste in the system consistently
Better quality and frequency of communications cross-functionally
Building competencies effectively across the teams to bridge silos-working. It is a continual process
Value to customer, faster and with quality outcomes

Do you think there are more critical ones, please highlight in comments. Additionally if you want more detailing on any of the above points, shout out, shall try elaborating that.

What is Apache Hadoop framework and its usage?

noreply@blogger.com (Unknown) — Sun, 15 Sep 2019 06:12:00 +0000

Apache Hadoop is an open source platform which provides data processing of distributed, large and scalable datasets in a reliable manner using simple programming models. It is built upon clusters of computers hence does data processing in a cost effective fashion. The data can be structured, semi structured or completely unstructured. These features make Apache Hadoop an ideal candidate for making large data lakes supporting big data analytics.

Apache Hadoop framework is consisted of the following modules:

a. Hadoop Common – it contains libraries and utilities needed by other Hadoop modules

b. Hadoop Distributed File System (HDFS) – it is a distributed file-system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster;

c. Hadoop YARN – it is a platform responsible for managing computing resources in clusters and using them for scheduling users' applications

d. Hadoop MapReduce – it is an implementation of the MapReduce programming model for large-scale data processing.

The intent of this framework is to distribute large chunks of data in different computers that are part of cluster and also handle any hardware failures in an automated fashion.

From Hadoop's usage/usecases perspective, it has an ability to provide an access to real time large datasets for making data driven decisions for your data scientist, line of business (LOB) owners and developers. Hadoop is helping in data science growth which combines machine learning, statistics, advanced analysis and programming. Hadoop optimizes and streamlines costs in enterprise data warehouse by moving “cold” data not currently in use to a Hadoop-based distribution.

What is Big Data Analytics?

noreply@blogger.com (Unknown) — Fri, 19 Apr 2019 15:25:00 +0000

Big Data, as the name suggests, refers to large and complex data sets which may be unstructured, semi structured or structured which are quite challenging to deal with traditional data-processing application software. These traditional database application software will not be able to capture, extract, analyze and process within tolerable time limits. In Big data, the main focus is on unstructured data and constantly changing data size from few dozen terabytes to many exabytes of data.

Big Data requires specific set of technologies and analytical methods to transform/make sense("value") out of such a high volume, high frequency of updated data(velocity) in various formats/variety(viz. numeric, textual or images/videos) having essential data quality(veracity). The objective is to derive some meaningful information from large data sets using advance tools/technologies.

Short Q&A for Python - II

noreply@blogger.com (Unknown) — Sat, 01 Dec 2018 01:34:00 +0000

Q1. What are four main types of namespaces in Python?
A: The four main types of namespaces in Python are defined as below:

Global,
Local,
Module and
Class namespaces.

Q2. Explain the difference between local and global namespaces.
A: The local namespaces are created within a function, whenever that function is called. Global name spaces are created when the program starts.

Q3. While defining a class, what does the __ init_O function do?

A:__ init_O function overrides any initialization from an inherited class, and is called when the class is instantiated.

Q4. What is a lambda operator?
A: A lambda operator is used to create anonymous functions and mostly used in cases where one wishes to pass functions as parameters. or assign them to variable names.

Q5. How does a function return values?
A: Functions return values using the return statement.

Q6. When would you use triple quotes as a delimiter?
A:Triple quotes ‘’”” or ‘“ are string delimiters that span multiple lines in Python. Triple quotes are usually used when spanning multiple lines, or enclosing a string that has a mix of single and double quotes contained therein.

Q7. What happens when a function doesn’t have a return statement? Is this valid?
A: Yes, it is valid. The function will return a None object. The end of a function is defined by a block of code being executed not by any explicit keyword.

Q8. Explain the use of break and continue in Python looping.
A: The break statement stops the execution of the current loop. and transfers control to the next block. The continue statement ends the current block’s execution and jumps to the next iteration of the loop.

Q9. When do you use a continue statement in a for loop?
A: The intent of continue statement is, “I’m done processing this specific item, move on to the next item.” So the moment when processing of a particular item is complete; in order to move on to next, without executing further processing in the block.

Q10. When do you use a break statement in a for loop?
A: Whenever in a loop, the purpose of looping is served, break statement is used. For example. after finding an item in a list searched for, there is no need to keep looping. The moment the break statement is encountered, moving on to the next block of code takes place.

Q11. How do you use GUI that comes with Python to test your code?
A:That is just an editor and a graphical version of the interactive shell.There is no automated testing.

Q12. Why is that none of my threads are not running? How can I make it work?
A: The moment main thread exits, all threads are killed. When main thread runs too quickly, gives threads no time to do any work.
A simple fix is to add a sleep to the end of the program that’s long enough for all the threads to finish.

Short questions and answers on Python - I

noreply@blogger.com (Unknown) — Tue, 20 Nov 2018 10:27:00 +0000

In 1991, Guido van Rossum released first edition of Python programming language which is a high level programming language that got popular gradually for many mission critical projects ever since its inception. Python has a dynamic type system and automatic memory management. It is supported within different operating systems through its interpreters. It supports multiple programming paradigms, including object-oriented, imperative, functional and procedural.

In this post and more such posts(shall keep list of questions to 10 per post so that readability is eased out), I tried compiling questions which have short answers attached to them in world of Python programming language. I hope Python enthusiasts shall find these interesting and in case you have any feedback or more questions to post, please let me know in comments section.

Q1. What are the key features of Python and what are the differences between Python and Java?
A: Key features of Python are listed as given below:
a. It supports multiple programming paradigms, including object-oriented, aspect-oriented, imperative, functional and procedural.
b. Uses dynamic typing and a combination of reference counting and a cycle-detecting garbage collector for memory management.
c. Python is designed to be highly extensible
d. A simpler, less-cluttered grammar. There is only one way and its philosophy says "there should be one—and preferably only one—obvious way to do it"
e. Interpreted language which has different OS dependent interpreters.
f. Open source technology
g. Large standard library
h. GUI can be developed using Python

Differences with Java:
a. Python is dynamically typed while Java is statically typed. In Java, it is imperative to declare objects explicitly and if one tries to assign them a wrong type then a typecasting error is thrown. In case of Python, an object can be of any type and can be assigned to any type dynamically.

b. Python is crisp and brief in expression while Java is verbose. It means same quanta of work will require more lines of codes in Java as compared to Python.

c. Python is easy to read as compared to Java

d. When it comes to speed Java is faster. As Python is interpreted, so it runs slower than its' counterparts.

e. From portability perspective, Java edges over Python as JVM is widely present.

f. In enterprise level critical data intensive applications, Java provides more robustness when it comes to handling database related operations than Python.

Both languages have their own benefits. It all boils down to what you choose for your project. Python is simple and concise whereas Java is fast and more portable. Python is dynamically-typed while Java is statically-typed.Both are powerful in their own areas, what is your decision while selecting them, is driven by your need and wish.

Q2. How is Python executed?
A: The execution of a Python program results in execution of the byte code on the Python Virtual Machine (PVM). Every time a Python script is executed, script converts into byte code. When a Python script is imported as a module, the byte code will be stored in the corresponding .pyc file which would need PVM for its execution.

Q3. What is the difference between .py and .pyc files?
A: When a Python module is loaded, the .py file is "byte compiled" to .pyc files. This is done to load module faster rather run faster. The timestamp is recorded in pyc files.

Q4. How do you invoke the Python interpreter for interactive use?
A: Python has two basic modes: script and interactive. In normal mode, the scripted and finished .py files are run in the Python interpreter.

Interactive mode is a command line shell which gives immediate feedback for each statement, while running previously fed statements in active memory. Interactive mode is a good way to play around and try variations on syntax.

Q5. How do you execute a Python Script?
A: You can simply call from command prompt:

python /filename.py

Please make sure python runtime is included in system PATH variable.

Q6. Explain the use of try: except raise, and finally:
A: In Python programs, exception handling is done through try, except, raise and finally statements. In try block of code, that snippet is written which is expected to throw an exception or error and the code that handles exception is written in except clause. The exceptions are raised when corresponding errors occur at run time, but we can forcefully raise it using the keyword raise. Every try statement in Python may have an optional finally clause which is executed no matter what, and is generally used to release external resources.

Q7. What happens if an error occurs that is not handled in the except block?
A: If an error happening in try block is not handled in except block, such a code may create problems in production environment(in severe mishandling situations, may crash production system too if error creates memory intensive leakages) since it is not properly tackled what to do when a certain error happens.

Q8. How are modules used in a Python program?
A: A module in Python is a file containing Python code. It can define functions, classes and variables. One can import module by using keyword called "import". Python has many standard, pre defined modules too which can be utilized as per need.

Q9. How do you create a Python function?
A: Keyword "def" marks the start of function header.

def function_name(parameters):

"""string to define what function does"""
statement(s)
One can call the function by just simply invoking it by name.

function_name(parameters)

Q10. How is a Python class created and how is it instantiated?
A: In python a class is created by the keyword 'class' while object is created using the constructor of the class. This object will then be called the instance of the class. In Python we create classes in the following manner.

class ClassName:

.
.
.

Instance of the class can be invoked something similar to given below:

instance = ClassName(arguments)

More questions on Python shall be posted in subsequent posts so keep checking this blog in near future.

Please share your comments or queries, if you have any.

Short questions and answers on PHP

noreply@blogger.com (Unknown) — Sun, 11 Nov 2018 16:17:00 +0000

PHP acronym stands for Hypertext Preprocessor which is an open source scripting language which is suited for web development and embedded in HTML. Its syntax resembles with Perl and C. The recommended versions of PHP are 7.1 and 7.2.
A very basic PHP sample is given as below:

Unlike any client side scripting language, PHP is executed at server side, generating HTML sent to client side. It offers many features and very simple to learn.

Here goes a list of Q&A on PHP:

Q1: What can PHP do?
A: PHP can be used for:
a. Server-side scripting for which you may need the PHP parser (CGI or server module), a web server and a web browser.
b. Command line scripting to run it without any server or browser.
c. Writing desktop applications wherein some advanced PHP features in client-side applications can be used using PHP-GTK
d. PHP is used on all major operating systems, e.g. Linux, many Unix variants (including HP-UX, Solaris and OpenBSD), Microsoft Windows, macOS, RISC OS, and probably others
e. One can output images, PDF files and even Flash movies (using libswf and Ming) generated on the fly, any text, such as XHTML and any other XML file.
f. It supports a wide range of databases. Using one of the database specific extensions (e.g., for mysql), or using an abstraction layer like PDO, or connect to any database supporting ODBC extension.
g. PHP supports talking to other services using protocols such as LDAP, IMAP, SNMP, NNTP, POP3, HTTP, COM (on Windows) and countless others. You can also open raw network sockets and interact using any other protocol.
h. PHP has useful text processing features like Perl compatible regular expressions (PCRE), and many extensions and tools to parse and access XML documents.

Q2: What’s the difference between the include() and require() functions?
A: The difference between include and require is when a file being included cannot be found: include will emit a warning ( E_WARNING ) and the script will continue, whereas require will emit a fatal error ( E_COMPILE_ERROR ) and halt the script.

Q3: How can we get the IP address of the client?
A: In general "$_SERVER['REMOTE_ADDR'];" is used in PHP to determine the IP address of the client.

Q4: What’s the difference between unset() and unlink()?
A: In PHP, unlink() is a function used for file system handling, primarily used to delete files.
unset() is a function for variable management in PHP. It will make a variable undefined/destroyed. unset() is used to null out the value of a given variable. It can be used to remove a single variable, multiple variables, or an element from an array.

Q5: What are the main error types in PHP and how do they differ?
A: Basically four types of errors in PHP, which are as follows:

a. Parse Error (Syntax Error): Unclosed quotes, Missing or Extra parentheses, Unclosed braces, Missing semicolon

b. Fatal Error: When asking PHP to do something which it is not designed to do e.g. try accessing undefined functions

c, Warning Error: It won't stop execution of a PHP script e.g. warning errors are to include a missing file or using the incorrect number of parameters in a function

d. Notice Error: Similar to warning errors, such as to include a missing file or using the incorrect number of parameters in a function

Q6: How can you enable error reporting in PHP?
A. There are two ways for viewing PHP errors.
1. display errors directly on your website (viewable from any web browser)
2. enable error logging to write the errors to a specified file (viewable inside a text file)

Q7: What’s the difference between using mysql_ functions and PDO?
A: mysql is deprecated function in PHP which provides an access to MySQL through a procedural way while PDO(PHP Data Object) is a general database abstraction layer with support for MySQL. PDO provides prepared statements and lot of flexibility while returning data.

Q8: Describe how inheritance works with PHP.
A: PHP uses inheritance principle in its object model.When one class extends another, the subclass inherits all of the public and protected methods from the parent class. They will retain their original functionality unless a class overrides those methods in parent class. PHP supports single inheritance and multi-level inheritance only.

Q9: Do you know what the PHP-FIG is? Describe it.
A: FIG stands for Framework Interoperability Group. The purpose of same is to increase collaboration among different stakeholders of projects with commonalities.

Q10: What are different PHP frameworks and their preferences of use?
A: The objective of PHP Frameworks is to provide a tool for building PHP based applications with agility. It helps in facilitating scalability and better control on maintenance. Some examples of PHP frameworks to help you build interoperable, agile applications are:
Symfony, CakePHP, Phalcon, Aura, Laravel, Fat-Free, Kohana

Q11: What are Traits?
A: Traits is a mechanism for code reuse in single inheritance languages like PHP. It reduces limitations of single inheritance by enabling a developer to reuse sets of methods freely in several independent classes living in different class hierarchies.

Q12: Can the value of a constant change during the script’s execution?
A: No, it cannot change during the execution of script.

Q13: Can you extend a Final defined class?
A: No. If a class is defined as Final, it cannot be extended. If a method is defined as final then it cannot be overridden in derived classes.

Q14: What are the __construct() and __destruct() methods in a PHP class?
A: Constructor an Destructor, like any other OOP language, in PHP5 can be defined using _construct() and _destruct() methods explicitly. Classes which have a constructor method call this method on each newly-created object.The destructor method is called as soon as there are no other references to a particular object.

Q15: How we can get the number of elements in an array?
A: Use the PHP count() or sizeof() function

Q16: What are the 3 scope levels available in PHP and how would you define them?
A: private: accessible within the class itself, public:accessible outside of the class, protected: accessible in parent and child classes

Q17: What are getters and setters and why are they important?
A: These are methods to declare and retrieve variable values. They play important role since they provide a central mechanism for a variable definition and retrieval.

Q18: What does MVC stand for and what does each component do?
A: Model View Controller: an application design pattern which separates data logic layer(model), presentation layer(view) and business logic layer(controller) from each other. It provides a segregation of responsibility for better code maintenance.

Q19: What are SQL Injections, how do you prevent them and what are the best practices?
A: SQL Injections are nefarious ways to modify database queries on the fly in such a way that it will make a database vulnerable to external threats. In order to prevent SQL Injection, one can use prepared statements and parameterized queries. In PHP, PDO(independent of underlying database) and MySQLi(specifically for MySQL).

Q20: Why would you use === instead of ==?
A: The detailed answer of same can be obtained in PHP Manual.

Q21: What are PSRs ? Why would you follow a PSR standard?
A: PSR stands for PHP Standard Recommendations, a PHP specification published by the PHP Framework Interop Group(PHP-FIG) similar to Java Specification Request(JSR) in case of Java. The objectives of PSRs are:
a. To serve the standardization of programming concepts in PHP
b. To enable interoperability of components
c. To provide a common technical basis for implementation of proven concepts for optimal programming and testing practices.

What are your key learnings while working in Scrum team?

noreply@blogger.com (Unknown) — Mon, 29 Oct 2018 03:11:00 +0000

In Scrum ways of working, it's always important to stay ahead of your game but when things go out of your control owing to the reasons you cannot do much about in such scenarios reorganizing your work as a team comes as a natural choice. Mostly things which did not go your way make you learn the most. In this post I'm writing down some general and most of the common learnings during scrum executions:

Don't work on fragile requirements as seen quite often, in immatured setup, customer keeps changing business requirements even when stories are in QA which results in rework and hence delays and frustrations within team. Start work once requirements are frozen and detailed enough. Any changes to once baselined business requirements must be dealt with a change request in subsequent sprints so as proper estimates can be arrived at.
Avoid too many Dev and QA reloops which results in waste of efforts. Both Dev and QA must have same understanding of requirements so that stories once in QA don't deviate from business requirements expectations which are established during requirements grooming stage. Integrate QA steps within sprints post Dev is done with their set of unit, function and regression testings with desired objectives. More rigorous and proactive testing mean better would be quality of shipped product.
Do all Scrum ceremonies without fail with the best industrial practices. Daily scrum meetings with Kanban boards, not more than 15 minutes. Respect of time by everyone is very important.
Daily updates into ALM tools must be religiously adhered, to get a grip on burndown happening as planned or not. The corrective measures can only be taken when data is correct and timely.
Scrum team members must have a right mindset towards ethos of Scrum. Every team goes through a process of forming, storming, norming and performing so building a solid team spirit stimulates productivity which happens through colocation and continuous communication. If teams are geographically distributed, relevant tools and business practices must be used to build effective collaboration.Do it with agility. Be critical to work at hand rather to people.
Scrum Master must quicken the turnaround time of blocker resolution which hampers team velocity. A timely decision must be arrived on stories that cannot be delivered due to blockers and communication must follow to relevant stakeholders.
Apply learnings observed during retrsperetros meetings in subsequent sprints with tangible goals explicitly defined. The repeat of certain steps if required must be made available to team through visual reminders readily accessible to them.

In case you have more learnings to add through your experiences, please share in comments on this post.

Short Q&A for Cloud computing

noreply@blogger.com (Unknown) — Sun, 28 Oct 2018 06:50:00 +0000

As cloud computing has matured over the years, business potential of same has become quite mainstream. During its inception years, data privacy protection was a major concern for most of the giant corporates who weren't willing to move their data in someone else's environment but gradually models of cloud transformation evolved and momentum has now become a necessity to cut down costs of ownership.
I wrote about a short and crisp introduction on what cloud computing is and why does one need it in one of my earlier posts:

What is cloud computing?

Now in this post, I shall be focusing on putting more short questions and answers that are frequently asked during preliminary phases of interview process. Hope you find them useful:

Q1: What is Infrastructure As A Service (IaaS)? Give some examples.

A: IaaS is one specific category of cloud service which provides instant virtualized computing resources(Servers and storage that are hosted in data center physical plant/building made accessible through networking firewalls/security) over the internet.One can quickly scale up and pay as per use of service. The cloud computing service provider manages the infrastructure. Different example of IaaS are Amazon Web Services (AWS), Microsoft Azure, Google Compute Engine (GCE),Cisco Metapod,Linode, Rackspace, DigitalOcean etc.

Q2: What is Platform As A Service (PaaS) and what are layers of PaaS? Give some examples.

A: PaaS is one specific category of cloud service which provides a platform(OS+Development tools over infrastructure(data-centers consisting of physical servers, network and security etc.)) which are available for a price over their web browsers for end users. Different examples of PaaS are AWS Elastic Beanstalk, Windows Azure, Heroku, Force.com, Google App Engine, Apache Stratos, OpenShift etc.

Q3: What is Software As A Service (SaaS)? Give some examples.
A: SaaS is one specific category of cloud service which provides software distribution model of third-party provided applications made accessible to customers over the Internet. SaaS runs on top of PaaS and IaaS combined. Different examples of SaaS are Google apps(Drive, Docs etc.), Dropbox, Cisco WebEx etc.

Q4: What are key differences in IaaS, PaaS and SaaS?

A: The summary of key differences(self explanatory) in IaaS, PaaS and SaaS can be summarized in table given below:

Q5: What are the basic characteristics of cloud computing?

A: There are following basic characteristics of cloud computing:

a. On-demand self-service: An end user can provision computing service automatically without requiring human interaction with each service provider.

b. Broad network access: services are available over the network and accessed through heterogeneous thin or thick client platforms (e.g., mobile phones, tablets, laptops, and workstations)

c. Resource pooling: Multiple consumers using a multi-tenant model can be serviced in provider’s pooled environment dynamically. A consumer remains oblivious of exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).Different resources include storage, processing, memory, and network bandwidth with different physical and virtual resources dynamically assigned and reassigned with demand.

d. Rapid elasticity: Scale up/down rapidly and in specific cases automatically.

e. Measured Service: Automatic control and optimize resources by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the provider and consumer of the utilized service.

Q6: What are the must essentials to be followed before going to cloud computing platform?

A: Essentials are:

Compliance issues
Data storage types
Data integrity adherence in the cloud
Ensuring availability and access
Protection from loss of data
Business continuity plan
Uptime – reduction of downtime

Q7: What are different layers in cloud architecture?

A: A cloud computing architecture is consisted of following components:

Front end: Fat client, thin client, mobile device
Back end: Servers, Storages
Cloud based delivery(IaaS, PaaS, SaaS)
Network(Intranet, internet, inter-cloud) and Security

Q8: What are the types of data used in cloud computing?

A: Emails, images, videos, contracts, blogs etc.

Q9:How can our company benefit from cloud computing?

A: The key cloud computing benefits are:

Cost savings/Reduced OpEx
Pay per use
User defined security model for cloud computing(Private, Hybrid, Public clouds)
Efforts and focus saved for more strategic initiatives of growth

Q10: How you can you best secure data while transporting in the cloud?

A: A strong data encryption technique is a way before moving data across networks to storage servers/devices to eliminate chances of data leaks.

Q11: Explain the Security management in terms of Cloud Computing.

A: There are two key pillars of security management of cloud computing:

Identity management: Identify and manage personnels' data who can access authorized cloud based services. Define rules for authentication for these authorized personnels.

Access management: Roles based access definition and management towards cloud based services

Q12: What is the difference between scalability and elasticity in cloud computing?

A: The scalability corresponds to growth seen in service capacity due to scale picked up by business demand while elasticity corresponds to flexiblity in system to commission or decommission resources capacity automatically/dynamically.

Q13: What are hybrid clouds?

A: A combination of public and private clouds is called hybrid cloud. It gives flexibility to organizations to decide what features or services they want to secure over others and take them to private cloud vs. rest to be put on public cloud.

Q14: What is the difference between cloud computing and mobile computing?

A: Mobile computing inherently uses cloud computing and depends on it for data migration.

What are different DevOps tools?

noreply@blogger.com (Unknown) — Fri, 26 Oct 2018 11:41:00 +0000

In journey of implementing DevOps culture in your work environment, it is quite an important step what kind of DevOps tools that you and your team pick. It is important to understand technical needs of your project may be it requires a petabyte-scale, data analytics infrastructure or specific architecture and processes drivers during development and operation phases. A great care must be done while selecting, benchmarking and constantly improving tool selection.

If you are a company that is building a solution on top of an open-source stack (ELK), teams have to get involved in the open source community, contributing to multiple projects while customizing tools to fit needs. DevOps tools can be classified in various categories based upon their specific features/usage during software development lifecycle management. The ones which are quite widely used in IT industry are enumerated as given below :

a. Build and Continuous Integration tools
Jenkins
First released in 2011, Jenkins is the leading open source automation server, which provides hundreds of plugins to support building, deploying and automating any projec. According to its own marketing, “can be used as a simple CI server or turned into the continuous delivery hub for any project”. E.g. Pipeline plugin means customers can implement their project’s entire build/test/deploy pipeline in a Jenkinsfile and store it alongside the code. It helps in continuous delivery and speeding deployment.There are other tools as well like Solano Labs and Gradle.

b.Infrastructure Performance and Monitoring tools
Nagios
There are many infrastructure monitoring solutions e.g. Zabbix,Nagios and many others open-source tools but Nagios is a stand out monitoring solution that is highly effective owing to its large community of contributors who create plugins for the tool. Nagios may not include all the abilities that are desired around the automatic discovery of new instances and services, so working these issues around with the community’s plugins help a great deal.

c.Version control tools
Github
If you have an excellent repository of source code then it will accelerate DevOps projects. Github is one with its effective search navigation and collaborative structure that provides a space for software developers to host their code online for free. It functions as a social site, where questions can be asked and code exchanges made. Github has a handy list of DevOps tools with one-line descriptions to help developers choose a relevant software. Each suggestion is colour-coded to indicate which language it is written in, saving search time.

d. Enterprise Container Platform
Docker
Docker, launched in 2013, is a popular enterprise platform provider, operating on Windows or Linux and compatible with any programming language. The toolkit will separate software, wrapping it in a file-system which facilitates automated deployments. It makes easier to host applications inside portable environments. Docker automates operating-system-level virtualization – and it’s integration-friendly, cooperating with IBM Cloud, AWS, Oracle Cloud, Azure and more. It is a powerful tool that promises to assist in transition to DevOps. It helps implementing the velocity, variation and visualization of systems thinking and amplifying feedback loops. It has a built-in container orchestration which uses clustering.It provides convenient integration with third-party systems with open APIs, plugins and drivers.

e.Software Configuration management tools
Software configuration management(SCM) is the task of tracking and controlling changes in the software. When something goes wrong then SCM can help determine what was changed and by whom. SCM practices include revision control and the establishment of baselines.

The SCM tools in recent times have become virtual appliances that can be instantiated as virtual machines.The tools can model and manage cloud-based virtual resources, including virtual appliances, storage units, and software bundles. Now developers are able to dynamically instantiate virtual servers and related resources.
Chef
Chef is a configuration automation tool which is quite popular in DevOps communities.As the environments managed grow, manual configuration and deployment practices can result in growing operational expenses at an alarming rate. With Chef, the infrastructure is defined as code, ensuring that configuration policy is flexible, versionable, testable and human readable. Servers managed by Chef are continuously evaluated against their desired state, ensuring that configuration drift is automatically corrected, and configuration changes are universally applied.
Ansible
Ansible is on of the simplest solution for configuration management available. It's designed to be minimal in nature, consistent, secure and highly reliable, with an extremely low learning curve for administrators, developers and IT managers.
Ansible configurations are simple data descriptions your infrastructure which are both human-readable and machine-parsable - ensuring everyone on team will be able to understand the meaning of each configuration task.It frees people of configuration management related tasks botheration and free their time for more strategic works. Ansible requires a password or SSH key in order to start managing systems. It can start managing them without installing any agent software, avoiding the problem of "managing the management" common in many automation systems.

f.Incident, change and problem management tools

Jira
Jira is was primarily an issue tracking software which helps DevOps team organized when it comes to dealing with issues. These days Atlassian has come up with plethora of Jira products meant for DevOps teams:

What is DevOps?

noreply@blogger.com (Unknown) — Wed, 24 Oct 2018 03:15:00 +0000

DevOps is a software development philosophy or a paradigm shift where development team and operation team break silos, collaborate better in order to ship a product or service to market in a very quick time. The main objective of DevOps is to shorten the lifecycle of development. As the name itself suggests it is a compound word coined by joining two words development and operation together.

Now question arises why DevOps? Weren't things functioning good enough before? The actual struggle arises when operation team, which is primarily responsible for a very high uptime of production environment, face sscaling of production environment. It brings challenges to operation guy with limitations of available tools efficiency to manage production environment better. They usually resist frequent changes to production to upkeep a stable environment for end customers and expect a development schedule in advance from development team so that they can manage changes better. If development team and operation team don't work in cohesion then it affects overall system effectiveness, efficiency and time to market of product/service.

DevOps propounds a change in mindset with new set of tools and skills. Both entities(Dev and Ops teams) must work together to obtain common goal - customer delight by breaking silos, thinking alike, proactive executions of duties and shared responsibilities to reach enhanced productivity.

DevOps promotes culture of automation in infrastructure, code, testing, workflows for continuous improvement in entire delivery lifecycle. The monitoring and measuring of system performance is also automated. The development of new features happen in chunks. It may span mostly once in 3 weeks or 4 weeks or may be days and hours based upon how much matured/robust overall ecosystem is for code deployments in production.

What is Kanban?

noreply@blogger.com (Unknown) — Tue, 23 Oct 2018 14:44:00 +0000

Kanban means signboard or billboard in Japanese. It is a scheduling system which is for lean manufacturing and just-in-time manufacturing (JIT). The father of Kanban is Taiichi Ohno, an industrial engineer at Toyota. He developed Kanban to improve manufacturing efficiency. It is also known as "Toyota nameplate system"

A goal of the kanban system is to limit the buildup of excess inventory at any point in production. Kanban helps in limiting the number of items waiting at supply points and then removing inefficiencies.

Kanban is used as one of the agile methodology for software development. Some key attributes are listed as given below:

Real time communication of capacity
Absolute transparency among teams
Building Kanban scheduling board as visually available tools for team, helping in better collaboration with transparency
Work in Progress (WIP) limits are set which define amount of work that can exist in each workflow.
Kanban canbe visualized with Jira software for efficient project development tasks definition, WIP limits help reducing overloading resources.

What is lean?

noreply@blogger.com (Unknown) — Sat, 20 Oct 2018 12:37:00 +0000

Lean is both a philosophy and discipline which focuses primarily on continuous improvement in process of providing value to customer. Before it was known as a business methodology it was more known in manufacturing world introduced by research team of Toyota deriving maximum possible value for end customer while reducing the waste of resources involved. The organized way of lean derive maximum customer value by:

- Eliminate waste(also known as Muda)
- Eliminate waste due to overburden/workload is too high(also known as Muri)

The continuous improvement(CI) is achieved with thorough examination of existing processes and a continuous search in process steps which are not adding any value to end customer. The people part is equally very important since process and people are intertwined and nothing concrete will come out from end customer perspective unless both work in absolute tandem continuously. Hence collaborative efforts to remove waste continuously is key to overall performance improvement. It aims for paradigm shift in overall organizational culture.

There are 8 types of wastes:

Defects
Over-Production
Waiting
Non-Utilized Talent
Transportation
Inventory
Motion
Extra-Processing

In world of softwares more than 50% of functionality are rarely or never used. On an average 45% of projects exceed budget, 7% runs over schedule and 56% delivers less value than expected.60%–90% of ideas do not improve the metric they were intended to improve. Imbibing lean principles with conscious efforts lead to value creation with a committed and self motivated teams.

What is Extreme Programming(XP)?

noreply@blogger.com (Unknown) — Thu, 27 Sep 2018 06:18:00 +0000

Extreme Programming (XP) is one of the agile methodologies which is focused on improving responsiveness to business requirement changes and software quality. It focuses on "releases" in short development cycles, improving productivity.
XP promotes pair programming focusing on extensive code reviews and unit testing. A flat management structure is another feature to cut down on bureaucracy for faster and efficient working. The basic idea is to take all relevant development steps to extreme level in order to improve overall quality of delivery whether it's development in pair, unit testing, automation of system test cases and their executions, test driven development. Acceptance testing is done and results are published and if bugs are found then test cases are written first surrounding bugs, essentially bugs are nothing but missed test cases.
XP only works on small teams of twelve or fewer people. This limitation can be overcome by breaking up project into smaller pieces and the team into smaller groups.
It promotes continuous design and release concept and can run efficiently and successfully with extremely commited senior technical worfkorce.

What is SCRUM?

noreply@blogger.com (Unknown) — Fri, 06 Apr 2018 09:04:00 +0000

SCRUM is an agile methodology/framework for software development. A SCRUM team is usually consisted of 3 to 9 developers who breaks down their work into timeboxed, usually called sprints of 2 to 3 weeks, user stories. The development is done in an iterative fashion in order to provide realized value/working product to end user in quick time.

There are typically three roles in SCRUM teams:

1. Product Owner(PO): Represents voice of customer and accountable that SCRUM team delivers value to the business. A product owner defines business requirements of a product in a breakdown fashion called a user story and collections of such user stories is called product backlog. These user stories are elaborated enough so that development team can start work on them during sprint execution. These user stories can be prioritized in a sprint during a sprint planning session. The product owner spends most of his time with business liaising with business stakeholders so that he bridges the gap from business functionalities perspective to development team. Product owner is a key communicator to business community whenever sprint review meetings are conducted with desired outcomes and also helps business stakeholders to understand the outcome of a sprint in case they are not able to make it to these meetings. A product owner also educates business on development process and challenges on their side. A product owner should not be taking any work of SCRUM Master who is primarily responsible for helping product owner in maintaining the product backlog in such a way that team can be continually working towards achieving business goals.

2. Development Team: Primarily responsible for development of shippable product incrementally. A team of 3 to 9 who is responsible for analysis, design, coding, unit testing, system and integration testing, building and deploying incremental shippable product. It is a self-organizing and mature setup of delivery team members who adhere to ethos of SCRUM methodology.

3. SCRUM Master(SM): SM is a facilitator. The primary job to remove any kind of blockers/impediments at the earliest so as development team works seamlessly to deliver shippable product incrementally. SM is not to be confused with a Project Manager or a typical team lead. SM is responsible for:
- Helping/Coaching development team to stick to SCRUM ethos and values
- Helps team to self organize, defining the definition of DONE for user stories
- Helps PO so as product backlog is loaded with user stories that are detailed enough to be developed when sprint is in execution.
- Eliminating blockers/impediments(internal/external)
- SCRUM ceremonies execution(Sprint Planning, Daily SCRUM, Product/Sprint Review Meeting, Retrospective and tracking actions thereof)

Core Java Interview Questions

noreply@blogger.com (Unknown) — Wed, 03 Jan 2018 16:38:00 +0000

Every year many thousand technocrats choose Java careers across the globe,after formally taking Java training classes,courses and/or Java certifications like OCJP etc.While appearing for technical interviews,these budding Java programmers face variety of technical questions, are asked to write code snippets to prove their mettle in relevant topics of Java.They are not only assessed on language basics but also on their familiarity with Java development tools, coding standards and guidelines,optimization of Java code,load tests,securing code through Java obfuscation and many more. It is their performance in these interviews which decides their career fate and is highly dependent on to-the-point and not too verbose answers.

Apart from technical questions, an interviewee faces a spate of personal questions as well.In post, Personal Interview Questions you will find variety of questions asked on personal front.In wired world, it is very important to know where your resources are lying and how credible are they.

This blog intends to provide a consolidated knowledge base for all netizens,budding Java Engineers or experienced ones, interested in Java technology.This blog will primarily focus on 'Core Java' related questions and later on advanced topics in Java EE space,Open source technologies and frameworks and more.At this point this is a nice idea to put a roadmap right away to put all these questions in an organized fashion. In order to access questions list on a particular topic,please click the links associated with it.It will open the questions list in the same browser window and in order to comeback to this page, while going through various posts, you have to click back button in your bowser window.Each answer for a question on various topics will open in a fresh window,while you are through with reading an answer,close the open window in order to comeback to the questions list.If you want to access Master List of Core Java Interview Questions then click here.

Java Language Fundamentals (Click Link To Access Questions' List)

Packages and their classes (Click Link To Access Questions' List)

Java and Database Access
JFC Swing
Collection APIs

Check more miscellaneous short questions on Core Java with short answers,here.

If you starve for more knowledge on Core Java,you may like to check out online e-books and other resources on Core Java or access all Online Java and Java EE resources. All suggestions,comments are welcome in order to make this space more productive and useful for all its audiences.

Read more on Core Java:
- Short Questions On Core Java
- Interview Questions on Java 5
- Enhancements in Java 7

Miscellaneous Core Java Questions With Short Answers

noreply@blogger.com (Unknown) — Tue, 02 Jan 2018 16:43:00 +0000

In a Java program, how can you divert program messages to the system console, but error messages, say to a file?

A. The class 'System has a variable out that represents the standard output, and the variable err that represents the standard error device. By default, they both point at the system console. This how the standard output could be re-directed:
```
Stream stream = new Stream(new FileOutputStream("error.txt"));
System.setErr(stream);
System.setOut(stream);
```
How do you know if an explicit object casting is needed?

A. In order to assign a superclass object to a variable of a subclass,one needs to do explicit casting. For example:
```
Person person;
Man man;
man = (Man)person;
```
While automatic casting happens when you typecast a subclass object as parent class object.

What's the difference between the methods sleep() and wait()

A. The code sleep(1000); puts thread aside for exactly one second. The code wait(1000), causes a wait of up to one second. A thread could stop waiting earlier if it receives the notify() or notifyAll() call. The method wait() is defined in the class Object and the method sleep() is defined in the class Thread.

Why would you use a synchronized block vs. synchronized method?

A. A synchronized blocks place locks for shorter periods than synchronized methods.

Can you write a Java class that could be used both as an applet as well as an application?

A. Yes. Add a main() method to the applet.

Can you call one constructor from another if a class has multiple constructors

A. Yes. Use this() syntax.

How will you convert a String array to an ArrayList object?

A.

String[] stringArray = new String[] {"x", "y", "Z"};
List list = Arrays.asList(stringArray);

Does it matter in what order catch statements for FileNotFoundException and IOExceptipon are written?

A. Yes, it does. The FileNoFoundException is inherited from the IOException. Exception's subclasses have to be caught first.

Can an inner class declared inside of a method access local variables of this method?

A. It's possible if these variables are final.

What can go wrong if you replace && with & in the following code:

String a=null;
if (a!=null && a.length()>10) {...}

A. A single ampersand here would lead to a NullPointerException.

When should the method invokeLater()be used?

A. To ensure that Swing components are updated through the event-dispatching thread.

What's the difference between a queue and a stack?

A. Stacks works by last-in-first-out rule (LIFO), while queues use the FIFO rule

You can create an abstract class that contains only abstract methods. On the other hand, you can create an interface that declares the same methods. So can you use abstract classes instead of interfaces?

A. Sometimes. But your class may be a descendant of another class and in this case the interface is your only option.

If you're overriding the method equals() of an object, which other method you might also consider?

A.hashCode()

You are planning to do an indexed search in a list of objects. Which of the two Java collections should you use: ArrayList or LinkedList?

A. ArrayList

How would you make a copy of an entire Java object with its state?

A. Have this class implement Cloneable interface and call its method clone().

How can you minimize the need of garbage collection and make the memory use more effective?

A. Use object pooling and weak object references.

There are two classes: A and B. The class B need to inform a class A when some important event has happened. What Java technique would you use to implement it?

A. If these classes are threads then consider notify() or notifyAll(). For regular classes one can use the Observer interface.

How will you sort a collection object?

A.

  
// Sort
Collections.sort(list);

// Case-insensitive sort
Collections.sort(list, String.CASE_INSENSITIVE_ORDER);

// Reverse-order sort
Collections.sort(list, Collections.reverseOrder  ());

// Case-insensitive reverse-order sort
Collections.sort(list, String.CASE_INSENSITIVE_ORDER);
Collections.reverse(list);

In a Java class, one has 10 variables. One wants to serialize only 3 variables,how can this be achieved?

A.Make variables as 'transient' which are not to be serialized.

What is cloud computing?

noreply@blogger.com (Unknown) — Thu, 28 Sep 2017 10:26:00 +0000

Cloud computing is a practice wherein series of servers hosted over internet providing every possible IT solution be it platform, infrastructure, storage, software/application as a service which can be made available or configured as on demand with minimum management efforts for specific period of time of use and one has to pay for service as per usage.

It is a cost effective way of co-utilizing resources which otherwise would be more expensive to obtain and use. Due to economy of scale it further makes cost effective appeal. Now most of top rung organizations are moving their work/business in private/hybrid/public cloud leveraging more time to focus on their core business rather investing resources on computing infrastructure and its maintenance. Amazon, Microsoft, Alibaba Google are major players in this space according to Gartner.

Collection APIs Related Interview Questions

noreply@blogger.com (Unknown) — Sat, 19 Aug 2017 02:39:00 +0000

You may like to read following posts before going through the list of questions mentioned on Collections APIs in this post:

Q. How will you synchronize a collection? A. The code sample given below elaborates on synchronization of various collection objects:

import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.SortedMap;
import java.util.SortedSet;
import java.util.TreeMap;
import java.util.TreeSet;


public class CollectionSynchronization {
public static void main(String[] args) {
Collection collection = Collections.synchronizedCollection(new ArrayList());
List list = Collections.synchronizedList(new ArrayList());
Set set = Collections.synchronizedSet(new HashSet());
Map map = Collections.synchronizedMap(new HashMap());
SortedMap sortedMap = Collections.synchronizedSortedMap(new TreeMap());
SortedSet sortedSet = Collections.synchronizedSortedSet(new TreeSet());
}
}

Q. Why is it advisable to override hashCode() method as well, when equals() method is overridden? A. When equals() method is overridden, then each object must have same hashcode for pair of objects which have equals() method overridden.If the .hashCode() method of your custom class doesn't work the same way your .equals() method works, the results of your code will be erroneous. Q. If a collection object has duplicate values then how will you convert into a collection having unique values? A. Here goes the listing:

import java.util.ArrayList;
import java.util.Iterator;
import java.util.Set;
import java.util.TreeSet;


public class CollectionExample {
public static void main(String[] args) {
 ArrayList list = new ArrayList();
 list.add("Apple");
 list.add("Apple");
 list.add("Apple");
 list.add("Mango");
 list.add("Guvava");
 list.add("Pineapple");
 Set set = new TreeSet(list);
 Iterator iterator = set.iterator();
   while (iterator.hasNext()) {
  String str= (String)iterator.next();
  System.out.println(""+str);
  }

 

}
}

Output:

Apple
Guvava
Mango
Pineapple

Q. How will you address the scenario given below to decide on collection object to work with: - Multiple keys with multiple values - Multiple keys with single value - Single key with multiple values - Single key with single value A. In the given scenario, HashMap will be the best choice. As long as the keys are unique, one can enter multiple values in an ArrayList and associate this with a key inside HashMap and retrieve those multiple values from unique key. Q.How to get a TreeSet from a TreeMap? A.The possibility of creating a TreeSet from the keyset of TreeMap.Check the example given as below:

import java.util.Iterator;
import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;


public class CollectionExample {
public static void main(String[] args) {
    TreeMap treeMap = new TreeMap();
    treeMap.put("1", "Apple");
    treeMap.put("2", "Mango");
    treeMap.put("3", "Guvava");
    treeMap.put("4", "Banana");
    treeMap.put("5", "Grapes");

    Set set = new TreeSet(treeMap.keySet());
    Iterator iterator = set.iterator();

    while (iterator.hasNext()) {
        String str = (String) iterator.next();
        System.out.println("" + str);
    }
}
}

Output:
1
2
3
4
5

Q.How a TreeSet different from HashSet? A.One line answer is : TreeSet contains-> Unique and Ordered/Sorted values(as per natural order or based on Comparator implemented logic).It is backed by TreeMap. While HashSet contains-> Unique values but not Ordered/Sorted The example given below will explain the same:

import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;


public class CollectionExample {
  public static void main(String[] args) {
      Set hashSet = new HashSet();
      Set treeSet = new TreeSet();

      addElements(hashSet);
      addElements(treeSet);

      System.out.println("HashSet:"+hashSet);
      System.out.println("TreeSet"+treeSet);
  }

  private static void addElements(Set aTreeSet) {
      aTreeSet.add(Integer.valueOf(6));
      aTreeSet.add(Integer.valueOf(2));
      aTreeSet.add(Integer.valueOf(2));
      aTreeSet.add(Integer.valueOf(4));
      aTreeSet.add(Integer.valueOf(7));
      aTreeSet.add(Integer.valueOf(7));
      aTreeSet.add(Integer.valueOf(1));
  }
}

The Output is:

HashSet:[2, 4, 6, 1, 7]
TreeSet[1, 2, 4, 6, 7]

Q.A practical problem, I want to choose a collection object which allows me to put only unique values inside it and in the same order as it is coming out of a database, here data from the database can contain duplicate values.What collection object will you choose? A.Choose:

LinkedHashSet uniqueData = new LinkedHashSet(dataFromDB);

Q. What is the initial capacity of Vector? A. 10. Try out the code given below:

import java.util.Set;
import java.util.Vector;


public class CollectionExample {
   public static void main(String[] args) {
       Vector vector = new Vector();
       int capacity = vector.capacity();
       System.out.println("Capacity:" + capacity);
   }
}

The Output is:

Capacity:10

An article on JavaWorld on Collections' API

Java EE,Open Source,Advanced Technologies Interview Questions

noreply@blogger.com (Unknown) — Sun, 02 Apr 2017 10:00:00 +0000

In continuation of earlier post on Java Interview Questions this post is focussed on interview questions in several Java EE technologies(erstwhile J2EE).Apart from Java EE, interview questions on most widely used open source technologies like Struts,Hibernate,Spring and Velocity and more will also be covered.

On constant requests by readers to put something on personal interview questions, consequently, you may like to read post on personal interview questions , prior talking about Java Enterprise Edition.

Sun Microsystems introduced Java EE 5 specification under JSR 244 and the final release was made public on May ,2006.Java EE has become a cornerstone technology for enterprise level web based solution development.It has more than three million downloads and more than twenty six vendors' application servers support this specification.It is significant to go through an introduction of Java EE 5 before jumping to questions list straightaway.You can capture essence of Java EE 5 here in detail.

If you directly want to access Master List of Java EE Interview Questions then click here.

The list of questions on various topics covered under Java EE hood is as follows(on the basis of different technologies covered under Java EE umbrella) :

Java EE 5(Click To Access Questions' List)
Java Servlets
Java Server Pages and JSP Standard Tag Libraries
Enterprise Java Beans
Java Message Services
Java APIs for XML Parsing
Java Connector Architecture
Java Mail
Java Transaction APIs

Open Source Technologies/Emerging Frameworks:
Struts
Hibernate Framework
Spring Framework
Velocity
Apache AXIS
Test Driven Development (TDD).

Architectures:
Service Oriented Architecture

OthersJava Design Patterns

If anyone of you has come across a very unique experience in terms of technical and personal questions asked, feel free to share your experience.

Keep sharing your resources for ongoing learning...

Learn more about Java EE, here.