How Big a Data should be to be called as Bigdata ? - Introduction to Big Data

2017-04-18T00:46:00.000-07:00

Hello all, welcome to my Blog Base2Pro.

So right now, you are in a mood to understand the concept of "Bigdata". We are living in a world of data and you are getting buzzed with the word "Bigdata". From the name, you can simply understand that it is a data which is very very very big, in one word humongous volume of data. Now the question is how Big a data should be to be called as "Bigdata".

In this Blog we are going to answer this question and also discuss some others topic related to "Bigdata", like

Since when the "Bigdata" Issue come up?

Who generates this humongous amount of data?

Why this large amount of "Bigdata" is so important?

Why our current database system can't handle this "Bigdata" problem?

What solution for this issue is available now?

In the family of data, the smallest member or unit is "bit". A "bit" can only be 2 variants 0/1, another way you can say it's a presentation of state ON or OFF as machine can only understand this. 8(Eight) "bit" together called as "byte" and 1024 "byte"' makes 1 "Kilobyte", likely 1024 "Kilobyte" form 1 "Megabyte". The same way the next unit of measurement of data "Gigabyte" is defined by 1024 "Megabyte" and so on we get next measurement units, "Terabyte", "Petabyte", "Exabyte", "Zettabyte". The largest unit to measure data defined till date is "Yottabyte". 1 "Yottabyte" is equal to trillion of trillion byte. How much exactly, See the below table.

· 1 Bit = Binary Digit

· 8 Bits = 1 Byte

· 1024 Bytes = 1 Kilobyte = 1,024 (210) bytes

· 1024 Kilobytes = 1 Megabyte = 1,048,576 (220) bytes

· 1024 Megabytes = 1 Gigabyte = 1,073,741,824 (230) bytes

· 1024 Gigabytes = 1 Terabyte = 1,099,511,627,776 (240) bytes

· 1024 Terabytes = 1 Petabyte = 1,125,899,906,842,624 (250) bytes

· 1024 Petabytes = 1 Exabyte = 1,152,921,504,606,846,976 (260) bytes

· 1024 Exabytes = 1 Zettabyte = 1,180,591,620,717,411,303,424 (270) bytes

· 1024 Zettabytes = 1 Yottabyte = 1,208,925,819,614,629,174,706,176 (280) bytes

· 1024 Yottabytes = 1 Brontobyte (Proposed)

· 1024 Brontobytes = 1 Geopbyte (Proposed)

So let's return to our main question,

But the answer is not specifically defined, there is no definitive number to get the medal of "Bigdata". According to Wikipedia "Big data is a term for data sets that are so large or complex that traditional data processing application software is inadequate to deal with them.". In simple word, we can say the data can be called as Big or Small depending on machine we are using to process or store the data. Right now, 100 GB data is a "Bigdata" for a normal Smartphone but it’s not a matter for computer or laptop. Same way as 100 TB can't be processed in a single computer but it can be easily be processed and stored by a server.

The amount of data produced is increasing rapidly day by day. Just to give you a simple idea here is an example.

In 2007 the amount of information or data stored worldwide was almost 300 "Exabyte" and that amount grows to 2.8 "Zettabyte" in 2012 and it is expected to grow 35 "Zettabyte" by 2020.

Data is getting created in exponential rate.

According to Moore's law the storing and processing power of our systems are getting doubled in every 2 years which leads us to this situation.

They say today's cell phone have more processing power than the rocket NASA used to go to moon in 1960.

By analysis we can see that all the data get created since beginning to the year 2000 is less than the data get produced in a minute today and all the data ever been created 95% of that had been created and in last 5 years.

So, the next question that comes up in our mind is that, since when we are facing this huge or Bigdata issue and the correct answer is since the beginning of this computer era.

At first when people or specifically scientist, use Mainframe computer, they used Magnetic tape to store the data and those were very heavy and costly so they store only the data which worth that value, but as time passed there comes Harddisk, Floppydisk, CD, DVD, SD, SSD storage, that not only reduce the cost of the data storage also decrease the processing time of data.

As storing and processing data get easier and cheaper, we started to store more and more data, even sometime we store exact duplicate data. If you search Youtube you would get thousand and thousand videos which have exact same replica in other websites.

Now the time for third question, who creates this much data that is difficult to process even in today's computer with such high-end infrastructure.

Today computer is used in every sector and every part of our daily life, from Retail to Social Media, from Healthcare to Financial Sector.

60 hour of video get uploaded every 1 second in YouTube and YouTube currently content more than 500 "Petabyte" of data and how much it is just to give you an idea, it will take more than 30,000 years to watch all the video currently in YouTube (and that also Only if you can watch at a stretch without taking any break).

Except this we also getting Large data from different sector like "Airplane-Simulation", "3D Modelling", "Automation" "Location Data", "Personal Data", "CC Camera", "Medical Research", "Logistic", "Financial Firms", "Stock Exchange" and also from different "Astronomical Research" like the "Radio Telescopes" generates 1 "Exabyte" of data in every 4 days.

This leads us to another question that "Is all the data is so important?" Previously we would like to say, "No not at all, we should only save that data which have high value" but unfortunately we was wrong, now we know every data, every piece and bit of data is important in one or other way.

All these data are important for analysis and to determine the next step for future. Like the "Retail Shop" like to know what is strategy to take in next month's Sale. "Health-care" wants to know which medicine will be best for the patient in the certain condition. "Social Media" wants to know what would you like to share, what would you like to watch and who might be your friend OR what more personal suggestion they could give to you.

Till the time in early 21st century we had only consider the data that was important to run a business or process. The "Retail Shop" was only storing the list of products and their prices, but omitting the record what user brought from them. Now time changed, they want to give the user a more personalized Offer or Service. So, they need to know what that customer like to buy OR to which brand he/she have a soft corner. Because giving wrong product in wrong time OR wrong place can bring disaster to a company, so they want to know more about their customer from their social platform and other activities performed by the same person.

But all of this information are not available in a very structure way so that they can be processed by our "Conventional" "Relational" "Database" "Management" "System".

In reality, 90% of today's data are "Unstructured" data. Also, there are "Semi-structured" data like log, processing info, email that is generated by different technical work also need to be stored for future reference. As we are moving more and more towards "Unstructured" data and using "Relational Database" we can only store different structure document like "Financial figure" "Stock Record", "Transactional Data" and even if we can store them but it take high processing power to process the same, which is also very costly and even doesn't give enough high performance.

So, we need a solution to store and analysis the huge amount of "Unstructured", "Semi-structured" and even large "Structured" data. That is why engineer have come up with a solution to handle this "Bigdata" issue by using No-SQL Database.

Currently there are many different type of No-SQL Database available like "Hbase", "CouchDB", "MongoDB", "Casandra", "Redis" etc. You can get the full list of DBs in DB-Engines.com. There are many Giant technical companies are working on the same space like, "IBM", "Oracle", "Google". "IBM" have introduced "IBM Analytics", "Oracle" that come up with solution like "Oracle Bigdata", "Google" have created a cloud based analysis that is "Google Cloud Platform" to work on "Bigdata" and to do more analysis.

These solutions are greatly accepted by most of the large company who deals with high amount of data everyday like "Google", "Yahoo", " Facebook", " eBay", " Amazon" etc. By these solutions huge data can be stored very cost effectively as the service runs on community hardware and also the processing time become significantly less.

As more and more data gets generated the people and companies start to move towards cloud platform and the cloud based analysis which we will discuss later on this Blog, but for now just know that clouds are typically a database more specifically No-SQL database on a server located remotely.

The most popular solution what we get in used these days is "HADOOP" which runs on 3-V principle. These are the characteristics given by IBM to define the process of "Bigdata" first one is "Volume", that indicates a data should be high-volume data. "Velocity", the processing time of that high-volume data should be minimum. "Variety", all type of data should be able to process by the solution of "Bigdata".

Currently there are many distribution available for the open source Hadoop among them Four are dominating the market and have most of the partner associate with them, those are "Cloudera", "Hortonworks", "Mapr" and also "Amazon Web Services"

I would like to conclude this topic with the phase what people say,

"DATA" is as raw unorganised fact is in and of itself worthless.

"INFORMATION" which is potentially valuable concept based on "DATA".

"KNOWLEDGE" is what we understand based upon "INFORMATION".

"WISDOM" is the effective use of "KNOWLEDGE" in decision making.

Hope you like this Post. Please post your feedback and suggestion in the comment Section below. You can check out my Youtube Channel Base2Pro for video on this topic and other. Please share this with all friends, family and anyone who might like this topic, Ok share with everyone. And subscribe to my Youtube channel and Blog to get notified for all my new video and post that is coming line up.

Thanks again and Stay tuned to learn more in fun way.