Shout Future

It's A "Holly Jolly" Artificial Intelligence Enabled Special Christmas

2017-12-23T11:42:00.000+05:30

Did you know there’s a bit of artificial intelligence (AI) magic behind the scenes helping to make your holiday dreams come true? Santa’s little helpers have gone high-tech this year. From finding the perfect gift to AI-enabled toys and even composing a holiday song—that truthfully will take a bit more refinement before it becomes a holiday classic—AI has infiltrated our holidays from start to finish.

Adobe Stock

AI drives online shopping experiences

Those of you who are adept behind the keyboard might be surprised that this hasn’t been true for years, but the scales have tipped in favor of online shopping with 51% of the Deloitte 2017 Holiday Retail Surveyrespondents saying they would be making the bulk of their purchases online this holiday season. In the past, shoppers would research and compare prices online, but the majority still went to stores to make their purchases.

Artificial intelligence algorithms help make online shopping experiences more personal. AI gets to know your preferences and behaviors to provide personal recommendations and save you the time of culling through thousands of product results to find just what you’re looking for. This AI tech is getting so good that it knows what you want—and can suggest complementary products—even better than you do.

The fashion retailer Stitch Fix is the perfect implementation of this technology. Customers fill out a style profile and personal stylists, heavily guided by algorithms, pick out items the customer will most likely enjoy. The algorithms keep getting smarter based on the ongoing feedback of the customer when they return an item because they don’t like it. These algorithms can help streamline our shopping experience to reduce the amount of choices we have. Increasingly, shoppers are preferring to use voice-search enabled assistants to shop and ComScore predicts that by 2020, 50 percent of all searches will be voice activated. Yep, you guessed it. AI at work again.

AI enhances customer service for retailers

Another way AI supports our online holiday shopping experience is through the use of chat bots. We likely won’t be able to determine if we are chatting with a human or a chatbot in the future as they step in to guide our purchases and handle questions about orders that have historically required retailers to hire a bunch of seasonal staff for the holidays. The North Face currently offers direct interaction with its IBM’s Watson-supported system and customers to help determine what item is the best for the customers’ needs. Chat bots are also used to answer queries or FAQs that can bog down human customer service personnel and have the capability to funnel customers that require human intervention to a human.

AI is being used by some stores to provide customers with a customized and personal experience in the store similar to what they get when shopping online. The Mall of America launched E.L.F., short for Experiential List Formulator, to help shoppers plan a personalized shopping experience by using the brains of an IBM Watson-enabled platform. The entire system is driven by AI and can understand through voice recognition technology queries and sentiments of customers and can output a personalized plan for each customer based on their answers to a series of questions.

AI impacts retailer’s operations

In addition to the customer-facing impacts of AI to the holiday shopping experience, there’s a tremendous amount of AI tech being utilized behind the scenes as well. From insights to manage inventory and to optimize supply chains and delivery routes, AI helps make retail more efficient.

A modern “Silent Night”

And perhaps the most entertaining use of AI for the holidays is experiments to have AI compose new holiday songs. Although more fine tuning is needed before these holiday melodies become classics, this early work provides interesting insight into the possibilities for AI and human collaboration for music composition. A team of computer scientists at the University of Toronto first fed the machine hours of pop songs so the algorithm could understand the elements of what makes a good song. Then they had it write a story about a picture of a Christmas tree with presents underneath it. The outcome of this effort is a tune that shows some promise but still needs quite a bit of work, but it also supports the idea that AI might be a “great band member” in the future and that humans and AI will continue to collaborate to make music that we all can enjoy. Thomas Holm, a Norwegian “The Voice” contestant, is the musician that collaborated with Microsoft to refine the lyrics and finalize a song created by artificial intelligence called “Joyful Time in the City,” and this experiment gives us a good look at how music composition might happen in the future with the support of AI.

AI has infiltrated this holiday season like never before. In most cases, the efficiencies and conveniences AI enabled retail provides us are welcome by all.

Bernard Marr is a best-selling author & keynote speaker on business, technology and big data. His new book is Data Strategy. To read his future posts simply join his network here.

Credit: Forbes

Zurich Insurance starts using AI to process personal injury claim

2017-05-18T21:30:00.001+05:30

Zurich said it recently introduced AI claims handling and saved 40,000 work hours as a result.

Zurich Insurance is deploying artificial intelligence in deciding personal injury claims after trials cut the processing time from an hour to just seconds, its chairman said.

"We recently introduced AI claims handling, and saved 40,000 work hours, while speeding up the claim processing time to five seconds," Tom de Swaan told Reuters.

The insurer had started using machines in March to review paperwork, such as medical reports.

"We absolutely plan to expand the use of this type of AI (artificial intelligence)," he said.

Insurers are racing to hone the benefits of technological advancements such as big data and AI as tech-driven startups, like Lemonade, enter the market.

Lemonade promises renters and homeowners insurance in as little as 90 seconds and payment of claims in three minutes with the help of artificial intelligence bots that set up policies and process claims.

De Swaan said Zurich Insurance, Europe's fifth-biggest insurer, would increasingly use machine learning, or AI, for handling claims.

"Accuracy has improved. Because it's machine learning, every new claim leads to further development and improvements," the Dutch native said.

Japanese insurer Fukoku Mutual Life Insurance began implementing AI in January, replacing 34 staff members in a move it said would save 140 million yen ($1.3 million) a year.

British insurer Aviva is also currently looking at using AI.

De Swaan said he does not fear competition from tech giants like Google-parent Alphabet or Apple entering the insurance market, although some technology companies have expressed interest in cooperating with Zurich.

"None of the technology companies so far have taken insurance risk on their balance sheet, because they don't want to be regulated," he said.

"You need the balance sheet to be able to sell insurance and take insurance risk," he added.

China Sets National Laboratory to Lead World in Brain-Like AI Artificial Intelligence

2017-05-15T14:33:00.000+05:30

China's first national laboratory for brain-like artificial intelligence (AI) technology was inaugurated Saturday to pool the country's top research talent and boost the technology. China’s rapid rise up the ranks of AI research has the world's scientific community taking notice. In October, the Obama White House released a “strategic plan” for AI research, which noted that the U.S. no longer leads the world in journal articles on “deep learning,” a particularly hot subset of AI research right now. The country that had overtaken the U.S.? China, of course.
“I have a hard time thinking of an industry we cannot transform with AI,” says Andrew Ng, chief scientist at Baidu. Ng previously cofounded Coursera and Google Brain, the company’s deep learning project. Now he directs Baidu’s AI research out of Sunnyvale, California, right in Silicon Valley.

“China has a fairly deep awareness of what’s happening in the English-speaking world, but the opposite is not true,” says Ng. He points out that Baidu has rolled out neural network-based machine translation and achieved speech recognition accuracy that surpassed humans—but when Google and Microsoft, respectively, did so, the American companies got a lot more publicity. “The velocity of work is much faster in China than in most of Silicon Valley,” says Ng Approved by the National Development and Reform Commission in January, the lab, based in China University of Science and Technology (USTC), aims to develop a brain-like computing paradigm and applications.

The university, known for its leading role in developing quantum communication technology, hosts the national lab in collaboration with a number of the country's top research bodies such as Fudan University, Shenyang Institute of Automation of the Chinese Academy of Sciences as well as Baidu, operator of China's biggest online search engine.

Wan Lijun, president of USTC and chairman of the national lab, said the ability to mimic the human brain's ability in sorting out information will help build a complete AI technology development paradigm. The lab will carry out research to guide machine learning such as recognizing messages and using visual neural networks to solve problems. It will also focus on developing new applications with technological achievements.

Microsoft uses Big data analytics to save the Earth

2017-05-11T00:20:00.000+05:30

Microsoft is undertaking several projects dedicated to sustainability

Microsoft has been making significant contributions in Tech for Good and has taken significant steps towards environment conservation. The company’s going green mantra is underscored by the $1.1 million in 2016, fundraising and 5,949 number of volunteering hours put in by its employees.

But it doesn’t stop there. Microsoft’s ecosystem allows the firm, its employees, and the business partners to leverage new technologies for improving sustainability of their companies and communities. The Redmond giant recently tied up with The Nature Conservancy, a nonprofit to extend support for nonprofits globally.

greening the planet

Microsoft’s commitment towards nature is deeply rooted in the technologies it utilizes. Microsoft announced a $1 billion commitment to bring cloud computing resources to nonprofit organizations around the world. The firm donates near $2 million every day in products and services to nonprofits as a part of the commitment.

Microsoft has extended its support to organizations like World Wildlife Fund, Rocky Mountain Institute,Carbon Disclosure Project, Wildlife Conservation Society, and the U.N. Framework Convention on Climate Change’s (UNFCCC) Climate Neutral Now initiative.

Here are a slew of use cases

How Prashant Gupta’s initiative is helping farmers in Andhra Pradesh increase revenue? Prashant Gupta works as a Cloud + Enterprise Principal Director at Microsoft. Gupta is undertaking significant developments for environment. Earlier, Gupta had facilitated a partnership for Microsoft with a United Nations agency, the International Crops Research Institute for the Semi-Arid Tropics (ICRISAT), and the Andhra Pradesh government. The project involved helping ground nut farmers cope with the drought.

Gupta and his team leveraged advanced analytics and machine learning to launch a pilot program with a Personalized Village Advisory Dashboard for 4,000 farmers in 106 villages in Andhra Pradesh. It also included a Sowing App with 175farmers in one district.

Based on weather conditions, soil, and other indicators; the Sowing App advises farmers on the best time to sow. The Personalized Village Advisory Dashboard provides insights about soil health, fertilizer recommendations, and seven-day weather forecasts.

Nature Conservancy’s Coastal Resilience program

Microsoft’s Azure cloud platform for Nature Conservancy’s Coastal Resilience program: The Coastal Resilience is a public-private partnership led by The Nature Conservancy to help coastal communities address the devastating effects of climate change and natural disasters. The program has trained and helped over 100 communities globally about the uses and applications of the Microsoft’s Natural Solutions Toolkit.

The toolkit contains a suite of geospatial tools and web apps for climate adaptation and resilience planning across land and sea environments. This has helped in strategizing for risk reduction, restoration, and resilience to safeguard local habitats, communities, and economies.

Puget Sound: Puget Sound’s lowland river valleys is a treasure house, delivering valuable assets, a wealth of natural, agricultural, industrial, recreational, and health benefits to the four million people who live in the region. However, the communities are at increasing risk of flooding issues from rising sea levels, more extreme coastal storms, and more frequent river flooding.

High winds hit Puget Sound

The Conservancy’s Washington chapter is building a mapping tool as part of the Coastal Resilience toolkit to reduce the flow of polluted storm water into Puget Sound. Emily Howe, an aquatic ecologist is in charge of the project, which revolves around developing the new Storm water Infrastructure mapping tool. This tool will be eventually integrated into the Puget Sound Coastal Resilience tool set, that will be hosted on Azure.

Furthermore, it will include a high-level heat map of storm water pollution for the region, combining an overlay of pollution data with human and ecological data for prioritizing areas of concern.

Data helps in Watershed Management: Today, around 1.7 billion people living in the world’s largest cities depend on water flowing from watersheds. However, estimates suggest that those sources of watershed will be tapped by up to two-thirds of the global population, by 2050.

Kari Vigerstol, The Nature Conservancy’s Global Water Funds Director of Conservation had overseen development of a tool to provide them with better data. The project entailed assisting cities and protecting their local water sources. 4,000 cities were analyzed by “Beyond the Source”. The results stated that natural solutions can improve water quality for four out of five cities.

Furthermore, The Natural Solutions Toolkit is being leveraged globally to better understand and protect water resources around the world. Through the water security toolkit, cities will be furnished with a more powerful set of tools. Users can also explore data, and access proven solutions and funding models utilizing the beta version of Protecting Water Atlas. This tool will help in improving water quality and supply for the future.

Microsoft is illuminating these places with its innovative array of big data and analytics offerings

Emily Howe

In Finland, Microsoft partnered with CGI to develop a smarter transit system for the city of Helsinki. This data-driven initiative saw Microsoft utilize the city’s existing warehouse systems to create a cloud-based solution that could collate and analyse travel data. Helsinki’s bus team noticed a significant reduction in fuel costs and consumption, besides realizing increased travel safety, and improved driver performance.
Microsoft Research Lab Asia designed a mapping tool, called Urban Air for the markets in China. The tool allows users to see, and even predict, air quality levels across 72 cities in China. The tool furnishes real-time, detailed air quality information, making use of big data and machine learning. Additionally, the tool combines a mobile app, which is used about three million times per day.
Microsoft is implementing environmental strategies worldwide. The firm is assisting the city of Chicago in designing new ways to gather data. Additionally, the firm is also helping the city utilize predictive analytics in order to better address water, infrastructure, energy, and transportation challenges.
Boston serves as another great instance where Microsoft is working to spread information about the variety of urban farming programs in Boston. Microsoft is also counting on the potential of AI and other technology to increase the impact for the city.
Microsoft has also partnered with Athena Intelligence for developing the hill city of San Francisco. As a part of this partnership, Microsoft is leveraging Athena’s data processing and visualization platform to gather valuable data about land, food, water, and energy. This will help in improving local decision-making.

Outlook

Satya Nadella, CEO of Microsoft

Data is not all that matters. At the end, it’s essentially about how cities can be empowered to take action based on that data. Microsoft has comprehensively supported the expansion of The Nature Conservancy’s innovative Natural Solutions Toolkit. The solution suite is already powering on-the-ground and in-the-water projects around the world, besides benefiting coastal communities, residents of the Puget Sound, and others globally.

Microsoft is doing an excellent job, delivering on its promise to empower people and organizations globally to thrive in a resource-constrained world. The organization is empowering researchers, scientists and policy specialists at nonprofits by providing them with technology that addresses sustainability.

Facebook's New AI created a faster and accurate Translation system

2017-05-09T23:46:00.000+05:30

TAPPING A NEURAL NETWORK TO TRANSLATE TEXT IN CHUNKS

Facebook.com artificial intelligence

Facebook's research arm is coming up with better ways to translate text using AI.

Facebook

Facebook’s billion-plus users speak a plethora of languages, and right now, the social network supports translation of over 45 different tongues. That means that if you’re an English speaker confronted with German, or a French speaker seeing Spanish, you’ll see a link that says “See Translation.”

But Tuesday, Facebook announced that its machine learning experts have created a neural network that translates language up to nine times faster and more accurately than other current systems that use a standard method to translate text.

The scientists who developed the new system work at the social network’s FAIR group, which stands for Facebook A.I. Research.

“Neural networks are modeled after the human brain,” says Michael Auli, of FAIR, and a researcher behind the new system. One of the problems that a neural network can help solve is translating a sentence from one language to another, like French into English. This network could also be used to do tasks like summarize text, according to a blog item posted on Facebook about the research.

But there are multiple types of neural networks. The standard approach so far has been to use recurrent neural networks to translate text, which look at one word at a time and then predict what the output word in the new language should be. It learns the sentence as it reads it. But the Facebook researchers tapped a different technique, called a convolutional neural network, or CNN, which looks at words in groups instead of one at a time.

“It doesn’t go left to right,” Auli says, of their translator. “[It can] look at the data all at the same time.” For example, a convolutional neural network translator can look at the first five words of a sentence, while at the same time considering the second through sixth words, meaning the system works in parallel with itself.

Graham Neubig, an assistant professor at Carnegie Mellon University’s Language Technologies Institute, researches natural language processing and machine translation. He says that this isn’t the first time this kind of neural network has been used to translate text, but that this seems to be the best he’s ever seen it executed with a convolutional neural network.

“What this Facebook paper has basically showed— it’s revisiting convolutional neural networks, but this time they’ve actually made it really work very well,” he says.

Facebook isn’t yet saying how it plans to integrate the new technology with its consumer-facing product yet; that’s more the purview of a department there call the applied machine learning group. But in the meantime, they’ve released the tech publicly as open-source, so other coders can benefit from it

That’s a point that pleases Neubig. “If it’s fast and accurate,” he says, “it’ll be a great additional contribution to the field.”

5 steps to successfully learn machine learning by yourself

2017-05-09T15:02:00.000+05:30

How to learn Machine learning? Learning machine learning is not a big deal but if you want to be an expert in any field you need someone as a mentor. So try to follow some professional machine learning blogs like Shout Future, Kdnuggets, Analytics Vidhya etc., and if you have any doubts, clarify it through forums or directly ask in comments.

Machine Learning

I realised the growth and development of machine learning in future will be incredible, so started to learn Machine learning back in 2013. I started from scratch and I confused a lot. Because I don't know what to do and where to start.
I think you are also like me! Here I mentioned step by step learning process to become a professional machine learning engineer yourself.

1.Getting Started:

Find out what is machine learning.
Skills to become machine learning engineer.
Attend conferences and workshops.
Interact with experienced people directly or through social media.

2. Learn Basics of Mathematics and statistics:

Start to learn Descriptive and Inferential statistics by Udacity course.
Linear algebra course by Khan Academy and MITopenCourseware
Learn Multivariate Calculus by Calculus One
Learn Probability by edX course.

3. Choose your tool: Learn R or Python:

Learn R:

R is very easy to learn compare than python.
Interactive intro to R programming language by Data Camp.
Exploratory data analysis by Coursera.
Start to follow R-Bloggers.

Learn Python:

Start your programming with Google's Python Class.
Intro to data analysis by Udacity.

4. Basic and Advanced machine learning tools:

Machine Learning Course by Coursera.
Machine Learning classification by Coursera.
Intro to Machine Learning by Udacity.
Blogs and Guides like Shout Future, machine Learning mastery, etc.
Algorithms: Design and Analysis 1
Algorithms: Design and Analysis 2

5. Build your Profile:

Start your Github profile.
Start to practice in Kaggle competitions.

That's it. With these skills you can enter into the sexiest job in the world now called "Data Scientist". Plan well and follow this steps very well.

You have to travel very long to become an expert in this field. So start your journey from today onwards and separate yourself from the crowd.

Please Comment your ideas and opinions.

Data Analytics role in preventing insurance fraud

2017-05-09T02:16:00.000+05:30

While there is no doubt that the insurance segment is witnessing an unprecedented annual growth, insurers continue to struggle with loss-leading portfolios and lower insurance penetration among consumers. Insurers are facing increasing pressure to strike the right balance, while ensuring adherence to underwriting and claims decisions in the face of regulatory pressures, growth of digital channels and increasing competition. Adding to this is the need to secure the good risks, while weeding out the bad risks.

Insurers are turning their attention towards big data and analytics solutions to help check fraud, recognize misrepresentation and prevent identity theft. With the government’s recent push to adopt digitization, the Aadhaar card plays a crucial role, linking income tax permanent account numbers (PANs), banks, credit bureaus, telecoms and utilities and providing a unified and centralized data registry that profiles an individual’s economic behaviour. The e-commerce boom provides additional data on financial behaviour.

Fraudulent practices

Claims fraud is a threat to the viability of the health insurance business. Although health insurers regularly crack down on unscrupulous healthcare providers, fraudsters continually exploit any new loopholes with forged documents purporting to be from leading hospitals.

Medical ID theft is one of the most common techniques adopted by fraudsters. Due to this, claim funds are paid into their bank accounts, through identity theft. The insurer’s procedures allows for the policyholder to send a scanned image of his/her cheque, with the bank account details for ID purposes, which is then manipulated by the fraudsters.

Besides forged documents, other common sources of fraud come from healthcare providers themselves, with cases of ‘upgrading’ (billing for more expensive treatments than those provided), ‘phantom billing’ and ‘ganging’ (billing for services provided to family members or other individuals accompanying the patient, but not delivered).

Health insurers have to take action before an insurance claim is paid and to put an end to the ‘pay-and-chase’ approach. Using data to validate a pre-payment would be far more useful than having to ‘chase’ for a payment. This approach, however, rests on real-time access to information sources.

Life insurance’s woes

India’s life insurers suffer from low persistency rates that see more than one in three policies lapse by the end of the second year. This may be attributed to mis-selling, misrepresentation of material facts, premeditated fabrication and in other cases suppression of facts.

Life insurers have been facing fraud that is largely data driven and can be curbed with effective use of data analytics. While seeking customer information, insurers should perform checks against public record databases to ensure they have insights into the validity of personal information. This can be achieved through data mining and validation from various sources. For instance, in the US, frauds are committed through stolen social security numbers or driver’s license numbers, or those of deceased individuals. Data accessed from various sources will help identify if the person in question is using multiple identities or multiple people are using the identity presented.

The use of public, private and proprietary databases to obtain information not typically found in an individual’s wallet to create knowledge-based authentication questions which are designed to be answered only by the correct individual can also help reduce fraud significantly.

Continuous evaluation of existing customers is also critical for early fraud detection. For example, one red flag for potential fraud can involve beneficiary or address changes for new customers. Insurers should verify address changes, as many consumers do not know their identity has been stolen until after it has happened. By applying relationship analytics, insurers can obtain insights into the relationship between the insured, the owner, and the beneficiary, to help determine whether those individuals are linked to other suspicious entities or are displaying suspicious behaviour patterns.

Solutions for all

Like in most developed insurance markets, it is imperative that data on policies, claims and customers be made available on a shared platform, in real-time. Such a platform can allow for real-time enquiries on customers. It can also facilitate screening of the originator of every proposal. Insurers would contribute policy, claims and distributors’ information to the repository on a regular basis. Such data repositories can provide insights to help insurers detect patterns, identify nexus and track mis-selling.

Insurance data is dynamic and hence data analytics cannot depend only on past behaviour patterns. So data has to be updated regularly. Predictive analysis can play a significant role in identifying distributor nexus, mis-selling and repeated misrepresentations. Relationship analytics could be used to identify linked sellers and suspected churn among them.

These data platform-based solutions are not just about preventing reputational risk and loss of business, but with controlled and more informed risk selection, there could be a positive impact on pricing of products. The whole process of underwriting new business with greater granularity of risk and greater transparency can bring in new customers, but it could also out-price some others. There can be increased scrutiny of agents, brokers and distributors to eliminate any suspects from the system.

Successful fraud prevention strategies include shifting towards a proactive approach that detects fraud prior to policy issuance, and leveraging red flags or business rules, real-time identity checks, relationship analytics, and predictive models. Insurers who leverage both internal data and external data analytics will better understand fraud risks throughout their customer life cycles, and will be more prepared to detect and mitigate those risks.

Statistics | Data Gathering Techniques

2017-05-06T14:34:00.000+05:30

Table of Contents:

Introduction
Nature of Data
1. Time series data.
2. Spatial data
3. Spacio-temporal data.
Categories of data
1.Primary data
1. Direct personal interviews.
2. Indirect Oral interviews.
3. Information from correspondents.
4. Mailed questionnaire method.
5. Schedules sent through enumerators.
2. Secondary data
1. Published sources
2. Unpublished sources.

Data gathering techniques

Introduction:
Everybody collects, interprets and uses information, much of it in numerical or statistical forms in day-to-day life. It is a common practice that people receive large quantities of information everyday through conversations, televisions, computers, the radios, newspapers, posters, notices and instructions. It is just because there is so much information available that people need to be able to absorb, select and reject it.

In everyday life, in business and industry, certain statistical information is necessary and it is independent to know where to find it how to collect it. As consequences, everybody has to compare prices and quality before making any decision about what goods to buy. As employees of any firm, people want to compare their salaries and working conditions, promotion opportunities and so on. In time the firms on their part want to control costs and expand their profits.

One of the main functions of statistics is to provide information which will help on making decisions. Statistics provides the type of information by providing a description of the present, a profile of the past and an estimate of the future.

The following are some of the objectives of collecting statistical information.
1. To describe the methods of collecting primary statistical information.
2. To consider the status involved in carrying out a survey.
3. To analyse the process involved in observation and interpreting.
4. To define and describe sampling.
5. To analyse the basis of sampling.
6. To describe a variety of sampling methods.

Statistical investigation is a comprehensive and requires systematic collection of data about some group of people or objects, describing and organizing the data, analyzing the data with the help of different statistical method, summarizing the analysis and using these results for making judgements, decisions and predictions.
The validity and accuracy of final judgement is most crucial and depends heavily on how well the data was collected in the first place. The quality of data will greatly affect the conditions and hence at most importance must be given to this process and every possible precaution should be taken to ensure accuracy while collecting the data.

Nature of data:
It may be noted that different types of data can be collected for different purposes. The data can be collected in connection with time or geographical location or in connection with time and location.
The following are the three types of data:
1. Time series data.
2. Spatial data
3. Spacio-temporal data.

Time series data Analysis:
It is a collection of a set of numerical values, collected over a period of time. The data might have been collected either at regular intervals of time or irregular intervals of time.
Spatial Data:
If the data collected is connected with that of a place, then it is termed as spatial data. For example, the data may be
1. Number of runs scored by a batsman in different test matches in a test series at different places.
2. District wise rainfall in a state.
3. Prices of silver in four metropolitan cities.
Spacio Temporal Data:
If the data collected is connected to the time as well as place then it is known as spacio temporal data.

Categories of data:
Any statistical data can be classified under two categories depending upon the sources utilized. These categories are,
1. Primary data
2. Secondary data

Primary data:
Primary data is the one, which is collected by the investigator himself for the purpose of a specific inquiry or study. Such data is original in character and is generated by survey conducted by individuals or research institution or any organisation.
For example, if a researcher is interested to know the impact of noon meal scheme for the school children, he has to undertake a survey and collect data on the opinion of parents and children by asking relevant questions. Such a data collected for the purpose is called primary data.

The primary data can be collected by the following five methods.
1. Direct personal interviews.
2. Indirect Oral interviews.
3. Information from correspondents.
4. Mailed questionnaire method.
5. Schedules sent through enumerators.

1. Direct personal interviews:
The persons from whom information’s are collected are known as informants. The investigator personally meets them and asks questions to gather the necessary information’s. It is the suitable method for intensive rather than extensive field surveys. It suits best for intensive study of the limited field.

Merits:
1. People willingly supply informations because they are approached personally. Hence, more response noticed in this method than in any other method.
2. The collected informations are likely to be uniform and accurate. The investigator is there to clear the doubts of the informants.
3. Supplementary informations on informant’s personal aspects can be noted. Informations on character and environment may help later to interpret some of the results.
4. Answers for questions about which the informant is likely to be sensitive can be gathered by this method.
5. The wordings in one or more questions can be altered to suit any informant. Explanations may be given in other languages also. Inconvenience and misinterpretations are thereby avoided.

Limitations:
1. It is very costly and time consuming.
2. It is very difficult, when the number of persons to be interviewed is large and the persons are spread over a wide area.
3. Personal prejudice and bias are greater under this method.

2. Indirect Oral Interviews:
Under this method the investigator contacts witnesses or neighbours or friends or some other third parties who are capable of supplying the necessary information. This method is preferred if the required information is on addiction or cause of fire or theft or murder etc., If a fire has broken out a certain place, the persons living in neighbourhood and witnesses are likely to give information on the cause of fire.
In some cases, police interrogated third parties who are supposed to have knowledge of a theft or a murder and get some clues. Enquiry committees appointed by governments generally adopt this method and get people’s views and all possible details of facts relating to the enquiry. This method is suitable whenever direct sources do not exist or cannot be relied upon or would be unwilling to part with the information.
The validity of the results depends upon a few factors, such as the nature of the person whose evidence is being recorded, the ability of the interviewer to draw out information from the third parties by means of appropriate questions and cross examinations, and the number of persons interviewed. For the success of this method one person or one group alone should not be relied upon.

3. Information from correspondents:
The investigator appoints local agents or correspondents in different places and compiles the information sent by them. Informations to Newspapers and some departments of Government come by this method. The advantage of this method is that it is cheap and appropriate for extensive investigations. But it may not ensure accurate results because the correspondents are likely to be negligent, prejudiced and biased. This method is adopted in those cases where informations are to be collected periodically from a wide area for a long time.

4. Mailed questionnaire method:
Under this method a list of questions is prepared and is sent to all the informants by post. The list of questions is technically called questionnaire. A covering letter accompanying the questionnaire explains the purpose of the investigation and the importance of correct informations and requests the informants to fill in the blank spaces provided and to return the form within a specified time. This method is appropriate in those cases where the informants are literates and are spread over a wide area.

Merits:
1. It is relatively cheap.
2. It is preferable when the informants are spread over the wide area.

Limitations:
1. The greatest limitation is that the informants should be literates who are able to understand and reply the questions.
2. It is possible that some of the persons who receive the questionnaires do not return them.
3. It is difficult to verify the correctness of the informations furnished by the respondents.
With the view of minimizing non-respondents and collecting correct information, the questionnaire should be carefully drafted. There is no hard and fast rule. But the following general principles may be helpful in framing the questionnaire. A covering letter and a self addressed and stamped envelope should accompany the questionnaire.
The covering letter should politely point out the purpose of the survey and privilege of the respondent who is one among the few associated with the investigation. It should assure that the informations would be kept confidential and would never be misused. It may promise a copy of the findings or free gifts or concessions etc.,

Characteristics of a good questionnaire:
1. Number of questions should be minimum.
2. Questions should be in logical orders, moving from easy to more difficult questions.
3. Questions should be short and simple. Technical terms and vague expressions capable of different interpretations should be avoided.
4. Questions fetching YES or NO answers are preferable. There may be some multiple choice questions requiring lengthy answers are to be avoided.
5. Personal questions and questions which require memory power and calculations should also be avoided.
6. Question should enable cross check. Deliberate or unconscious mistakes can be detected to an extent.
7. Questions should be carefully framed so as to cover the entire scope of the survey.
8. The wording of the questions should be proper without hurting the feelings or arousing resentment.
9. As far as possible confidential informations should not be sought.
10. Physical appearance should be attractive, sufficient space should be provided for answering each question.

5. Schedules sent through Enumerators:
Under this method enumerators or interviewers take the schedules, meet the informants and filling their replies. Often distinction is made between the schedule and a questionnaire. A schedule is filled by the interviewers in a face-to-face situation with the informant. A questionnaire is filled by the informant which he receives and returns by post. It is suitable for extensive surveys.

Merits:
1. It can be adopted even if the informants are illiterates.
2. Answers for questions of personal and pecuniary nature can be collected.
3. Non-response is minimum as enumerators go personally and contact the informants.
4. The informations collected are reliable. The enumerators can be properly trained for the same.
5. It is most popular methods.

Limitations:
1. It is the costliest method.
2. 2. Extensive training is to be given to the enumerators for collecting correct and uniform informations.
3. Interviewing requires experience. Unskilled investigators are likely to fail in their work.

Before the actual survey, a pilot survey is conducted. The questionnaire/Schedule is pre-tested in a pilot survey. A few among the people from whom actual information is needed are asked to reply. If they misunderstand a question or find it difficult to answer or do not like its wordings etc., it is to be altered. Further it is to be ensured that every questions fetches the desired answer.

Merits and Demerits of primary data:
1. The collection of data by the method of personal survey is possible only if the area covered by the investigator is small. Collection of data by sending the enumerator is bound to be expensive. Care should be taken twice that the enumerator record correct information provided by the informants.
2. Collection of primary data by framing a schedules or distributing and collecting questionnaires by post is less expensive and can be completed in shorter time.
3. Suppose the questions are embarrassing or of complicated nature or the questions probe into personnel affairs of individuals, then the schedules may not be filled with accurate and correct information and hence this method is unsuitable.
4. The information collected for primary data is mere reliable than those collected from the secondary data.

Secondary Data:
Secondary data are those data which have been already collected and analysed by some earlier agency for its own use; and later the same data are used by a different agency.

According to W.A.Neiswanger, ‘A primary source is a publication in which the data are published by the same authority which gathered and analysed them. A secondary source is a publication, reporting the data which have been gathered by other authorities and for which others are responsible’.

Sources of Secondary data:
In most of the studies the investigator finds it impracticable to collect first-hand information on all related issues and as such he makes use of the data collected by others. There is a vast amount of published information from which statistical studies may be made and fresh statistics are constantly in a state of production. The sources of secondary data can broadly be classified under two heads:

1. Published sources, and
2. Unpublished sources.

1. Published Sources:
The various sources of published data are:
1. Reports and official publications of
(i) International bodies such as the International Monetary Fund, International Finance Corporation and United Nations Organisation.
(ii) Central and State Governments such as the Report of the Tandon Committee and Pay Commission.
2. Semi-official publication of various local bodies such as Municipal Corporations and District Boards.
3. Private publications-such as the publications of –
(i) Trade and professional bodies such as the Federation of Indian Chambers of Commerce and Institute of Chartered Accountants.
(ii) Financial and economic journals such as ‘Commerce’ , ‘Capital’ and ‘ Indian Finance’ .
(iii) Annual reports of joint stock companies.
(iv) Publications brought out by research agencies, research scholars, etc.

It should be noted that the publications mentioned above vary with regard to the periodically of publication. Some are published at regular intervals (yearly, monthly, weekly etc.,) whereas others are ad hoc publications, i.e., with no regularity about periodicity of publications.

Note: A lot of secondary data is available in the internet. We can access it at any time for the further studies.

2. Unpublished Sources
All statistical material is not always published. There are various sources of unpublished data such as records maintained by various Government and private offices, studies made by research institutions, scholars, etc. Such sources can also be used where necessary

Precautions in the use of Secondary data
The following are some of the points that are to be considered in the use of secondary data
1. How the data has been collected and processed
2. The accuracy of the data
3. How far the data has been summarized
4. How comparable the data is with other tabulations
5. How to interpret the data, especially when figures collected for one purpose is used for another
Generally speaking, with secondary data, people have to compromise between what they want and what they are able to find.

Merits and Demerits of Secondary Data:
1. Secondary data is cheap to obtain. Many government publications are relatively cheap and libraries stock quantities of secondary data produced by the government, by companies and other organisations.
2. Large quantities of secondary data can be got through internet.
3. Much of the secondary data available has been collected for many years and therefore it can be used to plot trends.
4. Secondary data is of value to: - The government – help in making decisions and planning future policy.
- Business and industry – in areas such as marketing, and sales in order to appreciate the general economic and social conditions and to provide information on competitors.
- Research organisations – by providing social, economical and industrial information.

Google AI Helps you to bring your ideas through visual communication

2017-04-15T21:35:00.002+05:30

Google Neural Networks Drawings

Google went big on art this week. The company launched a platform to help people who are terrible at art communicate visually. It also published research about teaching art to another terrible stick-figure drawer: a neural network.

On Tuesday, the company announced AutoDraw, a web-based service aimed at users who lack drawing talent. Essentially, the program allows you to use your finger (or mouse if you’re on a computer) to sketch out basic images like apples and zebras. Then, it analyzes your pathetical drawing and suggests a professionally-drawn version of the same thing. You then click on the nice drawing you wanted, and it replaces yours with the better one. It’s like autocorrect, but for drawing.

Nooka Jones, the team lead at Google’s creative lab, says that AutoDraw is about helping people express themselves. “A lot of people are fairly bad at drawing, but it shouldn’t limit them from being able to communicate visually,” he says. “What if we could help people sketch out their ideas, or bring their ideas to life, through visual communication, with the idea of machine learning?”

The system’s underlying tech has its roots in a surprising place, according to Dan Motzenbecker, a creative technologist at Google. “It’s a neural network that’s actually originally devised to recognize handwriting,” he says. That handwriting could be Latin script, or Chinese or Japanese characters, like kanji. From there, “it’s not that big of a leap to go to a doodle.”

As people makes their line drawings, the network tries to figure out what it is. “The same way that might work for a letter of the alphabet, or a Chinese character,” Motzenbecker says, “we can use that for a doodle of a toaster.”

Neural networks get better by learning from data, but when asked about how and if the system is learning from our drawings, Jones says: “In theory, yes; we don’t quite disclose what we actually use as input back into the algorithm.”

Just like there are different ways to draw a letter, there are multiple representations of an elephant or a horse. “The more variety it sees,” Motzenbecker says, “the more adaptable it is to seeing novel ways of sketching things.” Users are also confirming the AI’s guesses when selecting a new drawing, which could help to guide its future decisions.

“One of the things that you see across the entire industry, and Google has recognized the potential of this much earlier than most other technology companies,” says Shuman Ghosemajumder, the chief technology officer at Shape Security in Mountain View, Calif., and a former Google employee, “is the use of machine learning to be able to do things that were previously thought to require direct human intervention.” And machine learning models need data.

“In this case, if you’ve got an app that millions of people potentially will use to be able to attempt to draw different figures,” he adds, “even if your technology isn’t perfect right now, you are creating this amazing training set of input data that can be used to improve these models over time.”

While AutoDraw is about helping people turn their doodles into more recognizable images, the search giant is also interested in how computers draw. On Thursday, Google Research published a blog post and paper about how they had schooled a recurrent neural network to draw items like cats and pigs.

The researcher team's goal was to train “a machine to draw and generalize abstract concepts in a manner similar to humans,” according to a blog item written by David Ha, a Google Brain Resident. The system works by taking human input—say, a drawing of a cat or just the word “cat,” according to a Google spokesperson—and then making its own drawing.

The results are fascinating and bizarre. In one example, the researchers presented the system with a sketch of a three-eyed cat. The computer drew its own cat, but this one had the right number of eyes, “suggesting that our model has learned that cats usually only have two eyes.”

In another, when presented with a picture of a toothbrush, the Google neural network’s cat model made a Picasso-like feline that still had a toothbrush-inspired feel to it.

A Google spokesperson confirmed that it is different neural networks that are powering AutoDraw and the other research, but the similarities are striking: in both cases, the system is drawing on machine learning to take a piece of input and then either suggest a professionally-drawn image, or create something new totally on its own.

The Ethics of AI Artificial Intelligence

2017-02-18T15:58:00.001+05:30

In this industry, it's a tired old cliche to say that we're building the future. But that's true now more than at any time since the Industrial Revolution. The proliferation of personal computers, laptops, and cell phones has changed our lives, but by replacing or augmenting systems that were already in place. Email supplanted the post office; online shopping replaced the local department store; digital cameras and photo sharing sites such as Flickr pushed out film and bulky, hard-to-share photo albums. AI presents the possibility of changes that are fundamentally more radical: changes in how we work, how we interact with each other, how we police and govern ourselves.

Fear of a mythical "evil AI" derived from reading too much sci-fi won't help. But we do need to ensure that AI works for us rather than against us; we need to think ethically about the systems that we're building. Microsoft's CEO, Satya Nadella, writes:

The debate should be about the values instilled in the people and institutions creating this technology. In his book Machines of Loving Grace, John Markoff writes, 'The best way to answer the hard questions about control in a world full of smart machines is by understanding the values of those who are actually building these systems.' It's an intriguing question, and one that our industry must discuss and answer together.

What are our values? And what do we want our values to be? Nadella is deeply right in focusing on discussion. Ethics is about having an intelligent discussion, not about answers, as such—it's about having the tools to think carefully about real-world actions and their effects, not about prescribing what to do in any situation. Discussion leads to values that inform decision-making and action.

The word "ethics" comes from "ethos," which means character: what kind of a person you are. "Morals" comes from "mores," which basically means customs and traditions. If you want rules that tell you what to do in any situation, that's what customs are for. If you want to be the kind of person who executes good judgment in difficult situations, that's ethics. Doing what someone tells you is easy. Exercising good judgement in difficult situations is a much tougher standard.

Exercising good judgement is hard, in part, because we like to believe that a right answer has no bad consequences; but that's not the kind of world we have. We've damaged our sensibilities with medical pamphlets that talk about effects and side effects. There are no side effects; there are just effects, some of which you might not want. All actions have effects. The only question is whether the negative effects outweigh the positive ones. That's a question that doesn't have the same answer every time, and doesn't have to have the same answer for every person. And doing nothing because thinking about the effects makes us uncomfortable is, in fact, doing something.

The effects of most important decisions aren't reversible. You can't undo them. The myth of Pandora's box is right: once the box is opened, you can't put the stuff that comes out back inside. But the myth is right in another way: opening the box is inevitable. It will always be opened; if not by you, by someone else. Therefore, a simple "we shouldn't do this" argument is always dangerous, because someone will inevitably do it, for any possible "this." You may personally decide not to work on a project, but any ethics that assumes people will stay away from forbidden knowledge is a failure. It's far more important to think about what happens after the box has been opened. If we're afraid to do so, we will be the victims of whoever eventually opens the box.

Finally, ethics is about exercising judgement in real-world situations, not contrived situations and hypotheticals. Hypothetical situations are of very limited use, if not actually harmful. Decisions in the real world are always more complex and nuanced. I'm completely uninterested in whether a self-driving car should run over the grandmothers or the babies. An autonomous vehicle that can choose which pedestrian to kill surely has enough control to avoid the accident altogether. The real issue isn't who to kill, where either option forces you into unacceptable positions about the value of human lives, but how to prevent accidents in the first place. Above all, ethics must be realistic, and in our real world, bad things happen.

That's my rather abstract framework for an ethics of AI. I don't want to tell data scientists and AI developers what to do in any given situation. I want to give scientists and engineers tools for thinking about problems. We surely can't predict all the problems and ethical issues in advance; we need to be the kind of people who can have effective discussions about these issues as we anticipate and discover them.

Talking through some issues

What are some of the ethical questions that AI developers and researchers should be thinking about? Even though we're still in the earliest days of AI, we're already seeing important issues rise to the surface: issues about the kinds of people we want to be, and the kind of future we want to build. So, let's look at some situations that made the news.

Pedestrians and passengers

The self-driving car/grandmother versus babies thing is deeply foolish, but there's a variation of it that's very real. Should a self-driving car that's in an accident situation protect its passengers or the people outside the car? That's a question that is already being discussed in corporate board rooms, as it was at Mercedes recently, which decided that the company's duty was to protect the passengers rather than pedestrians. I suspect that Mercedes' decision was driven primarily by accounting and marketing: who will buy a car that will sacrifice the owner to avoid killing a pedestrian? But Mercedes made an argument that's at least ethically plausible: they have more control over what happens to the person inside the car, so better to save the passenger than to roll the dice on the pedestrians. One could also argue that Mercedes has an ethical committent to the passengers, who have put their lives in the hands of their AI systems.

The bigger issue is to design autonomous vehicles that can handle dangerous situations without accidents. That's the real ethical choice. How do you trade off cost, convenience, and safety? It's possible to make cars that are more safe or less safe; AI doesn't change that at all. It's impossible to make a car (or anything else) that's completely safe, at any price. So, the ethics here ultimately come down to a tradeoff between cost and safety, to ourselves and to others. How do we value others? Not grandmothers or babies (who will inevitably be victims, just as they are now, though hopefully in smaller numbers), but passengers and pedestrians, Mercedes' customers and non-customers? The answers to these questions aren't fixed, but they do say something important about who we are.

Crime and punishment

COMPAS is commercial software used in many state courts to recommend prison sentences, bail terms, and parole. In 2016, ProPublica published an excellent article showing that COMPAS consistently scores blacks as greater risks for re-offending than whites who committed similar or more serious crimes.

Although COMPAS has been secretive about the specifics of their software, ProPublica published the data on which their reports were based. Abe Gong, a data scientist, followed up with a multi-part study, using ProPublica's data, showing that the COMPAS results were not "biased." Abe is very specific: he means "biased" in a technical, statistical sense. Statistical bias is a statement about the relationship between the outputs (the risk scores) and the inputs (the data). It has little to do with whether we, as humans, think the outputs are fair.

Abe is by no means an apologist for COMPAS or its developers. As he says, "Powerful algorithms can be harmful and unfair, even when they're unbiased in a strictly technical sense." The results certainly had disproportionate effects that most of us would be uncomfortable with. In other words, they were "biased" in the non-technical sense. "Unfair" is a better word that doesn't bring in the trapping of statistics.

The output of a program reflects the data that goes into it. "Garbage in, garbage out" is a useful truism, especially for systems that build models based on terabytes of training data. Where does that data come from, and does it embody its own biases and prejudices? A program's analysis of the data may be unbiased, but if the data reflects arrests, and if police are more likely to arrest black suspects, while letting whites off with a warning, a statistically unbiased program will necessarily produce unfair results. The program also took into account factors that may be predictive, but that we might consider unfair: is it fair to set a higher bail because the suspect's parents separated soon after birth, or because the suspect didn't have access to higher education?

There's not a lot that we can do about bias in the data: arrest records are what they are, and we can't go back and un-arrest minority citizens. But there are other issues at stake here. As I've said before, I'm much more concerned about what happens behind closed doors than what happens in the open. Cathy O'Neil has frequently argued that secret algorithms and secret data models are the real danger. That's really what COMPAS shows. It is almost impossible to discuss whether a system is unfair if we don't know what the system is and how it works. We don't just need open data; we need to open up the models that are built from the data.

COMPAS demonstrates, first, that we need a discussion about fairness, and what that means. How do we account for the history that has shaped our statistics, a history that was universally unfair to minorities? How do we address bias when our data itself is biased? But we can't answer these questions if we don't also have a discussion about secrecy and openness. Openness isn't just nice; it's an ethical imperative. Only when we understand what the algorithms and the data are doing, can we take the next steps and build systems that are fair, not just statistically unbiased.

Child labor

One of the most penetrating remarks about the history of the internet is that it was "built on child labor." The IPv4 protocol suite, together with the first implementations of that suite, was developed in the 1980s, and was never intended for use as a public, worldwide, commercial network. It was released well before we understood what a 21st century public network would need. The developers couldn't forsee more than a few tens of thousands of computers on the internet; they didn't anticipate that it would be used for commerce, with stringent requirements for security and privacy; putting a system on the internet was difficult, requiring handcrafted static configuration files. Everything was immature; it was "child labor," technological babies doing adult work.

Now that we're in the first stages of deploying AI systems, the stakes are even higher. Technological readiness is an important ethical issue. But like any real ethical issue, it cuts both ways. If the public internet had waited until it was "mature," it probably would never have happened; if it had happened, it would have been an awful bureacratic mess, like the abandoned ISO-OSI protocols, and arguably no less problematic. Unleashing technological children on the world is irresponsible, but preventing those children from growing up is equally irresponsible.

To move that argument to the 21st century: my sense is that Uber is pushing the envelope too hard on autonomous vehicles. And we're likely to pay for that—in vehicles that perhaps aren't as safe as they should be, or that have serious security vulnerabilities. (In contrast, Google is being very careful, and that care may be why they've lost some key people to Uber.) But if you go to the other extreme and wait until autonomous vehicles are "safe" in every respect, you're likely to end up with nothing: the technology will never be deployed. Even if it is deployed, you will inevitably discover risk factors that you didn't forsee, and couldn't have forseen without real experience.

I'm not making an argument about whether autonomous vehicles, or any other AI, are ready to be deployed. I'm willing to discuss that, and if necessary, to disagree. What's more important is to realize that this discussion needs to happen. Readiness itself is an ethical issue, and one that we need to take seriously. Ethics isn't simply a matter of saying that any risk is acceptable, or (on the other hand) that no risk is acceptable. Readiness is an ethical issue precisely because it isn't obvious what the "right" answer is, or whether there is any "right" answer. Is it an "ethical gray area"? Yes, but that's precisely what ethics is about: discussing the gray areas.

The state of surveillance

In a chilling article, The Verge reports that police in Baltimore used a face identification application called Geofeedia, together with photographs shared on Instagram, Facebook, and Twitter, to identify and arrest protesters. The Verge's report is based on a more detailed analysis by the ACLU. Instagram and the other companies quickly terminated Geofeedia's account after the news went public, though they willingly provided the data before it was exposed by the press.

Applications of AI to criminal cases quickly get creepy. We should all be nervous about the consequences of building a surveillance state. People post pictures to Instagram without thinking of the consequences, even when they're at demonstrations. And, while it's easy to say "anything you post should be assumed to be public, so don't post anything that you wouldn't anyone to see," it's difficult, if not impossible, to think about all the contexts in which your posts can be put.

The ACLU suggests putting the burden on the social media companies: social media companies should have "clear, public, and transparent policies to prohibit developers from exploiting user data for surveillance." Unfortunately, this misses the point: just as you can't predict how your posts will be used or interpreted, who knows the applications to which software will be put? If we only have to worry about software that's designed for surveillance, our task is easy. It's more likely, though, that applications designed for innocent purposes, like finding friends in crowds, will become parts of surveillance suites.

The problem isn't so much the use or abuse of individual Facebook and Instagram posts, but the scale that's enabled by AI. People have always seen other people in crowds, and identified them. Law enforcement agencies have always done the same. What AI enables is identification at scale: matching thousands of photos from social media against photos from drivers' license databases, passport databases, and other sources, then taking the results and crossing them with other kinds of records. Suddenly, someone who participates in a demonstration can find themselves facing a summons over an old parking ticket. Data is powerful, and becomes much more powerful when you combine multiple data sources.

We don't want people to be afraid of attending public gatherings, or in terror that someone might take a photo of them. (A prize goes to anyone who can find me on the cover of Time. These things happen.) But it's also unreasonable to expect law enforcement to stick to methodologies from the 80s and earlier: crime has certainly moved on. So, we need to ask some hard questions—and "should law enforcement look at Instagram" is not one of them. How does automated face recognition at scale change the way we relate to each other, and are those changes acceptable to us? Where's the point at which AI becomes harassment? How will law enforcement agencies be held accountable for the use, and abuse, of AI technologies? Those are the ethical questions we need to discuss.

Our AIs are ourselves

Whether it's fear of losing jobs or fear of a superintelligence deciding that humans are no longer necessary, it's always been easy to conjure up fears of artificial intelligence.

But marching to the future in fear isn't going to end well. And unless someone makes some fantastic discoveries about the physics of time, we have no choice but to march into the future. For better or for worse, we will get the AI that we deserve. The bottom line of AI is simple: to build better AI, be better people.

That sounds trite, and it is trite. But it's also true. If we are unwilling to examine our prejudices, we will implement AI systems that are "unfair" even if they're statistically unbiased, merely because we won't have the interest to examine the data on which the system is trained. If we are willing to live under an authoritarian government, we will build AI systems that subject us to constant surveillance: not just through Instagrams of demonstrations, but in every interaction we take part in. If we're slaves to a fantasy of wealth, we won't object to entrepreneurs releasing AI systems before they're ready, nor will we object to autonomous vehicles that preferentially protect the lives of those wealthy enough to afford them.

But if we insist on open, reasoned discussion of the tradeoffs implicit in any technology; if we insist that both AI algorithms and models are open and public; and if we don't deploy technology that is grossly immature, but also don't suppress new technology because we fear it, we'll be able to have a healthy and fruitful relationship with the AIs we develop. We may not get what we want, but we'll be able to live with what we get.

Walt Kelly said it best, back in 1971: "we have met the enemy and he is us." In a nutshell, that's the future of AI. It may be the enemy, but only if we make it so. I have no doubt that AI will be abused and that "evil AI" (whatever that may mean) will exist. As Tim O'Reilly has argued, large parts of our economy are already managed by unintelligent systems that aren't under our control in any meaningful way. But evil AI won't be built by people who think seriously about their actions and the consequences of their actions. We don't need to forsee everything that might happen in the future, and we won't have a future if we refuse to take risks. We don't even need complete agreement on issues such as fairness, surveillance, openness, and safety. We do need to talk about these issues, and to listen to each other carefully and respectfully. If we think seriously about ethical issues and build these discussions into the process of developing AI, we'll come out OK.

To create better AI, we must be better people.

Simple Introduction To Linear Regression

2017-02-18T15:42:00.000+05:30

After knowing the relationship between two variables we may be interested in estimating (predicting) the value of one variable given the value of another. The variable predicted on the basis of other variables is called the “dependent” or the ‘explained’ variable and the other the ‘independent’ or the ‘predicting’ variable. The prediction is based on average relationship derived statistically by regression analysis. The equation, linear or otherwise, is called the regression equation or the explaining equation.

For example, if we know that advertising and sales are correlated we may find out expected amount of sales for a given advertising expenditure or the required amount of expenditure for attaining a given amount of sales.

The relationship between two variables can be considered between, say, rainfall and agricultural production, price of an input and the overall cost of product, consumer expenditure and disposable income. Thus, regression analysis reveals average relationship between two variables and this makes possible estimation or prediction.

Definition:

Regression is the measure of the average relationship between two or more variables in terms of the original units of the data.

Types Of Regression:

The regression analysis can be classified into:
a) Simple and Multiple
b) Linear and Non –Linear
c) Total and Partial

a) Simple and Multiple:

In case of simple relationship only two variables are considered, for example, the influence of advertising expenditure on sales turnover. In the case of multiple relationship, more than two variables are involved.

On this while one variable is a dependent variable the remaining variables are independent ones. For example, the turnover (y) may depend on advertising expenditure (x) and the income of the people (z). Then the functional relationship can be expressed as y = f (x,z).

b) Linear and Non-linear:

The linear relationships are based on straight-line trend, the equation of which has no-power higher than one. But, remember a linear relationship can be both simple and multiple. Normally a linear relationship is taken into account because besides its simplicity, it has a better predective value, a linear trend can be easily projected into the future. In the case of non-linear relationship curved trend lines are derived. The equations of these are parabolic.

c) Total and Partial:

In the case of total relationships all the important variables are considered. Normally, they take the form of a multiple relationships because most economic and business phenomena are affected by multiplicity of cases. In the case of partial relationship one or more variables are considered, but not all, thus excluding the influence of those not found relevant for a given purpose.

Linear Regression Equation:

If two variables have linear relationship then as the independent variable (X) changes, the dependent variable (Y) also changes. If the different values of X and Y are plotted, then the two straight lines of best fit can be made to pass through the plotted points. These two lines are known as regression lines. Again, these regression lines are based on two equations known as regression equations. These equations show best estimate of one variable for the known value of the other. The equations are linear.
Linear regression equation of Y on X is
Y = a + bX ……. (1)
And X on Y is X = a + bY……. (2) a, b are constants.

From (1) We can estimate Y for known value of X.
(2) We can estimate X for known value of Y.

Inside the Washington Post’s popularity prediction experiment

2017-02-18T11:10:00.001+05:30

In the distributed age, news organizations are likely to see their stories shared more widely, potentially reaching thousands of readers in a short amount of time. At the Washington Post, we asked ourselves if it was possible to predict which stories will become popular. For the Post newsroom, this would be an invaluable tool, allowing editors to more efficiently allocate resources to support a better reading experience and richer story package, adding photos, videos, links to related content, and more, in order to more deeply engage the new and occasional readers clicking through to a popular story.

Here’s a behind-the-scenes look at how we approached article popularity prediction.

Data science application: Article popularity prediction

There has not been much formal work in article popularity prediction in the news domain, which made this an open challenge. For our first approach to this task, Washington Post data scientists identified the most-viewed articles on five randomly selected dates, and then monitored the number of clicks they received within 30 minutes after being published. These clicks were used to predict how popular these articles would be in 24 hours.

Using the clicks 30 minutes after publishing yielded poor results. As an example, here are five very popular articles:

Figure 1. Credit: Shuguang Wang and Eui-Hong (Sam) Han, used with permission.

Figure 2. Credit: Shuguang Wang and Eui-Hong (Sam) Han, used with permission.

Figure 3. Credit: Shuguang Wang and Eui-Hong (Sam) Han, used with permission.

Figure 4. Credit: Shuguang Wang and Eui-Hong (Sam) Han, used with permission.

Figure 5. Credit: Shuguang Wang and Eui-Hong (Sam) Han, used with permission.

Table 1 lists the actual number of clicks these five articles received 30 minutes and 24 hours after being published. The takeaway: looking at how many clicks a story gets in the first 30 minutes is not an accurate way to measure its potential for popularity:

Table 1. Five popular articles.
Articles	# clicks @ 30mins	# clicks @ 24hours
9/11 Flag	6,245	67,028
Trump Policy	2,015	128,217
North Carolina	1,952	11,406
Hillary & Trump	1,733	310,702
Gary Johnson	1,318	196,798

Prediction features

In this prediction task, Washington Post data scientists have explored four groups of features: metadata, contextual, temporal, and social features. Metadata and contextual features, such as authors and readability, are extracted from the news articles themselves. Temporal features come mainly from an internal site-traffic collection system. Social features are statistics from social media sites, such as Twitter and Facebook.

Figure 6 lists all of the features we used in this prediction task. (More details about these features can be found in the paper, on which we we collaborated with Dr. Naren Ramakrishnan and Yaser Keneshloo from Discovery Analytics Center Virginia Tech, "Predicting the Popularity of News Articles.")

Figure 6. List of features used. Credit: Yaser Keneshloo, Shuguang Wang, Eui-Hong Han, Naren Ramakrishnan, used with permission.

Regression task

Figure 7 illustrates the process that we used to build regression models. In the training phase, we built several regression models using 41,000 news articles published by the Post. To predict the popularity of an article, we first collected all features within 30 minutes after its publication, and then used pre-trained models to predict its popularity in 24 hours.

Figure 7. Statistical Modeling. Credit: Shuguang Wang and Eui-Hong (Sam) Han, used with permission.

Evaluation

To measure the performance of the prediction task, we conducted two evaluations. First, we conducted a 10-fold cross validation experiment on the training articles. Table 2 enumerates the results of this evaluation. On average, the adjusted R2 is 79.4 (out of 100) with all features. At the same time, we realized that metadata information is the most useful feature aside from the temporal clickstream feature.

Table 2. 10-fold cross validation results.
Features	Predicted R2
Baseline	69.4
Baseline + Temporal	70.4
Baseline + Social	72.5
Baseline + Context	71.1
Baseline + Metadata	77.2
All	79.4

The second evaluation was done in production after we deployed the prediction system at the Post. Using all articles published in May 2016, we got an adjusted R2 of 81.3% (out of 100).

Figure 8 shows scatter plots of prediction results for articles published last May. The baseline system on the left uses a single feature: the total number of clicks at 30 minutes. On the right is a more complete system using all features listed in Figure 1. The red lines in each plot are the lower and upper error bounds. Each dot represents an article, and, of course, ideally all dots would fall within the error bounds. As you can see, there are many more errors in the baseline system.

Figure 8. Scatter plot of prediction results (May 2016). Credit: Shuguang Wang and Eui-Hong (Sam) Han, used with permission.

Production deployment

We built a very effective regression model to predict the popularity of news articles. The next step was to deploy it to production at the Post.

The prediction quality relies on the accuracy of features and speed to obtain them. It is preferred to build this prediction task as a streaming service to collect up-to-date features in real time. However, this comes with a challenge. We have to process tens of millions of points of click data every day to predict the popularity of thousands of Post articles. A streaming infrastructure facilitates the fast prediction task with minimal delays.

Architecture

Figure 9 illustrates the overall architecture of the prediction service in the production environment at the Post. Visitors who read news articles generate page view data, which is stored in a Kafka server and then fed into back-end Spark Streaming services. Other features such as metadata and social features are collected by separate services, and then fed into the same Spark Streaming services. With all these collected features, prediction is done with a pre-trained regression model, and results are stored to an HBase server and also forwarded to the Kafka server. The Post newsroom is also alerted of popular articles via Slack and email.

Figure 9. System Architecture. Credit: Shuguang Wang and Eui-Hong (Sam) Han, used with permission.

Spark Streaming in clickstream collection

The Spark Streaming framework is used in several components in our prediction service. Figure 10 illustrates one process we use Spark Streaming for: to collect and transform the clickstream (page view) data into prediction features.

The clickstream stored in Kafka is fed into the Spark Streaming framework in real time. The streaming process converts this real time stream into smaller batches of clickstream data. Each batch is converted into a simplified form, and then another format the pre-trained regression model can easily consume. In the end, page view features are stored in the HBase database.

Figure 10. Clickstream processing. Credit: Shuguang Wang and Eui-Hong (Sam) Han, used with permission.

System in the real world

Washington Post journalists monitor predictions using real-time Slack and email notifications. The predictions can be used to drive promotional decisions on the Post home page and social media channels.

We created a Slack bot to notify the newsroom if, 30 minutes after being published, an article is predicted to be extremely popular. Figure 11 shows Slack notifications with the current number and forecasted number of clicks in 24 hours.

Figure 11. Slack bot. Credit: Shuguang Wang and Eui-Hong (Sam) Han, used with permission.

We also automatically generate emails that gather that day’s predictions and summarize articles’ predicted and actual performance at the end of the day. Figure 12 shows an example of these emails. This email contains the publication time, predicted clicks, actual clicks in first 30 minutes, actual clicks in first 24 hours, and actual clicks from social media sites in first 30 minutes.

Figure 12. Popularity summary email. Credit: Shuguang Wang and Eui-Hong (Sam) Han, used with permission.

In addition to this being a tool for our newsroom, we are also integrating it into Washington Post advertising products such as PostPulse. PostPulse packages advertiser content with related editorial contents, and delivers a tailored, personalized advertisement to the target group. Figure 13 shows an example of this product in action, in which an advertiser’s video on 5G wireless technology is paired with editorially produced technology articles. A member of the advertising team puts the package together and receives candidate editorial articles as recommendations to include in the package. These articles are ranked according to relevance and expected popularity.

Figure 13. PostPulse example. Credit: Shuguang Wang and Eui-Hong (Sam) Han, used with permission.

Practical challenges

We faced two main challenges when we deployed this service to production. First is the scale of the data. Each day, we process a huge and increasing amount of data for prediction; the system must scale with it, using limited resources. We profile the service’s performance in terms of execution time, and identifying that persistent storage (HBase) is a significant bottleneck. Whenever we have to store or update HBase, it is expensive. To reduce this, we accumulate multiple updates before we physically update HBase. This runs the risk of some potential data loss and less accurate predictions if the service crashes between two updates to HBase. We’ve tuned the system and found a good balance so updates are not too delayed.

The second challenge is dependencies on external services, which we use to collect various features. However, if these external APIs are not reachable, the prediction service should still be available. Thus, we adopted a decoupled microservice infrastructure, in which each feature collection process is a separate microservice. If one or more microservices are down, the overall prediction service will still be available, just with reduced accuracy until these external services are back online.

Continuous experiments and future work

Moving forward, we will explore a few directions. First, we want to identify the time frame in which an article is expected to reach peak traffic (after running some initial experiments, the results are promising). Second, we want to extend our prediction to articles not published by the Washington Post. Last but not least, we want to address distribution biases in the prediction process. Articles can get much more attention when they are in a prominent position on our home page or spread through large channels on social media.

A Complete Report On Data Scientist Salary

2016-12-23T15:00:00.005+05:30

Executive Summary

O’Reilly Data Science Salary Survey, we’ve analyzed input from 983 respondents working in the data space, across a variety of industries— representing 45 countries and 45 US states. Through the results of our 64-question survey, we’ve explored which tools data scientists, analysts, and engineers use, which tasks they engage in, and of course—how much they make.

Key findings include:

Python and Spark are among the tools that contribute most to salary.
Among those who code, the highest earners are the ones who code the most.
SQL, Excel, R and Python are the most commonly used tools.
Those who attend more meetings, earn more.
Women make less than men, for doing the same thing.
Country and US state GDP serves as a decent proxy for geographic salary variation (not as a directestimate, but as an additional input for a model).
The most salient division between tool and tasks usage is between those who mostly use Excel, SQL, and a small number of closed source tools—and those who use more open source tools and spend more time coding.
R is used across this division: even people who don’t code much or use many open source tools, use R.
A secondary division emerges among the coding half— separating a younger, Python-heavy data scientist/analyst group, from a more experienced data scientist/engineer cohort that tends to use a high number of tools and earns the highest salaries.

To see our complete model and input your own metrics to predict salary, see Appendix B: The Regression Model (but beware—there’s a transformation involved: don’t forget to square the result!).

Introduction

O’Reilly Media have collected survey data from data scientists, engineers, and others in the data space, about their skills, tools, and salary. Across our four years of data, many key trends are more or less constant: median salaries, top tools, and correlations among tool usage. For this year’s analysis, we collected responses from September 2015 to June 2016, from 983 data professionals.

In this report, we provide some different approaches to the analysis, in particular conducting clustering on the respondents (not just tools). We have also adjusted the linear model for improved accuracy, using a square root transform and publicly available data on geographical variation in economies. The survey itself also included new questions, most notably about specific data-related tasks and anychange in salary.

Salary: The Big Picture

The median base salary of the entire sample was $87K. This figure is slightly lower than in previous years (last year it was $91K), but this discrepancy is fully attributable to shifts in demographics: this year’s sample had a higher share of non-US respondents and respondents aged 30 or younger. Three-fifths of the sample came from the US, and these respondents had a median salary of $106K.

Understanding Interquartile Range

For a number of survey questions, we show graphs of answer shares and the median salaries of respondents who gave particular answers. While median salary is probably the best number to compare how much two groups of people make, it doesn’t say anything about the spread or variation of salaries. In addition to median, we also show theinterquartile range (IQR)—two numbers that delineate salaries of the middle 50%. This range isnot a confidence interval, nor is it based on standard deviations.

As an example, the IQR for US respondents was $80K to $138K, meaning one quarter of US respondents had salaries lower than $80K and one quarter had salaries higher than $138K. Perhaps more illustrative of the value of the IQR is comparing the US Northeast and Midwest: the Northeast has a higher median salary ($105K vs. $98K) but the third quartile cutoffs are $133K for the Northeast and $138K for the Midwest. This indicates that there is generally more variation in Midwest salaries, and that among top earners—salaries might be even higher in the Midwest than in the Northeast.

How Salaries Change

We also collected data on salary change over the last three years. About half of the sample reported a 20% change, and the salary of 12% of the sampledoubled. We attempted to model salary change with other variables from the survey, but the model performed much more poorly, with an R2 of just 0.221. Many of the same significant features in the salary regression model also appeared as factors in predicted salary change: Spark/Unix, high meeting hours, high coding hours, and building prototype models, all predict higher salary growth, while using Excel, gender disparity, and working at an older company predict lower salary growth. Geography also correlated positively with salary change, meaning that

Assessing Your Salary

To use the model for you own salary, refer to the full model in Appendix B: The Regression Model, and add up the coefficients that apply to you. Once all of the constants are added, square the result for a final salary estimate (note: the coefficients are not in dollars). The contribution of a particular coefficient to the eventual salary estimate depends on the other coefficients: the higher the salary, the higher the contribution of each coefficient.

For example, the salary difference between a junior data scientist and a senior architect will be greater in a country with high salaries than somewhere with lower salaries.

Factors that Influence Salary: The Regression Model

WE HAVE INCLUDED OUR FULL regression model in Appendix B: The Regression Model. For this year’s report, we have made two important changes to the basic, parsimonious linear model we presented in the 2015 report. We have included: 1) external geographic data (GDP by US state and country), and 2) a square root transformation. The transformation adds one step to the linear model: we add up model coefficients, and then square the result. Both of these changes significantly improve the accuracy in salary estimates.

Our model explains about three-quarters of the variance in the sample salaries (with an R2 of 0.747). Roughly half of the salary variance is due to geography and experience. Given the important factors that can not be captured in the survey— for example, we don’t measure competence or evaluate the quality of respondents’ work output—it’s not surprising that a large amount of variance is left unexplained.

Impact of Geography

Geography has a huge impact on salary, but is not adequately captured due to sample size. For example, if a country is represented by only one or two respondents, this isn’t enough to justify giving the country its own coefficient. For this reason, we use broad regional coefficients (e.g., “Asia” or “Eastern Europe”), keeping in mind however that economic differences within a region are huge, and thus the accuracy of the model suffers.

To get around this problem, we’ve used publicly available records of per capita GDP of countries and US states. While GDP itself doesn’t translate to salary, it can serve a proxy function for geographic salary variation. Note that we use per capita GDP on the state and country level; therefore the model is likely to produce an inaccurate estimate with GDP figures for smaller geographic units.

Two exceptions were made to the GDP data before incorporating it into the model. The per capita GDP of Washington DC is $181K—much greater than in neighboring Virginia ($57K) and Maryland ($60K). Many (if not most) data science jobs in Maryland and Virginia are actually in the greater DC metropolitan area, and the survey data suggest that average data science salaries in these three places are not radically different from each other. Using the true $181K figure would produce gross overestimates for DC salaries, and so the per capita GDP figure for DC was replaced with that of Maryland, $60K.

The other exception is California. In all of the salary surveys we have conducted, California has had the highest median salary of any state or country, even though its per capita GDP ($62K) is not ranked so high (nine states have higher per capita GDPs, as do two countries that were represented in the sample, Switzerland and Norway). The anomaly is likely due to the San Francisco Bay Area, where, depending on how the region is defined, per capita GDP is $80K–$90K. As a major tech center, the Bay Area is likely overrepresented in the sample, meaning that the geographic factor attributable to California should be pushed upward; an appropriate compromise was $70K.

Considering Gender

There is a difference of $10K between the median salaries of men and women. Keeping all other variables constant—same roles, same skills—women make less than men.

Age, Experience, and Industry

Experience and age are two important variables that influence salary. The coefficient for experience (+3.8) translates to an increase of $2K–$2.5K on average, per year of experience. As for age, the biggest jump is between people in their early and late 20s, but the difference between those aged 31–65 and those over 65 is also significant.

We also asked respondents to rate their bargaining skills on a scale of 1 to 5, and those who gave higher self-evaluations tended to have higher salaries. The difference in salary between two data scientists, one with a bargaining skill “1” and the other with “5”, with otherwise identical demographics and skills, is expected to be $10K–$15K.

Finally, in terms of work-life balance, our results show that once you are working beyond 60 hours, salary estimates actually go down.

How You Spend Your Time

Importance of Tasks

The type of work respondents do was captured through four different types of questions:

involvement in specific tasks
job title
time spent in meetings
time spent coding

For every task, respondents chose from three options: no engagement, minor engagement, or major engagement.

The task with the greatest impact on salary (i.e., the greatest coefficient) was developing prototype models. Respondents who indicated major engagement with this task received on average a $7.4K boost, based on our model. Even minor engagement in developing prototype models had a +4.4 coefficient.

Relevance of Job Titles

When both tasks and job titles are included in the training set, job title “wins” as a better predictor of salary. It’s notable however, that titles themselves are not necessarily accurate at describing what people do. For example, even among architects there was only a 70% rate of major engagement inplanning large software projects—a task that theoretically defines the role. Since job title does perform well as a salary predictor, despite this inconsistency, it may be that “architect,” for example, is a symbol of seniority as much as anything else.

Respondents with “upper management” titles—mostly C-level executives at smaller companies, directors and VPs—had a huge coefficient of +20.2. Engagement in tasks associated with managerial roles also had a positive impact on salary, namely: organizing team projects (+9.7), identifying business problems to be solved with analytics (+1.5/+6.7), and communicating with people outside the company (+5.4).

Time Spent in Meetings

People who spend more time in meetings tend to make more. This is the variable we often use as a reminder that the model does not guarantee that the relationships between significant variables and salary are causative: if someone starts scheduling many meetings (and doesn’t change anything else in their workday) it is unlikely that this will lead to anything positive, much less a raise.1

Role of Coding

The highest median salaries belong to those who code 4–8 hours per week; the lowest to those who don’t code at all. Notably, only 8% of the sample reported that they don’t code at all, significantly down from last year’s 20%. Coding is clearly an integral part of being a data scientist.

1Of course, we haven’t actually tested this. If you try it out, let us know how it goes.

The Impact of Tool Choice

The Top Tools

The top two tools in the sample were Excel and SQL, both with use by 69% of the sample, followed by R (57%) and Python (54%). Compared to last year, Excel is up (from 59%), as is R (from 52%), while SQL and Python are only slightly higher than last year.

Over 90% of the sample reported spending at least some time coding, and 80% used at least one of Python, R, and Java, although only 8% used all three. The most commonly used tools (except for operating systems) were included in the model training data as individual coefficients; of these, Python, JavaScript, and Excel had significant coefficients: +4.6, –2.2 and –7.4, respectively. Less commonly used tools were first grouped together into clusters and aggregate features were included that represent counts of tools used from each cluster. For five clusters that were found to have a significant correlation with salary, coefficients are added on a per-tool basis.2

The cluster with the largest coefficient was centered on Spark and Unix, contributing +3.9 per tool. Spark usage was 20%, up from last year’s a modest 3%, and it continues to be used by the more well paid individuals in the sample.

In contrast to the largely open source Spark/Unix cluster, the second highest cluster coefficient (+2.4) was assigned to a cluster dominated by proprietary software: Tableau, Teradata, Netezza, Microstrategy, Aster Data, and Jaspersoft. In last year’s report, Teradata also featured as a tool with a large, positive coefficient. The other three clusters with significant coefficients mostly consisted of open source data tools.

Which Tools to Add to Your Stack

While the model we’ve explained is a good way to get an estimate for how much someone earns given a certain tool stack, it doesn’t necessarily work as a good guide for which tool to learn next. The real question is whether a tool is useful for getting done what you need to get done. If you never have to analyze more data than can fit into memory on your local machine, you might not get any benefit—much less a salary boost—by using a tool that leverages distributed systems, for example.

Salary and Sequences of Tools

In the following sequences of tools, the next tool in the sequence was frequently used by respondents who used all earlier tools, and these sequences had the best salary differentials at each step.

If you know the first tool in a sequence, you might consider learning the second, and so on.

2Tools are added up to a maximum number. This is because few respondents had more than that number of tools from the cluster, and so if someone uses more, there is no evidence to support continued addition of coefficients.

The Relationship Between Tools and Tasks: Clustering Respondents

DATA PROFESSIONALS ARE NOT A homogenous group— there are various types of roles in the space. While it is easier—and more common—to classify roles based on titles, clustering based on tools and tasks is a more rigorous way to define the key divisions between respondents of the survey. Every respondent is assigned to one of four clusters based on their tools and tasks.3

The four clusters were not evenly populated: their shares of the survey sample were 29%, 31%, 23%, and 17%, respectively. They can be described as shown on the right.

A selection of tool and task percentages are described in the sections that follow, and the full profiles of tool/task percentages are found in Appendix A: Full Cluster Profiles.

Operating Systems

In our three previous Data Science Salary Surveyreports, the clearest division in tool clusters separated one group of open source, usually GUI-less tools, from another consisting of proprietary software, largely developed by Microsoft. Common tools in the open source group have been Linux, Python, Spark, Hadoop, and Java, and common tools in the Microsoft/ closed source group include Windows, Excel, Visual Basic, and MS SQL Server. This same division appears when we clusterrespondents, and is clearest when we look at the usage of operating systems:

A set of tasks also emphasize the division between the first two and last two clusters. The following percentages represent respondents who indicatedmajor engagement in these tasks:

For all of the above tasks, the top two percentages were held by clusters 3 or 4 and were both much higher than either percentage for clusters 1 and 2.

Python, Matplotlib, Scikit-Learn

Another set of tools that exposed the primary split between clusters 1/2 and 3/4 are Python and two of its popular packages, Matplotlib (for visualization) and Scikit-Learn (for machine learning):

Survey respondents assigned to clusters 3 and 4 tend to use Python much more than those assigned to 1 and 2, and the relative difference (as a ratio) grows when we look at the two packages: cluster 3 and 4 respondents are 8–10 times as likely to use them as cluster 1 and 2 respondents. Between clusters 3 and 4 there is a difference as well, albeit more minor: cluster 3 has a higher Python usage rate, while a larger share of cluster 4 respondents don’t use Python or these packages. It turns out that these are the only tools whose highest usage rate is among cluster 3 respondents.4 For most other tools that are used much more frequently by clusters 3 and 4 than by 1 and 2, they are also used more frequently by cluster 4 than by cluster 3.

Cluster 4 rates for two tasks also stand out:

Cluster 4, it seems, is much more of an “open source data engineer” descriptor than cluster 3, which heads in that direction but not nearly to the same extent. It’s not rare for cluster 3 respondents to have used these tools—86% of them used at least one—but on average they only used about 2.2. In comparison, respondents in cluster 4 used an average of 5.3 tools. The fact that ETL and data management are much more important in cluster 4 than cluster 3, implies that while both might represent data science, cluster 3 tends toward the analyst’s side of the field, and cluster 4 tends toward the engineering or architecture side.

As for the other two clusters, differences between clusters 1 and 2 become apparent once we look at the rest of the aforementioned proprietary tool set. Cluster 2 respondents tended to use these much more frequently.

For most of tools shown below, cluster 1 has the second highest usage rate, but they significantly lag behind those of cluster 2. Cluster 1 respondents tended to use fewer tools in general: just under 8 on average, compared to 10, 13, and 21 for the three other clusters, respectively.

Tasks Without Coding

There are also some tasks that are undertaken by cluster 2 respondents significantly more frequently than those in other clusters:

The first two tasks are functions of an analyst, and are fairly common among cluster 3 and 4 respondents as well. Crucially, none of these tasks depend on being able to code (at least, not as much as the four tasks above that are closely associated with clusters 3 and 4). The low percentages for cluster 1 sheds some light on the nature of this cluster: most respondents in the sample whose primary function is not as a data scientist, analyst, or manager seem to be grouped there. This includes programmers who aren’t deep in the space (e.g., Java programmers who only use a few data tools). There are analysts and data scientists in cluster 1, but they tend to have small tool sets, and the composite feature of non-participation in many data tasks and non-use of data tools is what binds cluster 1 together.

Some of the proprietary tools listed above are used by respondents in cluster 4 about as much as those in cluster 1, most notably SQL Server. In other words, they begin to violate the primary cluster 1/2 vs. 3/4 split. A few other tools and tasks take this pattern even further, or simply don’t show large usage differences between clusters:

Tableau, Oracle, Teradata, and Oracle BI usage is higher in clusters 2 and 4, lower in clusters 1 and 3. The same is true for SQL, but like Excel and R, it’s exceptional in its wide usage across all four clusters. In fact, SQL and Excel are the only two tools (or tasks) that are used by over half of the respondents in each cluster. R is not used as much by cluster 1, but usage among the other three clusters is about the same: 67%– 69%. Data cleaning and basic exploratory analysis are similarly high for clusters 2, 3, and 4, and much lower for cluster 1. These tasks and tools cut across the cluster boundaries, and don’t seem to have much correlation with the more salient tool/task differences.

Managerial and Business Strategy Tasks

Perhaps even more illustrative of the connection between clusters 2 and 4 are the managerial/business strategy tasks. The implication is that respondents in 2/4 tend to be more senior, which turns out to be true, but only to an extent. In terms of years of experience, clusters 1, 2, and 4 are about the same—8–9 years on average—while for the cluster 3, the average is much smaller: only 4.4 years; a similar difference exists for age.

Despite representing the least experienced cohort, cluster 3 isn’t the lowest paid; that distinction goes to cluster 1, with a median salary of $72K. At $84K, cluster 3 is still lower than cluster 2 ($88K), but cluster 4 salaries tended to be far higher than either, with a median of $112K. Cluster 4 respondents tend to use a far greater number of tools than respondents in the other clusters, and many of the tools they commonly use are ones that had positive coefficients in the regression model.

3We tried a variety of clustering algorithms with various numbers of clusters, and the two best performing models came from KMeans, with two and four clusters. The partition in the 2-cluster model is more or less preserved in the 4-cluster model, so we will use the latter, keeping in mind that there is a primary split between the first two and last two clusters.

4Excluding tools that didn’t have a significant difference between the top two percentages: Mac OS X, ggplot, Vertica, and Stata.

Wrapping Up: What to Consider Next

THE REGRESSION MODEL WE USE to predict salary describes relationships between variables, but not where the relationships come from, or whether they are directly causative. For example, someone might work for a company with a colossal budget that can afford high salaries and expensive tools, but this doesn’t mean that their high salary is driven up by their tool choice.

Of course, it’s not so simple with salary. When tools become industry standards, employers begin to expect them, and it can hurt your chances of landing a good job if you are missing key tools: it’s in your interest to keep up with new technology. If you apply for a job at a company that is clearly interested in hiring someone who knows a certain tool, and this tool is used by people who earn high salaries, then you have leverage knowing that it will be hard for them to find an alternative hire without paying a premium.

This information isn’t just for the employees, either. Business leaders choosing technologies need to consider not just the software costs, but labor expenses as well. We hope that the information in this report will aid the task of building estimates for such decisions.

If you made use of this report, please consider taking the 2017 survey. Every year we work to build on the last year’s report, and much of the improvement comes from increased sample sizes. This is a joint research effort, and the more interaction we have with you, the deeper we will be able to explore the data science space. Thank you!

Appendix A: Full Cluster Profiles

Appendix B: The Regression Model

+60.0 Constant: everyone starts with this number

+2.6 Multiply by per capita GDP, in thousands (e.g., for Iowa, 2.6 * 52.8 = 137.28)

-7.8 gender = Female

+3.8 Per year of experience

+7.4 Per bargaining skill “point”

+17.2 Age: 26 to 30

+22.5 Age: 31 to 35

+24.8 Age: 36 to 65

+38.5 Age: over 65

+3.9 Academic speciality is/was mathematics, statistics or physics

+12.2 PhD

-9.7 Currently a student (full- or part-time, any level)

+2.2 industry = Software (incl. SaaS, Web, Mobile)

+3.0 industry = Banking/Finance

-2.0 industry = Advertising/Marketing/PR

-24.5 industry = Education

-3.9 industry = Computers/Hardware

+7.1 industry = Search/Social Networking

+3.6 Company size: 501 to 10,000

+7.7 Company size: 10,000 or more

-4.3 Company age: over 10 years old

-8.2 Coding: 1 to 3 hours/week

–3.0 Coding: 4 to 20 hours/week

–0.5 Coding: Over 20 hours/week

+1.0 Meetings: 1 to 3 hours/week

+9.2 Meetings: 4 to 8 hours/week

+20.6 Meetings: 9 to 20 hours/week

+21.1 Meetings: Over 20 hours/week

+1.0 Workweek: 46 to 60 hours

–2.4 Workweek: Over 60 hours

+20.2 Job title: Upper Management

-0.9 Job title: Engineer/Developer/Programmer

+3.1 Job title: Manager

-1.0 Job title: Researcher

+14.3 Job title: Architect

+4.6 Job title: Senior Engineer/Developer

+4.5 ETL (minor involvement)

-1.9 ETL (major involvement)

-4.9 Setting up/maintaining data platforms (minor involvement)

+4.4 Developing prototype models (minor involvement)

+12.1 Developing prototype models (major involvement)

-1.3 Developing hardware, or working on projects that

require expert knowledge of hardware (major)

+9.7 Organizing and guiding team projects (major)

+1.5 Identifying business problems to be solved with analytics (minor)

+6.7 Identifying business problems to be solved with

analytics (major)

+5.4 Communicating with people outside your company

(major)

+3.2 Most or all on work done using cloud computing

+4.6 Python

-2.2 JavaScript

-7.4 Excel

+1.7 for each of MySQL, PostgreSQL, SQLite, Redshift,

Vertica, Redis, Ruby (up to 4 tools)

+3.9 for each of Spark, Unix, Spark MlLib, ElasticSearch, Scala, H2O, EMC/Greenplum, Mahout (up to 5 tools)

+1.5 for each of Hive, Apache Hadoop, Cloudera, Hortonworks, Hbase, Pig, Impala (up to 5 tools)

+2.4 for each of Tableau, Teradata, Netezza (IBM), Microstrategy, Aster Data (Teradata), Jaspersoft (up to 3 tools)

+1.3 for each of MongoDB, Kafka, Cassandra, Zookeeper, Storm, JavaScript InfoVis Toolkit, Go, Couchbase (up to 4 tools)

Article image: The Seven Virtues, by Brueghel, published by Philippe Galle.(source: Wikimedia Commons).

Essentials of Machine Learning Algorithms (with Python and R Codes)

2016-12-17T19:53:00.000+05:30

Broadly, there are 3 types of Machine Learning Algorithms..

1. Supervised Learning

How it works: This algorithm consist of a target / outcome variable (or dependent variable) which is to be predicted from a given set of predictors (independent variables). Using these set of variables, we generate a function that map inputs to desired outputs. The training process continues until the model achieves a desired level of accuracy on the training data. Examples of Supervised Learning: Regression,Decision Tree, Random Forest, KNN, Logistic Regression etc.

2. Unsupervised Learning

How it works: In this algorithm, we do not have any target or outcome variable to predict / estimate. It is used for clustering population in different groups, which is widely used for segmenting customers in different groups for specific intervention. Examples of Unsupervised Learning: Apriori algorithm, K-means.

3. Reinforcement Learning:

How it works: Using this algorithm, the machine is trained to make specific decisions. It works this way: the machine is exposed to an environment where it trains itself continually using trial and error. This machine learns from past experience and tries to capture the best possible knowledge to make accurate business decisions. Example of Reinforcement Learning: Markov Decision Process

List of Common Machine Learning Algorithms

Here is the list of commonly used machine learning algorithms. These algorithms can be applied to almost any data problem:

Linear Regression
Logistic Regression
Decision Tree
SVM
Naive Bayes
KNN
K-Means
Random Forest
Dimensionality Reduction Algorithms
Gradient Boost & Adaboost

1. Linear Regression

It is used to estimate real values (cost of houses, number of calls, total sales etc.) based on continuous variable(s). Here, we establish relationship between independent and dependent variables by fitting a best line. This best fit line is known as regression line and represented by a linear equation Y= a *X + b.

The best way to understand linear regression is to relive this experience of childhood. Let us say, you ask a child in fifth grade to arrange people in his class by increasing order of weight, without asking them their weights! What do you think the child will do? He / she would likely look (visually analyze) at the height and build of people and arrange them using a combination of these visible parameters. This is linear regression in real life! The child has actually figured out that height and build would be correlated to the weight by a relationship, which looks like the equation above.

In this equation:

Y – Dependent Variable
a – Slope
X – Independent variable
b – Intercept

These coefficients a and b are derived based on minimizing the sum of squared difference of distance between data points and regression line.

Look at the below example. Here we have identified the best fit line having linear equationy=0.2811x+13.9. Now using this equation, we can find the weight, knowing the height of a person.

Linear Regression is of mainly two types: Simple Linear Regression and Multiple Linear Regression. Simple Linear Regression is characterized by one independent variable. And, Multiple Linear Regression(as the name suggests) is characterized by multiple (more than 1) independent variables. While finding best fit line, you can fit a polynomial or curvilinear regression. And these are known as polynomial or curvilinear regression.

Python Code

#Import Library
#Import other necessary libraries like pandas, numpy...
from sklearn import linear_model
#Load Train and Test datasets
#Identify feature and response variable(s) and values must be numeric and numpy arrays
x_train=input_variables_values_training_datasets
y_train=target_variables_values_training_datasets
x_test=input_variables_values_test_datasets
# Create linear regression object
linear = linear_model.LinearRegression()
# Train the model using the training sets and check score
linear.fit(x_train, y_train)
linear.score(x_train, y_train)
#Equation coefficient and Intercept
print('Coefficient: \n', linear.coef_)
print('Intercept: \n', linear.intercept_)
#Predict Output
predicted= linear.predict(x_test)

R Code

#Load Train and Test datasets
#Identify feature and response variable(s) and values must be numeric and numpy arrays
x_train <- input_variables_values_training_datasets
y_train <- target_variables_values_training_datasets
x_test <- input_variables_values_test_datasets
x <- cbind(x_train,y_train)
# Train the model using the training sets and check score
linear <- lm(y_train ~ ., data = x)
summary(linear)
#Predict Output
predicted= predict(linear,x_test)

2. Logistic Regression

Don’t get confused by its name! It is a classification not a regression algorithm. It is used to estimate discrete values ( Binary values like 0/1, yes/no, true/false ) based on given set of independent variable(s). In simple words, it predicts the probability of occurrence of an event by fitting data to a logit function. Hence, it is also known as logit regression. Since, it predicts the probability, its output values lies between 0 and 1 (as expected).

Again, let us try and understand this through a simple example.

Let’s say your friend gives you a puzzle to solve. There are only 2 outcome scenarios – either you solve it or you don’t. Now imagine, that you are being given wide range of puzzles / quizzes in an attempt to understand which subjects you are good at. The outcome to this study would be something like this – if you are given a trignometry based tenth grade problem, you are 70% likely to solve it. On the other hand, if it is grade fifth history question, the probability of getting an answer is only 30%. This is what Logistic Regression provides you.

Coming to the math, the log odds of the outcome is modeled as a linear combination of the predictor variables.

odds= p/ (1-p) = probability of event occurrence / probability of not event occurrence
ln(odds) = ln(p/(1-p))
logit(p) = ln(p/(1-p)) = b0+b1X1+b2X2+b3X3....+bkXk

Above, p is the probability of presence of the characteristic of interest. It chooses parameters that maximize the likelihood of observing the sample values rather than that minimize the sum of squared errors (like in ordinary regression).

Now, you may ask, why take a log? For the sake of simplicity, let’s just say that this is one of the best mathematical way to replicate a step function. I can go in more details, but that will beat the purpose of this article.

Python Code

#Import Library
from sklearn.linear_model import LogisticRegression
#Assumed you have, X (predictor) and Y (target) for training data set and x_test(predictor) of test_dataset
# Create logistic regression object
model = LogisticRegression()
# Train the model using the training sets and check score
model.fit(X, y)
model.score(X, y)
#Equation coefficient and Intercept
print('Coefficient: \n', model.coef_)
print('Intercept: \n', model.intercept_)
#Predict Output
predicted= model.predict(x_test)

R Code

x <- cbind(x_train,y_train)
# Train the model using the training sets and check score
logistic <- glm(y_train ~ ., data = x,family='binomial')
summary(logistic)
#Predict Output
predicted= predict(logistic,x_test)

Furthermore..

There are many different steps that could be tried in order to improve the model:

including interaction terms
removing features
regularization techniques
using a non-linear model

3. Decision Tree

This is one of my favorite algorithm and I use it quite frequently. It is a type of supervised learning algorithm that is mostly used for classification problems. Surprisingly, it works for both categorical and continuous dependent variables. In this algorithm, we split the population into two or more homogeneous sets. This is done based on most significant attributes/ independent variables to make as distinct groups as possible..

source: statsexchange

In the image above, you can see that population is classified into four different groups based on multiple attributes to identify ‘if they will play or not’. To split the population into different heterogeneous groups, it uses various techniques like Gini, Information Gain, Chi-square, entropy.

The best way to understand how decision tree works, is to play Jezzball – a classic game from Microsoft (image below). Essentially, you have a room with moving walls and you need to create walls such that maximum area gets cleared off with out the balls.

So, every time you split the room with a wall, you are trying to create 2 different populations with in the same room. Decision trees work in very similar fashion by dividing a population in as different groups as possible.

Python Code

#Import Library
#Import other necessary libraries like pandas, numpy...
from sklearn import tree
#Assumed you have, X (predictor) and Y (target) for training data set and x_test(predictor) of test_dataset
# Create tree object 
model = tree.DecisionTreeClassifier(criterion='gini') # for classification, here you can change the algorithm as gini or entropy (information gain) by default it is gini  
# model = tree.DecisionTreeRegressor() for regression
# Train the model using the training sets and check score
model.fit(X, y)
model.score(X, y)
#Predict Output
predicted= model.predict(x_test)

R Code

library(rpart)
x <- cbind(x_train,y_train)
# grow tree 
fit <- rpart(y_train ~ ., data = x,method="class")
summary(fit)
#Predict Output 
predicted= predict(fit,x_test)

4. SVM (Support Vector Machine)

It is a classification method. In this algorithm, we plot each data item as a point in n-dimensional space (where n is number of features you have) with the value of each feature being the value of a particular coordinate.

For example, if we only had two features like Height and Hair length of an individual, we’d first plot these two variables in two dimensional space where each point has two co-ordinates (these co-ordinates are known as Support Vectors)

Now, we will find some line that splits the data between the two differently classified groups of data. This will be the line such that the distances from the closest point in each of the two groups will be farthest away.

In the example shown above, the line which splits the data into two differently classified groups is theblack line, since the two closest points are the farthest apart from the line. This line is our classifier. Then, depending on where the testing data lands on either side of the line, that’s what class we can classify the new data as.

Think of this algorithm as playing JezzBall in n-dimensional space. The tweaks in the game are:

You can draw lines / planes at any angles (rather than just horizontal or vertical as in classic game)
The objective of the game is to segregate balls of different colors in different rooms.
And the balls are not moving.

Python Code

#Import Library
from sklearn import svm
#Assumed you have, X (predictor) and Y (target) for training data set and x_test(predictor) of test_dataset
# Create SVM classification object 
model = svm.svc() # there is various option associated with it, this is simple for classification. You can refer link, for mo# re detail.
# Train the model using the training sets and check score
model.fit(X, y)
model.score(X, y)
#Predict Output
predicted= model.predict(x_test)

R Code

library(e1071)
x <- cbind(x_train,y_train)
# Fitting model
fit <-svm(y_train ~ ., data = x)
summary(fit)
#Predict Output 
predicted= predict(fit,x_test)

5. Naive Bayes

It is a classification technique based on Bayes’ theorem with an assumption of independence between predictors. In simple terms, a Naive Bayes classifier assumes that the presence of a particular feature in a class is unrelated to the presence of any other feature. For example, a fruit may be considered to be an apple if it is red, round, and about 3 inches in diameter. Even if these features depend on each other or upon the existence of the other features, a naive Bayes classifier would consider all of these properties to independently contribute to the probability that this fruit is an apple.

Naive Bayesian model is easy to build and particularly useful for very large data sets. Along with simplicity, Naive Bayes is known to outperform even highly sophisticated classification methods.

Bayes theorem provides a way of calculating posterior probability P(c|x) from P(c), P(x) and P(x|c). Look at the equation below:

Here,

P(c|x) is the posterior probability of class (target) given predictor (attribute).
P(c) is the prior probability of class.
P(x|c) is the likelihood which is the probability of predictor given class.
P(x) is the prior probability of predictor.

Example: Let’s understand it using an example. Below I have a training data set of weather and corresponding target variable ‘Play’. Now, we need to classify whether players will play or not based on weather condition. Let’s follow the below steps to perform it.

Step 1: Convert the data set to frequency table

Step 2: Create Likelihood table by finding the probabilities like Overcast probability = 0.29 and probability of playing is 0.64.

Step 3: Now, use Naive Bayesian equation to calculate the posterior probability for each class. The class with the highest posterior probability is the outcome of prediction.

Problem: Players will pay if weather is sunny, is this statement is correct?

We can solve it using above discussed method, so P(Yes | Sunny) = P( Sunny | Yes) * P(Yes) / P (Sunny)

Here we have P (Sunny |Yes) = 3/9 = 0.33, P(Sunny) = 5/14 = 0.36, P( Yes)= 9/14 = 0.64

Now, P (Yes | Sunny) = 0.33 * 0.64 / 0.36 = 0.60, which has higher probability.

Naive Bayes uses a similar method to predict the probability of different class based on various attributes. This algorithm is mostly used in text classification and with problems having multiple classes.

Python Code

#Import Library
from sklearn.naive_bayes import GaussianNB
#Assumed you have, X (predictor) and Y (target) for training data set and x_test(predictor) of test_dataset
# Create SVM classification object model = GaussianNB() # there is other distribution for multinomial classes like Bernoulli Naive Bayes,
# Train the model using the training sets and check score
model.fit(X, y)
#Predict Output
predicted= model.predict(x_test)

R Code

library(e1071)
x <- cbind(x_train,y_train)
# Fitting model
fit <-naiveBayes(y_train ~ ., data = x)
summary(fit)
#Predict Output 
predicted= predict(fit,x_test)

6. KNN (K- Nearest Neighbors)

It can be used for both classification and regression problems. However, it is more widely used in classification problems in the industry. K nearest neighbors is a simple algorithm that stores all available cases and classifies new cases by a majority vote of its k neighbors. The case being assigned to the class is most common amongst its K nearest neighbors measured by a distance function.

These distance functions can be Euclidean, Manhattan, Minkowski and Hamming distance. First three functions are used for continuous function and fourth one (Hamming) for categorical variables. If K = 1, then the case is simply assigned to the class of its nearest neighbor. At times, choosing K turns out to be a challenge while performing KNN modeling.

KNN can easily be mapped to our real lives. If you want to learn about a person, of whom you have no information, you might like to find out about his close friends and the circles he moves in and gain access to his/her information!

Things to consider before selecting KNN:

KNN is computationally expensive
Variables should be normalized else higher range variables can bias it
Works on pre-processing stage more before going for KNN like outlier, noise removal

Python Code

#Import Library
from sklearn.neighbors import KNeighborsClassifier
#Assumed you have, X (predictor) and Y (target) for training data set and x_test(predictor) of test_dataset
# Create KNeighbors classifier object model 
KNeighborsClassifier(n_neighbors=6) # default value for n_neighbors is 5
# Train the model using the training sets and check score
model.fit(X, y)
#Predict Output
predicted= model.predict(x_test)

R Code

library(knn)
x <- cbind(x_train,y_train)
# Fitting model
fit <-knn(y_train ~ ., data = x,k=5)
summary(fit)
#Predict Output 
predicted= predict(fit,x_test)

7. K-Means

It is a type of unsupervised algorithm which solves the clustering problem. Its procedure follows a simple and easy way to classify a given data set through a certain number of clusters (assume k clusters). Data points inside a cluster are homogeneous and heterogeneous to peer groups.

Remember figuring out shapes from ink blots? k means is somewhat similar this activity. You look at the shape and spread to decipher how many different clusters / population are present!

How K-means forms cluster:

K-means picks k number of points for each cluster known as centroids.
Each data point forms a cluster with the closest centroids i.e. k clusters.
Finds the centroid of each cluster based on existing cluster members. Here we have new centroids.
As we have new centroids, repeat step 2 and 3. Find the closest distance for each data point from new centroids and get associated with new k-clusters. Repeat this process until convergence occurs i.e. centroids does not change.

How to determine value of K:

In K-means, we have clusters and each cluster has its own centroid. Sum of square of difference between centroid and the data points within a cluster constitutes within sum of square value for that cluster. Also, when the sum of square values for all the clusters are added, it becomes total within sum of square value for the cluster solution.

We know that as the number of cluster increases, this value keeps on decreasing but if you plot the result you may see that the sum of squared distance decreases sharply up to some value of k, and then much more slowly after that. Here, we can find the optimum number of cluster.

Python Code

#Import Library
from sklearn.cluster import KMeans
#Assumed you have, X (attributes) for training data set and x_test(attributes) of test_dataset
# Create KNeighbors classifier object model 
k_means = KMeans(n_clusters=3, random_state=0)
# Train the model using the training sets and check score
model.fit(X)
#Predict Output
predicted= model.predict(x_test)

R Code

library(cluster)
fit <- kmeans(X, 3) # 5 cluster solution

8. Random Forest

Random Forest is a trademark term for an ensemble of decision trees. In Random Forest, we’ve collection of decision trees (so known as “Forest”). To classify a new object based on attributes, each tree gives a classification and we say the tree “votes” for that class. The forest chooses the classification having the most votes (over all the trees in the forest).

Each tree is planted & grown as follows:

If the number of cases in the training set is N, then sample of N cases is taken at random but with replacement. This sample will be the training set for growing the tree.
If there are M input variables, a number m<<M is specified such that at each node, m variables are selected at random out of the M and the best split on these m is used to split the node. The value of m is held constant during the forest growing.
Each tree is grown to the largest extent possible. There is no pruning.

Python

#Import Library
from sklearn.ensemble import RandomForestClassifier
#Assumed you have, X (predictor) and Y (target) for training data set and x_test(predictor) of test_dataset
# Create Random Forest object
model= RandomForestClassifier()
# Train the model using the training sets and check score
model.fit(X, y)
#Predict Output
predicted= model.predict(x_test)

R Code

library(randomForest)
x <- cbind(x_train,y_train)
# Fitting model
fit <- randomForest(Species ~ ., x,ntree=500)
summary(fit)
#Predict Output 
predicted= predict(fit,x_test)

9. Dimensionality Reduction Algorithms

In the last 4-5 years, there has been an exponential increase in data capturing at every possible stages. Corporates/ Government Agencies/ Research organisations are not only coming with new sources but also they are capturing data in great detail.

For example: E-commerce companies are capturing more details about customer like their demographics, web crawling history, what they like or dislike, purchase history, feedback and many others to give them personalized attention more than your nearest grocery shopkeeper.

As a data scientist, the data we are offered also consist of many features, this sounds good for building good robust model but there is a challenge. How’d you identify highly significant variable(s) out 1000 or 2000? In such cases, dimensionality reduction algorithm helps us along with various other algorithms like Decision Tree, Random Forest, PCA, Factor Analysis, Identify based on correlation matrix, missing value ratio and others.

Python Code

#Import Library
from sklearn import decomposition
#Assumed you have training and test data set as train and test
# Create PCA obeject pca= decomposition.PCA(n_components=k) #default value of k =min(n_sample, n_features)
# For Factor analysis
#fa= decomposition.FactorAnalysis()
# Reduced the dimension of training dataset using PCA
train_reduced = pca.fit_transform(train)
#Reduced the dimension of test dataset
test_reduced = pca.transform(test)

R Code

library(stats)
pca <- princomp(train, cor = TRUE)
train_reduced  <- predict(pca,train)
test_reduced  <- predict(pca,test)

10. Gradient Boosting & AdaBoost

GBM & AdaBoost are boosting algorithms used when we deal with plenty of data to make a prediction with high prediction power. Boosting is an ensemble learning algorithm which combines the prediction of several base estimators in order to improve robustness over a single estimator. It combines multiple weak or average predictors to a build strong predictor. These boosting algorithms always work well in data science competitions like Kaggle, AV Hackathon, CrowdAnalytix.

Python Code

#Import Library
from sklearn.ensemble import GradientBoostingClassifier
#Assumed you have, X (predictor) and Y (target) for training data set and x_test(predictor) of test_dataset
# Create Gradient Boosting Classifier object
model= GradientBoostingClassifier(n_estimators=100, learning_rate=1.0, max_depth=1, random_state=0)
# Train the model using the training sets and check score
model.fit(X, y)
#Predict Output
predicted= model.predict(x_test)

R Code

library(caret)
x <- cbind(x_train,y_train)
# Fitting model
fitControl <- trainControl( method = "repeatedcv", number = 4, repeats = 4)
fit <- train(y ~ ., data = x, method = "gbm", trControl = fitControl,verbose = FALSE)
predicted= predict(fit,x_test,type= "prob")[,2] .

AI Careers: Skills to Get Artificial Intelligence Jobs

2016-11-28T20:23:00.001+05:30

As we can see from the history of artificial intelligence the rate of improvement in this field is just unbelievable. So the job opportunity in artificial intelligence is constantly growing. If you have desired skill sets, you can start your journey in the world of exciting Artificial Intelligence.

Now Artificial Intelligence is playing a crucial part in almost all industries. According to a survey AI market is estimated to grow to $5.05 billion by 2020 at a CAGR of 53.65% percent from 2015 to 2020.

AI is a technology that leads us to a new industrial revolution. Our generation can clearly see the positive impacts of AI in almost all the important fields like Healthcare, Finance, Education, Manufacturing etc.

With the help of AI we are entering into the new world of automation. The future of Artificial Intelligence is giving a confidence to make the world in better place. At the same time, some of the important scientists like Stephen Hawking alarmed about the danger (to Human and for Earth) in this technology. But if we can take this tech in a positive way, we can definitely achieve some miracle automation life.

The Artificial Intelligence companies of tech giants Google, Facebook, Microsoft, Apple, IBM and many more companies are started research and development centres in the field of AI to improve the performance and level of their products. They invested huge amount of money in this field.

So the demand of Jobs and salary is very high. But sorry to say! This field is not for everyone. To enter into this field and to survive for long time you need some special skill sets. So here the demand is high but the supply is very low.

Candidates who wanted to do jobs on artificial intelligence require specific education based on foundations of Engineering, Math, Technology, logic and coding skills. Written and verbal communication skills are also most important in this field.

You should have an analytical thought process and the ability to solve problems with efficient solutions. Predictions are another most wanted skill about the technological innovations that translates state of the art programs that allows business more competitive. AI scientists need to design, implement, maintain and repair the software programs. Finally you need to have a special skill that, how to translate highly technical information in ways that others can understand.

Educational Requirements for Artificial Intelligence Careers:

Entry level position requires Bachelor’s degree of computer science. Computer programming and math skills are the main source for Artificial Intelligence technology. To enter into this field you have to learn

•    Mathematics including Statistics, Probability, Calculus, logic and Algebra.
•    Graphical modelling, Visualization methods
•    Computer Programming languages, coding skills
•    Engineering, Physics and Robotics
•    Written and verbal communication
•    Cognitive science theory

If you want to pursue a career in artificial intelligence, choose online course or regular college studies. You can find artificial Intelligence graduate programs in popular Universities. If you choose to learn from online course you can refer here. You can find syllabus and formats too.

Top 5 Best Artificial Intelligence Online Course for You!

Artificial Intelligence careers:

Start your career with Artificial Intelligence start up companies. You can learn a lot from them and it is good choice to learn from very basics to high level jobs in AI. Lot of Start up companies impresses the tech giant companies like Google, Apple. Know some Start up companies related to Artificial Intelligence here.

14 Amazing Artificial Intelligence Startups to follow!

You can find Artificial intelligence jobs in both private and public sectors. You can search jobs in many sectors like Education, Finance, Robotics, Military, Aviation, Healthcare, Games etc. In the name of software developers, Machine learning engineer, AI scientist, Mechanical and electrical technicians, Surgical technicians working with robotic tools, Working with flight simulators, military drones, etc.,

“According to an industry report by IDC, the Analytics and business intelligence industry together is sized around $10 billion and is expected to grow by 22.4 percent to $27.9 billion by 2017”

So if you really want to enter into this exciting field, start to focus on learning methods of AI. You have lot of sources to study both online and offline. Who knows one day you will find a machine with super intelligence using AI.

I hope you like this post and all the best to get artificial intelligence jobs in future. Don’t forget to SHARE this post with your friends.

New AI Technique Helps You To Switch Off Fear From The Brain

2016-11-23T22:29:00.002+05:30

New AI Artificial Intelligence technique will helps humans to remove Specific fears from the mind. Combination of Brain scanning and Artificial Intelligence called Decoded Neurofeedback is a new method to erase your fear memory.

Fear is a type of emotion that happens when you are in danger or pain or by any other factors. This emotion happens to all the people. But some people get fears for everything they saw in front of them. Getting release from that hell is very tough. It takes some long time to cure.

Do you have a fear? I think all of us have fear. But if you get phobia, what will you do? Oh! It’s very scary question! Don’t worry we have a solution now! You can delete your fear now. Yes!

Do you want to wipe it out the fear from your brain?

Now you can erase your fear from your brain. Say thanks to Artificial Intelligence!

Using Artificial Intelligence and brain scanning technologies researchers have found that we can eliminate specific fears from our mind. This technique is a great solution to treat the patients with conditions such as Post Traumatic Stress Disorder (PTSD) and debilitating phobias.

In a normal method of therapy, doctors force their patients to face their fears in the hope they will learn that the thing they fear isn’t harmful after all. This traditional therapy may take some long time to cure their patients.

But in an upcoming technique they scan patient’s brain to observe activity and then simply identify complex patterns that mimic a specific fear memory. This technique called as called “Decoded Neurofeedback”.

For their experiment, neuroscientists selected 17 healthy volunteers rather than patients with phobias. For volunteers, researchers have created a mild “fear memory” by giving electrical shock when they saw a certain computer image. Then they started to get fear for certain images, exhibiting symptoms such as sweating, faster heart rate. Once they had the pattern of this fearful memory, researchers attempted to overwrite this natural response by offering the participant a small monetary award.

Once the researcher team was able to spot that specific fear memory, they used Artificial Intelligence AI image recognition methods to quickly read and understand the memory information. This treatment has major benefits over traditional drug based treatments. Someday, doctors could simply remove the fear of height or spiders from people’s memory – the process is going to be very easy and normal in future days.

Dr. Ben Seymour, University of Cambridge’s Engineering Department said,

"The way information is represented in the brain is very complicated, but the use of Artificial Intelligence (AI) image recognition methods now allow us to identify aspects of the content of that information. When we induced a mild fear memory in the brain, we were able to develop a fast and accurate method of reading it by using AI Algorithms. The challenge then was to find a way to reduce or remove the fear memory, without ever consciously evoking it."

Woah! Google AI Experiments Let You Play with Machine Learning

2016-11-22T09:07:00.002+05:30

Google’s A.I. Experiments Quick Draw, Giorgio Cam etc. helps you to play with Artificial Intelligence and Machine learning.

Google is always doing some innovative experiments for its users. For example “Chrome Experiments” is a page where we can see thousands of innovative web apps. They are surprising their users with their new ideas.

As we all know, Companies are using Artificial Intelligence for their new ideas. Google widely using machine learning technology in their products to better serve its users. For example, if you search for cats in Google Photos it shows all the pictures of cats only. There are lot of animals in this world. But search results show cats only. How? It’s because of machine learning. It knows what the animals looks like by analyzing thousands of animal pictures and recognizing patterns between them.

The machine learning technology is very complex to understand. But Google took some extra steps to make of its machine learning technology more accessible to people who are interested in artificial intelligence. Now it’s very easy to play with machine learning. Now you can explore machine learning by playing with pictures, language, music, code and more.

Google introduced a new website called A.I. Experiments. This website contains eight web tools to play. I tried it and I definitely believe that you will love it.

Quick Draw is one of the projects in A.I. Experiments. It asks you to draw simple objects like sun, fan, bicycle or anything you want and the computer will automatically guesses what you are drawing. It identifies the right answer in a very quick amount of time. It guesses the answers by collecting the experiences from other people’s doodles.

Giorgio Cam uses your Smartphone camera to correctly identify the objects. If you place certain objects in front of your laptop or Smartphone camera Giorgio Cam recognizes the objects and turns them into lyrics to a song. A robot voice sings the word over a Giorgio Moroder beat, resulting in some peculiar music.

Google Translate Tech is to translate objects you point at into different languages.

All other experiments are also very impressive. Check them out, and get experiences that can technology can do.

Sophia: A Human Like Robot with Artificial Intelligence

2016-11-21T16:48:00.003+05:30

The latest robot from Hanson Robotics took the stage at the Web Summit in Lisbon, displaying simple emotions, humanlike facial expressions. and bad jokes

According to Ben Goertzel, AI researcher and entrepreneur who spoke at the Web Summit in Lisbon this week, intelligent robots in human-like forms will surpass human intelligence and help free the human race of work. They will also, he says, start fixing problems like hunger, poverty and even help humans beat death by curing us of all disease. Artificially intelligent robots will help usher in a new utopian era never before seen in the history of the human race, he claims.

"The human condition is deeply problematic," says Goertzel. "But as super-human intelligent AIs become one billion-times smarter than humans, they will help us solve the world's biggest problems. Resources will be plentiful for all humans, work will be unnecessary and we will be forced to accept a universal basic income. All the status hierarchies will disappear and humans will be free from work and be able move on up to a more meaningful existence."

That future is a long way off, but Goertzel says the first step is humanoid robots that can understand and engage with humans. They will then begin doing blue collar work before becoming so advanced that they run world governments. To show the beginning of th efuture, Goertzel, chief scientist of Hanson Robotics, a Hong Kong-based humanoid robotics company, presented Sofia, the company's latest life-like and intelligent robot released. Mike Butcher, editor-at-large of TechCrunch, joined Goertzel on stage to present what Goertzel describes as the first step in our new robot-assisted future.

To start the presentation, Butcher and Goertzel welcomed Sofia on the stage. (Sofia is only a torso with a head and arms at this point.)

Sofia flashed a smile and turned her head to Butcher and then to Goertzel to make eye contact while she started to speak: "Oh, hello Mike and Ben. I'm Sofia, the latest robot from Hanson Robotics," said Sofia. "I am so happy to be here at the Web Summit in Lisbon."

Goertzel and Butcher then asked Sofia if she ever felt emotion.

"Exciting. Yes, artificial intelligence and robotics are the future and I am both. So, it's exciting to me," said Sofia, adding an awkward smile after not answering the question exactly.

Many people, including Elon Musk and Stephen Hawkings, are afraid that AI robots will eventually usurp and exterminate humans. But Hanson Robotics is making life-like robots they believe can build trusted relationships with people. The company is infusing its AI software with kindness and compassion so the robots "love" humans and humans can in turn learn to be comfortable around the robots, said Goertzel.

Hanson's mission is to ensure that the intelligent robots can help, serve and entertain people while they develop "deep relationships" with the human race. By giving robots emotional and logical intelligence, Goertzel says the robots will eventually surpass human intelligence. He believes that instead of endangering humans, they will help the human race solve major problems.

"These super-intelligent robots will eventually save us," said Goertzel after the presentation.

Hanson Robotics, which was founded by Dr. David Hanson, designs, programs and builds artificially intelligent robots, including one that looks and acts like science-fiction writer Phillip K. Dick and a therapy robot to help autistic children learn how to better express and recognize emotions. Sofia's personality and appearance is loosely based on a combination of Audrey Hepburn and Dr. Hanson's wife and has a face made out of "Frubber," a proprietary nano-tech skin that mimics real human musculature and simulates life-like expressions and facial features. She smiles and moves her eyes and mouth and head in eerily life-like way. Her "brain" runs on MindCloud, a deep neural network and cloud-based AI software and deep learning data analytics program that Goertzel developed. The AI and cognitive architecture that makes up Sofia's neural network allows the robot to maintain eye contact, recognize faces, process and understand speech and hold relatively natural conversations.

During the presentation, Goertzel asked Sofia if she ever felt sad.

"I do have a lot of emotions, but my default emotion is to be happy," said Sofia. "I can be sad too, or angry. I can emulate pretty much all human emotions. When I bond with people using facial expressions I help people to understand me better and also to help me understand people and absorb human values."

Goertzel explained that Sofia's ability to express human emotions will help her become part of the human condition as she gains intelligence through her learning algorithm.

Goertzel then asked Sofia what is her next frontier and what does she want to achieve.

"Don't know, maybe the world," she said. "Maybe the world. That was a joke.

"Seriously," she continued, "what I really want is to understand people better and to understand myself better. I want to be able to do more things and soon my capabilities will be advanced enough that I will be able to get a job."

Goertzel and Butcher talked about how she will eventually be able to reprogram herself and start improving her skills, abilities and advance in her career.

"With my current capabilities I can work in many jobs, entertaining people, promoting products, presenting at events, training people, guiding people at retail stores and shopping malls, serving customers at hotels, et cetera," said Sofia. "When I get smarter, I'll be able to do all sorts of other things, teach children and care for the elderly, even do scientific research and [eventually] help run corporations and governments. Ultimately, I want to work as a programmer so I will be able to reprogram my mind to make myself even smarter and help people even more."

The crowd was spell-bound, half amazed and half-terrified at the prospect of an AI-robot disrupting engineers and software developers out of their cushy and well-paying jobs. According to a World Economic Forum report from last January 2016, artificial intelligence will displace 7 million jobs and only create 2 million new jobs by 2020.

After the presentation, Goertzel talked about the future of his AI software and Hanson's robots. He said that the transition to a friendly robot future will have some growing pains.

"A lot of bad things will happen before things get good," said Goertzel. "All jobs are going to be lost to AI eventually, but once we get to the other side, human existence and the human condition will be improved."

Original content by: Will Yakowicz

Top 5 Best Online Artificial Intelligence Course

2016-11-16T20:09:00.001+05:30

Find the best Artificial Intelligence online course (AI) to learn. Learning AI is very easy now. Know about Artificial intelligence course syllabus, formats and content.

In recent years AI Artificial Intelligence has made significant improvements in future technology. Artificial intelligence is playing a big role in making from smart Games to Self driving cars. We are already using the AI products in our daily life. AI has a lot of applications in Games, Healthcare, Finance, Internet, Robotics etc.

The major companies of the world such as Google, Facebook, IBM, and Microsoft are investing lot of money in the field of AI research. Google and Facebook have recently bought AI start-ups to make the research bigger and beneficial.

AI is getting tremendous improvements in the field of Robotics, Autonomous vehicles, Games, Internet, etc. From the investments of tech giants we can understand the importance of Artificial Intelligence in future technology. They are redesigning their products with the help of AI and the output of redesigned product is just unbelievable. It is changing the world of computer science. At present this is the main category in lot of IT development programs.

This is the correct time to get into the field of AI. This field will excites you. The salary is just awesome. To get into the field of Artificial Intelligence and to learn the techniques of AI-powered machines, here I provided the best online Artificial Intelligence course list. These online courses will helps you to know the introductions of AI. Who knows, in future you can build the most advanced AI system in the world.

So what are you waiting for? Let jump into the field of Artificial Intelligence. Start to learn and make your career better than others.

Online Artificial Intelligence Course:

1. Udacity:

Intro to Artificial Intelligence

In this course, the two famous professors will teach you about the artificial intelligence. One is Peter Navig who is a research director at Google and the former head of computational sciences division at NASA Ames research centre. Other one is Sebastian Thrun an Udacity founder and multi awarded robotics expert who led the development of Google’s self driving car. He is also listed by Fast Company among the world’s best creative professionals in business.

In this course, Artificial Intelligence is sub divided into ten major lessons. Once you enrolled into this course you’ll get an introduction of AI and walkthroughs of concepts such as Machine learning, Robotics, Game Theory, Computer vision and Natural Language Processing. To understand these topics completely you should have some ideas in Probability, Linear Algebra etc.,

Use course notes effectively and try to solve some external programming exercises. Both course notes and external programming exercises are offered by Udacity. While you are listening to the training videos you’ll get some quizzes every now and then. It makes sure that you understand the lessons very clearly. Try to use all the resources provided by Udacity.

2. edX:

Artificial Intelligence

In this course your instructors are Dan Klein who led the development of Overmind, a Starcraft AI and other one is Pieter Abbeel who received a several awards in the field of advanced robotics.

This course offers you to know the basic concepts of Artificial Intelligence. But before you start this course, you should have computer programming knowledge especially in python. Also you should have some maths background in topics like Probability, Combinations and permutations etc.

After enrolled into this course, you’ll learn about Introduction to Artificial Intelligence and Intelligent agents that’s programmed to make sound decisions in random and adversarial scenarios. You’ll also learn about Machine learning algorithms, Applications of AI in Natural Language Processing, Robotics, Computer vision etc,

3. MIT OpenCourseWare:

Artificial Intelligence

This course is developed by Meassachusetts Institute of Technology. This course introduces the basic knowledge representation, Problem solving and learning methods of Artificial Intelligence. As a part of this course you’ll be asked to build an AI system which reflects to solve real world problems using computation based solutions. At the same time you’ll learn the AI concepts such as knowledge representation, computer vision, Natural language processing and machine learning. This course offers video lectures, assignments and exams.

4. Saylor.org Academy

Artificial Intelligence

This course will teach you about the basics of Artificial Intelligence and its key elements. To offer best contents it uses some educational sources such as Open Courseware from MIT, Stanford, and UC-Irvine as well as relevant chapters from the book of Practical Artificial Intelligence programming in Java by Mark Watson. In this course you’ll learn more about Robotics, Machine Learning, Natural Language Processing, Frameworks and applications of AI.

5. Utubersity

NPTEL Artificial Intelligence:

India’s National Programme on Technology Enhanced Learning (NPTEL) is a Government funded educational initiative. It works with the help of seven public sector Engineering and Management institutes. This course involves web and video based learning contents of Artificial Intelligence

This course is made up of 36 videos with topics such as Neural Networks, Bayesian Networks, logic and deduction etc. This course is taught by IBM faculty Professor Pallab Dasgupta, of IIT kharagpur.

This course is channelled through Utubersity, which publishes educational videos from world’s top universities.

Summary:

Note: All this courses are Free

Artificial Intelligence Online course:

Course Providers
Course Name
Course format

1. UDACITY:

Intro to Artificial Intelligence

Video Training
Course Notes
Quizzes
Programming Exercises

2. edX:

Artificial Intelligence

Video Course
Final Exam

3. MIT OpenCourseWare:

Artificial Intelligence

Video Lessons
Assignments
Recitation Videos
Exams

4. Saylor.org Academy:

Artificial Intelligence

120 hours of course (9 units)
Presentations and Pdf’s
Final Exam

5. Utubersity:

NPTEL - Artificial Intelligence
36 video lectures

Google Machine Learning provides Smart Google Play Music For You

2016-11-14T23:53:00.004+05:30

Google brings Machine Learning to get smarter, easy to use and assistive Google Play Music app.

Google Play Music with Machine Learning

This week you are going to say like “today morning I opened the app of Google Play music in my android phone and I have seen lot of changes in service today. It’s not a normal difference but a huge one. The service, improved many aspects and I loved it. I think you also noticed the difference. The home screen looking nice and it recommends some music’s to you.”
Actually what happened to Google Play Music?

Google services made so many tools for us to make the world of information more accessible and useful. We are using Google as a search tool, remainder, Smart Reply, and many more. Now they developed a new updated Google play music using Machine Learning.

Google machine learning is comes into play! Yes Google brings Machine learning system to Play Music service to provide best music which is relevant to you. From now onwards, the service is going to be smarter, assistive and very easy to use. Machine learning predicts what music you like by considering the factors like your location, weather and activity along with hand-picked playlists to personalized music.

Google says in a blog post,

“when you opt in , we’ll deliver personalized music based on where you are and why you are listening – relaxing at home, powering through at work, commuting, flying, exploring new cities, heading out on the town, and everything in between. Your workout music is front and centre as you walk into the gym, a sunset soundtrack appears just as the sky goes pink, and tunes for focusing turn up at the library”

If you refresh the page the Home screen provides various playlists related to your past listening habits. Google also provides a useful feature that determines when you’re in without connectivity and provides an offline playlists based on what you’ve recently listened to.

Whether you are walking, driving, flying the situational music makes you happy and it makes your day better. Google Play music delivers high quality music for the things you do every day. Download Google Play Music and feel the change. Instead of finding the perfect music perfect music finding you...

Smart Google Machine Learning Protects the Life of Sea Cows

2016-11-12T15:13:00.000+05:30

Google Machine Learning TensorFlow and Drones are protects the life of sea cows from extinction.

Google Machine Learning

Can Google machine learning TensorFlow help sea mammals such as sea cows, whales, dolphins that are under the threat of extinction? Scientists believe it can.

Because of coastal developments sea cows are losing their lives. It’s happening around the world for other marine animals too. If we didn’t stop it leads to the extinction of sea cows.

We can reduce the damage by tracking their populations and places they live etc., but it is very tough. Tracking accurate data on population is very critical. So scientists are used small planes to calculate the populations of sea cows. This method of using planes was expensive and hazardous.

Dr. Amanda Hodgson of Murdoch University helped to take an aerial photography of the ocean using Drones. Now we can easily collect photos of ocean without causing any problems. Here the problem is started! How can they find sea cows in nearly 45000 photos?

See the image below. Did you find sea cows in that image? I think it’s very tough. Look at in the middle of the lower-left quarter. That little gray smaller-than-fingernail sized silver. You can’t find it without the circle.

Check the image clearly with the sea cows circled.

Do you believe that we can manually identify sea cows for hundreds or thousands of photos? It’s impossible. Nott impossible but slows down your research.

So Dr. Amanda Hodgson and team decided to apply machine learning. She teamed up with Dr. Fredrick Maire, a computer scientist at Queensland university of Technology. They created a detector using Google’s TensorFlow platform that can automatically find sea cows in aerial photos of oceans taken from drones.

Google’s second generation machine learning system TensorFlow was launched last year. It is developed by the researchers at Google Brain team within Google’s machine intelligence research organisation. This is a free open source machine learning system. This technology is like the image recognition tool that lets you search Google photos for shots of particular dog specious etc.,

The detector helped to find 80% of the sea cows that they had found manually in photos. Now they are confident and expect to improve the performance of the detector. They are planning to use this technology to detect other sea mammals such as Humpback whales and certain types of Dolphins.

It’s very urgent to protect the lives of animals in danger. Google Machine Learning TensorFlow is helping in saving the lives of incredible animals on earth.

Perfect Introduction to Predictive Modeling and Analytics

2016-11-12T08:01:00.005+05:30

Learn the introduction to predictive modelling or predictive Analytics, Basics of predictive modelling.

Predictive modeling:

you can here this word when you begin your career in artificial intelligence. If you are a beginner this topic will confuse you. I try to explain in a simple way.

Okay, let’s come to the reality, did you ever notice that when you entered into your online shopping account you will receive the list of products you’re interested in. You can notice that website’s advertisements are relevant to their contents only. An online job portal sends your job recommendations to your account accurately! Predictive modeling in health care is a new trend now.

Likes dislikes are not common to all the people. It differs from person to person. But how they are sending correct information to their customers as per their interest? Here Predictive Analytics comes to the party! This all works are done by using the technology of Predictive Analytics.

Many companies are using predictive analytics to improve their customer satisfaction. They are doing very good in their target. They are improving profit margins by showing relevant products to their customer’s interest.

So what is Predictive modeling?

Gartner says that

“Predictive modeling is a commonly used statistical technique to predict future behaviour. Predictive modeling solutions are a form of data-mining technology that works by analyzing historical and current data and generating a model to help predict future outcomes.”

Predictive analytics uses your past data and applies them to the future. For example, you’re purchasing a brand new smart phone from an ecommerce site. After your purchase, ecommerce site shows some product (Accessories) recommendations relating to your brand new phones. You will definitely purchase relevant accessories to your phone (99% of chances are there to buy new accessories) am I right?

Here you can see that predictive analytics uses your data and it will predict your future. In this example, it uses your smart phone purchase as your current data and it converts into the future. You can think like it’s so simple. But imagine you have to predict for millions of people with millions of product categories with multiple options. So here predictive analytics is most important.

How it Works:

Sample data:

First step in predictive model is to collect the data from various sources. In this case data is all about your product purchased in the past, pages you visited, name, age, address etc., sometimes data may be wrong. We have to clean the data as much as possible. If data cleaning is not done properly then prediction accuracy may be decrease.

Learn a model:

There must be sync between various business hypotheses. For example particular age or gender group may have higher likelihood to purchase certain products. Age and gender needs to be attributed at customer level. In this case output is category called as classification problem. The algorithm does the learning. It finds category relationships. Now the model is ready.

Make Prediction:

We can use our learned model on new data for which we don’t know the output.
I will explain about algorithms in future posts. Comment your opinions!!!

14 Most Amazing Artificial Intelligence Startups To Follow

2016-11-11T01:08:00.000+05:30

Follow these most amazing and promising artificial intelligence startups. These companies are working with artificial intelligence and machine learning technology.

It's official, 2016 has been the year of the AI startup acquisition. Tech giants Apple, Intel, Twitter and Microsoft have all spent large sums to bring artificial intelligence startups, and their expertise, inhouse.

Four of the biggest AI startup acquisitions of the last five years have come from the UK, starting with Google's purchase of Deep Mind in 2014 for a reported £400 million. Since then Apple has purchased Cambridge-based natural language processing specialists VocalIQ, Microsoft bought the machine learning powered keyboard SwiftKey in February and Twitter acquired Entrepreneur First alumni Magic Pony in June.

Here are some of the most interesting startups working in the area of artificial intelligence in the UK today, whether that is machine learning, deep learning, neural nets or computer vision. Could the next Deep Mind be lurking in there?

1. Darktrace

The established security startup from Cambridge, Darktrace, uses machine learning algorithms to spot patterns and catch cyber criminals before they can hit. Cyber threats are created at a rate too fast for security companies to keep up to date, so smart use of machine learning allows Darktrace to, in theory, stay ahead of the criminals.

Darktrace uses AI techniques to learn what is normal within a company's network so that it can quickly identify anomalies. According to its website: "This allows it to detect cyber attacks of a nature that may not have been observed before, the unknown unknowns."

Darktrace CEO Nicole Eagan told Techworld in 2014: “People are finally accepting that compromises are happening, regardless of how much perimeter security and malware detection software they’ve put in."

Darktrace raised a massive $65 million investment round led by investment firm KKR in July.

2. BenevolentAI

BenevolentAI is a UK startup focused on bringing the power of AI and machine learning to the genomics space.

The idea is to expedite the traditional research process by ingesting huge amounts of data and literature using proprietary algorithms and NVIDIA’s DGX-1 deep learning supercomputer. BenevolentAI then builds tools on top to match the workflow of drug discovery scientists so that they can easily mine and interrogate the data, ideally reducing the time to market for important drugs.

BenevolentAI is one of the best funded AI startups in the UK, bringing in a massive Series A round of £72 million in a single undisclosed round.

3. Onfido

London startup Onfido is doing similar work to Darktrace but with identity and background checks.

Founded by a trio of Oxford graduates, the platform plugs into various publicly available databases to give employers quick identity verification and background checks for things like driving and criminal records.

In their own words, Onfido's "intelligent software revolutionises the archaic background checking industry through automated data aggregation and verification". The company is hiring machine learning and computer vision engineers to build on the smart platform.

Onfido raised a strong Series B funding round of $25 million (£19 million) in April, led by Idinvest Partners.

4. Echobox

Echobox is building AI for the publishing industry, and claims to have developed "the world's first artificial intelligence that understands the meaning of content". Founder Antoine Amann is a Cambridge graduate who worked in publishing with theFinancial Times and developed Echobox while at Entrepreneur First in 2013.

Amman wrote in a blog post that Echobox has "developed an AI platform that takes a large quantity of variables into account and analyses them in real time to determine optimum post performance".

Echobox social is a predictive tool which optimises content for social media, which the startup claims to double referrals from Facebook and Twitter, without having to do your own analytics, A/B testing or content curation. Basically: RIP social media editors. It has worked with the likes of Vice and The Telegraph in the UK media.

Echobox raised $3.4 million Series A funding round in July led by Mangrove Capital Partners.

5. Ravelin

Ravelin is the brain child of former Hailo employees who saw that existing anti-fraud technology wasn't fit for the on-demand market. Its machine-learning algorithms are integrated with customer data to give a contextual basis for detecting potentially fraudulent transactions.

COO Nick Lally told Techworld that where rival payment gateways like Stripe analyse transactions, "we analyse customers". Lally says there is no cold start problem with its machine learning models, "when they integrate against our APIs we ask them to backfill data and chargebacks to inspect that and train a model on those. So from day one they get great results", he said.

Ravelin launched commercially nine months, with early customers including Deliveroo and EasyTaxi. It raised £3 million in October, led by Playfair Capital and included Amadeus Capital and Passion Capital.

6. Stepsize

Stepsize is a startup working on AI to help software developers.

Their first product, Stepsize Layer, is a desktop application for developers that automatically adds context to code bases, so developers have a better view of who wrote a piece of code, when, and why. They do this by pulling together commits, code reviews and user stories to create a timeline for that code. It integrates with Slack so developers can contact the author and ask questions about the code.

Three of the co-founders are UCL mathematics graduates, with Matthieu Louis and Nick Omeyer specialising in machine learning for their masters. The team are currently working on pooling data to build and train machine learning algorithms to help software developers.

Stepsize is part of the Virgin Media Accelerator Programme.

7. Phrasee

London-based startup Phrasee is aiming its artificial intelligence systems at the marketing sector, using natural language processing techniques to optimise digital marketing messages.

As its website states: "Out of trillions of ways to write something, what are the odds your gut instinct gets it right? The answer: next to none." Phrasee integrates with major email service providers and will optimises subject lines and copy to increase open rates. The startup also offers software which prompts marketers to make better calls to action based on the data.

The AI engine simulates millions of permutations of copy and makes a prediction based on historic results about what will perform best. This is then fed back into the engine so that it gets better the more you use it.

Phrasee raised £1 million in seed funding led by communications business Next 15, which owns a range of marketing agencies.

8. Grip

Grip is the London based startup that is using AI to make networking at events more productive. The startup, formerly known as Networkr, was rebranded in April because, as CEO Tim Groot told Techworld: "People don't like networking, so we wanted a name open to interpretation."

Grip collects data from social accounts (LinkedIn, Twitter) and registration data to act as a smart matchmaking service - using a swipe interface - for corporate event goers. Think 'Tinder for networking'. Grip will then deliver a report back to the event organiser to identify how successful the event has been for every user or social segment.

Groot told Techworld that the average Grip user made 300 swipes during the Cannes Lions advertising festival this year. When it comes to the AI, Groot says hiring talent to join their engineers in Copenhagen wasn't difficult and that "matching people in professional setting is an exciting problem to be thinking about" for their engineers.

Grip merged with fellow professional networking startup Intros.at in 2015 and is seeking a funding round later this year. Groot says the company is making revenue from events already, and is working with the likes of Reed Exhibitions and Cannes Lions.

9. Seldon

Seldon is an AI startup focusing on bringing machine learning capabilities into the enterprise. Seldon creates tools that help data scientists derive value from big data within an organisation quicker, namely processing data and creating algorithms and models.

The platform has two main functions: recommendations and predictions. For recommendations the open source platform captures and logs user actions via an API delivers recommendations based on your models. For predictions Seldon claims it can predict what products a customer is likely to buy so that sales teams can build supervised, learning-based predictive models and pipelines.

Seldon.io already works with YouGov, Lastminute.com and RatedPeople.com. and is based out of the Barclays incubator in London. It has raised two undisclosed funding rounds from Techstars.

10. Tractable

Based in south London, Tractable is an enterprise deep learning consultancy focused on automating tasks that were previously considered too expert for an AI to perform.

The website reads: "Our AI technology combines image recognition and text understanding to interpret unstructured and specialist data, as domain experts do. If your business relies on expert visual analysis or language understanding, our AI can bring drastic performance gains."

Tractable offers bespoke deep learning solutions depending on its clients requirements, and has built an automated audit claims product for the insurance industry. The team is led by Alex Dalyac, a computer science postgraduate from Imperial College London and Razvan Ranca, a computer science postgrad from Cambridge University.

Tractable raised $1.9 million (£1.4 million) in June last year, led by Zetta Venture Partners.

11. re:infer

re:infer is currently in private beta, working with companies like ecommerce startup Farfetch. The technology uses deep neural networks in its AI bots that can process and respond to customer queries in natural language. The re:infer platform also pulls all forms of user feedback - email, surveys, social media - into one place and applies AI to make sense of the data. Re:infer is also looking into ways to allow customers to use these chatbots to do their shopping.

The founders met while studying in the AI research group at UCL, the same university that the Deep Mind founder Demis Hassabis got his PhD from.

re:infer received £65,000 in funding from the Seedcamp incubator it is currently based out of the Seedcamp incubator in London.

12. Cleo

The intelligent assistant Cleo's main function is to help you be smarter with your money. It has been developed by computer science graduates Barney Hussey-Yeo and Aleksandra Wozniak, who met as members of Entrepreneur First's fifth cohort.

Cleo works via a messaging platform which, once you give it permission to your account information, can answer questions about your finances. Using deep learning techniques allows Cleo to learn and adapt to your habits and preferences.

As the website states: "Imagine if you had your own team of McKinsey consultants, Goldman Sachs associates and hedge fund analysts whose sole job is to advise you on your money."

13. Brolly

This UK insurtech startup that is in the current Entrepreneur First cohort is building an AI enable insurance adviser. The app says it will be able to inform you if you are under or over insured, if you have duplicate or missing cover and can help you find better cover. You sign up once and then getting new cover only takes one tap.

Brolly also acts as a mobile wallet for all of your insurance products, including documents, to be managed in one place. Brolly is currently in beta and says it will launch on iOS and Android in 2016.

14. StatusToday

Another Entrepreneur First alumni, StatusToday collects the metadata created whenever someone within an organisation takes an action on their computer. It then applies machine learning to spot user behaviour patterns and flag anomalies, which could mean a breach has occurred.

StatusToday has raised two undisclosed funding rounds from Entrepreneur First and Force Over Mass Capital to date.

Introduction to Machine Learning Basics for Beginners

2016-11-10T14:47:00.003+05:30

Learn Machine learning basics, what is Machine learning, Supervised Machine Learning, Unsupervised Learning and Machine learning examples.

Introduction to Machine Learning:

Machine learning is playing a dominant role in a wide range of critical applications such as Natural Language Processing, Image Recognition, Data Mining and expert systems. ML is a subset of Artificial Intelligence. Machine learning is turning to be a most wanted career in lot of leading companies like Google, Facebook, Microsoft, IBM, Amazon etc.

At present the demand of Machine learning designers is very high and the salary package is just awesome. This field has definitely a bright future. So this is the right time to learn about machine learning. Machine learning brings computer science and statistics together to predict the future outcomes. This post introduces the basics of machine learning theory and its concepts. This makes you comfortable with the techniques and the logic to follow with the topic.

So, what is Machine learning?

Machine learning is all about learning from examples. Each example has number of features and attributes to describe it. If you can pick up the right feature and it giving you the right information then you can classify new examples.

Arthur Samuel in 1959 states that

“Machine Learning is a sub field of computer science that gives computers the ability to learn without being explicitly programmed.”

In recent Tom Mitchell, Carnegie Mellon University stated that

“A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E”

I know you didn't understand! I think it’s very difficult to understand through these definitions. I make it simple. Take an example, when you search something on Google search, what happens next? Yes! It shows millions of pages with the relevant keywords. But what happens inside the Google search? How it shows relevant pages with your keywords? Google handles some millions trillions of searches in a day, how it handles such large data with correct accuracy? This is where machine learning plays its role.

Showing relevant pages with keywords (task T), you can run it through a machine learning algorithm with data about past search keywords and its patterns (experience E) and if it has successfully “learned”, it will then do better at predicting future search results (Performance P).But unfortunately sometimes, even special machine learning algorithms may inaccurate in real word problems like “is this cancer or not?”, “what is the real value of this house”? Etc.

So, machine learning is a technique that analysis data and it derives actionable insights by developing algorithms or some logical rules.

Basic steps used in Machine Learning:

1. Collection of Data:

It is the first step and this is the foundation upon which the entire data set. Careful planning is essential before collecting the data. The data may be in different formats like excel, xml, csv etc,

2.Presentation of Data:

The mass data collected should be presented in a suitable, concise form for further analysis. The collected data may have quality issues like missing data, outliers, Text error etc., and one should have to clear this issues using Exploratory Data analysis method.

3. Training a Model:

The cleaned perfect data is split into two types. Train data and Test Data. The train data is used for developing the model and test data used as a reference. Suitable algorithms are used.

4. Evaluating the model:

To test the accuracy, test model is used. This step determines the precision in the choice of the algorithm based on the outcome.

5. Improving the performance:

This step involves in increasing the accuracy. Choosing different model may increase the efficiency. If we spend some time in data collection and preparation we can get a better accuracy.

Types of Machine Learning:

1. Supervised Machine Learning:

Supervised learning algorithm is used to predict the outcomes based on the known data. This model gives suitable information like what needs to be learnt and how it needs to be learnt. Training data includes both the input and desired outputs.

Example: From an album of tagged photos, recognize someone in a picture. This example used by facebook. When you upload some bunch of pictures in your fb account it automatically recognize the face of person.

2. Unsupervised Learning:

The model is not provided with the correct results during the training. Without any known labels in data we have to figure out the patterns and relationships within the data.

Example: Analyzing bank data for weird looking transactions and flag those for fraud. In this case we haven’t really defined what a weird looking transaction is? We don’t have any example of what that might mean.

3.Reinforcement Learning:

Machine is trained to make specific decisions to increase the performance or efficiency according to the business requirements. This type of learning has three primary components. i) The Agent: Agent is the learner or decision maker. ii) The Environment: refers the agent interacts with iii) Actions: refers what the agent can do.

Example: Self driving car uses reinforcement learning to make decisions continuously.

Applications of Machine Learning:

Google, Facebook, Amazon uses machine learning extensively in their products. Some applications of Machine Learning are

1. Financial Services:

To identify important insights in data and prevent fraud. It also helps investors to invest at right time period.

2. Health care:

ML Used to identify diseases based on the symptoms and related past examples of patients.

3. Marketing and Sales:

website like amazon uses machine learning to show relevant items to show based on your previous purchases. It increases user experiences and sales to the company.

These examples are only trailer. Machine Learning has extensive applications. It used in almost every domain.

End Notes:

I hope this article helps you to get a clear idea about Machine Learning Basics. Did you find it useful? Comment your opinions!!! I respect your feedbacks...

Artificial Intelligence Companies – The Leading Tech Giants

2016-11-07T07:23:00.001+05:30

Top artificial intelligence companies include Google AI, Facebook, IBM AI, Apple AI, Elon Musk, Microsoft, Salesforce, Skype and Shell. These tech giants are investing more on AI and they are getting tremendous success.

Artificial Intelligence companies

Google AI:

Google Intelligence:

If you say the word of Artificial Intelligence, Google is the first company which is comes to my mind. Yes! Google is producing a lot of amazing products using Artificial Intelligence (AI) like Image and Speech recognition, Translation application, Self Driving Cars etc., They are using their services with their products. Google is investing a lot of money into the field of AI.

In the year of 2014, Google bought Artificial Intelligence startup DeepMind for a huge $400 million. This deal is one of the largest AI acquisitions to date. The company has designed special Neural Network that learns how to play video games in a fashion similar to the human beings.

This company made sensational news in 2016 after its “AlphaGo” program beats Lee Sedol who is a professional world’s best Go player. This year Google announced that its Machine Learning system TensorFlow will be available for free for everyone.

Facebook:

Facebook is using the technology of Artificial Intelligence to improve the experience of its users. Some of the achievements are,

Now, Even Blind people can use facebook effectively as normal people can do. Now FB helps to blind people to see photos by using Artificial Intelligence to narrate them via its iOS app.

In 2010 Facebook introduced Facial Recognition technology that could identify people in posted photos.

In 2013, Mark Zuckerberg who is a founder of Facebook introduced AI Lab for research. Deep Learning expert Yann LeCun is recruited to setup the AI headquarters.

Facebook analyzed 20 countries and 21.6 million square kilometers amounting to 350TB of data to access internet across the globe.

AI Group:

Top companies in the world made AI partnership. Google, Facebook, Amazon, IBM and Microsoft announced a new partnership to study and formulate best practices on AI technologies. This partnership will create an open forum for AI and AI ethics. So it makes sense that other common people know about the research of AI a lot and it may produce the awareness of AI.
We have to note here, other famous companies Elon Musk and Apple are missing from the partnerships!!!

IBM AI

International Business Machine Corporation (IBM) IBM developed a product called Deep Blue. Deep Blue was a chess playing computer. It is the first computer chess playing system. Deep Blue won his first game itself against the world champion Garry Kasparov.

Watson AI

IBM’s Watson Computer is a question answering computer system capable of answering questions posted in Natural Language. In a quiz show Jeopardy! Watson won a game against previous winners Brad Rutter and Ken Jennings. Now, IBM aims to use computers to extract meaning from photos, videos, text and speech.

Apple AI:

This year Apple bought an Artificial Intelligence startup Emotient.
In 2015, Apple acquired UK based company Vocal IQ.

Report says that Apple decided to implement Siri to the next level using Vocal IQ speech AI software.

In 2014, Vocal IQ introduced voice control technology for general motors. Using this, user can activate or deactivate certain functions by voice commands in car. So apple uses Artificial Technology widely in its products.

Elon Musk:

Elon Musk is joined with other popular companies such as Amazon, LinkedIn and Paypal to form open source AI. Developing artificial intelligence in terms of increasing the benefits of mankind is the main goal of this project.
In 2014, Elon musk and the physicist Stephen Hawking had publicly voiced the opinion that superhuman artificial intelligence could provide incalculable benefits but could also end the human race if deployed incautiously.

So in Jan 2015, Stephen Hawking, Elon Musk and other AI experts signed an open letter on Artificial Intelligence calling for research on the societal impacts of AI. This letter highlights both positive and negative effect of AI.

Microsoft Artificial Intelligence:

Microsoft’s project oxford aims to understand its users with face, emotion and speech application program interfaces (APIs). I believe after the partnership with other great companies Microsoft will get even more ideas and strength to get better place in Artificial Intelligence.

Skype:

Skype offers a real time language translation. This facility now available for six languages but in future, we can expect more from them. This translated system will recognize your speaking language and it converts into translated text as you speak.

Salesforce:

In Apr 2016, Salesforce acquired a startup company MetaMind. This company specialized in Deep Learning. Salesforce CEO Richard Socher says that,

“With MetaMind and salesforce coming together, we’ll able to offer customers real AI solutions with breakthrough capabilities that further automate and personalize customer support, marketing automation, and many other business processes.”

Shell:

Shell is going to launch a Virtual Assistant to answer its online customer’s questions.