NatWest Business Hub Article

On May 10, the NatWest Business Hub published a great story about Amplyfi’s growth, market success, dedication to entrepreneurship, and long-term vision. In an interview with Chris Ganje, Amplyfi co-founder and CEO, the piece deep dives into why Amplyfi is growing its business in Cardiff, the challenges of hiring the best talent while maintaining culture, and what Ganje’s vision is for the future.

Amplyfi continues its trajectory as a fast-moving startup, and BrightPlanet continues to expand our Data-as-a-Service offerings partnership. If you have not read the full article, check it out at: http://www.natwestbusinesshub.com/content/4e12978a-f1b6-b77d-b8bd-3c46f9bafa62

Conclusion

BrightPlanet is the leader is providing deep Data-as-a-Service to our customers with open-source, web content through a simple-to-use service. Our customers do not need to worry about the complexities and details about harvesting, curating, and preparing data for analytics. Instead they can focus on what they do best — creating intelligence.

hbspt.cta.load(179268, ‘811862a0-8baf-4d9e-b1ca-3fa809ee8f97’, {});

BrightPlanet’s harvest engine and quality assurance solution is designed to be robust and fault tolerant to these types of website changes. Our harvest engine will continue to harvest and provide value without need for external human intervention; this is because we typically leverage unstructured keyword rules instead of hardwired rules.

In our previous post, named “All websites are not created equal”, we talked specifically about our techniques for harvesting the best quality content from websites. Today, we are going to discuss how we ensure that clients’ harvests continue to operate even after the websites change.

Keeping Your Harvests Simple

If there is one thing we have learned over the last 18 years of web scraping, it is to keep your havests simple. I don’t mean harvesting simple websites. Instead of worrying which text nodes to process and which to ignore while harvesting data, let your analytics and unstructured text rules do the heavy lifting.

Sites requiring a tremendous amount of custom steps, extractions, and hardwired rules are going to be the ones which break easily. BrightPlanet takes an approach of loosening up harvest rules to ensure we’re pulling content, but then tightening restrictions on the analytics, qualifications, and post-harvest filtering.

For example, we often restrict a harvest based on URL path, using either substring matching or regular expression, instead of defining a series of rules to determine which links should be followed. It may sound simple, but it is very effective. If you are still picking up false-positive links, use unstructured keyword filters to polish the data set.

Leaving Rules As Unstructured

There is only so much you can do with harvest filters; sometimes you need to jump into the unstructured content to finalize content quality.

In its simplest form, we leverage large keyword lists and require webpages to have one or more of the keywords to keep the content. Our list processing system allows us to easily process up to thousands of keywords.

A great example of this filtering is used to determine if a webpage is selling a product, such as a pharmaceutical drug. We can provide a list of tens of thousands of drug names and then ensure that at least one of them is on the page, otherwise the page is rejected.

Quality Assurance Checks

After the harvest events are defined for a project, we integrate ongoing monitoring to ensure harvests are producing an expected number of documents over time. This will vary from project to project, source to source, and day to day. Using a standard deviation calculation and historic trending data, we can easily tell if a harvest needs to be reviewed or is operating within its expected boundaries.

Conclusion

BrightPlanet is the leader is providing deep Data-as-a-Service to our customers with open-source, web content through a simple-to-use service. Our customers do not need to worry about the complexities and details about harvesting, curating, and preparing data for analytics. Instead they can focus on what they do best – creating intelligence.

hbspt.cta.load(179268, ‘811862a0-8baf-4d9e-b1ca-3fa809ee8f97’, {});

In our previous post named “We harvest a lot of websites for our clients, but how do we know which sites to harvest in the first place?”, we talked specifically about finding valuable websites. Today, we are going to discuss how we decide the best ways to harvest content for our clients.

Data-as-a-Service

Determining What Content to Harvest

Before we setup any harvests, we always sit down with our clients and perform a client onboarding process, or harvest audit, to determine exactly which data makes the most sense to harvest. It is not sufficient to know only the website domain; we need to know which information to extract from those websites.

Think of a large news website, such as CNN. It is not practical, nor is it useful, to harvest the entire website. For example, if our client is looking for North Korean threats we will target only the sections within CNN which focus on that topics, such as world news, while excluding sports, weather, lifestyle, etc.

Once we have defined our target content, each site may need to be reviewed and processed individually, depending on how targeted the data must be. Each project is different; sometimes a broad harvest with sufficient filtering is enough.

Each harvest event can be defined with term filters, URL filters, domain filters, depth filters, and more. Filters are single items, multiple items, regular expressions, or even a massive list of keywords. There are no practical limits to our filtering system and many filters are applied as the harvest runs, further optimizing harvest efficiency.

Choosing the Right Harvest Techniques

BrightPlanet’s harvest solution contains many different techniques and harvesters allowing us to pick and choose the most efficient way to harvest content for each website. We are not limited to a simple web crawl. This allows us to spend less time harvesting content and more time processing the data, a topic we previously covered here.

Since we utilize multiple harvest engines, we can easily choose the correct harvesting techniques for the website and client’s content needs.

For examples, going back to our earlier CNN example, say we are only interested in North Korean missile testing. Instead of harvesting thousands of irrelevant world news documents looking for the needle in a haystack, we can leverage our Deep Web harvest engine to customize a search of the CNN website looking for specific keywords, ordered by publication date, and filtered by sub-sections and keywords. Now we’re only harvesting only extremely relevant documents

Our Data Acquisition Engineers may even leverage different harvest techniques for the same website, if necessary. Perhaps we need to perform an initial harvest of archived content and then perform daily updates of new content. It is not necessary to constant harvest old data; we would create one harvest to grab all content on the website. A second harvest (typically pointed to an RSS feed or home page) would monitor for new documents.

Advanced Deep Web and Dark Web Tips

Targeting makes a huge difference when it comes to Dark Web content. Dark Web websites often cover many topics, many of them being irrelevant to our client’s needs. Having the ability to quickly filter only relevant content channels allows us to be more efficient and also prevents our harvesters from being blocked – critical when harvesting Dark Web sites.

Another technique we often leverage involves a multi-pass of the same website to pre-process content. Once an initial pass is performed, we will process the data that was harvested to build intelligence into how we should monitor the site over a longer period of time. This per-processed data is typically thrown away since it is not relevant enough to provide value.

Deep Web and Dark Web content may also be scattered with irrelevant content, or content meant to obfuscated valuable data. Leveraging a broad harvest without filtering is often used to target relevant content which may then be re-harvested using additional filters in a new harvest event. This allows us to curate higher quality data with little additional work.

Conclusion

BrightPlanet is the leader is providing deep Data-as-a-Service to our customers with open-source, web content through a simple-to-use service. Our customers do not need to worry about the complexities and details about harvesting, curating, and preparing data for analytics. Instead they can focus on what they do best — creating intelligence.

hbspt.cta.load(179268, ‘811862a0-8baf-4d9e-b1ca-3fa809ee8f97’, {});

1. Challenges which come with harvesting open-source content
2. Solutions for those challenges
3. Specific project use-cases, such as harvesting job postings

Eagle Alpha provides a full-service platform enabling asset managers to obtain solutions from alternative data providers, such as BrightPlanet.

The interview is only available to subscribers of Eagle Alpha’s newsletter, or for partners within Eagle Alpha’s internal partnership portal. If you’d like to jump on their mailing list, you can sign up here.

hbspt.cta.load(179268, ‘811862a0-8baf-4d9e-b1ca-3fa809ee8f97’, {});

The first phase of all projects – we have three phases total – is harvesting. It is not uncommon for a single project to have 10,000-20,000 websites from which we harvest content. While a large volume of websites is impressive, the more important element is knowing which websites are the most appropriate for the current task. This is where our Data Acquisition Engineers become a critical part of the solution.

Techniques for Identifying Websites

BrightPlanet has developed several techniques for locating websites; some are trade secrets whiles others are just common sense. We are going to use the term “website” to refer to a single site that is all within the same domain name. Like “brightplanet.com” or “cnn.com”. As you can imagine, a single website may have a few pages or a several billion pages, it all depends on the site. A useful trick for estimating the number of pages in a website is to do a Google search using only the website parameter. (ex: site:brightplanet.com)

Occasionally clients possess a full list of websites they want to harvest. Clients will typically have a partial list of websites. However, even these lists will usually require some type of curation or quality check.

Most often, our Data Acquisition Engineers will work with a client to define what a ‘good’ website looks like. This will help identify and filter out websites which may be relevant, but not on target with a client’s needs.

As you might have guessed, one way to identify new sites is to “search” for them using surface web sites like Google or DuckDuckGo. Unlike an analyst who might need to search, iterate, search again, and then review each website by hand, we can do that quickly using our Deep Web harvesting and some filter rules. This is extremely effective because we can iterate so fast with our harvest engine.

Locating online lists or directories of similar sources. If you have ever tried to find the “best video editing software” online, you know people love to make lists. Using known entities, we can quickly locate other sites referencing related sources. From there, we can easily harvest and qualify the sites listed.

Advanced Techniques Used For Some Deep Web Harvesting

Those are pretty straightforward, and probably obvious, ways to locate websites. Some projects need a much greater depth to locate valid websites. Here are a few creative techniques we have used on projects.

Monitoring new purchased domains. Each day, a list of the newly purchased (and newly expired) domains is generated. Since we can harvest and validate many sites at a time, we can use these lists in combination with good filtering and validation rules.

Diving deep into topical blogs or messages boards is a great way to find hidden gems. Often, people will post links which get buried deep within these sites. Being able to go deep, and validate them, allows us to locate sites that might otherwise be missed.

Social media contains a wealth of links, but they typically need to be “exploded” since they will be shortened for tracking purposes. Again, our harvest engine and social connectors make it possible to search, locate, harvest, extract, and validate these links.

Conclusion

BrightPlanet is the leader is providing deep Data-as-a-Service to our customers with open-source, web content through a simple-to-use service. Our customers do not need to worry about the complexities and details about harvesting, curating, and preparing data for analytics. Instead they can focus on what they do best — creating intelligence.

hbspt.cta.load(179268, ‘811862a0-8baf-4d9e-b1ca-3fa809ee8f97’, {});

• URL
• web page title
• document harvest date
• full text of news article
• extracted named entities: including crime, other threats, events, weapons, people, diseases, companies, countries, places, etc.
• document and individual entity-level sentiment

The data below was visualized and shared using Tableau.

Named Entities Reveal Granular Patterns in Data

The named entities we’re focusing on in this dashboard are threats, events, weapons, and people. By default, we see the total count of entities over time. Between June 2015 and January 2018, there were about 3,800 news articles mentioning Lashkar-e-Taiba. Looking at the line chart, March 2016 and November 2017 jump out as months with increased activity. Clicking on any single month will filter all the data down to that month. Overall, we can see entities which frequently appear in this dataset include:

• “attack”
• “kill”
• “terrorism”
• “arrest
• “India”
• “Pakistan”
• “Hafiz Saeed” (co-founder of Lashkar-e-Taiba)

Click the image to go to interactive visualization

Matching Crimes to Locations with Entity Relationships

Knowing individual entities which appear in a document is helpful, but seeing how those entities relate to other nearby entities of interest can provide real insight. BrightPlanet uses Rosoka Series 6 as its entity tagging engine. One convenient feature of the software is automatically tagging relationships between nearby entities. The yellow dots on the map correspond to news articles containing a Crime-to-Location relationship. Hovering over the southern-most yellow dot on the map, we can see that the crime entity “theft” was found nearby the place entity of “North Paravoor”. If we jumped to that article from The New Indian Express, we would find the targeted sentence,

“Anoop, a native of North Paravoor, is the seventh accused in the case related to theft of ammonium nitrate, nitrate mixer and electric detonator from Thuruthiyil Traders, a shop functioning in Perumbavoor.”

These are just a few examples of the insights that can be found by adding structure to unstructured web content by tagging named entities. Please click into the interactive Tableau visualization to explore the nuggets of insight which can be found from a targeted search of news data.

Develop Business Insight through Unstructured Web Content with BrightPlanet

If you think your organization would benefit from trends and insights derived from access to a repository of news data with over 15 million articles from thousands of unique sources, check out BrightPlanet’s Global News Data Feed for a 30-day free trial.

hbspt.cta.load(179268, ‘811862a0-8baf-4d9e-b1ca-3fa809ee8f97’, {});

These collections often contain data from dozens, hundreds, even thousands of websites using various techniques to optimize the best content from each website. All websites are not created equally; there is a very low chance the same harvesting technique will work for every website.

Data-as-a-Service

Techniques for Harvesting Content

BrightPlanet has developed several techniques for harvesting data. One of the easiest for people to understand would be a Surface Web harvest, or a hyperlink crawl. Start with a single URL, then follow the outgoing links from that URL, and repeat. This is a typical process for collecting web data and has is utilized in nearly every project we do.

Other techniques increase the harvest complexity, such as a Deep Web harvest. Instead of starting with a URL and following links, our Deep Web harvester can interact with a website’s internal search database, asking it different questions, each time collecting and harvesting the results as they are returned.

Our extensive harvest engine also supports many other techniques, such as a Dark Web harvester, REST-API harvester, and even a custom scripted engine. Each of these techniques allow many variations for diving deep into websites and extracting only relevant content, avoiding noise in the final dataset.

Curating Deep Web Content

As each document is harvested, it is normalized into a consistent stream of data and prepared for curation. It is important the content is properly prepared to optimize entity relationships. If data gets mangled, it can be mismatched and produce poor results. Garbage in, garbage out.

Each project will have its own curation steps defined because what works for a fraud project will be useless for a reputation management project. Our Rosoka integration allows us to be very flexible and extensive when it comes to creating custom named entity recognition solutions.

We covered entity extraction features in a previous blog post.

Data Delivery through REST APIs

Lastly, data needs to be delivered to the analytics platform for further processing. At this point, the data is properly prepared for a third-party solution to ingest the data. The easiest, and most common way to transfer data is via our REST API service.

Typically, our Data-as-a-Service ends with a REST API that our clients or partners can easily integrate into a dashboard, service, analytics engine, data visualization, reports, or even a simple data alert.

Conclusion

BrightPlanet is the leader is providing deep Data-as-a-Service to our customers with open-source, web content through a simple-to-use service. Our customers do not need to worry about the complexities and details about harvesting, curating, and preparing data for analytics. Instead they can focus on what they do best — creating intelligence.

hbspt.cta.load(179268, ‘811862a0-8baf-4d9e-b1ca-3fa809ee8f97’, {});

Visualizing Bitcoin News Data

The first insight to jump out was the high correlation between the Bitcoin price and the number of news mentions, as you can see when those two measures are visualized with a scatterplot. The r-squared value of the linear regression line is 0.95, indicating a high level of predictability. The greater the price of Bitcoin, the more news articles regarding cryptocurrency are produced.

Click the image to view the interactive visualization

Using the coloring of the marks on both the scatterplot and treemap charts, which indicate the “Year” attribute of the monthly data, you can see the slow rise in popularity of Bitcoin in 2015 and 2016 before the explosion in 2017. Every month before May 2017 is clumped together in the bottom-left of the scatterplot, indicating a price range around $1,000 and a news article count between 100-350 articles per month.

Diving into Deep Web content

The dual-axis chart at the bottom of the dashboard shows the relationship trends between Bitcoin price and news article sentiment. BrightPlanet uses Rosoka Series 6 for entity tagging and extraction. The Rosoka data point we’re using in this dashboard is total sentiment at the document-level. To smooth out the trendlines and avoid large outliers, both sentiment and price were visualized as a 7-day moving average. The insights aren’t as striking in this chart, but you can generally see increases and decreases in sentiment shortly following corresponding increases and decreases in Bitcoin price.

If you hover over any single day on the line charts, you will see a “Top 10 Articles by Sentiment Intensity” chart. This ranking shows the top 10 news articles for that day based on difference from a “neutral” sentiment rating.

In addition to the two main measures, Price and Sentiment, shown by the line charts, we also added two additional measures to the view by using the “size” attribute of each line to show the volume of both news articles and Bitcoin trading. On the left-side (older) of the line chart, the lines are noticeably skinny. The skinniness indicates fewer news articles and a lower volume of Bitcoin trading. Towards the right-side (more recent) of the line chart, the lines grow thicker as both news mentions and Bitcoin trading volume increase.

Develop Business Insight through Unstructured Web Content with BrightPlanet

If you think your organization would benefit from trends and insights derived from access to a repository of news data with over 15 million articles from thousands of unique sources, check out BrightPlanet’s Global News Data Feed for a 30-day free trial.

hbspt.cta.load(179268, ‘811862a0-8baf-4d9e-b1ca-3fa809ee8f97’, {});

[Photo credit from Max Pixel].

Hundreds of languages also make up areas of the Deep Web and Dark Web. If businesses want to gain complete data insight into their organization, it becomes necessary to effectively harvest and interpret data from languages other than English.

Learn how BrightPlanet partners with Rosoka to utilize unstructured data mining and Deep Web search to harvest data in nearly any language, interpreting its meaning to provide deeper insight into the opportunities and threats that exist for businesses today.

Harvesting Foreign Deep Web Data

BrightPlanet’s process for harvesting web data in a foreign language is the same process it uses to harvest a web page in English.

For example, an online article about leukemia written in English is harvested the same as an article about leukemia written in Dutch, Arabic, or Portuguese. BrightPlanet navigates each page and stores and archives all text from that page. This process works with any language that can be written using characters of some sort online.

Curating Foreign Data to Identify Patterns

While harvesting foreign data isn’t too difficult, the challenge comes when analyzing documents of different languages and comparing them for similar and contrasting content.

BrightPlanet works with text analytics solutions partner, Rosoka, to enrich unstructured data that has been harvested through Deep Web search or Dark Web search through the process of entity extraction.

Rosoka has the ability to detect key entities in multiple languages, meaning it is able to extract main keywords and themes from content in over 200 languages.

If Rosoka harvested three different articles about leukemia written in English, Dutch, and Portuguese, it would be able to recognize the main theme of the disease without needing to perform a full-size machine extraction, saving valuable time.

Another advantage to using Rosoka and BrightPlanet to harvest data in other languages is our ability to normalize extracted tags into one instance, regardless of language. For example, even though the three example articles mentioned above are all written in different languages, we are able to create a common link between the three, simply referring to “leukemia” instead of each individual tag.

Not only can Rosoka identify common keywords and themes among different languages, it can also identify sentiments such as mood and intensity of voice among entities and entire documents. This feature allows organizations to dig deeper into their data, discovering the passions and decisions that are the driving force behind data points.

Once you have this harvested data, BrightPlanet works Rosoka to give you the content in the language you primarily use, from English to Russian.  

hbspt.cta.load(179268, ‘73981f45-acaa-4b80-9f0b-2e2f59b50016’, {});

Develop Business Insight through Unstructured Data Mining of Foreign Languages

Harvesting foreign language data can be an overwhelming topic to address for businesses. Many people don’t always realize the potential increased business insight that comes with foreign language entity extraction.

BrightPlanet works to provide the best possible foreign language data extraction services that allow you to protect your business against possible domestic and foreign dangers such as fraud, while also giving you insight into potential business opportunities.

No matter what your data harvest needs are, BrightPlanet can help. Schedule a consultation with one of our Data Acquisition Engineers and learn how you can increase your business intelligence.

hbspt.cta.load(179268, ‘811862a0-8baf-4d9e-b1ca-3fa809ee8f97’, {});

We recognize the power of data, and work to find new ways to harvest content from the Deep Web and Dark Web that can help businesses with their data goals. Here are four data trends we’ll be watching in 2018.

1. Growth in Artificial Intelligence

Artificial intelligence systems (AI) have seen a resurgence in growth the past few years. One use for AI comes in the form of chatbots. These chatbots are those often annoying pop-ups that ask you vague questions. When you reply, their response (if you receive one at all) is usually unhelpful.  

While chatbots don’t have the best reputation, that may change for the better in 2018. No longer do chatbots just scan for keywords and pump unintelligible responses to complex questions, they really do understand what you are asking, and can quickly route you to where you need to go.

Chatbots are interesting, but they aren’t the only thing that’s affected by AI.

AI is going to make a huge impact in the intelligence community through the building of content models that can better detect valuable content versus background or secondary content.

BrightPlanet has seen these models work very well in a few of our projects, as they classify content based on how similar or dissimilar it is to a curated model. We believe that AI is ready to analyze all open source data in 2018, giving BrightPlanet clients the ability to see patterns and quality indicators within their data.

2. Expansion of Unstructured Data Mining

BrightPlanet has seen major advancements in unstructured data mining in recent years, with more advancements to come within the industry in 2018.

BrightPlanet works closely with text analytics solutions company, Rosoka, which will see many updates in the coming year, including updates to their entity extraction system.

The key to processing quality web content relies entirely on the ability to extract and create meaningful metadata through the unstructured data mining process. As we see services like Amazon Web Services (AWS) and Google invest more resources into artificial intelligence, the ability to process unstructured web content is also being improved.

One specific industry that we see adopting more unstructured content in 2018 is the banking and financial services industries, or Fintech as it is being referred to today.

Being able to predict trends in the economy using unstructured text analytics solutions can help businesses identify potential opportunities and risks within the financial services industry. To learn more about uncovering trends within the financial services industry (as well as the auto industry), watch our trends and influencers webinar, held in November, 2017.

hbspt.cta.load(179268, ‘8a74934b-aef7-46f6-9f21-94c305bbd5ad’, {});

3. Improved Data Visualizations

Tableau, BrightPlanet’s default data visualization solution for the past three years, is making improvements to their data modeling, storage, and flexibility in data viewing in 2018. As these updates are implemented, it will be easier than ever to turn metadata into beautiful data visualizations with real meaning and intelligence.

Since unstructured data sets can have billions or tens-of-billions of data points, visualizing the data at scale has never been more important. Tableau recognizes the growing importance of having accurate data visualizations, and is adding an optimized cache to be built around these visuals.

Tableau, along with other data visualization options, are leveraging back-end REST APIs to properly prepare and organize data for optimal visualization. These tools will continue to improve and be a huge asset to all open source projects in 2018.

4. The Rise of the Chief Data Officer

We are optimistic that 2018 will be a strong year for organizations in all industries to seriously invest time and resources into their data analytics and data harvest processes.

With an increased emphasis on data, businesses will need to have people within their organization, such as a Chief Data Officer, whose primary job duty is to monitor and improve their data through analytics, collection, visualization, and open source content.

We welcome the well deserved recognition of future Chief Data Officers everywhere, and can’t wait to talk to them about how to best leverage open source web content for their business intelligence needs.

BrightPlanet: Your Answer to Data Harvest Questions in 2018 and Beyond

Data trends come and go, but one thing that won’t change in 2018 is BrightPlanet’s commitment to helping clients achieve their data harvest goals, no matter how simple or complex.

Looking to improve your data harvest process in 2018? Let BrightPlanet help. Schedule a free consultation with one of our Data Acquisition Engineers today!

hbspt.cta.load(179268, ‘811862a0-8baf-4d9e-b1ca-3fa809ee8f97’, {});