My Tech Notes

The KNIME Example

2013-12-23T23:05:00.000-08:00

The KNIME Platform

→ KNIME loads and integrates data from diverse data sources:
• Different data bases
• Various file formats (CSV, XML, SDF, etc.)
→ KNIME provides huge repository of modules for easy-to-use, modular
• Data preprocessing
• Data fusion
• Data transformation

Over 1000 native and imbedded nodes included:

Statistics
Data Mining
Time Series
Image Processing
Neighborgrams
Web Analytics
Text Mining
Network Analysis
Social Media Analysis
WEKA
R
Database Support
ETL
Text Processing
Data Generation
XML Read/Write
PMML Read / Write
Social Media Analysis
Business Intelligence
Community Nodes
3rd Party Nodes
Advanced Visualization

→ KNIME Text Processing

The KNIME Textprocessing feature enables to read, process, mine and visualize textual data in a conveniet way. It provides functionality from

natural language processing (NLP)
text mining
information retrieval.

To process texts with the KNIME Text Processing feature, usually six different steps need to be

accomplished. These steps are:

Readingand parsing: Text input/output

1. Enrichment: Named entity recognition

2. Preprocessing: Filtering and manipulation

3. Frequencies: Word counting and keyword extraction

4. Transformation: Bow and vector representation

5.Visualization: Tag cloud

Advantages using Knime in text Mining Process

=> Text mining process reveals new heterogeneous insights into social media customer behavior, which would not have been possible by using either technique on its own.

=> Combining sentiment analysis from online forum posts together with reference structures from the quotation network has allowed us to position negative and positive users in context with their relative weight as influences or followers in the underlying discussion forum.

=> This approach can also be enhanced by including additional relevant data sources regarding particular focus areas, such as company and product names, political parties, known users of products, and so on.

=> The additional data would further enhance the method capability of identifying, segmenting and drilling down to interesting user groups.

=> This technique introduces additional features into the user data, to improve the process of analytic investigation as popularized by the CRISP (Cross Industry Standard Process) methodology.

=> we see this technique as a basis for further data exploration. By using the KNIME platform, which allows not only traditional data mining, predictive analytics and machine learning, but also text mining and network analysis, we have been able to quickly and effectively create this new insight with a minimum of delay.

Disadvantages using Knime in text Mining Process

=> If we are using some nodes like extracting data from Document data extractor, Ngram creator it will take much time to process data from file.(Note:If file having more than 1000 lines)

=> If we are using Huge data the process will take around hours of time, if system configuration is low.

=> To Quick process of a Work flow we have to insert filtered documents.

=> We have to use appropriate node at preprocessing step other wise Redundant data occurs at Out put file.

Example::

The Nodes I used in Example,

→ PDF Parser: read all PDF files of a certain directory

→ POS Tagger:Tags words with part of speech tags.

→ BoW creator: Transforms a list of documents into a bag of words.

→ Term to String: Converts a term into a string.

→ Kuhlen Stemmer: Stems words with the Kuhlen algorithm.

→ POS Filter: Filters terms with specific part of speech tags.

→ Porter Stemmer: Stems terms with the Porter algorithm.

→ Replacer: Replaces in terms a certain character sequence by another one.

→ Stop word Filter: Filters terms which are stop words.

→ Number Filter: Filters term which are numbers.

→ Dict Replacer: Replaces certain substrings of terms with a particular replacement all specified in a dictionary.

→ Punctuation Erasure: Removes all punctuation marks from terms.

→ TF (Term Frequency): Computes the term frequency of terms.

→ IDF (Inverse Document frequency):

→ Chi-square keyword extractor: Extracts keywords, estimated as relevant by the Chi-square measure.

→ Frequency Filter: Filters terms with a certain frequency.

→ Tag Cloud: Displays a bag of words as tag cloud.

Tag cloud:

The KNIME Platform

2013-12-23T22:49:00.002-08:00

The KNIME Platform

KNIME [naim] is a user-friendly graphical workbench for the entire analysis process: data access, data transformation, initial investigation, powerful predictive analytics, visualisation and reporting. The open integration platform provides over 1000 modules (nodes), including those of the Knime Community and its extensive partner network.

The architecture of KNIMEwas designed with three main principles in mind.

.Visual, interactive framework

.Modularity

.Easy extensibility

KNIME Text Processing

→ IO subcategory of the Text Processing category. For formats such as DML, SDML, PubMed (XML format), PDF, Microsoft Word, and flat files, parser nodes are available.

→Stepwise execution ,Data table column types and conversionare specialties of KNIME.

→ KNIME implements its work flows graphically

→ Knime help to transforms a list of documents into a bag of words

→ After parsing the text and converting it into document cells, the documents and the included information can be enriched.

→ In the preprocessing step, terms are filtered and manipulatedin order to get rid of terms that do not contain content.

→ After preprocessing is finished, frequencies of terms in documents and the complete corpus can be computed.

→ Before regular KNIME nodes can be applied on the documents, the textual data has to be transformed into numerical data.

→ Knime also supports keyword extractor node analysis documentsand extracts relevant keywords using co-occurrence statistics.

→ Way to visualize long textsin tables withKnime Desktop tool

→ Access additional data sources outsideof KNIME

→ Scripting using languages like Perl, Python, Matlab, R, and Javain KNIME

→ Knime supports Algorithmslike Mining Algorithms (Hierarchical Clustering ,K-means , Fuzzy –c-Means ),Decision Tree Algorithm, Item sets / Association Rules ,Borgelt’s Algorithms

→ Custom node for scheduled execution is done in KNIME Server package.

→ The filter nodes offer a user-friendly way to filter out columns and rows without modifying any SQL statement by hand.

→ Programmers can easily write their own loop nodesby simply implementing an additional interface.

→ Almost all basic KNIME nodesthat create a model represent it in PMML

→ In conjunction with the integration of powerful libraries such as the Weka machine learning and the R statistics software, it constitutes a feature rich platform for various data analysis tasks.

Conclusion:Knime is a modular environment, which enables easy visual assembly and interactive execution of a data pipeline. It is designed as a teaching, research and collaboration platform, which enables simple integration of new algorithms and tools as well as data manipulation or visualization methods in the form of new modules or nodes.

How to use Pivote in sql

2013-11-21T23:55:00.002-08:00

Transpose a table in SQL
1) create a demo table with two fields
2) insert columns into demo table.
EX:

Pivote Query:

For Example:
select AccountID,AccountNumber from demo

select [1-10],[1-12],[1-19],[1-25],[1-26],[1-27],[1-33],[1-35],[1-36],[1-38] from
(
select AccountID,AccountNumber from
demo)src
pivot
( MAX(AccountID) For AccountNumber in ( [1-10],[1-12],[1-19],[1-25],[1-26],[1-27],[1-33],[1-35],[1-36],[1-38])) piv;

=> Now Pivote table Shown as:

A Cultural evolution

2012-02-21T09:51:00.001-08:00

"While millions in India don't even have a toilet to use, there are still hundreds of thousands with enough money to buy the latest technology - often spending much more than people in the US do "