Lateral-thinking

Navigating the Data Lake: Deploying Hadoop/HDFS and Spark on Kubernetes

2024-04-24T15:14:00.002+05:30

Introduction:

In the modern era of data-driven decision-making, organizations are constantly seeking innovative solutions to harness the power of big data. One such solution is the deployment of Hadoop/HDFS and Spark on Kubernetes—a strategic endeavor that requires careful planning and execution. In this blog post, I will explore the process of deploying these technologies on Kubernetes, unlocking new possibilities for data management and analysis. This is the same solution that we used when we started to build our no code/low code data pipeline and visualization platform DataSetu. For Datasetu we needed to create a Data lake for processing huge dataset with high performance. This how we went about it,

Chapter 1: Setting the Foundation with Kubernetes

Before embarking on our journey to the data lake, it's essential to establish a solid foundation. Kubernetes serves as the cornerstone of our infrastructure, providing the orchestration and scalability needed to manage our distributed systems efficiently. With Kubernetes in place, we are ready to proceed with confidence, knowing that our environment is robust and reliable.

Chapter 2: Deploying Hadoop/HDFS: Building the Data Fortress Our first destination in the data lake is the realm of Hadoop/HDFS—a proven solution for distributed storage and processing. By deploying Hadoop components on Kubernetes, we construct a formidable fortress to safeguard our data assets. With meticulous attention to detail, we configure HDFS settings to ensure fault tolerance and data integrity, laying the groundwork for seamless data management.

Chapter 3: Igniting Innovation with Spark As we venture deeper into the data lake, we encounter the dynamic landscape of Apache Spark—a powerful engine for large-scale data processing and analytics. Deploying Spark on Kubernetes enables us to leverage its advanced capabilities while maintaining flexibility and scalability. With Spark as our catalyst, we unlock new possibilities for real-time insights and predictive analytics, driving innovation across our organization.

Chapter 4: Optimizing Performance and Reliability In the ever-evolving world of big data, optimization is key to maintaining a competitive edge. We invest time and resources into fine-tuning our Kubernetes environment, optimizing resource utilization and maximizing performance. Through careful monitoring and proactive maintenance, we ensure that our Hadoop/HDFS and Spark clusters operate at peak efficiency, delivering reliable results and driving business value.

Chapter 5: Documenting the Journey As our deployment journey nears its conclusion, we take a moment to reflect on the lessons learned and achievements gained. Documentation becomes our compass, guiding future endeavors and informing best practices. By capturing insights and sharing experiences, we contribute to the collective knowledge of the data community, paving the way for continued innovation and success.

Conclusion: Deploying Hadoop/HDFS and Spark on Kubernetes is a strategic initiative that empowers organizations to harness the full potential of big data. By leveraging these technologies in a Kubernetes environment, businesses can achieve greater agility, scalability, and efficiency in their data operations. We have seen that with DataSetu platform. As we navigate the data lake, let us embrace the challenges and opportunities that lie ahead, knowing that with the right tools and expertise, the possibilities are limitless.

How to use Generative AI on custom data sets across industry to build domain specific use cases

2024-03-21T16:07:00.000+05:30

Generative AI is transforming the landscape of machine learning by enabling the creation of new content that can mimic human-like capabilities. One of the most powerful forms of generative AI is the Large Language Model (LLM), which can understand and generate human language with remarkable proficiency.
A key feature of LLMs is their ability to be re-trained or fine-tuned on custom data sets. This allows organizations to tailor the model's responses to specific domains or use cases. For instance, a legal firm could train an LLM on legal documents to assist in drafting contracts, while a medical research company could train it on scientific papers to generate new research hypotheses.
The process of re-training an LLM involves several steps. First, a suitable data set must be curated. This data set should be large enough to cover the desired scope and diverse enough to enable the model to learn various patterns and nuances. Next, the model must be fine-tuned, which involves adjusting the model's parameters so that it better aligns with the new data set. Finally, the re-trained model must be evaluated to ensure that it performs as expected on tasks relevant to the new domain. Sounds easy?
Re-training LLMs with custom data sets not only enhances their applicability but also improves their accuracy and efficiency in generating relevant content. As generative AI continues to evolve, the ability to customise and adapt these models will become increasingly important for businesses looking to leverage AI for competitive advantage.
Fine-tuning an LLM (Large Language Model) involves several critical steps to adapt the model to a specific domain. Here's a simplified overview of the process:
1) Data Preparation: Gather a dataset that is representative of the domain or task for which the model will be fine-tuned. This dataset should include examples of the type of content the model will generate or analyze.
2) Model Selection: Choose a pre-trained LLM that best fits the size and scope of your data. Larger models may require more data but can potentially yield better results.
3) Parameter Adjustment: Modify the model's hyperparameters, such as learning rate, batch size, and number of epochs, to optimize the training process.
4) Training: Use the custom dataset to train the model, allowing it to learn from the new examples and adjust its internal weights accordingly.
5) Evaluation: Test the fine-tuned model on a separate validation set to measure its performance and make any necessary adjustments.
6) Deployment: Once satisfied with the model's performance, deploy it to start generating or analyzing content as required.
Keep in mind, fine-tuning is an iterative process that may require multiple rounds of adjustment and evaluation to achieve the desired performance.

Custom data sets for re-training LLMs can vary widely depending on the specific needs and goals of the project. Here are some examples:
1) Customer Service Data: Transcripts of customer service calls or online chat logs can be used to train an LLM to understand and respond to customer inquiries.
2) Legal Documents: A collection of contracts, legal briefs, and case law can help an LLM learn the language of legal discourse for applications like automated contract analysis.
3) Medical Records: Anonymized patient records, treatment plans, and medical research papers can train an LLM to assist with medical diagnosis or literature review.
4) Technical Manuals: Manuals and documentation for specific products or technologies can help an LLM understand technical language and assist with customer support.
5) Literary Works: A corpus of literary texts can be used to train an LLM to generate creative writing or analyze literary styles.
6) Research Articles: A dataset of scientific articles from a particular field can train an LLM to summarize research findings or generate new research ideas.
We need to ensure that each of these data sets are carefully prepared so that they are representative, comprehensive, and appropriately formatted for the training process.

Let's consider a scenario where we have a dataset of customer service transcripts that we want to use to fine-tune an LLM. Below is a simplified example of how you might select, load, and train the model using this data:
This shows a basic framework for fine-tuning an LLM on customer service data. It includes steps for loading the data, preparing datasets, initialising the model and tokenizer, defining training arguments, and training and evaluating the model.

#python code

import pandas as pd
from transformers import AutoModelForCausalLM, AutoTokenizer, TextDataset, DataCollatorForLanguageModeling
from transformers import Trainer, TrainingArguments

# Load the dataset
data = pd.read_csv('customer_service_transcripts.csv')
texts = data['transcript'].tolist()
# Select a subset of the data for training
train_texts = texts[:int(len(texts) * 0.9)]
test_texts = texts[int(len(texts) * 0.9):]
# Save the training and test sets
with open('train_data.txt', 'w') as f:

f.write('\n'.join(train_texts))

with open('test_data.txt', 'w') as f:

f.write('\n'.join(test_texts))

# Load a pre-trained model and tokenizer
model_name = 'gpt2'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Prepare the datasets
train_dataset = TextDataset(tokenizer=tokenizer,file_path='train_data.txt',block_size=128)
test_dataset = TextDataset(tokenizer=tokenizer,file_path='test_data.txt',block_size=128)
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

# Define training arguments
training_args = TrainingArguments(
output_dir='./results',
overwrite_output_dir=True,
num_train_epochs=3,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
eval_steps=500,
save_steps=1000,
warmup_steps=500,
prediction_loss_only=True)

# Initialize the Trainer
trainer = Trainer(
model=model,
args=training_args,
data_collator=data_collator,
train_dataset=train_dataset,
eval_dataset=test_dataset)

# Train the model
trainer.train()

# Evaluate the model
trainer.evaluate()

What is big data ? and what role Hadoop has to play?

2011-04-22T20:03:00.000+05:30

In recent times we have been hearing , reading (in many blogs) quite often about cloud computing ,big data and HPC ie. high performance computing. Undoubtedly, these are the buzz words for this decade. If you have been in the industry for quite some time then you can recall last decade was web 2.0 decade.
So what is big data? Is it a new framework or new data modelling technique or part of NO SQL movement?
I have seen many times people relate big data to one of these or they are confused about it. Same is with HADOOP, it is thought as a NO SQL database framework.
Big data is nothing but just huge amount of data which requires mining. Per wiki, Big Data is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target currently ranging from a few dozen terabytes to many petabytes of data in a single data set.So big data is a hovering problem for organizations which deals in high volumes of data and requires storage,search, sharing, analytics and visualizing like Google Search, Facebook, Amazon or twitter. And with time this data will only grow and the analytics or mining would become bigger a problem.

Now the question could be how HADOOP comes in to picture? and are NO SQL databases are part of the problem?
Big data can be any kind of data either in RDBMS, NO SQL or software logs, sensors. Big Data requires exceptional technologies to efficiently process large quantities of data within tolerable elapsed times. Technologies being applied to Big Data include massively parallel processing (MPP) databases, datamining grids etc.
However, if we take google or facebook example. Google has it's own NO SQL database Bigtable which is nothing but a big hashmap (key-value pair) or it is data in JSON objects form. Similarly Facebook uses Cassandra for storing it's users data. The advantage of using these kind of databases is that the retrieval of data is considerably faster than the traditional RDBMS. Hence the through put is higher even with 100 millions of records.
So if it is that fast then where is the problem?
The problem is this data is unstructured unlike RDBMS where you just need to write the correct SQL to retrieve the desired data (using the grid computing or parallel computing). Hence , searching ,analyzing or visualization is a bigger problem on this kind of data.
Here we have MapReduce to rescue. MapReduce is a programming model and implementation developed by Google (Patented) for processing massive-scale, distributed data sets. And Hadoop is an Apache initiative inspired from Google's MapReduce. The framework is inspired by the map and reduce functions commonly used in functional programming.
Per Wiki, MapReduce is a framework for processing huge datasets on certain kinds of distributable problems using a large number of computers (nodes), collectively referred to as a cluster (if all nodes use the same hardware) or as a grid (if the nodes use different hardware). Computational processing can occur on data stored either in a filesystem (unstructured) or within a database (structured).

How does MapReduce works?
For a more theoritical aspect, you can look at below figure,

Hence hadoop is basically a framework for enabling powerful parallel processing of huge data. Figure below shows how MapReduce can be utilized for analytics purpose,

References

[1] Hadoop Wiki: http://wiki.apache.org/hadoop/PoweredBy
[2] Apache Hadoop: http://hadoop.apache.org/
[3] Hadoop tutorial : http://www.ibm.com/developerworks/java/library/j-javadev2-15/index.html
[4] Blog : http://cleanclouds.wordpress.com/2011/03/28/big-data-with-hadoop-cloud

class.forName(String className) used in JDBC unfolded

2011-01-15T16:21:00.000+05:30

During many interviews I have taken , I have observed even seasoned programmers find it difficult to explain the use of class.forname() method while making a JDBC connection object. In this post, I am just trying to explain this,
Here is a simple JDBC code,

import java.sql.*;
public class JdbcConCode {
public static void main(String args[]) {
Connection con = null;
String url = "jdbc:mysql://localhost:3306/nilesh";
String driver = "com.mysql.jdbc.Driver";
String user = "root";
String pass = "nilesh";
try {
Class.forName(driver);
con = DriverManager.getConnection(url, user, pass);
System.out.println("Connection is created....");
//TODO
//rest of the code goes here
} catch (Exception e) {
System.out.println(e);
}
}
}

Let's go through the code above, the url indicates the location of the database schema, here the database is nilesh.
Then there is Driver which is the fully qualified JDBC driver class name.
Other two variables are used to declare the username and passwords to access the database. Just for the purpose of this code and to maintain simplicity ,I have mentioned all these variables inside the class itself, generally, this kind of details are retrived through the properties or xml files.Let's not delve into those details, the actual code starts after the try{ block,
class.forName(String className)

Basically, here the forName() call tries to locate, load and link the driver class to JDBC API. With in java , dynamic loading is acheived by class.forName() and it is the basis of core reflection functionality provided by Java.
Class.forName attempts to load the class whose name is
given in its only argument (driver class), and returns the Class instance representing that class. In the
event that the Class could not be found, resolved, verified, or loaded, Class.forName
throws one of several different Exceptions, all of which are listed in the javadoc page for
java.lang.Class [1].
Back to the code, here the class.forName() is called as,
class.forName("com.mysql.jdbc.Driver");

Using the complete package name , it first locates the class binary in the classpath and then loads that as a object representation in memory.
Once the Driver class is loaded in memory , it's static code block is executed which has something like,

static {
try {
java.sql.DriverManager.registerDriver(new Driver());
} catch (SQLException E) {
throw new RuntimeException("Can't register driver!");
}
}

That is the Driver gets registered with JDBC api as soon as class.forName() executes. So it calls a static method in the java.sql.DriverManager class which apparently registers a copy of itself when it loads.

Since the Driver is registered the next line of code just get us the Connection() class object to work with database. But thats not that straight forward.
We all know that the DriverManager class returns a database connection given a JDBC URL string, a username and a password. However, in order to create that connection, the
DriverManager class has to know which database driver you want to use. It does that by iterating over the array (internally a Vector) of drivers that are registered with it (ie: the registerDriver(Driver driver) method ) and calls the acceptsURL(url)) method on each driver in the array, effectively asking the driver to tell it whether or not it is compatible to handle the JDBC calls.
So this explains how the class.forName() works for you.

Exporting/Saving Putty/SecureCRT/SecureFX sessions on Windows!!

2010-04-25T16:28:00.001+05:30

This is my first stint with blogging and I thought why not just start with sharing some useful information/tricks that can save you some time.

To begin with , here are couple of tips that could save you some time.

Actually, I had once this problem where I have to take backup and move my work data to a new laptop and as we normally do , I also put my data on some external drive and copied it to the new box. During all this exercise I realized that by this I can only back up the data, what about the work I am doing..my configurations and installations...? What the heck...I need to spend lot of time to redo all that...crap.

I was using Putty, SecureCRT and SecureFX to log on to client side boxes and to work upon. And all three had at least 30 saved sessions having IP/user/pwd information, reconfiguring those sessions would be a daunting tasks. I did some research "well by that I meant googling also..:)" and here is how a little information can help you..

If you are using Putty on windows...
Putty actually does not have any config or ini file , it writes all the user created sessions in windows registry. You can find the sessions saved in putty at HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\Sessions location. That's it...we are almost done. Just export this key and save it to some location or external drive if you plan to move to some different windows box. Last thing you need to do is import this key to any other windows system where putty is running.

If you are using SecureFx/SecureCRT on Windows...
SecureCRT saves the user sessions information under windows user home directory. \AppData\Roaming\Van Dyke Technologies\SecureCRT\Config\sessions. You just need to copy the sessions folder and save it to some location. Just as a word of caution..., with Windows 7, Microsoft has changed the users home directory to C:\Users\. The same is c:\documents and settings\ in WindowsXP and prior versions.
I guess you know what is coming next..I need not to explain how to copy the sessions folder to the same location on any other windows box where you want to setup secureCRT/SecureFX and yes...you do not want to redo all the session creations..:)

Please leave you comments..how did you find it? and remember.. this is my first blog..:)