Information Well

Basic CLI Application Using Golang

noreply@blogger.com (Kapil Garg) — Fri, 20 Sep 2024 15:59:00 +0000

Writing a Basic CLI Application Using Go

Go is a great language for building command-line tools due to its simplicity and powerful standard library. In this article, we will go through how to create a simple CLI app in Go that accepts commands and flags from the user.

Step 1: Install Go

Before you start, ensure that Go is installed on your machine. You can download it from here. Once installed, verify by running:

go version

Step 2: Set Up Your Project

Create a new folder for your project and navigate into it:

mkdir cli-app
cd cli-app

Step 3: Write Your CLI App

In your project folder, create a file called main.go. This is where we'll write our CLI code.


package main

import (
    "flag"
    "fmt"
    "os"
)

func main() {
    name := flag.String("name", "World", "a name to say hello to")
    flag.Parse()

    if len(os.Args) < 2 {
        fmt.Println("Usage: go run main.go --name [your name]")
        os.Exit(1)
    }

    fmt.Printf("Hello, %s!\n", *name)
}

Step 4: Run the Application

You can now run your CLI app by using the following command:

go run main.go --name=John

This will output:

Hello, John!

If you omit the --name flag, it will default to:

Hello, World!

Step 5: Add More Functionality

You can add more flags to extend your CLI tool. For example, to add a flag for age:

age := flag.Int("age", 0, "your age")

Then use it in the output:

fmt.Printf("Hello, %s! You are %d years old.\n", *name, *age)

This simple guide demonstrates how easy it is to build a basic CLI application using Go. By using Go's flag package, you can create commands and flags to handle user input efficiently. With more advanced packages, you can build even more powerful CLI tools.

Why Interfaces are Awesome in Go

noreply@blogger.com (Kapil Garg) — Sun, 08 Sep 2024 08:36:00 +0000

Why Interfaces are Awesome in Go Language

Go, or Golang, is known for being a simple, yet powerful programming language. One of the core concepts that makes Go stand out is interfaces. Interfaces in Go help you write flexible, reusable, and maintainable code. They allow different types to share common behaviors without worrying about the exact type behind the scenes.

In this article, we'll dive into why interfaces are awesome, with easy-to-understand examples. We’ll also give some tips on how to make the best use of them in your Go programs.

What is an Interface?

An interface in Go defines a set of method signatures (rules), but it doesn't provide the implementation. Instead, any type that implements these methods can be said to "satisfy" the interface. This allows you to write code that can work with different types as long as they share certain behaviors.

Basic Example of an Interface

package main

import "fmt"

// Define an interface with one method
type Speaker interface {
    Speak() string
}

// Define a type that implements the Speaker interface
type Dog struct{}

func (d Dog) Speak() string {
    return "Woof!"
}

// Another type that implements the Speaker interface
type Cat struct{}

func (c Cat) Speak() string {
    return "Meow!"
}

func main() {
    var animals []Speaker
    // Dog and Cat both implement the Speaker interface
    animals = append(animals, Dog{}, Cat{})
    for _, animal := range animals {
        fmt.Println(animal.Speak())
    }
}

In this example, we define a Speaker interface that requires a Speak() method. Both Dog and Cat implement this interface by providing their own versions of the Speak() method. When we loop through a slice of Speaker types, the correct Speak() method for each type is called.

This flexibility is what makes interfaces so powerful. You can handle multiple types without needing to know their exact types, as long as they implement the required behavior.

Why are Interfaces Awesome?

1. Encapsulation of Behavior

With interfaces, you can focus on behavior rather than specific types. In the example above, the Speaker interface doesn't care whether it's a Dog, Cat, or some other type. It only cares that the type knows how to Speak().

2. Flexibility

Interfaces let you write flexible code. You can add new types that satisfy the interface without changing existing code. This makes it easier to extend and maintain your programs.

For example, if you later decide to add a Bird type, it can easily be added to the animals list without changing anything else:

type Bird struct{}

func (b Bird) Speak() string {
    return "Chirp!"
}

Just add Bird{} to the slice of animals, and it works perfectly:

animals = append(animals, Bird{})

3. Loose Coupling

Interfaces allow you to decouple different parts of your code. The code using the interface doesn't need to know the details of the types that implement it. This leads to more modular code, which is easier to test and reuse.

Making Best Use of Interfaces

Now that we’ve seen why interfaces are great, let’s talk about how to use them effectively.

1. Design for Behavior, Not Data

When designing interfaces, focus on what the type should do, not how it stores data. Instead of thinking, "I need an interface for animals with legs," think, "I need an interface for things that can Speak()." This encourages flexible design and avoids overcomplicating your code.

2. Keep Interfaces Small

Go encourages small, focused interfaces. A good rule of thumb is to keep interfaces with just one or two methods. This makes them easier to implement and encourages composability.

Here’s a good example of small interfaces:

type Writer interface {
    Write(p []byte) (n int, err error)
}

type Reader interface {
    Read(p []byte) (n int, err error)
}

Both Writer and Reader interfaces have just one method, which makes them easy to implement. You can combine them in more complex interfaces when needed:

type ReadWriter interface {
    Reader
    Writer
}

3. Use Empty Interface with Caution

Go has an "empty" interface, interface{}, which can hold any type. This is sometimes useful, but it should be used sparingly. Overusing interface{} can lead to code that’s hard to understand and error-prone, because you lose the benefits of type safety.

func PrintAnything(i interface{}) {
    fmt.Println(i)
}

While this works for any type, it's better to use specific interfaces or types whenever possible to keep your code clear and maintainable.

4. Use Type Assertions to Access Underlying Types

Sometimes, you might need to access the underlying type that implements an interface. You can do this using type assertions. Here's an example:

func SpeakLoudly(s Speaker) {
    fmt.Println(s.Speak())

    if dog, ok := s.(Dog); ok {
        fmt.Println("This is a dog, let's make it louder: WOOF!")
    }
}

func main() {
    SpeakLoudly(Dog{})
    SpeakLoudly(Cat{})
}

In this example, we use a type assertion (s.(Dog)) to check if the Speaker is of type Dog and handle it differently.

Conclusion

Interfaces are awesome in Go because they provide flexibility, loose coupling, and allow for clean, maintainable code. By focusing on behavior rather than concrete types, you can write code that works with a wide range of types, making it easier to extend and maintain over time.

To make the best use of interfaces in Go:

Design interfaces around behavior.
Keep them small and focused.
Avoid overusing the empty interface.
Use type assertions wisely when necessary.

By following these practices, you can make the most out of Go’s powerful interface system and write cleaner, more efficient code.

Happy coding! 🎉

How to Start Learning Go Language: A Beginner's Guide

noreply@blogger.com (Kapil Garg) — Thu, 05 Sep 2024 05:24:00 +0000

How to Start Learning Go Language: A Beginner's Guide

Introduction to Go Language

Go, also called Golang, is a programming language developed by Google in 2009. It’s simple, fast, and great for building reliable software. If you’re interested in learning programming or want to try a new language, Go is a great choice. This article will help you understand how to start learning Go in a simple and easy way.

Why Learn Go?

Before we jump into how to learn Go, let’s look at why Go is popular:

Simple syntax: Go is easier to learn compared to other programming languages like C++ or Java.
Fast performance: Go runs quickly, making it great for web servers and large systems.
Great for beginners: Go has built-in tools that make coding easier, especially for new programmers.
Growing demand: More companies are using Go, so learning it can help in your career.

Steps to Start Learning Go

1. Set Up Go on Your Computer

To learn Go, you need to install it on your computer. Follow these steps:

Visit the official Go website to download the Go installer.
After downloading, follow the instructions to install Go.
To make sure Go is installed, open your terminal or command prompt and type:

go version

If it shows the version of Go, you're ready to start coding!

2. Understand the Basics of Go

Now that you’ve installed Go, it’s time to understand the basics. Here are some key concepts you should start with:

Variables: Learn how to store and use data in Go.
Functions: Go uses functions to organize code into reusable blocks.
Control structures: Learn how to use if-else, for loops, and switch statements.
Go’s syntax: Go uses simple syntax, so it’s easier for beginners to read and write.

You can find a good introduction to Go's syntax in the Go Tour, an interactive tutorial from the Go team.

3. Practice by Writing Small Programs

One of the best ways to learn any programming language is by writing code. Start with simple programs, like:

Printing “Hello, World!” in Go:

package main

import "fmt"

func main() {
    fmt.Println("Hello, World!")
}

Writing a program that adds two numbers:

package main

import "fmt"

func main() {
    var a int = 5
    var b int = 10
    fmt.Println("Sum:", a+b)
}

Keep practicing with small projects to build confidence. For more practice, visit Go by Example, which has examples and explanations for many Go programs.

4. Explore Go’s Standard Library

Go has a rich standard library that makes it easy to handle tasks like working with files, networking, and more. You can use Go’s documentation to explore different libraries and learn how to use them. Check out the Go Documentation to see all the built-in packages.

5. Work on Real-World Projects

Once you’ve got a grip on the basics, you can move on to building larger projects. Try to build simple applications like:

A to-do list app
A basic web server
A command-line tool

These projects will help you understand Go's power and flexibility. You can also contribute to open-source Go projects on GitHub.

6. Join Go Communities

Learning from others can speed up your progress. Join Go communities where you can ask questions, share knowledge, and learn from other developers. Here are some good places to start:

Go Forum: A great place to ask questions and interact with other Go learners.
Reddit Go Community: Useful discussions, tips, and news.
Go Slack Channel: Join the Gophers Slack and connect with Go developers.

Helpful Resources

Here are some additional learning resources:

Go Official Documentation – Complete documentation from the Go team.
A Tour of Go – An interactive tutorial for beginners.
Go by Example – Simple examples for learning Go’s syntax.
Exercism.io Go Track – Solve exercises and get feedback from mentors.
Go’s Standard Library – Explore the built-in libraries in Go.

Final Thoughts

Learning Go can be a fun and rewarding experience. With its simple syntax and powerful tools, Go is perfect for beginners and experienced programmers alike. Start slow, practice writing code, and use the resources mentioned above to build your skills. As you grow more comfortable with Go, try solving real-world problems and joining the Go community.

Happy coding!

The new generation storage: SSD

noreply@blogger.com (Kapil Garg) — Thu, 10 Jan 2019 14:59:00 +0000

This time we'll be learning about something which has already been in the market for a long time but still many of us doesn't understand its importance or how it is significantly better than its counterparts. That new thing is Solid State Drive (SSD). SSD is just another storage media like your pendrive and harddisk but it is way faster than either of them. In this post, we will learn what really makes an SSD faster than other storage media and how it can change our life forever.

What really is an SSD ?
What makes SSD better ?
Role of SSD in laptops

What really is an SSD ?

SSD is a flash device just like pendrives, micro sd cards and embedded storage of our mobile phones. A flash device is basically made of transistors and semiconductors as compared to harddisk which has magnetic disk and moving parts. I may introduce some heavy terms here which may not make sense to you much but keep reading. So like i was saying, flash devices have billions of transistors which are used to store data. Multiple transistors are joined to form a kind of a cell. A cell either can be NAND cell or NOR cell based on the configuration. One cell can hold one of 2 states (there are cells which can hold one of many states but lets stick to basic cell) either one or zero. As we all know (alright, some of us know), that everything is binary when it comes to computers, a combination of cell can store any value. All you have to do is to turn some cells ON and turn some cells OFF to store your particular value. This is achieved by passing a current to those cells and thus changing their states.

What makes SSD better ?

As you know now that storing a value in a cell is just a matter of passing the current, one may wonder how much time does it take. Well, the answer is 0 seconds, 0 milliseconds, 0 microseconds. Yes, that happens in nanoseconds. That said, now lets talk about harddisk. Harddisk is a giant blunder of magnetic disks and moving arms. It served the purpose of storing data to a good time until people realized that computers can perform better if all the data required by applications is already in RAM because fetching it from harddisk is a slow process and not even the fastest processor can do any good with it. So the bottom line is, storage media is the bottleneck for high performance. The way harddisks operate is itself so optimized that there is no way to get something better out of it. If you ever open a harddisk (or see the diagram below), you will notice that it has one pin which needs to go to the location where the requested data is stored. This involves movement of the pin and is a microseconds job.

On the other hand, SSD just requires current to be passed to some specific cells which is a nanoseconds job. SSD itself has its own RAM and a simple processor which keep things moving. Thus SSD can serve better and can provide data at a much faster rate than traditional harddisk drives.

Role of SSD in laptops

Like i said earlier, SSDs' write and read speed is pretty much higher than harddisks and applications which run on our laptop constantly require memory to write and read stuff. Even when you start an application, it boots up when processor reads the binary from the storage. So SSD can not even make your applications run smoothly but it can also improve applications startup time. I myself has experienced this on my intel-i3 laptop where my OS (ubuntu 18.04) is installed on one of samsung SSD. The boot up time has improved to 2-3 seconds and even android studio runs like a charm without GPU (But that has something to do with 8GB RAM also). So if you go to buy a laptop for your college work then i would suggest you to go for a laptop which has all basic parts like good enough processor(i5), may be GPU card, decent screen, which can come at a decent price of 40k. Just make sure it has additional RAM slot. Then spend additional 12k to buy more RAM and one 256GB internal SSD. Go to some laptop repair shop and ask him to replace your HDD with your brand new SSD and your CD player with the HDD he just removed. That way you can install your applications and OS (like windows 10) on your SSD and keep your other stuff like movies on HDD. Also ask the repairman to install that extra RAM to the additional RAM slow in your laptop. Just make sure you buy the RAM of the same frequency as the already installed one. At the end, you will have a powerful laptop in under 55k. Companies offer laptops with same configuration at a 15-20k higher price which is not at all required.

This was it for today's post fellas. If you need any help selecting the laptop or which SSD to buy, just drop a comment and i will reply faster than SSD serves data :D.

Hash Tables : Designing Hash Functions

noreply@blogger.com (Kapil Garg) — Thu, 22 Feb 2018 15:24:00 +0000

In the last post, we saw how collision in handled in a hash table. Now its time to explore some real world scenarios because come on, what is the use of learning all this if we can't apply it in real world softwares. As we learned that open addressing and chaining were both good in different scenarios only if we met a condition that the keys are uniformaly distributed. But its not the application job to pass distributed keys to hash tables, its our job to map any key to some random location. We do this using a hash function (or prehash function, whatever term you prefer). A hash function should map the keys as uniform as possible. So lets discuss some decent hash functions to some high end hash functions.

Desining Hash Functions

Hash Functions do the job of mapping a key to some location in hash table. This is achieved by doing some fixed operations on the key and generating a number between 0 to the capacity of table. Hash functions highly determine the overall complexity of hash table for insert and search operations because as we saw in open addressing if a lot of keys are mapped to same location then it will shorty form a clusture there and then inserting a new key or searching an old key will not be a O(1) operation anymore. So we want our hash function to have some properties. Lets see what they are.

Properties of Hash Functions

Desirable properties of a hash function are:

Uniform Distribution: Each key should be equally likely hashed to any slot of the table independent of where other keys hash. This means that hashing of a key should not be related to hash table size or placement of other keys.
Time Independent: Hash values of keys should be changed over time. This implies to the fact that if i hash my key now and if i hash the same key one year later, the resulting hash values in both the cases should be same.

Basic Hash Functions

As we saw what we want in our hash functions, we are ready to write some. We are making one assumption that the key is always integer. Later we'll see how to convert any object to an integer atleast in c/c++. Some of the hash functions which are used commonly are described below.

Division Method: In this method, we take modulo of key with table size. This will generate a number between 0 and size of table. This may create a problem if keys and size of table have common factors, then only some of table space will be utilized.
Multiplication Method: Opposite of the division method is multiplication method. In this method we multiply the key with some FIXED number and take out a bunch of bits.
Universal Hashing: In this method, we compute a value by multiplying the key with prime number and then taking the mod with size of table. The worst case possibility of colliding two keys is 1/(size of table) which is ideal situation.

Cryptographic Hash Functions

Hash Functions are not only used to compute the location of a key in a table but also to convert any given key or should we say any data to a fixed length digest. This digest is unique for each key and thus it is made sure that no 2 keys have same digest. Such type of hash functions are used in cryptography to encrypt a message or a password to transfer it safely from one place to another. Cryptographic hash functions must have a property other than the above two and that is they should be one way. This means that it must not be possible to recover the key from its digest. There are many widely used hash functions which are either used for encryption or just to compute checksum. Two most commonly used and famous hash functions are MD and SHA. We won't go in detail how these work because there are libraries to use these hash functions. Lets see what are those.

MD (Message Digest): The MD family comprises of hash functions MD2, MD4, MD5 and MD6. It was adopted as Internet Standard RFC 1321. It is a 128-bit hash function. MD5 was the most popular hash function for some years until in 2004, collisions were found in MD5. Since then it is not recommended to use but still people use it for less sophisticated applications where collisions are allowed.
SHA (Secure Hash Function): Family of SHA comprise of four SHA algorithms; SHA-0, SHA-1, SHA-2, and SHA-3. SHA-1 is most widely used hash function of this family. It is also employed in SSL (Secure Socket Layer).

This was it for this post. We'll see in the final post of this series how python manages it dictionaries and lists. Also share this article to your nerd friends to make them aware of where the world is heading. Keep Hashing

Hash Tables : Handling Collision

noreply@blogger.com (Kapil Garg) — Sun, 11 Feb 2018 06:11:00 +0000

We saw in the previous post how a simple hash table can be constructed without much effort. But, and that is a big But, that table was prone to collisions and could result in corrupted data. Of course we don't want a data structure which can corrupt our data even if it is faster. So to avoid data corruption we will study some collision resolution techniques and will see how to evenly distribute a key over the table to minimize collision. Lets continue the journey.

Collision Resolution

First of all, what is collision ? Collision is a phenomenon which happens when 2 or more distinct keys are mapped to the same hash table entry. But we don't want that so using some technique, we relocate that key to some other empty location. The technique used to do that is called collision resolution technique. There are 2 types of techniques in use. One is called Chaining and Other is Open Addressing. Open Addressing has further more types but the concept is basically the same. Lets start with chaining first and see how it can be useful.

Chaining

As the name suggests, In chaining technique , we create a list of colliding keys for every hash entry location. So our hash table is basically an array of pointers which are pointing to the corresponding list. Whenever 2 distinct keys are mapped to same location, we append them to that location's list. The below picture will make it more clear.

Now the question comes about searching for a key in chained table. As you can see in the above picture, we cannot find a key in one go. We'll first have to find the entry location in hash table and then will have to traverse the whole list associated with that location. Each list node has the original key and thus we can compare our key with node's key to find the match.

Chaining may seem to solve the problem of collision and in reality it does but there is an overhead of memory. Each new list node must have a pointer to the next node. Also you may have observed in the above picture that a bad hashing technique may try to place all the keys in one location, thus making search a linear operation. Thus we can get amortized O(1) time for get, set and search operations if our hashing technique evenly distributes the keys

Open Addressing

After seeing how the chaining works, its time to explore another collision resolution technique which utilizes the hash table to its full potential. As we saw in chaining, a lot of keys can be placed in one location and then many other locations remain empty and not fully utilized. Open addressing solves this problem. There are many types of open addressing techniques but in this post we will look at Linear Open Addressing

In linear open addressing, we first find a key location inside hash table where it should sit using hash function. Then we check whether that location is empty or not. If its empty, place the key there otherwise, find the next empty location linearly in circular motion, i.e., go to the start to the table if there is no empty location till the end of the table. After finding an empty entry, place the key there. The following picture will help you understand this better.

Now lets to come to search. Search operation is somewhat similar to insert operation. When we want to search a key in table, we first find its location using hash function. After that, we linearly go through all the keys and match with our key until we find it. If we encounter an empty entry during our search or if we are back to where we started, it implies that the key is not present. As you can imagine, linear probing may result in clustures if our hash function is not uniformly distributive enough. The other type of open addressing which tries to avoid clustures is Quadratic Probing. In this we don't go in linear fashion instead we go in steps of power of 2. Like we first go 1 step to find next empty bucket, 2 steps if not found, 4 steps if no empty location found, 8 steps and so on.

As we saw in this post Open addressing solves memory problem but chaining is faster for high load factors [load factor = (total number of keys)/(number of entries in table)]. In real world cases, people prefer open addressing as Open addressing is usually faster than chained hashing when the load factor is low because you don't have to follow pointers between list nodes. If the load factor approaches 1, open addressing becomes very very slower. But usually, the table size is kept double the number of total keys in the table and thus open addressing achieves good performance. This was it for this post, in the next post, we'll see how python manages its dictionaries and how we ensure that table size is always double the number of entries. Stay tuned. Also share this article to your friends if this helped you in any way.

Hash Tables : Scratching the Surface

noreply@blogger.com (Kapil Garg) — Sat, 03 Feb 2018 11:58:00 +0000

Have you ever worked with key-value data stores where you can attach one value to a key and store it in some form to perform easy lookups ? If yes, then i guess you have already seen hash tables in action. Hash Tables are so common these days that you don't even realise when you start using them but not a lot of people are familiar with the internals of hash tables and how these can be used in real world problems like cryptography. This a series of articles all about hashing. So sit tight and enjoy the journey.

What Problems Does Hashing Solve ?

Hash Tables are the most popular data structure in computer science. We forget to worry about time complexities of finding our data in hash table as its basically constant. But it doesn't show how hashing solves real world problems. Following are some problems which are being solved by hashing only.

Computer Cache is internally a hash table with keys being the tag bits (refer to cache post)
Python Dictionaries are implemented using hashing scheme only.
Associative Arrays in any programming language such as php are maintained using hashing.
Cryptographic Hashing uses hash functions to digest a variable length message to a fixed length string.

Apart from the above scenarios, you can use hash tables anywhere you may find appropriate. As hash tables store data in unordered fashion, we can't use it in sorting applications.

Simple Implementation of Hash Table

We can consider a hash table as an array where each array entry holds the original key and value pair. If the key itself is a value then there is no need to store value separately. For any hash table to work, you need some way to map keys to array indices. This mapping is done by a hash function which can be anything you want it to be. Desirable properties of a hash function are:

Uniform Distribution of keys over hash table. We'll see shortly why is that needed.
Mapping of one key to one hash table entry must not change over time.

Considering the above properties in mind, we can build our hash function using many simple arithmetic techniques. One of them is modulo operator. Assume we are managing integer keys and there is a value associated to each key. Thus we need our hash table to perform constant time lookup of integers. This can be achieved by using an integer array. Our hash function which maps a key to one hash entry location performs the modulo operation on the key and returns the index in the array where the key should be placed. Following is the implementation of such a hash table in c++.

#include <iostream>
class HashTable{
public:
    static const int CAPACITY = 10;
private:
    int* entries;
    bool* valid_map;
    int size;
public:
    Table(){
        entries = new int[CAPACITY];
        valid_map = new bool[CAPACITY];
        size = 0;
    }
    void insert(int key, value){
        int index = hash(key);
        entries[index] = value;
        valid_map[index] = true;
        size++;
    }
    void delete(int key){
        int index = hash(key);
        valid_map[index] = false;
        size--;
    }
    bool exists(int key){
        int index = hash(key);
        return valid_map[index];
    }
    int find(int key){
        int index = hash(key);
        if(valid_map[index] == true){
            return entries[index];
        }
        return -1;
    }
private:
    int hash(int key){
        return key%CAPACITY;
    }
};

As you see above, our hash table is capable of storing 10 keys. But there is one major problem with the above approach. There is no collision handling. If two keys map to same location then second key will overwrite the first one. We'll see how can we tackle Collisions in the next post and will look at how python handles hashing of objects.

Cache : Replacement, Mapping and Writing

noreply@blogger.com (Kapil Garg) — Mon, 27 Mar 2017 19:05:00 +0000

In the previous article, we discussed about cache and its properties. We also learnt some concepts and terminology related to cache. If you haven't read that article, i will recommend you to read that article first. This article contains many terms which i've discussed in that article. In this article, we will learn cache replacement strategies, cache mapping techniques and cache writing techniques. Below is the index of this complete cache tutorial.

Cache Introduction
Cache Organisation
Understanding a few Concepts
Cache Replacement Strategies
- Random Replacement
- Least Recently Used (LRU) Replacement
Cache Mapping Techniques
The Tedious Task of Cache Writing
- Write Through
  - No Allocate (write around)
  - Allocate
- Write Back
Basic Cache Simulator

Cache Replacement Strategies

In the previous article, we looked at the process of searching data for an address in the cache but there was 1 step missing in that process. We didn't look at the scenario of cache miss. In case of a cache miss, we look at other invalid entry on that index page (note down that we are talking about an index page not the whole index table, recall the terminology from previous article if you want). We add the missing entry in cache index from main memory. But when there is no invalid entry in our cache index then we will have to replace some existing entry to make room for our new entry. Replacing an existing entry is a frequent task and thus plays an important role in determining cache efficiency. There are majorly 2 techniques for finding the entry which is to be replaced by the new one.

Random Replacement

Using this strategy, we find any random entry on that index page and replace it with the new one. This strategy has the advantage of being easy to implement but it may also increase cache misses if we are not lucky enough. A better random algorithm can ensure some improvement but still its random.

Least Recently Used (LRU) Replacement

This is the most widely adopted technique and is somewhat more intuitive. In this strategy, we select the entry which has not been used for a while on the selected page. This means that the oldest used entry is replaced by the new one. This is intuitive because there is no worth of keeping an entry in the index which is lying stale. This strategy ensures better cache performance but it is also complex to implement mainly in hardware. But the algorithm used in strategy can be adopted in many different problem domains such as recommended product listing.

Cache Mapping Techniques

There are total 3 mapping techniques all of which have their own advantages and their disadvantages but all of them have many things in common like tag bits, line bits, word bits, byte select bits etc. The only thing which differs in these techniques is cache entries per index page. This also determines the total number of index pages or total number of cache lines. Correct Cache line is determined by line bits. Below is a list of formulas which you can derive easily if you understood the previous article

Formula for total number of cache lines (index pages) :
(cache_index_size)/(number_of_entries_per_page)

Formula for total number of byte select bits :
log₂(word_size)

Formula for total number of word bits :
log₂(number_of_words_per_block)

Formula for total number of line bits :
log₂(number_of_cache_lines)

Formula for total number of tag bits :
total_address_bits - number_line_bits - word_bits - byte_select_bits

Direct Mapped Cache

In this technique, there is only 1 index entry per index page. This implies that we won't have to perform a linear search for our tag bits. Once we find the correct page , we examine its tag bits to check if this is the address we are looking for.

Set Associative Mapped Cache

In this technique, we explicitly define the number of entries in one index page. For eg, in a 16 item cache index, if we set number of entries per page = 2 then there will be total 8 index pages (cache lines = 16/2). Thus we will have 3 line bits. Once we find the correct page, there will be 2 entries. We will have to check both of them to determine whether we have a cache hit or a cache miss. If we denote the number of entries per line by N then the technique is said to be N-way set associative mapping technique. Our previous example has a 2-way set associative mapped cache.

Fully Associative Mapped Cache

In this technique, there is only one index page which contains all the index entries. Thus we don't require line bits for this technique. This technique make most use of unoccupied entries but it also introduce inefficiency due to search needed to match each entry against given address.

The Tedious Task of Cache Writing

Cache writing is the process of updating cache data after computations. Cache writing is somewhat a complex task as compared to reading because we need to update main memory as well to avoid data inconsistency.

Write Through

This is the technique in which we update the main memory along with cache memory on each update. This technique keeps the main memory always in sync with cache. So basically whenever we want to update some address data, we search it in the cache, update it in cache if there is a cache hit and update it in main memory too. But there are 2 variants of this technique based on the action when there is a cache miss.

No Allocate (write around) In case of cache miss, we simply update the main memory without writing anything to cache.
Allocate In case of cache miss, we store index entry and data in cache and update the main memory as well.

Write Back

In this technique, main memory is not updated until the cache entry needs to be replaced. To improve the efficiency further, we maintain a dirty bit associated with each cache index entry. This dirty bit represents whether the cache data has been modified or not. In case when the cache entry needs to be replaced, we update the main memory only if this dirty bit is set to 1 otherwise we just replace the entry.

This was all about cache and its related concepts. We successfully summed up all this in 2 articles. This post is more theory and less pictures so i won't blame you if you found it too boring to read, but it also clears all the basic fundamentals about cache and will make you ready to implement your own cache simulator. If you think this article helped you in some way then do share it. Also if you have any doubts, feel free to comment it below.

Wondering Cache

noreply@blogger.com (Kapil Garg) — Fri, 17 Mar 2017 12:21:00 +0000

Some of you may have already heard about the computer cache in your curriculum or some of you may have not. But this article is intended for all of those who want to understand cache concepts. The next part of it consists of developing a cache simulator using C programming language. I believe that this article will give a clear insight of cache and how it works. So lets see what we will be learning in this series of articles.

Cache Introduction

The first thing we should know what are we talking about. Cache is a highly efficient (really fast) memory which can bring data to processor at a much more speed than our typical RAM. CPU cache memory operates between 10 to 100 times faster than RAM, requiring only a few nanoseconds to respond to the CPU request. Cache operates faster because it resides usually on the same chip as the processor. But it makes them expensive. Also caches are smaller in size (around 512KB to 3MB) whereas computer RAM can be upto 16GB. The reason why caches are smaller is because it makes the mapping techniques faster and also reduces cost for a normal computer. So cache behaves a temporary storage for RAM data. Computer processor looks for data in cache first and if it is not present, then it gets the data from RAM and stores it in cache for future reference.

Cache Organisation

As we know now that cache is smaller than RAM, it cannot store all the RAM data. In RAM memory, a memory cell number serves the purpose of index for the memory which is also our physical address. For example, if we want to store x on some address 32 then this means that we just go to 32nd cell number into RAM and writes our data. So we don't need to consider an index table for RAM because RAM is all the memory we have got.

But the same address cannot work for cache memory because lets say our cache has only 16 cells. This means we need to create a seperate index table for our cache memory into which we will search for cache cell number which contains data for given physical address. For our earlier example, if our cache has only 16 memory cells and we want to search for data for address 32 then we will look into cache index and find which cache cell holds data for address 32. One way would be to have a 16-entry index table where each entry represents the cache cell number and contains the physical address for which the data is stored in that particular cell. The following picture will give you a better idea than my words (But read my words again after looking at the picture, you will understand what i am saying).

So this is how it works.There are several ways to find a physical address in cache index. One of which would be to search all index entries one by one but that will make our cache inefficient. Thats why we use mapping techniques which maps RAM addresses to Cache addresses and make it efficient. The next article will make you understand what are the main mapping techniques are and how they work.

Understanding a few Concepts

There are 3 mapping techniques all of which have their advantages and disadvantages. But all of them have some things in common. Each technique breaks a physical address into some components which we will discuss first. If you don't understand what these terms mean then its not a problem as the picture will make their meanings more clear. There are also some formulas related but you don't need to cram them. You will understand how these formulas are derived on your own.

Word: This is the unit of data which is transferred for each operation.
Word Length: This is the maximum number of bytes allowed to transfer in one operation. For eg., In a 32-bit CPU, word length is 4 bytes (32 bits/8 bits)
Block: A group of words which is transferred between cache and main memory is called a block. This implies that whenever the requested data is missing from cache then a whole block of words is transferred from main memory to cache instead of just one word. The significance of transferring a whole block comes from the fact that most of the times data is stored sequentially by a program (stack memory) and thus transferring a whole block reduces the chances of next failed request. So all data transfer between cache and main memory happens in forms of blocks but data transfer between any memory and CPU happens in word.
Cache Hit: When the processor requests data from cache and finds it in the cache, then it is said to be a cache hit.
Cache Miss: Just the opposite of cache hit. When requested data is not in cache then it is a cache miss. Caches are designed to reduce cache misses and to increase cache hits. In case of a cache miss during data read, the required data is read from main memory after transfering a block of data to cache.
Byte Select Bits: These are the lower significance bits of an address which are used to select a specific byte of a word to select for operation. This is internal for a CPU. That is when a CPU asks for some data, whole word is transferred and which byte of that word is used for operation is out of our concern. For eg., In a 16-bit CPU, total 2 bytes (word length) are transferred for each request. This means only 1 bit (log(word_length)) is used for byte select. If the address is of 16 bits (64KB RAM) then we look at only more significant 15 bits (address bits - byte select bits) to look for data in RAM and transfer the 2 bytes regardless of what the last bit was. So if CPU asks for address XXX0010 or XXX0011 (XXX - first 12 bits) then we look for same location that is XXX001 (last bit ignored) and send both bytes in each case i.e. XXX0010 , XXX0011

Cache Line bits: Consider the cache index is index of some book and it contains several pages. So our index is written on several pages. Lines bits determine the page number on which to search for our physical address. This way we save our time by limiting our search to only one page instead of many. Remember that more entries on one page means that we will have to match each entry against given address. This means a linear search for the address is necessary after finding the page number of our memory address. Number of line bits depend on mapping techniques which we will cover in next article. Line bits are next bits after byte select bits which help us to limit our search for address in cache index
Tag Bits: After finding the page number, we will have to search our address. But we do not need to store the whole address as index entries because we already know some part of it which are the line bits and word bits. So we store the remaining bits of the address (excluding byte select, word and line bits) which are called the tag bits. Hence an address is broken into 4 parts : tag bits, line bits, word bits, byte select bits.
Word Bits: As the data transfer between main memory and cache memory happens in blocks, some bits are needed to transfer the required word from cache to processor. These are the word bits. Cache index tells the address of memory block in cache memory. The correct word which needs to be transferred is decided by word bits.
Valid Bit: This bit is used to indicate whether the data stored in the cache is valid or not (1- valid, 0-invalid). We need this bit in our index so we can keep track of memory locations which contain valid data. If we need to clear the cache, we can just set valid bit of each entry to 0
Dirty Bit: This bit is important in write back policy. When some data is loaded into cache and it is updated then it is marked as dirty. So that when the CPU needs to replace (which happens frequently) the data , it can save the updated data into RAM before replacing it.

Now lets see how the whole process of reading data from cache takes place

Cache index contains several pages depending on the mapping technique
Each entry of index contains tag bits, valid bit and dirty bit
Entry number represents the location of block in cache memory.
Processor sends a physical address to cache of which the data is required.
Physical address is broken into components in the following manner (Starting from lower bits)
- Byte Select Bits = log(word_length)
- Word Bits = log(number_of_words_per_block)
- Line Bits = log(number_of_pages)
- Tag Bits = remaining_bits_of_address
Correct Index Page is found using line bits
Each entry of the page is matched with tag bits
Correct word is found by adding word bits to matched entry.
Found word is transferred to processor if it is valid (valid=1)

This was pretty much about the terms we need to understand before moving on. Make sure you understand these concepts because in the next article, these terms will be repeating a lot.

Image Search Engine Using Python

noreply@blogger.com (Kapil Garg) — Thu, 12 Jan 2017 15:27:00 +0000

Images provide a lot more information than audio or text. Image processing is the prime field of research for robotics as well as search engines. In this article we will explore the concept of finding similarity between digital images using python. Then we will use our program to find top 10 search results inside a dataset of images for a given picture. It won't be as good as google's search engine because of the technique we will be using to find similarity between images. But what we are going to make will be pretty cool. So lets start.

Setting up the Environment
Our Algorithm
How the code looks
Lets build the GUI
Additional Techniques

Setting up the Environment

The code we are going to write requires a few tools which we need to install first. I will try to be as precise as i can and if you get stuck into installing some tool then you can drop a comment below and i will help you sort out the problem. So here are the tools and the steps to install those tools in ubuntu (16.04 but should work on any version). The steps are not independent so follow them accordingly.

Install virtualenv
- sudo pip install -g virtualenv
Create environment directory and activate virtual environment.
- mkdir env
- virtualenv env
- cd env
- source bin/activate
Install Numpy
- pip install numpy
Install Pillow (Python image library)
- pip install pillow
Install Tkinter (For GUI)
- sudo apt-get install python-tk

Our Algorithm

The approach we will be using includes finding euclidean distances between color histograms of images. This is a very basic approach and it will help us to search images using their colors and not using their features. So it is possible that with this approach the best search result for a zebra might be a yin-yang but that depends on the dataset actually. So this is not a bad approach at all. Lets see what the step by step process looks like :

Create the color histogram of each image in the dataset
Create the color histogram of the image to be searched
Calculate the euclidean distance between the histogram of image to be searched and histograms of the images in the dataset.
Select the smallest 10 distances and those are the search results.

This might sound a little confusing because we don't know about a couple of things in the algorithm. Like color histogram and euclidean distance. Let us understand these things.

What is color histogram:
Color histogram is spectrum of each possible color space of the image containing the number of pixels for a specific color component. Consider a simple RGB format image in which each pixel has one red component, one green and one blue component. So each pixel of the image can be represented by a tuple of 3 values (red, green, blue). Now the range of these values vary from 0 to 255 and thus forms a unique color by selecting any value for 3 components. Thus (0,0,0) represents white color and (255,255,255) represents black. The total possible values for each color component are 256 and thus we have a total of 768 possible color components (these are not total number of colors but only the number of the color components).

Now the color histogram for this image will be a table having 768 entries in which each entry contains the number of pixels containing that component
What is Euclidean Distance
It is nothing special but the ordinary distance between 2 vectors. If you know about vectors then you should have known how the distance between 2 vectors is calculated. But in case you dont know then lets assume that we have 2 vectors with N-dimensions. So each vector has N components. The distance between these 2 vectors can be calculated as below.

How the code looks

So far we have set up our environment and have learnt about the alogorithm we are going to use. Now its time to write actual code. Create 2 files with names hist.py and show_images.py. Below is the code for the hist.py file.

from PIL import Image
from numpy import *
import os
import show_images

DATASETDIR = '/path/to/your/dataset/directory/'

def perform_search(filename):
 #create an image object using Image.open method for the given image
 im = Image.open(filename)

 #we can use histogram method of image object to automatically build our histogram
 #we then convert the histogram array to numpy array to perform calculations
 search_histo = array(im.histogram())

 #create an empty list to store distances
 dist = []

 #get all the images of the dataset directory
 files = os.listdir(DATASETDIR)

 #declare the structure of the data for your dist list
 #It is only to perform sorting using numpy
 dtype = [('name', 'S100'), ('distance', float)]

 #Now we calculate euclidean distance between our search_histo and all images histograms
 for file in files:
  imob = Image.open(os.path.join(DATASETDIR, file))
  histo = array(imob.histogram())
  
  #Euclidean Distance Calculation
  try:
   diff = histo - search_histo
   sq = square(diff)
   total = sum(sq)
   result = sqrt(total)
   dist.append((file, result))
  except ValueError:
   pass

 
 #convert our list to numpy array with given data type
 distance = array(dist, dtype=dtype)

 #sort the array in increasing order to get top 10 results
 sort_dist = sort(distance, order='distance')

 top10 = sort_dist[:11]
 
 #show the result images in a window
 show_images.show_images(top10[1:])

Lets build the GUI

We have our backbone ready. Now we just need to show the images for which we require some GUI library. You can use pyqt or tkinter. But for this article we are going to use tkinter. If you have followed the first section then you already have tkinter installed into your system but if its not then i suggest you to install it first. Lets build our GUI program. The following code is for show_images.py

from Tkinter import *
from PIL import Image, ImageTk
import os

DATASETDIR = '/path/to/your/dataset/directory/'

class MainFrame(Frame):
    def __init__(self, parent, *args, **kw):
  Frame.__init__(self, parent, *args, **kw)            

  # create a canvas object and a vertical scrollbar for scrolling it
  vscrollbar = Scrollbar(self, orient=VERTICAL)
  vscrollbar.pack(fill=Y, side=RIGHT, expand=False)

  canvas = Canvas(self, bd=0, highlightthickness=0,
                  yscrollcommand=vscrollbar.set)
  canvas.pack(side=LEFT, fill=BOTH, expand=True)
  vscrollbar.config(command=canvas.yview)

  # reset the view
  canvas.yview_moveto(0)

  # create a frame inside the canvas which will be scrolled with it
  self.interior = interior = Frame(canvas)
  interior_id = canvas.create_window(0, 0, window=interior,
                                     anchor=NW)

  # track changes to the canvas and frame width and sync them,
  # also updating the scrollbar
  def _configure_interior(event):
      # update the scrollbars to match the size of the inner frame
      size = (interior.winfo_reqwidth(), interior.winfo_reqheight())
      canvas.config(scrollregion="0 0 %s %s" % size)
      if interior.winfo_reqwidth() != canvas.winfo_width():
          # update the canvas's width to fit the inner frame
          canvas.config(width=interior.winfo_reqwidth())
  interior.bind('<Configure>', _configure_interior)

  def _configure_canvas(event):
      if interior.winfo_reqwidth() != canvas.winfo_width():
          # update the inner frame's width to fill the canvas
          canvas.itemconfigure(interior_id, width=canvas.winfo_width())
  canvas.bind('<Configure>', _configure_canvas)


def show_images(top10, root=None): 
 root = Tk()
 root.title("Similar Images")
 root.imageframe = MainFrame(root)
 root.imageframe.pack(fill=BOTH, expand=True)

 images = []
 imagetks = []
 imagepanels = []
 r=0

 size = 128, 128

 for image in top10:
  imagename = os.path.splitext(image[0])[0]
  img = Image.open(os.path.join(DATASETDIR, image[0]))
  img.thumbnail(size)
  images.append(img)
  imgtk = ImageTk.PhotoImage(images[-1])
  imagetks.append(imgtk)
  panel = Label(root.imageframe.interior, image=imagetks[-1])
  imagepanels.append(panel)
  imagepanels[-1].grid(row=r, column=0)
  r = r+1

 root.mainloop()

I wish i could tell you why the code for GUI looks like so but then we will be going off topic. I will consider writing another article on making GUI applications with tkinter but till then you can use the above code as it is. Make sure you have specified the dataset directory in your both program files. So now that we have our main program and our gui program, its time to test the program.

Run the python interpreter using the command python
Now import our hist module using statement import hist
call hist.perform_search with the full path to the image to be searched
You should see a nice window showing the search results.

Additional Techniques

The technique we used above is pretty simple and might not provide results on the basis of other features of images like the shape, orientation and scaling. So you might tempt to use a better search technique. I know 2 other methods which involves using the following concepts.

Thats it for now. If you encounter any problem or have a question then don't hesitate to drop a comment below. Your feedback is also valuable so tell us your thoughts about the article in the comments.

Authentication System : Basic Design

noreply@blogger.com (Kapil Garg) — Sun, 04 Sep 2016 14:26:00 +0000

Being a system administrator, it is our main concern to protect users' data from theft and corruption. Regarding theft, we focus in building a sophisticated but secure authentication system which can fulfill all the basic needs we discussed in the last article. In the previous 2 articles, we learned about authentication and authorization. Also we understood the complications in building a secure system which can prevent tresspassing. But we have not code anything yet to build something. So in this article we will be designing an authentication system which will provide the functionalities we want and which will help us understand how the tech giants like google have taken security to another level. We will be writing code in php but the same applies to other languages also.

What we'll be using
Blueprint
Building the system
Running the system
Additional Care

What we'll be using

Lets see what are the languages we will be using and database plus some extra libraries

Languages	PHP, Javascript, HTML, CSS
Database	MySQL
Libraries & Frameworks (optional)	Fingerprintjs, jquery, bootstrap, validatejs

If you dont have php and mysql in your system then you should install them first before writing the code. You can download xampp.

Blueprint

Now lets see the database schema and the strategy we will use to build our system.

User (id [integer], email [varchar], password [varchar])
Session (session_id [varchar], token [varchar], user_id [integer] , expire_time [datetime])

Our system will authenticate the user based on the token stored in his machine and will prevent intrusuions.

To prevent sniffing, the application should be using SSL which encrypts the data sent over network between the client and the server. But we can still do a lot to protect our system even if some man in the middle steals our data.
Create the database tables according to the schema described above.
When user visits a page, check for the `session_id` and `login` cookie. Two cases are possible
Case #1 : If cookies are not present, ask for user credentials.
Search for those credentials in the `User` table.
If the credentials are correct then generate 2 random strings. One being the `session_id` and other being the `token`.
Create a new entry in the `Session` table with the values (session_id, token, user_id, expire_time). The `expire_time` can be any time you wish to allow a session being idle. For example: expire_time = current_time + 24 hours.
Send the session_id and token along with the response as `session_id` and `login` cookies respectively.
Case #2:If `session_id` cookie is present, search for the `session_id` in the database table.
If match is not found or the session has expired, ask for user credentials.
If session is found and active, match the `login` cookie value with the matched session `token`.
If match is not found, report a theft and delete all session data related to that user.
If match is found then authenticate the user and update the token value to some other random string.
Increase the expire time with 24 hours plus current system time.
Send the newly generated token along with the response and update the `login` cookie.
Make sure the cookies you save are set to be HTTP only so that XSS attacks can be prevented.

Now lets assume that someone steals the cookies while they were being sent to the server. Now he is able to impersonate the user But if he makes some request to the server with the stolen data then the access token will be changed and when the user tries to visit some page a theft is reported because the user has invalidated token and the user and theif both gets logged out of the system. So far so good.

But the problem arises when the theft occurs during the last request of the user. Now the user has left the application without logging out and some malicious person has the user's last authentication token. Now he is fully capabale of using the system for as long as he wants. We will see what can we do to prevent this and also how can we implement "remember me" functionality in our system. But before that lets build a system with the above strategy in mind.

Building the System

The system we will be designing is a pretty basic one but i will tell you what more you can add to take it to next level. Below are the components of the system with outlined code. The whole source code can be viewed on github fetch-info-auth repository

HTML login form : (A simple form with e-mail and password field only)

<form action="http://fetchinfo.com/fetch-info-auth/public/html/login.php" method="post">

 <input type="text" name="email" maxlength="30" placeholder="username">

 <input type="password" name="password" maxlength="30" placeholder="password">

 <input type="submit" value="Submit">

</form>

Main Authentication Script

This script first calls another script to check whether the user has an authenticated cookie or not. If cookie is not verified then it checks whether the user has supplied his credentials or not. If credentials are provided then it verifies them and sends appropriate data.

    function auth(){

  $result = authCookie() ;

  if(is_integer($result)){

   if(isset($_POST['email'])){

    $email = $_POST['email'] ;

    $pass = $_POST['password'] ;

    $conn = connect() ;

    $query = "Select * from user where email='".$email."' and password='".$pass."'" ;

    $result = mysqli_query($conn, $query) ;

    if(mysqli_num_rows($result)==1){

     

     $uid = mysqli_fetch_row($result)[0] ;

     $new_token = generateRandomString(100) ;

     $new_time = date('Y-m-d H:i:s', strtotime(date('Y-m-d H:i:s'))+86400) ;

     $new_session = generateRandomString(100) ;



     

     $query = "insert into session values('".$new_session."','".$new_token."',".$uid.",'".$new_time."')" ;

     

      if( mysqli_query($conn, $query)){

      return [$new_session, $new_token] ;

     }

     return globals("UNAUTHENTICATED") ;

    }

    return globals("UNAUTHENTICATED") ;

   }

   return globals("UNAUTHENTICATED") ;

  }

  else{

   return $result ;

  }

 }

Cookie Authentication Script

This script checks whether a user has authenticated cookie or not. If the verification succeeds then it generates a new token and returns the new token which is returned by the main authentication script and the html pages override the existing cookie with the new generated token.

 function authCookie(){

  $conn = connect() ;

  if(isset($_COOKIE['session_id'])){

   $session_id = $_COOKIE['session_id'] ;

   $query = "select * from session where session_id='".$session_id."'" ;

   $result = mysqli_query($conn, $query) ;

   if(mysqli_num_rows($result) == 1){

    $token = mysqli_fetch_row($result)[1] ;

    if(isset($_COOKIE['login'])){

     $login = $_COOKIE['login'] ;

     echo "Session: ".$session_id."<br/> Token: ".$token."<br/> Cookie: ".$login ;

     if($token == $login){

      $new_token = generateRandomString(100) ;

      $new_time = date('Y-m-d H:i:s', strtotime(date('Y-m-d H:i:s'))+86400000) ;

      $query = "update session set token='".$new_token."', expire_time='".$new_time."' where session_id='".$session_id."'" ;

      $result = mysqli_query($conn, $query) ;

      return [$session_id, $new_token] ;

     }

     else{

       return globals("THEFT") ;

     }

    }

    else{

     return globals("UNAUTHENTICATED") ;

    }

   }

   return globals("UNAUTHENTICATED") ;

  }

  return globals("UNAUTHENTICATED") ;

 }

A successful verification of either the cookie or the credentials lands the user to the home page which only contains a hello message in this case. For convinience, the session_id cookie and the token cookie values are displayed on each page so that you can see what is happening. Each refresh to the home page updates the token as well. There are many other scripts as well. So if you want to see the whole source code, you can visit the github fetch-info-auth repository

Running the System

Now lets see how can we quick run our system so that we can understand things more clearly. Here i am making some assumptions which are :

You have apache installed
You have mysql installed
You have php installed
You have created the database according to the schema discussed above.

All things settled down, now clone the repository from github and configure the apache localhost to view pages from this cloned repository. Also changed the connection settings in repository/script/auth/connect.php to point to your mysql database. If all goes accordingly then you will be able to see the login.php page. To do the login, you will have to create an account i.e. to add a user in the user table according to the schema we discussed above. Just create the account and do the required login and you will be redirected to the home page where you will have the option to logout.

Please note that this system currently does not erase the session data on detecting theft. There are also other loop holes which you can cover in the methods by changing the response based on the flags sent by the authentication functions. Also note that the code assumes that the database schema is exactly what is defined above. I will try to generalise the code in the future.

Additional Care

We have just created a basic system which only provides the required functionalities but does not fully protect the system. There are a couple of things we can do increase the security.

Use encryption over your application by using SSL protocol to prevent session hijacking.
Limit the number of login attempts to prevent dictionary attacks.
In case of persistent login ("remember me") option, different approach is required.
Use scripts to prevent SQL injection attacks.
Make your cookies to be read by http only to prevent cross-site scripting attacks.

I hope you learned something new here. In case you found some incorrect information or you think it can be improved then do share your thoughts in the comments. If you have any query then you can drop it in comments and i will try to resolve it.

Authentication: A step to security

noreply@blogger.com (Kapil Garg) — Fri, 12 Aug 2016 12:13:00 +0000

Here we are, continuing our discussion on authorization and authentication. If you have red my previous article then you would have known by now how important these concepts are and why these need to be implemented efficiently. But in the previous article, we only talked about them and didn't look at how to actually implement them. Cybersecurity is so vast field that we can't understand all of its complexities in one article but what we can do is we can start with some pretty basic stuff and move along to provide our application some sort of security. So in this article we'll see how can we implement authentication checks in server-client systems and specifically in web applications. But for other type of applications like desktop apps, mobile apps you can follow the same procedure although the technology would be different. Lets have a look what we are gonna learn today:-

Our Goal
General Procedure
Answers to some General Questions
Cookie Theft

Our Goal

Just keep aside what we are going to do for a second and think about what we actually want. We want some mechanism which can ensure that the user requesting a particular web page is actually some authenticated user. And by authenticated user we mean that our system recognizes the user. Currently we are not concerned about the actual user, we are only concerned whether the credentials supplied to our system by the user are authenticated or not. So, ideally when a malicious person has stolen one of our user's identity we should not let the thief into our system but as we talked earlier, start with some simple stuff and then move along. So just let that theif in because the credentials he is providing are authenticated.

Also we do not want to bother the user by asking for his credentials again and again for each request so we want to ensure some way of automatic login. But http being a stateless protocol, there is only one way of automatic login and that is to store some information about the connection/session on client's machine. The data stored on client machine can be stolen easily but we are not concerned with that right now. So to summarize our goal, we want to have the following features in our mechanism.

Authentication check for the provided credentials.
Provide automatic login facility by using any method.
Prevent Credentials theft. (optional)
Detect credentials theft (optional)

General Procedure

Do you think the authentication systems which the big players like google, facebook, amazon etc. implement will actually be that simple which we talked about earler. Hell No. But as we are in a learning phase we can start to build something atleast. And as every thing people build at first there needs to be step by step procedure to do that. The process i am going to tell you is not a standard so it is totally upto you to follow it. Here is a basic flow diagram of the process and following it is the description of the diagram.

Check whether information(could be stored in a cookie, file or anything else) exists on client's machine or not.
If not then ask for user credentials
Authenticate the credentials.
If credentials are authenticated then let the user in otherwise repeat step 2.
If information is present then validate the information on the server.
If information is not valid then delete the information and repeat step 2.
If information is valid then authenticate the user based on that information.
If authentication is done properly then let the user in.
If error occurs while authenticating then delete the information and repeat step 2.

As you may notice, if any problem occurs we ask for user credentials no matter what. There is a lot going in the above procedure. There are some things which are not clear and you might be wondering about those things. Some of the questions coming up your mind will be :

What information to store ?
How to store the information ?
How to validate the information ?
How to authenticate the user based on that information ?

Before tackling these questions, lets limit ourselves to web applications only. So the platform on which our application can perform will be a browser. Now lets answer the questions one by one.

Answers to our Questions

As we already know the procedure to implement it, we first need to know the answers to our question and then we will move forward to see what are the potential vulnerabilities in the simple procedure and how to remove them to a greater extent.

What information to store ?

If we don't care about the security then we can simply store the user_id and password so that we can directly look these things up into our database and see if we found a match but the problem with this approach is that we are compromising our user's data security. If that information gets stolen then not only the malicious user will be able to see the data but will also be able to delete it because he has the password of the real user.

So we need to store something else. One thing we can do is whenever a user logs into the system by providing his credentials then we generate a one time token which will be stored on the clients machine and also a table entry would be made in a seprate table which holds the token, associated user_id. We can match the token in the table and if it exists we see the user to which the token belongs to and assumes that this is the user who is making the request. Some people may already have sniffed the potential vulnerability here but there is a lot we can do about it. We'll talk about all the techniques in the next part. Lets just stick to the basics here.

How to store the information ?

Speaking regarding to the web application we can store the information either in the cookie or local storage. Local storage is not cross-browser compatible and your application may work somewhere and may not somewhere. So cookies will be our primary candidate to store the information.

How to validate the information ?

Considering what is the information you are storing and what technique you are using, you can validate it differently. For example, if you are using the token technique then you can add another field in the table which holds the expire time of the token and validate whether the token has been expired or not. Or if you are using signed cookies then first you can validate whether the signature matches or not and then move forward to validate the data stored in the cookie.

How to authenticate the user based on that information ?

This also depends on the technique you are using for authentication. If you are storing the user_id and password then you can authenticate the user by finding an entry for those things in the table or if you are storing token then you can find the token in the table and check its associated user_id. Basically, you are authenticating user associated with the information you stored on client's machine.

Cookie Theft

The main concern related to store information in a cookie is about the security of the cookie. Because there is currently no simple way to distinguish 2 browsers, a server will treat the same cookie present on 2 different browsers as if they are present on one machine. So if our cookie gets stolen then we can't prevent the thief to enter into our system. But there are some javascript libraries which can generate a unique string for a unique browser and the uniqueness of the string increases as we consider more parameters while generating the string. We'll see the authentication technique in the next article and how to prevent cookie theft and what to do in case our cookie data is compromised. But first see the method using which a cookie can be stolen.

Cross Site Scripting

Using this method, an attacker can inject script into the page by some means or using redirection techniques by which he can capture the cookies related to that page. If it looks something unimaginable then there is an article which explains how you can steal someone's cookies using cross site scripting.

Packet Capture

I don't know whether you have heard of the term "packet capture" earlier or not so let me explain it. Packet capture means capturing the data traffic going through some network. Using a tool like wireshark, we can capture each and every byte of data which is sent over the connected network. This way you can actually see what page users are visiting and what headers, cookies etc are exchanged during the connection. Once you get the cookies, you can set them on your machine and voila, you have stolen the user's identity. There is another nice article on how to do that Session Hijacking

There is a lot more to what just we learnt today. Some concepts like signed cookies, encrytion, request and response headers becomes useful when we want to make our system more secure. Also there are some authentication techniques which can be used to minimize data theft but that is a whole lot to write in one article so you'll have to read the next article where we will discuss authentication techniques. You can ask anything by dropping a comment below and i'll try to solve your problem. Also, suggestions are always welcome. If you found any information misleading or wrong then please inform by commenting.

Authentication and Authorization

noreply@blogger.com (Kapil Garg) — Mon, 08 Aug 2016 08:30:00 +0000

Authentication and Authorization are 2 independent concepts both of which are constituents of security. Authentication , in a vague manner, means to recognize someone and Authorization depicts to give that person some permissions. To understand it more better, a really simple example would be to know these concepts regarding to operating systems. When we log into our computer the OS checks whether our credentials match to any user of the computer. That is what we call authentication. Once the OS finds a match, it logs us into the computer. Now if we want to open a file then the operating system checks whether we have permission to do so or not. This is the time when the OS is authorizing us. If we have permission then OS will open the file for us otherwise it won't. So in this post we'll see how can we implement these features in any system and take safety measures to prevent intrusions. This post can be a bit boring but its actually a building block of what we'll learn in the future posts and it gives an idea of how things work.

Authentication
Authorization
Server-Client Systems
Best Practices

Authentication

As we talked earlier, authentication is recognizing the user. If we think about it a bit then we'll notice that authentication should be done before authorization. Its clear that you dont want to check whether a person is allowed to do something before checking if he is even allowed to enter the premises. Authentication can be done in a pretty straight forward manner. User enters his credentials, we check whether we have some registered user with those credentials and allow him in if we found a match. All we need to implement this type of authentication is a user-credentials table where we can search for the user.

Authorization

Authorization deals with the user's permissions. There are some ways how it works.

User can only see the data what he is allowed to take action on.
User can see all the data. He tries to take action on arbitrary data and then we check whether he is authorized to do that or not.
Mixture of above methods. User only see what he is allowed and if he takes action on it then we check again whether he is allowed or not.

Why do we actually need the third way ? The first two ways looks fine but why a double check. The reason for a double check is to prevent intrusions. So if someone actually seeing the allowed data but he uses some hacks to get information of the hidden data then we must ensure that he should not get it. So when a user is allowed to take actions without our provided interface then there must be a double check. In case where the system functions are totally hidden from a user where he is restricted to use only the provided interface then the first method would do fine.

But the second method is actually not supposed to be used in a system where information about the existence of the data is also sensitive. Also it might result in a lot of unauthorized actions and would confuse the user what he is actually allowed to do. So the first or third method to implement authorization would work fine. To implement authorization all we need is access-control table which stores the information about the role and its allowed permissions. The problem increases when we want fine control over a user permission for each data. We'll talk a lot about authorization in the next article and would see how to actually implement it for a server-client based system using an SQL based database and a scripting language like php or python.

Server-Client Systems

Server-Client systems are the systems in which one machine asks for data and other machine serves the data. There can be an API using which a client machine can ask for data and the server machine needs to implement authentication and authorization of the client requests so that it can prevent illegal access to the data. In a server-client system, if the connections are stateless i.e. the server disconnects the client after serving its request then with each request the server will have to authenticate the user first. Also there is a possiblity of exploiting the server without using the given API and thus intruding in the server system. To provide a user-friendly environment, client machines should not ask for user credentials for each and every request, instead they should do it automatically. But doing this on the client machine would require to store the credentials on the machine which ,in the future, can be stolen by any person. So never store the credentials on the client machine. There are 2 protocols which are used in a client-server system.

Stateless: In this protocol, the server is not required to retain session information or status about each client for the duration of multiple requests.
Stateful: A protocol which requires keeping of the internal state on the server. So if the connection is made then it is kept alive for multiple requests and the server does not need to do authentication again and again.

Both these protocols have their ups and downs but we are concerned with only authentication here. Most of the application use stateless protocol because it is not exhausting i.e. it does not use resources for only a limited clients. Once a request is served, it is open for other requests. But we are stuck to the problem of repeated authentication in a user-friendly manner. The solution to this problem is that we can generate an access token for each logged in user and then use that token to do the authentication on the server. Normally using only a token cannot prevent what we call "session hijacking" so we'll need to do a lot of more work. In the future posts, we'll see how to actually implement these features in a system.

Best Practices

These practices are actually for developers who have aleary implemented authentication and authoriation in their system. So lets see what we can do to make our system faster and secure.

Indexing: I hope you all know what indexing means regarding to a database. So always try to use indexing on the field which will be mostly searched. Like if you have user_id, user_name and password fields in your records and you will always want to search for a user using its user_name then apply indexing on that field. It will reduce the database response time.
Never store sensitive information on client machine.
Encrypted connections: Use encryption to secure the data you sent over a connection so that even if that data is compromised you'll be safe.
Understand wrong attempts: If user is providing wrong credentials over and over again then there are chances that a script might be running to tresspass the system. So lock the account in that case, inform the user whose credentials are tried to be used and then unlock the account only when the original user asks.
Use Captcha: By implementing captcha you are preventing the scripts from attempting false logins.
Store for lesser duration: If you dont have a "remember me" feature on your login system then store the session data for a shorter duration so that if a client is disconnected for a longer time then he'll have to provide his credentials again. A simple strategy would be to increase the session expire time by some amount, lets say 30 minutes, on every request. So if there is gap more than 30 minutes between two successive requests for the same session then decline that request and ask for the credentials again. This way even if the access token is compromised in the future, data will be safe.

This much for now. In the next article we'll write some code to implement authentication in a client-server system and then we'll see how to implement authorization for different user roles in the system. If you have any suggestion then just drop a comment. You are free to ask anything if you want. Also don't forget to subscribe to get notified about new articles.

MVC: At Its Core

noreply@blogger.com (Kapil Garg) — Sat, 18 Jun 2016 11:56:00 +0000

Designing an application requires a lot of effort And without a proper structure there are chances that your application may fail during some operations. That is the reason of choosing a framework to accomplish the task of building an application. A framework enforces you to use the defined application structure and methods so that you can easily maintain your code and refactor it if needed. MVC , short form for Model View Controller is a designing pattern in which an application is divided into 3 components i.e. Models, Views and Controllers. (The word Application infers to all type of applications like mobile apps, desktop apps, web apps etc.). A framework which follows the MVC pattern is known as a MVC framework and there are a lot of frameworks for each type of application written in most of the languages. Normally people start learning about the framework without understanding the concept behind them. Each MVC framework follows the same pattern i.e. the MVC pattern. So if we learn about MVC then we can pick any framework of our choice and it'll take lesser time to understand their API. So in this article, we'll focus on MVC pattern rather than the framework. These are things we'll be learning today.

MVC Components

As i mentioned earlier, using MVC pattern means that you'll have to break your application in 3 components. You'll be wondering why that is even necessary ? The reasons are many but to list a few, it helps you write maintainable code by structuring your code in different modules, easily testable, easily scalable, easy to understand by others and list is going.. Still if you don't understand the importance of MVC then lets check out the components and then you'll be able to visualise the modularity. The components of a MVC based application are modals, views and controllers.

Modals :
This is the component which is responsible to modify the data stored by your application. The data can be stored anywhere like in a database, in a file, in dynamic arrays etc. This component provides a handy API which can be used by other parts of your application to send data to database, retrieve data from database etc. The data definition (eg. database schema) and the data store (eg. MYSQL connections) connections are also managed by the Modal component. You just tell modal what to store and it takes care of the rest of the details like where, when, how to store.
Views :
This is the component which displays the data. It does nothing else but only displays the data. You must be thinking that why we need a component for displaying the data if we can simply output it ? Actually a view is a reusable fragment of code so you can use a same view to diaplay 2 different types of data. For example, you might need to show price of an item in rupees if user is indian and in dollars if he is american (Its just an example. I obviously know that rupees and dollars are currencies of many more countries.) So instead of returning the value and the sign wrapped in a label widget(GUI apps) or in HTML tags(web apps), you can simply create a view which shows a value and a sign and you can pass your data to it.
Controllers :
Controllers are the machinery which actually runs your application. It forms a bridge between the views and the modals. Controllers are the ones which asks the modals for some data, performs some operations on it and then send them to views. In a non-mvc application, what most people do is either they put all of their code in controllers or use any 1 more component along with controller like MC or VC. Thats when applications start to become unstable.

MVC Flow

The flow of an MVC application is really important for us to understand. We already know about the components so now lets see how they work so that we can visualise what exactly is going on. Below is a simple block diagram describing the placement of the components.

The main thing of your application is the routes using which a user can access your application views. From the perspective of a web app, a route is the url which user clicks and a controller will be called to perform an action. Consider a route to be a trigger to call a controller for handling. In web apps, triggers are urls and in mobile/desktop apps, triggers are buttons or actions (swipe). So routes are defined to trigger the controllers. Lets see the ongoing process when a user logs into an app.

Lets suppose user just opens your app (may be url :'/home' or just click app icon).
A controller is called to display the login form. That controller does nothing but returns a view containg a login form.
The returned view displays the login form to the user.
User fills out the form and clicks the signin button (trigger)
A different controller is called which recieves the data. It queries the modal to check for the existence of the user with the given credentials.
Modal checks for the user's existence and returns result to the controller.
Controller checks the result. If user exists then it return dashboard view otherwise it returns the login view again

Popular MVC Frameworks

The work flow of a login process which we saw earlier is a very basic one. There are a lot of things which frameworks handle for us like session management, request/responses, cache, cookies, database connections, security, form handling etc. So instead of building an application from the scratch without a framework, we should use frameworks for performance and reliability. There are hundreds of frameworks avaiable in the market for different languages and for different purposes. I'll list a few here but you are welcome to suggest any popular framework which is missing.

Framework	Language	Type
Express	Javascript	Web
Django	Python	Web
Laravel	PHP	Web
Angular	Javascript	Web
ASP.NET	C#/VB	Mobile/Web
Ruby on Rails	Ruby	Web
Spring	Java	Any
Ionic	Web languages	Hybrid mobile
Catalyst	Perl	Web
Phonegap	Web languages	Hybrid mobile

These are a lot more frameworks other than the above listed. But these are the ones which are widely used. If you want to share some more frameworks then you can write them in comments. Also it would be nice if you take a moment in sharing the article if you learnt something. Happy coding.

Debugging with gdb

noreply@blogger.com (Kapil Garg) — Sat, 07 May 2016 13:30:00 +0000

Debugging a program is a real tedious job which can consume time more than you took to write the program. And if you are not on the right track of debugging then you are probably gonna end up writing the whole program again. If you have red my previous article on debugging (The Art of Debugging) then you already have an idea of the debugging process and what are some basic methods to do it. We also talked about how are those basic techniques not gonna help us with larger and trickier programs. To make our job easy with those programs, there is a debugging tool : GNU Debugger or simply gdb. GDB is a powerful debugging tool with all the features you require while debugging your program. There are some basic set of features which are available in every debugging tool so if you want to use some other debugger then you'll find it easy to use after you get to know about gdb. There is some great stuff we'll be studying today so sit tight. Let's have a look at the index we'll be covering.

Indroduction to GDB
Debugging Terminology
My Favourite gdb commands
Debugging a program

Introduction to GDB

We have already seen its full form so now lets see what actually gdb is. GDB is like any other debugging tool which helps you to find out what is going inside your program. It helps you to check for variable states, memory states, stack trace and much more. GDB was actually made for GNU operating system but then it was made portable and now it can run on any UNIX-based operating system(Linux, Ubuntu, Redhat, Lubuntu) and windows operating system. It supports C and C++ programs. Although it also supports D, Fortran and 1-2 more languages but the support is partial. So its best if you use it to debug C and C++ programs. GDB requires some special symbols in your program to let it know about the program but don't worry, you won't have to add those symbols yourself. The GNU GCC compiler has a flag -g3 or -g which inserts the symbols automatically. We'll see all of this in the later section. GDB can do mainly four things to help you catch the bug.

Start your program, specifying anything that might affect its behavior.
Make your program stop on specified conditions.
Examine what has happened, when your program has stopped.
Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.

We'll see all these things in action while we will debug our own program. Till then lets just get ourselves acquainted with gdb.

Debugging Terminology

It will be very helpful if we first learn about some concepts before working with a debugger. We can't get to know about all the features a debugger provide because there are too many but we can enrich ourselves with some common and basic features.

Breakpoints: A break point is the point at which the debugger will stop your program. For eg., when you run your program in gdb, you can set a breakpoint at a function so that gdb stops at that function and you can execute your program line by line from there.
Watchpoints: A watch point is a break point which is not specific to a line in your program, instead it stops the program when some change happens to the supplied expression or variable. For instance, you can set a watch point to stop the program execution whenever your variable crosses a specific value.
Stepping: Stepping means to execute your program step by step. GDB has 2 types of step. We'll see both of them in the next section

There are more but those are self explaining so we don't need to elaborate them.

My Favourite gdb commands

GDB runs on commands. You tell it what to do, when to do by commands. There are a lot of commands of gdb but we won't use much of them frequently. But there are some commands which are used again and again and those commands provide those basic features which almost every debugging tool provides. So if we get handy with these frequently used commands then we'll able to work our way with most of the other debuggers. At this point, it is not necessary that you understand all these but soon we'll be using them.

help: Get a description of gdb’s commands.
run: Runs your program. You can give it arguments that get passed in to your program just as if you had typed them to the shell. Also used to restart your program from the beginning if it is already running.
quit: Leave gdb, killing your program if necessary.
break: Set a breakpoint. Some examples: - break somefunction stops before executing the first line somefunction. - break 117 stops before executing line number 117.
watch: Set a watchpoint. Some example: - watch myvar stops when the myvar value changes. watch myvar>100 stops when myvar value crosses 100 if it is initially less than 100.
list: Show part of your source file with line numbers (handy for figuring out where to put breakpoints). Examples: - list somefunc lists all lines of somefunc. - list 117-123 lists lines 117 through 123.
next: Execute the next line of the program, including completing any procedure calls in that line.
step: Execute the next step of the program, which is either the next line if it contains no procedure calls, or the entry into the called procedure.
finish: Continue until you get out of the current procedure (or hit a breakpoint). Useful for getting out of something you stepped into that you didn’t want to step into.
cont: (Or continue). Continue until (a) the end of the program, (b) a fatal error like a Segmentation Fault or Bus Error, or (c) a breakpoint. If you give it a numeric argument (e.g., cont 1000) it will skip over that many breakpoints before stopping.
print: Print the value of some expression, e.g. print i.
display: Like print, but runs automatically every time the program stops. Useful for watching values that change often.

Debugging a program

We have already learnt a lot about gdb. Now lets see it in action. In this section we'll debug a simple program which has a function to swap 2 numbers in the array. There are some bugs in the program so lets find out those using gdb. (Note that we could use basic techniques of debugging also in this case.)

#include<stdio.h>
void swap(int a, int b){
    int temp = a ;
    a = b ;
    b = temp ;
}
int main(){
    int i, arr[4]={1, 2, 3, 4} ;
    printf("Before Rotating left: ") ;
    for(i=0 ; i<4 ; i++){
        printf("%d ", arr[i]) ;
    }
    printf("\n") ;
    for(i=0 ; i<4 ; i++){
        swap(arr[i], arr[i+1]) ;
    }
    printf("After Rotating left: ") ;
    for(i=0 ; i<4 ; i++){
        printf("%d ", arr[i]) ;
    }
    printf("\n") ;
    return 0 ;
}

Save the above program in a file gdb.c. Now lets compile the above program but this time with a special flag to insert debugging symbols for gdb. Execute this command in the directory where gdb.js is saved gcc -g gdb.c It will generate an executable with name a.out in the same directory. Try to run the executable by ./a.out. What did just happen ? Our program is doing nothing. The list is not rotating. But why ? Lets ask gdb to help us.

Run the gdb by gdb a.out. It will start the gdb prompt where we'll start debugging our program. Now type run and execute. It will run the whole program without breaking at some point. It is just like normally executing our program. Lets add some breakpoints by typing break main break swap. Now run the program. Firstly it will stop at mainThere is nothing much to see at main starting point. So continue executing by typing cont. Now it will stop at swap function which is called at first iteration of the loop. You can see the passed in parameter values. a=1, b=2. Now lets see if the numbers are being swapped or not. Execute next 3 times and check the values of a and b by typing print a print b. Well the values have been swapped. Good. So our function is doing well. Now lets see whether the array values are swapping or not.

Execute next and you will be back to your loop. Lets see our array values by print arr[0] print arr[1].Oops. The values haven't been changed. But whats wrong ? Our function is doing well so what's the problem. Okay i got it. And i hope you too have got the problem. We are not passing our array values by reference. We are just passing them by value. So lets change that.

void swap(int *a, int *b){
    int temp = *a ;
    *a = *b ;
    *b = temp ;
}
//
//previous code
//
        swap(&arr[i], &arr[i+1]) ;
//
//rest code

Again execute the code and see the results. Our expected list is 2 3 4 1 but we are getting 2 3 4 0. Where that 1 is going. How it got converted to zero. Let me not reveal the suspense. I hand it over to you guys to find out what the second bug is. Hint : Iterate the whole loop and see whats going on inside the swap loop. Just use the basic commands and find out the bug. Many of you have already gotten it by just looking but still i will recommend you to find it using gdb so that you can get familiar with it.

Did you find this article a bit too much about the gdb ? Well its just the top of it. There are so much of it which we have not even touched. So if you want to go further with gdb then you can visit the GDB documentaion page. Also if you want a simple ebook with above gdb commands and their example then you can subscribe to our newsletter so that i can mail you that ebook. One more thing, sharing is caring. So share this if you think its worth reading or share your thoughts about it by dropping a comment below.

The Art of Debugging

noreply@blogger.com (Kapil Garg) — Sat, 30 Apr 2016 16:43:00 +0000

Writing unorganised code for something to work is an easy task, just put anything anywhere and see if the result is coming or not. If the result comes out in first attempt then you are really lucky but normal people like me are not. Then what we should do to check where the problem is occuring, we put some print statements (or console.log) to check the result in various places and observe where the output is deviating from its path. Well, it itself is a good practise of debugging the code but it does not work always especially not with large projects. So what should we do then ? According to me, we should give debugger a chance to find the bugs in our program. Debugging is an important part of writing programs because it teaches you to write unit tests for the parts of the program where it can fail.

To perform debugging, we need a debugger and thankfully we have one, gdb (GNU debugger). In this article we will start by seeing some basic debugging techniques which involves logging the output and assertions. Then in the next article, we'll use gdb which will do the heavy lifting for us and will tell us what is going in our program internally. So the topics we'll be covering in this article will be :-

Method of debugging and What to avoid
Basic debugging
- output logging
- assertions

Method of debugging and what to avoid

The basic approach to debugging our program follows the sequence of following steps :

Know what your program is supposed to do.
Detect when it doesn’t.
Fix it.

This seems pretty simple, right ? In reality it is not that much simple. Detection the bugging point is a tedious job itself. Also there are some tempting mistakes which we do when we see wrong output. First mistake is, we don't follow the first step and just randomly tweak the code until it works. Don't do this because it will eventually mess up your code and even if it works(which i doubt), you will end up with a hard-to-maintain code. The second mistake is to attempt to intuit where things are going wrong by staring at the code or the program’s output. Like if see a segmentation fault and we know that we have used pointers in our program then we keep staring at the pointers code and try to see where the reference is going wrong. Stop doing this and let the computer find itself the mistake using debugging tools.

Basic debugging

Usually there are programs which consist only 50-100 lines of code and 2-3 functions then there basic debugging techniques can be applied to find the bug easily. If basic debugging doesn't help you then move straight to use gdb instead of applying these techniques too much. The 2 basic methods of basic debugging are:

Logging output to console
Assertions

Lets see them one by one

Output Logging

Output logging basically means printing the output our program generates bewteen various parts. In C, we use printf function to print out the variables and we can see their state in between our program lifecycle. But priting the variables using printf has some disadvantages. Some of these disadvantages are:

We can't change the output without editing the code.
A lot of print statements can create mess and we'll get confused which part is generating which output.
The output can be misleading: in particular, printf output is usually buffered, which means that if your program dies suddenly there may be output still in the buffer that is never flushed to stdout. This can be very confusing, and can lead you to believe that your program fails earlier than it actually does

There are some rules of thumb to follow when using output logging technique.

Use fprintf: Instead of printf statement, use fprintf(stderr, ...). It will keep your regular output seperate from debugging output.
Flush the output: If you still want to use printf then call fflush(stdout) to prevent buffering problem.
Use #ifdef: Wrap your code inside #ifdef It will allow you to turn your debugging code on/off according to your needs.

Lets see an example of it in a quicksort program.

#include<stdio.h>
#define DEBUG
void qsort(int *arr, int l, int r)
{
 int i ;
    if(r>l)
    {
        int index = partition(arr, l, r ) ;
        #ifdef DEBUG
        for(i=0 ; i<5 ; i++){
            printf("%d ", arr[i]) ;
        }
        printf("\n") ;
        fflush(stdout) ;
        #endif
        qsort(arr, l, index) ;
        qsort(arr, index, r) ;
    }
}
int partition(int *arr, int l, int r)
    {
    int i=l, j=l ;
    int pivot = arr[r-1] ;
    while(j<r)
        {
        if(arr[j]<pivot)
            {
            int temp = arr[i] ;
            arr[i++] = arr[j] ;
            arr[j] = temp ;
        }
        j++ ;
    }
    int temp = arr[r-1] ;
    arr[r-1] = arr[i] ;
    arr[i] = temp ;
    return i ;
}
int main(){
    int i, arr[5] ;
    for(i=0 ; i<5 ; i++){
        scanf("%d", &arr[i]) ;
    }
    qsort(arr, 0, 5) ;
    for(i=0 ; i<5 ; i++){
        printf("%d ", arr[i]) ;
    }
    return 0 ;
}

This is a simple quicksort function but there is some error in it. To debug it, i have added a loop which prints my array after each partition in the qsort function. Also i have wrapped that piece of code inside a #ifdef block. I have switched on the debugging by defining DEBUG using #define DEBUG. You can comment that line if you want to turn off debugging(which means the code inside #ifdef block won't execute). Now if you try to run it then you will infinite array prints. Also those arrays are all sorted. So it suggests that our code is working fine but it is not stopping after sorting the array. What might be the reason for endless recursion. One can think may be our recursion condition is not right. And that is the bug in our program. If you change the recursion condition if(r>l> to if(r>l+1), you program will start behaving correctly. Now you can turn the debugging off by commenting #define debug.

Assertions

We just saw the ouput logging technique and how to use it smartly and carefully. Now lets see another technique of basic debugging. In this technique we'll use C assert library. Every non-trivial C program should include <assert.h>, which gives you the assert macro. The assert macro tests if a condition is true and halts your program with an error message if it isn’t. A basic example could be:

#include<stdio.h>
#include<assert.h>
int main(){
 assert(2+2 == 5) ;
 return 0 ;
}

When you will run the above program, the assertion will fail and you will see an error message. Assertions are useful because you won't have to change your code to get more debugging output. Assertions can be used to verify assumptions made by the program and print a diagnostic message if this assumption is false.

So far we have seen how can we debug our sweet simple programs but when our programs grow in size, they become more nasty. And to handle those, we'll be needing a more powerful tool to help us. In the next article, we'll see that mighty tool (gdb) and will see how to use it, when to use it and other tools to use along with gdb. If you have any query then drop a comment below and i'll be willing to help you out.

Understanding Python Decorators

noreply@blogger.com (Kapil Garg) — Thu, 28 Apr 2016 17:32:00 +0000

If you have ever wondered what those @somethingmean above a python function or method then you are going to have your answers now. This @something line of code is actually called a decorator. I have red from various articles about them but some of them were not able to clarify the concept of a decorator and what we can achieve with them. So in this post we'll learn a lot about python decorators. Here is a list of topics we'll be covering.

What is python decorator
Understanding the concept
Multiple decorators on same function
class method decorator
Where can we use decorators

What is python decorator

A python decorator is nothing but a function which accepts your given function as a parameter and returns a replacement function. So its like something this

def decorator(your_func):
    def replacement(your_func_args):
        #do some other work
    return replacement

@decorator
your_func(your_func_args):
    #your_func code

Now when your_func gets called then the function which actually gets executed will be replacement. Note that the parameter pattern of the replacement function must be same to that of your function. To understand what is happening, lets read the next section.

Understanding the core concept

It is very common in python to assign function objects, to pass function objects as arguments and to call functions using function refernces. What all these mean is you can do all of the following in python.

def any_func():
    #do some work

your_func = any_func

your_func()

Python decorators are based on the above concept and using a python decorator is equivalent to writing the following code.

def decorator(func):
    def replacement(args):
        #do some other work

    return replacement

def your_func(args):
    #do some work

your_func = decorator(your_func) #assign the replacement function to your given function. So now your_func points to replacement
your_func(args) #call the replacement function

As you can see, a decorator does nothing special, it just overrides your given function definition. And whenever your_func will be called then replacement will be executed. Lets see a simple example now which is very helpful while understanding decorators.

def encoder(func):
    def decorator(html_string): #function which replace < and > with html codes and calls the passed in function for formatting
        html_string = html_string.replace(r'<', r'&lt;')
        html_string = html_string.replace(r'>', r'&gt;')
        result = func(html_string)
        return result
    return decorator

@encoder
def para_wrapper(html_string): #our function which simply wraps the html string with a paragraph tag
    return '<p>{0}</p>'.format(html_string)

In this example what is happening is we use a decorator @encoder which replaces the function para_wrapper with decorator. Just remember that before the decorator assigns the new function to your function it gets the original copy of the function via the func argument. So we can call our original function using that reference. Now look what decorator is doing. It firstly makes the replacements in the html_string and then call the original function for its formatted output because that is what we want to send to the caller. Then it returns the output which our function generates for the new html_string. It is one of the use of the decorators. We'll see in the later section what other purposes can a decorator be used for.

Multiple decorators(Nested)

Decorators can easily be nested for multi-level uses. Nesting the decorator simply means nesting the function references.Lets see how nested decorator look

@decorator3
@decorator2
@decorator1
def your_func(args):
    #do some work

#This is equivalent to the following code

your_func = decorator3(decorator2(decorator1(your_func)))

Nested decorators are helpful when there is one decorator which you can't modify and you want to add some input/output formatting or something then the easy way to do this is to add one more decorator.

Class method decorators

Class method decorators are used for class methods. These decorators are not any special ones. As i mentioned earlier that the parameter list of the replacement function must be same as the given function, in a class method decorator the first parameter of the replacement function is self which is the reference to the current object so you can use current objects properties inside the replacement function if you want. All other things are the same. Lets see some skeleton code

def greet(func):
    def replacement(self):
        return 'Hello'+func()
    return replacement

class Person(object):
    def __init__(self, args):
        #initialisation

    @greet
    def name(self):
        return self.firstname +' '+ self.lastname

Where can we use decorators

Python decorators can be used for a lot of scenarios. Some of them are

Input Verification
Input modification
Output Modification

Lets see each of the use case one by one. I will show you the code skeleton only. Input Verification Suppose you want to verify the input arguments before executing the main function. Now you can write the verification code inside your main function also but suppose there are a lot of functions which require input verification then it is smart decission to write a function to do that. This is how our code will look.

def your_decorator(your_func):
    def replacement(your_func_args):
        #do the input verification work.
        if(verification_pass):
            return your_func(your_func_args)
        else:
            return #You can return anything you want like an error message or raise an exception

    return replacement

@your_decorator
def your_func(your_func_args):
    #do what you want with the verified input

Input Modification: We have already seen an example of this in the previous section where we wrote a decorator to replace the < and > symbols with html codes.
Output Modification Now suppose you want to modify the output returned by your function. For example, your function returns a data dictionary but you want to add some default keys to the data dictionary before returning the output. Something like in a client-server session where you want to add session id and expire time in the data dictionary. Now you can add those fields right in your function but as the number of function grows this becomes tedious and we won't want to repeat the code. So lets write a decorator to add the data to the output of your function.

def response(func):
    def decorator(func_args):
        result = func(func_args)
        result['session_id'] = key
        result['expire_time'] = time
        return result

    return decorator

@resposne
def your_func(your_func_args):
    data = {}
    #fill data dictionary
    return data

These were some simple cases where we can use decorators. There are some advanced uses of decorator also but those are only the modification of the above cases.

See More of C

noreply@blogger.com (Kapil Garg) — Sun, 27 Mar 2016 14:52:00 +0000

Why more of c ?
Function Pointers
Search and Sort
Multi-Threaded Programming
Time Functions
Random Numbers
String Conversion
Process Creation and System commands execution
DataBase Handling
CGI Scripting
More Links

Why digging more of C ?

Being a high level language and the features C support there is not a need to ask why should we learn more about C. C programs are relatively faster than most of other high level languages like python or java. That may be because C programs are compiled first or may be because C libraries and fucntions are more hardware oriented. But whatever is the reason, C programs are way better. I can show you an example of it to you. There is a program which corrects the spelling of a mis-spelled word like it changes "somethinh" to "something". Now here is the comparison between the programs written in C and than that written in python

Property	Program in C	Program in Python	Result
Lines of code	184	21	Python is shorter
Test with file 6.9 MB	0m0.892s	0m1.911s	C is 114% faster
Test with file 50 MB	0m6.896s	0m17.892s	C is 159% faster
Test with file 100 MB	0m14.474s	1m25.579s	C is 491% faster
Test with file 168 MB	0m44.627s	killed	Uncomparable

Spell corrector program

So its pretty obvious that C program wins. But most of us don't know what we can do with C other than writing a main() function declaring some structs etc. There is more to that. I assume that you have basic understanding of C and its concepts like functions, pointers, structs, I/O. Now lets see what other things can we do with C. I won't write in detail about these things nor i will write sample programs but i will definitely guide you to some useful sites where you can pick this stuff from.

Function Pointers

This is a whole new topic for some of us. We have red about functions, we know about pointers but what is a function pointer. Well from the name it is inferable that a function pointer is kind of a pointer which points to a function. But still how is it useful ? Well its useful in many ways. Have you heard about callback functions (Functions which are called after execution of a function) Callback functions are passed as arguments in other languages like javascript and python but we didn't know earlier that the same thing can be happened with C also. Just like array name represents a pointer to it, function name alone represents a pointer to it. We can assign them, pass them as arguments and call them whenever we want. Lets look at a simple example.


   #include<stdio.h>
typedef void (*f_pointer)(void) ;
void wrapper(cps doer, cps callback)
{
 (*doer)() ;
 (*callback)() ;
}
void func(){
 printf("I am in doing function\n" ) ;
}
void tobecalled(){
 printf("I am in callback") ;
}
int main(){
 f_pointer doer, callback;
        doer = func ;
 callback = tobecalled ;
 wrapper(doer, callback) ;
 printf("\nI am in main\nExiting\n") ;
}

Searching and Sorting

Some of you have already heard about it but for those who haven't, there is a good news. C stdlib provides 2 methods for optimized searching and sorting and those are qsort() and bsearch() which implements quicksort and binary search algorithm respectively. Here is a complete description about these methods and what parameters they take qsort and bsearch in c. In qsort() method definition, you will see that there is pointer to a function being passed as an argument.

Multi-Threaded Programming

Yes you saw right. We are not talking about java. We are talking about C which is also capable of creating and managing threads. All this is done by pthread library which have some useful functions for thread creation and thread management. For thread management, there are semaphores, mutex variables and condition variables. Here is the webpage where you can find further details. Pthread Details

Time Functions

Whenever we had to dealt with time it seemed so complicated because time functions return some unknown structure and data. Python has a now() function which prints current date and time. There is a similar function in C time.h library which is time() which returns a time data structure from which we can extract current data and time. Not only this, we can manipulate time structures, compare them and use anywhere we want (specifically in our databases). Here is the link to complete time functions C time functions

Random Numbers

Random numbers are much useful when creating test files for our programs and generating random numbers looks like a magical task which only more advanced languages can do like java, python, perl etc. But you would like to hear that C also has the functions which can generate random numbers (pseudo-random actually). The main functions are srand() and rand(). First of these is used to plant a seed to set range of random numbers. rand() is used to generate those random numbers. There are more functions to get random numbers. Here is the link for further reading about random numbers Random Numbers using C.

String conversion

We see in java how we can convert string to integer using new Integer(string). This can also be done using C string.h library. There are many functions but lets see some of them. atoi() converts string to integer. atol() converts string to long int. Similarly we can convert character from upparcase to lowercase using tolower() and from lowercase to upparcase using touppar(). For details of string conversion and character testing i would prefer the same source i posted for random numbers. The next section to random numbers is string conversion.

Process Creation and System commands execution

This is specific to Unix-like operating systems. In C you can create another process from within your C program and can hand over some task to it just like threads. But the created process will be totally independent of other processes. You can then apply Interprocess Communication techniques to get the result produced by the forked process. C is very unix friendly because Unix is totally built using C. Unix like operating systems like linux also supports this functionality and you can create another processes there too. The main functions related to process creation are fork() wait() kill().

Just like creating another process for random task, you can execute another file from within your program using execv() and its variants. If you are using linux, ubuntu etc then just do man exec and you will see other variants of execv(). For detailed description of process creation, i would suggest this documentation Process Management using C

DataBase Handling

Okay so we have pulled ourselves into some serious stuff. Now the things i will be talking will be dependent upon some libraries, APIs etc. I am writing about all these because these things are fascinating and there is no harm in knowing about them. We can write programs in C which can connect to database and handle the data. As C is fast relative to other languages, this seems really helpful but there is one thing C lacks and that is faster string parsing. That is the reason we feel need for python, perl etc. because these languages provide faster and efficient string parsing functions. Still mysql API for C exist and can be used for database handling. Those who are interested can visit MySQL website which contains the API documentation.

CGI Scripting

CGI is the interface which makes it possible for any language to serve as a scripting language. ANy language that can handle strings can be used as a scripting language. Writing CGI scripts using C may seem to be really fun task but i have red somewhere that it is not secure so i don't think any website uses C to write CGI programs in C. Still we can do it and if you are interested to know then there is a library with which you can write CGI scripts in C. Here is the link to that library API CGI library for C

Some other Interesting Sources

We saw so many things which we can accompalish using C but still there is more. I am posting some links to interesting C stuff

Learn Git & Github - 4

noreply@blogger.com (Kapil Garg) — Sun, 03 Jan 2016 13:53:00 +0000

Git Tree Navigation And Management: Lesson-4

Topics

How to change remote repository
Commit navigation in git

Changing Remote Repository

In the previous lesson, we learned how to make commits. If totally followed the lesson then i am sure you world have made some commits also. Now change your working directory to your project directory. Check you are on master and Use git pull origin master to update your master. Since Now, we were syncing our local repository with our forked repository. But what we need is to keep updated with the original project repository. So we need to add the url of original project as our main remote repository. We can do so by typing git remote add main https://github.com/kapilgarg1996/fetch-info-github.git. If you have forked another project then just add its clone url. Now type git remote -v and you will notice that there are 2 remote repositories. One is origin which is your forked repo and another is main which is original project repo. You have control only over forked repo so you can "push" your changes to "origin" only. To update your forked with original project , you need to perform these 2 steps. Perform these steps when you are on your master branch

Pull from main repo by git pull main master
Push to your origin repo by git push origin master

This way, your local project and forked project will be in sync with the original project.

There are times when you make many commits and then realise that somewhere back you made some wrong changes. Then you can go to that commit and edit that commit. Or If you want to start a branch from a specific previous commit, you can go to that commit and start a branch from there.I told you in previous lesson, how to find a commit id. But if you dont recall then go to previous lesson and read about commit.

Jump to a specific commit

Before doing anything, create a test branch so that you can harmlessly run these commands. So i am assumming now that you have the id of the commit to which you want to go to. Now just type git rebase -i commit-id. This will bring you to that commit. It will open up an editor showing you the commits made after your given commit.Now lets what can we do in that editor.

Creating a branch from that commit

To create a branch from that commit, replace the word "pick" before that commit with "edit". Not press Ctrl + X and it will prompt to exit. Type "y" and press "Enter". Now the HEAD is at the commit id you supplied. Type git branch branch-name to create branch. Now to bring HEAD back to latest commit or to continue the work flow, type git rebase --continue.

Editing a Commit

Now lets see how to edit a specific commit.Editing a commit means editing the changes that commit introduced. To do so, just rebase to that commit id and replace "pick" with "edit" in the editor. Exit the editor and Now change whatever you want to change. Now you can either make multiples commits there or you can ammend that commit only by typing git commit --amend. Ammending a commit will keep the same commit but new changes you put.Now again continue your work flow by git rebase --continue

Merging 2 commits on 1

This is sometimes the case when you have created so many commits and want to merge some because some commits reflect only a change of a line or two. Lets see how to merge commits. Go to the specific commit and then the editor will open. The first commit in the editor will be your given commit. To merge the commits, replace "pick" with "sqaush" of the commits which you want to merge. Those commits will be merged with the previous commit whose command is "pick". Then exit the editor but this time it will ask you to edit commit message if you want. Then continue your workflow and check your git log. You will see that the commits have been merged.

Removing/Deleting Commits

This feature reflects the real power of git. Git is made so that you can control what changes you need and what you don't. Also, reverting the changes should be simpler than a text editor. In git we can remove commits and can undo the changes that commit made so far. Let me tell you how to delete specific commit or whole commit range in git.Suppose you have commit history like:

commit id34
commit id33
commit id32
commit id31
commit id30
commit id29

And you want to delete commits id31, id32. Lets suppose your branch is "branch1". Now to delete those commits type git rebase --onto id30 id32 branch1. Understand this code carefully because putting wrong commit ids can delete some important commits.

Pick/Move specific commit

Lets suppose that you have a fully working branch. Now you create a test branch to add some features. You made some important commits and then some not-so-important commits. Now if you want to add 1 or 2 specific commits of test branch on your master branch then there are two options. One is moving the commits to the master branch and second option is picking the commits from test and placing them on master. Lets have a look on both of them

Pick commits :You can use cherry-pick command. Just type git cherry-pick commit-id. It will place the commit specified on the top of your current working branch.
Moving Commits :You can move commits but before moving you must make sure that in the range of the commits, the last commit must be on top. For example, for this commit history :
- commit id34
- commit id33
- commit id32
- commit id31
- commit id30
- commit id29
If you want to move commits id31 and id32 then go to commit id32 and create a branch from there. Lets say the branch name is "temp". Now temp will look like:
- commit id32
- commit id31
- commit id30
- commit id29
Now you can move id31 and id32 to master by typing git rebase --onto master id31 temp

This is all for this lesson and for this guide. Whatever you have learned in these four lessons is sufficient to handle git and github.If you come across any problem then you can dig into stackoverflow or drop a comment below. If you have come across a problem called "merge conflict" then i would recommend you to visit This to see how to resolve merge conflicts. Also, i remind you to subscribe you to our newsletter which will update you about future posts directly by e-mail. So signup now..

If you face any problem then feel free to drop a comment below and i'll tell you a loophole.

Learn Git & Github - 3

noreply@blogger.com (Kapil Garg) — Sat, 02 Jan 2016 14:28:00 +0000

Git Commands: Lesson-3

Topics

What we learned in previous lesson
Git Terminology
Git Commands for project development

What we learned in Previous Lesson

In the previous lesson, we learned about github and how can we use it along with git to create an open source project(A project which is free to use and distribute). We set up our project on github and cloned it in our machine to modify it locally(On our machine only). Now either you have cloned fork of an existing project or your own project, the development process is nearly the same. I'll point out the difference wherever applicable.

Git Terminology

In first lesson, we talked about 3 most occuring terms of git which are branch, commit and head. Now Lets talk about some more terms which are associated with git

Index :Index is the current state of the project for git. If you have changed some files then you will have to add them to git index so that git can track those files for future changes. If you don't add some file to index then it won't be tracked. Also it is necessary to add files to index to make commits to those files.
Push :Pushing means sending the changes we made on our machine in our project to the hosting service or remote repository which is our github repository. So what does it actually mean that if we changed some files of our project and we want to upload the changes to github then it is called pushing to the remote.
Pull :Pulling means getting the current state of the remote branch. There can be lots of branches in the github project. If a team is working on a project and some developers have brought changes to some files of a branch then you can update your local project(Project on your machine) by pulling those changes. So if you have forked an existing project then what you do to keep your branches updated is you pull the changes from the original project branch to your local project branch then you push those changes to your github forked project.
Rebase : Rebasing means putting your commits over the main project commits. For example, if you are editing some files and make some commits but before you push them to your github project some team member has updated original github project then what you would want will be to put your changes on the top to prevent conflicts between your changes and other changes which that team member brought. So in this case we pull that developers's changes with a rebase which will update our local project with the main project and then put our local commits on top of commit history. Now when you will push your changes then your changes will be the most recent ones.

There can be some other terms also which you may notice while searching for a answer but those terms are actually the command names which we'll learn in the next section.

Git Commands

Git is actually a command line software which means you will type commands in terminal or git-bash to tell what to do to git. So to use git, you must be familiar with many of its commands. In this section, i'll tell you some git commands with which we'll continue our project development. If you have set up your project on your machine then you are ready to go but if you have not then i'll advise you to go to previous lesson and set it up first.Also make sure that your present working directory is your project directory otherwise running the following commands will throw error. Also i recommend you to fork and clone fetch-info-github repository so that you can make sure that nothing gets messed up and i can visualise what you are seeing right now.

Before we start, lets set our usename and email on git by typing git config --global user.name your-name and git config --global user.email your-email

branch :We have cloned the project. So first lets see what are branches do we have initially. The command git branch will list the branches which are present in out working tree. Currently there should be only master branch. So lets create one for development purposes. You should not change the master branch directly because you can have it as reference point from where to continue the work. Always create a new branch for some further development. Make commits to that branch. If everything works fine then merge that branch with master and push it or push it to some other branch on github and merge it there with master and pull the remote master to your local master. But for now, lets just create a branch by typing git branch -l any-branch-name. Now run git branch again and you will see that there is your new branch now along with your master. You can delete a branch by typing git branch -d branch-name
checkout :Checkout is used to navigate your git tree. It can help you switch your working branch, create a new branch and move between commits. Here we'll limit ourself to only switching and creating branch. Moving between commits and doing stuff there will be covered in next lesson. First check your working branch by git branch. Working branch will be colored. Now switch to some other branch by typing git checkout branch-name. Again check your working branch. You can also create branches with checkout command by typing git checkout -b new-branch-name. So far so good.
log and reflog :Log and Reflog commands are used to see the git history. Type git log and it will show the commits made so far along with their authors and time stamp. If you want to see associated file names then type git log --name-status. To see the movement of HEAD pointer you can view reference log by typing git reflog
status :Status command tells you the current status whether there are some untracked files(those files which haven't been added to git index) or any merge conflicts. To check status type git status
add :Now lets start some real stuff. Create a branch named "test" by checkout command. Check your working branch and it must be "test". Now create any file you want or open any file you want to modify. Change some code or write some code in new file and save the file. Now check status of your project and this time you will see some red lines telling you that your modified file or created file hasn't been added to tracked list. To add that file type git add "file name". You can also add the whole directory to index by typing git add .. Now again check the status. It will show that files have been added to index but changes haven't been commited yet.
commit :Now lets commit our changes. Type git commit -m any-message-or-change-description. NOw check status again and it will show that everything is fine. Check your log by log command and you'll see that your commit is on the top of history. The first 7 characters of colored line in git log is the commit id. For example, if you see something like commit 18b67246fa50778bd64fe0462bce45357fc5abd5 then your commit id is 18b6724.
pull :Okay so we have made our first commit. But we are not sure whether we have the updated version of the branch or not. To update the project, we will pull the remote project changes by typing git pull --rebase origin master This way our project will be synced with the remote project and our commits will be on top if there are no merge conflicts. Merge conflicts resolution will be covered in later lesson.
push : Now lets push our commits to the remote project. In your case, your forked project. We do so by typing git push origin branch-name. Then you'll be prompted with github username and password then it will upload your changes to desired branch.

Finally we have made some commits and have pushed them to remote project. Now if you have created a new project then your changes will be in your branch and you can merge them with master directly on github. But if have a forked project then the changes will be uploaded to the branch of your forked project. To let the developers of original project know about your changes and to request them to add those changes on their original project, you'll have to create a pull request directly on github. All these minute details will be covered in next lesson and we'll also learn to navigate git tree so that we can precisely control our project versions. So stay with us. You can subscribe to our newsletter just by entering your e-mail address and you'll get notified about the next lesson by your e-mail directly. So what are you waiting for..Sign up Now...

Please don't forget to respond to confirmation mail sent to you for this subscription. If you don't click the link sent to you, you won't be subscribed.

If you have any doubt or need any help then just a comment below.

Learn Git & Github - 2

noreply@blogger.com (Kapil Garg) — Fri, 01 Jan 2016 15:31:00 +0000

Learn Github: Lesson-2

Topics

Difference between Git and Github
Geting started with github
1. Work on existing github project
2. Creating a new project on github

Difference between git and github

In the first lesson we learned how git works, Now we will see what is github. People often get confused between git and github. These two are entirely different things providing the same functionality. The major difference between git and github is that git works on your machine and github is a cloud service. You can see github as a cloud version of github where an entire team can work on a project. So there is not a point of choosing either one of them. You use git only if you have a project on which you are working alone and then you use github along with git if you are handling a team project. Its not mandatory to have a team to use github. If you want to share your code to people then you can push/upload it to github also. Don't worry about getting it changed by developers because the project will be in your hands and you will be the person to choose which changes should your project accept.

Getting started with Github

Lets see how can we use github. There are 2 reasons to get to github.

Either you want to work on an existing github project
You want to start your own project and want to share it with the world.

Work on existing github project

So we'll consider the first reason first. Suppose you want to work on an existing project. Then you'll definitely have your desired project url but in case you haven't decided yet then you can go to this Fetch-info Github Tutorial repository (Open it in a new tab so you can go along). Every project uploaded on github is called a repository. Now when you have opened the wep page. The first thing to do is "Fork the project" by clicking a button "Fork" on right side in the top See Image. When you fork a project, github creates a copy of that project under your account which you can modify anytime but those changes won't reflect in the original project.

Lets assume that you have forked the project and now the code url will be something like https://github.com/your-username/fetch-info-github.Now check a text field containing a link aside a button "Download Zip" See Image. Copy the link from the field. Its the link from which you can download that project on your machine. Downloading a project is called cloning in github terminology.

Once you have copied the link, open your terminal/git-bash on your machine and go the directory where you want to download your project. Now type git clone your-link. Now git will download the project along with the edit history of that project which means all of its commits history. Go to your forked project webpage and see a drop-down menu in left with label "branch" See Image. If you click it then you will see that there are more than one branch. One branch being the master and other being alpha if you have forked my dummy repository. Now go back to terminal and change your directory to your project name which git downloaded.

Once inside the directory, type git branch. You will notice that there is only one branch "master". Its advantageous becuase all other branches are normally either for testing or development. The branch which matters is "master". So you have your project set up on your machine. You can type git log to see what commits have been happened. This is just set up. Main development on that project will continue later when we'll be discussing about git commands so sit tight. Also these steps are for people who want to contribute to existing projects. Those who want to create a new project read the next section.

Creating a new project on github

Creating a new project on github is not a difficult task to do. Just click the "+" button on the top right corner of the webpage aside your "profile pic" See Image. There you will be presented with many options but we'll click "New Repository" because we want to create a project. When you will click the "New Repository" button, a page will open which will ask for project name. Enter the name which doesn't exist earlier. Now add description of your project. Public repositories is our option because we haven't paid for the private service. Now if you want a Readme file in your project which will contain the description then click "Initialise with Readme". Choosing a licence is very important if you are doing some series development. Here is something which can help you Choose License But if you are doing some basic development then you can choose MIT license which distribute your project for research purposes.

When you have done all this then click "Create Repository". Then you will have your project. Now check a text field containing a link aside a button "Download Zip". Copy the link from the field. Its the link from which you can download that project on your machine. Downloading a project is called cloning in github terminology.

Okay folks. Now you have set up your project. In the next lesson, we will study some basic git commands and will use them to develop our project further.

Learn Git & Github

noreply@blogger.com (Kapil Garg) — Thu, 31 Dec 2015 09:01:00 +0000

Learn Git : Lesson 1

Why Git or any VCS

Image from git-scm

Version control system is a software which maintains the records of changes done to a project and also the states of the project which you can switch anytime. As a programmer, it sometimes happens to me when i write a piece of code and it works but when i try to optimize it then it gets broken. Also i could not be able to traceback the changes and get my working code back. If this happens to you also then you might undo tens of times to get back the previous code.

But what will you do if your project contains multiple files with hundred lines of code in each file. If you now brong some changes to more than one file then its not easily possible to get back the working code. Here comes the importance of a version control system. In a VCS, you can save the current state and continue to work on a different state but it doesn't mean that vcs creates a copy of the current state. No. It maintains the record of the changes bring to a project. It maintains a history which you can navigate to get what state you want.

About this guide

Git has becoming very popular nowadays as the open source softwares are emerging in the market so a lot of developers want to know what exactly git is. Not only knowing about the git, learning and understanding it is also needed to develop large softwares/products with a team. There are many tutorials, guides, video lectures are available to learn git then why this one more guide ? The reason i am writing this guide is that after learning git from various guides, we'll still be wandering stackoverflow to get help for our unique scenarios. So what's wrong with other guides ?
All other guides lack one common thing, User perspective. Many of the guides are written to provide the theoretical knowledge of git commands but don't tell the other uses of the same command. In this guide, i'll not only cover the git basic commands, but will also tell you various tips and tricks to get your task done in git. You won't just get to know about git commands but will understand what these commands do and can do. Not only the commands, how git works is also important to understand.
So this guide will make you fall in love with git. Its not that other VCS are not good but i know about only git and other VCS also work in the same way but the syntax gets different. Below is the list of topics, i'll be covering.

Topics

How Git works
Get yourself on Github
Using commands the smart way

How git works

Git and many vcs have such terms in their terminology like branches, commits, head, master etc. We'll use these terms and will see how actually git works.

Branch :A branch is a state of the project on which you are working. For example, suppose you create a project and name it P1.0. Now you want to modify some code without breaking the working code then what you do is you create a branch and name is what you want. Now whatever changes you bring to that branch does not effect the branch P1.0. In this way you can safely work on your differnt branches without breaking anything.
Commit :Saving a change is called commit. If you change something in your branch then you should commit/save it so that it gets recorded it git history. An important advice:- Commits should be small changes. If you have changed a lot in your project and you commit it. Then you find out that it has broken the code then you'll have to write the whole changes again to fix it. Instead of doing large commits, you should change small, commit it, then change small, commit it and so on. This way you can navigate git history to get to the commit where the code seems to broken. 10 smaller commits are much much better than 1 large commit.
Head :Git maintains a pointer called HEAD to the current working branch and commit. When you navigate the git history then what you do is you actually change this HEAD pointer to that branch or that commit. So HEAD is just a pointer.

Git Tree

Now lets have a look at the git tree.

In this tree, you can see that the nodes of this tree are commits which you do in a branch. There you can see 3 branches. In this tree i have a growing branch which has stable versions of my project which are 1.0, 1.2 and 1.6. Then there are 2 branches for testing phase of my versions which are 1.0-test and 1.2-test. The main thing about git is that you can jump between branches and commits whenever you want. In the 3rd lesson, i will tell you how to navigate git history. So stay tuned.

Python : Learn it my way

noreply@blogger.com (Kapil Garg) — Sun, 20 Dec 2015 15:07:00 +0000

Why Python

The first question we should ask ourself before learning something is "Why should we learn it".I don't know about other stuff but i can answer why should you learn python. Python is a scripting as well as programming language. What does that mean is we can use it for web servers, softwares etc. Also it has a vast open source community behind it which provides thousands of libraries, frameworks and applications based on python.Today, development is moving forward to use languages like python, perl , ruby etc. So if we want to keep ourself updated with latest technologies then we'll have to learn them.Python also becomes handy for GUI development.Learning python is considered to be easier than any other object oriented language.

Is this a python tutorial ?

Obviously not. This is not a python tutorial. I am just gonna show you the right path to get yourself into python. When i first faced python(during sublime plugin development), it seems so scary that i skipped the plugin development and moved to something else. But recently i found out some useful resources to learn python from and it took me a week to learn python sufficient for software development and obviously coding.So if you want to find out the right tutorials and stuff to learn python then keep reading otherwise you can skip all of this post.

Particular Resources

There are thousands of tutorials on the internet about the language but some of them don't start from the right point. Also finding right books is a hard task to do. So the first step you are gonna do which i did, just visit google python class web page and there you will learn the basics of pyhton in just one day.

Google Python Class

Then the next step is optional, you can attend codecademy course to practise what you learned at google python class. In that course you will come to learn about functions and classes as well.

Codecademy Course

Then the main part comes in. Getting a really good book or e-book. There are many books out there but many of them are just not so good. The books which i studied are:-

Learn Python the Hard Way
Dive into Python
Rapid GUI Programming with Python and Qt: The Definitive Guide to PyQt Programming

All these books are good to read but the 3rd one is concise about python programing and the Dive into Python is a very elaborated material. The first one is actually teach you python in a simple way but its not much of use.

Want these e-books ?

If you want these e-books right in your e-mail inbox then just fill up the subscription form and i'll send you the books within 24 hours. Subscription to this blog is totally free and i don't spam my subscribers mail box so feel free to take advantage of this blog's free subscription.

Get your machine ready

Python is a interpreted language so you will be needing a python interpreter which you can easily download from their website Python Download. Ubuntu users don't need to install it at all becuase it comes pre-installed. How to run your programs is well explained in google class. If you want to know anything else then just drop a comment below and i'll help you asap.

C-Mate : Your companion for C in Sublime Text

noreply@blogger.com (Kapil Garg) — Fri, 04 Sep 2015 16:34:00 +0000

What is C-Mate ?

When it comes to writing code then either we choose an IDE like codeblocks or ecllipse, OR we use some text editor like notepad++ or sublime text. Sublime text is a very powerful, flexible and free text editor. You can install any package you want in sublime which help you writing codes easily. C-Mate is also such a sublime package which will help you write C code efficiently on ubuntu. It consists of many useful snippets, pre-coded functions and a powerful build system.

Why C-Mate

Although there is a pre-installed c++ sublime package which helps you writing c as well as c++ codes but the feature it lags is the code running in the terminal. People like writing code in sublime but when it comes to build and run the code then they need to go the directory containing the code file and then build and run it from there. It gets so annoying when we debug the code. No sublime package allows to run the code within the sublime. Here C-Mate makes the exception. Its build system is capable of running the code in terminal from within the sublime.

Also it comes with some pre-coded functions which noone wants to write again and again but need to write them unwillingly. These functions are min(int *), swap(int, int) and much more. You all just need to start typing the name of the function and it will write the function declaration and function body of that function.

C-Mate also consists many useful shortcuts/snippets which speed up you writing. Like include, for loop, while loop, main function, structure, typedef and many more.

How to get it ?

It has not come to the sublime package install yet but it will be there soon. Till that time, you'll need to install it manually. Don't worry, you just need to follow 2 steps.

Download the C-Mate from here. Download C-Mate
Copy the downloaded file to /home/.config/sublime-text-3/Installed Packages/. Thats it
Now just type your c code and press ctrl+b and select C-Mate.

How to use it ?

After the installation, just start typing your code and C-Mate will start helping you. When you are finished your code, just press ctrl+b and select C-Mate. You wont have to select C-Mate every time. Its just for the first time.

Associated keys

ctrl+b : Build the code
ctrl+shift+b : Run the code in the terminal.

Some snippets keywords

main : for main() function
inc : include statement
for : for loop
while : while loop
type : typedef
.......and many more

Note : It is to note that the application file of your code will be create in the same directory in which the code file reside. And its name will be the same as of the code file but without the .c extension. This means that you can create multiple application files in the same directory without the fear of losing the previous one.

In the near future, i'll try to incorporate auto function declaration feature in C-Mate. If you want to contribute to C-Mate then you can visit its . If you have any doubt regarding its functionality then just comment it out below and i'll be happy to solve your query.

Why to Use Ubuntu

noreply@blogger.com (Kapil Garg) — Mon, 20 Jul 2015 08:16:00 +0000

Why Ubuntu

Brief Intro to Ubuntu

This section follows the legacy of article writting in which an intro to the subject is necessary. Ubuntu, as you all know, is an operating system under linux distribution. Ubuntu started as a command like operating system but now it offers the gui also to accompalish all the tasks. Ubuntu is open source software so it comes free. Using it is easy and it suits best for development process. The next section of this article describes the benefits come along with ubuntu.

What comes in the box ?

There are so much benefits which ubuntu brings. But listing all of them is not necessary. The following few things about ubuntu will be enough to persuade you to use it or atleast give it a try.

Open Source: What it means is that it is totally free of cost. It also specifies that there are thousands of developers sitting who are willing to help you anytime you are stuck with your machine. New updates come regularly and bugs are tackled as soon as they are confirmed. Not only this, thousands of applications built for ubuntu are also open source. So you can think, there is a family of open source softwares which come along with ubuntu for free.
Fully Customizable: This is also one of the best features which only an open source software can provide. You can customize the code as you like(If you want to but why would you). If you have ever rooted your phone then you know whats it like to have a fully customizable operating system.
Powerful file handling: It is incredible the way we can handle our files in ubuntu. Just one command and we can make our file restricted. Just one command and we can delete the whole operating system files. Password protected files, hidden files, directories and links are easy to create and delete.File compression and decompression can easily be done either via gui or cli.
Incredible Package Management: The reason why developers prefer linux based operating system is its awesome package management capabilities. Ubuntu has got a
Advanced Packaging Too(apt)
which can install almost any package in your system. apt-get install is the most basic command we use to install/update the applications.
Superb Network Management: When we think about networking or network sharing then our thoughts are very limited to routers, adapters and ip addresses. But what is behind all these is the management of these. In ubuntu, network management is so handy that you can learn it easily. Changing IP addresses is just a child's play. You can learn more about networking through various sources which can be found at the end of this article which is about to come.

Getting Ubuntu on your machine

After reading all of the above points, you are definitely thinking of trying ubuntu atleast once. In this section i'll tell you how to get it ready on your machine. There are basically 3 methods to use ubuntu.A detailed description of each is not necessary in this article because there are already the official documentation of each method on the internet. But still to reduce your google work, i have posted the links to those documentations at the end of the article.

Using Virtual machine: Virtual machine is a software which can run system images (iso files) without affecting the other operating system in which it is running. Its the safest and easiest method to run ubuntu. Just install the virtual machine and run the iso file using it. The detailed documentaion is available on howtogeek website.(link is at the bottom). Its good to use virtual machine to get started with ubuntu but virtual machine degrades the performance of the OS, so use it temporarily.
Using Live CD/pendrive: Live CD is another safer option to try ubuntu but its just for the trial basis. You can look and try the ubuntu features and work for as long as the cd/pendrive is inserted in your machine.
Dual boot: Once you have tried ubuntu then if you want it permanently then dual booting is the only option. It will install the grub(a software to manage os booting) and you can switch between windows and ubuntu by just restarting. However, the process of dual booting is longer than the rest two.

So thats all. I love ubuntu and thought everyone should give it a try, so wrote this article. If you are also an ubuntu lover then encourage your friends by sharing this article to them or if you are new to ubuntu and have some questions then comment it below and i'll defintely try to help you out. In the next article, i'll highlight some wonderful ubuntu commands. Till then, Hakuna Matata.

Helpful Links

Download Ubuntu
Virtual Machine Installation
Live CD method
Dual Booting
Ubuntu wiki