MQH Blog

Writing Less Simple, Yet Stupid Filesystem Using FUSE in C

Sat, 28 Sep 2019 00:00:00 +0300

In the previous tutorial we learned how to write Simple, Stupid Filesystem (SSFS) by using FUSE, we have covered the basics of FUSE to write a really simple filesystem which is able to list root directory’s files, return the attributes of available files and read the content of a file. In the current tutorial we are going to extend what we have discussed and create a Less Simple, Yet Stupid Filesystem (LSYSFS) which is able to create new directories, create new files and write some content in the files.

Please note that some parts of this tutorial is already covered in SSFS tutorial, therefore, I recommend you to read it first. Also, please make sure that FUSE is installed on your Linux machine and you are ready to go. I used version 2.9.7 of FUSE in this tutorial, which also runs SSFS with no issues.

What “Less Simple, Yet Stupid Filesystem (LSYSFS)” will be able to do?

We have mentioned previously that LSYSFS will be able to do the following:

Create new directories by implementing the function do_mkdir for the event mkdir and introducing a simple array to maintain the list of directories created by the user.
Create new files by implementing the function do_mknod for the event mknod and introducing a simple array to maintain the list of files created by the user.
Write to files by implementing the function do_write for the event write and introducing a simple array to maintain the contents of the files.

As SSFS, LSYSFS is an in-memory filesystem. That means all the information of the filesystem (e.g. created directories and files) will reside in the memory and will not be stored persistently on the disk, once the filesystem is unmounted, all of its contents will go.

For the sake of simplicity, the new files and directories can be only created in the root directory of the filesystem “/”, we will call this limitation “root-level-only objects” to ease our discussions later on. An object is either a directory or a file. However, the concept will be the same if you want to make it possible to create files and directories in a sub-directory, but more advanced data structure should be used instead.

Including Headers

We need to start our C file with the following code to include needed headers:

#define FUSE_USE_VERSION 30

#include <fuse.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <time.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>

Initializing Data Structures

Before working with FUSE API, we are going to work on our data structures and their auxiliary functions. The goal of these data structure is to keep the list of directories and files which are created by the user, also, the content of these files will be maintained by one of these data structures. Because we aim to create a simple filesystem here, our data structures will be simple arrays.

char dir_list[ 256 ][ 256 ];
int curr_dir_idx = -1;

char files_list[ 256 ][ 256 ];
int curr_file_idx = -1;

char files_content[ 256 ][ 256 ];
int curr_file_content_idx = -1;

The first array is “dir_list” which maintains the names of directories that have been created by the user, the second array “files_list” maintains the names of files that have been created by the user and the third array “files_content” maintains the contents of the files. As you can see all of them are arrays of strings, each array can store 256 strings and each string has the maximum length of 256 bytes. That means, we can only create 256 directories and 256 files in LSYSFS. Furthermore, the length of filename, directory name or file content can be only 256 character.

The variables “curr_dir_idx”, “curr_file_idx” and “curr_file_content_idx” indicate the current index of each array. Initially, there is no entries in any of these arrays, so we start with -1 for each index. Now, let’s define some functions for these arrays to make our life eaiser when we start dealing with FUSE.

void add_dir( const char *dir_name )
{
	curr_dir_idx++;
	strcpy( dir_list[ curr_dir_idx ], dir_name );
}

The first function is “add_dir” which adds a new directory to the list of directories. It takes the name of the directory that we wish to create. As you can see, we first increment the current index of directories then we copy the name of the new directory to the new position of the array. So, when the first directory is created, the value of “curr_dir_idx” which is -1 will be 0 and the name of the first directory will be copied to the position 0 (which is, as you now, the first position of the array in C) of the array dir_list. Let’s move to the second function.

int is_dir( const char *path )
{
	path++; // Eliminating "/" in the path
	
	for ( int curr_idx = 0; curr_idx <= curr_dir_idx; curr_idx++ )
		if ( strcmp( path, dir_list[ curr_idx ] ) == 0 )
			return 1;
	
	return 0;
}

Given a path, the function “is_dir” is going to check if this path is a directory available in the filesystem by trying to find the name of directory in the directories list “dir_list”, if there is a directory with such name, “is_dir” returns 1, otherwise it returns 0.

Now, to understand the line “path++” we need to discuss the point of root-level-only objects further. When the user needs to deal with an object (a file or directory) in the filesystem, the full path of this object will be sent to our filesystem by FUSE. For example, when the user wishes to write some content to the file “bar” which resides in the directory “foo”, FUSE sends a write request to our filesystem with the full path “/foo/bar” of the file.

LSYSFS supports root-level-only objects, the means there will be no files or directories inside a sub-directory as in the example of “/foo/bar”, that means any path recieved by LSYSFS will be in the following format “/[Object Name]”, which indicates that the object resides in the root directory of the filesystem. And as you have seen in the function “add_dir” (and as you will see later in the function “add_file”), the names of the objects will be stored in the data structures instead of their paths, therefore, we need to get rid of the first slash in the received path which indicates that the object resides in the root directory. That’s exactly what the line “path++” does. After this line, we will have only the name of the object, and we can easily try to find it by using a simple sequential search on the array “dir_list” and that’s implemented in the for loop block of “is_dir” function.

Well, that’s it with the functions that relate to directories list. Let’s now move to the data structures of both files list and files contents.

void add_file( const char *filename )
{
	curr_file_idx++;
	strcpy( files_list[ curr_file_idx ], filename );
	
	curr_file_content_idx++;
	strcpy( files_content[ curr_file_content_idx ], "" );
}

Same as “add_dir”, the function “add_file” adds a new file in the filesystem by adding a new entry in the array “files_list” with the name of the new file. It also initializes the content of this file to be an empty string and stores this content in the array “files_content” as we have mentioned before.

You can see here that the files and their contents are related to each other by the file index which should be the same in both arrays “files_list” and “files_content”, that is, if a file in index x of the array “files_list”, its content should be in the same index x in the array “files_content”.

For example, let’s assume that the first file named “foo” is created in the filesystem by “add_file” function, its file index will be 0 in “files_list” array which stores its name “foo”. Also, its initial content which is an empty string will be in the same file index (0) in contents array “file_content”, so to refer to the file “foo” in our filesystem, we use its file index which is 0, in the way we can obtain both its name and its content. This method reminds us a little bit with inode. Let’s move to the next function.

int is_file( const char *path )
{
	path++; // Eliminating "/" in the path
	
	for ( int curr_idx = 0; curr_idx <= curr_file_idx; curr_idx++ )
		if ( strcmp( path, files_list[ curr_idx ] ) == 0 )
			return 1;
	
	return 0;
}

Same as “is_dir”, this function takes a path and see if there is a file with the same name exists in the filesystem. It returns 1 if the file exists, otherwise 0 will be returned. No further explanation required for this function since it resembles “is_dir” with the only difference that it works with “files_list” array instead of “dir_list” array.

int get_file_index( const char *path )
{
	path++; // Eliminating "/" in the path
	
	for ( int curr_idx = 0; curr_idx <= curr_file_idx; curr_idx++ )
		if ( strcmp( path, files_list[ curr_idx ] ) == 0 )
			return curr_idx;
	
	return -1;
}

Given a path, the function “get_file_index” returns the file index that is valid to be used with the both arrays “files_list” and “files_content”. Same as “is_file”, it uses sequential search on the array “files_list” to find if there is some file which has the given name. If such file is found in the array its index in the array will be returned. If no such file, -1 will be returned.

void write_to_file( const char *path, const char *new_content )
{
	int file_idx = get_file_index( path );
	
	if ( file_idx == -1 ) // No such file
		return;
		
	strcpy( files_content[ file_idx ], new_content ); 
}

The final function of data structures manipulation is “write_to_file”, it takes the path of the file that the user wishes to write some content on and the content, then it writes this content on the file.

As a first step, “write_to_file” tries to obtain the file index by using the predefined function “get_file_index”. If no such file, it does nothing. Otherwise, it is going to use the file index to copy the new content to the contents array “files_content”.

Well, we have declared our data structures, and written the functions that help us in manipulating these data structure. We are ready now to implement LSYSFS with FUSE.

Implementing Less Simple, Yet Stupid Filesystem

To write a LSYSFS, we need to implement functions for the following FUSE events: getattr, readdir, read, mkdir, mknod and write. The first three has have covered with details previously in SSFS tutorial. Therefore, we are going to pass through them with less details.

Implementing “do_getattr”

The function of “getattr” event is the one which provides the information about a specific object for the operating system. It’s the one that tells the operating system the type of the object in question if it is a file, directory or something else. Also, it tells about the owner, group and permission bits of the object. This function is a vital one that without it we can’t have a functional filesystem. Let’s start implementing it with the name “do_getattr”.

static int do_getattr( const char *path, struct stat *st )
{

The first parameter is the path of the object (file or directory) that the operating system is asking about, while the second parameter is the structure will be filled with object’s information. Let’s continue.

	st->st_uid = getuid();
	st->st_gid = getgid();
	st->st_atime = time( NULL );
	st->st_mtime = time( NULL );

Here we fill the following information about the object in question: its owner, its group, last access time and last modification time. Due to the simplicity, we are filling information that are not real in these fields. Of course, in real filesystem, this kind of information will be stored in some data structure and should be accurate.

	if ( strcmp( path, "/" ) == 0 || is_dir( path ) == 1 )
	{
		st->st_mode = S_IFDIR | 0755;
		st->st_nlink = 2; // Why "two" hardlinks instead of "one"? The answer is here: http://unix.stackexchange.com/a/101536
	}

After filling the basic information, we are checking if the object in question is a directory or not as you can see in the if statement. If the object in question is the root of the filesystem, we consider it as directory, otherwise we try to check our data structure if there is such directory in the filesystem or not by using the function “is_dir” which we have defined above. In case that the object in question is a directory, we fill “st_mode” field with the type “S_IFDIR” which means directory, also, the permission bits will be filled in the same field. Then, the number of hardlinks will be filled in the filed “st_nlink”.

	else if ( is_file( path ) == 1 )
	{
		st->st_mode = S_IFREG | 0644;
		st->st_nlink = 1;
		st->st_size = 1024;
	}

If the object in question isn’t a directory, we check our data structures to see if there is a file with such name. If the file is found, the type “S_IFREG” which means regular file will be filled in “st_mode” field, the permission bits, the number of hard links and finally the size of the file which will always be 1024 bytes for the sake of simplicity.

	else
	{
		return -ENOENT;
	}

Finally, if we didn’t find the object in question in our directories list and files list, that means there is no such object in the filesystem. Therefore, we need to tell the operating system that there is no such file/directory by returning -ENOENT. It is crucial to tell the operating system that a file/directory doesn’t exist to be able to create new files and directories.

	
	return 0;
}

Finally, when everything is fine, we return 0 as a success sign. The full code of “do_getattr” is the following:

static int do_getattr( const char *path, struct stat *st )
{
	st->st_uid = getuid(); // The owner of the file/directory is the user who mounted the filesystem
	st->st_gid = getgid(); // The group of the file/directory is the same as the group of the user who mounted the filesystem
	st->st_atime = time( NULL ); // The last "a"ccess of the file/directory is right now
	st->st_mtime = time( NULL ); // The last "m"odification of the file/directory is right now
	
	if ( strcmp( path, "/" ) == 0 || is_dir( path ) == 1 )
	{
		st->st_mode = S_IFDIR | 0755;
		st->st_nlink = 2; // Why "two" hardlinks instead of "one"? The answer is here: http://unix.stackexchange.com/a/101536
	}
	else if ( is_file( path ) == 1 )
	{
		st->st_mode = S_IFREG | 0644;
		st->st_nlink = 1;
		st->st_size = 1024;
	}
	else
	{
		return -ENOENT;
	}
	
	return 0;
}

More about the parameters, return values and errors of getattr is here.

Implementing “do_readdir”

The function of the event readdir provides the operating system with the list of files/directories that reside in a given directory. Our implementation of this function is going to traverse through the two arrays “dir_list” and “files_list” and return their content, and due to LSYSFS’s root-level-only objects limitation, this is going to happen only when the operating system asks about the files/directories inside the root directory. Let’s start implementing “do_readdir”.

static int do_readdir( const char *path, void *buffer, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi )
{
	filler( buffer, ".", NULL, 0 ); // Current Directory
	filler( buffer, "..", NULL, 0 ); // Parent Directory

We mentioned earlier that the parameter “buffer” is where we fill the list of available files/directories that reside in the directory of question. Also the parameter “filler” is a function provided by FUSE that we can use to fill the “buffer”. You can see in the beginning of the function we filled the buffer with the two directories “.” and “..” as we have done with SSFS. Let’s continue.

	if ( strcmp( path, "/" ) == 0 )
	{
		for ( int curr_idx = 0; curr_idx <= curr_dir_idx; curr_idx++ )
			filler( buffer, dir_list[ curr_idx ], NULL, 0 );
	
		for ( int curr_idx = 0; curr_idx <= curr_file_idx; curr_idx++ )
			filler( buffer, files_list[ curr_idx ], NULL, 0 );
	}

Now, we check if the directory in question is the root directory of LSYSFS, we start to traverse through the array “dir_list” and fill its content in the buffer, then we do the same with “files_list” array.

	
	return 0;
}

Finally, we return 0 to indicate that everything is fine. The full code of “do_readdir” is the following:

static int do_readdir( const char *path, void *buffer, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi )
{
	filler( buffer, ".", NULL, 0 ); // Current Directory
	filler( buffer, "..", NULL, 0 ); // Parent Directory
	
	if ( strcmp( path, "/" ) == 0 ) // If the user is trying to show the files/directories of the root directory show the following
	{
		for ( int curr_idx = 0; curr_idx <= curr_dir_idx; curr_idx++ )
			filler( buffer, dir_list[ curr_idx ], NULL, 0 );
	
		for ( int curr_idx = 0; curr_idx <= curr_file_idx; curr_idx++ )
			filler( buffer, files_list[ curr_idx ], NULL, 0 );
	}
	
	return 0;
}

Implementing “do_read”

The function of the event read is called when the operating system wants to read the content of a specific file. In this function we are going to depend on the array “files_content” to get the content of a file.

static int do_read( const char *path, char *buffer, size_t size, off_t offset, struct fuse_file_info *fi )
{
	int file_idx = get_file_index( path );
	
	if ( file_idx == -1 )
		return -1;

As we mentioned, the file index is needed to know in which index the required content is reside in “files_content”. To get the index we use the function “get_file_index” which we have already defined above. When the file doesn’t exists -1 will be returned.

	char *content = files_content[ file_idx ];
	
	memcpy( buffer, content + offset, size );
		
	return strlen( content ) - offset;
}

After locating the file index of the file in question, we get its content from “files_content” array, then copy the content to the “buffer” which is provided as a parameter. Finally, the number of bytes that have been read will be returned. The details of size and offset parameters can be found in the previous tutorial. The full code of “do_read” is the following:

static int do_read( const char *path, char *buffer, size_t size, off_t offset, struct fuse_file_info *fi )
{
	int file_idx = get_file_index( path );
	
	if ( file_idx == -1 )
		return -1;
	
	char *content = files_content[ file_idx ];
	
	memcpy( buffer, content + offset, size );
		
	return strlen( content ) - offset;
}

More about the parameters, return values and errors of read is here.

Implementing “do_mkdir”

Now, we start with the first event that has not been used previously in SSFS, it is mkdir which will be called when the user wishes to create a new directory. The function of mkdir event takes two parameters, the first one, as usual, is the path of the directory that the user wants to create. The second parameter is “mode” which specifies permission bits. The implementation of “do_mkdir” is too simple thanks to the predefined functions of LSYSFS’s data structures.

static int do_mkdir( const char *path, mode_t mode )
{
	path++;
	add_dir( path );
	
	return 0;
}

That’s it! We first eliminate the slash of the root directory path for the reason the we explained above, then, we simply call the function “add_dir” with the name of the new directory. “add_dir” is going to do the job by adding the name of the new directory in directories data structure “dir_list”. Finally, we return 0 to indicate that everything is fine.

More about the parameters, return values and errors of mkdir is here.

Implementing “do_mknod”

The function of mknod event is called when the user wishes to create a new file. The new file can be a regular file, device file or named pipe file. However, we are interested on regular files.

static int do_mknod( const char *path, mode_t mode, dev_t rdev )
{
	path++;
	add_file( path );
	
	return 0;
}

As in “do_mkdir”, the function “do_mknod” is also too simple and works the same way. It takes the following parameters: “path” which is the path of the new file, “mode” which specifices the permission bits and the type of the new file (regular, device or named pipe) and “rdev” which should be specified if the new file is a device file.

In the same manner of “do_mkdir”, the slash of the root directory will be eliminated from the path, and the function “add_file” will be used to add the new file to “files_list” array and initialize the content of the new file. Finally, the function returns 0 to indicate that everything is fine.

More about the parameters, return values and errors of mknod is here.

Implementing “do_write”

The last function that we are going to implement is “do_write” which is the function of write event. This function will be called when the user wishes to write content the a specific file.

static int do_write( const char *path, const char *buffer, size_t size, off_t offset, struct fuse_file_info *info )
{
	write_to_file( path, buffer );
	
	return size;
}

The parameter “buffer” contains the content that the user whishes to write to the file in “path”. Both “size” and “offset” are same as those on the function of “read” event, because LSYSFS is too simple, and the maximum size of the file is only 256 bytes we are not going to deal with these two parameters.

The function that we have defined above “write_to_file” will be used to copy the content of “buffer” to “files_content” data structure. The number of written bytes will be returned.

More about the parameters, return values and errors of write is here.

Filling “fuse_operations” & Telling FUSE About It

Now we fill “fuse_operations” structure and call the main function of FUSE which is going to run our filesystem:

static struct fuse_operations operations = {
    .getattr	= do_getattr,
    .readdir	= do_readdir,
    .read		= do_read,
    .mkdir		= do_mkdir,
    .mknod		= do_mknod,
    .write		= do_write,
};

int main( int argc, char *argv[] )
{
	return fuse_main( argc, argv, &operations, NULL );
}

Well, we have got Less Simple, Yet Stupid Filesystem! :-)

Compiling & Mounting The Filesystem

You can use GCC to compile LSYSFS as the following:

gcc lsysfs.c -o lsysfs `pkg-config fuse --cflags --libs`

The “pkg-config” part is going to provide the compiler the proper arguments to include “fuse” library. To mount the filesystem after compilation:

./lsysfs -f [mount point]

Using the option “-f” will let you see the debug messages which are printed using “printf”.

Source Code

You can find the source code of SSFS in my GitHub account here: https://github.com/MaaSTaaR/LSYSFS.

Demo

PrettyPi: A TODO List for Raspberry Touchscreen with Web Interface

Fri, 01 Sep 2017 07:17:00 +0300

I would like to introduce the first release of PrettyPi. It’s a todo list application which is intended to by used in Raspberry Pi with Touchscreen, it can be used with normal screens also. I needed a way to always remember my tasks by showing them in front of me all the time, to realize that I used Raspberry Pi with its official Touchscreen and PrettyPi and I got some good results :-)

PrettyPi has two components, the user interface which is intended to run on the touchscreen all the time. It shows your not-completed tasks and whether you’re working on some of them or not.

The second component of PrettyPi is the control panel. It’s a web interface that lets you control the user interface remotely. You could add tasks, delete them, marking them as done and set a task that you are currently working on.

Technologies

Of course without the huge contributions of open source community our work will be a lot harder, I would like to thank all contributors of the great open source community :-)

I used multiple technologies to implement PrettyPi, and here the major of them: The Programming Language Python, Kivy Framework, KivyMD Widgets, Python Arabic Reshaper Library, Flask Web Microframework, SQLite DBMS and AdmitLTE template which gave the control panel a really pretty shape :-)

Installation Guide on Raspbian

First, install Kivy on your Raspbian, here is the official guide. And make sure that SQLite3 is installed on your Raspberry Pi.

Installing KivyMD:

If git isn’t available in your Raspbian, install it first:

sudo apt-get install git

Then, clone KivyMD’s repository and install it:

git clone https://gitlab.com/kivymd/KivyMD
cd KivyMD
sudo python ./setup.py install

Installing Other Dependencies

You need pip to install the other dependencies, so if it isn’t in your Raspian install it:

sudo apt-get install python-pip

Now, install the dependencies

sudo pip install arabic_reshaper flask python-bidi

Configuring Kivy to Run With Raspberry Pi Official Touch Screen

Edit the file ~/.kivy/config.ini and in [input] section add the following lines (Source):

mouse = mouse
mtdev_%(name)s = probesysfs,provider=mtdev
hid_%(name)s = probesysfs,provider=hidinput

Installing PrettyPi

Get a copy of PrettyPi from here on your Raspberry Pi then extract the archive file. In PrettyPi’s parent directory you will find two files “run.sh” and “run_web.sh”. Set execute permission for both of them:

cd PrettyPi
chmod +x run.sh
chmod +x run_web.sh

From your touchscreen run the file “run_web.sh” first, then run the file “run.sh”. After running the second file the user interface of PrettyPi will be shown on the screen. If the first file executed successfully, then the control panel is running on port 5000. From any machine that is connected to the same network (or even from your Raspberry Pi) go to http://[Your-Raspberry-Pi-IP]:5000 and the installation page will be shown.

Fill the fields then click “Install”, after the completion of the installation go back to the URL http://[Your-Raspberry-Pi-IP]:5000 and use the username and password that you entered in the installation page and you’ll get the main page of the control panel.

Demo Video

Repository and License

PrettyPi’s license is MIT and its GitHub repository can be found here.

EasyJekyll: A GUI Frontend for Jekyll

Sat, 17 Sep 2016 10:04:21 +0300

Hi All. I would like to announce the first release 0.0.1 of EasyJekyll. It is a simple GUI front-end for Jekyll to let you edit your blog’s posts easily. Written using Java/Swing and its license is GNU GPL 3.

For now it lets you to:

Create new drafts, edit and publish them in your blog.
Edit you previous posts.
Manage the categories and tags of each post.
Manage the custom front-matters for each post.
Run your blog locally using Jekyll.
Deploy your blog to your webserver. The only available deployment option now is FTP.

You can download the binary/source code from GitHub

Screenshots

Using EasyJekyll

First Setup

After downloading the binary file from GitHub and extracting it you’ll find a jar file and a directory. Make sure that Java is already installed in your system. In Unix-like system you can use the following command to run EasyJekyll:

java -jar [Path to EasyJekyll-0.0.1.jar]

After the first run, EasyJekyll is going to request the path of your blog’s directory. If you haven’t created your blog, do that using Jekyll. Make sure that the path you provide is a path to a directory and it is exist in your filesystem.

After running EasyJekyll successfully go to “Settings” button, you can find it in the bottom of the main window, and fill the deployment information if you are planing to use FTP to deploy your blog. Also you need to fill the path of Jekyll’s executable file if it is different in your system. In the same “Settings” window you can change the blog directory to another one.

Working with Drafts

To create a new post you need first to create a new draft by using the button “New Draft” in the main window. It is going to open the editor. The first line on the editor is the title of the draft. When you would like to publish you draft simply press on the button “Publish” in the editor and the draft will be treated as a published post.

Press on “Deploy” button to publish your new changes to your webserver.

For Developers

If you would like to play with EasyJekyll you will need Apache Commons Net 3.5. The directory “assets” can be found in the binary distribution of EasyJekyll. It contains third-party fonts.

Have Fun! :-)

Live Storage Migration in Virtual Machines and VMWare Example

Wed, 25 May 2016 00:00:00 +0300

In Virtual Machine Monitors’ world there is a term called “Live Migration”. In this post I’m going to explain what do “Live Migration” and “Live Storage Migration” mean, then I’m going to show how does VMWare implements its Live Storage Migration and finally I’m going to share with you a simple implementation of VMWare’s Live Storage Migration.

The Meaning of Live Migration

Well, imagine that there is a running virtual machine - let’s call it VM1 - which has a webserver on it and serving some important websites that we need them to be up 24/7, at some point we need to do some upgrades on the computer which runs this virtual machine, let’s say, for example, we want to add more RAM for that computer or we want to update the kernel of the operating system and we need to reboot that computer. In this case the obvious solution is to shutdown the virtual machine then doing the upgrading process, but remember, downtime for the websites that we serve is not an option, maybe it is acceptable for few seconds but not for minutes.

In this situation we could use the “Live Migration”, the concept is simple, we are going to run a new virtual machine - let’s call it VM2 - on another computer then we run “Live Migration” process which is going to move the state of VM1 (source) to VM2 (destination), after the process finishes the webserver will be running on VM2 instead of VM1, so it is fine to shutdown VM1 to do the upgrade while VM2, which is working on another computer, serves those important websites.

And, What is Live Storage Migration?

According to the paper “The Design and Evolution of Live Storage Migration in VMware ESX”, early “Live Migration” solutions did not migrate the virtual disks of the source virtual machines but only the state of CPU and the main memory. “Live Storage Migration” means that also the virtual disks of the source virtual machine will be moved during the process.

VMWare ESX’s Implementation of Live Storage Migration

(Note: this part of the post uses the paper “The Design and Evolution of Live Storage Migration in VMware ESX” as a reference)

In VMWare ESX there are three techniques that implement Live Storage Migration: (1) Snapshotting, (2) Dirty Block Tracking (DBT) and (3) I/O Mirroring.

Snapshotting

In this technique a snapshot - let’s say a copy - of the virtual disk will be taken by the hypervisor, any new changes on the files will be written on this newly created snapshot while the base virtual disk will be copied to the destination virtual machine. When the copying process completes the snapshot will be written to the base disk, a new one will be created and the process will be repeated until the changes on the snapshot be as minimum as possible (according to some threshold) where the source virtual machine is suspended – to ensure that there are no more modifications on the source’s disk take place - to complete the migration process.

Dirty Block Tracking (DBT)

We know that the hard disks are divided into blocks. In this technique the migration process starts by copying the virtual disk of the source and while doing that the hypervisor is going to track any changes on the data of the disk and mark the block of this data as a dirty block. After the initial copying finishes the hypervisor examines the dirty blocks and moves them to the destination. This process will be repeated until reaching some threshold which causes the source virtual machine to be suspended and copying the remaining dirty blocks to complete the migration process.

I/O Mirroring

The most efficient technique is I/O Mirroring in terms of migration time and downtime. As usual, the process here starts by copying the source’s virtual disk and the offset where the copying process reached will be held - let’s call it cp_offet -. When a write process happens while the copying process is working there are three cases. (1) The modification is on an offset before cp_offset, (2) The modification is on the same offset as cp_offset or (3) The modification is on an offset after cp_offset.

In case (1), Obviously the offset which the changes must be written on is already copied into the destination. Therefore, the changes will be written on both source’s and destination’s disks. In case (2) the offset which the changes must be written on is under copying, the write process will be waiting for a while until the copying of this offset is done, then the write process starts and the new changes will be written on both source’s and destination’s disks. Finally in case (3) the data there has not been copied yet, therefore, it will be written only on the source’s disk and at some point of time it will be copied.

The Paper “The Design and Evolution of Live Storage Migration in VMware ESX”

In case you are interested in more details of those three techniques and their evaluations I recommend you to read the paper The Design and Evolution of Live Storage Migration in VMware ESX” which this post based on. Some really interesting parts have been omitted on this post in sake of simplicity - specially the hot blocks part -.

Example Implementation of I/O Mirroring

Under the supervision of Prof. Hussain Almohri I have written a simple implementation of I/O Mirroring in C and I used FUSE to track the offsets. I published the source code under GNU GPL in my GitHub account here

Writing a Simple Filesystem Using FUSE in C

Sat, 21 May 2016 03:00:00 +0300

FUSE (Filesystem in Userspace) is an interface that let you write your own filesystem for Linux in the user space. Of course being in the user space is a big advantage, you could use any of available libraries when you build your own filesystem in contrast of kernel space which needs a deep understand of the kernel which you are working with. Actually you can’t build a native filesystem which can store data to disk directly [1] instead you need to use an already exist filesystem to do so [2]. However; you still could build interesting stuff, for example GDFS is a filesystem which let you mount your Google Drive in your system and access the files there as normal files. You can find a list of filesystems that implemented using FUSE here. For me I found ext4fuse really interesting.

Personal Experience

In one of my projects I needed a way to track any modifications on some specific files and after a simple search I found the system call inotify. For the requirements of the project inotify was not enough, yes it is going to catch any modifications on the files but I also needed to know in which offset of the file the changes have been written and I found no way to do that with inotify, thus I decided to use FUSE where I could implement my own “write” function therefore I could track any changes on the files in question and easily get the offset of changes which was the critical information in the main part of the project.

Let’s Start!

First you need to make sure that FUSE is installed in your Linux box. Note that I’m using version 2.9.4 here, so it might be some simple differences if you’re using a newer version but the main concepts will be probably the same. FUSE has a structure called “fuse_operations” with the following definition [3]:

struct fuse_operations {
	int (*getattr) (const char *, struct stat *);
	int (*readlink) (const char *, char *, size_t);
	int (*getdir) (const char *, fuse_dirh_t, fuse_dirfil_t);
	int (*mknod) (const char *, mode_t, dev_t);
	int (*mkdir) (const char *, mode_t);
	int (*unlink) (const char *);
	int (*rmdir) (const char *);
	int (*symlink) (const char *, const char *);
	int (*rename) (const char *, const char *);
	int (*link) (const char *, const char *);
	int (*chmod) (const char *, mode_t);
	int (*chown) (const char *, uid_t, gid_t);
	int (*truncate) (const char *, off_t);
	int (*utime) (const char *, struct utimbuf *);
	int (*open) (const char *, struct fuse_file_info *);
	int (*read) (const char *, char *, size_t, off_t,
		     struct fuse_file_info *);
	int (*write) (const char *, const char *, size_t, off_t,
		      struct fuse_file_info *);
	int (*statfs) (const char *, struct statvfs *);
	int (*flush) (const char *, struct fuse_file_info *);
	int (*release) (const char *, struct fuse_file_info *);
	int (*fsync) (const char *, int, struct fuse_file_info *);
	int (*setxattr) (const char *, const char *, const char *, size_t, int);
	int (*getxattr) (const char *, const char *, char *, size_t);
	int (*listxattr) (const char *, char *, size_t);
	int (*removexattr) (const char *, const char *);
	int (*opendir) (const char *, struct fuse_file_info *);
	int (*readdir) (const char *, void *, fuse_fill_dir_t, off_t,
			struct fuse_file_info *);
	int (*releasedir) (const char *, struct fuse_file_info *);
	int (*fsyncdir) (const char *, int, struct fuse_file_info *);
	void *(*init) (struct fuse_conn_info *conn);
	void (*destroy) (void *);
	int (*access) (const char *, int);
	int (*create) (const char *, mode_t, struct fuse_file_info *);
	int (*ftruncate) (const char *, off_t, struct fuse_file_info *);
	int (*fgetattr) (const char *, struct stat *, struct fuse_file_info *);
	int (*lock) (const char *, struct fuse_file_info *, int cmd,
		     struct flock *);
	int (*utimens) (const char *, const struct timespec tv[2]);
	int (*bmap) (const char *, size_t blocksize, uint64_t *idx);
	int (*ioctl) (const char *, int cmd, void *arg,
		      struct fuse_file_info *, unsigned int flags, void *data);
	int (*poll) (const char *, struct fuse_file_info *,
		     struct fuse_pollhandle *ph, unsigned *reventsp);
	int (*write_buf) (const char *, struct fuse_bufvec *buf, off_t off,
			  struct fuse_file_info *);
	int (*read_buf) (const char *, struct fuse_bufvec **bufp,
			 size_t size, off_t off, struct fuse_file_info *);
	int (*flock) (const char *, struct fuse_file_info *, int op);
	int (*fallocate) (const char *, int, off_t, off_t,
			  struct fuse_file_info *);
};

You can see that all the fields of this structure are pointers to functions. Each one of them will be called by FUSE when a specific event happens on the filesystem; for instance when the user writes on a file the function which is pointed by the field “write” in the structure will be called. If you take a quick look on the fields of the structure and try to figure out what event each field represents you’re going to find yourself familiar with many of those events. For example, obviously the function of “mkdir” entry will be called when the user tries to create a new directory, “unlink” will be called when the user tries to delete a file and so on.

To implement your filesystem you need to use this structure and you need to define the functions of this structure then to fill the structure with the pointers of your implemented functions. Most of the functions here are optional; you don’t need to implement them all; even though some of them are essential for a functional filesystem (e.g. getattr). We’re going here to examine the most important functions that must be implemented to write our example simple filesystem and these functions are getattr, readdir and read.

The function of getattr event will be called when the system tries to get the attributes of the file. The comment on FUSE library source code says [3]: “Similar to stat(). The ‘st_dev’ and ‘st_blksize’ fields are ignored. The ‘st_ino’ field is ignored except if the ‘use_ino’ mount option is given.”

If we read the manual page of stat() we can see that it receives two parameters, the first one is the path of the file which its attributes must be returned, the second one is a pointer to the structure which will contain the attributes after the call of the function finishes. In case of success this function must return 0.

If you need more information about the attributes themselves I recommend the following links Unix Stat Command: How To Identify File Attributes, Stat (system call) and The meaning of the File Attributes

The function of readdir event will be called when the user tries to show the files and directories that reside in a specific directory. And as we can see from the names, the function of read event will be called when the system tries to read a chunk of data from a file.

Implementing Our Functions

What’s Our Example Filesystem Going To Do?

Nothing special nor useful! it’s a read-only filesystem, there are two files on it; namely “file54” and “file349”. The user can open these files and read their contents. I’m going to call it Simple Stupid Filesystem (SSFS).

Implementing “getattr”

Well, let’s start. Now we are going to implement the function of the event getattr, it is essential to write a functional filesystem. Once again the function of this event is going to return important information about each file that resides in our filesystem by filling a structure of type stat. In sake of simplicity we’re going to cover the essential fields of this structure just to make our filesystem works. maybe there are some uncovered fields in this tutorial that you are interested in, so I recommend you to read the documentation of the structure “stat”.

Before we start the real job let’s define FUSE version macro and include the required files to write SSFS:

#define FUSE_USE_VERSION 30

#include <fuse.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <time.h>
#include <string.h>
#include <stdlib.h>

The function “do_getattr” is the one which will be called when the system asks SSFS for the attributes of a specific file. We, of course, start with the header of the function:

static int do_getattr( const char *path, struct stat *st )
{

We can see that it takes two parameters and returns an integer. The first parameter is the path of the file which the system asked SSFS for its attributes. The other parameter is the “stat” structure which must be filled with the attributes of that file. On success the return value must be 0, otherwise it is -1 and “errno” must be filled with the right error code.

Well, let’s fill the first bunch of stat’s fields.

	st->st_uid = getuid();
	st->st_gid = getgid();
	st->st_atime = time( NULL );
	st->st_mtime = time( NULL );

The field “st_uid” represents the owner of the file in question, In SSFS the owner of all files/directories is the same user who mounted the filesystem. Also for “st_gid” which represents the owner group of the files/directories, it will be the same group of the user who mounted the filesystem. We use the functions getuid and getgid to get the user id and group id of the current user (the user who mounted the filesystem).

The field “st_atime” represents the last access time for the file in question and “st_mtime” represents the last modification time of the file. We fill both of them with the time which the function has been called in (the current time) by using the function time. Please note that the value of these two fields must be a Unix time.

Now, let’s fill the rest of the fields:

	if ( strcmp( path, "/" ) == 0 )
	{
		st->st_mode = S_IFDIR | 0755;
		st->st_nlink = 2;
	}
	else
	{
		st->st_mode = S_IFREG | 0644;
		st->st_nlink = 1;
		st->st_size = 1024;
	}
		
	return 0;

As you can see from the last piece of code that we fill both “st_mode” and “st_nlink” fields with different information according to the path of the file in question. The field “st_mode” specifies if the file is a regular file, directory or other. In addition, “st_mode” specifies the permission bits of that file. The field “st_nlink” specifies the number of hardlinks. Finally, the field “st_size” specifies the size of that file in bytes.

Back to the previous code we can see that the first block of the if-statement is going to execute when the file in question is the root directory of SSFS. In this case we fill “st_mode” with the macro S_IFDIR which indicates that the file in question is a directory, then we set the permission bits as the following: only the owner of the file could read, write and execute the directory, the group’s users and other users could only read and execute the directory, If you don’t know what I’m talking about you could read “Understanding Linux File Permissions”. After filling “st_mode” for the root directory we specify the number of hardlinks, you might wonder why 2 hardlinks not just 1, you can find the answer here.

The block of else will be executed for all files other than the root directory, in SSFS namely: “file54” and “file349”. In “st_mode” for those files we use the macro S_IFREG to indicate that they are just regular files, then we specify the permission bits as the following: the owner could read and write the file, the group’s users and other users could only read the file. After that we set the number of hardlinks as 1. And finally we specify the size of the files as 1024 bytes.

Now, we got a simple & stupid implementation of getattr, the whole code with comments and debug messages:

static int do_getattr( const char *path, struct stat *st )
{
	printf( "[getattr] Called\n" );
	printf( "\tAttributes of %s requested\n", path );
	
	// GNU's definitions of the attributes (http://www.gnu.org/software/libc/manual/html_node/Attribute-Meanings.html):
	// 		st_uid: 	The user ID of the file’s owner.
	//		st_gid: 	The group ID of the file.
	//		st_atime: 	This is the last access time for the file.
	//		st_mtime: 	This is the time of the last modification to the contents of the file.
	//		st_mode: 	Specifies the mode of the file. This includes file type information (see Testing File Type) and the file permission bits (see Permission Bits).
	//		st_nlink: 	The number of hard links to the file. This count keeps track of how many directories have entries for this file. If the count is ever decremented to zero, then the file itself is discarded as soon 
	//						as no process still holds it open. Symbolic links are not counted in the total.
	//		st_size:	This specifies the size of a regular file in bytes. For files that are really devices this field isn’t usually meaningful. For symbolic links this specifies the length of the file name the link refers to.
	
	st->st_uid = getuid(); // The owner of the file/directory is the user who mounted the filesystem
	st->st_gid = getgid(); // The group of the file/directory is the same as the group of the user who mounted the filesystem
	st->st_atime = time( NULL ); // The last "a"ccess of the file/directory is right now
	st->st_mtime = time( NULL ); // The last "m"odification of the file/directory is right now
	
	if ( strcmp( path, "/" ) == 0 )
	{
		st->st_mode = S_IFDIR | 0755;
		st->st_nlink = 2; // Why "two" hardlinks instead of "one"? The answer is here: http://unix.stackexchange.com/a/101536
	}
	else
	{
		st->st_mode = S_IFREG | 0644;
		st->st_nlink = 1;
		st->st_size = 1024;
	}
	
	return 0;
}

Implementing “readdir”

In readdir we could list the files/directories which are available inside a specific directory. In SSFS there is only one directory (the root directory). Let’s start with the header of do_readdir function:

static int do_readdir( const char *path, void *buffer, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi )
{

As you can see, do_readdir has five parameters, here; we are interested in the first three of them. The first parameter “path” is the path of the directory in question, that is, the directory which the system requested the list of files that reside under it. In the second parameter “buffer” we are going to fill the names of the files/directories which are available inside the directory in question. The third parameter “filler” is a function sent by FUSE and we could use it to fill the “buffer” with available files in “path”. The function returns 0 on success.

The declaration of “filler” is the following [3]:

typedef int (*fuse_fill_dir_t) (void *buf, const char *name,
				const struct stat *stbuf, off_t off);

The first parameter is a pointer to the buffer which we want to write the entry (filename or directory name) on. The second parameter is the name of the current entry. The third and the fourth parameters will not be covered here.

	filler( buffer, ".", NULL, 0 );
	filler( buffer, "..", NULL, 0 );

We filled the list of available entries in “path” with two entries: “.” which represents the current directory, while “..” represents the parent directory. It’s a known convention in Unix world. Let’s continue:

	if ( strcmp( path, "/" ) == 0 )
	{
		filler( buffer, "file54", NULL, 0 );
		filler( buffer, "file349", NULL, 0 );
	}
	
	return 0;

In case the directory in question is the root directory, we are going to add entries for the files: “file54” and “file349”. And that’s it!

The code of “do_readdir”:

static int do_readdir( const char *path, void *buffer, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi )
{
	printf( "--> Getting The List of Files of %s\n", path );
	
	filler( buffer, ".", NULL, 0 ); // Current Directory
	filler( buffer, "..", NULL, 0 ); // Parent Directory
	
	if ( strcmp( path, "/" ) == 0 ) // If the user is trying to show the files/directories of the root directory show the following
	{
		filler( buffer, "file54", NULL, 0 );
		filler( buffer, "file349", NULL, 0 );
	}
	
	return 0;
}

Implementing “read”

The last function we are going to implement is “read”. As we know, through this function the system could read the content of a specific file. Let’s start with the header:

static int do_read( const char *path, char *buffer, size_t size, off_t offset, struct fuse_file_info *fi )
{

The first parameter “path” is the path of the file which the system wants to read. In the second parameter “buffer” we are going to store the chunk which the system interested in, the third parameter “size” represents the size of this chunk and the fourth parameter “offset” is the place in the file’s content where we are going to start reading from. “do_read” must return the number of the bytes that have been read successfully.

For the clarity I’m going to give a more detailed example about the parameters “size” and “offset”. Let’s assume that we have a file with the following content: “Filesystem in Userspace (FUSE) is a software interface for Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code. This is achieved by running file system code in user space while the FUSE module provides only a “bridge” to the actual kernel interfaces.”.

If size = 35 and offset = 0, we are going to read the content from the first character, because offset = 0, and we are going to read the first 35 characters, so the content of “buffer” will be the following: “Filesystem in Userspace (FUSE) is a”.

Another example, let’s assume that size = 35 but the offset = 40, so we are going to skip the first 41 characters, because offset = 40, and start reading from the character 42 to 77, because the size = 35, the value of the “buffer” will be “ware interface for Unix-like comput”. Well, let’s start by defining some local variables:

	char file54Text[] = "Hello World From File54!";
	char file349Text[] = "Hello World From File349!";
	char *selectedText = NULL;

The first string is the content of “file54” and the second is the content of “file349”. The job of “selectedText” will be shown in a moment:

	if ( strcmp( path, "/file54" ) == 0 )
		selectedText = file54Text;
	else if ( strcmp( path, "/file349" ) == 0 )
		selectedText = file349Text;
	else
		return -1;

If the file in question is “file54” then we are going to assign the pointer of “file54” content to the “selectedText” which is obviously represents the content which will be returned to the system. The same is going to happen with “file349”. In case that the system requested to read another file an error will be sent because SSFS only has those to files.

	memcpy( buffer, selectedText + offset, size );
		
	return strlen( selectedText ) - offset;
}

In the last part of “do_read”, we are going to copy - into “buffer” - the content of the file in question using memcpy starting from the “offset” until reaching “size”. Then we return the number of bytes that have been read.

Filling “fuse_operations” & Telling FUSE About It

Now simply we fill “fuse_operations” structure and call the main function of FUSE which is going to run our filesystem:

static struct fuse_operations operations = {
    .getattr	= do_getattr,
    .readdir	= do_readdir,
    .read	= do_read,
};

int main( int argc, char *argv[] )
{
	return fuse_main( argc, argv, &operations, NULL );
}

And we got our first filesystem! :-)

Compiling & Mounting The Filesystem

You can use GCC to compile SSFS as the following:

gcc ssfs.c -o ssfs `pkg-config fuse --cflags --libs`

The “pkg-config” part is going to provide the compiler the proper arguments to include “fuse” library. To mount the filesystem after compilation:

./ssfs -f [mount point]

Using the option “-f” will let you see the debug messages which are printed using “printf”.

Source Code

You can find the source code of SSFS in my GitHub account here: https://github.com/MaaSTaaR/SSFS.

Extending SSFS

The second part of this tutorial which extends SSFS to create Less Simple, Yet Stupid Filesystem (LSYSFS) can be found here.

References

[1] FUSE in Wikipedia.

[2] The Source Code of “ext4fs”.

[3] The Source Code of “libfuse 2.9.4”.

Writing Telegram Bot with PHP using Guzzle

Sat, 16 Jan 2016 00:00:00 +0300

Several months ago Telegram started to support bots which is, in my opinion, really an interesting feature for a chat application to come with. Anyone could create his/her own bot and of course to make the bot functional a code should be written for it :-). In this post I’m going to explain how to write a simple Telegram bot with PHP using Guzzle HTTP Client. Your bot’s code must reside on your own server which must support HTTP connections and obviously PHP scripts; a shared host is enough but you will need a SSL/TLS if you’re planning to use web hook for you bot, more details about web hook will be presented in this post. You can read the official guide if you don’t know what exactly bots are.

Creating a new Bot

As a first step you’ll need to create a new bot which is a really easy step. There is an official bot called BotFather; by using it you could create you own bot. Just add BotFather to your Telegram contact list by clicking here then follow the instructions. It’s probably easier to use Telegram Desktop Client while you’re developing your bot.

After adding BotFather to your contact list use the command “/newbot” to create your new bot. BotFather will ask you few questions about the name of your new bot and after answering the questions the new bot will be created and a token for Web API of your bot will be provided to you. Keep the token in a secure place we will need it later when we start to write bot’s code.

Start Coding

Telegram provides us a HTTP API to program our bots and instead of using curl to consume those APIs we’re going to use Guzzle here. The installation guide of Guzzle can be found here. I assume that you will use Composer to install Guzzle so the path of Guzzle’s files in our examples will start with the directory “vendor”.

Initializing the HTTP Client

I assume that you installed Guzzle successfully in a directory which is dedicated to our bot’s code; Let’s create a new php file in this directory and call it “bot.php”. The first thing that we need to do is including the file autoload.php from Guzzle package; after that we’re going to use the keyword “use” to get rid of Guzzle’s namespace as the following:

require_once( 'vendor/autoload.php' );
use GuzzleHttp\Client;

Guzzle has a class called Client which helps us to open a connection with HTTP API’s URL and send requests to the server and receive the responses. Our bot URL has the following form: https://api.telegram.org/bot[Bot’s Token]/[To be called Method]. For the start we don’t care about the method’s part of the URL; we only need the first part which involves the bot’s token; by using it we could initialize the HTTP Client’s object.

$apiKey = ''; // Put your bot's API key here
$apiURL = 'https://api.telegram.org/bot' . $apiKey . '/';

$client = new Client( array( 'base_uri' => $apiURL ) );

In the first two lines we construct the API’s URL for our newly-created bot; then we create a new object of the class Client which is provided to use by Guzzle; in this object there are all needed methods to communicate with Telegram’s servers and consume the APIs. You can see that the constructor of class Client requires one parameter which is an array; this array has an element with the key “base_uri” which is the URL of the HTTP APIs. Well, we now initialized the HTTP client which resides in the object $client. Let’s start with the real work :-).

Getting Updates

(AKA: Is there anything new?)

Well, what we want to do now is getting the updates. When anybody sends a new message to our bot we call this message an “update”. There are two ways to get updates, the first one by using Telegram’s API method getUpdates. By using this way we need to manually refresh the page - which has the php code that handles the updates - which is of course not suitable way for production but it is a good way to be used in the development phase.

The other way is using the WebHook, in this way we write our code which handles the updates and set the page which has this code as a WebHook for our bot - by using the Telegram’s method setWebHook -, When our bot receives any new update, Telegram’s server is going to call the page which has been set as a WebHook instantly so the bot will response automatically to the updates.

For now we are going to talk about the first way - getUpdates - and later we are going to modify our code to be suitable to be used as WebHook.

getUpdates() method

Client class of Guzzle provides us the method “get” which sends a GET request to the URL which already initialized and because the Telegram’s method “getUpdates” is a GET method so we are going to use Guzzle’s “get”. We already initialized the client’s object and we stored this object in a variable called $client.

$response = $client->get( 'getUpdates' );
$updates = json_decode( $response->getBody() );

In the first line we call the method getUpdates which provides us the list of new messages. The method “get” returns an object which represents server’s response and if everything goes smoothly in the second line we’re going to get the body of the response which is a JSON that contains the list of all new messages, and by using PHP’s function json_decode we convert JSON to a PHP object which will be stored in “$updates”.

If you try to send a text message to your bot using Telegram then try to print the object $updates using var_dump( $updates ) you will get something like that:

object(stdClass)[32]
  public 'ok' => boolean true
  public 'result' =>
    array (size=1)
      0 =>
        object(stdClass)[30]
          public 'update_id' => int 475557067
          public 'message' =>
            object(stdClass)[27]
              ...

The format of var_dump may be shown differently with you and that’s because I use Xdebug in my development environment but anyway we can see from var_dump’s result that $updates object has two fields; first one has the name “ok” which is a boolean that shows if everything is fine or not, the second one which is more important has the name “result” which is an array of the new messages which have been sent to our bot. Each entry in this array represents an Update. We want to traverse through the list of results:

foreach ( $updates->result as $update )
{
	// Our next code will be here
}

At each iteration of foreach we will have an object “$update” which represents one update, and there are two fields in this object, the first one is “$update->update_id” which contains the identifier number of the current update, the second one which is more interesting is “$update->message” which is an object of Message and obviously this object represents the message which has been sent to our bot. It has many interesting fields which are all decribed on the official guide. Take a look here.

Sending Messages

(AKA: Responding to Updates)

Now let’s see how to let our bot to send messages to the users who contacted it. As an example we want our bot to send the message “Welcome to MQH Blog’s Bot” when the user sends “Hello” to it. In the previous code the object $update has the field “message” which, as we said previously, represents the message which has been sent by the user; the message’s object itself has many useful fields, but in our situation we’re interested in two of them. The first one is “text” and the second one is “chat”.

Obviously the field “text” contains the text of the message which has been sent by the user. The first step in our example is checking whether the user sent the message “Hello” to our bot and we can do that simply as the following (Note that the following code must be inside foreach’s block of the previous code which we already wrote in this post):

if ( $update->message->text == 'Hello' )
{
	echo 'Hello has been received';

	// Response's code will be here
}

Well, the second step is sending a response to the user and to do that we must use Telegram’s API method sendMessage which requires two parameters; the first one is “chat_id” which is a unique key that represents the user who we would like to send the message to. In our case the value of “chat_id” must be the id of the user who sent the message “Hello” to our bot and we could grab his/her id by using the field “chat” of the object “message”, “chat” itself is an object which has a field called “id” and that’s exactly the id which we need. The second parameter which is required by the method “sendMessage” is “text” which is the message we would like to send to the user who has been specified in the first parameter:

$client->post( 'sendMessage', array( 'query' => array( 'chat_id' => $update->message->chat->id, 'text' => "Welcome to MQH Blog's Bot" ) ) );

First of all, “sendMessage” is a POST method and because of that we used Guzzle’s method “post”. The first parameter is the name of the Web API method which we want to call. The second parameter is an array of one element that has the key “query” which is an array that contains the parameters that is required by the method “sendMessage”. You can see that we sent the two parameters “chat_id” and “text” to the method “sendMessage”.

The complete code until now:

<?php

require_once( 'vendor/autoload.php' );
use GuzzleHttp\Client;

$apiKey = ''; // Put your bot's API key here
$apiURL = 'https://api.telegram.org/bot' . $apiKey . '/';

$client = new Client( array( 'base_uri' => $apiURL ) );

$response = $client->get( 'getUpdates' );
$updates = json_decode( $response->getBody() );

foreach ( $updates->result as $update )
{
	if ( $update->message->text == 'Hello' )
	{
		echo 'Hello has been recieved';

		$client->post( 'sendMessage', array( 'query' => array( 'chat_id' => $update->message->chat->id, 'text' => "Welcome to MQH Blog's Bot" ) ) );
	}
}

?>

Responding Automatically with WebHook

Using WebHook we could set the page - which handles the new updates - to our bot and Telegram’s servers are going to call this page whenever a user sends a message to our bot. You can observe that the previous code need to be executed manually each time to let our bot to response to new “Hello” messages. With WebHook there is no need to execute the code manually; Telegram’s servers is going to call the code with every new message. The code will be the same with slight changes.

The first step is to use setWebHook to set the URL of the PHP page as a web hook for our bot. Note that the URL must starts with HTTPS which means TLS/SSL must be enabled in your server.

Let’s create a new PHP file and call it “SetWebHook.php”. In this file we’re going to call setWebHook which is a simple method that takes the URL of update’s handler page - the web hook - as a parameter.

<?php

require_once( 'vendor/autoload.php' );
use GuzzleHttp\Client;

$apiKey = ''; // Put your bot's API key here
$apiURL = 'https://api.telegram.org/bot' . $apiKey . '/';

$client = new Client( array( 'base_uri' => $apiURL ) );

$webHookURL = ''; // Your Web Hook's URL
$response = $client->post( 'setWebhook', array( 'query' => array( 'url' => $webHookURL ) ) );

?>

There is nothing new here. We created our HTTP Client object “$client” then we used it to call the POST method “setWebHook”. The variable “$webHookURL” must be filled with the URL of our bot’s web hook. Now let’s create a new PHP file and call it “WebHook.php” which will be the web hook of our example bot. The coming code will be same as the code from previous part with some modifications. First we include Guzzle and create the HTTP Client object as we did previously:

<?php

require_once( 'vendor/autoload.php' );
use GuzzleHttp\Client;

$apiKey = ''; // Put your bot's API key here
$apiURL = 'https://api.telegram.org/bot' . $apiKey . '/';

$client = new Client( array( 'base_uri' => $apiURL ) );

We mentioned many times earlier that Telegram’s servers are going to call our bot’s web hook when a new message recieved. And that will be done of course be sending a HTTP request (POST) to your server. The body of the request contains a Update object of the new message.

$update = json_decode( file_get_contents( 'php://input' ) );

Now we have Update object in the variable “$update” and we already dealt with this object in the previous part. No we can examine the content of the recieved message using “$update” as we did previously.

if ( $update->message->text == 'Hello' )
		$client->post( 'sendMessage', array( 'query' => array( 'chat_id' => $update->message->chat->id, 'text' => "Welcome to MQH Blog's Bot" ) ) );

The complete code of WebHook:

<?php

require_once( 'vendor/autoload.php' );
use GuzzleHttp\Client;

$apiKey = ''; // Put your bot's API key here
$apiURL = 'https://api.telegram.org/bot' . $apiKey . '/';

$client = new Client( array( 'base_uri' => $apiURL ) );

$update = json_decode( file_get_contents( 'php://input' ) );

if ( $update->message->text == 'Hello' )
		$client->post( 'sendMessage', array( 'query' => array( 'chat_id' => $update->message->chat->id, 'text' => "Welcome to MQH Blog's Bot" ) ) );

?>

Three Days with Java: Day 2

Fri, 28 Aug 2015 21:30:00 +0300

Here we continue what we have started in the previous post “Three Days with Java: Day 1”.

The Initial Position of Swing Application Window

Because I already wrote applications for GTK I have the habit of showing the main window of the application when the user starts it in a position which is close to the mouse pointer and I wanted to implement that with Swing.

We could do that by knowing the current position of the mouse then passing this position to the method which sets the position of the window. The class MoustInfo provides us a method called “getLocation”; by using it we could get the current location of the mouse pointer as the following:

MouseInfo.getPointerInfo().getLocation()

After that we could pass this location to the method setLocation which is available in our window’s object (which should be of type JFrame). Example:

JFrame mainWin = new JFrame( "Three Days with Java" );

// ... //

mainWin.setSize( 600, 600 );
mainWin.setDefaultCloseOperation( JFrame.EXIT_ON_CLOSE );

// Shows the window near to the mouse cursor
mainWin.setLocation( MouseInfo.getPointerInfo().getLocation() );

Right-to-Left Languages and Swing

If you application’s interface has been written in right-to-left language such as Arabic it’s probably a good idea to change the direction of the text for users’ convenience. In Swing there is a method called “setComponentOrientation” which is available in the class Component to do that. Let’s see the following example:

JPanel paneName = new JPanel();
paneName.setComponentOrientation( ComponentOrientation.RIGHT_TO_LEFT );
paneName.setBorder( BorderFactory.createTitledBorder( "Right-to-Left Text" ));

Note the second line, we passed a parameter which indicate the that JPanel which we just created has right-to-left direction; due this line the JPanel itself and the text “Right-to-Left Text” will be shown from right to left. If we remove the second line the direction will back to the default one.

Antoher example:

JTextField textName = new JTextField( 25 );
textName.setComponentOrientation( ComponentOrientation.RIGHT_TO_LEFT );

In this example we set the direction of a text field using the same method setComponentOrientation; when the user write some input in this text field the direction of the input will be from right to left and that will provide better user experience if he/she is going to write a text in a right-to-left language.

Swing’s JList

Let’s now discuss some classes which provided by Swing. I’ll start with JList which provides us a way to display a list for the user. In my application I used it to display a list of options for the user so he/she could select an item from the list and according to the selected item the application decides how to insert the data to database.

The use of JList is too simple. You just need to build an array of String which has the available items then pass this array to the constructor of JList. Example:

JList UnitsList = new JList( new String[] { "a", "b", "c", "d", "e" } );

win.add( UnitsList );

First we created an object of JList - which is stored in the variable UnitsList - and we passed an array of String which contains items from “a” to “e” as a parameter to the constructor of JList. After that we added this list to the window - which is represented by the object “win” and the example assumes that the type of this object is JFrame -.

To get the value which has been selected by the user we could use the method getSelectedValue which is provided by the class JList; it returns the value itself and not the index of the selected value. We could also use the method setSelectedIndex to indicate that some item on the list has been selected; we could use it for instance to automatically select the default item from the list then the user has the freedom to change the default selected value. Let’s assume that we want to set the item “c” in our previous example as the default item when the user runs the window which has our JList. The following line does that:

UnitsList.setSelectedIndex( 2 );

You can see that we passed 2 as a parameter; this value represents the index of the item; the indices in JList starts with 0 which means that the item “c” in our list has the index 2 and not 3.

Swing’s JScrollPane

In the JList’s part of this post we built a list with a specific number of items and when we put it in our window we knew exactly what is the best height to be set for our list to show it perfectly in the UI. Let’s assume that we need to build a dynamic list; that is we know nothing about the number of items which reside inside it. Because the height of our list in the UI must be static and our list’s items is dynamic; at some point the number of items will be big enough to not fit in the UI due the static height which decided statically. To handle this issue we need to set a scrollbar for our list in the UI and after that we could set a static height value for the list and add whatever number of items in it without worrying if all items will be shown for the user or not. Once the number of items be too large to be fit in the specified height the scrollbar will be enabled and the user could use it to navigate all available items in the list.

The class JScrollPane can be used to add scrollbars for Swing’s components. It’s simple to use; what we should do to use it is creating a new object of it and passing the object of the component - which we which to enable the scrollbar for it - as a parameter to the constructor then adding the object of JScrollPane in the interface - instead of the original component itself -. Examples are always better :-)

JFrame win = new JFrame( "Three Days with Java" );

win.setSize( 500, 200 );
win.setDefaultCloseOperation( JFrame.EXIT_ON_CLOSE );

// ... //

JList alpha = new JList( new String[] { "a", "b", "c", "d", "e", "h", "i",
				"j", "k", "l", "m", "n", "o", "p",
				"q", "r", "s", "t", "u", "v", "w",
				"x", "y", "z" } );

JScrollPane scrollPane = new JScrollPane( alpha );

win.add( scrollPane );

// ... //

win.setVisible( true );

If you run this example you will see that the window is too small to fit the whole items of the list; and due that a scrollbar will be shown on the list so you could navigate all items. You can run the following example which is exactly same as the current but doesn’t use JScrollPane:

JFrame win = new JFrame( "Three Days with Javaا" );

win.setSize( 500, 200 );
win.setDefaultCloseOperation( JFrame.EXIT_ON_CLOSE );

// ... //

JList alpha = new JList( new String[] { "a", "b", "c", "d", "e", "h", "i",
				"j", "k", "l", "m", "n", "o", "p",
				"q", "r", "s", "t", "u", "v", "w",
				"x", "y", "z" } );

win.add( alpha );

// ... //

win.setVisible( true );

And you tell me the difference :-)

JScrollPane can be used with any other Swing’s component and not only with JList. Let’s see the following example which uses JScrollPane with a text area:

JFrame win = new JFrame( "Three Days with Java" );

win.setSize( 200, 100 );
win.setDefaultCloseOperation( JFrame.EXIT_ON_CLOSE );

// ... //

JTextArea txt = new JTextArea();

JScrollPane scrollPane = new JScrollPane( txt );

win.add( scrollPane );

// ... //

win.setVisible( true );

Run the example and try to write text with multiple lines; at some point the scrollbar will be enabled to scroll you lines. In the following example the use of JScrollPane has been eliminated:

JFrame win = new JFrame( "Three Days with Java" );

win.setSize( 200, 100 );
win.setDefaultCloseOperation( JFrame.EXIT_ON_CLOSE );

// ... //

JTextArea txt = new JTextArea();

win.add( txt );

// ... //

win.setVisible( true );

If you run it and start to write multiple line text on the text area at some point the old lines will start to disappear and the only way to back to them again is by using the keyboard.

Swing’s JOptionPane

Sometime you need to show dialogs for the user and for that you could use JOptionPane which provides a way to show the most common types of dialog boxes.

Here I’m going to discuss the static method JOptionPane.showConfirmDialog and there are more such as showInputDialog, showMessageDialog and showOptionDialog; each method of them has a descriptive name and you maybe need to use a method other than “showConfirmDialog” for your application; you could read the official guide for more information.

The method JOptionPane.showConfirmDialog can be used to show a confirmation dialog for the user. Example:

int msg = JOptionPane.showConfirmDialog( null,
										"This identifier is already stored in database. Do you really want to update its information?",
										"Warning", JOptionPane.YES_NO_OPTION );

It’s obvious that the second parameter is the message which will be shown for the user. The third one is the title of the dialog box and the last one decides what are the buttons that will be shown for the user in this dialog box and in our case we would like to show two buttons: “Yes” and “No”.

The variable “msg” contains the response of the user which could be handled as the following:

if ( msg == JOptionPane.YES_OPTION )
	update = true;
else
	no_clicked = true;

The End. In case you have missed the first part of this post you can find it here. See you in the next part :-)

Three Days with Java: Day 1

Sat, 22 Aug 2015 23:15:00 +0300

Three years ago I started with my first project in Java. It was a desktop application which depends on database to store its data and retrieve them according to some business rules. Through the development of this project and because I worked a little bit with Java before, I had many questions of the kind “How to do X in Java?” of course I needed to do some researches to find the answers; that was easy due the simplicity of my questions :-). Eventually I summarized these questions and their answers in three parts post and we’re in the first part. I assume here that you already know Java and SQL for the part of database in this post.

Java and Database

Java has a standard API which is called Java Database Connectivity (JDBC) and it is used to access the database regardless of the database management system which is used in the backend.

As a first step you need to choose which database management system (DBMS) you’re going to use in your project, of course there are many options which are supported on JDBC from SQLite to Oracle. After deciding the what is your DBMS you need to find a JDBC driver for it by using your favorite search engine. Usually the JDBC driver is a JAR file (Java Library) which should be included in your project to use the driver in your code.

For my project I chose SQLite as a DBMS because, you know, it is simple and the project actually didn’t need a large-server-client-architecture-based DBMS. After a small search I found this driver. After downloading the JAR file I needed to include it in my project and that was a simple step with Eclipse. I just right-clicked my project and from “Build Path” sub-menu I selected “Add External Archives” and added the JAR file to my project which is after that has been shown in “Referenced Libraries”.

Loading the Driver and Connecting to the Database

That was the initial step of working with database on Java. You just need to find the right JDBC driver and include it in your project. Let’s move to the coding part, first you need to import the package java.sql in your source code then you could start with the following code.

try {
	Class.forName( "org.sqlite.JDBC" );
} catch (ClassNotFoundException e) { }

Well, what does this code do? simply it loads the JDBC driver. As you can see we used the static method Class.forName here to load the driver; I’m going to talk about this method later in another section of this post. However; the method forName receives one parameter and the value of it should be found in the documentation of the JDBC driver which you depends on. In my case the value of SQLite’s JDBC driver was the one which is presented in the previous bunch of cde.

After loading the JDBC driver using Class.forName method we could start using the database in our code and obviously the first thing we want to do is connecting to the database:

try {
	Connection con = DriverManager.getConnection("jdbc:sqlite:db_file.db");
} catch (SQLException e) { e.printStackTrace(); }

In this code we used the class DriverManager which manages the drivers of JDBC. The method getConnection tries to connect to the database and in case of success it returns an object of Connection. The parameter which we passed to this method is the database URL which usually comes in the following format: jdbc:subprotocol:subname and this parameter also is a driver-specific. In our example you could see that the last part of the URL is the database name.

Let’s talk a little bit about the object which will be returned from the method DriverManager.getConnection; this object represents a session of the database; by using this object we could manipulate our database using SQL statements.

Preparing the SQL Statement

Now, we loaded the driver; got our connection to the database. The next logical step is manipulating our database. Inserting some data, retrieving them, updating them or deleting them; that could be done using SQL so we need a way to execute SQL statements. The object which had been returned by the method DriverManager.getConnection to do that which means that the interface Connection has the required methods to deal with SQL statements.

One of these methods is prepareStatement which has the following declaration - actually one of its declarations -:

PreparedStatement prepareStatement(String sql)

It requires one parameter which is the SQL statement and returns an object of type PreparedStatement.

As you know that SQL is just another programming language which is used as an interface for the databases. And we know that every code we write in a high-level language should be compiled to a language which the machine could understand and SQL is not an exception.

The job of the method prepareStatement is to compile the passed SQL statement and the result of the compilation process will be stored in an object of PreparedStatement which means that after using this method we got a compiled version of our SQL statement and we could use it several times with no need to recompile the original statement over and over again.

In addition the main purpose of the method prepareStatement is preparing the SQL statements which have parameters and we could say that it is the best way to prevent SQL Injections.

Retrieving Data

Let’s see an example:

try {
		PreparedStatement stat = con.prepareStatement( "SELECT * FROM employee WHERE id=?" );

		stat.setLong( 1, req_id );

		ResultSet res = stat.executeQuery();

		if ( res.next() )
		{
			String name = res.getString( "name" );
			String address = res.getString( "address" );
			int phone = res.getInt( "phone_number" );

			System.out.println( name + " " + address + " " + phone );
		}

		stat.close();

} catch (SQLException e) { System.out.println( e.getMessage() ); }

Let’s assume that we have a table called “employee” which has the column “id” as a primary key and we have a variable “req_id” which holds the id of the employee which we need to show his/her information. If you already know SQL so you can see nothing new in the SQL statement in the example but only the weird question mark. The question mark in this context tells JDBC that in this place we are going to pass a parameter and the value of this parameter must replace the question mark in this statement.

As we said the id of the employee which we want to show his/her information resides in the variable req_id; you can see in this line of the example we request from JDBC to change the question mark to the value which resides in “req_id”:

stat.setLong( 1, req_id );

You can observe that we used a method which is called setLong and we did that because the type of “req_id” is long. The name of the method should be changed according to the value which we need to pass for the SQL statement. The first parameter of setLong is the position of the question mark which we would like to replace in the SQL statement. Assume that we have a SQL statement which has two parameters - which mean two question marks - the first question mark shows in the SQL statement has position 1 and the second has position 2 and so on.

Well, why we use this way to pass the values in SQL statements? simply to get rid of SQL Injections (and maybe there are more advantages which I missed).

What we done until now is compiling the SQL statement; we need to execute the query to get the result of the query and that’s done by the following line:

ResultSet res = stat.executeQuery();

We can see that the method executeQuery returns an object of type ResultSet and here where query’s result resides. This object has a cursor which initially points on position before the first row of the result; we could use the method next to move this cursor one step forward. The method next returns true when there is a row otherwise it returns false.

To get the value of a column we could use one of the getters which are available in ResultSet as we can see in this part of the example:

if ( res.next() )
{
	String name = res.getString( "name" );
	String address = res.getString( "address" );
	int phone = res.getInt( "phone_number" );

	System.out.println( name + " " + address + " " + phone );
}

In the first line we check if there is any data fetched from the database and we move the cursor one step forward and in the block of the if statement we get the values of the row according to the table’s columns and stores these value in some local variables. You can see that the parameters of getter methods represent the name of the column on the database.

Inserting and Updating Data

Let’s now talk about inserting data. We can use the already explained prepareStatement way to prepare our insert statement as the following:

try
{
	PreparedStatement stat = null;

	stat = con.prepareStatement( "INSERT INTO employee VALUES( ?, ?, ? )" );

	stat.setString( 1, name );
	stat.setString( 2, address );
	stat.setInt( 3, phone );

	int effected_rows = stat.executeUpdate();

	stat.close();

	if ( effected_rows > 0 )
		System.out.println( "Inserted" );
	else
		System.out.println( "Error" );
} catch (SQLException e) { e.printStackTrace(); }

There are only two differences here from retrieving data. First we used the method executeUpdate instead of executeQuery and this method returns the number of effected rows and in our case it should be one and according to this value we could tell if the execution of the SQL statement succeed or failed.

To update data we could use the same way but we just need to change the SQL statement.

What exactly is Class.forName method?

(AKA: Welcome to “Reflection” World!)

We already used the method Class.forName in the previous part to load the database driver. In this part we are going to take a look at it. The method forName is a static method which resides in a class called Class; this method load classes dynamically at the run-time; that is while the application is running.

Java has a feature which is called “Reflection”; usually the method forName is used to take advantage of this feature. Simply “Reflection” allow the programmer to access to all information that related to a specific class at the run-time. For example we could use reflection to get the list of all class’s methods also we can call them at run-time.

Let’s see an example:

import java.lang.reflect.Method;

public class Main
{
	public static void main(String[] args)
	{
		String cName = args[ 0 ];

		try {
			Class<?> myClass = Class.forName( cName );

			Method[] methods = myClass.getDeclaredMethods();

			for( Method k: methods)
			{
				System.out.println( k.getName() );
			}
		} catch (ClassNotFoundException e) { }
	}
}

This example works as the following: It should be run from the command line and a class name should be passed as an argument; the class will be loaded at the run-time; that is the name of the class is unknown until the user runs the code. After loading the required class our code is going to print the name of all declared method in the class. Let’s assume that we have the following class:

public class RefTest
{
	public void one()
	{
		two();
	}

	private void two()
	{
		System.out.println( "Hello World!" );
	}
}

If we run the main example and pass the class name “RefTest” as an argument for it we could see it prints the name of the methods of “RefTest” and they are “one” and “two”. We could pass the name of the example’s class itself which is “Main” and it will print “main”.

As you can see in the main example that the method forName receives one parameter which is the name of the class we would like to load at run-time; it returns an object of type Class which represents a Java class and the object itself represents the class which we loaded it at run-time and in our case it is “RefTest”. There are many methods in “Class” which we could use to get any information about the loaded class.

We already used one of “Class” methods in our main example and this method is “getDeclaredMethods” which is obviously returns the list of declared methods in the loaded class. This list will be an array of type Method which is another class that represents a Java method. We used the method “getName” from class Method to get the name of loaded class’s methods.

How to reach application’s files?

In some cases you may include some files in you application and you would like to reach to these files to use them in you application’s code. These files could be images for instance. In my case the SQLite database file was included with the application itself and I needed a way to reach the location of my application’s directory in the user’s computer so I could reach the database file to establish the connection with it. The following snippet was the solution:

String path = new File( Main.class.getProtectionDomain().getCodeSource().getLocation().getPath() ).getParent();

Note that Main is the name of the class which has this line on it. In this way you could find the location of your application’s executable file so you could locate your files in the same directory of the executable file and also you may create sub-directories to organize your hierarchy.

How to call a method before closing the application?

In Swing applications that can be done by simply bind some callback to the window listener as the following:

JFrame mainWin = new JFrame( "Example" );

mainWin.addWindowListener( (WindowListener) this );

Then we could override the method “windowClosing” and write inside it the code which we would like to execute when the user close the application:

@Override
public void windowClosing(WindowEvent arg0)
{
	// Do something when the use close the main window ...
}

The other solution which is applicable when we don’t use Swing is the following:

Runtime.getRuntime().addShutdownHook(new Thread(new Runnable()
{
    public void run()
    {
		// Do something
    }
}));

Using this way when the application is closed a new thread will be started and the code which resides in the method “run” will be executed.

The Look & Feel of the Swing’s application

Swing has its own Look & Feel but you probbably would like to use the user’s operating system’s Look & Feel for your application. The following code does this trick:

try
{
	UIManager.setLookAndFeel( UIManager.getSystemLookAndFeelClassName() );
} catch (ClassNotFoundException e) {}
catch (InstantiationException e) {}
catch (IllegalAccessException e) {}
catch (UnsupportedLookAndFeelException e) {}

See you in the second part :-).

Welcome to MQH Blog!

Fri, 21 Aug 2015 20:55:00 +0300

Welcome to Mohammed Q. Hussain’s blog. I’m going to talk about programming stuff here :-). Enjoy!