Sensing...

Python code: Download shared file from google drive

2018-01-01T00:38:00.000-08:00

This python script downloads file from googledrive by using the shareable link of the file.

import requests
import sys

def download_file_from_google_drive(id, destination):
    URL = "https://docs.google.com/uc?export=download"

session = requests.Session()

response = session.get(URL, params = { 'id' : id }, stream = True)
    token = get_confirm_token(response)

if token:
        params = { 'id' : id, 'confirm' : token }
        response = session.get(URL, params = params, stream = True)

save_response_content(response, destination)

def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            return value

return None

def save_response_content(response, destination):
    CHUNK_SIZE = 32768

with open(destination, "wb") as f:
        for chunk in response.iter_content(CHUNK_SIZE):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)

if __name__ == "__main__":

# set default values if no input is given at acommandline
 outputFileName = "world13e.tif"
 googleFileID = "1H583z6qDwmrbb9_OYRL05EA2FKVqY8C0"

# READ INPUT FROM COMMANDLINE
 inputLine = sys.argv[1]

if len(sys.argv) == 2: # only google drive link given
  
  # set googleFileID from command line
  junk,googleFileID = inputLine.split('&id=') #https://docs.google.com/uc?export=download&id=1H583z6qDwmrbb9_OYRL05EA2FKVqY8C0
  
 elif len(sys.argv) > 2:
  junk,googleFileID = inputLine.split('&id=')
  outputFileName = sys.argv[2]
 
 download_file_from_google_drive(googleFileID, outputFileName)
 print 'Download Complete!'

# USAGE
# python download.py  'SharableLink of google drive file' ' outputFileName'
# python download.py 'https://docs.google.com/uc?export=download&id=1H583z6qDwmrbb9_OYRL05EA2FKVqY8C0' 'world1.tif'

However, a single command can also be used to download files from google drive using wget command as shown below

t-test with minimum number of samples

2017-12-29T02:20:00.000-08:00

There are two different ways to justify the use of the t-test.

Your data is normally distributed and you have at least two samples per group
You have large sample sizes in each group

If either of these cases hold, then the t-test is considered a valid test. So if you are willing to make the assumption that your data is normally distributed (which many researchers who collect small samples are), then you have nothing to worry about.

However, someone might reasonably object that you are relying on this assumption to get your results, especially if your data is known to be skewed. Then the question of sample size required for valid inference is a very reasonable one.

As for how large a sample size is required, unfortunately there's no real solid answer for that; the more skewed your data, the bigger the sample size required to make the approximation reasonable. 15-20 per group is usually considered reasonable large, but as with most rules of thumb, there exist counter examples: for example, in lottery ticket returns (where 1 in, say, 10,000,000 observations is an EXTREME outlier), you would literally need somewhere around 100,000,000 observations before these tests would be appropriate.

Source: stackexchange

Error mounting NTFS drive from Ubuntu

2014-07-24T19:51:00.003-07:00

I came across a new problem, at least for me, in ubuntu 14.04, while trying to mount the NTFS drives. It was just saying that I did not turned off windows system properly and therefore the filesystem is corrupt. I could mount the file system in the readonly mode.

sudo mount -t "ntfs" -ro "uhelper=udisks2,nodev,nosuid,uid=1000,gid=1000,dmask=0077,fmask=0177" "/dev/sda3" "/media/bidur/d"

For that, I have to create a new directory beforehand i.e. "/media/bidur/d".

Just now, I came across a way to deal with the problem by using ntfsfix commad:

sudo ntfsfix /dev/sda3

Python Pipe

2012-12-06T22:29:00.000-08:00

Pipes can be used in python using the popen command. os.popen(command[, mode[, bufferSize]]). The default mode is read which can also be explicitly indicated by "r" as shown in the example below. The parameter bufferSize can be 0 (no buffering), 1 (line buffering), negative(system default buffering) and positive values greater than 1 (number defines the buffer size to be used).

calculate hop count in a graph using python

2012-11-19T07:24:00.002-08:00

The following program finds out the hop count of the nodes in a connected network. Hop Count means the number of point to point connections in a graph of network. For example, in the graph shown below the hop count from node a to node b is one, node b to node c is two, etc.

This program assumes the graph given above.
usage: getHopCount(dictNeigh, node1,node2)
Where:
dictNeigh: dictionary of list of neighbor of each node in the network
node1 and node2 are the nodes for which hop count is calculated by the program
Exception: if the nodes are not connected then it returns Zero

import  copy

dictNeigh = dict() # dictNeigh defines the neighbors 
dictNeigh['a'] = ['b','c','d'] # neighbors of 'a'
dictNeigh['b'] = ['a']# neighbors of 'b'
dictNeigh['c'] = ['a','d']
dictNeigh['d'] = ['a','c','e']
dictNeigh['e'] = ['d','f']
dictNeigh['f'] = ['e','g','h']
dictNeigh['g'] = ['f']
dictNeigh['h'] = ['f']

def getHopCount(dictNeigh, node1, node2):
    
    hopCount = 0
    listNodes = copy.deepcopy(dictNeigh[node1])
    listNodeParsed = copy.deepcopy(dictNeigh[node1])
    listNodeParsed.append(node1) # put self node as parsed
    
    listPrevNodes = copy.deepcopy(dictNeigh[node1])

for i in range (0, len(dictNeigh.keys())):

if node2 in listNodes:
            hopCount = i +1
            return hopCount

else:            
            for node in listPrevNodes: 
            # copy all 1 hop neighbor of immediate neighbor             
                listNodes = copy.deepcopy(list( set(listNodes) | set(dictNeigh[node]) ))

listNodes = copy.deepcopy(list(set(listNodes) - set(listNodeParsed))) 
            # remove the nodes already seen
            listNodeParsed = copy.deepcopy(list(set(listNodeParsed) | set(listNodes)))
            listPrevNodes = copy.deepcopy(listNodes)       
        
    return 0

print 'Hop:' +str( getHopCount(dictNeigh, 'b','d'))

ubuntu (10.04) apache web server not running due to port conflict

2012-11-05T20:46:00.002-08:00

Sometimes the port assigned to apache2 may be used by some other services. In such cases, if we try to live apache2 server then it fails. The following message is shown in ubuntu 10.04 in my case:
Starting web server apache2
apache2: Could not reliably determine the server's fully qualified domain name, using 0.0.0.3 for ServerName (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down Unable to open logs

Here is one of the ways to overcome this problem:

1. see who is using the port 80 :

sudo lsof -i :80

2. get the pid of the services in 1. E.g.: 1577
3. kill all the services in 2. :

sudo kill 1577

4. restart apache2:

sudo /etc/init.d/apache2 restart

Display dynamic content in Google Earth using KML

2012-10-17T21:59:00.000-07:00

Keyhole Markup Language (KML) is a xml standard which can be used to define what and how to present geographic data in Google Earth and Google Maps. Useful examples to start can be browsed via https://developers.google.com/kml/documentation/kml_tut.
I have to display KML data in googleearth. And the KML file has to be loaded dynamically and the updates in the kml file should be shown in the googleearth on the fly.
For this we have to Create a Network Link.
Goto Add -> Network Link

Then specify the details as shown in the figure.
To specify the frequency of updating the data, go to Refresh tab.

The code for dynamicKML.kml is as follows:

Note that, we are using a local file junk.kml (junk.kml and dynamciKML.kml reside in the same directory) to define the content to be displayed in the google earth interface. This can even be a remote file in which case we have to specify the remote file URL. The code for junk.kml is as follows.

Now, we can change the content of junk.kml and the changes will be seen in the google earth interface dynamically.

python code for querying own IP address

2012-10-12T01:37:00.003-07:00

The socket.gethostname() function is supposed to give the IP address of the machine but it may not give the real IP address but gives 127.0.0.1. Even it is said that removing 127.0.0.1 from the /etc/hosts file (in ubuntu) can give the real IP but it could not help in my case.

import socket

nodeName = socket.gethostname()
info = socket.getaddrinfo(nodeName, None, 0, socket.SOCK_STREAM)
print [x[4][0] for x in info] # prints 127.0.0.1

I came through another way which can print the real IP as well:

import commands
NodeIPS = commands.getoutput("/sbin/ifconfig | grep -i \"inet\" | grep -iv \"inet6\" | " +
"awk {'print $2'} | sed -ne 's/addr\:/ /p'")
print NodeIPS # prints 127.0.0.1 and realIP(eg. 202.145.12.22)

Note: code worked for ubuntu 12.04 and python 2.7

Sometimes its handy to have the IP address as list items. The following code does the work.

import os
listIPs=[]
f=os.popen("/sbin/ifconfig | grep -i \"inet\" | grep -iv \"inet6\" | " + "awk {'print $2'} | sed -ne 's/addr\:/ /p'")
for i in f.readlines():
listIPs.append(i.rstrip('\n'))
print listIPs

Open file with unlisted application in ubuntu 12.04

2012-10-11T21:52:00.000-07:00

In ubuntu 12.04,
While browsing files in nautilus, if we want to open file with unlisted applications, for e.g. open a file with idle(but idle is not listed in the list of open with ). Enter the following command in the terminal:

gksudo gedit /usr/share/applications/idle.desktop

Then remove the line:

Exec=/usr/bin/idle -n

Replace it with:

Exec=idle %U

Now we can see idle listed in the list of open with applications.

To set it as default application to open a file.
Go to the file properites, select idle and click set as default.

Metaheuristics

2012-07-26T19:44:00.000-07:00

Generally speaking heuristics aim to find a good enough solution to a problem in a reasonable amount of time. These are mostly based on experiences and common sense. Metaheutistics iteratively tries to improve a candidate solution with respect to a given measure of quality.They rarely make assumptions about the problem being optimized They can search very large spaces of candidate solutions. Whereas they do not guarantee the finding of an optimal solution. Some of the popularly used metaheuristics for optimization problems are: • Simulated Annealing (SA), • Particle Swarm Optimization (PSO), • Differential Evolution (DE), • Genetic Algorithm (GA), • Evolutionary Strategy (ES).

secure shell (ssh)

2012-07-23T10:43:00.000-07:00

Two machines running a server and a client running respectively can be connected securely via ssh.

ssh user@serverIP

example:

ssh tom@203.159.11.42

Every time a user logs in to the server user have to provide password for getting into the server.Whereas passwordless authentication using the public key is considered more secure while loggin to a remote server. For that the following steps should be followed:

1. Generate public/private rsa key pair:

ssh-keygen -t rsa

When password is asked just hit a "ENTER" to leave it empty

2.Now use ssh to create a directory ~/.ssh as user (e.g. "tom") on server(e.g. 203.159.11.42).

ssh tom@203.159.11.42 -p .ssh

3.Copy key to remote server

cat .ssh/id_rsa.pub | ssh tom@203.159.11.42 'cat >> .ssh/authorized_keys'

After finishing these three steps, login to the remote server with ssh will not require password.

using sed from linux shell for file editing

2012-03-25T01:40:00.002-07:00

#Delete empty lines from a file

sed '/^$/d' file.txt

# do the same in the source file itself

sed -i '/^$/d' file.txt

# copy contents from a file from 3rd line

sed -n '3,$p' file.txt

#copy specific lines from a file(e.g. copy line 2 to 4)

sed -n 2,4'p' file.txt

Installing NS-3 in Ubuntu

2012-01-14T00:02:00.000-08:00

Installing testing it and running NS-3 in Ubuntu 10.04

NS-3 is a discrete-event network simulator basically for simulating Internet systems.
#The list of Dependency for ns 3.12(sorry if something is missing):

sudo apt-get install gcc g++ python
sudo apt-get install gcc g++ python python-dev
sudo apt-get install mercurial
sudo apt-get install bzr
sudo apt-get install gdb valgrind
sudo apt-get install gsl-bin libgsl0-dev libgsl0ldbl
sudo apt-get install flex bison
sudo apt-get install g++-3.4 gcc-3.4
sudo apt-get install tcpdump
sudo apt-get install sqlite sqlite3 libsqlite3-dev
sudo apt-get install libxml2 libxml2-dev
sudo apt-get install libgtk2.0-0 libgtk2.0-dev
sudo apt-get install libgtk2.0-0 libgtk2.0-dev
sudo apt-get install vtun lxc
sudo apt-get install uncrustify
sudo apt-get install doxygen graphviz imagemagick
sudo apt-get install texlive texlive-pdf texlive-latex-extra texlive-generic-extra texlive-generic-recommended
sudo apt-get install python-sphinx dia texlive texlive-pdf texlive-latex-extra texlive-extra-utils texlive-generic-recommended
sudo apt-get install python-pygraphviz python-kiwi python-pygoocanvas libgoocanvas-dev
sudo apt-get install libboost-signals-dev libboost-filesystem-dev

#Download and build development version:

hg clone http://code.nsnam.org/ns-3-allinone
cd ns-3-allinone
./download.py

instead of using the above method, you can directly get the ns-3 tarball and untar it and follow the following:

./build.py
cd ns-3-dev
./waf distclean
./waf configure OR ./waf configure --enable-examples --enable-tests
./waf build

Now test the installation of NS-3 (we expect all the tests to run successfully):

./test.py OR ./test.py -c core (test core modules only)

other useful options:

./waf -d optimized configure
./waf --help shows you other options

# To run the first program on NS-3, waf is required. Waf is a Python-based
framework for configuring, compiling and installing applications. It is already
included in the ubuntu repository. The syntax to run any NS-3 script is:

./waf - -run filename

Now, we can run a very basic program on NS-3 without using any extra files and extra codes:

./waf - -run hello-simulator

# Writing your own NS-3 programs:
The scratch/ directory is supposed to contain the user written programs and they are automatically build while running the waf.
For example, let us copy wifi-adhoc.cc from the example/ directory to the scratch/ directory.

cp ns-3.12.1/examples/wireless/wifi-adhoc.cc ns-3.12.1/scratch/wifi-adhoc.cc
./waf
./waf --run scratch/wifi-adhoc

To visualize the program :

./waf --run scratch/wifi-adhoc --visualize

To plot the output :

./waf --run scratch/wifi-adhoc | gnuplot

Graph: Ms Excel 2010

2012-01-09T19:25:00.000-08:00

To plot a graph in Microsoft Excel 2010 by taking the column data as axis in the graph:
1.Highlight the column data that you wish to plot.
2.Insert-> scatter graph

If you wish to plot a chart of X1 and X2 against Y
X2 X1 Y
88 123 100
188 234 200
350 345 300
500 456 400
Select these columns, goto insert and then click scatter chart. This will give the following chart:

But this is not what we want. Now plot X2 and Y , then plot X1 and Y. Then copy the curve from a plot and paste the curve to another plot.

If you wish to edit the style of the plot. For e.g. changing the thickness of the graph line:
1. Right click on the graph -> format data series -> line style -> width (you can specify the thickness value here)

If you wish to scale the range of x-axis and y-axis data values:
1. Double click on th ex-axis -> axis options -> set the minimum/maximum values ( you can also change other values as you need)

If you wish to label some data points:
1. Double click on the curve -> add data label at specific points

If you wish to provide label for the x-axis and y-axis:
1. click on the graph -> Layout -> AxisTitle -> Primary Horizontal/Vertical Axis Title -> write your title
2.similar process for giving chart name as well.

VirtualBox in ubuntu10.04

2012-01-07T20:51:00.000-08:00

VirtualBox is a virtualization tool which allows us to install and run different OS like windows and linux. I had tried installing ubuntu10.04 over ubuntu 10.04. To install it in Ubuntu (10.04 for now):
i. go to Ubuntu Software Center and search for Oracle VM VirtualBox, then click install.
ii. Goto virtualbox.org -> Downloads and get the package that you need for your system (For e.g. ".deb" package for Ubuntu)
iii. Install virtualbox from commandline:

sudo apt-get update
sudo apt-get install virtualbox-4.1

If writing the version number fails then trying without version number might work:

sudo apt-get install virtualbox

After installing the Virtualbox, we can install any Operating System over it. Installing a new OS is more or less similar to installing an OS in a new computer but you need to do some more before proceeding to install the new OS:
i. Open the VirtualBox window, then Press New button and create a new Virtual Machine.
ii. Specify the Name (e.g. ubu10.04)for it and select the OS Type (e.g. Linux )and version (e.g. Ubuntu)
iii. Next, set the memory and virtual hard disk for the new OS to use.

Installing Guest Additions is an imporatant part of VirtualBox:
[Source: http://bit.ly/zJ2DYg]
i. Add the userName to virtualbox user group:

(in the host OS)
sudo usermod -a -G vboxusers userName

ii. Under the Devices menu in VirtualBox, you can select Install Guest Additions and follow the instructions
iii. Dont forget to install the guest utils in guest OS:

sudo apt-get install build-essential

iv. Enable folder sharing, clipboard sharing and Screen resizing feature:

(in the host OS)
sudo apt-get install virtualbox-guest-dkms

I used this command sicne I had problem while mounting shared folder.
v. Create a folder for sharing between host OS and guest OS in the Guest:

mkdir /mnt/u10.04 # create a folder named u10.04
chmod 777 /mnt/u10.04 # make it accessible by all user

vi. Create a folder for sharing between host and guest Os in the Host:

mkdir /home/user/Desktop/u10.04

vii. Goto Devices -> shared Folders -> New and specify the folder path (browse the link to the Host folder created in step vi) and provide the folder name.
viii. Run the following command to mount the shared folder:

sudo mount -t vboxsf -o uid=1000,gid=1000 u10.04 /mnt/u10.04

Removing -o uid=1000,gid=1000 options from above command makes the mounted folder to be writeable only by the root. If there is problem in mounting, then sometimes changing vboxsf to vboxfs does the trick.

ix. If you wish to mount the shared folder during the boot, then run the following command in the command line:

sudo gedit /etc/rc.local

Now, copy the following command in the file and save it.

mount -t vboxsf -o uid=1000,gid=1000 u10.04 /mnt/u10.04

Now, you may need to rebbot your guest OS for all the changes to make an effect.

Post Scripts in blogs

2012-01-03T19:55:00.000-08:00

Posting scripts in the blogs is not straightforward. I came across some ways to do so:
Method #1: Simple Approach:

This will display the script as follows:

Method#2: Better Approach:
1. Login to your Blogger account.
2. Goto Design and then Edit HTML. Then you will see the script for your current blog template.
3. Backup your current template by clicking at "Download Full Template". You may skip this step if you are confident that you do not need it afterwards.
4. Find ]]></b:skin> in the code and add the following code just before ]]></b:skin>:

.code {color: #006AB0; border : 1px solpd #DfADADA;padding : 5px 5px;font-family : Consolas, "Courier New", Courier, mono, serif;font-size : 12px;background-color : #fAfAfA; width : 90%;overflow : auto;margin : 10px 0 10px 10px; border-left : 20px solid #DADADA;max-height : 500px;min-height : 16px;line-height : 16px;}
.code:hover {background-color : #f3f3f3;}

5. Save the template.

To post code in the blog just follow this example(NO Spaces within the div tag):

This will display as follows:

Print("This is my code in codebox");

But sometimes you may require to do more things if you want to display symbols like "<" , ">" ,"&", etc. In that case you have to used the following:
for < use & lt; (no spaces between & and lt)
for > use & gt; (no spaces between & and gt)
for & use & amp; (no spaces between & and amp)
for " use & quot; (no spaces between & and quot)
for space use & nbsp; (no spaces between & and nbsp)
for ± use & plusmn; (no spaces between & and plusmn)

[Sources:mainly from http://bit.ly/zpC2RX ]

Search File with specific content in Ubuntu

2012-01-03T03:15:00.000-08:00

The following command can be used to search any files that is under /home/myFolder and which has the content 'my search string'. The man page for the given command can be browsed for more detail information.

grep -i -r -n 'my search string' /home/myFolder

Where:
-i=ignore case, -r=recursive, -n= give line number

JDK6 in Ubuntu 10.04

2012-01-01T18:01:00.000-08:00

To install JDk 6 (Java Developement Kit 6 ) in ubuntu 10.04 from the command line, we just need to use the following command. This will install all other dependencies automatically.

sudo apt-get install openjdk-6-jdk

shutdown

2011-12-29T00:31:00.000-08:00

We may think that why to bother typing commands when using the GUI to shutting down the computer is easy and straight. But at time we may want to shut down the computer after some time when the download finishes or some other task completes. But we cannot wait until that time. So we can use shutdown command in such cases. For example, if you want to shut down the computer at 2:12 pm then you can use the following command in your ubuntu box:

sudo shutdown -h 14:12 "Going to Shutdown at 2:12 pm"

Kile

2011-12-28T09:58:00.000-08:00

Though the default installation of Kile, a latex editor, in ubuntu works fine for simple things, but at times it needs extra packages (for e.g. algorithm.sty, etc). In that case we can install the following extra scientific packages in Ubuntu linux as follows:

sudo apt-get install texlive-science

Convert .nrg format to .iso format

2011-12-27T19:33:00.000-08:00

A .nrg file is image file format of Nero Burning ROM.

installing .nrg to .iso converter

sudo apt-get install nrg2iso

Usage:

nrg2iso image.nrg image.iso

Installing new fonts to Ubuntu Linux

2011-12-27T19:29:00.000-08:00

Here is an example to install the font 'Times_New_Roman.ttf':
1.copy font the new font to </usr/share/fonts/truetype>

sudo cp Times_New_Roman.ttf /usr/share/fonts/truetype

2. Restart your machine OR just Clear the font cache

sudo fc-cache -fv

Hide drive icon from your Ubuntu Desktop

2011-12-27T08:28:00.000-08:00

It is useful when we wish to configure our desktop item to be stable. Sometimes its annoying when the drive icons appear and disappear when we mount and unmount them. So follow the following steps to get rid of this situation:

1. Hit Alt+F2 then the "Run Application" window will apper.

2. Then type "gconf-editor" and press Enter so as to get the Configuration Editor.

3. Browse to apps > nautilus > desktop.

4. Check off "volumes_visible" then you are done

Restore items from trash to current directory via commandline in Ubuntu Linux

2011-12-27T08:21:00.000-08:00

Once I had to look for command line alternatives to restore items from trash since I was not able to do it from the GUI.

mv ~/.local/share/Trash/files/myFile.doc ./

How to Empty trash from commandline in Ubuntu Linux

2011-12-27T08:14:00.000-08:00

rm -rf /home/username/.local/share/Trash/info/*

rm -rf /home/username/.local/share/Trash/files/*