my little note

David Beazley - Modules and Packages: Live and Let Die!

2015-11-03T03:33:00.001-08:00

If you are Python developer don't miss this section from PyCon 2015 by David Beazly. This section is important for every python developer.
Why it's important? if you don't know about how python module, namespace and packaging in general manner you will die :-).
This section not talk about how to create package, but more general about what is package and module it self.
Good explanations by David, this section also compare which one that should and shouldn't do by developer.
for more information you can watch PyCon 2015 By David Beazley - Modules and Packages: Live and Let Die!

Android HTML5 App, how really it's?

2015-11-02T20:42:00.000-08:00

In this post I just want to share HTML5 development, it's not using framework like apache cordova or something else. I just want to share how to communicate or interfacing between javascript and android java native code.

When I wrote this post, I use Android Studio Version 1.4.

For complete project you can download from this link https://www.dropbox.com/s/02mor7925l1l3bg/MyApplication.zip?dl=0

Create new project with no activity using android studio

Edit AndroidManifest.xml to be like this,

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.amru.myapplication" >

    <uses-permission android:name="android.permission.INTERNET" />

    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:supportsRtl="true"
        android:theme="@style/AppTheme" >
        <activity android:name=".MainActivity" >
            <intent-filter>
                <action android:name="android.intent.action.MAIN"/>
                <category android:name="android.intent.category.LAUNCHER"/>
            </intent-filter>
        </activity>
    </application>

</manifest>

Then add android assets folder inside main, same level with android res folder,
Create assets folder with some folder inside assets folder:

html (for html page)
css (for css style)
js (for javascript function)

Create HTMLUtil class for read html page to parse .html page.

Then put this line into HTMLUtil class

package com.example.amru.myapplication;

import android.content.Context;
import android.content.res.AssetManager;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;

/**
* Created by amru on 10/28/15.
*/
public class HTMLUtil {
    public static String ASSET_DIR = "file:///android_asset/";
    public static String HTML_PREFIX = ".html";
    public static String HTML_DIR = "html/";
    public static String HTML_MIME = "text/html";
    public static String UTF8_ENCODING = "UTF-8";

    private Context _context;

    public HTMLUtil(Context context){
        this._context = context;
    }

    // filename without .html prefix
    public String readHTML(String filename){
        StringBuilder html = new StringBuilder();
        try {
            AssetManager assetManager = this._context.getAssets();
            InputStream inputStream = assetManager.open(HTMLUtil.HTML_DIR + filename + HTMLUtil.HTML_PREFIX);
            BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));

            String line;
            while((line = bufferedReader.readLine()) != null){
                html.append(line);
            }
        }catch (IOException e){
            e.printStackTrace();
        }

        return html.toString();
    }
}

Create JsInterface class, and put this line

package com.example.amru.myapplication;

import android.app.AlertDialog;
import android.content.Context;
import android.content.DialogInterface;
import android.webkit.JavascriptInterface;

/**
* Created by amru on 10/28/15.
*/
public class JsInterface {
    private Context _context;

    public JsInterface(Context context){
        this._context = context;
    }

    // don't forget user @JavascriptInterface if you want call it from javascript
    // this is mandatory start from api 16
    @JavascriptInterface
    public void sayHello(){
        AlertDialog.Builder alert = new AlertDialog.Builder(this._context);
        alert.setTitle("halo");
        alert.setMessage("this from native dialog...");
        alert.setNegativeButton("Close", new DialogInterface.OnClickListener() {
            public void onClick(DialogInterface dialog, int which) {
                dialog.dismiss();
            }
        });
        alert.show();
    }

    // parameter should be in json
    // func is function name
    // call javascript function from native activity
    // need call from post runnable, make sure it's thread safe :)
    public static void callJS(String func, String param){
        final StringBuilder stringBuilder = new StringBuilder();
        stringBuilder.append("javascript:");
        stringBuilder.append(func);
        stringBuilder.append("(");
        stringBuilder.append(param);
        stringBuilder.append(")");
        MainActivity.WEB_VIEW.post(new Runnable() {
            @Override
            public void run() {
                MainActivity.WEB_VIEW.loadUrl(stringBuilder.toString());
            }
        });
    }
}

On MainActivity class put this line,

package com.example.amru.myapplication;

import android.app.Activity;
import android.os.Bundle;
import android.view.View;
import android.webkit.WebChromeClient;
import android.webkit.WebView;

public class MainActivity extends Activity {

    private HTMLUtil _mHTMLUtil;
    public static WebView WEB_VIEW;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        this._mHTMLUtil = new HTMLUtil(this);

        MainActivity.WEB_VIEW = new WebView(this);
        MainActivity.WEB_VIEW.setWebChromeClient(new WebChromeClient());

        MainActivity.WEB_VIEW.getSettings().setJavaScriptEnabled(true);
        MainActivity.WEB_VIEW.getSettings().setAllowContentAccess(true);
        MainActivity.WEB_VIEW.getSettings().setAllowFileAccess(true);
        MainActivity.WEB_VIEW.getSettings().setBuiltInZoomControls(false);
        MainActivity.WEB_VIEW.setWebChromeClient(new WebChromeClient());
        MainActivity.WEB_VIEW.loadDataWithBaseURL(HTMLUtil.ASSET_DIR + HTMLUtil.HTML_DIR,
                this._mHTMLUtil.readHTML("page"),
                HTMLUtil.HTML_MIME,
                HTMLUtil.UTF8_ENCODING,
                null);
        setContentView(MainActivity.WEB_VIEW);

        // register JsInterface class as native interface
        // we can call via NativeInterface.method_name
        MainActivity.WEB_VIEW.addJavascriptInterface(new JsInterface(this), "NativeInterface");

        // let's try to call javascript from native
        JsInterface.callJS("callMeJava", "");
    }
}

For testing the HTML5 App, we need to create some html pages and javascript code. Create page.html, fn.js inside html folder and js folder.

page.html

<html>
    <head>
        <script type="text/javascript" src="file:///android_asset/js/fn.js"></script>
    </head>
    <body>
        <button id="mybutton" onclick="testAlert();">click !</button>
    </body>
</html>

fn.js

// call native interface function
function testAlert(){
NativeInterface.sayHello();
}

// this will call from java native
function callMeJava(){
alert('call js function from java native.');
}

Then, we can start to test the app prototype. When app loaded it will trigger callMeJava function from fn.js. The method call from oncreate activity just test case only.

through this line on MainActivity

// let's try to call javascript from native
JsInterface.callJS("callMeJava", "");

Let's test call from javascript to native,

Through this line (JsInterface class)

// call native interface function
function testAlert(){
NativeInterface.sayHello();
}

// this will call from java native
function callMeJava(){
alert('call js function from java native.');
}

Thanks.

Python Access Java Through JNIus

2015-10-27T22:30:00.000-07:00

Can python access java library or class?, yes it's possible. There are some project to do this, the popular one is PyJNIus and this project under Kyvi project https://github.com/kivy/pyjnius.

How it's work? PyJnius wrap java using JNI call so python can call it, don't worry if you don't know about it. Me to :-)

To install PyJNIus, we need to install

Java JDK installed (OpenJDK will do)
Python 3.4 (In this case I use 3.4 version)
Setup JAVA_HOME environment
Setup JRE_HOME environment

After install python, we need to install cython, use pip or easy install. In this case I use pip.

$ sudo pip install cython

After cython installed, then install PyJNIus through php or easy install. In this case I use pip

$ sudo pip install PyJnius

After PyJnius installed then try import jnius from python, and let learn little thing about calling java from python.

On linux should be no problem when importing jnius,

Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from jnius import autoclass
>>>

But on OSX we sometime meet this error when import jnius

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/barry/anaconda/envs/ovation-testing/lib/python2.7/site-packages/ovation/__init__.py", line 8, in <module>
from jnius import autoclass, cast, JavaException
File "/Users/barry/anaconda/envs/ovation-testing/lib/python2.7/site-packages/jnius/__init__.py", line 12, in <module>
from .jnius import *
ImportError: dlopen(/Users/barry/anaconda/envs/ovation-testing/lib/python2.7/site-packages/jnius/jnius.so, 2): Library not loaded: @rpath/libjvm.dylib
Referenced from: /Users/barry/anaconda/envs/ovation-testing/lib/python2.7/site-packages/jnius/jnius.so
Reason: image not found

If you found that problem, you need to add this to .profile in your home file, and add this line then restart your terminal then try again to import jnius.

export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:$(/usr/libexec/java_home)/jre/lib/server

now you can import jnius on python interpreter.

On windows, I have no luck to install PyJnius through pip or easy install using MinGW as compiler. But I still struggling with it :-(.

now let's test to access some java library

Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import jnius
>>> from jnius import autoclass
>>> Stack = autoclass('java.util.Stack')
>>> stack = Stack()
>>> stack.push('hello')
'hello'
>>> stack.push('world')
'world'
>>> stack.pop()
'world'
>>> stack.pop()
'hello'
>>> System = autoclass('java.lang.System')
>>> System.out.println('Hello World')
Hello World

for more learning stuff you can refer to:
http://pyjnius.readthedocs.org/en/latest/
http://kivy.org/planet/2012/08/pyjnius-accessing-java-classes-from-python/

with this project, it's possible to develop android app using python. To create android package we can use http://python-for-android.readthedocs.org/en/latest/

How to make android emulator run faster?

2015-10-25T05:44:00.001-07:00

Android emulator running too slow? this is main problem when develop android application :-). I always trying to find other solution rather than using original google android emulator.

After do some googling and learn little about virtualization technology, I got little knowledge how to run android emulator using intel virtualization technology (VT) this feature must be activated through BIOS.

We need to access bios and activate intel virtualization technology, keep in mind that not all processor and motherboard support intel virtualization technology. But if intel virtualization technology exist in your bios setting you can use it to run android emulator using x86 image.

If your processor and motherboard support intel virtualization technology install haxm through android sdk manager

But if your computer doesn't support inter haxm don't worry just skip haxm installation step, then go to android sdk installation folder and goto extra/intel/Hardware_Accelerated_Execution_Manager then install using silent_install.bat must be using command prompt with administrator mode:

D:\SDK\android-sdk-windows\extras\intel\Hardware_Accelerated_Execution_Manager>silent_install.bat -m 512

-m parameter is memory will be used by haxm manager, don't use memory more than 1024MB, after installation succes you will get message in command prompt will return success message.

Next step install image x86 both, x86 image system and x86 google api image system.

Then setup your android virtual device (AVD), setup using one of x86 image system, don't forget to Use GPU Host

That's it, and let you got fast android emulation.

Python LinkedIn REST API Implementation

2015-10-20T23:42:00.003-07:00

LinkedIn using OAuth Version 2.0 it's different process compare with OAuth Version 1.0. Newer version is more easy to implement we don't really need to create parameter with oauth_ prefix and generate signature on client side, for more information about oauth version 2.0 we can refer to official oauth page http://oauth.net/2/.

All request should use https protocol and redirect url must be accessible if not we can't get code for generate access token. The step is simple just 2 step only.

1. Generate grant access url, from here we will redirected to our redirect url and we will get code for authorization code
2. Generate access token using that code with POST method and we will get token access with specific time expired

We can use that token access to call API to provider. Token access will expired in specified time and we need to renew access token with both step.

You can grab pyLinkedinAPI from my github repository https://github.com/amru-rosyada/pyLinkedinAPI

Please if you use it, don't forget to give us feedback. Next project I will try to implement Facebook, G+, Instagram and Flickr as far as I have little time to make it :D

Python Tumblr REST API Implementation

2015-10-20T23:41:00.003-07:00

In previous post I was implement python twitter API, today I will talk something about tumblr API implementation.

I guess twitter, tumblr, linkedin and yahoo used same oauth method and api implementation. I use pyTwitterAPI base code to implement tumblr rest API and still work but with little additional parameter.

Tumblr is yahoo like API implementation as we know tumblr was acquired by yahoo :-) so if it's worked with tumblr, it's also will work with yahoo API :D

The main different with twitter API call is tumblr add api_key parameter which is consumer key. So if I want to call tumblr rest API, I must pass api_key=MY_CONSUMER_KEY in api url.

ex:

http://api.tumblr.com/v2/blog/scipsy.tumblr.com/info?api_key=fuiKNFp9vQFvjLNvx4sUwti4Yb5yGutBN4Xh10LXZhhRKjWlV4

Most of tumblr api call need base hostname as parameter. For example my tumblr domain is http://xyzmind.tumblr.com, so I need to call it with http://api.tumblr.com/v2/blog/xyzmind.blogspot.com/info?api_key=fuiKNFp9vQFvjLNvx4sUwti4Yb5yGutBN4Xh10LXZhhRKjWlV4. It's also work with tumblr custom domain :D

You can grab my python tumblr API implementation from my github repository https://github.com/amru-rosyada/pyTumblrAPI

Let me know if something wrong, may you want to contribute with my project.

Python Twitter REST API Implementation

2015-10-20T23:40:00.003-07:00

Before read this article maybe you need to read my previous article http://walkinsmile.blogspot.com/2015/04/python-oauth-implementation-its-not.html

I already wrote some codes in a day to implement wrapper for twitter REST API (OAuth V1.0 and REST API V1.1), this code will simplify process to call twitter REST API from python code.

It's not that complex and not that hard and just little code to help other developer if they need to simplify twitter REST API call.

I use python version 3.4 :-), you may need some modifications and tuning to run this code on older python, especially older than version 3.

You can grab my code on my github repository free to use, modify and redistribute https://github.com/amru-rosyada/pyTwitterAPI

Understanding python decorator in easy way

2015-10-20T23:39:00.003-07:00

I still have enough energy to write something on my blog :D after one week trip to Jakarta for some work reason. I am so tired but I want to write and share something about python decorator before I sleep :-).

Lets see how python function work,

Code 1:

def myfunc(a):
print(a)

decorator = myfunc
decorator('It\'s Works')

Output 1:

It's Works

Code 2:

def myfunc(*args):
print(args)

decorator = myfunc
decorator('It\'s', 'Works', 'Yeah', 'Must be ok')

Output 2:

("It's", 'Works', 'Yeah', 'Must be ok')

Code 3:

def myfunc(**kwargs):
print(kwargs)

decorator = myfunc
decorator(a=1, b=2, c=3)

Output 3:

{'a':1, 'b':2, 'c':3}

Inner Function
Inner function is function inside function or we can call it nested function. Look at code bellow:

def myfunc(a):
    def inner_myfunc():
        print(a)

    return a

decorator = myfunc
decorator('inner myfunc')

Output should be:

'inner myfunc'

from the code we can se that, argument or parameter a is accessible from inner function.

I have some question why I always wrote decorator = myfunc? I just want to show python function can be assigned to var as alias or pointer.

@some_decorator
def myfunc():
return a

it's mean:
myfunc = some_decorator(myfunc)

Lets try to the real code with decorator implementation:

# define logger function
# we will this for as decorator
def logger(func):
    def inner(*args, **kwargs): #1
        print "Arguments were: %s, %s" % (args, kwargs)
        return func(*args, **kwargs) #2

    return inner

# use the decorator
@logger
def myfunc1(x, y=1):
    return x * y

# use the decorator
@logger
def myfunc2():
    return 2

After define function now try to call myfunc1 and myfunc2:

# call function
myfunc1(5, 4)

# output should be
Arguments were: (5, 4), {}
20

# call function
myfunc1(1)

# output should be
Arguments were: (1,), {}
1

# call function
myfunc2()

# output should be
Arguments were: (), {}
2

If you need more explanation you can comment this post, just comment and I will try to answer :-).

Let Sleep.....

Python OAuth implementation it's not that hard!

2015-10-20T23:38:00.005-07:00

Did you know about OAuth ? I will explain about OAuth from some resources,

Wikipedia: http://en.wikipedia.org/wiki/OAuth
OAuth is an open standard for authorization. OAuth provides client applications a 'secure delegated access' to server resources on behalf of a resource owner. It specifies a process for resource owners to authorize third-party access to their server resources without sharing their credentials. Designed specifically to work with Hypertext Transfer Protocol (HTTP), OAuth essentially allows access tokens to be issued to third-party clients by an authorization server, with the approval of the resource owner, or end-user. The client then uses the access token to access the protected resources hosted by the resource server. OAuth is commonly used as a way for web surfers to log into third party web sites using their Microsoft, Google, Facebook or Twitter accounts, without worrying about their access credentials being compromised.

If you want to know more about OAuth standard you can refer to http://oauth.net/

In this post I will use Python version 3.4,

Something you need to know about OAouth implementation:

Know about REST API from service provider like twitter, facebook, google, tumblr, yahoo etc. They have different method when request using oauth. some provider using POST only or GET only or both.
Know about OAuth standard at least you have read it
Know about term which is used by OAuth like oauth_callback, oauth_nonce, oauth_consumer_key, oauth_consumer_secret, oauth_token, oauth_token_secret, oauth_timestamp, oauth_signature, oauth_signature_method and oauth_version
that's it

OAuth using CONSUMER_KEY and CONSUMER_SECRET to generate token, then this token will be use to access API from service provider.
I will use twitter rest api for this implementation, but this will be useable for common OAuth standard :-) don't worry. Before we start you must create application on twitter follow this twitter app registration.

1. If you already have account you can sign in, but if you don't have you need register to twitter

2. After sign in, you will get create application button if you don't have any application. Then just create new app

3. Fill your application name, description, website and callback. Callback for what? callback is url that will call after user success and granted your application. For now we don't really care about this we just want to create oauth impletentation to get token and token secret

4. After finish your app creation, then scooll down on your app detail. You will get some information about url for request token, authorization, access token.

5. Go to tab Keys and Access Tokens and The important thing is you need to retrieve CONSUMER_KEY and CONSUMER_SECRET from your app. Sorry I don't want show my consumer secret :D.

Ok then lets we start, create implementation of OAuth is not hard the only we need is follow the rule.
I already create basic code, using twitter api for testing my code implementation you can grab it from my github repository here https://github.com/amru-rosyada/pyTwitterAPI.

Thats the basic code, but I will add more for retrieve or post status. The basic is enough if you want to implement other request. follow this twitter rest api.

Regards,

How to request POST and GET using Python ?

2015-10-20T23:38:00.001-07:00

I want to share about HTTP request using python, HTTP request can be done using POST or GET method. The main different between POST and GET is on HTTP header, POST method need Content-Type: application/x-www-form-urlencoded definition when sending data.

If you want to know about HTTP header you can refer to wiki page http://en.wikipedia.org/wiki/List_of_HTTP_header_fields or W3 page http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html.

But I will not talk about HTTP Protocol standard in this post, I only will share about how to request POST and GET using python (I use python version 3.4).

POST Method:

# import url Request to create request object
from urllib.request import Request

# import urlopen to send request to web site
from urllib.request import urlopen

# for post request need define Content-Type : application/x-www-form-urlencoded on http header
header = {'Content-Type':'application/x-www-form-urlencoded'}

# create request object
# data query need to encode to byte, it is save to use standart ISO-8859-1
req = Request('http://localhost:8888/p/authenticate/google', 'firstname=amru&lastname=rosyada'.encode('ISO-8859-1'), header)

# send post request
post_response = urlopen(req)

GET Method:
GET code snippet is same with POST but without application/x-www-form-urlencoded on http header data parameter should be None of left it blank.

# import url Request to create request object
from urllib.request import Request

# import urlopen to send request to web site
from urllib.request import urlopen

# create request object
# data query need to encode to byte, it is save to use standart ISO-8859-1
req = Request('http://localhost:8888/p/authenticate/google?firstname=amru&lastname=rosyada', None)

# send post request
get_response = urlopen(req)

Thanks :D

Python in a Minute

2015-10-20T23:37:00.002-07:00

Today I will share about how to learn python in a minute, in this post I will use python version 3 only and discuss about python keyword, using python module, python mode, basic syntax, operator, function, python data type and class.

This post is not deep concept, but about basic concept before we start deep learning into python.

Python is easy programming language to learn, powerful programming language. It has efficient high-level data structures and a simple but effective approach to object-oriented programming. Python’s elegant syntax and dynamic typing, together with its interpreted nature, make it an ideal language for scripting and rapid application development in many areas on most platforms.

Learning python is not hard because it's "human language" :-) it's mean that python language is more readable, portable, clean, easy to develop and easy to extends.

Ok lets start don't wasting time :D.

Python Mode
Python have two mode which is interpreter mode and scripting mode, we don't need to compile the code because compilation process to byte code will be handled by python interpreter.

You need to install python interpreter before you can start using python, can be downloaded from here python.org. Choose the right one for you distribution OS.

Interpreter mode can be access from terminal emulator in linux and mac or cmd in windows. Most linux distro include python interpreter on each their distribution.

From terminal emulator or cmd you simply type python you will get in to python interpreter depend on your system.

Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

With interpreter method you can use it as calculator like sum, div, mod etc

Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+1
2
>>> 3*3
9
>>> 3**4
81
>>> 6%2
0
>>>

The second one is script method, you can save python code in .py extension and run it with python interpreter. Let start with hello world example :D, open your text editor (notepad, notepad++, sublime text, emacs or nano up to you which one do you choose or you prefer using other text editor) create helloWorld.py then save it. Put this line into helloWorld.py and save it.

a = 'Hello World, this is my first python script'
print(a)

Open terminal in the same folder where helloWorld.py file saved. Then type python helloWorld.py

amrus-MacBook-Pro:Desktop amru$ python helloWorld.py
Hello World, this is my first python script

Ok, now we know about interpreter mode and scripting mode. :-) super easy isn't it?

Python Keyword
The following identifiers are used as reserved words, or keywords of the language, and cannot be used as ordinary identifiers. They must be spelled exactly as written here:

False, class, finally, is, return, None, continue, for, lambda, try, True, def, from, nonlocal, while, and, del, global, not, with, as, elif, if, or, yield, assert, else, import, pass, break, except, in, raise

Basic Syntax
Python use indentation for block code, I recommended using space rather than tab so it can be portable if change to other text editor. Change from your text editor setting and change tab to 4 spaces or depend on your preference.

Python syntax compare with other language:

C language:
// this is line comment in C
/*
* this is block comment in C
*/
int add(int a, int b){
return a*b;
}

PHP language:
// this is line comment in PHP
/*
* this is block comment in PHP
*/
function add($a, $b){
return $a*$b;
}

Java language:
// this is line comment in Java
/*
* this is block comment in Java
*/
public int add(int a, int b){
return a*b;
}

Python language:
# this is line comment in C
"""
this is block comment in C
"""
def add(a, b):
return a*b

Which one is simpler? :D. Yes python is simpler and clean. I don't mean that other language is bad but I just want to show to you python is more simpler than other.

Keep in mind python using indentation for block code, let try other sample create fibonacci.py, put this code and save it.

a, b = 0, 1
while b < 10:
print(b)
a, b = b, a+b

then run python fibonacci.py, the execution output should be

1
1
2
3
5
8

Python Operator
Like in other programming language python have some operator, this table will explain operator in python

Operation	Syntax	Function
Addition	`a + b`	`add(a, b)`
Concatenation	`seq1 + seq2`	`concat(seq1, seq2)`
Containment Test	`obj in seq`	`contains(seq, obj)`
Division	`a / b`	`truediv(a, b)`
Division	`a // b`	`floordiv(a, b)`
Bitwise And	`a & b`	`and_(a, b)`
Bitwise Exclusive Or	`a ^ b`	`xor(a, b)`
Bitwise Inversion	`~ a`	`invert(a)`
Bitwise Or	`a \| b`	`or_(a, b)`
Exponentiation	`a ** b`	`pow(a, b)`
Identity	`a is b`	`is_(a, b)`
Identity	`a is not b`	`is_not(a, b)`
Indexed Assignment	`obj[k] = v`	`setitem(obj, k, v)`
Indexed Deletion	`del obj[k]`	`delitem(obj, k)`
Indexing	`obj[k]`	`getitem(obj, k)`
Left Shift	`a << b`	`lshift(a, b)`
Modulo	`a % b`	`mod(a, b)`
Multiplication	`a * b`	`mul(a, b)`
Negation (Arithmetic)	`- a`	`neg(a)`
Negation (Logical)	`not a`	`not_(a)`
Positive	`+ a`	`pos(a)`
Right Shift	`a >> b`	`rshift(a, b)`
Slice Assignment	`seq[i:j] = values`	`setitem(seq, slice(i, j), values)`
Slice Deletion	`del seq[i:j]`	`delitem(seq, slice(i, j))`
Slicing	`seq[i:j]`	`getitem(seq, slice(i, j))`
String Formatting	`s % obj`	`mod(s, obj)`
Subtraction	`a - b`	`sub(a, b)`
Truth Test	`obj`	`truth(obj)`
Ordering	`a < b`	`lt(a, b)`
Ordering	`a <= b`	`le(a, b)`
Equality	`a == b`	`eq(a, b)`
Difference	`a != b`	`ne(a, b)`
Ordering	`a >= b`	`ge(a, b)`
Ordering	`a > b`	`gt(a, b)`

Checking syntax is easy, we can use python interpreter mode, ex:

amrus-MacBook-Pro:Desktop amru$ python
Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> # for a + b
... 1 + 2
3
>>> # for a - b
... 1 - 2
-1
>>> # concat two string
... 'hello' + ' world'
'hello world'
>>> # shift right operation
... 1 >> 2
0
>>> # shift left operation
... 1 << 2
4
>>> # test if object in list
... 'a' in ['a', 'b', 'c']
True
>>>

Python Data Type
Python have powerful built in data type dict, list, set, frozenset and tuple

Python Data Type : Dict
Dict is dictionary data type must contain pair key and value, initialization of dict using {}

amrus-MacBook-Pro:Desktop amru$ python
Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = {'hello':'world', 1:'oke', 'yes':10, 'dict_in_dict':{}, 'list_in_dict':[1, 2, 3, 4], 'tuple_in_dict':(1, 2, 3, 4)}
>>> a.get('hello')
'world'
>>> a.get('tuple_in_dict')
(1, 2, 3, 4)
>>> a.get('list_in_dict')
[1, 2, 3, 4]
>>> a['hello'] = 'guys'
>>> a.get('hello')
'guys'
>>> a['add new item'] = 'new item added'
>>> a.get('add new item')
'new item added'
>>>

As we can see python dictionary can be mixed value with string, object, int and other python objects.

Python Data Type : List
List in python is like array in C or C++ but more powerful, you can mixed value inside list not only one data type. List python support append, insert, sort and other operation.

amrus-MacBook-Pro:Desktop amru$ python
Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = [1, 'oke', 4, 0.3, 'yeah']
>>> a[0]
1
>>> a[2:]
[4, 0.3, 'yeah']
>>> a[2:-1]
[4, 0.3]
>>> a.append('append to last')
>>> a
[1, 'oke', 4, 0.3, 'yeah', 'append to last']
>>> del a[0]
>>> a
['oke', 4, 0.3, 'yeah', 'append to last']
>>> a.pop(2)
0.3
>>> a
['oke', 4, 'yeah', 'append to last']
>>>

list initialization using [], list item can be retrieve using list[index]. Negative index will search from last list item. List[-1] mean get last item, List[0:3] mead get slice of list from index 0 until index 3.

Python Data Type : Tuple
Tuple is like List the difference between both data type is Tuple immutable, the value cannot be change. To initialize tuple use ().

amrus-MacBook-Pro:Desktop amru$ python
Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = (1, 2, 3, 4.3, 'iya')
>>> a[1]
2
>>> a[0:3]
(1, 2, 3)
>>> a[0] = 2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>>

See last command a[0] = 2 will raise error, because tuple is immutable

Python Data Type : Set, Frozenset
A set object is an unordered collection of distinct hashable objects. Common uses include membership testing, removing duplicates from a sequence, and computing mathematical operations such as intersection, union, difference, and symmetric difference.

amrus-MacBook-Pro:Desktop amru$ python
Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = set([1, 2, 3, 4, 5])
>>> b = set((3, 4))
>>> a.union(b)
{1, 2, 3, 4, 5}
>>> a.difference(b)
{1, 2, 5}
>>> b.issubset(a)
True
>>>

As we can see when create set parameter should be iterable object like tuple or list.

Python Loop
Python only have two type of loop which is for and while.

Python Loop : For

amrus-MacBook-Pro:Desktop amru$ python
Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> for i in [1, 2, 3]:
... print(i)
...
1
2
3
>>> for i in range(3, 10):
... print(i)
...
3
4
5
6
7
8
9
>>> for i in (1, 2, 3):
... print(i)
...
1
2
3
>>> d = {'mami':'halo', 'papi':'halo'}
>>> for i in d:
... print(d.get(i))
...
halo
halo
>>>

Python Loop : While

amrus-MacBook-Pro:Desktop amru$ python
Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = 10
>>> a = 3
>>> while a > 0:
... a -= 1
... print(a)
...
2
1
0

Python Import Module
We can import external module or internal python module using import keyword.

amrus-MacBook-Pro:Desktop amru$ python
Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> from datetime import date
>>> date.today()
datetime.date(2015, 4, 15)
>>> datetime.datetime.today()
datetime.datetime(2015, 4, 15, 12, 36, 27, 307647)
>>>

Let create external simple module, create testModule.py then put this code.

def sum_it(a, b):
print(a+b)
def pow_it(a, b):
print(a**b)

Open python interpreter in folder contain testModule.py file then do this stuff.

amrus-MacBook-Pro:Desktop amru$ python
Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from testModule import *
>>> sum_it(1, 3)
4
>>> pow_it(2, 5)
32
>>>

Finally we can call function from other file.

Python Function
To define python function we use def keyword, python function can have default parameter to prevent ordering fault. Better if your function have default parameter :D
Let we do little stuff with python function.

amrus-MacBook-Pro:Desktop amru$ python
Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> # let we define functin here
... def devide_it(a, b):
... return a/b # return result a devide by b
...
>>> # let we call the function
... devide_it(10, 3)
3.3333333333333335
>>> # let try create function and use default parameter
... def set_name(first_name='amru', last_name='rosyada'):
... print(first_name + ' ' + last_name)
...
>>> # let call function without parameter
... set_name()
amru rosyada
>>> # let call function with parameter
... set_name('David', 'Beckham')
David Beckham
>>>

Python Class
As high level language python also support OOP Concept(Object Oriented Programming). If you don't know about OOP you must learn it from other source.
Ok let we take a look to python class,

Python 3.4.3 (default, Mar 10 2015, 14:53:35)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> # let we define a class
... class ParenClass():
... def __init__(self): # this is constructor of the class
... self.public_paren_var = 'a'
... self.__private_paren_var = 'private a'
... def public_method(self):
... print('public method')
... def __private_method(self):
... print('private method')
...
>>> p = ParenClass()
>>> # let retrieve public variable
... p.public_paren_var
'a'
>>> # we cannot retrieve private variable of class
... # private variable or method use prefix '__'
>>> p.__private_paren_var
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'ParenClass' object has no attribute '__private_paren_var'
>>> # let we call public method
... p.public_method()
public method
>>> # we can't call private method
... # private method/function use prefix '__'
... p.private_method()
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
AttributeError: 'ParenClass' object has no attribute 'private_method'
>>>

Ok, that is the basic of python and now we can learn more deep about python concept. Let me know if you need some assist contact me for free.

Setup Python on windows platform

2015-10-20T23:36:00.003-07:00

Hello guys, today I will share how to setup and install python on windows platform. Before installation process you need to download python installer for windows platform at https://www.python.org/downloads/windows/ I recommended to download python 3 latest version.

If you already finished download, run installer by double click your python installer and follow the wizzard :-).

Until you found this page, don't forget select Add python.exe to Path you need this to enable and insert python path on Windows Environment.

After installation process finish then we need to check python from command prompt but you need to logout and re-login to your windows system.

Open cmd and type python, you should got this python interpreter.

Type pip to check python package installer

Type easy_install --help to check other package installer method

Thanks you hope it will help.

No more Python 2.8

2015-10-20T23:35:00.003-07:00

I use python for my daily programming stuff and I am so happy I can use it in my daily programming stuff, Python language it so beautiful, effective and clean.

Why it's beautiful? because it's easy to learn, easy to implement, easy to extends (we can extends python using C or C++). It's good programming language for people who want to learn basic programming concepts. But don't forget to have deep concept in C programming language :D, C is important thing and one thing you need to know about python it made from C.

Why it's effective? python syntax and idiom is well mature. We can create complicated stuff with python using few lines of code. Python built in method is complete you don't really need third party library if you not use it on specific purpose.

Why it's clean? Python functions have no explicit begin or end, and no curly braces to mark where the function code starts and stops. The only delimiter is a colon (:) and the indentation of the code itself. Python block of code is separate with indentation not curly braces so it clean and you will always got clean code structure.

But python 2.7 is the last python 2 development and will not supported in the future python release, because python development team decided to support python 3 version only. Don't worry python 3.4 is best python version ever, you can convert code from python 2.7 to python 3. :-) smart.

SQLite3 Review – Great for Beginners and Those Learning SQL

2015-02-03T23:43:00.000-08:00

Essentialsql.com SQLite3 is a compact free database you can use easily create and use a database. Though SQLite3 is not a full-featured database, it supports a surprisingly large set of the SQL standard, and is ideal for those just starting to learn SQL as well for developers that need a simple database engine to plug into their applications. As such, SQLite has become very popular with smart phone developers.

Learn SQL with SQLite
There are several reasons I feel that SQLite3 is suitable for beginners.

The first is that you don’t need to be an expert to install and configure it. In fact, getting SQLite3 to run is as simple as downloading the program and then running a simple command.

sqlite3 EssentialSQL.db

This is all that is needed to start up the database engine and start using the essential SQL database.

Second, the software is a really simple command line interface. Now you may think that is a fault, but it isn’t, as our goal is to learn SQL. And to really learn SQL you need to understand its commands and instructions (syntax). SQLite lets you work distraction free with the essence of SQL.

Also, SQLite runs many different computer systems such as Apple OS X, Linux, and Windows. This was important to me, since I wanted to provide everyone with easy to follow SQL lessons, and didn’t’ want to leave anyone out.

The last reason I recommend SQLite to beginners is that it is free to use.

What Others Say about SQLite3
There are other free database engines you can use to learn SQL. The most popular are Microsoft SQLExpress, MySQL, and Oracle Express.

Each of these are excellent databases and I have no trouble recommending them to anyone. If you look at the Database Engine rankings, you’ll see these are the top three databases in popularity. All three are full featured, and truly capable or supporting very large databases.

Once you learn SQL using SQLite, my recommendation is to pick one of these three products to continue your education.

What one would you choose next? I would pick the database that your company or department uses. Chances are it is either Oracle or Microsoft SQL. If you do a lot of work on the web, then I would gravitate towards MySQL.

SQLite3 Benefits
We covered some of the ways SQLite is suited for learning SQL; however, its benefits go beyond that. Though SQLite is a compact database, it is meant for serious applications! Airbus, the airplane manufacturer, even uses it on some flight software for their A350 line of aircraft. Some benefits of SQLite3 include:

Small and Self-Contained

No additional programs or components are required for it to run. The database engine can run on any modern PC or Smartphone

SQLite doesn’t require a separate computer process to run. IT doesn’t rely on windows services, background daemons, nor separate computer hardware.

TRANSLATION: It is simple to install and use.

Portable
It is really simple to share databases. Just copy one file to do so. This is what makes it easy for me to provide you with the essential SQL sample database.

A SQLite database file is at home on Windows as much as it is OS X.

Standards Aware
SQLite is based on the standard SQL language. Some features are omitted (see below), but those implemented closely adhere to standards.

This means what you learn, will easily translate to other database engines.

Public domain
SQLite isn’t owned by any company. Instead of a software license, the code comes with a simple blessing:

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.

Isn’t that refreshing?

SQLite Cons
There are disadvantages to SQLite. Since it is so compact and meant to run in a small space, some trade-offs were made.

Limited Data types
SQLite doesn’t support the date or timestamp data types. This is a disadvantage as many SQL puzzles, thus learning opportunities, are out of our reach in the beginning.

SQL Language Support
There are several features not supported in SQLite. Right Outer Joins aren’t supported nor are Full Outer Joins. Since Left Outer Joins are supported, and are similar to Right Outer Joins, the opportunity to learn isn’t lost.

It also isn’t easy to alter an existing table nor are there some referential integrity checks. These are pretty important features to have to manage a production database.

Size Limitations
SQLite isn’t meant to support extremely large databases. It isn’t able to scale to hundreds of users nor store gigabytes of data. It isn’t suitable where you expect a large number of users to simultaneously modify data (high concurrency), nor where there is a large volume of transactions.

In general I would say most reviewers have positive things to say about SQLite3. All pretty much agree that SQLite isn’t meant to power your next ERP system (a large database), but is well suited for a personal database or one hosting several users. SQL does what it does well. SQLite just works.

While searching for a minimal database, alkoksha.org, found SQLite3 to be the best choice.

SQLite.org recommends not using SQLite for a client/server application. We agree.

Learning More About SQLite3
If you are curious to learn more about SQLite I would recommend you start here to begin our free course on learning SQL using SQLite3. The course assumes you know nothing about programming, so anyone, really anyone who can drag a file onto their computer, can do the course.

If you want read more about SQLite3, then I would recommend Using SQLite, by Jay A. Kreibich (affiliate link). This book is well received by the community. It is definitely more technical than my training classes, but is a great way to get into more detail about SQLite.

solving a start job is running for create volatile files and directories : manjaro linux

2015-01-26T18:40:00.000-08:00

I have used Manjaro Linux which is Arch Linux based system for several month. Manjaro using roling system update, so we don't need to install new distribution to use latest version. We only need to upgrade through update manager.

After several times I used roling update then I restart my system boot time take long time, and plymouth like freeze. When I press escape to view error message on boot time. I got this message.

a start job is running for create volatile files and directories

I search internet but I got nothing to solve my problems, then I try to view my manjaro setting then I take look on kernel section, I know that my kernel is need to be upgrade. Manjaro system will notif recommended kernel version for installed system if newer kernel exist.

Install Recommended kernel from kernel section, after installation finish I restart my system and it's like a magic all error message and boot time back to normal :).

#good luck have fun

The first Ubuntu Phone will go on sale in Europe in the second week of February.

2015-01-21T20:17:00.000-08:00

Read original post here.

Made by Spanish device company Bq, the handset will mark the first time that an Ubuntu Phone is sold directly to consumers

The First Phone: Specifications

For its first Ubuntu Phone Bq is launching a repurposed version of the its popular Aquaris e4.5 handset preloaded with Ubuntu for Phones.
Unlike the version that developers can (and have) been downloading and flashing to their Nexus devices, the commercial version of Ubuntu on phones will ship with a number of differences in software, including new ‘Aggregator Scopes’, ‘Today Scope’, and support for paid content.
Specifications wise the Aquaris e4.5 is a capable daily driver. It’s a stylish, well built and dependable handset.

4.5-inch screen (qHD resolution @ 540×960)
1.3 GHz Quad Core ARM Cortex A7 (MediaTek)
Mali 400 GPU @ 500 MHz (MediaTek)
8GB eMMC Storage
1GB RAM
2150 mAh Battery
Dual micro-SIM

No, it’s not the most powerful of device but, honestly, that’s good Ubuntu’s debut device needs to be accessible, to balance price and performance so that hesitant developers and enthusiasts — the lifeblood of any burgeoning platform — can pick one up to tinker with without needing to sell a body part.

Although Canonical hasn’t given us any firm price tag is has been suggested to us, with a wry wink and a cheeky smile, that buyers certainly won’t pay more than the ~€149 or so that Bq currently sells the Android version at.

If anything, it is likely to sell for less.

Looking Ahead

The Bq Handset In Hand

The elephant in the room is the USA. Initially Bq will only sell their Ubuntu phone in Europe, and that through their website. US plans are yet to be divulged, and it’s not clear if it will be Bq, Meizu or an as yet unannounced partner jumping through the hoops required to launch a cellular device there.
Looking ahead, I think it’s likely that we’ll see Bq launch an Ubuntu version of its 4G-equipped Aquaris 5 model later in the year. The model is a little larger than the e4.5 and features an HD IPS screen screen and marginally faster processor. It’s this device that was launched at Bq’s ‘mystery press event’ last month, alongside several new Android tablets.
A video advert directed by renowned Spanish filmmaker Álex de la Iglesia premiered at that event — an advert I’ve a hunch we’ll be seeing a little more in the near future… ;)

Manjaro XFCE Post Installation Tips and Tricks

2015-01-20T03:49:00.000-08:00

To read original article go to here.

It's nearly a year since my last post, a lot of things had happened in between, Placement preparation, couple of interviews, six month internship in Blackberry Platform at Payoda Technologies and finally completion of my studies. During this period a lot of distros had been released but I stayed with the stable Opensuse, but then I felt KDE version of opensuse is little bit heavy, which can be felt during both start-up and during normal day to day usage(say Eclipse+amarok+chrome) even with 4GB RAM and i3 Processor.

I started my quest for "The best GNU/Linux OS" with distrowatch.com. Since I wanted a lighter distro but not certainly like Puppy/Tiny and no Ubuntu/Debian forks then by default I left with Arch. But the fear of two earlier failed attempts to install Arch forced me to try arch fork than arch. Seeing Manjaro Linux description "Manjaro Linux is a desktop-oriented, user-friendly distribution based on Arch Linux" , I thought I would give it a try. Downloaded and Installed manjaro-xfce-0.8.5.1-x86_64 from Manjaro site.

The first thing I noticed with Manjaro is, it's fast start-up and shutdown time, it is comparatively faster than opensuse. Here are the some of tricks which I had done to achieve what's in their description user-friendly distribution. NOTE: No Graphics driver installation tips are given because author himself followed the steps listed in Beginner's guide and finally ended with blank screen even though he only have Intel's Integrated graphics card.

1. Synchronize and Update Manjaro repositories using pacman

Pacman is the package manager used in Manjaro/Arch.


sudo pacman -Syu

For handy pacman tips refer @YF-Dogan's thread from manjaro forum and the wiki.

2.Mounting windows partition with Read and write permission.

i)To achieve write permissions for other users than root, you will need to install ntfs-3g,
which also will install the fuse package in order to work.

   sudo pacman -Sy ntfs-3g

ii) List all available disk partitions

 [blizzard@linuxbox Desktop]$ blkid
 /dev/sda1: LABEL="System Reserved" UUID="F088C67D88C64238" TYPE="ntfs" 
 /dev/sda2: UUID="3C804D34804CF644" TYPE="ntfs" 
 /dev/sda4: UUID="A80E38E70E38B064" TYPE="ntfs" 
 /dev/sda5: UUID="1da4f64f-3576-4caf-8cb8-e46f0514cd21" TYPE="ext4" 
 /dev/sda6: LABEL="swap" UUID="28b7a025-ffc8-41aa-9244-7d12061af511" TYPE="swap" 
 /dev/sda7: UUID="a7944df5-3f69-4d94-bab6-a4fe76ee276f" TYPE="ext4" 
 /dev/sr1: UUID="2012-03-22-12-00-00-00" LABEL="Mobile Partner" TYPE="iso9660"

iii) Create Mount point

[blizzard@linuxbox Desktop]$ sudo mkdir -v /mnt/Storage
 mkdir: created directory ‘/mnt/Storage’

iv) Add an /etc/fstab entry for any partition listed in `blkid`
e.g) For entry /dev/sda4: UUID="A80E38E70E38B064" TYPE="ntfs" as shown below:

 #/dev/sda4
 UUID=A80E38E70E38B064 /mnt/Storage ntfs-3g  defaults,umask=077,uid=1000 0 0

3. Install Chromium (Open source version of Google Chrome).

  sudo pacman -S chromium.

4. Install other necessary utilities

    sudo pacman -S vim clementine osmo libreoffice-impress pinta xchm

vim - Text editor.
clementine - Music player.
osmo - Personal calendar, note taking, todo list application.
libreoffice-impress - Presentation tool.
pinta - An alternative to GIMP(Photo editing).
xchm - CHM reader.

5. Automounting Removable disks

Goto Settings->Settings Manager -> Removable Drives and Media -> Mount removable drives when hot-plugged.

6. Installing Dropbox

$ sudo yaourt -S dropbox dropbox-cli

$ dropboxd &

Refer: http://pfigue.github.io/blog/2012/12/24/installing-dropbox-in-arch-linux/

7.Desktop and Workspace Shortcut Keys
By default the shortcut key used to minimize desktop is ctrl+alt+d. You can change the shortcut key by going to Settings->settings Manager->Window Manager and assigning the shortcut key.

The shortcut keys for "workspace+" and "Move window to workspace+" can be assigned based on the user preference. My favorite shortcut keys for "workspace[0-4]" are "<Alt>+[1-4]" and "Move window to workspace[1-4]" are "<super>+[1-4]".

8.Setting shortcut for Terminal

Goto Settings->settings Manager->Keyboard->Application Shortcuts,click Add-> Type "xfce4-terminal" and click "ok", after that assign whatever shortcut you want.

8. Stop Gedit from creating backup copy of files during each save
Goto Gedit Edit->Preferences->Editor-> Uncheck "Create a backup copy of files before saving".

9. Enable Touchpad Horizontal and Vertical scrolling.

 $sudo gedit /etc/X11/xorg.conf.d/50-synaptics.conf

Add the following content

Section "InputClass"
 Identifier "touchpad catchall"
 Driver "synaptics"
 MatchIsTouchpad "on"
 Option "TapButton1" "1"
 Option "VertEdgeScroll" "1"
 Option "HorizEdgeScroll" "1"
EndSection

For details refer Manjaro Wiki and Arch Wiki.

10. Installing Uncomplicated Firewall(ufw)

The launchpad description of ufw says "Ufw stands for Uncomplicated Firewall, and is program for managing a netfilter firewall. It provides a command line interface and aims to be uncomplicated and easy to use.". As the name says the program is easy to install and configure it in manjaro.

  sudo yaourt -S gufw

For detailed configuration steps follow the @handy's post from Manjaro forum and wiki.

For more please refer

Manjaro Forum
Manjaro Wiki
irc.freenode.net - #manjaro

Customizing Manjaro

Some Random Learnings

XNOISE "GstError parsed: configured audiosink audiobin is not working." sudo pacman -S gst-plugins-base
Fix for Multimedia mute button problem
Enable Multimedia Keys support for XNoise Goto Edit-> Preferences -> Settings#Addditional Section->Media Control Keys(Plugin)
UTC/local time zone conflicts when dual booting with windows 7

Manjaro Linux: Arch For People Who Don’t Have Time

2015-01-17T21:38:00.000-08:00

I am manjaro Linux user, and want to share how arch linux based system transform into friendly distro for end user.

This article originaly from make use of site.

There are plenty of things that make Arch Linux highly appealing to users: it’s always up-to-date, it’s a rolling release, and there’s tons of software available for it in its repositories. But what isn’t so appealing is the learning curve and pure difficulty of setting up an Arch system. If you want the best aspects of Arch, without the bad parts, you need Manjaro Linux.

About Manjaro Linux

Manjaro Linux is an Arch-based distribution, meaning that it runs on the same backbone and the same repositories as Arch itself. It also implements the rolling release upgrade model, meaning that you never have to perform a major upgrade from “release 1″ to “release 2″ – just update your packages and you’ll be up-to-date.

However, unlike Arch, it doesn’t require that users build up systems on their own. This isn’t to say that setting up the Arch way is a bad thing, or that it’s too difficult – there’s plenty of documentation for the job. But some people simply don’t have the time, regardless of their skill level.
There’s a lot of fuss among Arch users whether they should support Manjaro, and while some believe that it goes against everything Arch stands for, I think it’s a good option for those who want it.
Also unlike Arch, there are some defaults when it comes to the included software. For example, Manjaro defaults to the Xfce desktop environment (which is lightweight and awesome), although official Openbox,(a very minimal desktop environment), and KDE (a desktop environment with lots of eye candy) versions are available as well. Other desktop environments, such as Gnome, are available as “Community Editions”.

Don’t Fear the Beta!

Technically speaking, Manjaro is still a beta distribution – its version sits at 0.8.9 at time of writing. This shouldn’t push away any potential users. The main reason why Manjaro is still considered a beta distribution is because of the Manjaro additions to the otherwise stable Arch packages that are installed. Things like the Manjaro installer and the Pacman (package manager) graphical frontend are still beta, but everything else that is on the system are stable versions of the software that Arch offers in its repositories.

The User Experience

When you first launch the distribution, you’ll be greeted with a simple message welcoming you to the system. It’ll also provide you with several buttons that send you to certain websites, or the installer. Otherwise, you can close this message and begin exploring.

Manjaro comes with a familar applications menu which initially displays your favorite applications, and then all of the usual categories. Navigation is easy and pretty enjoyable, primarily because it doesn’t get in your way. The default software is pretty much what you’d expect for a regular full Linux distribution – Firefox, Thunderbird, and LibreOffice are all included. This is slightly surprisingly for a Xfce desktop – distributions using it tend to choose more lightweight applications as the defaults. Manjaro also includes GIMP, Steam, and VLC Media Player.
By the way, for those who are exploring and happen to come upon a prompt asking for root permissions, the root password in the live environment is “manjaro”.

Installing Software

Of course, you’ll be interested in knowing how installing packages goes. It worked just fine for me – no issues whatsoever. Manjaro actually uses its own repositories rather than piggybacking off of Arch’s repositories, but it imports Arch’s packages into its own to maintain compatibility. It also allows the project to perform some more testing than Arch does before putting their own “stable” label on the packages. Maintaining compatibility with Arch also allows it to use the Arch User Repository, the one repository where anyone can submit their own packages that aren’t found in the official repositories. Thankfully, but Manjaro’s repositories, as well as the AUR, can be accessed from the graphical package manager that comes with Manjaro — just enable the AUR checkbox and you’ll be looking at all the packages from both sources.

Conclusion

Again, I really think that Manjaro is a great solution for those who like Arch’s updated packages, rolling release updates, and massive repositories but don’t enjoy the time requirements of installing everything from the ground up and maintaining the system via configuration files. If you’re not sure whether you’d be interested in using Manjaro, the best way to find out is to try it out from a USB flash drive or within a virtual machine.
What’s your opinion of Manjaro? Do you think it could grow into a major distribution? Let us know in the comments!

Scary Steam for Linux bug erases all the personal files on your PC

2015-01-17T21:20:00.001-08:00

Original Article From pcworld

If you’re a Steam fan running Linux, the last thing you’ll want to do in the next few days is mess with your Steam files. Users on Valve’s GitHub Steam for Linux page are complaining about a nasty bug that has the potential to wipe out every single personal file on your PC. Even worse, users say the bug will even wipe out documents on USB connected drives. So much for local backups.
The impact on you at home: The obvious implication if you’re running Steam on Linux is to be wary of the program right now. As a precaution, don’t connect any local external hard drives while you’re running Steam. Users complaining of this bug appear to have moved their .steam or ~/.local/share/steam directories, or invoked Steam’s Bash script with the —reset option enabled.

UPDATE: Valve gave us the following statement: "So far we have had a handful of users report this issue, after they manually moved their Steam install. We have not been able to reproduce the reported issue, but we are adding some additional checks to ensure this is not possible while we continue to investigate. If anyone else has experienced this or has more information, they should email linux@valvesoftware.com."

Ouch

Steam’s bug appears to be caused by a line in the Steam.sh Bash script: rm -rf “$STEAMROOT/“*. That command is a basic Bash instruction that tells the computer to remove the STEAMROOT directory and all its sub-directories (folders).
That’s all well and good, but the issue is that if the STEAMROOT folder is not there then the computer interprets the command as rm -rf “/“*, as first reported by Bit-Tech. If you’re not familiar with Bash, that command tells the system to delete everything on your hard drive starting from the root directory.
The saving grace for Linux users is that you can only erase files you have write permissions over. That means the system itself can’t be erased, but pretty much all of a user’s files—including photos and personal documents—would be at risk.
Ironically, the instruction at issue is preceded by a comment from the developer: # Scary!.
Indeed.
Editor's note: This article originally published on 1/16/15, but was updated on 1/17/15 to include Valve's statement.

Porting Code to Python 3 with 2to3

2015-01-17T07:45:00.000-08:00

Diving In

So much has changed between Python 2 and Python 3, there are vanishingly few programs that will run unmodified under both. But don’t despair! To help with this transition, Python 3 comes with a utility script called 2to3, which takes your actual Python 2 source code as input and auto-converts as much as it can to Python 3. Case study: porting chardet to Python 3 describes how to run the 2to3 script, then shows some things it can’t fix automatically. This appendix documents what it can fix automatically.

Unicode string literals

In Python 2, print was a statement. Whatever you wanted to print simply followed the print keyword. In Python 3, print() is a function. Whatever you want to print, pass it to print() like any other function.

Notes	Python 2	Python 3
①	`print`	`print()`
②	`print 1`	`print(1)`
③	`print 1, 2`	`print(1, 2)`
④	`print 1, 2,`	`print(1, 2, end=' ')`
⑤	`print >>sys.stderr, 1, 2, 3`	`print(1, 2, 3, file=sys.stderr)`

To print a blank line, call print() without any arguments.
To print a single value, call print() with one argument.
To print two values separated by a space, call print() with two arguments.
This one is a little tricky. In Python 2, if you ended a print statement with a comma, it would print the values separated by spaces, then print a trailing space, then stop without printing a carriage return. (Technically, it’s a little more complicated than that. The print statement in Python 2 used a now-deprecated attribute called softspace. Instead of printing a space, Python 2 would set sys.stdout.softspace to 1. The space character wasn’t really printed until something else got printed on the same line. If the next print statement printed a carriage return, sys.stdout.softspace would be set to 0 and the space would never be printed. You probably never noticed the difference unless your application was sensitive to the presence or absence of trailing whitespace in print-generated output.) In Python 3, the way to do this is to pass end=' ' as a keyword argument to the print() function. The end argument defaults to '\n' (a carriage return), so overriding it will suppress the carriage return after printing the other arguments.
In Python 2, you could redirect the output to a pipe — like sys.stderr — by using the >>pipe_name syntax. In Python 3, the way to do this is to pass the pipe in the file keyword argument. The file argument defaults to sys.stdout (standard out), so overriding it will output to a different pipe instead.

Unicode string literals

Python 2 had two string types: Unicode strings and non-Unicode strings. Python 3 has one string type: Unicode strings.

Notes	Python 2	Python 3
①	`u'PapayaWhip'`	`'PapayaWhip'`
②	`ur'PapayaWhip\foo'`	`r'PapayaWhip\foo'`

Unicode string literals are simply converted into string literals, which, in Python 3, are always Unicode.
Unicode raw strings (in which Python does not auto-escape backslashes) are converted to raw strings. In Python 3, raw strings are always Unicode.

`unicode()` global function

Python 2 had two global functions to coerce objects into strings: unicode() to coerce them into Unicode strings, and str() to coerce them into non-Unicode strings. Python 3 has only one string type, Unicode strings, so the str() function is all you need. (The unicode() function no longer exists.)

Notes	Python 2	Python 3
	`unicode(anything)`	`str(anything)`

`long` data type

Python 2 had separate int and long types for non-floating-point numbers. An int could not be any larger than sys.maxint, which varied by platform. Longs were defined by appending an L to the end of the number, and they could be, well, longer than ints. In Python 3, there is only one integer type, called int, which mostly behaves like the long type in Python 2. Since there are no longer two types, there is no need for special syntax to distinguish them.
Further reading: PEP 237: Unifying Long Integers and Integers.

Notes	Python 2	Python 3
①	`x = 1000000000000L`	`x = 1000000000000`
②	`x = 0xFFFFFFFFFFFFL`	`x = 0xFFFFFFFFFFFF`
③	`long(x)`	`int(x)`
④	`type(x) is long`	`type(x) is int`
⑤	`isinstance(x, long)`	`isinstance(x, int)`

Base 10 long integer literals become base 10 integer literals.
Base 16 long integer literals become base 16 integer literals.
In Python 3, the old long() function no longer exists, since longs don’t exist. To coerce a variable to an integer, use the int() function.
To check whether a variable is an integer, get its type and compare it to int, not long.
You can also use the isinstance() function to check data types; again, use int, not long, to check for integers.

<> comparison

Python 2 supported <> as a synonym for !=, the not-equals comparison operator. Python 3 supports the != operator, but not <>.

Notes	Python 2	Python 3
①	`if x <> y:`	`if x != y:`
②	`if x <> y <> z:`	`if x != y != z:`

A simple comparison.
A more complex comparison between three values.

`has_key()` dictionary method

In Python 2, dictionaries had a has_key() method to test whether the dictionary had a certain key. In Python 3, this method no longer exists. Instead, you need to use the in operator.

Notes	Python 2	Python 3
①	`a_dictionary.has_key('PapayaWhip')`	`'PapayaWhip' in a_dictionary`
②	`a_dictionary.has_key(x) or a_dictionary.has_key(y)`	`x in a_dictionary or y in a_dictionary`
③	`a_dictionary.has_key(x or y)`	`(x or y) in a_dictionary`
④	`a_dictionary.has_key(x + y)`	`(x + y) in a_dictionary`
⑤	`x + a_dictionary.has_key(y)`	`x + (y in a_dictionary)`

The simplest form.
The in operator takes precedence over the or operator, so there is no need for parentheses around x in a_dictionary or around y in a_dictionary.
On the other hand, you do need parentheses around x or y here, for the same reason — in takes precedence over or. (Note: this code is completely different from the previous line. Python interprets x or y first, which results in either x (if x is true in a boolean context) or y. Then it takes that singular value and checks whether it is a key in a_dictionary.)
The + operator takes precedence over the in operator, so this form technically doesn’t need parentheses around x + y, but 2to3 includes them anyway.
This form definitely needs parentheses around y in a_dictionary, since the + operator takes precedence over the in operator.

Dictionary methods that return lists

In Python 2, many dictionary methods returned lists. The most frequently used methods were keys(), items(), and values(). In Python 3, all of these methods return dynamic views. In some contexts, this is not a problem. If the method’s return value is immediately passed to another function that iterates through the entire sequence, it makes no difference whether the actual type is a list or a view. In other contexts, it matters a great deal. If you were expecting a complete list with individually addressable elements, your code will choke, because views do not support indexing.

Notes	Python 2	Python 3
①	`a_dictionary.keys()`	`list(a_dictionary.keys())`
②	`a_dictionary.items()`	`list(a_dictionary.items())`
③	`a_dictionary.iterkeys()`	`iter(a_dictionary.keys())`
④	`[i for i in a_dictionary.iterkeys()]`	`[i for i in a_dictionary.keys()]`
⑤	`min(a_dictionary.keys())`	no change

2to3 errs on the side of safety, converting the return value from keys() to a static list with the list() function. This will always work, but it will be less efficient than using a view. You should examine the converted code to see if a list is absolutely necessary, or if a view would do.
Another view-to-list conversion, with the items() method. 2to3 will do the same thing with the values() method.
Python 3 does not support the iterkeys() method anymore. Use keys(), and if necessary, convert the view to an iterator with the iter() function.
2to3 recognizes when the iterkeys() method is used inside a list comprehension, and converts it to the keys() method (without wrapping it in an extra call to iter()). This works because views are iterable.
2to3 recognizes that the keys() method is immediately passed to a function which iterates through an entire sequence, so there is no need to convert the return value to a list first. The min() function will happily iterate through the view instead. This applies to min(), max(), sum(), list(), tuple(), set(), sorted(), any(), and all().

Modules that have been renamed or reorganized

Several modules in the Python Standard Library have been renamed. Several other modules which are related to each other have been combined or reorganized to make their association more logical.

`http`

In Python 3, several related HTTP modules have been combined into a single package, http.

Notes	Python 2	Python 3
①	`import httplib`	`import http.client`
②	`import Cookie`	`import http.cookies`
③	`import cookielib`	`import http.cookiejar`
④	`import BaseHTTPServer import SimpleHTTPServer import CGIHttpServer`	`import http.server`

The http.client module implements a low-level library that can request HTTP resources and interpret HTTP responses.
The http.cookies module provides a Pythonic interface to browser cookies that are sent in a Set-Cookie: HTTP header.
The http.cookiejar module manipulates the actual files on disk that popular web browsers use to store cookies.
The http.server module provides a basic HTTP server.

`urllib`

Python 2 had a rat’s nest of overlapping modules to parse, encode, and fetch URLs. In Python 3, these have all been refactored and combined in a single package, urllib.

Notes	Python 2	Python 3
①	`import urllib`	`import urllib.request, urllib.parse, urllib.error`
②	`import urllib2`	`import urllib.request, urllib.error`
③	`import urlparse`	`import urllib.parse`
④	`import robotparser`	`import urllib.robotparser`
⑤	`from urllib import FancyURLopener from urllib import urlencode`	`from urllib.request import FancyURLopener from urllib.parse import urlencode`
⑥	`from urllib2 import Request from urllib2 import HTTPError`	`from urllib.request import Request from urllib.error import HTTPError`

The old urllib module in Python 2 had a variety of functions, including urlopen() for fetching data and splittype(), splithost(), and splituser() for splitting a URL into its constituent parts. These functions have been reorganized more logically within the new urllib package. 2to3 will also change all calls to these functions so they use the new naming scheme.
The old urllib2 module in Python 2 has been folded into the urllib package in Python 3. All your urllib2 favorites — the build_opener() method, Request objects, and HTTPBasicAuthHandler and friends — are still available.
The urllib.parse module in Python 3 contains all the parsing functions from the old urlparse module in Python 2.
The urllib.robotparser module parses robots.txt files.
The FancyURLopener class, which handles HTTP redirects and other status codes, is still available in the new urllib.request module. The urlencode() function has moved to urllib.parse.
The Request object is still available in urllib.request, but constants like HTTPError have been moved to urllib.error.

Did I mention that 2to3 will rewrite your function calls too? For example, if your Python 2 code imports the urllib module and calls urllib.urlopen() to fetch data, 2to3 will fix both the import statement and the function call.

Notes	Python 2	Python 3
	`import urllib print urllib.urlopen('http://diveintopython3.org/').read()`	`import urllib.request, urllib.parse, urllib.error print(urllib.request.urlopen('http://diveintopython3.org/').read())`

`dbm`

All the various DBM clones are now in a single package, dbm. If you need a specific variant like GNU DBM, you can import the appropriate module within the dbm package.

Notes	Python 2	Python 3
	`import dbm`	`import dbm.ndbm`
	`import gdbm`	`import dbm.gnu`
	`import dbhash`	`import dbm.bsd`
	`import dumbdbm`	`import dbm.dumb`
	`import anydbm import whichdb`	`import dbm`

`xmlrpc`

XML-RPC is a lightweight method of performing remote RPC calls over HTTP. The XML-RPC client library and several XML-RPC server implementations are now combined in a single package, xmlrpc.

Notes	Python 2	Python 3
	`import xmlrpclib`	`import xmlrpc.client`
	`import DocXMLRPCServer import SimpleXMLRPCServer`	`import xmlrpc.server`

Other modules

Notes	Python 2	Python 3
①	`try: import cStringIO as StringIO except ImportError: import StringIO`	`import io`
②	`try: import cPickle as pickle except ImportError: import pickle`	`import pickle`
③	`import __builtin__`	`import builtins`
④	`import copy_reg`	`import copyreg`
⑤	`import Queue`	`import queue`
⑥	`import SocketServer`	`import socketserver`
⑦	`import ConfigParser`	`import configparser`
⑧	`import repr`	`import reprlib`
⑨	`import commands`	`import subprocess`

A common idiom in Python 2 was to try to import cStringIO as StringIO, and if that failed, to import StringIO instead. Do not do this in Python 3; the io module does it for you. It will find the fastest implementation available and use it automatically.
A similar idiom was used to import the fastest pickle implementation. Do not do this in Python 3; the pickle module does it for you.
The builtins module contains the global functions, classes, and constants used throughout the Python language. Redefining a function in the builtins module will redefine the global function everywhere. That is exactly as powerful and scary as it sounds.
The copyreg module adds pickle support for custom types defined in C.
The queue module implements a multi-producer, multi-consumer queue.
The socketserver module provides generic base classes for implementing different kinds of socket servers.
The configparser module parses INI-style configuration files.
The reprlib module reimplements the built-in repr() function, with additional controls on how long the representations can be before they are truncated.
The subprocess module allows you to spawn processes, connect to their pipes, and obtain their return codes.

Relative imports within a package

A package is a group of related modules that function as a single entity. In Python 2, when modules within a package need to reference each other, you use import foo or from foo import Bar. The Python 2 interpreter first searches within the current package to find foo.py, and then moves on to the other directories in the Python search path (sys.path). Python 3 works a bit differently. Instead of searching the current package, it goes directly to the Python search path. If you want one module within a package to import another module in the same package, you need to explicitly provide the relative path between the two modules.
Suppose you had this package, with multiple files in the same directory:

chardet/
|
+--__init__.py
|
+--constants.py
|
+--mbcharsetprober.py
|
+--universaldetector.py

Now suppose that universaldetector.py needs to import the entire constants.py file and one class from mbcharsetprober.py. How do you do it?

Notes	Python 2	Python 3
①	`import constants`	`from . import constants`
②	`from mbcharsetprober import MultiByteCharSetProber`	`from .mbcharsetprober import MultiByteCharsetProber`

When you need to import an entire module from elsewhere in your package, use the new from . import syntax. The period is actually a relative path from this file (universaldetector.py) to the file you want to import (constants.py). In this case, they are in the same directory, thus the single period. You can also import from the parent directory (from .. import anothermodule) or a subdirectory.
To import a specific class or function from another module directly into your module’s namespace, prefix the target module with a relative path, minus the trailing slash. In this case, mbcharsetprober.py is in the same directory as universaldetector.py, so the path is a single period. You can also import form the parent directory (from ..anothermodule import AnotherClass) or a subdirectory.

`next()` iterator method

In Python 2, iterators had a next() method which returned the next item in the sequence. That’s still true in Python 3, but there is now also a global next() function that takes an iterator as an argument.

Notes	Python 2	Python 3
①	`anIterator.next()`	`next(anIterator)`
②	`a_function_that_returns_an_iterator().next()`	`next(a_function_that_returns_an_iterator())`
③	`class A: def next(self): pass`	`class A: def __next__(self): pass`
④	`class A: def next(self, x, y): pass`	no change
⑤	`next = 42 for an_iterator in a_sequence_of_iterators: an_iterator.next()`	`next = 42 for an_iterator in a_sequence_of_iterators: an_iterator.__next__()`

In the simplest case, instead of calling an iterator’s next() method, you now pass the iterator itself to the global next() function.
If you have a function that returns an iterator, call the function and pass the result to the next() function. (The 2to3 script is smart enough to convert this properly.)
If you define your own class and mean to use it as an iterator, define the __next__() special method.
If you define your own class and just happen to have a method named next() that takes one or more arguments, 2to3 will not touch it. This class can not be used as an iterator, because its next() method takes arguments.
This one is a bit tricky. If you have a local variable named next, then it takes precedence over the new global next() function. In this case, you need to call the iterator’s special __next__() method to get the next item in the sequence. (Alternatively, you could also refactor the code so the local variable wasn’t named next, but 2to3 will not do that for you automatically.)

`filter()` global function

In Python 2, the filter() function returned a list, the result of filtering a sequence through a function that returned True or False for each item in the sequence. In Python 3, the filter() function returns an iterator, not a list.

Notes	Python 2	Python 3
①	`filter(a_function, a_sequence)`	`list(filter(a_function, a_sequence))`
②	`list(filter(a_function, a_sequence))`	no change
③	`filter(None, a_sequence)`	`[i for i in a_sequence if i]`
④	`for i in filter(None, a_sequence):`	no change
⑤	`[i for i in filter(a_function, a_sequence)]`	no change

In the most basic case, 2to3 will wrap a call to filter() with a call to list(), which simply iterates through its argument and returns a real list.
However, if the call to filter() is already wrapped in list(), 2to3 will do nothing, since the fact that filter() is returning an iterator is irrelevant.
For the special syntax of filter(None, ...), 2to3 will transform the call into a semantically equivalent list comprehension.
In contexts like for loops, which iterate through the entire sequence anyway, no changes are necessary.
Again, no changes are necessary, because the list comprehension will iterate through the entire sequence, and it can do that just as well if filter() returns an iterator as if it returns a list.

`map()` global function

In much the same way as filter(), the map() function now returns an iterator. (In Python 2, it returned a list.)

Notes	Python 2	Python 3
①	`map(a_function, 'PapayaWhip')`	`list(map(a_function, 'PapayaWhip'))`
②	`map(None, 'PapayaWhip')`	`list('PapayaWhip')`
③	`map(lambda x: x+1, range(42))`	`[x+1 for x in range(42)]`
④	`for i in map(a_function, a_sequence):`	no change
⑤	`[i for i in map(a_function, a_sequence)]`	no change

As with filter(), in the most basic case, 2to3 will wrap a call to map() with a call to list().
For the special syntax of map(None, ...), the identity function, 2to3 will convert it to an equivalent call to list().
If the first argument to map() is a lambda function, 2to3 will convert it to an equivalent list comprehension.
In contexts like for loops, which iterate through the entire sequence anyway, no changes are necessary.
Again, no changes are necessary, because the list comprehension will iterate through the entire sequence, and it can do that just as well if map() returns an iterator as if it returns a list.

`reduce()` global function

In Python 3, the reduce() function has been removed from the global namespace and placed in the functools module.

Notes	Python 2	Python 3
	`reduce(a, b, c)`	`from functools import reduce reduce(a, b, c)`

`apply()` global function

Python 2 had a global function called apply(), which took a function f and a list [a, b, c] and returned f(a, b, c). You can accomplish the same thing by calling the function directly and passing it the list of arguments preceded by an asterisk. In Python 3, the apply() function no longer exists; you must use the asterisk notation.

Notes	Python 2	Python 3
①	`apply(a_function, a_list_of_args)`	`a_function(*a_list_of_args)`
②	`apply(a_function, a_list_of_args, a_dictionary_of_named_args)`	`a_function(a_list_of_args, *a_dictionary_of_named_args)`
③	`apply(a_function, a_list_of_args + z)`	`a_function(*a_list_of_args + z)`
④	`apply(aModule.a_function, a_list_of_args)`	`aModule.a_function(*a_list_of_args)`

In the simplest form, you can call a function with a list of arguments (an actual list like [a, b, c]) by prepending the list with an asterisk (*). This is exactly equivalent to the old apply() function in Python 2.
In Python 2, the apply() function could actually take three parameters: a function, a list of arguments, and a dictionary of named arguments. In Python 3, you can accomplish the same thing by prepending the list of arguments with an asterisk (*) and the dictionary of named arguments with two asterisks (**).
The + operator, used here for list concatenation, takes precedence over the * operator, so there is no need for extra parentheses around a_list_of_args + z.
The 2to3 script is smart enough to convert complex apply() calls, including calling functions within imported modules.

`intern()` global function

In Python 2, you could call the intern() function on a string to intern it as a performance optimization. In Python 3, the intern() function has been moved to the sys module.

Notes	Python 2	Python 3
	`intern(aString)`	`sys.intern(aString)`

`exec` statement

Just as the print statement became a function in Python 3, so too has the exec statement. The exec() function takes a string which contains arbitrary Python code and executes it as if it were just another statement or expression. exec() is like eval(), but even more powerful and evil. The eval() function can only evaluate a single expression, but exec() can execute multiple statements, imports, function declarations — essentially an entire Python program in a string.

Notes	Python 2	Python 3
①	`exec codeString`	`exec(codeString)`
②	`exec codeString in a_global_namespace`	`exec(codeString, a_global_namespace)`
③	`exec codeString in a_global_namespace, a_local_namespace`	`exec(codeString, a_global_namespace, a_local_namespace)`

In the simplest form, the 2to3 script simply encloses the code-as-a-string in parentheses, since exec() is now a function instead of a statement.
The old exec statement could take a namespace, a private environment of globals in which the code-as-a-string would be executed. Python 3 can also do this; just pass the namespace as the second argument to the exec() function.
Even fancier, the old exec statement could also take a local namespace (like the variables defined within a function). In Python 3, the exec() function can do that too.

`execfile` statement

Like the old exec statement, the old execfile statement will execute strings as if they were Python code. Where exec took a string, execfile took a filename. In Python 3, the execfile statement has been eliminated. If you really need to take a file of Python code and execute it (but you’re not willing to simply import it), you can accomplish the same thing by opening the file, reading its contents, calling the global compile() function to force the Python interpreter to compile the code, and then call the new exec() function.

Notes	Python 2	Python 3
	`execfile('a_filename')`	`exec(compile(open('a_filename').read(), 'a_filename', 'exec'))`

`repr` literals (backticks)

In Python 2, there was a special syntax of wrapping any object in backticks (like `x`) to get a representation of the object. In Python 3, this capability still exists, but you can no longer use backticks to get it. Instead, use the global repr() function.

Notes	Python 2	Python 3
①	`x`	`repr(x)`
②	`'PapayaWhip' + `2``	`repr('PapayaWhip' + repr(2))`

Remember, x can be anything — a class, a function, a module, a primitive data type, &c. The repr() function works on everything.
In Python 2, backticks could be nested, leading to this sort of confusing (but valid) expression. The 2to3 tool is smart enough to convert this into nested calls to repr().

`try...except` statement

The syntax for catching exceptions has changed slightly between Python 2 and Python 3.

Notes	Python 2	Python 3
①	`try: import mymodule except ImportError, e pass`	`try: import mymodule except ImportError as e: pass`
②	`try: import mymodule except (RuntimeError, ImportError), e pass`	`try: import mymodule except (RuntimeError, ImportError) as e: pass`
③	`try: import mymodule except ImportError: pass`	no change
④	`try: import mymodule except: pass`	no change

Instead of a comma after the exception type, Python 3 uses a new keyword, as.
The as keyword also works for catching multiple types of exceptions at once.
If you catch an exception but don’t actually care about accessing the exception object itself, the syntax is identical between Python 2 and Python 3.
Similarly, if you use a fallback to catch all exceptions, the syntax is identical.

☞You should never use a fallback to catch all exceptions when importing modules (or most other times). Doing so will catch things like KeyboardInterrupt (if the user pressed Ctrl-C to interrupt the program) and can make it more difficult to debug errors.

`raise` statement

The syntax for raising your own exceptions has changed slightly between Python 2 and Python 3.

Notes	Python 2	Python 3
①	`raise MyException`	unchanged
②	`raise MyException, 'error message'`	`raise MyException('error message')`
③	`raise MyException, 'error message', a_traceback`	`raise MyException('error message').with_traceback(a_traceback)`
④	`raise 'error message'`	unsupported

In the simplest form, raising an exception without a custom error message, the syntax is unchanged.
The change becomes noticeable when you want to raise an exception with a custom error message. Python 2 separated the exception class and the message with a comma; Python 3 passes the error message as a parameter.
Python 2 supported a more complex syntax to raise an exception with a custom traceback (stack trace). You can do this in Python 3 as well, but the syntax is quite different.
In Python 2, you could raise an exception with no exception class, just an error message. In Python 3, this is no longer possible. 2to3 will warn you that it was unable to fix this automatically.

`throw` method on generators

In Python 2, generators have a throw() method. Calling a_generator.throw() raises an exception at the point where the generator was paused, then returns the next value yielded by the generator function. In Python 3, this functionality is still available, but the syntax is slightly different.

Notes	Python 2	Python 3
①	`a_generator.throw(MyException)`	no change
②	`a_generator.throw(MyException, 'error message')`	`a_generator.throw(MyException('error message'))`
③	`a_generator.throw('error message')`	unsupported

In the simplest form, a generator throws an exception without a custom error message. In this case, the syntax has not changed between Python 2 and Python 3.
If the generator throws an exception with a custom error message, you need to pass the error string to the exception when you create it.
Python 2 also supported throwing an exception with only a custom error message. Python 3 does not support this, and the 2to3 script will display a warning telling you that you will need to fix this code manually.

`xrange()` global function

In Python 2, there were two ways to get a range of numbers: range(), which returned a list, and xrange(), which returned an iterator. In Python 3, range() returns an iterator, and xrange() doesn’t exist.

Notes	Python 2	Python 3
①	`xrange(10)`	`range(10)`
②	`a_list = range(10)`	`a_list = list(range(10))`
③	`[i for i in xrange(10)]`	`[i for i in range(10)]`
④	`for i in range(10):`	no change
⑤	`sum(range(10))`	no change

In the simplest case, the 2to3 script will simply convert xrange() to range().
If your Python 2 code used range(), the 2to3 script does not know whether you needed a list, or whether an iterator would do. It errs on the side of caution and coerces the return value into a list by calling the list() function.
If the xrange() function was inside a list comprehension, the 2to3 script is clever enough not to wrap the range() function with a call to list(). The list comprehension will work just fine with the iterator that the range() function returns.
Similarly, a for loop will work just fine with an iterator, so there is no need to change anything here.
The sum() function will also work with an iterator, so 2to3 makes no changes here either. Like dictionary methods that return views instead of lists, this applies to min(), max(), sum(), list(), tuple(), set(), sorted(), any(), and all().

`raw_input()` and `input()` global functions

Python 2 had two global functions for asking the user for input on the command line. The first, called input(), expected the user to enter a Python expression (and returned the result). The second, called raw_input(), just returned whatever the user typed. This was wildly confusing for beginners and widely regarded as a “wart” in the language. Python 3 excises this wart by renaming raw_input() to input(), so it works the way everyone naively expects it to work.

Notes	Python 2	Python 3
①	`raw_input()`	`input()`
②	`raw_input('prompt')`	`input('prompt')`
③	`input()`	`eval(input())`

In the simplest form, raw_input() becomes input().
In Python 2, the raw_input() function could take a prompt as a parameter. This has been retained in Python 3.
If you actually need to ask the user for a Python expression to evaluate, use the input() function and pass the result to eval().

`func_*` function attributes

In Python 2, code within functions can access special attributes about the function itself. In Python 3, these special function attributes have been renamed for consistency with other attributes.

Notes	Python 2	Python 3
①	`a_function.func_name`	`a_function.__name__`
②	`a_function.func_doc`	`a_function.__doc__`
③	`a_function.func_defaults`	`a_function.__defaults__`
④	`a_function.func_dict`	`a_function.__dict__`
⑤	`a_function.func_closure`	`a_function.__closure__`
⑥	`a_function.func_globals`	`a_function.__globals__`
⑦	`a_function.func_code`	`a_function.__code__`

The __name__ attribute (previously func_name) contains the function’s name.
The __doc__ attribute (previously func_doc) contains the docstring that you defined in the function’s source code.
The __defaults__ attribute (previously func_defaults) is a tuple containing default argument values for those arguments that have default values.
The __dict__ attribute (previously func_dict) is the namespace supporting arbitrary function attributes.
The __closure__ attribute (previously func_closure) is a tuple of cells that contain bindings for the function’s free variables.
The __globals__ attribute (previously func_globals) is a reference to the global namespace of the module in which the function was defined.
The __code__ attribute (previously func_code) is a code object representing the compiled function body.

`xreadlines()` I/O method

In Python 2, file objects had an xreadlines() method which returned an iterator that would read the file one line at a time. This was useful in for loops, among other places. In fact, it was so useful, later versions of Python 2 added the capability to file objects themselves.
In Python 3, the xreadlines() method no longer exists. 2to3 can fix the simple cases, but some edge cases will require manual intervention.

Notes	Python 2	Python 3
①	`for line in a_file.xreadlines():`	`for line in a_file:`
②	`for line in a_file.xreadlines(5):`	no change (broken)

If you used to call xreadlines() with no arguments, 2to3 will convert it to just the file object. In Python 3, this will accomplish the same thing: read the file one line at a time and execute the body of the for loop.
If you used to call xreadlines() with an argument (the number of lines to read at a time), 2to3 will not fix it, and your code will fail with an AttributeError: '_io.TextIOWrapper' object has no attribute 'xreadlines'. You can manually change xreadlines() to readlines() to get it to work in Python 3. (The readlines() method now returns an iterator, so it is just as efficient as xreadlines() was in Python 2.)

☃

`lambda` functions that take a tuple instead of multiple parameters

In Python 2, you could define anonymous lambda functions which took multiple parameters by defining the function as taking a tuple with a specific number of items. In effect, Python 2 would “unpack” the tuple into named arguments, which you could then reference (by name) within the lambda function. In Python 3, you can still pass a tuple to a lambda function, but the Python interpreter will not unpack the tuple into named arguments. Instead, you will need to reference each argument by its positional index.

Notes	Python 2	Python 3
①	`lambda (x,): x + f(x)`	`lambda x1: x1[0] + f(x1[0])`
②	`lambda (x, y): x + f(y)`	`lambda x_y: x_y[0] + f(x_y[1])`
③	`lambda (x, (y, z)): x + y + z`	`lambda x_y_z: x_y_z[0] + x_y_z[1][0] + x_y_z[1][1]`
④	`lambda x, y, z: x + y + z`	unchanged

If you had defined a lambda function that took a tuple of one item, in Python 3 that would become a lambda with references to x1[0]. The name x1 is autogenerated by the 2to3 script, based on the named arguments in the original tuple.
A lambda function with a two-item tuple (x, y) gets converted to x_y with positional arguments x_y[0] and x_y[1].
The 2to3 script can even handle lambda functions with nested tuples of named arguments. The resulting Python 3 code is a bit unreadable, but it works the same as the old code did in Python 2.
You can define lambda functions that take multiple arguments. Without parentheses around the arguments, Python 2 just treats it as a lambda function with multiple arguments; within the lambda function, you simply reference the arguments by name, just like any other function. This syntax still works in Python 3.

Special method attributes

In Python 2, class methods can reference the class object in which they are defined, as well as the method object itself. im_self is the class instance object; im_func is the function object; im_class is the class of im_self. In Python 3, these special method attributes have been renamed to follow the naming conventions of other attributes.

Notes	Python 2	Python 3
	`aClassInstance.aClassMethod.im_func`	`aClassInstance.aClassMethod.__func__`
	`aClassInstance.aClassMethod.im_self`	`aClassInstance.aClassMethod.__self__`
	`aClassInstance.aClassMethod.im_class`	`aClassInstance.aClassMethod.__self__.__class__`

`nonzero` special method

In Python 2, you could build your own classes that could be used in a boolean context. For example, you could instantiate the class and then use the instance in an if statement. To do this, you defined a special __nonzero__() method which returned True or False, and it was called whenever the instance was used in a boolean context. In Python 3, you can still do this, but the name of the method has changed to __bool__().

Notes	Python 2	Python 3
①	`class A: def __nonzero__(self): pass`	`class A: def __bool__(self): pass`
②	`class A: def __nonzero__(self, x, y): pass`	no change

Instead of __nonzero__(), Python 3 calls the __bool__() method when evaluating an instance in a boolean context.
However, if you have a __nonzero__() method that takes arguments, the 2to3 tool will assume that you were using it for some other purpose, and it will not make any changes.

Octal literals

The syntax for defining base 8 (octal) numbers has changed slightly between Python 2 and Python 3.

Notes	Python 2	Python 3
	`x = 0755`	`x = 0o755`

`sys.maxint`

Due to the integration of the long and int types, the sys.maxint constant is no longer accurate. Because the value may still be useful in determining platform-specific capabilities, it has been retained but renamed as sys.maxsize.

Notes	Python 2	Python 3
①	`from sys import maxint`	`from sys import maxsize`
②	`a_function(sys.maxint)`	`a_function(sys.maxsize)`

maxint becomes maxsize.
Any usage of sys.maxint becomes sys.maxsize.

`callable()` global function

In Python 2, you could check whether an object was callable (like a function) with the global callable() function. In Python 3, this global function has been eliminated. To check whether an object is callable, check for the existence of the __call__() special method.

Notes	Python 2	Python 3
	`callable(anything)`	`hasattr(anything, '__call__')`

`zip()` global function

In Python 2, the global zip() function took any number of sequences and returned a list of tuples. The first tuple contained the first item from each sequence; the second tuple contained the second item from each sequence; and so on. In Python 3, zip() returns an iterator instead of a list.

Notes	Python 2	Python 3
①	`zip(a, b, c)`	`list(zip(a, b, c))`
②	`d.join(zip(a, b, c))`	no change

In the simplest form, you can get the old behavior of the zip() function by wrapping the return value in a call to list(), which will run through the iterator that zip() returns and return a real list of the results.
In contexts that already iterate through all the items of a sequence (such as this call to the join() method), the iterator that zip() returns will work just fine. The 2to3 script is smart enough to detect these cases and make no change to your code.

`StandardError` exception

In Python 2, StandardError was the base class for all built-in exceptions other than StopIteration, GeneratorExit, KeyboardInterrupt, and SystemExit. In Python 3, StandardError has been eliminated; use Exception instead.

Notes	Python 2	Python 3
	`x = StandardError()`	`x = Exception()`
	`x = StandardError(a, b, c)`	`x = Exception(a, b, c)`

`types` module constants

The types module contains a variety of constants to help you determine the type of an object. In Python 2, it contained constants for all primitive types like dict and int. In Python 3, these constants have been eliminated; just use the primitive type name instead.

Notes	Python 2	Python 3
	`types.UnicodeType`	`str`
	`types.StringType`	`bytes`
	`types.DictType`	`dict`
	`types.IntType`	`int`
	`types.LongType`	`int`
	`types.ListType`	`list`
	`types.NoneType`	`type(None)`
	`types.BooleanType`	`bool`
	`types.BufferType`	`memoryview`
	`types.ClassType`	`type`
	`types.ComplexType`	`complex`
	`types.EllipsisType`	`type(Ellipsis)`
	`types.FloatType`	`float`
	`types.ObjectType`	`object`
	`types.NotImplementedType`	`type(NotImplemented)`
	`types.SliceType`	`slice`
	`types.TupleType`	`tuple`
	`types.TypeType`	`type`
	`types.XRangeType`	`range`

☞types.StringType gets mapped to bytes instead of str because a Python 2 “string” (not a Unicode string, just a regular string) is really just a sequence of bytes in a particular character encoding.

`isinstance()` global function

The isinstance() function checks whether an object is an instance of a particular class or type. In Python 2, you could pass a tuple of types, and isinstance() would return True if the object was any of those types. In Python 3, you can still do this, but passing the same type twice is deprecated.

Notes	Python 2	Python 3
	`isinstance(x, (int, float, int))`	`isinstance(x, (int, float))`

`basestring` datatype

Python 2 had two string types: Unicode and non-Unicode. But there was also another type, basestring. It was an abstract type, a superclass for both the str and unicode types. It couldn’t be called or instantiated directly, but you could pass it to the global isinstance() function to check whether an object was either a Unicode or non-Unicode string. In Python 3, there is only one string type, so basestring has no reason to exist.

Notes	Python 2	Python 3
	`isinstance(x, basestring)`	`isinstance(x, str)`

`itertools` module

Python 2.3 introduced the itertools module, which defined variants of the global zip(), map(), and filter() functions that returned iterators instead of lists. In Python 3, those global functions return iterators, so those functions in the itertools module have been eliminated. (There are still lots of useful functions in the itertools module, just not these.)

Notes	Python 2	Python 3
①	`itertools.izip(a, b)`	`zip(a, b)`
②	`itertools.imap(a, b)`	`map(a, b)`
③	`itertools.ifilter(a, b)`	`filter(a, b)`
④	`from itertools import imap, izip, foo`	`from itertools import foo`

Instead of itertools.izip(), just use the global zip() function.
Instead of itertools.imap(), just use map().
itertools.ifilter() becomes filter().
The itertools module still exists in Python 3, it just doesn’t have the functions that have migrated to the global namespace. The 2to3 script is smart enough to remove the specific imports that no longer exist, while leaving other imports intact.

`sys.exc_type`, `sys.exc_value`, `sys.exc_traceback`

Python 2 had three variables in the sys module that you could access while an exception was being handled: sys.exc_type, sys.exc_value, sys.exc_traceback. (Actually, these date all the way back to Python 1.) Ever since Python 1.5, these variables have been deprecated in favor of sys.exc_info(), which is a function that returns a tuple containing those three values. In Python 3, these individual variables have finally gone away; you must use the sys.exc_info() function.

Notes	Python 2	Python 3
	`sys.exc_type`	`sys.exc_info()[0]`
	`sys.exc_value`	`sys.exc_info()[1]`
	`sys.exc_traceback`	`sys.exc_info()[2]`

List comprehensions over tuples

In Python 2, if you wanted to code a list comprehension that iterated over a tuple, you did not need to put parentheses around the tuple values. In Python 3, explicit parentheses are required.

Notes	Python 2	Python 3
	`[i for i in 1, 2]`	`[i for i in (1, 2)]`

`os.getcwdu()` function

Python 2 had a function named os.getcwd(), which returned the current working directory as a (non-Unicode) string. Because modern file systems can handle directory names in any character encoding, Python 2.3 introduced os.getcwdu(). The os.getcwdu() function returned the current working directory as a Unicode string. In Python 3, there is only one string type (Unicode), so os.getcwd() is all you need.

Notes	Python 2	Python 3
	`os.getcwdu()`	`os.getcwd()`

Metaclasses

In Python 2, you could create metaclasses either by defining the metaclass argument in the class declaration, or by defining a special class-level __metaclass__ attribute. In Python 3, the class-level attribute has been eliminated.

Notes Python 2 Python 3

①

Notes	Python 2	Python 3
①	`class C(metaclass=PapayaMeta): pass`	unchanged
②	`class Whip: __metaclass__ = PapayaMeta`	`class Whip(metaclass=PapayaMeta): pass`
③	`class C(Whipper, Beater): __metaclass__ = PapayaMeta`	`class C(Whipper, Beater, metaclass=PapayaMeta): pass`

class C(metaclass=PapayaMeta):
    pass

unchanged

②

class Whip:
    __metaclass__ = PapayaMeta

class Whip(metaclass=PapayaMeta):
    pass

③

class C(Whipper, Beater):
    __metaclass__ = PapayaMeta

class C(Whipper, Beater, metaclass=PapayaMeta):
    pass

Declaring the metaclass in the class declaration worked in Python 2, and it still works the same in Python 3.
Declaring the metaclass in a class attribute worked in Python 2, but doesn’t work in Python 3.
The 2to3 script is smart enough to construct a valid class declaration, even if the class is inherited from one or more base classes.

Matters of style

The rest of the “fixes” listed here aren’t really fixes per se. That is, the things they change are matters of style, not substance. They work just as well in Python 3 as they do in Python 2, but the developers of Python have a vested interest in making Python code as uniform as possible. To that end, there is an official Python style guide which outlines — in excruciating detail — all sorts of nitpicky details that you almost certainly don’t care about. And given that 2to3 provides such a great infrastructure for converting Python code from one thing to another, the authors took it upon themselves to add a few optional features to improve the readability of your Python programs.

`set()` literals (explicit)

In Python 2, the only way to define a literal set in your code was to call set(a_sequence). This still works in Python 3, but a clearer way of doing it is to use the new set literal notation: curly braces. This works for everything except empty sets, because dictionaries also use curly braces, so {} is an empty dictionary, not an empty set.

☞The 2to3 script will not fix set() literals by default. To enable this fix, specify -f set_literal on the command line when you call 2to3.

Notes	Before	After
	`set([1, 2, 3])`	`{1, 2, 3}`
	`set((1, 2, 3))`	`{1, 2, 3}`
	`set([i for i in a_sequence])`	`{i for i in a_sequence}`

`buffer()` global function (explicit)

Python objects implemented in C can export a “buffer interface,” which allows other Python code to directly read and write a block of memory. (That is exactly as powerful and scary as it sounds.) In Python 3, buffer() has been renamed to memoryview(). (It’s a little more complicated than that, but you can almost certainly ignore the differences.)

☞The 2to3 script will not fix the buffer() function by default. To enable this fix, specify -f buffer on the command line when you call 2to3.

Notes	Before	After
	`x = buffer(y)`	`x = memoryview(y)`

Whitespace around commas (explicit)

Despite being draconian about whitespace for indenting and outdenting, Python is actually quite liberal about whitespace in other areas. Within lists, tuples, sets, and dictionaries, whitespace can appear before and after commas with no ill effects. However, the Python style guide states that commas should be preceded by zero spaces and followed by one. Although this is purely an aesthetic issue (the code works either way, in both Python 2 and Python 3), the 2to3 script can optionally fix this for you.

☞The 2to3 script will not fix whitespace around commas by default. To enable this fix, specify -f wscomma on the command line when you call 2to3.

Notes	Before	After
	`a ,b`	`a, b`
	`{a :b}`	`{a: b}`

Common idioms (explicit)

There were a number of common idioms built up in the Python community. Some, like the while 1: loop, date back to Python 1. (Python didn’t have a true boolean type until version 2.3, so developers used 1 and 0 instead.) Modern Python programmers should train their brains to use modern versions of these idioms instead.

☞The 2to3 script will not fix common idioms by default. To enable this fix, specify -f idioms on the command line when you call 2to3.

Notes	Before	After
	`while 1: do_stuff()`	`while True: do_stuff()`
	`type(x) == T`	`isinstance(x, T)`
	`type(x) is T`	`isinstance(x, T)`
	`a_list = list(a_sequence) a_list.sort() do_stuff(a_list)`	`a_list = sorted(a_sequence) do_stuff(a_list)`

Recursive File and Directory Manipulation in Python (Part 3)

2015-01-16T07:27:00.000-08:00

Repost from python central.

In Part 2 of this series we expanded our file-searching script to be able to search for multiple file extensions under a tree, and to write the results (all paths to files of matching extensions found) to a log file. Now that we've come to the final part of the series, we'll add more functionality (in the form of functions) to our script to be able to move, copy, and even delete the results of the search.
Before looking at the move/copy/delete functions, we'll first take the subroutine of logging the results to file and encapsulate that in a function also. The following is what that part of our script looked like before:

# The header in our logfile

loghead = 'Search log from filefind for files in {}\n\n'.format(

os.path.realpath(topdir))

# The body of our log file

logbody = ''

# Loop through results

for searchResult in found:

# Concatenate the result from the found dict

logbody += "<< Results with the extension '%s' >>" % searchResult

logbody += '\n\n%s\n\n' % '\n'.join(found[searchResult])

# Write results to the logfile

with open(logname, 'w') as logfile:

logfile.write(loghead)

logfile.write(logbody)

In order to put it into a function definition, we simply put the definition statement above, adding the appropriate arguments, and indent the rest accordingly (note how found changes to results and logname to logpath):

# Logging results for findfile

def logres(logpath, results):

# The header in our logfile

loghead = 'Search log from filefind for files in {}\n\n'.format(

os.path.realpath(topdir))

# The body of our log file

logbody = ''

# Loop through results

for searchResult in results:

# Concatenate the result from the results dict

logbody += "<< Results with the extension '%s' >>" % searchResult

logbody += '\n\n%s\n\n' % '\n'.join(results[searchResult])

# write results to the logfile

with open(logpath, 'w') as logfile:

logfile.write(loghead)

logfile.write(logbody)

For error-reporting purposes, we'll also define a small function to write an error log, and we'll see why shortly. Here is the function for that, it takes in 3 arguments and writes a list of strings to an error log:

def logerr(logpath, errlist, act):

loghead = 'List of files that produced errors when attempting to %s:\n\n' % act

logbody = '\n'.join(errlist)

with open(logpath, 'w') as log:

log.write(loghead + logbody)

With both of our logging functions defined, we'll now write our functions to perform batch operations on our results list from our file search. We will first look at how to perform a batch move on the files found from their original locations to a target directory. The function we will use to actually move the files is the move function from the shutil module (imagine that :P), so we'll want to add this statement to the beginning of our script:

1 2	# We'll use copy2 later from shutil import move, copy2

For our function definition, instead of acting directly on the found variable in the script, we'll have our method take in a dictionary of results and act on it. It also needs the path to the directory to move them to and an error log path, if we want it. It will also need a variable to store the errors (a list of path strings to files):

# Moving results

def batchmove(results, dest, errlog=None):

# List of results that produce errors

errors = []

Before writing the rest of the function definition, a few important notes on the move function—firstly, this function will move the source argument to the destination of the same type. This means that if the source path is a directory, the destination will also be a directory, and likewise for files. Secondly, if the destination exists and is a file, then the source must be a file or the function will fail. In other words, if the destination is a directory, the source (either file or directory) will be moved into the destination directory, but if the destination is a file, the source may only be a file (we can't move a directory into a file). With that said, all we need do is make sure the dest argument to batchmove is an existing directory, so we'll use a try statement after testing:

# Make sure dest is a directory!

if os.path.isfile(dest):

print("The move destination '%s' already exists as a file!" % dest)

exit(input('Press enter to exit...'))

elif not os.path.isdir(dest):

try:

os.mkdir(dest)

except:

print("Unable to create '%s' folder!" % dest)

exit(input('Press enter to exit...'))

else:

print("'%s' folder created" % dest)

This way, if the move fails, it will alert the user and wait before exiting. With our destination directory checked, we can add the core of the function: looping through results and moving each file. Here is the loop:

# Loop through results, moving every file to dest directory

for paths in results.values():

for path in paths:

path = os.path.realpath(path)

try:

# Move file to dest

move(path, dest)

except:

errors.append(path)

print('File move complete')

The keys in results are simply the file extensions searched for, so only the values are needed, and for each path in the current paths list in the values, we try to move it to our destination. If the move fails, the path is added to the error list. When the loop completes, a message is printed to the standard output.
After the loop completes, we'll want to log any errors encountered with our logerr method like so:

# Log errors, if any

if errlog and errors:

logerr(errlog, errors, 'move')

print("Check '%s' for errors." % errlog)

Finally, we'll have the script print a final message and exit:

1	exit(input('Press enter to exit...'))

Putting it all together, here is what our batchmove function looks like:

# Moving results

def batchmove(results, dest, errlog=None):

# List of results that produce errors

errors = []

# Make sure dest is a directory!

if os.path.isfile(dest):

print("The move destination '%s' already exists as a file!" % dest)

exit(input('Press enter to exit...'))

elif not os.path.isdir(dest):

try:

os.mkdir(dest)

except:

print("Unable to create '%s' folder!" % dest)

exit(input('Press enter to exit...'))

else:

print("'%s' folder created" % dest)

# Loop through results, moving every file to dest directory

for paths in results.values():

for path in paths:

path = os.path.realpath(path)

try:

# move file to dest

move(path, dest)

except:

errors.append(path)

print('File move complete')

# log errors, if any

if errlog and errors:

logerr(errlog, errors, 'move')

print("Check '%s' for errors." % errlog)

exit(input('Press enter to exit...'))

Now that we have the batchmove function, in order to define the batchcopy function, we need only change the function call of the innermost loop to copy2 (and messages accordingly), so the full definition would look like this:

# Copying results

def batchcopy(results, dest, errlog=None):

# List of results that produce errors

errors = []

# Make sure dest is a directory!

if os.path.isfile(dest):

print("The copy destination '%s' already exists as a file!" % dest)

exit(input('Press enter to exit...'))

elif not os.path.isdir(dest):

try:

os.mkdir(dest)

except:

print("Unable to create '%s' folder!" % dest)

exit(input('Press enter to exit...'))

else:

print("'%s' folder created" % dest)

# Loop thru results, copying every file to dest directory

for paths in results.values():

for path in paths:

path = os.path.realpath(path)

try:

# copy file to dest

copy2(path, dest)

except:

errors.append(path)

print('File copying complete')

# Log errors, if any

if errlog and errors:

logerr(errlog, errors, 'copy')

exit(input('Press enter to exit...'))

These two we've defined should be useful enough not to need a deletion function, but if we wanted one, we would only have to remove the dest check from batchmove and change the inner loop function call to os.remove like so:

# Deleting results -- USE WITH CAUTION!

def batchdel(results, errlog=None):

# List of results that produce errors

errors = []

# Loop thru results, DELETING every file!

for paths in results.values():

for path in paths:

path = os.path.realpath(path)

try:

# Delete file

os.remove(path)

except:

errors.append(path)

print('File deletion complete')

# Log errors, if any

if errlog and errors:

logerr(errlog, errors, 'delete')

exit(input('Press enter to exit...'))

This is just to show how we'd implement the deletion subroutine, but actually it's not recommended, because whatever files Python deletes are deleted permanently (not sent to the Recycle Bin!). It would be safer simply to move the files to a folder with batchmove and delete them from there, but of course the choice is up to you :). Now that we've defined these functions, all we'd need to do to utilize them is call them after the search loop with found as the results argument, and whatever paths to log files we want accordingly, so finding and moving our files around will be a breeze even if we don't know where they are!

Recursive File and Directory Manipulation in Python (Part 2)

2015-01-16T07:25:00.000-08:00

Repost from python central.

In Part 1 we looked at how to use the os.path.walk and os.walk methods to find and list files of a certain extension under a directory tree. The former function is only present in the Python 2.x, and the latter is available in both Python 2.x and Python 3.x. As we saw in the previous article, the os.path.walk method can be awkward to use, so from now on we'll stick to the os.walk method, this way the script will be simpler and compatible with both branches.
In Part 1 our script traversed all the folders under the topdir variable, but only found files of one extension. Let's now expand that to find files of multiple extensions in select folders under the topdir path. We'll first search for files of three different file extensions: .txt, .pdf, and .doc. Our extens variable will be a list of strings instead of one:

1	extens = ['txt', 'pdf', 'doc']

The . character is not included in these strings as it was in the ext variable as before, and we'll see why shortly. In order to save the results (the file names) we'll use a dictionary with the extensions as keys:

1 2	# List comprehension form of instantiation found = { x: [] for x in extens }

The other variables will remain the same for now; however, the script file itself will be placed in (and will execute from) my system's “Documents” folder, so the topdir variable will become that path.
Previously we tested for the extension with the str.endswith method. If we were to use it again we'd have to loop through the extension list and test with endswith for every file name, but instead we'll use a slightly different approach. For each file stepped on during the walk we'll extract the extension and then test for membership in extens. Here's how we'll extract it:

for name in files:

# Split the name by '.' & get the last element

ext = name.lower().rsplit(“.”, 1)[-1]

As with the previous part, we put this line inside the for loop that interates over the files list returned by os.walk. With this line we combined three operations: changing the case of the file name, splitting it, and extracting an element. Calling str.lower on the filename changes it to lowercase. The same as all strings in extens. Calling str.rsplit on name then splits the string into a list (from the right) with the first argument . delimiting it, and only making as many splits as the second argument (1). The third part ([-1]) retrieves the last element of the list—we use this instead of an index of 1 because if no splits are made (if there is no . in name), no IndexError will be raised.
Now that we've extracted the extension of name (if any), we can test to see if it's in our list of extensions:

1	if ext in extens:

This is why . doesn't precede any of the extension names in extens, because ext won't ever have one. If the condition is true, we'll add the name found to our found dictionary:

1 2	if ext in extens: found[ext].append(os.path.join(dirpath, name))

The above line will append the result path (dirpath joined to name returned from os.walk) to the list at the ext key in found. Now that we have changed the search extensions and list of results we also have to adjust how to save the results to our log file.
In the previous version (using os.walk) we simply opened a file at logname and wrote the results to the file. In this version we must loop through multiple categories in the results, one for each extension. We'll concatenate each result list in found to our results string, which we'll now identify as logbody. We'll also add a small header to the logfile, loghead:

# The header in our logfile

loghead = 'Search log from filefind for files in {}\n\n'.format(os.path.realpath(topdir))

# The body of our log file

logbody = ''

# Loop through results

for search in found:

# Concatenate the result from the found dict

logbody += "<< Results with the extension '%s' >>" % search

# Use str.join to turn the list at search into a str

logbody += '\n\n%s\n\n' % '\n'.join(found[search])

The format of the results can be whatever or however you like, but it is important that we loop through all of the results to get the full log. After the logbody is complete, we can write our log file:

# Write results to the logfile

with open(logname, 'w') as logfile:

logfile.write('%s\n%s' % (loghead, logbody))

Note: if any names/paths in the solution contain non-ASCII characters, we would have to change the open mode to wb and decode loghead and logbody (or encode if in Python 3.x) in order to save the logfile successfully.
Now we are finally ready to test our script. Running it on my system yields this log file (shortened):

Search log from filefind for files in C:\Python27\Lib\site-packages

<< Results with the extension 'pdf' >>

.\GPL_Full.pdf

.\beautifulsoup4-4.1.3\doc\rfc2425-v2.1.pdf

.\beautifulsoup4-4.1.3\doc\rfc2426-v3.0.pdf

<< Results with the extension 'txt' >>

.\README.txt

.\soup.txt

.\beautifulsoup4-4.1.3\AUTHORS.txt

.\beautifulsoup4-4.1.3\COPYING.txt

...

.\wx-2.8-msw-unicode\docs\CHANGES.txt

.\wx-2.8-msw-unicode\docs\MigrationGuide.txt

.\wx-2.8-msw-unicode\docs\README.win32.txt

...

.\wx-2.8-msw-unicode\wx\tools\XRCed\TODO.txt

<< Results with the extension 'doc' >>

This log tells us that in the C:\Python27\Lib\site-packages directory there are a few PDF files, many text files, and no ".doc" or Word files. It seems to work fine, and the extension search list can be changed easily, but what if we don't want to search in the "docs" directory under the wx-2.8-msw-unicode tree? After all, we know there will probably be lots of text files in there. We can ignore this directory by modifying the dirnames list in-place in the main walk loop. Because we might want to ignore more than one directory, we'll keep a list of them (this will come before the loop of course):

1 2	# Directories to ignore ignore = ['docs', 'doc']

Now that we have the list, we'll add this small loop inside the main walk loop (and before the loop over the file names):

# Remove directories in ignore

# Directory names must match exactly!

for idir in ignore:

if idir in dirnames:

dirnames.remove(idir)

This will edit dirnames in-place, so that the next iteration of the walk loop will no longer include the folders named in ignore. The full script with the new walk loop now looks like this:

import os

# The top argument for name in files

topdir = '.'

extens = ['txt', 'pdf', 'doc'] # the extensions to search for

found = {x: [] for x in extens} # lists of found files

# Directories to ignore

ignore = ['docs', 'doc']

logname = "findfiletypes.log"

print('Beginning search for files in %s' % os.path.realpath(topdir))

# Walk the tree

for dirpath, dirnames, files in os.walk(topdir):

# Remove directories in ignore

# directory names must match exactly!

for idir in ignore:

if idir in dirnames:

dirnames.remove(idir)

# Loop through the file names for the current step

for name in files:

# Split the name by '.' & get the last element

ext = name.lower().rsplit('.', 1)[-1]

# Save the full name if ext matches

if ext in extens:

found[ext].append(os.path.join(dirpath, name))

# The header in our logfile

loghead = 'Search log from filefind for files in {}\n\n'.format(

os.path.realpath(topdir)

)

# The body of our log file

logbody = ''

# loop thru results

for search in found:

# Concatenate the result from the found dict

logbody += "<< Results with the extension '%s' >>" % search

logbody += '\n\n%s\n\n' % '\n'.join(found[search])

# Write results to the logfile

with open(logname, 'w') as logfile:

logfile.write('%s\n%s' % (loghead, logbody))

With our new ignored files element, the log file turns out looking like this (shortened):
Search log from filefind for files in C:\Python27\Lib\site-packages

<< Results with the extension 'pdf' >>

.\GPL_Full.pdf

<< Results with the extension 'txt' >>

.\README.txt

.\soup.txt

.\beautifulsoup4-4.1.3\AUTHORS.txt

.\beautifulsoup4-4.1.3\COPYING.txt

...

.\beautifulsoup4-4.1.3\scripts\demonstration_markup.txt

.\wx-2.8-msw-unicode\wx\lib\editor\README.txt

...

.\wx-2.8-msw-unicode\wx\tools\XRCed\TODO.txt

<< Results with the extension 'doc' >>

Our ignore list worked just as we wanted it to, cutting out the full tree under the "docs" directory in wx-...-unicode. We can also see that the other ignore directory ("doc") cut out the other two PDF files from our PDF results, and for both directories we didn't need to name the full path (because the name won't be the full path in dirnames anyway). This can be convenient but always remember that this method will prune out any part of the tree under any name that matches one in the ignore list (to avoid this try using the dirpath and dirnames together to specify full paths to ignore, if you don't mind going through the trouble of naming the full path!).
Now that we've completed this version of our file/directory manipulation script, we can search for multiple file extensions under any tree fast and have a record of all those found with just a double-click. This is great if we simply want to know where all the files exist, but since they likely will not all be in the same folder, if we wanted to move/copy them all to the same folder or do something else with all of them simultaneously, looking through each line of the log file would not be preferable. This is why in the next part we'll look at how to upgrade our script to move, copy/backup, or alternatively erase all the files we are looking for.

Recursive File and Directory Manipulation in Python (Part 1)

2015-01-16T07:19:00.000-08:00

Repost from python central.

If you are looking to utilize Python to manipulate your directory tree or files on your system, there are many tools to help, including Python's standard os module. The following is a simple/basic recipe to assist with finding certain files on your system by file extension.
If you have had the experience of "losing" a file in your system where you don't remember its location and are not even sure of its name, though you remember its type, this is where you might find this recipe useful.
In a way this recipe is a combination of How to Traverse a Directory Tree and Recursive Directory Traversal in Python: Make a list of your movies!, but we'll tweak it a bit and build upon it in part two.
To script this task, we can use the walk function in the os.path module or the walk function in the os module (using Python version 2.x or Python 3.x, respectively).

Recursion with os.path.walk in Python 2.x

The os.path.walk function takes 3 arguments:

arg - an arbitrary (but mandatory) argument.
visit - a function to execute upon each iteration.
top - the top of the directory tree to walk.

It then walks through the directory tree under the top, performing the function at every step. Let's examine the function (which we'll define as "step") we use to print the path names of the files under top that have the file extension we can provide through arg.
Here is the definition of step:

def step(ext, dirname, names):

ext = ext.lower()

for name in names:

if name.lower().endswith(ext):

print os.path.join(dirname, name)

Now let's break it down line-by-line, but first it's very important to point out that the arguments given to step are being passed by directly the os.path.walk function, not by the user. The three arguments that walk passes on each iteration are:

ext - the arbitrary argument given to os.path.walk.
dirname - the directory name for that iteration.
names - the names of all files under dirname.

The first line of our step function is of course our declaration of the function, and inclusion of the default arguments that will be passed directly by os.path.walk.
The second line ensures our ext string is lowercase. The third line begins our loop of the argument names, which is a list type. The fourth line is how we retrieve the names of files with the extension we want, using the string method endswith to test for a suffix.
The final line prints the path of any file that passes the suffix (extension) test, concatenating the dirname argument to the name (with the appropriate system-dependent separator).
Now after combining our step function with the walk function, the script looks something like this:

# We only need to import this module

import os.path

# The top argument for walk. The

# Python27/Lib/site-packages folder in my case

topdir = '.'

# The arg argument for walk, and subsequently ext for step

exten = '.txt'

def step(ext, dirname, names):

ext = ext.lower()

for name in names:

if name.lower().endswith(ext):

print(os.path.join(dirname, name))

# Start the walk

os.path.walk(topdir, step, exten)

For my system I have wx_py installed in the site-packages for Python 2.7, the output looks like this:

.\README.txt

.\wx-2.8-msw-unicode\docs\CHANGES.txt

.\wx-2.8-msw-unicode\docs\MigrationGuide.txt

.\wx-2.8-msw-unicode\docs\README.win32.txt

......

.\wx-2.8-msw-unicode\wx\tools\XRCed\TODO.txt</blockquote>

Recursion with os.walk in Python 3.x

Now let's do the same using Python 3.x.
The os.walk function in Python 3.x works differently, providing a few more options than the other. It takes 4 arguments, and only the first is mandatory. The arguments (and their default values) in order are:

top

topdown(=True)

boolean

onerror(=None)

followlinks(=False)

boolean

The only one we are concerned with for now is the first. Aside from the arguments, perhaps the biggest difference in the two versions of the walk function is that the Python 2.x version automatically iterates over the directory tree, while the Python 3.x version produces a generator function. This means that the Python 3.x version will only go to the next iteration when we tell it to, and the way we will do that is with a loop.
Instead of defining a separate function to call as with step we will write the os.walk generator into the loop that went into the step function. Like the Python 2.x version, os.walk produces 3 values we can use for every iteration (the directory path, the directory names, and the filenames), but this time they are in the form of a 3-tuple, so we have to adjust our method accordingly. Other than that we won't change the extension suffix test at all, so the script ends up looking something like this:

import os

# The top argument for walk

topdir = '.'

# The extension to search for

exten = '.txt'

for dirpath, dirnames, files in os.walk(topdir):

for name in files:

if name.lower().endswith(exten):

print(os.path.join(dirpath, name))

Because my system's Python32/Lib/site-packages folder contains nothing special, the output for this one ends up being just:

1	.\README.txt

This will work the same way for whatever the "topdir" and "exten" strings are set to; however, this script simply prints the filenames to the window (in our examples the Python IDLE window), and if there are many files to print this leaves our interpreter (or shell) window many rows high—kind of a pain to scroll through. If we know that this is the case, it would be much easier to write the results to a text file we can look at anytime. We can do so easily if we incorporate a with statement (as in Reading and Writing Files in Python) like so:

1 2	with open(logpath, 'a') as logfile: logfile.write('%s\n' % os.path.join(dirname, name))

Let's see first how to incorporate it into the version Python 2.x script:

# We only need to import this module

import os.path

# The top argument for walk. The

# Python27/Lib/site-packages folder in my case.

topdir = '.'

# The arg argument for walk, and subsequently ext for step

exten = '.txt'

logname = 'findfiletype.log'

def step((ext, logpath), dirname, names):

ext = ext.lower()

for name in names:

if name.lower().endswith(ext):

# Instead of printing, open up the log file for appending

with open(logpath, 'a') as logfile:

logfile.write('%s\n' % os.path.join(dirname, name))

# Change the arg to a tuple containing the file

# extension and the log file name. Start the walk.

os.path.walk(topdir, step, (exten, logname))

As we can see above, not much has changed except for the third variable logname, and the third argument to os.path.walk. The with statement has replaced the print statement. Because of the nature of os.path.walk function, step is required to open up the log file, write to it, and close it every time it finds a file name; this won't cause any errors but is a bit awkward. We must also note that because the log file is opened up in append mode, it will not overwrite a log file that exists already, it will only append to the file. This means if we run the script 2 or more times in a row without changing the logname, the results for each run will be added to the same file, which may not be desirable.
The modified version Python 3.x script is much less awkward:

import os

# The top argument for walk

topdir = '.'

# The extension to search for

exten = '.txt'

logname = 'findfiletype.log'

# What will be logged

results = str()

for dirpath, dirnames, files in os.walk(topdir):

for name in files:

if name.lower().endswith(exten):

# Save to results string instead of printing

results += '%s\n' % os.path.join(dirpath, name)

# Write results to logfile

with open(logname, 'w') as logfile:

logfile.write(results)

In this version the name of each found file is appended to the results string, and then when the search is over, the results are written to the log file. Unlike the Python 2.x version, the log file is opened in write mode, meaning any existing log file will be overwritten. In both cases the log file will be written in the same directory as the script (because we didn't specify a full path name).
With that we have a simple script to find files of a certain extension under a file tree and log those results. In the parts that follow we'll build upon this adding functionality to search for multiple file types, avoid certain paths, and more.

30 Python Language Features and Tricks You May Not Know About

2015-01-15T19:24:00.000-08:00

I re post this one, from here, when I surfing other python programmer and developer.

Sometimes we don't know about secret of python code, it should be find by self exploring or should be discuss with other python programmer. :-) soo we will get new knowledges. Don't forget to share to other programmer if you have good tips or trick to tweak something. Let's make good knowledges share media with wathever you want.

1.1 Unpacking

>>> a, b, c = 1, 2, 3
>>> a, b, c
(1, 2, 3)
>>> a, b, c = [1, 2, 3]
>>> a, b, c
(1, 2, 3)
>>> a, b, c = (2 * i + 1 for i in range(3))
>>> a, b, c
(1, 3, 5)
>>> a, (b, c), d = [1, (2, 3), 4]
>>> a
1
>>> b
2
>>> c
3
>>> d
4

1.2 Unpacking for swapping variables

>>> a, b = 1, 2
>>> a, b = b, a
>>> a, b
(2, 1)

1.3 Extended unpacking (Python 3 only)

>>> a, *b, c = [1, 2, 3, 4, 5]
>>> a
1
>>> b
[2, 3, 4]
>>> c
5

1.4 Negative indexing

>>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> a[-1]
10
>>> a[-3]
8

1.5 List slices (`a[start:end]`)

>>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> a[2:8]
[2, 3, 4, 5, 6, 7]

1.6 List slices with negative indexing

>>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> a[-4:-2]
[7, 8]

1.7 List slices with step (`a[start:end:step]`)

>>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> a[::2]
[0, 2, 4, 6, 8, 10]
>>> a[::3]
[0, 3, 6, 9]
>>> a[2:8:2]
[2, 4, 6]

1.8 List slices with negative step

>>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> a[::-1]
[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
>>> a[::-2]
[10, 8, 6, 4, 2, 0]

1.9 List slice assignment

>>> a = [1, 2, 3, 4, 5]
>>> a[2:3] = [0, 0]
>>> a
[1, 2, 0, 0, 4, 5]
>>> a[1:1] = [8, 9]
>>> a
[1, 8, 9, 2, 0, 0, 4, 5]
>>> a[1:-1] = []
>>> a
[1, 5]

1.10 Naming slices (`slice(start, end, step)`)

>>> a = [0, 1, 2, 3, 4, 5]
>>> LASTTHREE = slice(-3, None)
>>> LASTTHREE
slice(-3, None, None)
>>> a[LASTTHREE]
[3, 4, 5]

1.11 Iterating over list index and value pairs (`enumerate`)

>>> a = ['Hello', 'world', '!']
>>> for i, x in enumerate(a):
...     print '{}: {}'.format(i, x)
...
0: Hello
1: world
2: !

1.12 Iterating over dictionary key and value pairs (`dict.iteritems`)

>>> m = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
>>> for k, v in m.iteritems():
...     print '{}: {}'.format(k, v)
...
a: 1
c: 3
b: 2
d: 4

Note: use dict.items in Python 3.

1.13 Zipping and unzipping lists and iterables

>>> a = [1, 2, 3]
>>> b = ['a', 'b', 'c']
>>> z = zip(a, b)
>>> z
[(1, 'a'), (2, 'b'), (3, 'c')]
>>> zip(*z)
[(1, 2, 3), ('a', 'b', 'c')]

1.14 Grouping adjacent list items using zip

>>> a = [1, 2, 3, 4, 5, 6]

>>> # Using iterators
>>> group_adjacent = lambda a, k: zip(*([iter(a)] * k))
>>> group_adjacent(a, 3)
[(1, 2, 3), (4, 5, 6)]
>>> group_adjacent(a, 2)
[(1, 2), (3, 4), (5, 6)]
>>> group_adjacent(a, 1)
[(1,), (2,), (3,), (4,), (5,), (6,)]


>>> # Using slices
>>> from itertools import islice
>>> group_adjacent = lambda a, k: zip(*(islice(a, i, None, k) for i in range(k)))
>>> group_adjacent(a, 3)
[(1, 2, 3), (4, 5, 6)]
>>> group_adjacent(a, 2)
[(1, 2), (3, 4), (5, 6)]
>>> group_adjacent(a, 1)
[(1,), (2,), (3,), (4,), (5,), (6,)]

1.15 Sliding windows (n-grams) using zip and iterators

>>> from itertools import islice
>>> def n_grams(a, n):
...     z = (islice(a, i, None) for i in range(n))
...     return zip(*z)
...
>>> a = [1, 2, 3, 4, 5, 6]
>>> n_grams(a, 3)
[(1, 2, 3), (2, 3, 4), (3, 4, 5), (4, 5, 6)]
>>> n_grams(a, 2)
[(1, 2), (2, 3), (3, 4), (4, 5), (5, 6)]
>>> n_grams(a, 4)
[(1, 2, 3, 4), (2, 3, 4, 5), (3, 4, 5, 6)]

1.16 Inverting a dictionary using zip

>>> m = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
>>> m.items()
[('a', 1), ('c', 3), ('b', 2), ('d', 4)]
>>> zip(m.values(), m.keys())
[(1, 'a'), (3, 'c'), (2, 'b'), (4, 'd')]
>>> mi = dict(zip(m.values(), m.keys()))
>>> mi
{1: 'a', 2: 'b', 3: 'c', 4: 'd'}

1.17 Flattening lists:

>>> a = [[1, 2], [3, 4], [5, 6]]
>>> list(itertools.chain.from_iterable(a))
[1, 2, 3, 4, 5, 6]

>>> sum(a, [])
[1, 2, 3, 4, 5, 6]

>>> [x for l in a for x in l]
[1, 2, 3, 4, 5, 6]

>>> a = [[[1, 2], [3, 4]], [[5, 6], [7, 8]]]
>>> [x for l1 in a for l2 in l1 for x in l2]
[1, 2, 3, 4, 5, 6, 7, 8]

>>> a = [1, 2, [3, 4], [[5, 6], [7, 8]]]
>>> flatten = lambda x: [y for l in x for y in flatten(l)] if type(x) is list else [x]
>>> flatten(a)
[1, 2, 3, 4, 5, 6, 7, 8]

Note: according to Python's documentation on sum, itertools.chain.from_iterable is the preferred method for this.

1.18 Generator expressions

>>> g = (x ** 2 for x in xrange(10))
>>> next(g)
0
>>> next(g)
1
>>> next(g)
4
>>> next(g)
9
>>> sum(x ** 3 for x in xrange(10))
2025
>>> sum(x ** 3 for x in xrange(10) if x % 3 == 1)
408

1.19 Dictionary comprehensions

>>> m = {x: x ** 2 for x in range(5)}
>>> m
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

>>> m = {x: 'A' + str(x) for x in range(10)}
>>> m
{0: 'A0', 1: 'A1', 2: 'A2', 3: 'A3', 4: 'A4', 5: 'A5', 6: 'A6', 7: 'A7', 8: 'A8', 9: 'A9'}

1.20 Inverting a dictionary using a dictionary comprehension

>>> m = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
>>> m
{'d': 4, 'a': 1, 'b': 2, 'c': 3}
>>> {v: k for k, v in m.items()}
{1: 'a', 2: 'b', 3: 'c', 4: 'd'}

1.21 Named tuples (`collections.namedtuple`)

>>> Point = collections.namedtuple('Point', ['x', 'y'])
>>> p = Point(x=1.0, y=2.0)
>>> p
Point(x=1.0, y=2.0)
>>> p.x
1.0
>>> p.y
2.0

1.22 Inheriting from named tuples:

>>> class Point(collections.namedtuple('PointBase', ['x', 'y'])):
...     __slots__ = ()
...     def __add__(self, other):
...             return Point(x=self.x + other.x, y=self.y + other.y)
...
>>> p = Point(x=1.0, y=2.0)
>>> q = Point(x=2.0, y=3.0)
>>> p + q
Point(x=3.0, y=5.0)

1.23 Sets and set operations

>>> A = {1, 2, 3, 3}
>>> A
set([1, 2, 3])
>>> B = {3, 4, 5, 6, 7}
>>> B
set([3, 4, 5, 6, 7])
>>> A | B
set([1, 2, 3, 4, 5, 6, 7])
>>> A & B
set([3])
>>> A - B
set([1, 2])
>>> B - A
set([4, 5, 6, 7])
>>> A ^ B
set([1, 2, 4, 5, 6, 7])
>>> (A ^ B) == ((A - B) | (B - A))
True

1.24 Multisets and multiset operations (`collections.Counter`)

>>> A = collections.Counter([1, 2, 2])
>>> B = collections.Counter([2, 2, 3])
>>> A
Counter({2: 2, 1: 1})
>>> B
Counter({2: 2, 3: 1})
>>> A | B
Counter({2: 2, 1: 1, 3: 1})
>>> A & B
Counter({2: 2})
>>> A + B
Counter({2: 4, 1: 1, 3: 1})
>>> A - B
Counter({1: 1})
>>> B - A
Counter({3: 1})

1.25 Most common elements in an iterable (`collections.Counter`)

>>> A = collections.Counter([1, 1, 2, 2, 3, 3, 3, 3, 4, 5, 6, 7])
>>> A
Counter({3: 4, 1: 2, 2: 2, 4: 1, 5: 1, 6: 1, 7: 1})
>>> A.most_common(1)
[(3, 4)]
>>> A.most_common(3)
[(3, 4), (1, 2), (2, 2)]

1.26 Double-ended queue (`collections.deque`)

>>> Q = collections.deque()
>>> Q.append(1)
>>> Q.appendleft(2)
>>> Q.extend([3, 4])
>>> Q.extendleft([5, 6])
>>> Q
deque([6, 5, 2, 1, 3, 4])
>>> Q.pop()
4
>>> Q.popleft()
6
>>> Q
deque([5, 2, 1, 3])
>>> Q.rotate(3)
>>> Q
deque([2, 1, 3, 5])
>>> Q.rotate(-3)
>>> Q
deque([5, 2, 1, 3])

1.27 Double-ended queue with maximum length (`collections.deque`)

>>> last_three = collections.deque(maxlen=3)
>>> for i in xrange(10):
...     last_three.append(i)
...     print ', '.join(str(x) for x in last_three)
...
0
0, 1
0, 1, 2
1, 2, 3
2, 3, 4
3, 4, 5
4, 5, 6
5, 6, 7
6, 7, 8
7, 8, 9

1.28 Ordered dictionaries (`collections.OrderedDict`)

>>> m = dict((str(x), x) for x in range(10))
>>> print ', '.join(m.keys())
1, 0, 3, 2, 5, 4, 7, 6, 9, 8
>>> m = collections.OrderedDict((str(x), x) for x in range(10))
>>> print ', '.join(m.keys())
0, 1, 2, 3, 4, 5, 6, 7, 8, 9
>>> m = collections.OrderedDict((str(x), x) for x in range(10, 0, -1))
>>> print ', '.join(m.keys())
10, 9, 8, 7, 6, 5, 4, 3, 2, 1

1.29 Default dictionaries (`collections.defaultdict`)

>>> m = dict()
>>> m['a']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'a'
>>>
>>> m = collections.defaultdict(int)
>>> m['a']
0
>>> m['b']
0
>>> m = collections.defaultdict(str)
>>> m['a']
''
>>> m['b'] += 'a'
>>> m['b']
'a'
>>> m = collections.defaultdict(lambda: '[default value]')
>>> m['a']
'[default value]'
>>> m['b']
'[default value]'

1.30 Using default dictionaries to represent simple trees

>>> import json
>>> tree = lambda: collections.defaultdict(tree)
>>> root = tree()
>>> root['menu']['id'] = 'file'
>>> root['menu']['value'] = 'File'
>>> root['menu']['menuitems']['new']['value'] = 'New'
>>> root['menu']['menuitems']['new']['onclick'] = 'new();'
>>> root['menu']['menuitems']['open']['value'] = 'Open'
>>> root['menu']['menuitems']['open']['onclick'] = 'open();'
>>> root['menu']['menuitems']['close']['value'] = 'Close'
>>> root['menu']['menuitems']['close']['onclick'] = 'close();'
>>> print json.dumps(root, sort_keys=True, indent=4, separators=(',', ': '))
{
    "menu": {
        "id": "file",
        "menuitems": {
            "close": {
                "onclick": "close();",
                "value": "Close"
            },
            "new": {
                "onclick": "new();",
                "value": "New"
            },
            "open": {
                "onclick": "open();",
                "value": "Open"
            }
        },
        "value": "File"
    }
}

(See https://gist.github.com/hrldcpr/2012250 for more on this.)

1.31 Mapping objects to unique counting numbers (`collections.defaultdict`)

>>> import itertools, collections
>>> value_to_numeric_map = collections.defaultdict(itertools.count().next)
>>> value_to_numeric_map['a']
0
>>> value_to_numeric_map['b']
1
>>> value_to_numeric_map['c']
2
>>> value_to_numeric_map['a']
0
>>> value_to_numeric_map['b']
1

1.32 Largest and smallest elements (`heapq.nlargest` and `heapq.nsmallest`)

>>> a = [random.randint(0, 100) for __ in xrange(100)]
>>> heapq.nsmallest(5, a)
[3, 3, 5, 6, 8]
>>> heapq.nlargest(5, a)
[100, 100, 99, 98, 98]

1.33 Cartesian products (`itertools.product`)

>>> for p in itertools.product([1, 2, 3], [4, 5]):
(1, 4)
(1, 5)
(2, 4)
(2, 5)
(3, 4)
(3, 5)
>>> for p in itertools.product([0, 1], repeat=4):
...     print ''.join(str(x) for x in p)
...
0000
0001
0010
0011
0100
0101
0110
0111
1000
1001
1010
1011
1100
1101
1110
1111

1.34 Combinations and combinations with replacement (`itertools.combinations` and `itertools.combinations_with_replacement`)

>>> for c in itertools.combinations([1, 2, 3, 4, 5], 3):
...     print ''.join(str(x) for x in c)
...
123
124
125
134
135
145
234
235
245
345
>>> for c in itertools.combinations_with_replacement([1, 2, 3], 2):
...     print ''.join(str(x) for x in c)
...
11
12
13
22
23
33

1.35 Permutations (`itertools.permutations`)

>>> for p in itertools.permutations([1, 2, 3, 4]):
...     print ''.join(str(x) for x in p)
...
1234
1243
1324
1342
1423
1432
2134
2143
2314
2341
2413
2431
3124
3142
3214
3241
3412
3421
4123
4132
4213
4231
4312
4321

1.36 Chaining iterables (`itertools.chain`)

>>> a = [1, 2, 3, 4]
>>> for p in itertools.chain(itertools.combinations(a, 2), itertools.combinations(a, 3)):
...     print p
...
(1, 2)
(1, 3)
(1, 4)
(2, 3)
(2, 4)
(3, 4)
(1, 2, 3)
(1, 2, 4)
(1, 3, 4)
(2, 3, 4)
>>> for subset in itertools.chain.from_iterable(itertools.combinations(a, n) for n in range(len(a) + 1))
...     print subset
...
()
(1,)
(2,)
(3,)
(4,)
(1, 2)
(1, 3)
(1, 4)
(2, 3)
(2, 4)
(3, 4)
(1, 2, 3)
(1, 2, 4)
(1, 3, 4)
(2, 3, 4)
(1, 2, 3, 4)

1.37 Grouping rows by a given key (`itertools.groupby`)

>>> from operator import itemgetter
>>> import itertools
>>> with open('contactlenses.csv', 'r') as infile:
...     data = [line.strip().split(',') for line in infile]
...
>>> data = data[1:]
>>> def print_data(rows):
...     print '\n'.join('\t'.join('{: <16}'.format(s) for s in row) for row in rows)
...

>>> print_data(data)
young               myope                   no                      reduced                 none
young               myope                   no                      normal                  soft
young               myope                   yes                     reduced                 none
young               myope                   yes                     normal                  hard
young               hypermetrope            no                      reduced                 none
young               hypermetrope            no                      normal                  soft
young               hypermetrope            yes                     reduced                 none
young               hypermetrope            yes                     normal                  hard
pre-presbyopic      myope                   no                      reduced                 none
pre-presbyopic      myope                   no                      normal                  soft
pre-presbyopic      myope                   yes                     reduced                 none
pre-presbyopic      myope                   yes                     normal                  hard
pre-presbyopic      hypermetrope            no                      reduced                 none
pre-presbyopic      hypermetrope            no                      normal                  soft
pre-presbyopic      hypermetrope            yes                     reduced                 none
pre-presbyopic      hypermetrope            yes                     normal                  none
presbyopic          myope                   no                      reduced                 none
presbyopic          myope                   no                      normal                  none
presbyopic          myope                   yes                     reduced                 none
presbyopic          myope                   yes                     normal                  hard
presbyopic          hypermetrope            no                      reduced                 none
presbyopic          hypermetrope            no                      normal                  soft
presbyopic          hypermetrope            yes                     reduced                 none
presbyopic          hypermetrope            yes                     normal                  none

>>> data.sort(key=itemgetter(-1))
>>> for value, group in itertools.groupby(data, lambda r: r[-1]):
...     print '-----------'
...     print 'Group: ' + value
...     print_data(group)
...
-----------
Group: hard
young               myope                   yes                     normal                  hard
young               hypermetrope            yes                     normal                  hard
pre-presbyopic      myope                   yes                     normal                  hard
presbyopic          myope                   yes                     normal                  hard
-----------
Group: none
young               myope                   no                      reduced                 none
young               myope                   yes                     reduced                 none
young               hypermetrope            no                      reduced                 none
young               hypermetrope            yes                     reduced                 none
pre-presbyopic      myope                   no                      reduced                 none
pre-presbyopic      myope                   yes                     reduced                 none
pre-presbyopic      hypermetrope            no                      reduced                 none
pre-presbyopic      hypermetrope            yes                     reduced                 none
pre-presbyopic      hypermetrope            yes                     normal                  none
presbyopic          myope                   no                      reduced                 none
presbyopic          myope                   no                      normal                  none
presbyopic          myope                   yes                     reduced                 none
presbyopic          hypermetrope            no                      reduced                 none
presbyopic          hypermetrope            yes                     reduced                 none
presbyopic          hypermetrope            yes                     normal                  none
-----------
Group: soft
young               myope                   no                      normal                  soft
young               hypermetrope            no                      normal                  soft
pre-presbyopic      myope                   no                      normal                  soft
pre-presbyopic      hypermetrope            no                      normal                  soft
presbyopic          hypermetrope            no                      normal                  soft

1.38 Start a static HTTP server in any directory

[10:26] $ python -m SimpleHTTPServer 5000
Serving HTTP on 0.0.0.0 port 5000 ...

1.39 Learn the Zen of Python

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

1.40 Use C-Style Braces Instead of Indentation to Denote Scopes

>>> from __future__ import braces

Good Luck Have Fun!

Python Performance Tips

2015-01-15T18:31:00.000-08:00

Have Been 5 year or may be more :D I used python for create many automation script. In this section I will share something about how to improve performance to speedup python code. This is originally I got from wiki.python.org

You can only know what makes your program slow after first getting the program to give correct results, then running it to see if the correct program is slow. When found to be slow, profiling can show what parts of the program are consuming most of the time. A comprehensive but quick-to-run test suite can then ensure that future optimizations don't change the correctness of your program. In short:

Get it right.
Test it's right.
Profile if slow.
Optimise.
Repeat from 2.

Certain optimizations amount to good programming style and so should be learned as you learn the language. An example would be moving the calculation of values that don't change within a loop, outside of the loop.

Choose the Right Data Structure

TBD.

Sorting

Sorting lists of basic Python objects is generally pretty efficient. The sort method for lists takes an optional comparison function as an argument that can be used to change the sorting behavior. This is quite convenient, though it can significantly slow down your sorts, as the comparison function will be called many times. In Python 2.4, you should use the key argument to the built-in sort instead, which should be the fastest way to sort.

Only if you are using older versions of Python (before 2.4) does the following advice from Guido van Rossum apply:

An alternative way to speed up sorts is to construct a list of tuples whose first element is a sort key that will sort properly using the default comparison, and whose second element is the original list element. This is the so-called Schwartzian Transform, also known as DecorateSortUndecorate (DSU).

Suppose, for example, you have a list of tuples that you want to sort by the n-th field of each tuple. The following function will do that.

def sortby(somelist, n):
    nlist = [(x[n], x) for x in somelist]
    nlist.sort()
    return [val for (key, val) in nlist]

Matching the behavior of the current list sort method (sorting in place) is easily achieved as well:

def sortby_inplace(somelist, n):
    somelist[:] = [(x[n], x) for x in somelist]
    somelist.sort()
    somelist[:] = [val for (key, val) in somelist]
    return

Here's an example use:

>>> somelist = [(1, 2, 'def'), (2, -4, 'ghi'), (3, 6, 'abc')]
>>> somelist.sort()
>>> somelist
[(1, 2, 'def'), (2, -4, 'ghi'), (3, 6, 'abc')]
>>> nlist = sortby(somelist, 2)
>>> sortby_inplace(somelist, 2)
>>> nlist == somelist
True
>>> nlist = sortby(somelist, 1)
>>> sortby_inplace(somelist, 1)
>>> nlist == somelist
True

From Tim Delaney

From Python 2.3 sort is guaranteed to be stable.

(to be precise, it's stable in CPython 2.3, and guaranteed to be stable in Python 2.4)

Python 2.4 adds an optional key parameter which makes the transform a lot easier to use:

# E.g. n = 1
n = 1
import operator
nlist.sort(key=operator.itemgetter(n))
# use sorted() if you don't want to sort in-place:
# sortedlist = sorted(nlist, key=operator.itemgetter(n))

Note that the original item is never used for sorting, only the returned key - this is equivalent to doing:

# E.g. n = 1
n = 1
nlist = [(x[n], i, x) for (i, x) in enumerate(nlist)]
nlist.sort()
nlist = [val for (key, index, val) in nlist]

String Concatenation

The accuracy of this section is disputed with respect to later versions of Python. In CPython 2.5, string concatenation is fairly fast, although this may not apply likewise to other Python implementations. See ConcatenationTestCode for a discussion.

Strings in Python are immutable. This fact frequently sneaks up and bites novice Python programmers on the rump. Immutability confers some advantages and disadvantages. In the plus column, strings can be used as keys in dictionaries and individual copies can be shared among multiple variable bindings. (Python automatically shares one- and two-character strings.) In the minus column, you can't say something like, "change all the 'a's to 'b's" in any given string. Instead, you have to create a new string with the desired properties. This continual copying can lead to significant inefficiencies in Python programs.

Avoid this

s = ""
for substring in list:
    s += substring

Use s = "".join(list) instead. The former is a very common and catastrophic mistake when building large strings. Similarly, if you are generating bits of a string sequentially instead of:

s = ""
for x in list:
    s += some_function(x)

use

slist = [some_function(elt) for elt in somelist]
s = "".join(slist)

Avoid:

out = "<html>" + head + prologue + query + tail + "</html>"

Instead, use

out = "<html>%s%s%s%s</html>" % (head, prologue, query, tail)

Even better, for readability (this has nothing to do with efficiency other than yours as a programmer), use dictionary substitution:

out = "<html>%(head)s%(prologue)s%(query)s%(tail)s</html>" % locals()

This last two are going to be much faster, especially when piled up over many CGI script executions, and easier to modify to boot. In addition, the slow way of doing things got slower in Python 2.0 with the addition of rich comparisons to the language. It now takes the Python virtual machine a lot longer to figure out how to concatenate two strings. (Don't forget that Python does all method lookup at runtime.)

Loops

Python supports a couple of looping constructs. The for statement is most commonly used. It loops over the elements of a sequence, assigning each to the loop variable. If the body of your loop is simple, the interpreter overhead of the for loop itself can be a substantial amount of the overhead. This is where the map function is handy. You can think of map as a for moved into C code. The only restriction is that the "loop body" of map must be a function call. Besides the syntactic benefit of list comprehensions, they are often as fast or faster than equivalent use of map.

Here's a straightforward example. Instead of looping over a list of words and converting them to upper case:

newlist = []
for word in oldlist:
    newlist.append(word.upper())

you can use map to push the loop from the interpreter into compiled C code:

newlist = map(str.upper, oldlist)

List comprehensions were added to Python in version 2.0 as well. They provide a syntactically more compact and more efficient way of writing the above for loop:

newlist = [s.upper() for s in oldlist]

Generator expressions were added to Python in version 2.4. They function more-or-less like list comprehensions or map but avoid the overhead of generating the entire list at once. Instead, they return a generator object which can be iterated over bit-by-bit:

iterator = (s.upper() for s in oldlist)

Which method is appropriate will depend on what version of Python you're using and the characteristics of the data you are manipulating.

Guido van Rossum wrote a much more detailed (and succinct) examination of loop optimization that is definitely worth reading.

Avoiding dots...

Suppose you can't use map or a list comprehension? You may be stuck with the for loop. The for loop example has another inefficiency. Both newlist.append and word.upper are function references that are reevaluated each time through the loop. The original loop can be replaced with:

upper = str.upper
newlist = []
append = newlist.append
for word in oldlist:
    append(upper(word))

This technique should be used with caution. It gets more difficult to maintain if the loop is large. Unless you are intimately familiar with that piece of code you will find yourself scanning up to check the definitions of append and upper.

Local Variables

The final speedup available to us for the non-map version of the for loop is to use local variables wherever possible. If the above loop is cast as a function, append and upper become local variables. Python accesses local variables much more efficiently than global variables.

def func():
    upper = str.upper
    newlist = []
    append = newlist.append
    for word in oldlist:
        append(upper(word))
    return newlist

At the time I originally wrote this I was using a 100MHz Pentium running BSDI. I got the following times for converting the list of words in /usr/share/dict/words (38,470 words at that time) to upper case:

Version Time (seconds)
Basic loop 3.47
Eliminate dots 2.45
Local variable & no dots 1.79
Using map function 0.54

Initializing Dictionary Elements

Suppose you are building a dictionary of word frequencies and you've already broken your text up into a list of words. You might execute something like:

wdict = {}
for word in words:
    if word not in wdict:
        wdict[word] = 0
    wdict[word] += 1

Except for the first time, each time a word is seen the if statement's test fails. If you are counting a large number of words, many will probably occur multiple times. In a situation where the initialization of a value is only going to occur once and the augmentation of that value will occur many times it is cheaper to use a try statement:

wdict = {}
for word in words:
    try:
        wdict[word] += 1
    except KeyError:
        wdict[word] = 1

It's important to catch the expected KeyError exception, and not have a default except clause to avoid trying to recover from an exception you really can't handle by the statement(s) in the try clause.

A third alternative became available with the release of Python 2.x. Dictionaries now have a get() method which will return a default value if the desired key isn't found in the dictionary. This simplifies the loop:

wdict = {}
get = wdict.get
for word in words:
    wdict[word] = get(word, 0) + 1

There were clear situations where one of the first two approaches was faster. It seems that all three approaches now exhibit similar performance (within about 10% of each other), more or less independent of the properties of the list of words.

Also, if the value stored in the dictionary is an object or a (mutable) list, you could also use the dict.setdefault method, e.g.

   4     wdict.setdefault(key, []).append(new_element)

You might think that this avoids having to look up the key twice. It actually doesn't (even in python 3.0), but at least the double lookup is performed in C.

Another option is to use the defaultdict class:

from collections import defaultdict

wdict = defaultdict(int)

for word in words:
    wdict[word] += 1

Import Statement Overhead

import statements can be executed just about anywhere. It's often useful to place them inside functions to restrict their visibility and/or reduce initial startup time. Although Python's interpreter is optimized to not import the same module multiple times, repeatedly executing an import statement can seriously affect performance in some circumstances.

Consider the following two snippets of code (originally from Greg McFarlane, I believe - I found it unattributed in a comp.lang.python python-list@python.org posting and later attributed to him in another source):

def doit1():
    import string ###### import statement inside function
    string.lower('Python')

for num in range(100000):
    doit1()

or:

import string ###### import statement outside function
def doit2():
    string.lower('Python')

for num in range(100000):
    doit2()

doit2 will run much faster than doit1, even though the reference to the string module is global in doit2. Here's a Python interpreter session run using Python 2.3 and the new timeit module, which shows how much faster the second is than the first:

>>> def doit1():
... import string
... string.lower('Python')
...
>>> import string
>>> def doit2():
... string.lower('Python')
...
>>> import timeit
>>> t = timeit.Timer(setup='from __main__ import doit1', stmt='doit1()')
>>> t.timeit()
11.479144930839539
>>> t = timeit.Timer(setup='from __main__ import doit2', stmt='doit2()')
>>> t.timeit()
4.6661689281463623

String methods were introduced to the language in Python 2.0. These provide a version that avoids the import completely and runs even faster:

def doit3():
    'Python'.lower()

for num in range(100000):
    doit3()

Here's the proof from timeit:

>>> def doit3():
... 'Python'.lower()
...
>>> t = timeit.Timer(setup='from __main__ import doit3', stmt='doit3()')
>>> t.timeit()
2.5606080293655396

The above example is obviously a bit contrived, but the general principle holds.

Note that putting an import in a function can speed up the initial loading of the module, especially if the imported module might not be required. This is generally a case of a "lazy" optimization -- avoiding work (importing a module, which can be very expensive) until you are sure it is required.

This is only a significant saving in cases where the module wouldn't have been imported at all (from any module) -- if the module is already loaded (as will be the case for many standard modules, like string or re), avoiding an import doesn't save you anything. To see what modules are loaded in the system look in sys.modules.

A good way to do lazy imports is:

email = None

def parse_email():
    global email
    if email is None:
        import email
    ...

This way the email module will only be imported once, on the first invocation of parse_email().

Data Aggregation

Function call overhead in Python is relatively high, especially compared with the execution speed of a builtin function. This strongly suggests that where appropriate, functions should handle data aggregates. Here's a contrived example written in Python.

import time
x = 0
def doit1(i):
    global x
    x = x + i

list = range(100000)
t = time.time()
for i in list:
    doit1(i)

print "%.3f" % (time.time()-t)

vs.

import time
x = 0
def doit2(list):
    global x
    for i in list:
        x = x + i

list = range(100000)
t = time.time()
doit2(list)
print "%.3f" % (time.time()-t)

Here's the proof in the pudding using an interactive session:

>>> t = time.time()
>>> for i in list:
... doit1(i)
...
>>> print "%.3f" % (time.time()-t)
0.758
>>> t = time.time()
>>> doit2(list)
>>> print "%.3f" % (time.time()-t)
0.204

Even written in Python, the second example runs about four times faster than the first. Had doit been written in C the difference would likely have been even greater (exchanging a Python for loop for a C for loop as well as removing most of the function calls).

Doing Stuff Less Often

The Python interpreter performs some periodic checks. In particular, it decides whether or not to let another thread run and whether or not to run a pending call (typically a call established by a signal handler). Most of the time there's nothing to do, so performing these checks each pass around the interpreter loop can slow things down. There is a function in the sys module, setcheckinterval, which you can call to tell the interpreter how often to perform these periodic checks. Prior to the release of Python 2.3 it defaulted to 10. In 2.3 this was raised to 100. If you aren't running with threads and you don't expect to be catching many signals, setting this to a larger value can improve the interpreter's performance, sometimes substantially.

Python is not C

It is also not Perl, Java, C++ or Haskell. Be careful when transferring your knowledge of how other languages perform to Python. A simple example serves to demonstrate:

% timeit.py -s 'x = 47' 'x * 2'
loops, best of 3: 0.574 usec per loop
% timeit.py -s 'x = 47' 'x << 1'
loops, best of 3: 0.524 usec per loop
% timeit.py -s 'x = 47' 'x + x'
loops, best of 3: 0.382 usec per loop

Now consider the similar C programs (only the add version is shown):

#include <stdio.h>

int main (int argc, char *argv[]) {
 int i = 47;
 int loop;
 for (loop=0; loop<500000000; loop++)
  i + i;
 return 0;
}

and the execution times:

% for prog in mult add shift ; do
< for i in 1 2 3 ; do
< echo -n "$prog: "
< /usr/bin/time ./$prog
< done
< echo
< done
mult: 6.12 real 5.64 user 0.01 sys
mult: 6.08 real 5.50 user 0.04 sys
mult: 6.10 real 5.45 user 0.03 sys

add: 6.07 real 5.54 user 0.00 sys
add: 6.08 real 5.60 user 0.00 sys
add: 6.07 real 5.58 user 0.01 sys

shift: 6.09 real 5.55 user 0.01 sys
shift: 6.10 real 5.62 user 0.01 sys
shift: 6.06 real 5.50 user 0.01 sys

Note that there is a significant advantage in Python to adding a number to itself instead of multiplying it by two or shifting it left by one bit. In C on all modern computer architectures, each of the three arithmetic operations are translated into a single machine instruction which executes in one cycle, so it doesn't really matter which one you choose.

A common "test" new Python programmers often perform is to translate the common Perl idiom

while (<>) {
    print;
}

into Python code that looks something like

import fileinput

for line in fileinput.input():
    print line,

and use it to conclude that Python must be much slower than Perl. As others have pointed out numerous times, Python is slower than Perl for some things and faster for others. Relative performance also often depends on your experience with the two languages.

Use xrange instead of range

This section no longer applies if you're using Python 3, where range now provides an iterator over ranges of arbitrary size, and where xrange no longer exists.

Python has two ways to get a range of numbers: range and xrange. Most people know about range, because of its obvious name. xrange, being way down near the end of the alphabet, is much less well-known.

xrange is a generator object, basically equivalent to the following Python 2.3 code:

def xrange(start, stop=None, step=1):
    if stop is None:
        stop = start
        start = 0
    else:
        stop = int(stop)
    start = int(start)
    step = int(step)

    while start < stop:
        yield start
        start += step

Except that it is implemented in pure C.

xrange does have limitations. Specifically, it only works with ints; you cannot use longs or floats (they will be converted to ints, as shown above).

It does, however, save gobs of memory, and unless you store the yielded objects somewhere, only one yielded object will exist at a time. The difference is thus: When you call range, it creates a list containing so many number (int, long, or float) objects. All of those objects are created at once, and all of them exist at the same time. This can be a pain when the number of numbers is large.

xrange, on the other hand, creates no numbers immediately - only the range object itself. Number objects are created only when you pull on the generator, e.g. by looping through it. For example:

xrange(sys.maxint) # No loop, and no call to .next, so no numbers are instantiated

And for this reason, the code runs instantly. If you substitute range there, Python will lock up; it will be too busy allocating sys.maxint number objects (about 2.1 billion on the typical PC) to do anything else. Eventually, it will run out of memory and exit.

In Python versions before 2.2, xrange objects also supported optimizations such as fast membership testing (i in xrange(n)). These features were removed in 2.2 due to lack of use.

Re-map Functions at runtime

Say you have a function

 class Test:
   def check(self,a,b,c):
     if a == 0:
       self.str = b*100
     else:
       self.str = c*100

 a = Test()
 def example():
   for i in xrange(0,100000):
     a.check(i,"b","c")

 import profile
 profile.run("example()")

And suppose this function gets called from somewhere else many times.

Well, your check will have an if statement slowing you down all the time except the first time, so you can do this:

 class Test2:
   def check(self,a,b,c):
     self.str = b*100
     self.check = self.check_post
   def check_post(self,a,b,c):
     self.str = c*100

 a = Test2()
 def example2():
   for i in xrange(0,100000):
     a.check(i,"b","c")

 import profile
 profile.run("example2()")

Well, this example is fairly inadequate, but if the 'if' statement is a pretty complicated expression (or something with lots of dots), you can save yourself evaluating it, if you know it will only be true the first time.

Profiling Code

The first step to speeding up your program is learning where the bottlenecks lie. It hardly makes sense to optimize code that is never executed or that already runs fast. I use two modules to help locate the hotspots in my code, profile and trace. In later examples I also use the timeit module, which is new in Python 2.3.

See the separate profiling document for alternatives to the approaches given below.

Profiling

There are a number of profiling modules included in the Python distribution. Using one of these to profile the execution of a set of functions is quite easy. Suppose your main function is called main, takes no arguments and you want to execute it under the control of the profile module. In its simplest form you just execute

import profile
profile.run('main()')

When main() returns, the profile module will print a table of function calls and execution times. The output can be tweaked using the Stats class included with the module. From Python 2.4 profile has permitted the time consumed by Python builtins and functions in extension modules to be profiled as well.

A slightly longer description of profiling using the profile and pstats modules can be found here (archived version):

http://web.archive.org/web/20060506162444/http://wingware.com/doc/howtos/performance-profiling-python-code

The cProfile and Hotshot Modules

Since Python 2.2, the hotshot package has been available as a replacement for the profile module, although the cProfile module is now recommended in preference to hotshot. The underlying module is written in C, so using hotshot (or cProfile) should result in a much smaller performance hit, and thus a more accurate idea of how your application is performing. There is also a hotshotmain.py program in the distribution's Tools/scripts directory which makes it easy to run your program under hotshot control from the command line.

Trace Module

The trace module is a spin-off of the profile module as of Python 2.0 you should find trace.py in the Tools/scripts directory of the Python distribution. Starting with Python 2.3 it's in the standard library (the Lib directory). You can copy it to your local bin directory and set the execute permission, then execute it directly. It's easy to run from the command line to trace execution of whole scripts:

% trace.py -t spam.py eggs

In Python 2.4 it's even easier to run. Just execute python -m trace.

There's no separate documentation, but you can execute "pydoc trace" to view the inline documentation.

Visualizing Profiling Results

RunSnakeRun is a GUI tool by Mike Fletcher which visualizes profile dumps from cProfile using square maps. Function/method calls may be sorted according to various criteria, and source code may be displayed alongside the visualization and call statistics.

An example usage:

runsnake some_profile_dump.prof

Gprof2Dot is a python based tool that can transform profiling results output into a graph that can be converted into a PNG image or SVG.

A typical profiling session with python 2.5 looks like this (on older platforms you will need to use actual script instead of the -m option):

python -m cProfile -o stat.prof MYSCRIPY.PY [ARGS...]
python -m pbp.scripts.gprof2dot -f pstats -o stat.dot stat.prof
dot -ostat.png -Tpng stat.dot

PyCallGraph pycallgraph is a Python module that creates call graphs for Python programs. It generates a PNG file showing an modules's function calls and their link to other function calls, the amount of times a function was called and the time spent in that function.

Typical usage:

pycallgraph scriptname.py

PyProf2CallTree is a script to help visualize profiling data collected with the cProfile python module with the kcachegrind graphical calltree analyser.

Typical usage:

python -m cProfile -o stat.prof MYSCRIPY.PY [ARGS...]
python pyprof2calltree.py -i stat.prof -k

Good luck have fun!

my little note

David Beazley - Modules and Packages: Live and Let Die!

Android HTML5 App, how really it's?

Python Access Java Through JNIus

How to make android emulator run faster?

Python LinkedIn REST API Implementation

Python Tumblr REST API Implementation

Python Twitter REST API Implementation

Understanding python decorator in easy way

Python OAuth implementation it's not that hard!

How to request POST and GET using Python ?

Python in a Minute

Setup Python on windows platform

No more Python 2.8

SQLite3 Review – Great for Beginners and Those Learning SQL

Small and Self-Contained

solving a start job is running for create volatile files and directories : manjaro linux

The first Ubuntu Phone will go on sale in Europe in the second week of February.

The First Phone: Specifications

Looking Ahead

Manjaro XFCE Post Installation Tips and Tricks

For more please refer

Manjaro Linux: Arch For People Who Don’t Have Time

About Manjaro Linux

Don’t Fear the Beta!

The User Experience

Installing Software

Conclusion

Scary Steam for Linux bug erases all the personal files on your PC

Ouch

Porting Code to Python 3 with 2to3

Diving In

Unicode string literals

Unicode string literals

unicode() global function

long data type

<> comparison

has_key() dictionary method

Dictionary methods that return lists

Modules that have been renamed or reorganized

http

urllib

dbm

xmlrpc

Other modules

Relative imports within a package

next() iterator method

filter() global function

map() global function

reduce() global function

apply() global function

intern() global function

exec statement

execfile statement

repr literals (backticks)

try...except statement

raise statement

throw method on generators

xrange() global function

raw_input() and input() global functions

func_* function attributes

xreadlines() I/O method

lambda functions that take a tuple instead of multiple parameters

Special method attributes

__nonzero__ special method

Octal literals

sys.maxint

callable() global function

zip() global function

StandardError exception

types module constants

isinstance() global function

basestring datatype

itertools module

sys.exc_type, sys.exc_value, sys.exc_traceback

List comprehensions over tuples

os.getcwdu() function

Metaclasses

Matters of style

set() literals (explicit)

`unicode()` global function

`long` data type

`has_key()` dictionary method

`http`

`urllib`

`dbm`

`xmlrpc`

`next()` iterator method

`filter()` global function

`map()` global function

`reduce()` global function

`apply()` global function

`intern()` global function

`exec` statement

`execfile` statement

`repr` literals (backticks)

`try...except` statement

`raise` statement

`throw` method on generators

`xrange()` global function

`raw_input()` and `input()` global functions

`func_*` function attributes

`xreadlines()` I/O method

`lambda` functions that take a tuple instead of multiple parameters

`nonzero` special method

`sys.maxint`

`callable()` global function

`zip()` global function

`StandardError` exception

`types` module constants

`isinstance()` global function

`basestring` datatype

`itertools` module

`sys.exc_type`, `sys.exc_value`, `sys.exc_traceback`

`os.getcwdu()` function

`set()` literals (explicit)

`buffer()` global function (explicit)

1.5 List slices (`a[start:end]`)

1.7 List slices with step (`a[start:end:step]`)

1.10 Naming slices (`slice(start, end, step)`)

1.11 Iterating over list index and value pairs (`enumerate`)

1.12 Iterating over dictionary key and value pairs (`dict.iteritems`)

1.21 Named tuples (`collections.namedtuple`)

1.24 Multisets and multiset operations (`collections.Counter`)

1.25 Most common elements in an iterable (`collections.Counter`)

1.26 Double-ended queue (`collections.deque`)

1.27 Double-ended queue with maximum length (`collections.deque`)

1.28 Ordered dictionaries (`collections.OrderedDict`)

1.29 Default dictionaries (`collections.defaultdict`)

1.31 Mapping objects to unique counting numbers (`collections.defaultdict`)

1.32 Largest and smallest elements (`heapq.nlargest` and `heapq.nsmallest`)

1.33 Cartesian products (`itertools.product`)

1.34 Combinations and combinations with replacement (`itertools.combinations` and `itertools.combinations_with_replacement`)

1.35 Permutations (`itertools.permutations`)

1.36 Chaining iterables (`itertools.chain`)

1.37 Grouping rows by a given key (`itertools.groupby`)