Hashing Algorithms

The most used algorithms to hash a file are MD5 and SHA-1. They are used because they are fast and they provide a good way to identify different files. The hash function only uses the contents of the file, not the name. Getting the same hash of two separating files means that there is a high probability the contents of the files are identical, even though they have different names.

MD5 File Hash in Python

The code is made to work with Python 2.7 and higher (including Python 3.x).

import hashlib

hasher = hashlib.md5()
with open('myfile.jpg', 'rb') as afile:
    buf = afile.read()
    hasher.update(buf)
    afile.close()
print(hasher.hexdigest())

The code above calculates the MD5 digest of the file. The file is opened in rb mode, which means that you are going to read the file in binary mode. This is because the MD5 function needs to read the file as a sequence of bytes. This will make sure that you can hash any type of file, not only text files.

It is important to notice the read function. When it is called with no arguments, like in this case, it will read all the contents of the file and load them into memory. This is dangerous if you are not sure of the file’s size. A better version will be:

MD5 Hash for Large Files in Python

import hashlib
BLOCKSIZE = 65536
hasher = hashlib.md5()
with open('anotherfile.txt', 'rb') as afile:
    buf = afile.read(BLOCKSIZE)
    while len(buf) > 0:
        hasher.update(buf)
        buf = afile.read(BLOCKSIZE)
print(hasher.hexdigest())

If you need to use another algorithm just change the md5 call to another supported function, e.g. SHA1:

SHA1 File Hash in Python

import hashlib
BLOCKSIZE = 65536
hasher = hashlib.sha1()
with open('anotherfile.txt', 'rb') as afile:
    buf = afile.read(BLOCKSIZE)
    while len(buf) > 0:
        hasher.update(buf)
        buf = afile.read(BLOCKSIZE)
print(hasher.hexdigest())

If you need a list of supported hash algorithms in your system use hashlib.algorithms_available. (Only works in Python 3.2 and superior). Finally, for another look into hashing, be sure to checkout the hashing Python strings article.

The post Hashing Files with Python appeared first on Python Central.

Now suppose you want to hash the string “Hello Word” with the SHA1 Function, the result is 0a4d55a8d778e5022fab701977c5d840bbc486d0.

Hash functions are used inside some cryptographic algorithms, in digital signatures, message authentication codes, manipulation detection, fingerprints, checksums (message integrity check), hash tables, password storage and much more. As a Python programmer you may need these functions to check for duplicate data or files, to check data integrity when you transmit information over a network, to securely store passwords in databases, or maybe some work related to cryptography.

I want to make clear that hash functions are not a cryptographic protocol, they do not encrypt or decrypt information, but they are a fundamental part of many cryptographic protocols and tools.

Some of the most used hash functions are:

• MD5: Message digest algorithm producing a 128 bit hash value. This is widely used to check data integrity. It is not suitable for use in other fields due to the security vulnerabilities of MD5.
• SHA: Group of algorithms designed by the U.S’s NSA that are part of the U.S Federal Information processing standard. These algorithms are used widely in several cryptographic applications. The message length ranges from 160 bits to 512 bits.

The hashlib module, included in The Python Standard library is a module containing an interface to the most popular hashing algorithms. hashlib implements some of the algorithms, however if you have OpenSSL installed, hashlib is able to use this algorithms as well.

This code is made to work in Python 3.2 and above. If you want to run this examples in Python 2.x, just remove the algorithms_available and algorithms_guaranteed calls.

First, import the hashlib module:

import hashlib

Now we use algorithms_available or algorithms_guaranteed to list the algorithms available.

print(hashlib.algorithms_available)
print(hashlib.algorithms_guaranteed)

The algorithms_available method lists all the algorithms available in the system, including the ones available trough OpenSSl. In this case you may see duplicate names in the list. algorithms_guaranteed only lists the algorithms present in the module. md5, sha1, sha224, sha256, sha384, sha512 are always present.

MD5

import hashlib
hash_object = hashlib.md5(b'Hello World')
print(hash_object.hexdigest())

The code above takes the “Hello World” string and prints the HEX digest of that string. hexdigest returns a HEX string representing the hash, in case you need the sequence of bytes you should use digest instead.

It is important to note the “b” preceding the string literal, this converts the string to bytes, because the hashing function only takes a sequence of bytes as a parameter. In previous versions of the library, it used to take a string literal. So, if you need to take some input from the console, and hash this input, do not forget to encode the string in a sequence of bytes:

import hashlib
mystring = input('Enter String to hash: ')
# Assumes the default UTF-8
hash_object = hashlib.md5(mystring.encode())
print(hash_object.hexdigest())

SHA1

import hashlib
hash_object = hashlib.sha1(b'Hello World')
hex_dig = hash_object.hexdigest()
print(hex_dig)

SHA224

import hashlib
hash_object = hashlib.sha224(b'Hello World')
hex_dig = hash_object.hexdigest()
print(hex_dig)

SHA256

import hashlib
hash_object = hashlib.sha256(b'Hello World')
hex_dig = hash_object.hexdigest()
print(hex_dig)

SHA384

import hashlib
hash_object = hashlib.sha384(b'Hello World')
hex_dig = hash_object.hexdigest()
print(hex_dig)

SHA512

import hashlib
hash_object = hashlib.sha512(b'Hello World')
hex_dig = hash_object.hexdigest()
print(hex_dig)

Using OpenSSL Algorithms

Now suppose you need an algorithm provided by OpenSSL. Using algorithms_available, we can find the name of the algorithm you want to use. In this case, “DSA” is available on my computer. You can then use the new and update methods:

import hashlib
hash_object = hashlib.new('DSA')
hash_object.update(b'Hello World')
print(hash_object.hexdigest())

Practical example: hashing passwords

In the following example we are hashing a password in order to store it in a database. In this example we are using a salt. A salt is a random sequence added to the password string before using the hash function. The salt is used in order to prevent dictionary attacks and rainbow tables attacks. However, if you are making real world applications and working with users’ passwords, make sure to be updated about the latest vulnerabilities in this field. I you want to find out more about secure passwords please refer to this article

import uuid
import hashlib

def hash_password(password):
    # uuid is used to generate a random number
    salt = uuid.uuid4().hex
    return hashlib.sha256(salt.encode() + password.encode()).hexdigest() + ':' + salt
    
def check_password(hashed_password, user_password):
    password, salt = hashed_password.split(':')
    return password == hashlib.sha256(salt.encode() + user_password.encode()).hexdigest()

new_pass = input('Please enter a password: ')
hashed_password = hash_password(new_pass)
print('The string to store in the db is: ' + hashed_password)
old_pass = input('Now please enter the password again to check: ')
if check_password(hashed_password, old_pass):
    print('You entered the right password')
else:
    print('I am sorry but the password does not match')

import uuid
import hashlib

def hash_password(password):
    # uuid is used to generate a random number
    salt = uuid.uuid4().hex
    return hashlib.sha256(salt.encode() + password.encode()).hexdigest() + ':' + salt
    
def check_password(hashed_password, user_password):
    password, salt = hashed_password.split(':')
    return password == hashlib.sha256(salt.encode() + user_password.encode()).hexdigest()

new_pass = raw_input('Please enter a password: ')
hashed_password = hash_password(new_pass)
print('The string to store in the db is: ' + hashed_password)
old_pass = raw_input('Now please enter the password again to check: ')
if check_password(hashed_password, old_pass):
    print('You entered the right password')
else:
    print('I am sorry but the password does not match')

The post Hashing Strings with Python appeared first on Python Central.

From Python’s official documentation, be sure to checkout Python’s mmap module:

A memory-mapped file object behaves like both strings and like file objects. Unlike normal string objects, however, these are mutable.

Basically, a memory-mapped (using Python’s mmap module) file object maps a normal file object into memory. This allows you to modify a file object’s content directly in memory. Since a memory-mapped file object also behaves like a mutable string object, you can modify the content of a file object like you modify the content of a list of characters:

• obj[1] = 'a' – Assigns a character ‘a’ to the the second character of the file object’s content.
• obj[1:4] = 'abc' – Assigns a character list ‘abc’ to a range of three characters starting at the second character of the file object’s content.

In a nutshell, memory-mapping a file with Python’s mmap module us use the operating system’s virtual memory to access the data on the filesystem directly. Instead of making system calls such as open, read and lseek to manipulate a file, memory-mapping puts the data of the file into memory which allows you to directly manipulate files in memory. This greatly improves I/O performance.

Comparison of Memory-Mapped Files vs Normal Files with Python

Assume we have a binary file test.out that is larger than 10 MB and there’s a certain kind of algorithm that requires us to process the data of the file in such a manner that needs us to repeat the process of:

• From the current position, seek 64 bytes and process the data at the beginning of current position.
• From the current position, seek -32 bytes and process the data at the beginning of the current position.

The actual process of the data is replaced by a pass statement since it does not affect the relative performance comparison between mmap and normal file access. The algorithm keeps processing the data of the file until it reaches a point that is larger than 10MB. The code that uses the normal file objects to perform the algorithm is listed in the file normal_process.py:

import os
import time

f = open('test.out', 'r')
buffer_size = 64
retract_size = -32
start_time = time.time()
while True:
    f.seek(buffer_size, os.SEEK_CUR)
    # Process some data starting at the current position
    pass
    f.seek(retract_size, os.SEEK_CUR)
    # Process some data starting at the current position
    pass
    if f.tell() > 1024 * 1024 * 10:
        break
end_time = time.time()
f.close()
print('Normal time elapsed: {0}'.format(end_time - start_time))

The code that uses mmap to process the data of the file is listed in the file mmap_process.py:

import os
import time
import mmap

f = open('test.out', 'r')
m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
buffer_size = 64
retract_size = -32
start_time = time.time()
while True:
    m.seek(buffer_size, os.SEEK_CUR)
    # process some data starting at the current position
    pass
    m.seek(retract_size, os.SEEK_CUR)
    # process some data starting at the current position
    pass
    if m.tell() > 1024 * 1024 * 10:
        break
end_time = time.time()
m.close()
f.close()
print('mmap time elapsed: {0}'.format(end_time - start_time))

Now you can compare normal_process.py and mmap_process.py in a simple for-loop in your shell:

for i in {1..3}
do
    python normal_process.py
    python mmap_process.py
done
normal time elapsed: 0.355199098587
mmap time elapsed: 0.296804904938
normal time elapsed: 0.371860027313
mmap time elapsed: 0.290856838226
normal time elapsed: 0.355377197266
mmap time elapsed: 0.305727958679

As you can see, mmap_process.py is about 17% faster than normal_process.py on average, because the seek function calls are performed directly against the memory in mmap_process.py while they are performed using filesystem calls in normal_process.py.

Modify a Memory-Mapped File in Python with mmap

In the following file mmap_write.py, we modify the content of a file write_test.txt using mmap and flush the changes back to disk:

import os
import mmap

f = open('write_test.txt', 'a+b')
m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_WRITE)
m[0] = 'n'
# Flush changes made to the in-memory copy of the file back to disk
m.flush()
m.close()
f.close()

Suppose write_test.txt contains one line of text:

$ cat write_test.txt
mmap is a cool feature!

After we run python mmap_write.py, its content will change to (note the first character of the sentence):

$ cat write_test.txt
nmap is a cool feature!

mmap Summary and Suggestions

Although mmap is a cool feature, keep in mind that mmap has to find a contiguous block of addresses in your process’s address space that is large enough to fit an entire file object. Suppose you’re working with large files on a system that does not have enough continuous memory region available to fit those files, then creating a mmap will fail. In additions, mmap does not work on certain special file objects like pipes and ttys.

The following list summarizes when you should use mmap:

• During multithreaded programming, if you have multiple processes accessing data in a read only way from the same file, then using mmap can save you lots of memory.
• mmap allows the operating system to optimize paging operations, which enables the program’s memory in pages to be efficiently re-used by the operating system.

The post Memory-Mapped (mmap) File Support in Python appeared first on Python Central.

Creating and Filling a QWebView

Content from a QWebView’s URL

You instantiate a QWebView like any othe QWidget, with an optional parent. There are a number of ways to put content into it, however. The simplest — and possibly the most obvious — is its load method, which takes a QUrl; the simplest way to construct a QUrl is with a unicode URL string:

web_view.load(QUrl('http://www.pythoncentral.org'))

That loads Python Central‘s main page in the QWebView control. Equivalent would be to use the setUrl method, like so:

web_view.setUrl(QUrl('http://www.pythoncentral.org'))

Load Arbitrary HTML with QWebView

There are other interesting ways to load content into a QWebView. You can load generated HTML into it using the setHtml method. For example, you can do something like this:

html = '''<html>
<head>
<title>A Sample Page</title>
</head>
<body>
<h1>Hello, World!</h1>
<hr />
I have nothing to say.
</body>
</html>'''

web_view.setHtml(html)

The setHtml method can take an optional second argument, the base URL of the document – the URL based on which any relative links contained in the document are resolved.

Other QWebView Content Types

A QWebView‘s content does not have to be HTML; if you have other browser-viewable content, you can put it in a QWebView using its setContent method, which accepts the content and optionally its MIME type and a base URL. If the MIME type is omitted, it will be assumed that it is text/html; no autodetection of MIME types has been implemented yet, though it is at least tentatively planned. I’ve not managed to find a list of MIME types that can be handled by the QWebView, but here is an example that works with a PNG file:

app = QApplication([])
win = QWebView()

img = open('myImage.png', 'rb').read()
win.setContent(img, 'image/png')

win.show()
app.exec_()

Interacting QWebView with JavaScript

Presenting a user with web-style content is useful in itself, but that content can be made interactive with JavaScript, which can be initiated from Python code.

A QWebView contains a QWebFrame object, which is useful to us right now for its evaluateJavaScript method. That method accepts a string of JavaScript, evaluates it in the context of the QWebView‘s content, and returns its value.

What values can be returned? The PySide documentation for QWebFrame, like that for PyQt and Qt itself, is not clear on that point. In fact, that information doesn’t appear to be available on the web at all, so I did some testing.

It appears that strings and booleans just plain work, and numbers, objects, undefined, and null work with caveats:

• Because JavaScript lacks separate integer and floating-point datatypes, numbers are consistently converted to Python floats.
• Objects are returned as Python dictionaries unless they are functions or arrays; functions are returned as useless empty dictionaries, and arrays become Python lists.
• undefined becomes None, sensibly enough.
• null becomes, less sensibly, ”. That’s right — an empty string.

Note especially the behavior regarding null and functions, as both can cause code that looks right to behave wrong. I see no better option for functions, but null is especially confusing; the only way to detect a null value from evaluateJavaScript is to do the comparison val === null on the JavaScript side before you return it to Python. (It is at this point that we collectively grieve over JavaScript’s ill-thought-out types.)

An important caution about evaluateJavaScript: it has all the security implications of JavaScript’s built-in eval, and should be used with the discretion that is so seldom displayed by front-end JavaScript coders. It would be far too simple, for example, to allow the execution of arbitrary JavaScript by naïvely building a string and sending it to evaluateJavaScript. Be careful, validate user input, and block anything that looks too clever.

Example of Evaluating JavaScript in a QWebView

Now, let’s throw caution to the wind and look at a simple example. It will show a form that allows the user to enter a first and last name. There will be a full name entry that is disabled; the user cannot edit it. There is a submit button, but it is hidden. (Don’t do this in real life, okay? This form is a usability and cultural-sensitivity disaster, and would be almost insultingly dumb to show in public.) We’ll supply a Qt button that fills out the full name entry, shows the submit button, and prints the full name to the console. Here’s the source of the example:

# Create an application
app = QApplication([])

# And a window
win = QWidget()
win.setWindowTitle('QWebView Interactive Demo')

# And give it a layout
layout = QVBoxLayout()
win.setLayout(layout)

# Create and fill a QWebView
view = QWebView()
view.setHtml('''
  <html>
    <head>
      <title>A Demo Page</title>

      <script language="javascript">
        // Completes the full-name control and
        // shows the submit button
        function completeAndReturnName() {
          var fname = document.getElementById('fname').value;
          var lname = document.getElementById('lname').value;
          var full = fname + ' ' + lname;

          document.getElementById('fullname').value = full;
          document.getElementById('submit-btn').style.display = 'block';

          return full;
        }
      </script>
    </head>

    <body>
      <form>
        <label for="fname">First name:</label>
        <input type="text" name="fname" id="fname"></input>
        <br />
        <label for="lname">Last name:</label>
        <input type="text" name="lname" id="lname"></input>
        <br />
        <label for="fullname">Full name:</label>
        <input disabled type="text" name="fullname" id="fullname"></input>
        <br />
        <input style="display: none;" type="submit" id="submit-btn"></input>
      </form>
    </body>
  </html>
''')

# A button to call our JavaScript
button = QPushButton('Set Full Name')

# Interact with the HTML page by calling the completeAndReturnName
# function; print its return value to the console
def complete_name():
    frame = view.page().mainFrame()
    print frame.evaluateJavaScript('completeAndReturnName();')

# Connect 'complete_name' to the button's 'clicked' signal
button.clicked.connect(complete_name)

# Add the QWebView and button to the layout
layout.addWidget(view)
layout.addWidget(button)

# Show the window and run the app
win.show()
app.exec_()

Try running it. Fill out a first and last name and click the button. You should see a full name and a submit button appear. Looking at the console, you should see the full name printed there as well.

This was a very heavily contrived example, but we can do much more interesting things: in the next installment, we’ll build a simple application that combines HTML in a QWebView and some other Qt widgets to usefully work with a web API.

The post PySide/PyQt Tutorial: QWebView appeared first on Python Central.

The program is going to receive a folder or a list of folders to scan, then is going to traverse the directories given and find the duplicated files in the folders.

This program is going to compute a hash for every file, allowing us to find duplicated files even though their names are different. All of the files that we find are going to be stored in a dictionary, with the hash as the key, and the path to the file as the value: { hash: [list of paths] }.

To start, import the os, sys and hashlib libraries:

import os
import sys
import hashlib

Then we need a function to calculate the MD5 hash of a given file. The function receives the path to the file and returns the HEX digest of that file:

def hashfile(path, blocksize = 65536):
    afile = open(path, 'rb')
    hasher = hashlib.md5()
    buf = afile.read(blocksize)
    while len(buf) > 0:
        hasher.update(buf)
        buf = afile.read(blocksize)
    afile.close()
    return hasher.hexdigest()

Now we need a function to scan a directory for duplicated files:

def findDup(parentFolder):
    # Dups in format {hash:[names]}
    dups = {}
    for dirName, subdirs, fileList in os.walk(parentFolder):
        print('Scanning %s...' % dirName)
        for filename in fileList:
            # Get the path to the file
            path = os.path.join(dirName, filename)
            # Calculate hash
            file_hash = hashfile(path)
            # Add or append the file path
            if file_hash in dups:
                dups[file_hash].append(path)
            else:
                dups[file_hash] = [path]
    return dups

The findDup function is using os.walk to traverse the given directory. If you need a more comprehensive guide about it, take a look at the How to Traverse a Directory Tree in Python article. The os.walk function only returns the filename, so we use os.path.join to get the full path to the file. Then we’ll get the file’s hash and store it into the dups dictionary.

When findDup finishes traversing the directory, it returns a dictionary with the duplicated files. If we are going to traverse several directories, we need a method to merge two dictionaries:

# Joins two dictionaries
def joinDicts(dict1, dict2):
    for key in dict2.keys():
        if key in dict1:
            dict1[key] = dict1[key] + dict2[key]
        else:
            dict1[key] = dict2[key]

joinDicts takes 2 dictionaries, iterates over the second dictionary and checks if the key exists in the first dictionary, if it does exist, the method appends the values in the second dictionary to the ones in the first dictionary. If the key does not exist, it stores it in the first dictionary. At the end of the method, the first dictionary contains all the information.

To be able to run this script from the command line, we need to receive the folders as parameters, and then call findDup for every folder:

if __name__ == '__main__':
    if len(sys.argv) > 1:
        dups = {}
        folders = sys.argv[1:]
        for i in folders:
            # Iterate the folders given
            if os.path.exists(i):
                # Find the duplicated files and append them to the dups
                joinDicts(dups, findDup(i))
            else:
                print('%s is not a valid path, please verify' % i)
                sys.exit()
        printResults(dups)
    else:
        print('Usage: python dupFinder.py folder or python dupFinder.py folder1 folder2 folder3')

The os.path.exists function verifies that the given folder exists in the filesystem. To run this script use python dupFinder.py /folder1 ./folder2. Finally we need a method to print the results:

def printResults(dict1):
    results = list(filter(lambda x: len(x) > 1, dict1.values()))
    if len(results) > 0:
        print('Duplicates Found:')
        print('The following files are identical. The name could differ, but the content is identical')
        print('___________________')
        for result in results:
            for subresult in result:
                print('\t\t%s' % subresult)
            print('___________________')

    else:
        print('No duplicate files found.')

Putting everything together:

# dupFinder.py
import os, sys
import hashlib

def findDup(parentFolder):
    # Dups in format {hash:[names]}
    dups = {}
    for dirName, subdirs, fileList in os.walk(parentFolder):
        print('Scanning %s...' % dirName)
        for filename in fileList:
            # Get the path to the file
            path = os.path.join(dirName, filename)
            # Calculate hash
            file_hash = hashfile(path)
            # Add or append the file path
            if file_hash in dups:
                dups[file_hash].append(path)
            else:
                dups[file_hash] = [path]
    return dups


# Joins two dictionaries
def joinDicts(dict1, dict2):
    for key in dict2.keys():
        if key in dict1:
            dict1[key] = dict1[key] + dict2[key]
        else:
            dict1[key] = dict2[key]


def hashfile(path, blocksize = 65536):
    afile = open(path, 'rb')
    hasher = hashlib.md5()
    buf = afile.read(blocksize)
    while len(buf) > 0:
        hasher.update(buf)
        buf = afile.read(blocksize)
    afile.close()
    return hasher.hexdigest()


def printResults(dict1):
    results = list(filter(lambda x: len(x) > 1, dict1.values()))
    if len(results) > 0:
        print('Duplicates Found:')
        print('The following files are identical. The name could differ, but the content is identical')
        print('___________________')
        for result in results:
            for subresult in result:
                print('\t\t%s' % subresult)
            print('___________________')

    else:
        print('No duplicate files found.')


if __name__ == '__main__':
    if len(sys.argv) > 1:
        dups = {}
        folders = sys.argv[1:]
        for i in folders:
            # Iterate the folders given
            if os.path.exists(i):
                # Find the duplicated files and append them to the dups
                joinDicts(dups, findDup(i))
            else:
                print('%s is not a valid path, please verify' % i)
                sys.exit()
        printResults(dups)
    else:
        print('Usage: python dupFinder.py folder or python dupFinder.py folder1 folder2 folder3')

The post Finding Duplicate Files with Python appeared first on Python Central.

As useful as that is, there is still something else that we could add that would make copying a directory much more user friendly!

Enter the progress bar from stage left. It’s a useful utility that all big name software uses to tell the user something is happening and where it is with the task it is doing.

So how do we incorporate this into copying a directory? Glad you asked, my question filled friend!

The Progress Bar

First, we’ll make a simple class to make printing progress 100 times easier as well as give us a nice reusable class to use in other projects that might need it.

Here we go. First we’ll add the __init__ method:

# -*- coding: utf-8 -*-
import sys

class ProgressBar(object):
    def __init__(self, message, width=20, progressSymbol=u'▣ ', emptySymbol=u'□ '):
        self.width = width

        if self.width < 0:
            self.width = 0

        self.message = message
        self.progressSymbol = progressSymbol
        self.emptySymbol = emptySymbol

That wasn’t too hard! We pass in the message we want to print, the width of the progress bar, and the two symbols for progress done and empty progress. We check that the width is always greater that or equal to zero, because we can’t have a negative length!

Simple stuff.

Alright, on to the hard part. We have to figure out how to update and render the progress bar to the output window without printing a new line every single time.

That’s where the carriage return (\r) comes in. The carriage return is a special character that allows the printed message to start at the beginning of a line when printed. Every time we print something, we’ll just use it at the beginning of our line and it should take care of our needs just fine. The only limitation this has is that if the progress bar happens to wrap in the terminal, the carriage return will not work as expected and will just output a new line.

Here is our update function that will print the updated progress:

    def update(self, progress):
        totalBlocks = self.width
        filledBlocks = int(round(progress / (100 / float(totalBlocks)) ))
        emptyBlocks = totalBlocks - filledBlocks

        progressBar = self.progressSymbol * filledBlocks + \
                      self.emptySymbol * emptyBlocks

        if not self.message:
            self.message = u''

        progressMessage = u'\r{0} {1}  {2}%'.format(self.message,
                                                    progressBar,
                                                    progress)

        sys.stdout.write(progressMessage)
        sys.stdout.flush()


    def calculateAndUpdate(self, done, total):
        progress = int(round( (done / float(total)) * 100) )
        self.update(progress)

What we just did is calculate the number of filled blocks using the total blocks needed. It looks a little complicated, so let me break it down. We want the number of filled blocks to be the progress divided by 100% divided by the total number of blocks. The calculateAndUpdate function is just to make our life a little easier. All it does is calculate and print the percentage done given the current number of items and the total number of items.

filledBlocks = int(round(progress / (100 / float(totalBlocks)) ))

That call to float is there to make sure the calculation is done with floating point precision, and then we round up the floating point number with round and make it an integer with int.

emptyBlocks = totalBlocks - filledBlocks

Then the empty blocks are simply the totalBlocks minus the filledBlocks.

progressBar = self.progressSymbol * filledBlocks + \
              self.emptySymbol * emptyBlocks

We take the number of filled blocks and multiply it by the progressSymbol and add it to the emptySymbol multiplied by the number of empty blocks.

progressMessage = u'\r{0} {1}  {2}%'.format(self.message,
                                            progressBar,
                                            progress)

We use the carriage return at the beginning of the message to set the print position to the beginning of the line and make a message formatted like: “Message! ▣ ▣ ▣ □ □ □ 50%”.

sys.stdout.write(progressMessage)
sys.stdout.flush()

And, finally, we print the message to the screen. Why don’t we just use the print function? Because the print function adds a new line to the end of our message and that’s not what we want. We want it printed just as we set it.

On with the show! To the copying a directory function!

Copying a Directory (Folder) with Progress in Python

In order to know our progress, we’ll have to sacrifice some speed. We need to count all of the files that we’re copying to get the total amount of progress left, and this requires us to recursively search the directory and count all of the files.

Let’s write a function to do that.

import os

def countFiles(directory):
    files = []

    if os.path.isdir(directory):
        for path, dirs, filenames in os.walk(directory):
            files.extend(filenames)

    return len(files)

Pretty straight forward. We just checked if the directory passed in is a directory, then if it is, recursively walk the directories and add the files in the directories to the files list. Then we just return the length of the list. Simple.

Next is the copying directories function. If you looked at the article mentioned at the top of this article, you’ll notice that I used the shutil.copytree function. Here we can’t do that because shutil doesn’t support progress updates, so we’ll have to write our own copying function.

Let’s do it!

First we create an instance of our ProgressBar class.

p = ProgressBar('Copying files...')

Next, we define a function to make directories that don’t exist yet. This will allow us to create the directory structure of the source directory.

def makedirs(dest):
    if not os.path.exists(dest):
        os.makedirs(dest)

If the directory doesn’t exist, we create it.

Now for the copy function. This one is a bit of a doozy, so I’ll put it all down in one place, then break it down.

import shutil

def copyFilesWithProgress(src, dest):
    numFiles = countFiles(src)

    if numFiles > 0:
        makedirs(dest)

        numCopied = 0

        for path, dirs, filenames in os.walk(src):
            for directory in dirs:
                destDir = path.replace(src,dest)
                makedirs(os.path.join(destDir, directory))
            
            for sfile in filenames:
                srcFile = os.path.join(path, sfile)

                destFile = os.path.join(path.replace(src, dest), sfile)
                
                shutil.copy(srcFile, destFile)
                
                numCopied += 1
                
                p.calculateAndUpdate(numCopied, numFiles)
        print

Firstly, we count the number of files.

def copyFilesWithProgress(src, dest):
    numFiles = countFiles(src)

Nothing too difficult about that.

Next, if there are actually files to copy, we create the destination folder if it doesn’t exist and initialize a variable to count the current number of files copied.

    if numFiles > 0:
        makedirs(dest)

        numCopied = 0

Then, we walk the directory tree and create all the directories needed.

        for path, dirs, filenames in os.walk(src):
            for directory in dirs:
                destDir = path.replace(src, dest)
                makedirs(os.path.join(destDir, directory))

The only tricky part here is that I replaced the source directory string in path, which is the root path, with the destination folder path.

Next, we copy files and update progress!

        for path, dirs, filenames in os.walk(src):
            (...)
            for sfile in filenames:
                srcFile = os.path.join(path, sfile)

                destFile = os.path.join(path.replace(src, dest), sfile)
                
                shutil.copy(srcFile, destFile)
                
                numCopied += 1
                
                p.calculateAndUpdate(numCopied, numFiles)

Here, we go through all the files, copy them, update the number of files copied so far, and then draw our progress bar.

That’s it! We’re done! Now you have a nice copying function that will alert you of the progress while copying files.

Moving a Directory (Folder) with Progress in Python

As a note, you may also change the shutil.copy function call in copyFilesWithProgress to shutil.move and have a move progress bar instead. All the other code would be the same, so I’m not going to rewrite it here. I’ll leave that up to you!

Until the next time, I bid you adieu.

The post How to Move/Copy a File or Directory (Folder) with a Progress Bar in Python appeared first on Python Central.

A common task when programming any web service is the construction of a solid database backend. In the past, programmers would write raw SQL statements, pass them to the database engine and parse the returned results as a normal array of records. Nowadays, programmers can write Object-relational mapping (ORM) programs to remove the necessity of writing tedious and error-prone raw SQL statements that are inflexible and hard-to-maintain.

ORM is a programming technique for converting data between incompatible type systems in object-oriented programming languages. Usually, the type system used in an OO language such as Python contains types that are non-scalar, namely that those types cannot be expressed as primitive types such as integers and strings. For example, a Person object may have a list of Address objects and a list of PhoneNumber objects associated with it. In turn, an Address object may have a PostCode object, a StreetName object and a StreetNumber object associated with it. Although simple objects such as PostCodes and StreetNames can be expressed as strings, a complex object such as a Address and a Person cannot be expressed using only strings or integers. In addition, these complex objects may also include instance or class methods that cannot be expressed using a type at all.

In order to deal with the complexity of managing objects, people developed a new class of systems called ORM. Our previous example can be expressed as an ORM system with a Person class, a Address class and a PhoneNumber class, where each class maps to a table in the underlying database. Instead of writing tedious database interfacing code yourself, an ORM takes care of these issues for you while you can focus on programming the logics of the system.

The Old Way of Writing Database Code in Python

We’re going to use the library sqlite3 to create a simple database with two tables Person and Address in the following design:

Note: If you want to checkout how to use SQLite for Python, you might want to have a look at the SQLite in Python series.

In this design, we have two tables person and address and address.person_id is a foreign key to the person table. Now we write the corresponding database initialization code in a file sqlite_ex.py.

import sqlite3
conn = sqlite3.connect('example.db')

c = conn.cursor()
c.execute('''
          CREATE TABLE person
          (id INTEGER PRIMARY KEY ASC, name varchar(250) NOT NULL)
          ''')
c.execute('''
          CREATE TABLE address
          (id INTEGER PRIMARY KEY ASC, street_name varchar(250), street_number varchar(250),
           post_code varchar(250) NOT NULL, person_id INTEGER NOT NULL,
           FOREIGN KEY(person_id) REFERENCES person(id))
          ''')

c.execute('''
          INSERT INTO person VALUES(1, 'pythoncentral')
          ''')
c.execute('''
          INSERT INTO address VALUES(1, 'python road', '1', '00000', 1)
          ''')

conn.commit()
conn.close()

Notice that we have inserted one record into each table. Run the following command in your shell.

$ python sqlite_ex.py

Now we can query the database example.db to fetch the records. Write the following code in a file sqlite_q.py.

import sqlite3
conn = sqlite3.connect('example.db')

c = conn.cursor()
c.execute('SELECT * FROM person')
print c.fetchall()
c.execute('SELECT * FROM address')
print c.fetchall()
conn.close()

And run the following statement in your shell.

$ python sqlite_q.py
[(1, u'pythoncentral')]
[(1, u'python road', u'1', u'00000', 1)]

In the previous example, we used an sqlite3 connection to commit the changes to the database and a sqlite3 cursor to execute raw SQL statements to CRUD (create, read, update and delete) data in the database. Although the raw SQL certainly gets the job done, it is not easy to maintain these statements. In the next section, we’re going to use SQLAlchemy’s declarative to map the Person and Address tables into Python classes.

Python’s SQLAlchemy and Declarative

There are three most important components in writing SQLAlchemy code:

• A Table that represents a table in a database.
• A mapper that maps a Python class to a table in a database.
• A class object that defines how a database record maps to a normal Python object.

Instead of having to write code for Table, mapper and the class object at different places, SQLAlchemy’s declarative allows a Table, a mapper and a class object to be defined at once in one class definition.

The following declarative definitions specify the same tables defined in sqlite_ex.py:

import os
import sys
from sqlalchemy import Column, ForeignKey, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship
from sqlalchemy import create_engine

Base = declarative_base()

class Person(Base):
    __tablename__ = 'person'
    # Here we define columns for the table person
    # Notice that each column is also a normal Python instance attribute.
    id = Column(Integer, primary_key=True)
    name = Column(String(250), nullable=False)

class Address(Base):
    __tablename__ = 'address'
    # Here we define columns for the table address.
    # Notice that each column is also a normal Python instance attribute.
    id = Column(Integer, primary_key=True)
    street_name = Column(String(250))
    street_number = Column(String(250))
    post_code = Column(String(250), nullable=False)
    person_id = Column(Integer, ForeignKey('person.id'))
    person = relationship(Person)

# Create an engine that stores data in the local directory's
# sqlalchemy_example.db file.
engine = create_engine('sqlite:///sqlalchemy_example.db')

# Create all tables in the engine. This is equivalent to "Create Table"
# statements in raw SQL.
Base.metadata.create_all(engine)

Save the previous code into a file sqlalchemy_declarative.py and run the following command in your shell:

$ python sqlalchemy_declarative.py

Now a new sqlite3 db file called “sqlalchemy_example.db” should be created in your current directory. Since the sqlalchemy db is empty right now, let’s write some code to insert records into the database:

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

from sqlalchemy_declarative import Address, Base, Person

engine = create_engine('sqlite:///sqlalchemy_example.db')
# Bind the engine to the metadata of the Base class so that the
# declaratives can be accessed through a DBSession instance
Base.metadata.bind = engine

DBSession = sessionmaker(bind=engine)
# A DBSession() instance establishes all conversations with the database
# and represents a "staging zone" for all the objects loaded into the
# database session object. Any change made against the objects in the
# session won't be persisted into the database until you call
# session.commit(). If you're not happy about the changes, you can
# revert all of them back to the last commit by calling
# session.rollback()
session = DBSession()

# Insert a Person in the person table
new_person = Person(name='new person')
session.add(new_person)
session.commit()

# Insert an Address in the address table
new_address = Address(post_code='00000', person=new_person)
session.add(new_address)
session.commit()

Save the previous code into a local file sqlalchemy_insert.py and run the command python sqlalchemy_insert.py in your shell. Now we have one Person object and one Address object stored in the database. Let’s query the database using the classes defined in sqlalchemy_declarative.py:

>>> from sqlalchemy_declarative import Person, Base, Address
>>> from sqlalchemy import create_engine
>>> engine = create_engine('sqlite:///sqlalchemy_example.db')
>>> Base.metadata.bind = engine
>>> from sqlalchemy.orm import sessionmaker
>>> DBSession = sessionmaker()
>>> DBSession.bind = engine
>>> session = DBSession()
>>> # Make a query to find all Persons in the database
>>> session.query(Person).all()
[<sqlalchemy_declarative.Person object at 0x2ee3a10>]
>>>
>>> # Return the first Person from all Persons in the database
>>> person = session.query(Person).first()
>>> person.name
u'new person'
>>>
>>> # Find all Address whose person field is pointing to the person object
>>> session.query(Address).filter(Address.person == person).all()
[<sqlalchemy_declarative.Address object at 0x2ee3cd0>]
>>>
>>> # Retrieve one Address whose person field is point to the person object
>>> session.query(Address).filter(Address.person == person).one()
<sqlalchemy_declarative.Address object at 0x2ee3cd0>
>>> address = session.query(Address).filter(Address.person == person).one()
>>> address.post_code
u'00000'

Summary of Python’s SQLAlchemy

In this article, we learned how to write database code using SQLAlchemy’s declaratives. Compared to writing the traditional raw SQL statements using sqlite3, SQLAlchemy’s code is more object-oriented and easier to read and maintain. In addition, we can easily create, read, update and delete SQLAlchemy objects like they’re normal Python objects.

You might be wondering that if SQLAlchemy’s just a thin layer of abstraction above the raw SQL statements, then it’s not very impressive and you might prefer to writing raw SQL statements instead. In the following articles of this series, we’re going to investigate various aspects of SQLAlchemy and compare it against raw SQL statements when they’re both used to implement the same functionalities. I believe at the end of this series, you will be convinced that SQLAlchemy is superior to writing raw SQL statements.

The post Introductory Tutorial of Python’s SQLAlchemy appeared first on Python Central.

What is a view?

In Django, a view is an endpoint that can be accessed by any client to retrieve data over the wire. For example, some views serve data rendered with a template to a web server that in turn serves the HTML page to a client. While other views serve JSON data to a web server and these views become part of a RESTful API.

In Django, views are simple Python functions or methods which return data to a front-end web server. When a request comes in, Django chooses a view by examining the request‘s URL after the domain name. In order to call the correct view when a request is made, you need to insert a list of URL patterns into myblog/urls.py. A URL pattern is a regular expression based string which specifies the generic form of a view‘s URL. For example, r'^posts/(?P\d{4})/$' matches all URLs whose parts after the domain name follow the pattern of a string ‘posts’ followed by a four-digits number.

Views for Our Django Website

In our blog website, we will write the following views:

• Post “index” view that displays a list of most recent posts or popular posts in the blog, just like the home page of Python Central
• Post “detail” view that displays the details of a Post, which is similar to each individual article page on Python Central

In the “index” view, a user should be able to see a list of latest Posts made by others. In the “detail” view, a user should be able to see the content of a Post as well as a list of Comments made by others.

Index View

Let’s write your first view “index” that shows a list of most recent Posts. Create a file myblog/views.py with the following content:

from django.http import HttpResponse

def index(request):
    return HttpResponse('This page shows a list of most recent posts.')

And modify the file myblog/urls.py:

from django.conf.urls import patterns, include, url

# Uncomment the next two lines to enable the admin:
from django.contrib import admin
admin.autodiscover()

urlpatterns = patterns('',
    # Add the following line to link the root URL to the function myblog.views.index()
    url(r'^$', 'myblog.views.index', name='index'),
    # url(r'^myblog/', include('myblog.foo.urls')),

    # Uncomment the admin/doc line below to enable admin documentation:
    url(r'^admin/doc/', include('django.contrib.admindocs.urls')),

    # Uncomment the next line to enable the admin:
    url(r'^admin/', include(admin.site.urls)),
)

Now you can access 127.0.0.1:8000 in your browser to view the content returned by myblog.views.index():

Since the return value of myblog.views.index is a string “This page shows a list of most recent posts.”, it shows up on this page. Next, we are going to modify the function myblog.views.index to return a list of Posts that are posted no earlier than two days ago:

from datetime import datetime, timedelta
from django.http import HttpResponse

from myblog import models as m

def index(request):
    two_days_ago = datetime.utcnow() - timedelta(days=2)
    recent_posts = m.Post.objects.filter(created_at__gt=two_days_ago).all()
    return HttpResponse(recent_posts)

Now you can refresh 127.0.0.1:8000 to see that no post is made less than two days ago:

Let’s go ahead and add a Post using the Admin interface at the Admin Site:

Click “Save” and the post admin page should show that a new post is added and there’re three posts total in the database now:

Now you can refresh the Home Page to see that one Post is returned:

Django Views Templates

Obviously, only displaying “Post object” is not very helpful to the users and we need a more informative page to show a list of Posts. This is when a template would be helpful to turn an useless page like the current one into a presentable one.

Create a new file myblog/templates/index.html with the following content:

{% if post_list %}
    <ul>
    {% for post in post_list %}
        <li>
            <a href="/post/{{ post.id }}/">{{ post.content }}</a>
            <span>{{ post.created_at }}</span>
        </li>
    {% endfor %}
    </ul>
{% else %}
    <p>No post is made during the past two days.</p>
{% endif %}

And modify the file myblog/views.py to use the Django template system to render the response of 127.0.0.1:8000:

from datetime import datetime, timedelta
from django.http import HttpResponse
from django.template import Context, loader 

from myblog import models as m
    
    
def index(request):
    two_days_ago = datetime.utcnow() - timedelta(days=2)

    # Retrieve a list of posts that are created less than two days ago
    recent_posts = m.Post.objects.filter(created_at__gt=two_days_ago).all()

    # Load the template myblog/templates/index.html
    template = loader.get_template('index.html')

    # Context is a normal Python dictionary whose keys can be accessed in the template index.html
    context = Context({
        'post_list': recent_posts
    })

    return HttpResponse(template.render(context))

Now you can refresh the Home Page again to see that one Post is rendered using the template in myblog/templates/index.html:

Although the current code of the view index() works, it’s a bit longer than necessary. You can shorten the code using a Django shortcut render:

from datetime import datetime, timedelta
from django.http import HttpResponse
from django.shortcuts import render
from django.template import Context
from myblog import models as m

def index(request):
    two_days_ago = datetime.utcnow() - timedelta(days=2)
    recent_posts = m.Post.objects.filter(created_at__gt=two_days_ago).all()
    context = Context({
        'post_list': recent_posts
    })
    # Render accepts three arguments: the request object, the
    # path of the template file and the context
    return render(request, 'index.html', context)

Django’s Detail View

Similar to the index view, the detail view shows a page that presents detailed information about a Post at URLs like /post/1/detail.html, where 1 is the id of a Post.

Similar to the index view, you need to write the URL pattern before writing the detail view. Modify the file myblog/urls.py in the following way:

from django.conf.urls import patterns, include, url                          

# Uncomment the next two lines to enable the admin:                          
from django.contrib import admin
admin.autodiscover()                                                         

urlpatterns = patterns('',                                                   
    url(r'^$', 'myblog.views.index', name='index'),
    # Map the view function myblog.views.post_detail() to an URL pattern
    url(r'^post/(?P<post_id>\d+)/detail.html$',
        'myblog.views.post_detail', name='post_detail'),
    # url(r'^myblog/', include('myblog.foo.urls')),                          
        
    # Uncomment the admin/doc line below to enable admin documentation:      
    url(r'^admin/doc/', include('django.contrib.admindocs.urls')),           
                                                                             
    # Uncomment the next line to enable the admin:                           
    url(r'^admin/', include(admin.site.urls)),                               
)

Then you can implement the view function in myblog/views.py:

from datetime import datetime, timedelta
from django.http import Http404, HttpResponse
from django.shortcuts import render
from django.template import Context
from myblog import models as m

    
def index(request):
    two_days_ago = datetime.utcnow() - timedelta(days=2)
    recent_posts = m.Post.objects.filter(created_at__gt=two_days_ago).all()
    context = Context({
        'post_list': recent_posts
    })
    return render(request, 'index.html', context)
                                                                             
    
# post_detail accepts two arguments: the normal request object and an integer
# whose value is mapped by post_id defined in r'^post/(?P<post_id>\d+)/detail.html$'
def post_detail(request, post_id):
    try:
        post = m.Post.objects.get(pk=post_id)
    except m.Post.DoesNotExist:
        # If no Post has id post_id, we raise an HTTP 404 error.
        raise Http404
    return render(request, 'post/detail.html', {'post': post})

Then we need to add a new template file at myblog/templates/post/detail.html:

<div>                                                                        
    <h1>{{ post.content }}</h1>                                              
    <h2>{{ post.created_at }}</h2>                                           
</div>

Now you can access the URL Post Detail:

If you access a Post whose id is not in the database, our detail view will show a HTTP 404 error page:

Summary

In this article, we wrote our first two views. The index view shows a list of posts that are created by users less than two days ago and the detail view presents the detailed content of a Post. Generally speaking, the process of writing a view involves:

• Write URL patterns for the view in myblog/urls.py
• Write the actual view’s code in myblog/views.py
• Write a template file in the directory myblog/templates/

The post Writing Simple Views for Your First Python Django Application appeared first on Python Central.

Using SQLite’s date and datetime Types

Sometimes we need to insert and retrieve some date and datetime types in our SQLite3 database. When you execute the insert query with a date or datetime object, the sqlite3 module calls the default adapter and converts them to an ISO format. When you execute a query in order to retrieve those values, the sqlite3 module is going to return a string object:

>>> import sqlite3
>>> from datetime import date, datetime
>>>
>>> db = sqlite3.connect(':memory:')
>>> c = db.cursor()
>>> c.execute('''CREATE TABLE example(id INTEGER PRIMARY KEY, created_at DATE)''')
>>>
>>> # Insert a date object into the database
>>> today = date.today()
>>> c.execute('''INSERT INTO example(created_at) VALUES(?)''', (today,))
>>> db.commit()
>>>
>>> # Retrieve the inserted object
>>> c.execute('''SELECT created_at FROM example''')
>>> row = c.fetchone()
>>> print('The date is {0} and the datatype is {1}'.format(row[0], type(row[0])))
# The date is 2013-04-14 and the datatype is <class 'str'>
>>> db.close()

The problem is that if you inserted a date object in the database, most of the time you are expecting a date object when you retrieve it, not a string object. This problem can be solved passing PARSE_DECLTYPES and PARSE_COLNAMES to the connect method:

>>> import sqlite3
>>> from datetime import date, datetime
>>>
>>> db = sqlite3.connect(':memory:', detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
>>> c = db.cursor()
>>> c.execute('''CREATE TABLE example(id INTEGER PRIMARY KEY, created_at DATE)''')
>>> # Insert a date object into the database
>>> today = date.today()
>>> c.execute('''INSERT INTO example(created_at) VALUES(?)''', (today,))
>>> db.commit()
>>>
>>> # Retrieve the inserted object
>>> c.execute('''SELECT created_at FROM example''')
>>> row = c.fetchone()
>>> print('The date is {0} and the datatype is {1}'.format(row[0], type(row[0])))
# The date is 2013-04-14 and the datatype is <class 'datetime.date'>
>>> db.close()

Changing the connect method, the database now is returning a date object. The sqlite3 module uses the column’s type to return the correct type of object. So, if we need to work with a datetime object, we must declare the column in the table as a timestamp type:

>>> c.execute('''CREATE TABLE example(id INTEGER PRIMARY KEY, created_at timestamp)''')
>>> # Insert a datetime object
>>> now = datetime.now()
>>> c.execute('''INSERT INTO example(created_at) VALUES(?)''', (now,))
>>> db.commit()
>>>
>>> # Retrieve the inserted object
>>> c.execute('''SELECT created_at FROM example''')
>>> row = c.fetchone()
>>> print('The date is {0} and the datatype is {1}'.format(row[0], type(row[0])))
# The date is 2013-04-14 16:29:11.666274 and the datatype is <class 'datetime.datetime'>

In case you have declared a column type as DATE, but you need to work with a datetime object, it is necessary to modify your query in order to parse the object correctly:

c.execute('''CREATE TABLE example(id INTEGER PRIMARY KEY, created_at DATE)''')
# We are going to insert a datetime object into a DATE column
now = datetime.now()
c.execute('''INSERT INTO example(created_at) VALUES(?)''', (now,))
db.commit()

# Retrieve the inserted object
c.execute('''SELECT created_at as "created_at [timestamp]" FROM example''')

Using as "created_at [timestamp]" in the SQL query will make the adapter to parse the object correctly.

Insert Multiple Rows with SQLite’s executemany

Sometimes we need to insert a sequence of objects in the database, the sqlite3 module provides the executemany method to execute a SQL query against a sequence.

# Import the SQLite3 module
import sqlite3
db = sqlite3.connect(':memory:')
c = db.cursor()
c.execute('''CREATE TABLE users(id INTEGER PRIMARY KEY, name TEXT, phone TEXT)''')
users = [
    ('John', '5557241'), 
    ('Adam', '5547874'), 
    ('Jack', '5484522'), 
    ('Monthy',' 6656565')
]

c.executemany('''INSERT INTO users(name, phone) VALUES(?,?)''', users)
db.commit()

# Print the users
c.execute('''SELECT * FROM users''')
for row in c:
    print(row)

db.close()

Please note that each element of the sequence must be a tuple.

Execute SQL File with SQLite’s executescript

The execute method only allows you to execute a single SQL sentence. If you need to execute several different SQL sentences you should use executescript method:

# Import the SQLite3 module
import sqlite3
db = sqlite3.connect(':memory:')
c = db.cursor()
script = '''CREATE TABLE users(id INTEGER PRIMARY KEY, name TEXT, phone TEXT);
            CREATE TABLE accounts(id INTEGER PRIMARY KEY, description TEXT);
            
            INSERT INTO users(name, phone) VALUES ('John', '5557241'), 
             ('Adam', '5547874'), ('Jack', '5484522');'''
c.executescript(script)

# Print the results
c.execute('''SELECT * FROM users''')
for row in c:
    print(row)

db.close()

If you need to read the script from a file:

fd = open('myscript.sql', 'r')
script = fd.read()
c.executescript(script)
fd.close()

Please remember that it is a good idea to surround your code with a try/except/else clause in order to catch the exceptions. To learn more about the try/except/else keywords, checkout the Catching Python Exceptions – The try/except/else keywords article.

Defining SQLite SQL Functions

Sometimes we need to use our own functions in a statement, specially when we are inserting data in order to accomplish some specific task. A good example of this is when we are storing passwords in the database and we need to encrypt those passwords:

import sqlite3 #Import the SQLite3 module
import hashlib

def encrypt_password(password):
    # Do not use this algorithm in a real environment
    encrypted_pass = hashlib.sha1(password.encode('utf-8')).hexdigest()
    return encrypted_pass

db = sqlite3.connect(':memory:')
# Register the function
db.create_function('encrypt', 1, encrypt_password)
c = db.cursor()
c.execute('''CREATE TABLE users(id INTEGER PRIMARY KEY, email TEXT, password TEXT)''')
user = ('johndoe@example.com', '12345678')
c.execute('''INSERT INTO users(email, password) VALUES (?,encrypt(?))''', user)

The create_function takes 3 parameters: name (the name used to call the function inside the statement), the number of parameters the function expects (1 parameter in this case) and a callable object (the function itself). To use our registered function, we called it using encrypt() in the statement.

Finally, PLEASE use a true encryption algorithm when you are storing passwords!

The post Advanced SQLite Usage in Python appeared first on Python Central.

To use webviews, you’ll obviously need to know some HTML, CSS and JavaScript, so basic familiarity with these languages is assumed in this article.

Webviews with SL4A: A Call and Two Hooks

When you first use webviews, you’ll find things work a bit differently than you’d expect coming from programming native graphical interfaces or programming web applications. Thankfully, things are much easier all round. For example, you don’t have to manage frame rates or buffers in your event loop as webviews are rendered by an engine, in SL4A’s case, WebKit, so the Python code doesn’t need to handle any of that. At the same time, you don’t have things like AJAX and websockets to manage; you just post simple events to and from the user interface. In many ways, you have the best of both worlds with webviews.

This is one of those things that’s easier to explain with some code, so we’ll just start with a hello-world and keep incrementing.

To build a webview based app, you need at least one HTML file and one Python script, which implement the view and an app to use it. For simplicity, we’ll call them index.html and main.py.

The Webview Call

Below is the content of index.html. The meta tag in the head is just there to tell the device not to try and rescale anything.

<!DOCTYPE HTML>
<html>
    <head>

        <meta name="viewport" id="viewport"
            content="width=device-width, target-densitydpi=device-dpi,
            initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0"
        />

    </head>

    <body>
        Hello World<be>
        with SL4a Webviews
    </body>
</html>

This is the content of main.py:

from android import Android
import time

droid = Android()
droid.webViewShow('file:///sdcard/sl4a/scripts/example/index.html')

time.sleep(4)

This script creates an Android object, called droid, which is standard procedure for SL4A apps ~ droid exposes the Scripting Layer API. The script then launches the webview using the droid’s webViewShow method, passing a local file URL as the only argument. Obviously, you need to edit that URL to point to your HTML file. The script then sleeps for 4 seconds, just so you have time to see the webview on your device before the Python script exits and kills it.

As you can see, launching a webview just takes a single call. To keep the program running indefinitely, and to manage communication between the Python script and the webview, the script will need an event loop, but there’s a couple of hooks you need to know about first.

The First Hook

Once you have your webview running, you need to pass messages to it and handle messages from it. Everything boils down to events.

Apart from the Android stuff, the following version of index.html is pretty basic. There’s an explanation of how it all works below.

<!DOCTYPE HTML>
<html>
    <head>
        <meta name="viewport" id="viewport"
            content="width=device-width, target-densitydpi=device-dpi,
            initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0"
        />

        <style>
            body { background-color: #EEE }
            #banner { text-align: center }
            #userin {
                font-size: 64px;
                position: absolute;
                left: 10px;
                right: 10px;
            }
        </style>

        <script>
            var droid = new Android();
            function postInput(input) {
                // Post the content of the text field to the python side if
                // the user has pressed enter (or soft keyboard equivalent)
                if (event.keyCode == 13)
                    droid.eventPost('line', input);
            }
        </script>
    </head>

    <body>

        <div id="banner">
            <h1>SL4A</h1>
            <h2>Text to Speech Machine</h2>
        </div>
    
        <input id="userin" type="text" spellcheck="false"
            autofocus="autofocus" onkeyup="postInput(this.value)"
        />

    </body>
</html>

Now, index.html has a textual banner at the top of the page, with a text field underneath that the user can type into. When the user enters some text, the string in the text field is passed to a function, postInput, which posts the string to the Python script as an event, using the Droid’s eventPost method. The first argument to eventPost, 'line', is currently irrelevant and could’ve been anything.

This was a good point to chuck in some simple CSS too, else the text field would have been really small and ugly.

Below is a new version of main.py. This version will launch the webview, then block and wait for an event. When it gets the event, it grabs the data, the string the user entered in the text field, and uses the Android Text to Speech Facade to read out the user input. It exits immediately afterwards, so it will only handle one event.

from android import Android
droid = Android()

droid.webViewShow('file:///sdcard/sl4a/scripts/example/index.html')

# Wait for an event from webview
event = droid.eventWait().result

# Get the data, the user's input
data = event['data']

# Do text to speech on the input
droid.ttsSpeak(data)

It’s easy to put the last three lines in a loop to handle multiple events, but it’s worth looking at the code a bit more first.

Line 6 assigns droid.eventWait().result to a variable, event. The eventWait method is blocking, it just waits for any event to be posted, then returns a dict that contains information about the event.

Note that all calls to the the Scripting Layer are RPC calls and communication is in JSON. It’s the incoming JSON that’s always returned, as a dict, by any call to the SL4A API. This is covered in detail elsewhere, but, in short, it’s the result that’s often all you care about, hence it’s commonplace to see API calls written like the one on Line 6.

Anyway, droid.eventWait().result, and therefore event in the script above, is a dict containing information about the event.

If the last three lines of main.py were inside a loop, it could handle multiple events, but then there’d be no way for the user to exit the app cleanly.

The following version of main.py can handle two types of event, line and kill events. The line events will be handled as before; the script just reads a line of user input out loud, but this time, this will happen inside a loop, iterating until a kill event occurs. A kill event will cause the script to exit, taking the webview with it.

The webview itself doesn’t currently have any way to post kill events, but that’s easily resolved later.

import sys

from android import Android
droid = Android()

droid.webViewShow('file:///sdcard/sl4a/scripts/example/index.html')

# Complete one iteration for each event
while True: 
    event = droid.eventWait().result
    
    if event['name'] == 'kill':
        sys.exit()
    elif event['name'] == 'line':
        droid.ttsSpeak(event['data'])

Note that the script references event['name'] and event['data']. The following JavaScript snippet just illustrates how events are posted from the front end. The call takes two arguments, both strings. The first is an event name, the second is some event data.

var droid = new Android();
droid.eventPost('name', 'data');

The arguments are passed into main.py as a dict, which it assigns to event. The first argument is available as event['name'], the second as event['data'].

The following version of index.html includes a little more HTML and CSS, adding a button that will post a kill event when it is pressed.

<!DOCTYPE HTML>
<html>
    <head>

        <meta name="viewport" id="viewport"
            content="width=device-width, target-densitydpi=device-dpi,
            initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0"
        />

        <style>
            body { background-color: #EEE }
            #banner { text-align: center }
            #userin {
                font-size: 64px;
                position: absolute; right:10px; left:10px;
            }
            #killer {
                font-size: 32px; padding: 16px;
                position: absolute; bottom:10px; right:10px; left:10px;
            }
        </style>

        <script>
            var droid = new Android();
            function postInput(input) {
                if (event.keyCode == 13)
                    droid.eventPost('line', input)
            }
        </script>

    </head>

    <body>
        <div id="banner">
            <h1>SL4A</h1>
        <h2>Text to Speech Machine 2</h2>
        </div>
    
        <input id="userin" type="text" spellcheck="false"
            autofocus="autofocus" onkeyup="postInput(this.value)"
        />

        <button id="killer" type="button"
                onclick="droid.eventPost('kill', '')"
        >KILL BUTTON</button>

    </body>
</html>

The app now has a webview that can post multiple types of event, distinguished by name, that carry data about the event to handler code in the Python event loop. Now that you understand how to post events ‘inward’, from JavaScript to Python, you just need to know how to post events ‘outward’, from Python to JavaScript. Then you have all the bits you need to do graphical user interfaces with webviews in SL4A.

The Second Hook

Posting an event from Python is really easy, especially if you know how it’s done in JavaScript. Basically, you just use the same API in the same way, with droid.eventPost('name', 'data'). Handling events in JavaScript is a little different though. JavaScript is event oriented, so the event loop is implicit. To handle events in JavaScript, you need to register callbacks. In this case, you register functions to handle events by name, using droid.registerCallback('name' some_function).

The following JavaScript snippet registers a simple callback, using a lambda, for handling spam events.

droid.registerCallback('spam', function(e) {
    alert(e.data)
});

Note that the function receives one argument, e, which is the event. It has all the properties you should now expect from an event object.

SL4A Webview: Complete Example

This final version of the example code implements a crude Python evaluator. It simply evaluates any Python expression the user enters and prints the result in the webview.

Doing this only requires a few small changes to index.html: In the body, there’s an empty div with an id named output, which is where the output of the evaluation will appear, and, in the style sheet, there’s a few simple tweaks for this output div. The only other change is to the JavaScript, where a callback is registered that will handle stdout events. The function just prints the event data to the output div, overwriting whatever might have been there before.

<!DOCTYPE HTML>
<html>
	<head>
	    <meta name="viewport" id="viewport"
	        content="width=device-width, target-densitydpi=device-dpi,
	        initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0"
	    />
	
	    <style>
	        body { background-color: #EEE }
	        #banner { text-align: center }
	        #userin {
	            font-size: 64px;
	            position: absolute;
	            right: 10px;
	            left: 10px;
	        }
	        #output {
	            font-size: 32px;
	            font-family: monospace;
	            padding: 20px;
	            position: absolute;
	            top: 250px;
	            }
	        #killer {
	            font-size: 32px;
	            padding: 16px;
	            position: absolute; bottom: 10px;
	            right: 10px;
	            left: 10px;
	        }
	    </style>
	
	    <script>
	        var droid = new Android();
	        
	        function postInput(input) {
	            if (event.keyCode == 13)
	                droid.eventPost('line', input)
	            
	            droid.registerCallback('stdout', function(e) {
	                document.getElementById('output').innerHTML = e.data;
	            });
	        }
	    </script>
	
	</head>

    <body>
	    <div id="banner">
	        <h1>SL4A Webviews</h1>
	        <h2>Example: Python Evaluator</h2>
	    </div>
	    
	    <input id="userin" type="text" spellcheck="false"
	           autofocus="autofocus" onkeyup="postInput(this.value)"
	    />
	    
	    <div id="output"></div>
	    
	    <button id="killer" type="button"
	            onclick="droid.eventPost('kill', '')"
	    >KILL BUTTON</button>
    </body>
</html>

On the Python side, the only change is in how line events are handled. Now, a function called line_handler takes the line, evaluates it, converts the result to a string and then passes the string to the webview as the data for a sdtout event.

import sys

from android import Android
droid = Android()

def line_handler(line):
    ''' Evaluate user input and print the result into a webview.
    This function takes a line of user input and calls eval on
    it, posting the result to the webview as a stdout event.
    '''
    
    output = str(eval(line))
    droid.eventPost('stdout', output)


droid.webViewShow('file:///sdcard/sl4a/scripts/example/index.html')

while True:
    event = droid.eventWait().result
    
    if event['name'] == 'kill':
        sys.exit()
    elif event['name'] == 'line':
        line_handler(event['data'])

Admittedly, this isn’t the most exciting or polished app in the world, but it does illustrate how the Events API can be used with webviews to create graphical user interfaces. You could use the code module to turn this example app into a Python interactive interpreter without too much work, or use ACE to build a code editor that’s better than the crappy SL4A built in one, or just replace the SL4A interface entirely.

There’s a ton of good JavaScript libraries available now days for building front-ends. With Python on Android underneath, you can get really creative.

Obviously, it’s all a bit of a hack, but it’s a nice one. For a dev who just needs to modify some phones, or a hobbyist wanting to put out a tablet RPG, really for anyone who just wants an nice easy way to do a graphical app on an Android device, webviews are ideal.

Note: The Events API is a little richer and a lot more powerful than this example demonstrates, but the call to webViewShow and the two hooks, posting events from JavaScript to the Python event loop and from Python to JavaScript callbacks, are all you need to build a webview based user interface.

Note: Remember, you’re still building a regular SL4A application, so you can use all the native Android UI components, like dialogues, toasts, notifications and so on, alongside webviews, giving you a nice and simple tool set for building rich Android applications with Python and HTML5.

The post Python for Android: Using Webviews (SL4A) appeared first on Python Central.