Is there a connection?

First of all, he is not quite sure whether there is a strong connection or not. In architecture, he says, there is a profound moral component which is we are not completely aware of in our profession as software engineers. In my opinion, software engineering is also have a moral component or it should have. Maybe it is not much emphasized as in architecture due to the fact that we love to concentrate on the technical side as it is mostly a technically challenging area. But after all, what we are building in software engineering is for humans, for a better living, for improving the quality of life. If there is a human factor, it boils down to one thing: morality, just as Christopher Alexander says.

Design Patterns

He also questions whether patterns in software is the same as the patterns he discovered in architecture. He says software patterns is a subset of pattern languages. He sees software patterns as a tool to facilitate the communication between software engineers. He stated three essential features that exist in pattern languages; moral component, creating coherence, and generativeness.

Moral component

He emphasises the vitality of moral component in pattern languages and the absence of it in software engineering. I think it is true that using patterns in software engineering does not directly improves the quality of life. But using patterns will contribute to building better software, even it is just better in a technical way. Also, maybe it will make the programmer feel better because the code will have a better structure or it will have life in it. It will, in turn, help powering the quality of life.

Creating coherence

He mentions a very important aspect of pattern languages. One of the aims of pattern languages is that creating coherence when patterns used together. This means, all the entities created using the pattern language should be in a way that creating harmony. On the other hand, if we look at the design patterns, there is no guarantee that what we will end up with is a coherent software. Software patterns seem to be more independent than patterns in architecture.

Generativeness

This is a larger and deeper concept than patterns. He defines generativity as a process that is capable of leading to a good structure. He says patterns should be generative thus allowing the process to produce an entity which is coherent and consistent with the context. I don't know exactly if we have this in software engineering but I think we should be after that. This is a very nice concept to have: a process or a path that leads to a good software.

For more about Stream of consciousness at Wikipedia.

BTW, Stream of Consciousness by Dream Theater is a great song.

Ertuğ Karamatlı

For Directed Studies I course at Bogazici University, Turkey

Abstract

Botnets are becoming a real challenge. New botnets are discovered everyday that employs more sophisticated methods to spread, defend, and attack. This article explores the techniques used by traditional botnets, current botnets, and also emerging threats by future botnets. Current botnets incorporate distributed network architectures and cryptography increasingly. Moreover, we present new techniques such as the use of strong cryptography, scripting languages, anonymity networks, embedded systems, steganography, and IPv6 tunneling.

Keywords: Botnets; P2P; Cryptography; Network Security; Information Security

Download (PDF)

Ertuğ Karamatlı, Supervised by Jean-Marc Desharnais

For Software Measures II course at Bogazici University, Turkey

Abstract

Increasing complexity is one of the difficulties that make software maintenance harder over time. This article discusses McCabe’s cyclomatic complexity then explores the ways of incorporating cyclomatic complexity in software maintenance and also how to take actions against increasing complexity.

Keywords: McCabe, Cyclomatic Complexity, Software Measurement, Software Maintenance, Software Engineering

Download (PDF)

Here is the command I used to find the search queries:

gunzip -c other_vhosts_access.log.* | grep cv\.html | grep "google\|bing\|search\|job\|project" | ccze -A

Then, I used whois command on IPs from interesting queries to locate where the queries are originating from.

Search strings are grouped by company name. Search query, search engine, OS, and browser combinations used are listed below.

Google

(inurl:CV | inurl:Vitae | inurl:Resume | intitle:CV | intitle:Vitae | intitle:Resume) AND Computer.Science AND ("Opensource" OR "Open Source") AND (C++ OR Java OR Python) AND (Linux OR Ubuntu OR kernel) -apply -jobs

• Search Engine: Google
• OS: Macintosh
• Browser: Chrome

(Python and c++) ("computer science"|CS) ((inurl:~resume) | (inurl:~vitae) | (intitle:~resume) | (intitle:~vitae)) -job -jobs -Ph.D

• Search Engine: Google
• OS: Windows
• Browser: Firefox

Microsoft

prototyping algorithms games cv resume

• Search Engine: Bing
• OS: Windows
• Browser: Internet Explorer

Amazon

resume 2010 (ca OR wa OR or) java -coffee c++ developer -job (unix OR linux) enterprise distributed (425 OR 415 OR 310 OR 206 OR 503 OR 509)

• Search Engine: Bing
• OS: Windows
• Browser: Firefox

Unknown origin but interesting

-inurl:dir -inurl:sample -inurl:samples -inurl:cvsamples -inurl:example -inurl:examples  scheduling (project OR management) (DoD OR security OR clearance)

• Search Engine: Recruiting.com (uses Google)
• OS: Windows
• Browser: Internet Explorer

(intitle:resume OR intitle:cv OR intitle:vitae OR inurl:resume OR inurl:cv OR inurl:vitae)  "programming competition" javascript

• Search Engine: Google
• OS: Windows
• Browser: Firefox

Conclusions

Google and Amazon used complicated and detailed queries while Microsoft preferred a simpler search string. Furthermore, there isn't much surprise in OS and browser choices. Google used its own browser Chrome and Microsoft used its own search engine Bing. Of course, these are based on my observations for last few months, they may use many different queries or browsers, I can't know for sure.

However, there is something very interesting about the usage of search engines for recruiting. Job seekers may apply SEO techniques to their CVs/Resumes. Preferably, optimizing for the company they want to work for, according to the search queries these companies use. RSEO or CVSEO might become popular in the near future.

Further Reading

• Searching for Candidates On Google: A Primer
• Google Search Strings for Recruiters Looking For Work
• Google Recruitment 101
• Is LinkedIn Actively Preventing Recruiters from Searching Profiles via Google?
• Recruiting.com
• Search engine optimization on Wikipedia
• Resume Information Extraction with Named Entity Clustering based on Relationships

blog comments powered by Disqus

Basic One-Liners

print [item*2 for item in [1, 2, 3]]
# prints: [2, 4, 6]

cond = True
print 'yes' if cond else 'no'
# prints: yes

Dynamic Function Arguments

def somefunc(*args, **kwargs):
    print 'args: %s' % str(args)
    print 'kwargs: %s' % kwargs

somefunc(1, 2, thirdarg=3)
# prints:
# args: (1, 2)
# kwargs: {'thirdarg': 3}

Pseudo Methods

class <span class="createlink">SomeClass</span>(object):
    def __getattr__(self, fname):
        def wrapper(*args, **kwargs):
            print fname, args, kwargs
        return wrapper

someobj = <span class="createlink">SomeClass</span>()
someobj.somemethod(1, 2, 3)
# prints: somemethod (1, 2, 3) {}

Dynamic Imports

def get_module(s):
    module = __import__(s)
    for i in s.split(".")[1:]:
        module = getattr(module, i)
    return module

module3 = get_module('module1.module2.module3')

Object-Dict Hybrid

class <span class="createlink">HybridObj</span>(object):
    def __getitem__(self, name):
        return self.__dict__[name]

    def __setitem__(self, name, value):
        self.__dict__[name] = value

    def set_all(self, dict):
        self.__dict__ = dict

    def get_all(self):
        return self.__dict__

obj = <span class="createlink">HybridObj</span>()
obj['mykey'] = 'myval'

print obj.mykey
# prints: myval

print obj.get_all()
# prints: {'mykey': 'myval'}

Error Recovery Pattern

app = App()

while True:
    try:
        app.main()
    except Exception:
        import StringIO, traceback
        output = StringIO.StringIO()
        traceback.print_exc(file=output)
        print output.getvalue()
        time.sleep(10)

Type Checking

obj = 3.14

if isinstance(obj, basestring):
    type = 'string'
elif isinstance(obj, (int, float)):
    type = 'number'
else:
    type = obj.__class__

print type
# prints: number

Decorators

def base_amount(value):
    def decorator(func):
        def wrapper(*args):
            a, b = args
            a += value
            return func(a, b)
        return wrapper
    return decorator

@base_amount(5)
def add(a, b):
    return a + b

print add(3, 4)
# prints: 12

Slice Notation

a = [1, 2, 3, 4, 5]

#copy a list
print a[:]
# prints: [1, 2, 3, 4, 5]

#copy in reverse
print a[::-1]
# prints: [5, 4, 3, 2, 1]

print a[1:-1]
# prints: [2, 3, 4]

print a[-2:]
# prints: [4, 5]

print a[:-2]
# prints: [1, 2, 3]

Functional style

map((lambda x: x*2), [1, 2, 3])
#prints: [2, 4, 6]

reduce(lambda x, y: x+y, [1, 2, 3])
#prints: 6

filter((lambda x: x>2), range(6))
#prints: [3, 4, 5]

Sorting according to a key

student_tuples = [
    ('john', 'A', 15),
    ('jane', 'B', 12),
    ('dave', 'B', 10),
]

print sorted(student_tuples, key=itemgetter(2), reverse=True)
# prints: [('john', 'A', 15), ('jane', 'B', 12), ('dave', 'B', 10)]

# taken from: http://wiki.python.org/moin/HowTo/Sorting/

Getting items from a dictionary according to a pattern

items = {
    'name': lambda x: x['user']['name'],
    'post_count': lambda x: x['posts']['count'],
}
dct = {
    'user': {
        'name': 'Someone',
    },
    'posts': {
        'count': 5,
    },
}

new_dct = {}
for n, fn in items.iteritems():
    new_dct[n] = fn(dct)

print new_dct
# prints: {'post_count': 5, 'name': 'Someone'}

Global Variables in Threads

import threading
global = threading.local()
global.somevar = 3

Multi-value return

def somefunc():
    return val1, val2

val1, val2 = somefunc()

Checking if a value exists in a list

if 'somestring' in ['this', 'list', 'of', 'strings']:
    print 'yes!'

Removing duplicate items in a list

print set(['a', 'b', 'a', 'c'])
# prints: ['a', 'b', 'c']

Raising Meaningful Errors

raise <span class="createlink">TypeError</span>('Some error about type %s' % (t,))
raise <span class="createlink">AttributeError</span>('Some error about attr %s' % (a,))
raise <span class="createlink">ValueError</span>('Some error about val %s' % (v,))

Updates

I have got some comments on Python reddit. I have fixed the errors, they are now working. I should have checked them, thanks for pointing them out!

blog comments powered by Disqus

Ertuğ Karamatlı and Selim Akyokuş

In INISTA 2010: Proceedings of International Symposium on INnovations in Intelligent SysTems and Applications, Kayseri, Turkey, June 2010

Abstract

This paper presents an effective method for resume information extraction. The information extraction process consists of 4 phases. In the first step, a resume is segmented into blocks according to their information types. In the second step, named entities are found by using special chunkers for each information type. In the third step, found named entities are clustered according to their distance in text and information type. In the fourth step, normalization methods are applied to the text.

Download (PDF)

It collects user actions using a JavaScript code that is placed in pages and hashes the collected data before sending to server periodically. Then your application requests generated random hashes from the server when needed.

User data utilized for now:

• Constant parameters: User agent, platform, screen width, screen height, screen color depth, referrer, current URL, and cookies.
• Dynamic parameters: JavaScript random function, current time, mouse moves, key presses, and key press timings.

All these parameters are hashed with SHA-256 before being sent to server, so there shouldn't be any privacy issue for end-users.

Server chooses 2 hashes received from clients randomly and hashes again with a secret key (salt) before sending to application.

I inspired by the USB True Random Number Generator that is posted to HN.

Try

My site (karamatli.com) runs UserEntropy since 06/27/2010. You can have a look at generated numbers here:

Download

You can download it in zip or tar format.

Get the source using Git:

$ git clone git://github.com/ertug/userentropy.git

UserEntropy on GitHub

How to contribute?

Fork it on GitHub. Have a look at TODO.

Authors

Ertug Karamatli (ertug@karamatli.com)

References

• Wikipedia, Entropy
• Wikipedia, SHA-2
• Data Flow Entropy Collector
• web.py

Object pooling is pre-allocating a number of objects to use later. It is done by creating a group of objects (pool) initially and when you need an object, you take one from the pool rather than creating it. When you are done with the object you send it back to the pool. It effectively means reusing of objects. This allows bypassing the object creation overhead and reducing the work of garbage collector. Thus, managing the memory yourself. This is, of course, should be used with care because if you really want to manage the memory, why use a high-level language like Python?

Despite the fact that many programs don't need object pooling, this technique can speed things up, especially in situations explained below.

When to use Object Pooling?

There are two main reasons to use object pooling:

• You need to create and destroy many objects quickly (i.e. limited lifetime of objects)
• Creating many objects that takes time to be initiated (i.e. class constructor does many things)

It is an optimization so it shouldn't be applied until there is a proved bottleneck. It will increase the overall complexity of your program and probably present additional bugs. Also, when you create many objects at start, your program may allocate too much memory throughout its lifetime. As a result, you should carefully analyze the trade-off here.

Sample Implementation

The harbor example is from Paul Graham's book ANSI Common Lisp. Here are two classes, one is using object pooling while the other is not.

class Ship():
    def __init__(self):
        self.name = None
        self.flag = None
        self.tons = None

class <span class="createlink">HarborWithPooling</span>():
    def __init__(self, count):
        self.pool = [Ship() for i in range(count)]
        self.harbor = {}

    def enter(self, name, flag, tons):
        ship = self.pool.pop()
        ship.name = name
        ship.flag = flag
        ship.tons = tons
        self.harbor[name] = ship

    def find_ship(self, name):
        return self.harbor[name]

    def leave(self, name):
        ship = self.harbor[name]
        self.pool.append(ship)
        del self.harbor[name]

class <span class="createlink">HarborWithoutPooling</span>():
    def __init__(self, count):
        self.harbor = {}

    def enter(self, name, flag, tons):
        ship = Ship()
        ship.name = name
        ship.flag = flag
        ship.tons = tons
        self.harbor[name] = ship

    def find_ship(self, name):
        return self.harbor[name]

    def leave(self, name):
        ship = self.harbor[name]
        del self.harbor[name]

Benchmark

Its execution time is measured while enter and leave methods are called 100,000 times in both classes. The test machine has an Intel Core 2 Duo 1.80GHz CPU and it is running Linux. Tested using Python 2.5.4. If you have time to test it on another version or OS, please let me know. You can download the complete source code here.

There is about %66 overall speed increase when using object pooling.

Conclusion

Using object pooling in Python increases total object creation and destruction rate more than two-fold. When used appropriately, it will help developing fast Python programs. Particularly, the programs that needs to be fast such as games.

Further Reading

• Wikipedia, Pool
• Brandon Corfman, Object pooling in Python
• Paul Graham, ANSI Common Lisp, 1996, pp. 226-227
• Brian Goetz, Java theory and practice: Urban performance legends, revisited

Pool image from: Stuck In Customs

blog comments powered by Disqus

Did you ever find yourself struggling to find out why an assignment is not working as you expected or a variable is being modified magically?

You should have heard of call-by-value and call-by-reference. These are evaluation strategies that many popular programming languages use. An evaluation strategy is a set of rules that defines how expressions are evaluated in a programming language.

Python uses a different evaluation strategy named call-by-sharing (or call-by-object). Actually call-by-sharing term isn't used much, there is inconsistency across sources. For example, Java programmers say Java is pass-by-value, on the other hand, Ruby programmers say Ruby is pass-by-reference, even though both have the same semantics. Call-by-sharing is generally not fully understood by beginner Python programmers. You can do many useful things without knowing much about call-by-sharing but you should know it to use the full power of Python.

Assignment always updates the reference and don't change the previous referenced object.

var = 3
var = 5

First assignment will create a variable named "var" in the local context then create an int object that has a value of 3. Finally it updates the reference of var to point to the new int object. Doing one more assignment will create a new int object again and updates the reference of var to point to this object. So it won't do anything directly to the previous object that has a value of 3. Additionally, if there isn't any reference to the old object like in this example, it will be garbage collected.

class A():
    pass

a = A()
a.var = 3
a.var = 5

Doing an assignment operation on an object variable will behave the same as previous example. Reference of "var" will change but reference of "a" won't be affected by the assignment.

There is actually two cases that makes it behave like call-by-value or call-by-reference. I will try to explain these cases.

Mimicking call-by-value

Call-by-value means the variable that you pass to a function will not change the original variable, instead use a copy. So, we need to create copies of variables.

To create a copy you can use the slice syntax if the object in question is iterable:

copy_of_var = var[:]

Or if it is a builtin:

copy_of_var1 = list(var1)
copy_of_var2 = dict(var2)
copy_of_var3 = set(var3)

Or as a last resort you can use the copy module.

You can create a copy of a variable before passing to a function or before returning a value from a function. Below are some examples.

Copy before passing:

def append_one(lst):
    lst.append(1)
    return lst

lst = [2, 3]
append_one(list(lst)) # here!

Copy before returning:

def append_one(lst):
    lst = list(lst) # here!
    lst.append(1)
    return lst

lst = [2, 3]
append_one(lst)

I think copying inside the function is more intuitive than the other but I don't see any side effect of using the other.

Mimicking call-by-reference

Call-by-reference means a function can modify the original variable that is passed from the context where the function call occurred. Changing items in an object does not change the object itself (object should mutable, of course). So if we wrap a variable inside a list, for example, it will be accessible both from the context of the function call and inside the function.

def add_one(n):
    n[0] += 1

n = [3]
add_one(n)
print n[0] # prints 4

Also, some of the builtin functions work "in place":

def sort_list(lst):
    lst.sort()

lst = [7, 3, 9]
sort_list(lst)
print lst # prints [3, 7, 9]

Further Reading

• Wikipedia, Evaluation strategy
• Stack Overflow, Python: How do I pass a variable by reference?
• Fredrik Lundh, Python Objects
• Fredrik Lundh, Call By Object

Updates

• 1 (16 May 2010): Fixed code samples to demonstrate mutable objects instead of immutable objects that doesn't reflect the idea correctly.
• 2 (16 May 2010): Added more information about call-by-sharing and an example to demonstrate how assignments work.

blog comments powered by Disqus