another day another vice another roll of the dice

Program config as a DNA strand

2013-04-20T03:01:00.001-07:00

This is a technical followup to the post about mind-altering programming languages, which concentrates, iterates and extends on the abstract DNA part.

Do you know how DNA looks like from the point of view of Python programmer?
DNA is a list. Of genes, and fillers in between.

    [ GENE, GENE, GENE, None, GENE, STUFF, GENE, .... ]

By the analogy with software, one GENE is one option. Here None or STUFF is something that is not recognized and not used in the process right now, but may be used later even in a different place for a different purpose. This leads to very interesting properties of configuration described as a DNA strand.

Property One: Everything is Optional and Configurable

Usually it is a tough choice what to include into your program config, because an option a day keeps users away. But with DNA config it doesn't matter - you can encode everything - every bit of flexibility you wanted, every feature-creeper, forward-thinking matter. Don't bother yourself with restrictions - include everything.

Property Two: Indefinite Logical Groups and Namespaces

To make sense out of tons of information in a DNA strand, you need to concentrate only on the parts that are relevant to you. This is done by "masking" (or marking) parts of the DNA code according to your current task. When you apply a mask, all GENEs of DNA that are not used are cleared. You can have multiple masks for different programs. An awesome feature-creeper would be to figure out mask automatically during the program run (program explicitly marks sequences what it reads and what it skips).

Property Three: Non-linear Option Identification

Because DNA is a list, you may think that you need to know option position to access it. This is not necessary. Index in a DNA is the most straightforward way to lookup an option value, and mask is a tool to help explore and identify its position. But the true power comes from the fact that a option can be identified by just analyzing content. Option can be a single GENE in specific format, or it can be identified as a sequence of GENEs that pass some validation logic. This makes is possible to write generic configuration analyzers, which can read these sequences to detect configuration problems, patterns and meaningful values to produce nice visualization out of them.

Property Four: Application Types

In real world™ DNA is packed in chromosomes. Humans have 46. Potatoes possess 48. Zombies are somewhere in between. In a virtual world of coding it makes sense to use one chromosome per application type. For example, every Python web framework has a lot of common parts, common features and minor details in these features, so a variety of such frameworks can be encoded with a single DNA strand.

Exercise

It may sound too abstract, but try to think how to build a shared DNA strand for two or more mini web application frameworks in Python. Just remember that DNA code includes both hardcoded application features and modifiable configuration options. And as usual look at BoxCar 2D for an inspiration.

Programming languages that alter your mind

2013-03-15T13:17:00.000-07:00

Today I went to Perl 6 site.
I left.
I can remember in details what brought me there.
I still shocked by the new world that opened before me.
Butterflies..

---

NOTE: This is a not very positive post `about life`. Don't waste you time if you don't like such stuff.

The text below contains depictions of deaths by coding, too much letters, biased speculations, insight into psychedelic culture and a long way to the conclusions you're may not be ready to face with. You've been warned.

---

Let's face it - programming languages alter your mind. They affect the way people think and solve problems. It creates problems in life and relationships. Languages are addictive, they provide a lot of fun to entertain one's brain well beyond solving simple crosswords and moving sprites of monsters on the screen through the means of other sprites guided by the mouse-type manipulator. And while programming languages are fun, they are also harmful and even toxic.

A language is toxic when it doesn't match your expectations, such as were mine that resulted in filling Python bug #17426 (which I called the subzero wart, but that doesn't really matter). A language is harmful when it requires you to remember yet another thing from its "funny" behavior, and for that purpose you need to sacrifice a mem cell that was dedicated to store the name of your girlfriend's favorite flowers. It is harmful if it affects your mood, and you get butthurt when you read bad things about it.

---

Programming can be fatal for a girlfriend, relatives and the person in general. Much like eating mushrooms. I was only at two funerals of people below the age of 25. Both of them died of pneumonia, because both were not attentive enough to the symptoms of the cold they got, and that's because their attention was completely absorbed by tasks they needed to complete for their jobs. Their body failed, because their brain was unable to dedicated resources to timely response to the alerts of life sustaining system.

That's why people quit programming. That's why salaries are so high. But that doesn't mean other jobs are not harmful either. Low salary can be much more harmful in many ways leading to low self esteem and to even more complicated problems with relatives, friends and society. Did I say mushrooms can do this too?

---

Programming can be fatal even for the people with positive strong attitudes to pursue their own goals in life. If taken without prescription or without negative sides of the daily jobs in bad companies, programming becomes addictive - fun for one who practices and a loss for the human society. Ignoring everything around, being busy with developing your own _____ (framework, game, startup, social network, ...). A loss that happens when a person is leaves the job to pursue its virtual dreams about programming. Dreams disconnected from reality. And reality is something that you've completely ignored and skipped while being buried under the pile of daily tasks at your big corporation. Did I say that mushrooms can cause this too?

---

Real life is tough. It requires interaction with people, responding to signals and basically you start to forget your ideas, dreams and become more far away from your goals. New ideas are less likely to visit you, because maintaining a business requires a mindset of a library with shallow functions, not a scientific library with modelling algorithms. Maintaining a business is reinventing the bicycle. Over and over, day after day. Pursuing your dreams is reinventing the bicycle. Over and over, day after day. That keeps people busy, and the progress slow.

Programming languages can make the world less harmful and more close to the reality. Programming languages that are accessible, that save time and don't have a tough legacy. Enabling people to adapt them, discuss them and invent their own. If people can not invent ideal programming languages by themselves, by calculating, by using maths and inductive reasoning - they should try genetic algorithms. Did I mention that mushrooms can mutate?

---

The more experience I gain, the less I want to learn the peculiarities and differences between the languages. I just want them to work and I'd gladly outsource the task of inventing such working language to genetic algorithm. Algorithm where I involved as a social actor that makes selection based on shared experience with other actors. Genetic algorithm is where you have a set of parameters written into one long line. Mutation is a change of some or all of these parameters - sometimes completely random, sometimes guided. If your programming language has parameters - syntax, functions, modules, features - you can describe it using this long line called DNA.

To apply genetic algorithm you need a selection method - this is a shared user experience. You need an identification for the language a DNA sequence, but also a registry of names for and versions for human consumption. You need to abandon backward compatibility excuses and start building reusable components that will become new parameters in the language DNA sequence.

---

It starts to get late. Tomorrow there will be a new day, new karma and new ideas, so I need to finish this today. If programming languages can alter you mind, you can use this mind to alter the language. It is well known that when you look into the abyss, the abyss also gazes into you, but it is less known that it is an iterative process. Did I mention that shrooms can give you powers?..

Let me say goodbye and wish you all the best, dear reader. I hope you had a good trip, and now enjoy the expanded state of your mind and consciousness. Did I mention mushrooms? Right.

Formatting API Anti-Pattern

2013-02-27T04:05:00.001-08:00

I was meditating over subset of core Python API that's dedicated to self-inspection in a running script. This API consists of inspect + traceback + sys.exc_* + magical locals() and ideally should give you a total understanding of Python state at a given point in your program. But that doesn't happen, because things are complicated, and here is why..

I wanted to get a chain of callers for the function I was debugging. I don't like pdb at all - it is so outdated for 2013 that deserves a separate GSoC project. I prefer to just insert debug statements into the code and after several weeks and attempts I came up with a helper to get the name of the caller that I used to log and debug such complex projects as SCons and Spyder. I am not as smart as everybody else, but even I didn't expect to spend weeks researching this basic stuff.

I think the reason historically is that people tried to produce the meaningful data using functional approach without taking due attention on the data quality itself. Is the data complete, sufficient and accessible? You can not say that from the manual. The manual (see inspect) encourages you to hack your way in the debris of various bits of data that elusively fail to create a complete picture - the puzzle just doesn't solve, but you can't see this, because you don't know what is the structure of the Python state at a given point in your program and what to look for. The API encourages you to find functions that join and analyse the bits, you fail and try again and in the end you come up with a function filled with uncertainties, that just does some formatting magic.

"Formatting API Anti-Pattern" may not be the best term for that, but it is the API (data structures, functions and other stuff) so complex and incomplete that you can not see this, and instead of that forced to write formatting functions just to make sense of this incomplete data. We are so used to the fact that "API provides abilities" that completely miss the point that it also "provides limitation", and the data in API can be incomplete, insufficient (incomplete for certain tasks) and not accessible (code becomes very complicated to handle).

How to fight this? Think about data first. What data should be available for inspection at a given point in your program when this program is paused? Can you see this data in the manual? If not, then you're dealing with "Formatting API".

Ghosts in the shell

2013-02-17T09:28:00.001-08:00

At the beginning of the era human controllers were dominating in the shell. Now only their ghosts are controlling processes, launching programs and executing tasks server side. Quite often without any shells at all.

Updated the CHAOS speccy.

Using getopt with optparse (or how to move from getopt gradually)

2012-12-14T05:58:00.000-08:00

TL;DR: https://bitbucket.org/techtonik/scons/commits/bcb60b

SCons has a very old and interesting codebase with a lots of outdated and unusual stuff that makes it more difficult to extend. One such thing is getopt library, which is a predessor for Optik library (written by Greg Ward) now better known as optparse.

So I wanted to replace getopt with optparse, but didn't want to change everything in one step, because I didn't have time to check every option. Instead I decided to parse options I needed with optparse and leave everything else to the old getopt engine.

getopt only needs a list of arguments to work. sys.argv[1:] to be exact. This is also the second half of result returned by OptionParser.parse_args() function. The only problem was to teach OptionParser to ignore unknown options and leave them in arguments. Strange thing, but Optik examples included this user story, completely ignored in optparse documentation. To make this long user story short, you need to subclass OptionParser to use getopt with optparse:

# "Pass-through" option parsing -- an OptionParser that ignores
# unknown options and lets them pile up in the leftover argument
# list.  Useful to gradually port getopt to optparse.

from optparse import OptionParser, BadOptionError

class PassThroughOptionParser(OptionParser):
    def _process_long_opt(self, rargs, values):
        try:
            OptionParser._process_long_opt(self, rargs, values)
        except BadOptionError, err:
            self.largs.append(err.opt_str)
    def _process_short_opts(self, rargs, values):
        try:
            OptionParser._process_short_opts(self, rargs, values)
        except BadOptionError, err:
            self.largs.append(err.opt_str)

parser = PassThroughOptionParser(add_help_option=False)
parser.add_option('-a', '--all', action='store_true',
                      help="Run all tests.")
(options, args) = parser.parse_args()

#print "options:", options
#print "args:", args

Now pass args down to the getopt call and you're all set.

P.S. In argparse you can use ArgumentParser.parse_known_args() function.

Update 2013-02: For humane option parsing you should definitely see docopt library.

Good reference on Python magic methods

2012-12-05T15:19:00.002-08:00

I've just stumbled upon this manual about Python magic methods and it's really amazing. Definitely a good refresher and highly recommended.

http://www.rafekettler.com/magicmethods.html

/me wonders if the same engineering technique can be applied to official documentation corpus..

Cinematic journey approach for Python development

2012-11-20T04:15:00.000-08:00

Quotes page (fixed in stone) is silent about the one who said that Python, compared to other languages, allows to directly put thoughts into the code. I couldn't disagree with this, but taking idealistic approach, this was more true with Python 2 when coding on a system level, but not so true with the great coming of the web and i18n. So, what's wrong now? I don't have a clean and up to the point answer, because many people still think that there is nothing wrong with the Python. Probably the right question is: why Python is not better than it is now?

This one of the complicated questions nobody is able to answer fully. 42 is the answer, but does the question clear enough? The question is probably too complex for a good technical answer and should undergo decomposition. The decomposition can be achieved by clarifying. What means "better"? More easy to code in. Why is it hard to code? Here goes a list of problems...

...

Well, there is no list. Therefore there is no visibility, and without visibility no answer is possible. Gain visibility into the list of problems that make Python not-as-good as we want it to be is the primary step to take to make all subsequent steps reasonably grounded for a good party quest (and sane development roadmap for community to focus on).

Historically there were several driving forces behind Python development - mailing lists, bug reports and PEPs. PEPs more than the bugs. Mailing lists somewhere in between (YMMV).

ML were good until people had a lot of time to follow up. Bugs are good at tracking status of things, but they are tuned for fixing things and scratching issues, so language research naturally falls out of context in bug tracker interface. PEPs.

PEP is a good thing that helped to free Python core from featurecreep damage, provided a basis for discussions over a long period of time and insight into decisions over the language development. But PEPs start to fail, and the reason why they do this is the lack of time and energy to iterate over them. Most people can't say if technology is good or bad before testing it (version control as an example), and PEPs with lengthy pieces of design detail assume prior experience with the problem, require thorough imagination to see if the solution will play well.

PEPs require a lot of concentration - the resource of a big shortage nowadays, especially of professional grade. Which is not a surprise if you look at how good HR and management technologies are developed in modern world to keep people busy and involved. We can only hope that collective minds of big corp.s are somehow bugged with the problem and look for solutions to divert their resource flow to improve the grounds they are standing on. Let's hope that community can back up their support, and also somehow bugged with the problem about how to lower barriers of requirements, responsibility, experience and technical expertise for occasional community member, a student or elderly accountant, to be useful in Python development process. Lets's hope that both parties are interested enough to constantly improve ways to use the resource flow to the fullest extent possible.

There are two things that can be help here (and make Python better that it is now) - first one is to improve visibility. It takes its roots in cinematic industry and it's called scenario. Second one is to improve the process and it is a best practice developed over the time by user experience professionals. This one named customer journey map.

What keeps me away from putting my thoughts into code when I write Python?

"""Python forces me to maintain a lowest level structure of my writing - the indented layout, a good thing. Although this also comes with a pain while debugging, because Gangam style multiline comments require me to remember to indent them as well.""" - this is a scenario. You can add various metrics to it, such as:

"""I have only 7 operational attention slots in my mind, and one constantly falls out, because I have to pay attention to complicated commenting requirement.""" - the metrics here directly influences how deep one person can operate at any given moment. It is basically that multiline comments with strings are stealing concentration.

"""Those indentation errors are driving me mad every time I forget to indent multiline comment for debugging.""" - this says that a person uses iterative approach to debug problems, often commenting a lot, and probably in production environment using non-tuned editor. That's another scenario where Python comment hack doesn't play well.

Scenarios have two good qualities - they are short and can be conflicting between each other. PEP is on the other side - it is self-sufficient. To notice that PEP is contradictory - you need to attentively and thoroughly read it or write it yourself. It takes a lot of time. Scenarios are somewhat emotional, they are easy to remember and refer to. This makes it possible to concentrate on conflicting scenarios, outline conflicting points and concentrate all work around them rather than around vague opinions, which makes the whole process of looking for compromises (or good solutions) more fun and involving.

To summarize, the scenario is a good title to remember and a short story to tell. What is the difference between scenario and a StackOverflow question? Question may not have a story, scenario may not contain questions. What's the difference between scenario, use case and user story? "Use case" is an enterprise slang, "user story" is an agile term. Both may have some definitions. Scenario is just scenario, like in movie. You should replay it to see how it works. Scenario is for humans, it is less formalized and comes with emotions included (YMMV).

Let's skip to another example of problem with Python usability on a higher level - packaging - and present another tool from usability domain that can help with analyzing processes in general.

What's wrong with Python packaging that everybody constantly rewrites it?

I didn't intend to include it here first, but a half an hour ago I spotted this article - http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/ If distutils/setuptools had a scenario database for packaging, it could be possible to analyze limitations of Python in regard to each scenario. This analysis is similar to PEP, but not necessary a proposal and not necessary so extensive. Scenario may contain a history of the problem, a short description, summary and link to other conflicting scenarios. The role of scenario database is to aid decision making process and an easy reference for new people facing the same problems.

Scenarios can be universal and it is a good analysis tool. You can substitute Ruby for Python and look how good this specific workflow looks for the different system.

"""I can't list installed Python packages, why?""" - does anybody have a link? """I can't find the answer""", and that's another scenario about usefulness of scenarios.

So, to fix packaging there should be a way to operate with scenarios. There should be at least a list of scenarios (or better indented tree), so that (y-)hackers of a new packaging tool could go over it, think about their approach, tick checkboxes and hopefully, spot and bring to the surface this "Essential Packaging Restraint" that eats a whole generations of people. The point is to spot the problem before starting to code.

The scenario DB will help, but there is another usability tool that can make packaging, bug tracking and other development processes more streamlined (less time consuming, more fun and engaging). This tool is called Customer Journey Map, and it shows to people, who are not experiencing any problems with the process, where those problems are for somebody else. This map is also a good starting point in web site redesigns, conference organizations, all kinds of activities that involve people, or more specific, a single person named "Customer", barriers this guy is facing and steps to remove these barriers.

I can't extend to a great detail about CJM in this post due to time constraints. I was impressed by a presentation of awesome UXpresso team, there might be a video available, but it is likely Russian only, and I've heard of at least one major Python company (wargaming.net) that uses it extensively, so I can only give you a pointer for now. It will be interesting to make presentation of this technology for Python contribution process and talk about CJM at PyCon, but I am unlikely to afford the participation costs, so somebody else should do this.

About Environment

2012-07-09T10:51:00.000-07:00

In Python applications environment is often an ambiguous term that needs clarification. In general sense `the environment` is system environment with PATH and friends accessible from os.environ within Python. But in Python applications it can mean different things.

In Trac `the environment` is a directory with settings, database and other files related to one Trac site.

In SCons `the environment` is a structure in memory that holds dependency trees, helper functions, builders and other stuff. It is written to disk only for caching.

Quite many other applications have some kind of environment for their own purpose with meanings close to either Trac or SCons, which often confuses newbies or strangers who are not aware about the context. Software development clearly needs more specific terms in English to make people write and read in the same language without those excessive contexts.

Spyder IDE Internals: Highlighting in 2.2.0dev

2012-05-26T01:56:00.001-07:00

UPD: Highligher support in code editor have been improved, so its instance can now be traced easily.

The entrypoint to syntax highlighting in Spyder IDE is located in CodeEditor widget at spyderlib/widgets/sourcecode/codeeditor.py

CodeEditor widget is basically file content in one of main tabs. The whole stack of tabs is called EditorStack. So editors are grouped in editor stacks, each editor renders one file, and each editor has its own highligher created from assigned self.highlighter_class

Highlighers are implemented with Qt's QSyntaxHighlighter. No pygments, nothing like this, so can't say if Qt is faster, but it should be. Default self.highligher_class for every code editor is TextSH. The actual instance is created by set_language() method called from setup_editor(). If setup_editor() is not called, the highlighter can be unset, but frankly I don't know what's the purpose of using such editor.

if set, syntax highlighter (self.highlighter) is responsible for:

coloring raw text data inside editor on load
coloring text data when editor is cloned
updating document highlight on line edits
providing color palette (scheme) for the editor
providing data for Outliner

self.highlighter is not responsible for:

background highlight for current line
background highlight for search / current line occurrences

Enjoy hacking.

Multidimensional programming

2012-04-28T05:28:00.000-07:00

From time to time I reverse some piece of open source code just to understand how it works. The biggest problem with that is the amount of things I need to juggle in my head until they find a place on the canvas of reversed blueprint. If there are too many - I give up or put the project on the back burner. This was with Stackless and Twisted, but I really glad to finally get to them.

Quite often is starts with some bug that seems easy to fix and I go for it. In ideal world it shouldn't take more than 15 minutes for studying the code to gain understanding what should be changed, but it is also important to have a confidence that the change won't break anything.

I won't tell you how to deal with that complexity, but I'd like to share an idea that I've got from Large Hadron Collider. =) Let me tell you that story...

I always had troubles trying to squeeze more than 3 dimensions into my head. I could imagine a dot moving along X or Y axes easily, could imagine it moving along Z axis, but everything more than that caused a confusion.

I could have my brain exploded when some time ago a friend of mine tried to explain what's going on in LHC. He said that physics and mathematicians are trying to figure out how many dimensions our universe has. They assume that our universe has more than 4 dimensions (3 coordinates and 1 time value). In fact they argue that the truth is somewhere between 9 and 26!

I am not a fan of scientific theories - my neurons are pretty calm to that matter, but when you just need to understand, because the person in front of you is knowledgeable and tries all the best to explain - here is when you start to feel colliding brain cells under your skull. To the honor of my friend, he didn't try to build-up suspense (as I would probably do before explaining properly) and managed to draw a clear picture in my mind. A standard plot with two axes - X and Y, and a dot.

"Look at this dot on X/Y plane. This dot has coordinates in two dimensions - X and Y", - he explained. "These are only two you can see here. But this dot is from the real world, so it also has Z coordinate", - he added putting a label "Z=0.1" next to the dot. "This dot also have speed", - he drew "V=0.1 m/s". "Now we can see values for all 4 traditional dimensions of our dot, but we can add more. There are many things that we can classify our dot. It probably has color.", - he drew a color box and a hex value. "It has temperature", - a new label "t = 25С" appeared in the column. "There are 6 already, and you can add your own.". I immediately imagined a dot travelling through canvas of "Stars!" game with all those labels nearby that constantly change their values as the dot moved. "Wow! Now I see", - that was a nice feeling - I must admit I can be pretty dumb sometimes. =) It didn't make a science fan out of me, but did make an important short-circuit in the depths of my head, which popped up a few months later..

A few months later.

As it usually happens in software development you need at least some basic design before setting down to code, and with time design phase completely faded from our process into "who made it first - wins" motto. It is hard to argue with that, so I didn't. But as a result after some time, the stability of our releases dropped. People were not communicating, and started to forget about some aspects of our system that could fire at any time in completely unexpected places. Even though all our code undergoes reviews, the reviewers tend to forget about those aspect as well. The process went out of control - we couldn't keep all the aspects in our heads when planning for the next feature to be released, and I found myself in same state I was when trying to understand the complexity of multidimensional string theories. That lead me to the idea that we need to control the amount of aspects that we need to keep in mind when coding and restructure our architecture to keep those aspects at minimum. This will help us to regain sense of confidence into what we are doing, and save some money from the bills from the nearest bar on the name of our release manager.

So, the `multidimensional programming` concept means that at any point of your code there are multiple things that you should be aware of. These things are the 'dimensions', and the more you have - the more complex your application is. Basically, these are the things that can be broken by any change at this place. Good application architecture is orthogonal - you can work at only one dimension at a time without thinking too much about all others. But you need to know all of them anyway to gain a sense of confidence.

For example, a recent change in Spyder IDE requires me to rename some file. This breaks the code, which I grep and fix - that's one dimension. But it will also likely to break a translation for the strings in that file, because the string is now at a different place. I imagine that nobody will be interested to check and translate the same stuff over and over again, so that's one more thing I'd like to avoid, so I should keep that in mind.

Another example are web applications. You need to keep in mind 'user privileges', type of HTTP request ('ajax', ...) and response ('json', ...) required. You need to make sure `critical errors` are handled and reported, and `static files` are correctly served by web server. You need to save incomplete `data between requests`, and cleanup it where possible. Make sure there is sufficient `XSRF protection` and `browser compatibility`. There are a lot more to it, and so far these have nothing to do with the logic of your web application. Frameworks help to deal with that, but inside they are still multidimensional. If framework is not flexible for you - that probably means it tries to keep some dimensions orthogonal, and there could be a good reason for that.

Maybe that's not much, but at least now you can argument that Large Hadron Collider experiments have much in common with software engineering, and when somebody asks about your job - you can proudly state that you're on par with scientists with their string theory, but in your own big enterprise application universe. =)

Rietveld architecture: AppEngine/Django request processing

2012-02-17T03:24:00.000-08:00

Just a quick note/reminder of the request handling flow in AppEngine environment for mixed AE/Django application such a Rietveld . Hopefully it provides a good entrypoint to understand how AppEngine works. I use Rietveld as an example, because this project is basically born to show how to run Django on AE.

Rietveld is a Django application that is run by AppEngine. Let's leave all Django stuff aside and learn how AppEngine loads and initializes Python applications first.

Import and execution in Python web apps

In PHP when your code is executed, it is read and interpreted (executed) from start to finish every time a new request arrives. In Python the code is read only once (imported), executed and on subsequent requests only the part that handles request is invoked over and over. The catch is that first request is always different in Python.

What happens when application is uploaded to AppEngine?

Step 1. Standard AppEngine application loading and initialization sequence

AppEngine reads app.yaml to understand how to load application (which version of Python it requires and which URLs are handled by which Python scripts)

AppEngine initializes application by creating an instance for it

Then it looks at URL and executes script that shoud process this URL according to app.yaml

This stuff is actual for every AppEngine application.

Example: --- app.yaml from Rietveld project --->

It is the entrypoint to understand any AppEngine app. If you want to know what is called when you request an URL - first thing to do is to look there.

Step 2. Python code to fine-tune AppEngine params and configure request handler

All requests in Rietveld (except static files) are handled by main.py. It does the following:

Imports appengine_config.py that in turn:

Initializes and tunes Appstats tool
Chooses version of Django to use (1.2 currently)
Configures Django to read settings.py with Rietveld specific parameters

Adds logger for all exceptions
Removes Django's DB rollback event handler (because Rietveld doesn't use DB layer of Django)
Creates request handler using Django
Passes handler to AE's run_wsgi_app() util to give Django control to process request

Django didn't fire at this point, and nothing magical happened.

Step 3. Request handling magic

Request handling starts with run_wsgi_app() - this magical function implicitly imports appengine_config.py to read its own settings behind the scenes and then gives control to Django handler created earlier.

Django reads its settings.py mentioned earlier and processes options before executing anything application/request specific:

Configures middlewares - that's important, because they provide such things as user object in request:

django.middleware.common.CommonMiddleware - doesn't seem to be used (docs)
django.middleware.http.ConditionalGetMiddleware - not sure why it is needed
codereview.middleware.AddUserToRequestMiddleware - this one also fetches user-specific parameters from Account record
codereview.middleware.PropagateExceptionMiddleware - logs and rewrites exceptions to be more user-friendly

Sets urls.py to be the ROOT_URLCONF - mapping between URLs and handler functions in views.py
Enables django.core.context_processors.request which adds `request` object to templates
Configures template loaders
Configures file uploads
Configures URL to generate path to static files as `/static/`
Rietveld own constants like incoming email address are also defined here

After all above is done, Django handler starts processing the request:

It looks into urls.py to find what function should process requested URL
urls.py is a redirect to codereview/urls.py with actual mapping, so it reads the latter as well
Finds associated function name and calls this function from views.py

And that's basically the entrypoint that you need to start hacking Rietveld/Django and AppEngine.

Python based VNC client in browser

2011-11-12T14:09:00.001-08:00

What I really like in open source is when people reuse existing stuff to bring up something new. Today Arkaitz Jimenez brought up a way to connect to VNC server with browser using hybrid solution from Python, JavaScript, gevent and WebSockets.

The exciting thing that this project uses patched version of python-vnc-viewer written by Chris Liechti that I've uploaded to Google Code, because it had to be updated to work with modern versions of required libraries. It is nice to see this effort brought up something cool today.

How to make disk in memory in Linux

2011-08-28T23:02:00.000-07:00

Disks in memory are useful for one major reason - speed. It is invaluable for test execution. While the idea is not new, there was no incentive to explore it until I've run upon tmpfs reference in Ubuntu Wiki.

For example, to get 2Gb of space for files in RAM, edit /etc/fstab to add the following line:
tmpfs /var/ramspace tmpfs defaults,size=2048M 0 0 /var/ramspace is now the place to store your files in memory.

Audio output on Windows with pure Python + ctypes

2011-07-03T07:08:00.000-07:00

I want to make a simple walkie-talkie for a local LAN to have fun with my friends. I've created Raw Audio Socket spec to keep it really simple. We have a lot of different operating systems, so it would be nice to have a single implementation that can be run cross-platform. I chose Python and started with getting the sound playing on Windows.

Python is known for its easy integration with C libraries and, indeed, there is a lot of Python bindings calling audio libraries written in C from Python. However, it requires the library to be compiled for your specific platform. I chose to avoid dependencies on any C code and instead call Windows WinMM Multimedia API directly with ctypes module.

Thanks to an excellent tutorial by David Overton, here is the pure Python proof of concept that plays standard CD Audio PCM 44.1kHz 16bit Stereo sample from external file to the default sound device.


"""
Implementation of Raw Audio Socket server spec in pure Python
http://code.google.com/p/rainforce/wiki/RawAudioSocket

"""

import sys

#-- CHAPTER 1: CONTINUOUS SOUND PLAYBACK WITH WINDOWS WINMM LIBRARY --
#
# Based on tutorial "Playing Audio in Windows using waveOut Interface"
# by David Overton

import ctypes
from ctypes import wintypes


# 1. Open Sound Device

# --- define necessary data structures from mmsystem.h
HWAVEOUT = wintypes.HANDLE
WAVE_FORMAT_PCM = 0x1
WAVE_MAPPER = -1
CALLBACK_NULL = 0
MMSYSERR_NOERROR = 0

class WAVEFORMATEX(ctypes.Structure):
  _fields_ = [
    ('wFormatTag',  wintypes.WORD),
      # 0x0001 WAVE_FORMAT_PCM. PCM audio
      # 0xFFFE The format is specified in the WAVEFORMATEXTENSIBLE.SubFormat
      # Other values are in mmreg.h 
    ('nChannels',   wintypes.WORD),
    ('SamplesPerSec',  wintypes.DWORD),
    ('AvgBytesPerSec', wintypes.DWORD),
      # for WAVE_FORMAT_PCM is the product of nSamplesPerSec and nBlockAlign
    ('nBlockAlign', wintypes.WORD),
      # for WAVE_FORMAT_PCM is the product of nChannels and wBitsPerSample
      # divided by 8 (bits per byte)
    ('wBitsPerSample', wintypes.WORD),
      # for WAVE_FORMAT_PCM should be equal to 8 or 16
    ('cbSize',      wintypes.WORD)]
      # extra format information size, should be 0
# --- /define

# Data must be processes in pieces that are multiple of
# nBlockAlign bytes of data at a time. Written and read
# data from a device must always start at the beginning
# of a block. Playback of PCM data can not be started in
# the middle of a sample on a non-block-aligned boundary.

hwaveout = HWAVEOUT()
wavefx = WAVEFORMATEX(
  WAVE_FORMAT_PCM,
  2,     # nChannels
  44100, # SamplesPerSec
  705600,# AvgBytesPerSec = 44100 SamplesPerSec * 16 wBitsPerSample
  4,     # nBlockAlign = 2 nChannels * 16 wBitsPerSample / 8 bits per byte
  16,    # wBitsPerSample
  0
)

# Open default wave device
ret = ctypes.windll.winmm.waveOutOpen(
  ctypes.byref(hwaveout), # buffer to receive a handle identifying
                          # the open waveform-audio output device
  WAVE_MAPPER,            # constant to point to default wave device
  ctypes.byref(wavefx),   # identifier for data format sent for device
  0, # DWORD_PTR dwCallback - callback mechanizm
  0, # DWORD_PTR dwCallbackInstance - user instance data for callback
  CALLBACK_NULL # DWORD fdwOpen - flag for opening the device
)

if ret != MMSYSERR_NOERROR:
  sys.exit('Error opening default waveform audio device (WAVE_MAPPER)')

print "Default Wave Audio output device is opened successfully"


# 2. Write Audio Blocks to Device

# --- define necessary data structures
PVOID = wintypes.HANDLE
WAVERR_BASE = 32
WAVERR_STILLPLAYING = WAVERR_BASE + 1
class WAVEHDR(ctypes.Structure):
  _fields_ = [
    ('lpData', wintypes.LPSTR), # pointer to waveform buffer
    ('dwBufferLength', wintypes.DWORD),  # in bytes
    ('dwBytesRecorded', wintypes.DWORD), # when used in input
    ('dwUser', wintypes.DWORD),          # user data
    ('dwFlags', wintypes.DWORD),
    ('dwLoops', wintypes.DWORD),  # times to loop, for output buffers only
    ('lpNext', PVOID),            # reserved, struct wavehdr_tag *lpNext
    ('reserved', wintypes.DWORD)] # reserved
# The lpData, dwBufferLength, and dwFlags members must be set before calling
# the waveInPrepareHeader or waveOutPrepareHeader function. (For either
# function, the dwFlags member must be set to zero.)
# --- /define

class AudioWriter(object):
  def __init__(self, hwaveout):
    self.hwaveout = hwaveout
    self.wavehdr = WAVEHDR()

  def write(self, data):
    self.wavehdr.dwBufferLength = len(data)
    self.wavehdr.lpData = data
    
    # Prepare block for playback
    if ctypes.windll.winmm.waveOutPrepareHeader(
         self.hwaveout, ctypes.byref(self.wavehdr), ctypes.sizeof(self.wavehdr)
       ) != MMSYSERR_NOERROR:
      sys.exit('Error: waveOutPrepareHeader failed')

    # Write block, returns immediately unless a synchronous driver is
    # used (not often)
    if ctypes.windll.winmm.waveOutWrite(
         self.hwaveout, ctypes.byref(self.wavehdr), ctypes.sizeof(self.wavehdr)
       ) != MMSYSERR_NOERROR:
      sys.exit('Error: waveOutWrite failed')

    # [ ] calculate sleep delay based on sample length
    # iii [ ] Measure CPU usage spike during wait without delay
    import time
    time.sleep(1)

    # Wait until playback is finished
    while True:
      # unpreparing the header fails until the block is played
      ret = ctypes.windll.winmm.waveOutUnprepareHeader(
              self.hwaveout,
              ctypes.byref(self.wavehdr),
              ctypes.sizeof(self.wavehdr)
            )
      if ret == WAVERR_STILLPLAYING:
        import time
        time.sleep(1)
        continue
      if ret != MMSYSERR_NOERROR:
        sys.exit('Error: waveOutUnprepareHeader failed with code 0x%x' % ret)
      break


# [ ] it's no good to read all the PCM data into memory at once
data = open('95672__Corsica_S__frequency_change_approved.raw', 'rb').read()

aw = AudioWriter(hwaveout)
aw.write(data)


# x. Close Sound Device

ctypes.windll.winmm.waveOutClose(hwaveout)
print "Default Wave Audio output device is closed"

#-- /CHAPTER 1 --

Windows also provides DirectSound API, but it looks too complicated for me ATM. The code above is also available from https://bitbucket.org/techtonik/audiosocket and marked with 0.1 tag. You may expect to find further modifications there.

Why Subversion conflict resolution suxx?

2011-05-19T09:25:00.000-07:00

Because conflict resolution choice is confusing (for non-English natives for sure):

$ svn up
U    bitten\slave.py
G    bitten\tests\queue.py
Conflict discovered in 'bitten/queue.py'.
Select: (p) postpone, (df) diff-full, (e) edit,
        (mc) mine-conflict, (tc) theirs-conflict,
        (s) show all options:

Should you select "theirs conflict" or "mine confict" to fix your code? In situation when you feel that incoming change is right you may want to select "mine conflict" as the change that conflicts and should be removed as a result. But the correct answer it to select "theirs conflict" as the change to be incorporated in your code. Subversion developers think that you're are not removing conflicts, but incorporating them into your code. If they could think of user action - it is removing or resolving conflict in high-level terminology, or selecting correct change in low-level description.

Let's see the help:

Conflict discovered in 'bitten/slave.py'.
Select: (p) postpone, (df) diff-full, (e) edit,
        (mc) mine-conflict, (tc) theirs-conflict,
        (s) show all options: s
  (e)  edit             - change merged file in an editor
  (df) diff-full        - show all changes made to merged file
  (r)  resolved         - accept merged version of file
  (dc) display-conflict - show all conflicts (ignoring merged version)
  (mc) mine-conflict    - accept my version for all conflicts (same)
  (tc) theirs-conflict  - accept their version for all conflicts (same)
  (mf) mine-full        - accept my version of entire file (even non-conflicts)
  (tf) theirs-full      - accept their version of entire file (same)
  (p)  postpone         - mark the conflict to be resolved later
  (l)  launch           - launch external tool to resolve conflict
  (s)  show all         - show this list

Frankly, I can't understand how (mc) is different from (mf) if both versions accept all changes. (same) marker doesn't add much sense either. But even if help doesn't describe anything - these options were still added for a reason, and if we meditate a bit on them starting from the bottom comment to (mf) help, it becomes clear that:

1. (mf) throws away any incoming changes for the given file even if some of them were merged successfully
2. that means (mc) probably keeps the changes that were merged successfully
3. that means (df) shows more changes than (dc) output

Recommendations? Introduce `change` and `conflict` terminology earlier, i.e. show info like `3 changes, 2 conflicts`. It will also allow incremental conflict resolution, like if you (e)dited the counter is changed to `4 changes, 1 conflict` etc. Final recommendation if to replace `(ignoring merged version)` with `(ignoring merged changes)` string. There are surely more ways to improve this.

Chrome logo was awesome

2011-04-29T10:03:00.000-07:00

This is a Chrome 11 logo.

This one is the old variant.

Finding unused files in a SCons project with Process Monitor

2011-04-06T21:37:00.000-07:00

New technologies are born and die, but one things remains in your project - their files. Quite often you have no idea about where are they used, and attempt to remove them may lead to serious consequences.

Fortunately, if your project is managed by fine grained build system such as SCons, if your build scripts are not globbing too much, there are chances you can find files that are not participating in the builds.

Here is how to do this on Windows using Process Monitor tool that intercepts all system calls including file access.

While build systems are usually common for C/C++ and Java projects, it is possible to add fine-grained file usage control for any project. For example, SCons itself is written entirely in Python, it could run directly from the source checkout or build distributives from checkout. But instead, it uses build procedure to copy all necessary files from checkout into separate directory and do stuff from there.

Thanks to that it is possible to see which files are no more actual. While it is possible to compare checkout source tree and copied directory trees, I'll go through the hells of monitoring system file access in a source tree during the build process using Process Monitor (FileMon in the past). Linux should have similar tools too - let me know how are they called.

The process is the following:

Start Process Monitor
Stop incoming event flood by (un)clicking Capture (Ctrl-E) button
Open Filter (Ctrl-L) dialog to add some filters

SCons build is started by bootstrap.py script from a root of SCons source checkout. The script is executed by python executable, so I add python.exe process name to the filter. I know that bootstrap.py copies files from src/ subdirectory, so it is the directory I need to monitor, so I add this dir to filters too.

Go Tools -> File Summary...

There is a list of paths catched by Process Monitor when listening to system calls. They are already filtered, but additional filters can be applied using bottom left button to make information even more useful.

Export to CSV using Save...

Exported CSV is not very useful without some postprocessing. I used the following a script to compare the list of paths in CSV to actual src/ directory contents. This gives me names of files that were not touched during build at all.

SRCDIR = "C:\\p\\python\\scons\\src"
CSVLIST = 'accessed_bootstrap_files.CSV'

import csv
import os

reader = csv.reader(open(CSVLIST))
header = reader.next()
pathidx = header.index("Path")
pathset = set([row[pathidx] for row in reader])

#for row in pathset:
#  print row

fileset = set()
for root, dirs, files in os.walk(SRCDIR):
  fileset.update( [os.path.join(root, f) for f in files] )
  if '.svn' in dirs:
    dirs.remove('.svn')  # don't visit .svn directories

if len(pathset & fileset) == 0:
  print 'Error: File sets do not intersect at all'

print "Files not found in source directory tree:"
for f in (pathset - fileset):
  if not os.path.isdir(f):
    print f

print
print "Untouched files in source directory tree:"
for f in sorted(fileset - pathset):
  if not os.path.isdir(f):
    print f

I've found a few interesting things about SCons. Core tests are mixed with source files in repository checkout. They are not copied during bootstrap build. There are also few setup.py files, post-install script and announcement that don't participate in the build.

Here is the output of the above script:



Files not found in source directory tree:

<Total>



Untouched files in source directory tree:

C:\p\python\scons\src\.aeignore

C:\p\python\scons\src\Announce.txt

C:\p\python\scons\src\engine\.aeignore

C:\p\python\scons\src\engine\SCons\.aeignore

C:\p\python\scons\src\engine\SCons\ActionTests.py

C:\p\python\scons\src\engine\SCons\BuilderTests.py

C:\p\python\scons\src\engine\SCons\CacheDirTests.py

C:\p\python\scons\src\engine\SCons\DefaultsTests.py

C:\p\python\scons\src\engine\SCons\EnvironmentTests.py

C:\p\python\scons\src\engine\SCons\ErrorsTests.py

C:\p\python\scons\src\engine\SCons\ExecutorTests.py

C:\p\python\scons\src\engine\SCons\JobTests.py

C:\p\python\scons\src\engine\SCons\MemoizeTests.py

C:\p\python\scons\src\engine\SCons\Node\.aeignore

C:\p\python\scons\src\engine\SCons\Node\AliasTests.py

C:\p\python\scons\src\engine\SCons\Node\FSTests.py

C:\p\python\scons\src\engine\SCons\Node\NodeTests.py

C:\p\python\scons\src\engine\SCons\Node\PythonTests.py

C:\p\python\scons\src\engine\SCons\Optik\.aeignore

C:\p\python\scons\src\engine\SCons\PathListTests.py

C:\p\python\scons\src\engine\SCons\Platform\.aeignore

C:\p\python\scons\src\engine\SCons\Platform\PlatformTests.py

C:\p\python\scons\src\engine\SCons\SConfTests.py

C:\p\python\scons\src\engine\SCons\SConsignTests.py

C:\p\python\scons\src\engine\SCons\Scanner\.aeignore

C:\p\python\scons\src\engine\SCons\Scanner\CTests.py

C:\p\python\scons\src\engine\SCons\Scanner\DirTests.py

C:\p\python\scons\src\engine\SCons\Scanner\FortranTests.py

C:\p\python\scons\src\engine\SCons\Scanner\IDLTests.py

C:\p\python\scons\src\engine\SCons\Scanner\LaTeXTests.py

C:\p\python\scons\src\engine\SCons\Scanner\ProgTests.py

C:\p\python\scons\src\engine\SCons\Scanner\RCTests.py

C:\p\python\scons\src\engine\SCons\Scanner\ScannerTests.py

C:\p\python\scons\src\engine\SCons\Script\.aeignore

C:\p\python\scons\src\engine\SCons\Script\MainTests.py

C:\p\python\scons\src\engine\SCons\Script\SConscriptTests.py

C:\p\python\scons\src\engine\SCons\SubstTests.py

C:\p\python\scons\src\engine\SCons\TaskmasterTests.py

C:\p\python\scons\src\engine\SCons\Tool\.aeignore

C:\p\python\scons\src\engine\SCons\Tool\JavaCommonTests.py

C:\p\python\scons\src\engine\SCons\Tool\PharLapCommonTests.py

C:\p\python\scons\src\engine\SCons\Tool\ToolTests.py

C:\p\python\scons\src\engine\SCons\Tool\f03.xml

C:\p\python\scons\src\engine\SCons\Tool\msvsTests.py

C:\p\python\scons\src\engine\SCons\UtilTests.py

C:\p\python\scons\src\engine\SCons\Variables\BoolVariableTests.py

C:\p\python\scons\src\engine\SCons\Variables\EnumVariableTests.py

C:\p\python\scons\src\engine\SCons\Variables\ListVariableTests.py

C:\p\python\scons\src\engine\SCons\Variables\PackageVariableTests.py

C:\p\python\scons\src\engine\SCons\Variables\PathVariableTests.py

C:\p\python\scons\src\engine\SCons\Variables\VariablesTests.py

C:\p\python\scons\src\engine\SCons\WarningsTests.py

C:\p\python\scons\src\engine\SCons\cppTests.py

C:\p\python\scons\src\engine\setup.cfg

C:\p\python\scons\src\engine\setup.py

C:\p\python\scons\src\script\.aeignore

C:\p\python\scons\src\script\scons-post-install.py

C:\p\python\scons\src\script\setup.cfg

C:\p\python\scons\src\script\setup.py

C:\p\python\scons\src\test_aegistests.py

C:\p\python\scons\src\test_files.py

C:\p\python\scons\src\test_interrupts.py

C:\p\python\scons\src\test_pychecker.py

C:\p\python\scons\src\test_setup.py

C:\p\python\scons\src\test_strings.py

Hope this helps clean up your projects too.

P.S. I wish there was a Python script replacement for Process Monitor, or at least that it could be controlled from command line.

Asynchronous input from Windows console

2011-03-29T06:03:00.000-07:00

too long didn't read - https://bitbucket.org/techtonik/async-console-input - public domain/MIT example for Windows implemented in pure Python (ctypes)

What is asynchronous input?

Given: The program need to terminate immediately when a subprocess exits or user hits 'q'. It should not eat 100% CPU time.

Blocking synchronous input:

import sys, subprocess
from msvcrt import getch, kbhit

p = subprocess.Popen([r"notepad"], shell=True)

while True:
  char = getch()
  if char == 'q':
    sys.exit('terminated by user')
  if p.poll() != None:
    sys.exit('terminated by child exit')

This one doesn't work (doesn't exits immediately when child process terminates), because getch() blocks execution key is pressed, and even if child process exits, it won't be detected until some key is pressed.

Non-blocking synchronous input (polling):

import sys, subprocess
from msvcrt import getch, kbhit

p = subprocess.Popen([r"notepad"], shell=True)

while True:
  while kbhit():
    if getch() == 'q':
      sys.exit('terminated by user')
  if p.poll() != None:
    sys.exit('terminated by child exit')

This works ok, but constant polling uses 100% of CPU resources. It is possible to insert time.sleep() instructions to reduce the carbon footprint, but these crutches will greatly slow down the console when you add z-type to console to spend your time until the background process is finished.

Asynchronous console input on Windows with Python

Asynchronous input allows your program to receive notification from operating system when an input is available. This means that you define events your program needs to react (not necessary console events), send this list to operating system and put your program into wait mode. When system sees this event, your wait function returns and it's possible to inspect/filter event to a greater detail.

Windows provides WaitForMultipleObjects wait function, and in Linux I believe it is select call. If you've tried to understand how Twisted works, but couldn't - will it make more clear if I say that twisted reactor is a wait function? What you do when you code with twisted is just configuring events to react, making chains of them so that one event reacts to another event. This allow to build very interesting, complex and de-coupled asynchronous applications in a couple of days.

O.k. Getting back from Twisted to the asynchronous console input on Windows. Below is the full source code that uses WaitForMultipleObjects. I am afraid that's minimal complete example possible to build with ctypes using Windows API. Tested with Python 2.5

Non-blocking asynchronous input from console:
https://bitbucket.org/techtonik/async-console-input

"""
Example of non-blocking asynchronous console input using
Windows API calls in Python.  This can become handy for
async console tools such as IRC client.

Public domain or MIT license
by anatoly techtonik 


Notes:
  1. WaitForMultipleObjects is used to listen for the
     signals from process and stdin handles
  2. When handle is signalled it remains in this state
     until reset
  3. msvcrt.* keyboard functions don't clear signalled
     state from stdin handle, that's why console API
     functions are used to clear the input buffer
     instead of kbhit()/getch() loop
"""


import ctypes
import ctypes.wintypes
import subprocess

# open notepad in separate process and monitor its execution
# at the same time asynchronously processing events from
# standard input without wasting 100% CPU on looping

# OpenProcess desired access flag
#   "the right to use the object for synchronization. This
#    enables a thread to wait until the object is in the
#    signaled state"
SYNCHRONIZE=0x00100000L
# Constant to get stdin handle with GetStdHandle() call
STD_INPUT_HANDLE = -10
# Constant for infinite timeout in WaitForMultipleObjects()
INFINITE = -1

# --- processing input structures -------------------------
# INPUT_RECORD structure
#  events:
EVENTIDS = dict(
  FOCUS_EVENT = 0x0010,
  KEY_EVENT = 0x0001,      # only key event is handled
  MENU_EVENT = 0x0008,
  MOUSE_EVENT = 0x0002,
  WINDOW_BUFFER_SIZE_EVENT = 0x0004)
EVENTS = dict(zip(EVENTIDS.values(), EVENTIDS.keys()))
#  records:
class _uChar(ctypes.Union):
  _fields_ = [('UnicodeChar', ctypes.wintypes.WCHAR),
              ('AsciiChar', ctypes.wintypes.c_char)]
class KEY_EVENT_RECORD(ctypes.Structure):
  _fields_ = [
    ('keyDown', ctypes.wintypes.BOOL),
    ('repeatCount', ctypes.wintypes.WORD),
    ('virtualKeyCode', ctypes.wintypes.WORD),
    ('virtualScanCode', ctypes.wintypes.WORD),
    ('char', _uChar),
    ('controlKeyState', ctypes.wintypes.DWORD)]
class _Event(ctypes.Union):
  _fields_ = [('keyEvent', KEY_EVENT_RECORD)]
  #  MOUSE_EVENT_RECORD        MouseEvent;
  #  WINDOW_BUFFER_SIZE_RECORD WindowBufferSizeEvent;
  #  MENU_EVENT_RECORD         MenuEvent;
  #  FOCUS_EVENT_RECORD        FocusEvent;
class INPUT_RECORD(ctypes.Structure):
  _fields_ = [
    ('eventType', ctypes.wintypes.WORD),
    ('event', _Event)]
# --- /processing input structures ------------------------



np = subprocess.Popen([r"notepad"],
       stdin=subprocess.PIPE,
       stdout=subprocess.PIPE,
       stderr=subprocess.PIPE,
       shell=True)
# OpenProcess returns handle that can be used in wait functions
# params: desiredAccess, inheritHandle, processId
nph = ctypes.windll.kernel32.OpenProcess(SYNCHRONIZE, False, np.pid)
print("Started Notepad with pid=%s, handle=%s" % (np.pid, nph))

ch = ctypes.windll.kernel32.GetStdHandle(STD_INPUT_HANDLE)

handles = [ch, nph]

ctypes.windll.kernel32.FlushConsoleInputBuffer(ch)
eventnum = ctypes.wintypes.DWORD()
eventread = ctypes.wintypes.DWORD()
inbuf = (INPUT_RECORD * 1)()

print "[q]uit, [s]top console processing, launch bro[w]ser"
stopflag = False
while not stopflag and nph in handles:
    print "Waiting for handles %s.." % handles

    HandleArrayType = ctypes.wintypes.HANDLE * len(handles)
    handle_array = HandleArrayType(*handles)
    # params: count, handles, waitAll, milliseconds
    ret = ctypes.windll.kernel32.WaitForMultipleObjects(
            len(handle_array), handle_array, False, INFINITE)

    if handles[ret] == ch:
        """
        # msvcrt won't work, because it doesn't reset
        # signalled state of stdin handle
        import msvcrt
        while msvcrt.kbhit():
            print "key!"
            print msvcrt.getch()
        continue
        """
        # --- processing input ---------------------------
        ctypes.windll.kernel32.GetNumberOfConsoleInputEvents(
          ch, ctypes.byref(eventnum))
        for i in range(eventnum.value):
          # params: handler, buffer, length, eventsnum
          ctypes.windll.kernel32.ReadConsoleInputW(
            ch, ctypes.byref(inbuf), 2, ctypes.byref(eventread))
          if EVENTS[inbuf[0].eventType] != 'KEY_EVENT':
            print EVENTS[inbuf[0].eventType]
            pass
          else:
            keyEvent = inbuf[0].event.keyEvent
            if not keyEvent.keyDown:
              continue
            char = keyEvent.char.UnicodeChar.lower()
            #print char, keyEvent
            if char == 'q':
              print('[q] key pressed. Exiting..')
              stopflag = True
            elif char == 's':
              handles.remove(ch)
            elif char == 'w':
              import webbrowser
              webbrowser.open('http://techtonik.rainforce.org')
            #print char
        # --- /processing input --------------------------

    elif handles[ret] == nph:
        print("Notepad is closed. Exiting..")
        handles.remove(nph)
    else:
        print("Warning: Unknown return value '%s'" % ret)

ctypes.windll.kernel32.FlushConsoleInputBuffer(ch)
ctypes.windll.kernel32.CloseHandle(nph)
print "Done."

Where can it be useful?

Writing first Windows console IRC client in Python and make it cross-platform? Network log viewers with keyboard shortcuts? Real-time roguelikes? Matrix sniffer screensaver? I don't know - its your time. Hope I saved you some though.

Enjoy! If you want to enhance this stuff - feel free to join ever empty https://groups.google.com/forum/#!forum/rainforce for public discussion.

User Pattern: Button Status

2011-03-22T07:24:00.000-07:00

Introduction into User Patterns

Many of you know about Design Patterns of software development, invented to make complicated solutions look less complicated. The problem with Design Patterns is that they describe solutions to coding problems - not for user problem. People abusing Design Patterns tend to align the functionality of application to patterns, sacrificing features users wanted. The application that looks well-designed in the eyes of developers, in fact makes even simple features hard to implement or just takes too much time. As time passes, users become dissatisfied and eventually lose all interest in the product.

People tried to approach the problem from the other end with User Interface Design guidelines. This often made developers unhappy, because their Design Patterns don't explain how to work with web and Ajax application where "business logic" becomes shared between client and server. Developers wanted to see how different frameworks approach this stuff and to make the development process more intuitive. For this reason there should be a way to see how some typical features/functionality are implemented in different frameworks and this should be more detailed than a wiki or a blog in 15 minutes.

So, let me introduce User Patterns. User Pattern is a very abstract story, missing details about design decisions, but detailed enough to provide implementation and see how _simple_, _extensible_ and _maintainable_ this implementation is.

Button Status User Pattern

User story: I want a page with a button and a status field displayed. When I click the button, the status should be updated.

Description: There are five levels in the quest for the implementation of this pattern.
Level 1. The status is blank - it is fetched/calculated only when the button is pressed.
Level 2. The status is initially available - and it is refreshed/calculated when the button is pressed.
Level 3. When refresh/recalculation fails, the error value is displayed in status field.
Level 4. When refresh/recalculation fails, the error value is displayed in status field along with user friendly message in separate field
Level 5. The status value is stored in application database - when the button is pressed, it is recalculated, saved and only when displayed to the user
Level 6. When recalculation/save or fetch operation fails, user should be presented with friendly message explaining the error and further action (i.e. report problem link)

I'm looking forward to create samples for AppEngine webapp, classic Django and Django-nonrel. Feel free to share your snippets.

[PySide] Python for Qt version 1.0.0~beta4 "I have altered the deal" released

2011-01-21T13:57:00.000-08:00

I can't see this post on planet.python.org and can't resist to share the good news about new PySide release.

The PySide team is happy to announce the fourth beta release of PySide:
Python for Qt. New versions of some of the PySide toolchain components
apiextractor, generatorrunner, shiboken, libpyside, pyside-tools have been
released as well.

Like the others, this is a source code release only; we hope our community
packagers will be providing provide binary packages shortly. To acquire the
source code packages, refer to our download wiki page [1] or pull the relevant
tagged versions from our git repositories [2].

Major changes since 1.0.0~beta3
===============================

This is a bug fix release. Since beta3, a total of 47 high-priority
bugs have been fixed. See the list of fixed bugs at the end of this message.

Path towards 1.0 release
========================

There are still some outstanding bugs in our Bugzilla [3]. To have these
fixed, we plan to do other beta in two weeks. The beta cycle will continue
until we have all P2 bugs fixed.


About PySide
============

PySide is the Nokia-sponsored Python Qt bindings project, providing
access to not only the complete Qt 4.7 framework but also Qt Mobility,
as well as to generator tools for rapidly generating bindings for any
Qt-based libraries.

The PySide project is developed in the open, with all facilities you'd
expect from any modern OSS project such as all code in a git repository
[2], an open Bugzilla [5] for reporting bugs, and an open design
process [4]. We welcome any contribution without requiring a transfer of
copyright.

List of bugs fixed
==================

624     button click emit doesn't work
484     Error compiling QtContacts 1.1 (problems with const QList)
498     powerStateChanged-SIGNAL not emitted!
509     Can't use Shiboken when both Debug and Released are installed.
528     Connecting to SIGNAL positionUpdated fails
552     Segmentation fault when using QUiLoader and QTabWidget
553     A warning against using QUILoader is needed in the documentation
560     Lack of QtCore.Signal documentation
582     Python slots don't get called when they have a custom decorator
589     Crash related to QGraphicsProxyWidget and QVariant
592     shiboken.dll produces a segmentation fault when reloading a PySide module
608     Photoviewer example missing license boilerplates and shebang lines
609     Python site-packages path cannot be customized
610     QWidgetItemV2 not exposed to Python
626     Problem building PySide on OS X (qabstractfileengine_wrapper.cpp:
No such file or directory)
406     Unable to send instant messages using QMessageService
458     Doesn't build with QtMobility 1.1.0~beta2
487     Support QContactDetailFieldDefinition.setAllowableTypes
497     Miising __lt__ operators in QtMobility::QGeoMapObject
499     QFeedbackHapticsInterface and QFeedbackFileInterface are broken
511     QPainter doesn't respect Qt.NoPen
522     example/threads/mandelbrot.py crashes on exit
523     QWidget.winId() returns PyCObject (expected unsigned long)
530     Importing division from future breaks QPoint division
531     sessionProperty "ConnectInBackground" does not work
539     MCC and MNC interchanged
541     QTableWidget.itemAt(row, col) always returns item at 0, 0.
550     Can't call PySide slot from QtScript when the args are a list of anything.
556     QGraphicsScene.addItem performs very poorly when the scene has >10000 items
562     pyside-uic does not generate some layers properties
568     List insertion time grows with list size
574     In docs of QUuid there's documentation for a function called
"operator QString"
575     Strange behaviour of QTextEdit.ExtraSelection().cursor
584     python pickle module can't treat QByteArray object of PySide
591     QtCore.QRect has no attribute "getRect()" in Windows binary
611     enum values lack a tp_print implementation
614     FAil to register 2 objects in the same address
619     never automatically delete a QWidget that has no parent widget and
is visible
620     QAbstractItemModel.createIndex(int,int,PyObject*) does not
increment refcount
621     QGLWidget.bindTexture(QString) does not bind the texture correctly
622     PPA pyside is broken on Ubuntu 10.10
623     QGLWidget.bindTexture(QPixmap, GLenum, GLenum) is missing
625     QFileDialog return a tuple instead of a unicode
628     pyside-uic can't effect "headerVisible" attribute for QTreeView
and QTreeWidget
232     [FTBFS] Fails to build on hurd-i386 (Test "lock" hangs for more
than 191 minutes)
255     Test qtscripttools_debugger segfaults on ia64
298     Contact subtype not correctly set



References
==========

[1] http://developer.qt.nokia.com/wiki/PySideDownloads
[2] http://qt.gitorious.org/pyside
[3] http://bugs.openbossa.org/
[4] http://www.pyside.org/docs/pseps/psep-0001.html

Thanks
PySide team.

Python GUIniverse

2011-01-18T14:16:00.000-08:00

I am on the PySide of the Force.

Injecting SVN bindings into Mercurial on Windows

2011-01-08T05:52:00.000-08:00

If you need to add Subversion bindings for your Mercurial, which was installed using ordinary installer (and not TortoiseHg), then the following script can help to put all files in place.

https://bitbucket.org/techtonik/py2exe-relibzip/src/f15f8d032b5b/reinject.py

Unpack svn bindings, add all *.py files from svn/ and libsvn/ directories into the root of library.zip archive *with* these directories. Then call script with path to unpacked libsvn/. The script will generate some *.dll and *.py files in current directory. Put *.py into library.zip:/libsvn/ and *.dll into Mercurial installation directory. You may need to also copy libsvn_swig_py-1.dll into Mercurial dir.

The script can be tuned to insert C extensions to any py2exe distribution compiled with bundle_files=3

Got mail? Part II

2010-12-10T04:10:00.000-08:00

See Part I

Validating SSL server certificate with Python 2.x

2010-11-10T07:50:00.000-08:00

SSL stands for Secure Sockets Layer and is designed to create secure connection between client and server. Secure means that connection is encrypted and therefore protected from eavesdropping. It also allows to validate site identity when connecting with HTTPS protocol.

However, there is a bug in ssl module from standard library of Python 2.x, that allows successful MITM attack using valid certificate from other site. Basically, module checks when connecting that server certificate is valid and correctly signed by root certificate, but it does not check that certificate actually belongs to the site, i.e. that site name matches the name specified in certificate.

It is still possible to validate server identity in Python 2.6 manually. Let's start with illustration of the vulnerability. The following snippet should fail - it replaces HOST "www.google.com" to connect to with its IP address. If you try to use this IP in Chrome like https://74.125.232.50 - it will show an error, but ssl library will not throw exception.

import socket

import ssl



HOST = "www.google.com"

PORT = 443



# replace HOST name with IP, this should fail connection attempt,

# but it doesn't in Python 2.x

HOST = socket.getaddrinfo(HOST, PORT)[0][4][0]

print(HOST)



# create socket and connect to server

# server address is specified later in connect() method

sock = socket.socket()

sock.connect((HOST, PORT))



# wrap socket to add SSL support

sock = ssl.wrap_socket(sock,

  # flag that certificate from the other side of connection is

  # required and should be validated when wrapping 

  cert_reqs=ssl.CERT_REQUIRED,

  # file with root certificates

  ca_certs="cacert.pem"

)

You will need cacert.pem file with root certificates. Just grab the latest version from http://curl.haxx.se/ca/cacert.pem This code above won't give you any error. Replace HOST value with https://www.debian-administration.org/ to check that certificate validation actually works. This site's certificate is not signed by any root certificates from "cacerts.txt", so you get an error.

To validate that a certificate matches requested site, you need to check commonName field in the subject of the certificate. This information can be accessed with getpeercert() method of wrapped socket.

import socket

import ssl



HOST = "www.google.com"

PORT = 443



# replace HOST name with IP, this should fail connection attempt

HOST = socket.getaddrinfo(HOST, PORT)[0][4][0]

print(HOST)



# create socket and connect to server

# server address is specified later in connect() method

sock = socket.socket()

sock.connect((HOST, PORT))



# wrap socket to add SSL support

sock = ssl.wrap_socket(sock,

# flag that certificate from the other side of connection is

# required and should be validated when wrapping 

cert_reqs=ssl.CERT_REQUIRED,

# file with root certificates

ca_certs="cacerts.txt"

)



# manual check of hostname

cert = sock.getpeercert()

for field in cert['subject']:

if field[0][0] == 'commonName':

certhost = field[0][1]

if certhost != HOST:

raise ssl.SSLError(

    "Host name '%s' doesn't match certificate host '%s'"

    % (HOST, certhost))