I lost an hour of my life today trying to figure out why Sphinx would not document a class inside a module specified by :automodule:.

I learnt two things today.

Firstly, Sphinx will not document a class automatically as part of :automodule: if the class itself doesn’t have a docstring. You can force the class to be displayed by having an additional :autoclass:directive though.

If the class does have a doc string, it will be displayed including the documentation of its regular (ones without a leading _ in the name) methods. It will still not show the constructor though.

To display the constructor, I first thought I would add the :special-members: option to the :automodule: directive. But that didn’t work very well, as it showed various unnecessary private functions as well, e.g. _weakref.

I finally discovered that the best way to get the constructor documentation displayed was using the autoclass_content config in the sphinx config file.

autoclass_content = 'both'

As per the documentation,

This value selects what content will be inserted into the main body of an autoclass directive. The possible values are:

• “class”: Only the class’ docstring is inserted. This is the default. You can still document __init__ as a separate method using automethod or the members option to autoclass.
• “both”: Both the class’ and the __init__ method’s docstring are concatenated and inserted.
• “init”: Only the __init__ method’s docstring is inserted.

This might be a very esoteric topic for most people, but since I could
not find information about this anywhere, I decided to document this
in a post.

Here is the problem. I use Jira at work, and today, I needed
to close a bunch of tickets based on a search result. Now, searching
or doing batch operations is simple enough from the browser, but a
small detail made the exercise impossible via the web UI.

Our Jira project requires a field to be filled while closing the
ticket – the time spent in the ticket. This breaks Jira in all sorts
of ways – the batch operation doesn’t work, some of the email-to-jira
interface at work breaks as well.

So I looked at doing this via the Jira Python library. But
it didn’t work as expected.

>>> from jira import JIRA
>>>
>>> jira = JIRA("https://JIRA_URL", basic_auth=('USER_NAME', 'Password'))
>>>
>>> issue = jira.issue("ISSUE-123")
>>>
>>> [(t['id'], t['name']) for t in jira.transitions(issue)]  # What are the workflows available?
[(u'4', u'Start Progress'), (u'5', u'Resolve Issue'), (u'2', u'Close Issue'), (u'711', u'Planning'), (u'751', u'Blocked'), (u'801', u'To Monitor')
>>>
>>> jira.transition_issue(issue, '2')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
...
jira.exceptions.JIRAError: JiraError HTTP 400
    text: Time Spent is required
    url: ...
    response headers = {...}
    response text = {"errorMessages":["Time Spent is required"],"errors":{}}

The documentation on transitions mentioned that
we could add fields in the call to jira.transitions(). That didn’t
work as well.

>>> jira.transition_issue(issue, '2', timespent="1h")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
...
jira.exceptions.JIRAError: JiraError HTTP 400
    text: Field 'timespent' cannot be set. It is not on the appropriate screen, or unknown.
    url: ...
    response headers = ...
    response text = {"errorMessages":[],"errors":{"timespent":"Field 'timespent' cannot be set. It is not on the appropriate screen, or unknown."}}

So I scoured the Internet for a long time, till I found this post
about how to do it via the REST API – the only official
interface to Jira. Here is what needs to be sent as the body to the
POST request.

{
    "transition": {
        "id": "2"
    },
    "update": {
        "worklog": [
            {
                "add": {
                    "timeSpent": "2m"
                }
            }
        ]
    }
}

I felt that to be odd, till I looked at both the api documentation
for transition and the doc for transition using the
python library and I found out why I have not been
successful till now.

The REST api supports two ways to update the issue while doing a
transition – you can set certain fields using the fields option, or
you can use the update option to do more complex changes.

The comment in the Python code revealed that the update method has
not yet been implemented.

    def transition_issue(self, issue, transition, fields=None, comment=None, **fieldargs):

        # TODO: Support update verbs (same as issue.update())

That put me in a bind. I had only one way to hack around this problem
now – using the REST api for the specific operation I wanted, and the
Python library for the rest of the work – ugly, but works for now,
till the Python library is complete.

So here was the final solution that did what I wanted.

from jira import JIRA
import requests

jira = JIRA("https://JIRA_URL", basic_auth=('USER_NAME', 'PASSWORD'))

d = {}
d["transition"]={"id": "2"}
d["update"]={"worklog": [{"add": {"timeSpent": "1h"}}]}

s = requests.Session()
s.auth = ("USER", "PASSWORD")
s.headers.update({"Content-Type": "application/json"})

j="https://JIRA_URL"

issue_list = jira.search_issues("assignee = currentUser() AND resolution = Unresolved  and status != Closed and updatedDate < '2015-10-01' and project='PROJECT' ORDER BY updatedDate DESC")

for i in issue_list:
    print i
    s.post(j+"/rest/api/2/issue/"+i.key+"/transitions", data=json.dumps(d))

“In view of several representations received regarding interpretation and scope of section 3(k) of the Patents Act 1970 (as amended), the Guidelines for Examination of Computer Related Inventions… are kept in abeyance till discussions with stakeholders are completed and contentious issues are resolved,” the Controller General of Patents said in a notification issued last week.

Again, this is a temporary measure and given the intensive lobbying that happens behind doors, it could still be revised. But thanks to Ispirt, The Software Freedom Law Center, this is being considered seriously by the office.

As the article points out, software patents are ravaging the US software industry. For the big guys, they are used akin to the Mutually Assured Destruction mentality of the cold war, with every company keeping a vast portfolio of patents to fight back with if they are attacked for the same. But there is a big impact on smaller companies too. Bloomberg reports that:

Big companies are not the only ones being clobbered. According to a 2012 study by Boston University Law School professors Michael J. Meurer and James Bessen, some 90 percent of all patent-troll lawsuits are aimed at small and midsize companies. And these companies, when faced with unfathomable potential legal costs, often pay off the trolls just to make them go away.
 
Overall, Meurer and Bessen found, this abusive system is draining billions of dollars annually from the economy, transferring an estimated $29 billion in 2011 alone from the bank accounts of companies that produce things to patent trolls. Of this, small and midsize businesses, the types that are often at the cutting edge of innovation, paid about 37 percent of the total–money that could have been put to much better use.

The Internet Engineering Task Force(IETF) has finally created a standard for when a page has been taken down due to legal reasons. The new status code, 451, indicates that a host has received a legal demand to deny access to a resource.

Via TheNextWeb

The avro python library uses schemas, and can store data in a compact binary format using both deflate and snappy compression. I wanted to test out how compact the serialization format is as compared to say, CSV.

I set up a Python virtual environment using the nifty virtualenv wrapper and installed the Python Avro library and the snappy library. On my Ubuntu system, the python snappy library is available from the package repository, but I used the Pypi one anyway. I had to install the c development package (libsnappy-dev) as a prerequisite though.

$ mkvirtualenv avro

$ workon avro

(avro) $ pip install avro

# Remember that it is 'python-snappy', and not just 'snappy', which is a
# completely different library
(avro) $ pip install python-snappy

And I was ready for my code. First I decided on a simple data set: a weather data set which was part of the Pandas cookbook.

Based on this, I created this simple schema.

{
"namespace": "net.sandipb.avro.example.weather",
"type": "record",
"name": "Reading",
"doc": "Weather reading at a point in time",
"fields": [
{"name": "time", "type": "int", "doc": "Seconds since epoch"},
{"name": "temp", "type": "float", "doc": "Temperature in Celsius"},
{"name": "dew_point_temp", "type": "float", "doc": "Dew point temperature in Celsius"},
{"name": "humidity", "type": "int", "doc": "Relative humidity %"},
{"name": "wind_speed", "type": "float", "doc": "Wind speed in km/h"},
{"name": "visibility", "type": "float", "doc": "Visibility in km"},
{"name": "pressure", "type": "float", "doc": "Atmospheric pressure in kPa"},
{"name": "weather", "type": "string", "doc": "Weather summary"}
]
}

Now to actually process the file, I created this Python script to read the CSV file and to write to three different avro files, each with a different compression format – null (no compression), deflate and snappy.

Here are some relevant parts of the code:

# Load the avro schema to validate and serialize data
schema = avro.schema.parse(open("weather.avsc").read())

# Open using a avro container class, providing the schema to use and the
# compression to use
writer_deflate = DataFileWriter(open("weather_data_deflate.avro", "wb"), DatumWriter(), schema, codec="deflate")

# Write a dict with the data
writer_deflate.append(row)

# close the container
writer_deflate.close()

The result of this was … unexpected. The deflate codec showed more compression than snappy.

$ ls -lSh *.avro *.csv
-rw-rw-r-- 1 sandipb sandipb 492K May 20 01:34 weather_2012.csv
-rw-rw-r-- 1 sandipb sandipb 317K May 20 02:29 weather_data_null.avro
-rw-rw-r-- 1 sandipb sandipb 178K May 20 02:29 weather_data_snappy.avro
-rw-rw-r-- 1 sandipb sandipb 121K May 20 02:29 weather_data_deflate.avro

It is possible that a larger dataset could show better compression for snappy. This is the first time I have used this codec, I need to read up a bit more about it. I will try this experiment with a larger dataset and update this post in the future.

In Python, dictionaries with default values is not part of the core language, and need to be imported from a standard library. For some reason I still don’t know, you cannot set a simple default value, you need supply a function object which is going to create a value for you.

So if I want to create a default dictionary in Python, you need to do something like this:

>>> from collections import defaultdict
>>> d = defaultdict(lambda: 5)

Now when you access any non-existent key in this hash, the hash gets magically populated with that key and the default value. See below.

>>> if d["one"]: print "not empty"
...
not empty
>>>
>>> print dict(d)
{'one': 5}
>>>

Ruby hashes support default values in the core language. So you can do something like:

>> d = Hash.new(5)
=> {}

>> puts d["a"]
5
=> nil

>> d
=>> {}

Wait! See the difference? Evaluating a non-existing hash position is Ruby returns the default value, but unlike Python, it doesn’t set the value!

After it drilled down to this quirk, I looked around and found some interesting articles.

This article from 2008 by David Black was particularly interesting. He points out in this article how this Hash behaviour breaks a very popular Ruby idiom.

Normally, the idiomatic Ruby way to initialize or return a variable goes like this:

>> d = Hash.new
=> {}

>> d["name"] ||= "Skye"
=> "Skye"

>> d
=> {"name"=>"Skye"}

However, if you use a Hash with a default value, it breaks this idiom because evaluating a non-existing key returns the default value. However, you would intuitively expect the ||= operator to at least set the missing key to the default value! But that doesn’t happen due to the peculiar treatment to that operator by Ruby. Ruby evaluates A ||= B NOT to A = A || B, but A || A=B. This never lets the value to be assigned to keys in default hashes.

>> d = Hash.new("May")
=> {}

>> d["name"] ||= "Skye"
=> "May"

>> d
=> {}

Some gotcha lurking there in an otherwise beautiful language.

Another nice ref:
http://www.rubyinside.com/what-rubys-double-pipe-or-equals-really-does-5488.html

Here is Pagerduty’s summary of the bug.

After more than a month of tireless research and testing, we have finally got to the bottom of our ZooKeeper mystery.

Corruption during AES encryption in Xen v4.1 or v3.4 paravirtual guests running a Linux 3.0+ kernel, combined with the lack of TCP checksum validation in IPSec Transport mode, which leads to the admission of corrupted TCP data on a ZooKeeper node, resulting in an unhandled exception from which ZooKeeper is unable to recover.

Jeez. Talk about a needle in a haystack… Even after all this, we are still unsure where precisely the bug lies. Despite that fact, we’re still pretty satisfied with the outcome of the investigation. Now all we need to do is work around it.

A troubling development:

Apple has patented a piece of technology which would allow government and police to block transmission of information, including video and photographs, from any public gathering or venue they deem “sensitive”, and “protected from externalities.”

In other words, these powers will have control over what can and cannot be documented on wireless devices during any public event.

And while the company says the affected sites are to be mostly cinemas, theaters, concert grounds and similar locations, Apple Inc. also says “covert police or government operations may require complete ‘blackout’ conditions.”

And those who think that this is not coming for Android in the future are deluded. If Apple managed to get this technology into the field, it is only a matter of time that Android handset manufacturers are forced to incorporate this as well. If the technology exists, in today’s post 9/11 world, it is difficult to resist government pressure on such matters.

Of course, it would be interesting to see the security features for this tech, as this is very likely to be abused – by repressive governments (read, every one) as well as criminal enterprise (recording-free drug zones everybody?)

Who said the field of security cannot have humour! An Android app to control the commode in Japan (you know the land of fully programmable toilets, I kid you not) has announced a vulnerability because the bluetooth pairing code is hardcoded.

The alert goes:

An attacker could simply download the “My Satis” application and use it to cause the toilet to repeatedly flush, raising the water usage and therefore utility cost to its owner. Attackers could cause the unit to unexpectedly open/close the lid, activate bidet or air-dry functions, causing discomfort or distress to user.

(via Sophos Security)

Curious about several peculiar Apple related 404 errors for images in my web server logs, I decided to find what is going on, and became knowledgeable about yet another nugget that I really didn’t want to know. (sigh)

The puzzle was these log lines frequently coming up in my web server access logs.

...
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon.png HTTP/1.1" 404
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon.png HTTP/1.1" 404
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon.png HTTP/1.1" 404
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon.png HTTP/1.1" 404
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon-114x114-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon-114x114.png HTTP/1.1" 404
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon.png HTTP/1.1" 404
...

I first thought that I got something in my web site theme wrong, and grepped everywhere but could not spot this URL anywhere. When it didn’t turn up anything, I searched on the web for a bit and this is what I found out.

These queries are sent out by mostly Apple devices, and maybe some Android ones too, just to be compatible.

These URLs are the equivalent of favicons in Apple’s world. So instead of implementing the standard favicon definition using “link” tags, Apple apparently implemented their own convention.

So, if you have OCD about these matters (404s in logs) like me, and really lazy on the other hand (the Apple standards will require you to create several versions of these icons) you should create a 144x144 PNG file with your favicon and place it in the root of your website with both these names – apple-touch-icon.png and apple-touch-icon-precomposed.png. That will show them apples. Heh. Umm. I will end here.

References:

1. Stackoverflow: Why am I getting error for apple-touch-icon-precomposed.png
2. Ghacks: Why Webmasters Should Analyze Their 404 Error Log
3. Mathias Bynens: Everything you always wanted to know about touch icons
4. Apple’s guidelines on creating “web clips”