PyDev adventures

PyDev crowdfunding finished

2013-05-14T13:16:00.000-07:00

Ok, the PyDev crowdfunding has just finished -- and it reached its basic goal :).

The funding was divided in a series of perks, 610 people contributed and the final distribution for them (monetary-wise) was:

Altruist: 1.5% (Thank you)
Early adopter: 17.95% (LiClipse regular license -- no access to betas)
Believer: 11.92% (LiClipse regular license -- access to tracker and betas)
PyDev Knight: 23.48% (Get a vote in the PyDev tracker and contribute only to PyDev itself)
Oracle: 22.61% (LiClipse perpetual license)
Bronze Sponsor: 2.86% (10 regular licenses)
Silver Sponsor: 19.68% (10 perpetual licenses)

I must say that the funding changed quite a bit. Initially there was no PyDev Knight perk (which is a perk targeted only towards PyDev, whereas the others are targeted at PyDev and LiClipse). So, this was the result of listening to feedback and adjusting the campaign (also, initially I didn't put enough details on PyDev and added to much emphasis on LiClipse -- which is something adjusted during the campaign).

There is also one thing I failed to check properly before : Paypal fees are not the same everywhere. Here in Brazil it's 7.4% + 0.3US$ (whereas in US it's 2.9%+0.3US$). So, in the end, 12% went to Paypal/Indiegogo, whereas I was planning something around 8% (and there are still taxes to pay, but this is something to be expected anyways, so, in the end the net result for me will be something as 73% -- but just because I have a company setup for that, had I done it as a person it'd be around 63% here in Brazil -- I just hope the government does use it's share properly -- grin).

I still think it was worth as a way for a one time funding as everything is already setup and to know if there's interest before starting it -- which I think is the main point of a crowdfunding campaign: discover if something is viable before delving into it (not that failing on a crowdfunding is always a sure sign that something won't work, but in this specific case, with a product that is already mature, and with an existing community, I think it's how I'd see it).

Also, during that process I've set up a new tracker at https://sw-brainwy.rhcloud.com for both PyDev and LiClipse. It's a tracker I had done for fun some time ago (and yes, I know it's one of the seven sins of a programmer), but as I'd have to customize any solution for what I want, I think it's nice. The idea is that anyone will be able to create tickets and comment on those, but for voting which tickets will be done next you'll have to become a PyDev Knight (I still haven't sent the access for existing PyDev Knights to vote, but I'll do it during the upcoming month -- as you can see, I still have some things to set up).

A note on the tracker: it's hosted on RedHat's OpenShift, and it's called Brainwy because that's the company I created to keep PyDev going :)

Mainly it's a Javascript-based tracker where it's easy to see the whole picture and move things around. If you're curious, you can use an offline version at https://sw-brainwy.rhcloud.com/demo/, because as a user in the PyDev/LiClipse trackers you won't really be able to move things around, just add ticket, comment on existing ones and vote (once you have a vote) -- you can use it as much as you want, but note that I don't plan on adding support to it or actually distributing it differently -- I've already enough supporting PyDev and LiClipse :)

All in all, it was definitely a good outcome and it was nice having so much support from so many people!

Thank you everyone!

Fabio

PyDev crowdfunding basic funding reached!

2013-04-30T08:29:00.001-07:00

The initial target of the crowdfunding for PyDev has been reached. Thank you everyone who helped in reaching that target!

Now, please don't stop sharing and helping it get to the higher proposal levels (having more there will enable more things to be done).

Also, I've had some people ask me privately on how things will proceed afterwards... well, with this funding, I'll be able to work in PyDev for the upcoming year, so, during that time I'll create a structure so that people that like it can keep supporting it so that it can move forward year after year.

Also, I hope LiClipse itself turns out to be successful, as the idea is that earnings in LiClipse will also help in getting PyDev going on -- if things turn out pretty well, I'll probably be able to get more people to help there too :)

PyDev crowdfunding / LiClipse initially supported languages

2013-04-16T04:34:00.000-07:00

Right now (30+ days elapsed on the funding), things are looking good, the PyDev/LiClipse funding already got to 70% of its target (with 354 supporters). Thank you everyone! Please keep sharing about it ;) (http://igg.me/at/liclipse)

On the PyDev side, the new tracker is still not up, but I'm working on it (so, please buffer your requests until the tracker is up... I'll provide proper links once that's done).

Regarding the initial languages to be supported in LiClipse, I'm sharing below the results of the poll so far (note: only languages with more than 15 answers are shown).

python 152
javascript 106
html 98
css 93
django templates 70
cython 53
c++ 47
c 39
jinja2 37
java 33
coffescript 27

On the total there were 50 languages voted. There was one I didn't even know about (the Ren'Py language http://www.renpy.org/ ).

I'll still leave the poll there until the end of the funding (vote on poll) to decide on the initial supported languages, but note that even if a language is not initially there, I'll put up proper docs and provide support so that anyone should be able to easily add a language there ;)

PyDev / LiClipse funding (and initially supported languages)

2013-04-04T14:20:00.000-07:00

Ok, 20 days on the crowdfunding elapsed and the funding is already at 57% (with 258 funders)! So, the funding is going pretty well so far. Thank you everyone!

Also, I'd like to give special thanks for the current Silver sponsors (http://holdenweb.com - http://kichwacoders.com) and the Bronze sponsor (http://wobe-systems.com).

Still seeking Gold and Platinum sponsors though... so, if there's a company that is a heavy user of PyDev -- or maybe the future LiClipse -- and would like some space in the PyDev / LiClipse homepage...

Also, I've put up a poll on google docs to help in deciding which will be the languages supported initially in LiClipse. You can vote at: LiClipse Languages Poll.

Note that I still have not put up a new tracker for PyDev itself (but I'm working on that).

Crowdfunding PyDev / LiClipse status

2013-03-25T09:19:00.002-07:00

So far, the experience on the crowdfunding has been really positive. In 10 days of funding, there are already 168 funders and 41% of the basic funding goals have been reached!

Also, it was nice to gather feedback and talk to many PyDev users out there.

Regarding PyDev, reactions were pretty positive, but regarding LiClipse, they were initially a bit mixed.

On the dark theme, it was either a love or hate reaction, but just to note, although that's one of the major points of LiClipse, it will definitely not force you to use a dark theme -- it'll be an opt in, not opt out -- and I expect that part to be the easy / fast part in the proposal :)

As for the languages support, initially it was a bit more neutral (with some ups and downs), and I think initially I did a bad job on explaining why I think that was needed, so, I've updated http://igg.me/at/liclipse to explain the 'why' part too :)

Note that as a result of talking to PyDev users, I did change the funding a bit too: there's now a special perk (named PyDev Knight) which empowers users who enjoy PyDev and only really care about a better Python support and not a dark theme or other languages (even if related to Python, such as Django Templates, Mako, etc).

So, I can only thank everyone so far for contributing and spreading the word on the funding!

PyDev funding and LiClipse?

2013-03-16T04:35:00.005-07:00

I got some feedback related to the creation of LiClipse and its relation with PyDev in the current crowdfunding proposal (http://igg.me/at/liclipse/), so, I'd like to explain how I believe things fit together.

1. So, is LiClipse a fork of Eclipse?

Definitely not. The idea behind LiClipse is having 2 things:

A way to theme Eclipse itself better and have some other UI improvements (I'm truly annoyed by not having what I'd consider a professional dark theme in Eclipse right now, so, I'd like to take those matters in my own hands).
An editor which should be able to support lots of languages out of the box. Think something closer to other all-purpose editors -- as opposed to IDEs -- such as Notepad++, Vi, TextMate, Sublime, etc. i.e.: the idea is supporting lots of languages out of the box, so, the idea is having it resembling formats such as ultraedit wordfiles or TextMate language files.

To be clear, the idea is not to replace the more advanced editors inside Eclipse for each language, but to provide a lightweight way to deal with any language -- usually you work with 1 or 2 main languages, for which you'll have the plugins you need, but sometimes, when you just want to open a file in a language you work seldomly and may not need/want to install a bulkier plugin, LiClipse would be a good addition to your toolbox (LiClipse is a short for "Lightweight Eclipse" BTW), and for me, not having this is a major shortcoming of Eclipse itself (and it may be an alternative for people who want less features and more speed).

Also, that should be doable without having to create a fork of Eclipse (although some theming issues may need to be resolved at Eclipse itself -- but on those parts, things should be fixed at that level, not on LiClipse).

2. How does that relate to PyDev?

Well, PyDev is very tightly bound to the environment it works on (Eclipse), and I'd like to solve some of the issues I see in it to improve its ecosystem as a whole, which IMHO is something I see needed for PyDev and Eclipse itself to keep moving forward.

3. But won't that divert too many resources out of PyDev itself?

For this proposal, my plan is spending my time 50/50 (I don't think LiClipse is a major undertaking -- all the pieces are out there, it's mostly a matter of stitching them together), although during that time, yes, it'll divert some resources from PyDev, but as I think that having it is very important for PyDev itself (as well as other language), I see the issues being tackled as very serious shortcomings of Eclipse which hinder the adoption of Eclipse itself (thus affecting PyDev directly).

4. I still think LiClipse is not a good idea and would like to support only PyDev.

Please e-mail me with those thoughts. I really believe LiClipse is needed for PyDev to keep getting traction, but if you feel that's not the case, please, please share your thoughts with me.

Also, just to make things clear, if the funding at http://igg.me/at/liclipse/ does not succeed, I won't really be able to support PyDev itself anymore (personally, I really want it to succeed, but I see the funding as the community speaking, so, if after 10 years working on it the PyDev community doesn't see it as a worthy goal or doesn't trust me enough to support me on LiClipse while properly maintaining PyDev, well, I really need to hear it and move on -- as a note, until now, I see the funding as a huge success, getting to 10% in the first day, so, if everyone keeps helping a bit there, it'll be awesome :)

Keeping PyDev alive through crowdfunding

2013-03-14T09:01:00.002-07:00

Ok, I just started a crowdfunding project (at www.indiegogo.com) for the continued development of PyDev (and improvements on Eclipse overall).

In this post, I hope to shed some light on why this funding is needed.

First, a little bit of history just to give some context: I've been developing PyDev for more than 10 years already (wow, I just realized that as I started the campaign) and it's definitely a pretty successful project (if it's not the most used Python IDE, it's definitely among the top ones).

So, backtrack a few years...

PyDev started doing success and I created a commercial extension to enable me to work more time on PyDev itself. After some time, Aptana acquired it and I joined them. Unfortunately, Aptana itself didn't turn out very well: it focused on its main product (the IDE) without generating any revenue from it, while trying to make side projects worthy enough to cover for everything (but those side projects weren't successful enough for that).

Enter Appcelerator: it acquired Aptana out of the need to provide a deeper integration for its main product (which is Titanium). So, for some time, it did back up the PyDev development (along with the other languages), but in the end, their main product is Titanium and the tools around it, so, PyDev itself wasn't seen as relevant enough to be kept supported (to be clear, they're still hosting the homepage and downloads, but not backing the development itself). As such, in the end of the last year they stopped supporting its development.

So, that's where the project is at now: it's (IMHO) a pretty successful, but unable to generate revenue for its continued support. Given this scenario I decided to create a crowdfunding project to ask for the community to provide resources to make that happen.

In the funding, I expanded its reach a bit, on what I think are the main issues with the PyDev environment right now. So, the idea is focusing on a nicer dark UI, usability, speed and memory and providing a way to easily have other editors out of the box in a lightweight implementation (Python is pretty strong in the web, so, one of the weak points right now is actually not the Python editing itself, but related web languages, such as CoffeScript and JavaScript).

So, please help in funding (and sharing) at http://igg.me/at/liclipse to keep PyDev going strong!

Interrupting a Python thread with signals

2013-01-17T07:29:00.000-08:00

First off, in Python, as appears to be common knowledge, signals can only be received by the main thread, and usually you need to do synchronization work with Queues or something else to work with other threads, so, what I'll describe below is a different approach to the problem which in effect will give you a way of interrupting a thread in Python from anywhere.

First off, I'll show the code and explain it afterwards:

import time
import sys
import threading

class SigFinish(Exception):
    pass

def throw_signal_function(frame, event, arg):
    raise SigFinish()

def do_nothing_trace_function(frame, event, arg):
    # Note: each function called will actually call this function
    # so, take care, your program will run slower because of that.
    return None

def interrupt_thread(thread):
    for thread_id, frame in sys._current_frames().items():
        if thread_id == thread.ident:  # Note: Python 2.6 onwards
            set_trace_for_frame_and_parents(frame, throw_signal_function)

def set_trace_for_frame_and_parents(frame, trace_func):
    # Note: this only really works if there's a tracing function set in this
    # thread (i.e.: sys.settrace or threading.settrace must have set the
    # function before)
    while frame:
        if frame.f_trace is None:
            frame.f_trace = trace_func
        frame = frame.f_back
    del frame


class MyThread(threading.Thread):

    def run(self):
        # Note: this is important: we have to set the tracing function
        # when the thread is started (we could set threading.settrace
        # before starting this thread to do this externally)
        sys.settrace(do_nothing_trace_function)
        try:
            while True:
                time.sleep(.1)
        except SigFinish:
            sys.stderr.write('Finishing thread cleanly\n')


thread = MyThread()
thread.start()
time.sleep(.5)  # Wait a bit just to see it looping.

interrupt_thread(thread)
sys.stderr.write('Joining\n')
thread.join()  # Joining here: if we didn't interrupt the thread before, we'd be here forever.
sys.stderr.write('Finished\n')

So, there you have it: a hacky approach which will prevent your code from being properly debugged :)

-- note that sys.settrace could be changed for sys.setprofile to achieve the same result -- in which case your program could be debugged but not profiled (which may be a better approach but will make the signal be raised only on function calls, not on line calls).

To sum it up, we use the tracing mechanisms that's intended for debuggers (or profilers) to actually throw the signal for us. This means that we have to enable the tracing on that thread (which will make that thread to execute a bit slower) and set the tracer function to a function which will actually throw the signal.

IMO, it works in a hackish (but nice) way... not sure if I'd use that in a production environment, but, there you have it: interruptible threads in Python (yes, you still have the limitation of the GIL: a thread won't be interrupted while calling some atomic operation that doesn't release the GIL)... Now, if only there was a Python signal library better integrated with the Python interpreter so that it checked for signals itself (probably in a way close to the tracing function itself with less overhead as only a single additional check for a null variable would be needed -- and otherwise a signal would be thrown), this hack wouldn't be needed in the first place -- but I'll leave that to someone else reading this :)

Python to Javascript translation

2013-01-07T05:46:00.001-08:00

I'm currently doing a project with a backend in Python and a client mostly in Javascript -- still not public, but if everything turns out right I'll post about it later on -- and there are some algorithms I'd like to reuse in both, but without going to a full Python in browser style (such as Pyjs or brython)... Ideally I'd like something like Coffescript (but with a Python-like syntax so that a subset of Python can be used for both Python and Javascript generation).

-- Sidenote: I decided I really have to grasp Javascript in order to find out how things work before trying an alternative language which compiles down to Javascript -- so far my experience has been nice. I found that I end up programming much more in a functional style and use classes in a very limited way (haven't felt the need for inheritance so far), although I feel that it's hard to find resources on what would be the proper way of doing things (but in the plus side, there are LOTS of javascript-related resources, so, I've hardly found myself stuck while learning it).

Anyways, I have some algorithms I've written in Python which I'd like to reuse in Javascript without having to use huge runtime with it (I'm restricting that to simple functions) and searching for a translator, I've found Pyvascript, which is a Python-like language which compiles to Javascript (so, there's only a subset of Python that can really be used, which is ok as I'm not after the ones that attempt to do full Python in Javascript and end up with a huge emulated runtime), but it does fit my needs properly with some minor changes (and now I have some code I can use in both Python and Javascript at the same time). Just to note, it doesn't seem like the project is active right now, but it does work fine for me :)

Python: get parent process id (pid) in windows

2013-01-02T05:41:00.001-08:00

Below is code to monkey-patch the os module to provide a getppid() function to get the parent process id in windows using ctypes (note that on Python 3.2, os.getppid() already works and is available on windows, but if you're on an older version, this can be used as a workaround).

import os
if not hasattr(os, 'getppid'):
    import ctypes

    TH32CS_SNAPPROCESS = 0x02L
    CreateToolhelp32Snapshot = ctypes.windll.kernel32.CreateToolhelp32Snapshot
    GetCurrentProcessId = ctypes.windll.kernel32.GetCurrentProcessId

    MAX_PATH = 260

    _kernel32dll = ctypes.windll.Kernel32
    CloseHandle = _kernel32dll.CloseHandle

    class PROCESSENTRY32(ctypes.Structure):
        _fields_ = [
            ("dwSize", ctypes.c_ulong),
            ("cntUsage", ctypes.c_ulong),
            ("th32ProcessID", ctypes.c_ulong),
            ("th32DefaultHeapID", ctypes.c_int),
            ("th32ModuleID", ctypes.c_ulong),
            ("cntThreads", ctypes.c_ulong),
            ("th32ParentProcessID", ctypes.c_ulong),
            ("pcPriClassBase", ctypes.c_long),
            ("dwFlags", ctypes.c_ulong),

            ("szExeFile", ctypes.c_wchar * MAX_PATH)
        ]

    Process32First = _kernel32dll.Process32FirstW
    Process32Next = _kernel32dll.Process32NextW

    def getppid():
        '''
        :return: The pid of the parent of this process.
        '''
        pe = PROCESSENTRY32()
        pe.dwSize = ctypes.sizeof(PROCESSENTRY32)
        mypid = GetCurrentProcessId()
        snapshot = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0)

        result = 0
        try:
            have_record = Process32First(snapshot, ctypes.byref(pe))

            while have_record:
                if mypid == pe.th32ProcessID:
                    result = pe.th32ParentProcessID
                    break

                have_record = Process32Next(snapshot, ctypes.byref(pe))

        finally:
            CloseHandle(snapshot)

        return result

    os.getppid = getppid

Prototyping a class in Python

2012-12-14T08:08:00.001-08:00

I'm not sure what's the name for this technique (for me it resembles what is called prototyping in javascript -- I know it's not the same, but the fact is that you're building your class outside of it, although in Python it's still in its declaration step).

Anyways, the idea is the following: you want to change the class attributes before it becomes final -- there are things you just can't do in Python after a class is already declared (Python uses this technique for creating properties -- in my specific use-case I'm using it to overcome some limitations that Django has in its model inheritance with fields, but I've already used this many times).

The idea is the following: you get the frame which is being used inside the class declaration and change the locals in it so that the final created class will have the things you declared.

I.e.: This code:


import sys

def prototype_class(frame=None):
    if frame is None:
        frame = sys._getframe().f_back

    frame.f_locals['new_attribute'] = 'New attribute'

class MyNewClass(object):
    prototype_class()

print MyNewClass().new_attribute

Is the same as:


class MyNewClass(object):
    new_attribute = 'New attribute'

-- Just more complicated and more flexible -- in my case, it'll properly help me to choose how to create and customize the fields of a Django model class without having to copy and paste a bunch of code.

On a separate note, blogspot just sucks for code... why can't they simply create an option to add a piece of code? I'm now manually putting my code in html as described in http://stackoverflow.com/a/8697421/110451 -- it'd certainly be trivial for the blogspot devs to put a button which would do that for me right? (colors would be nicer, but this is the easiest things that just works for me and I'd already settle for it if blogger added it -- I know blogger has the quotes, but I want the code at least on a box with a monospaced font).

Python tricks: making sure a function is only called once

2012-12-07T04:43:00.000-08:00

Sometimes I want a function to be called only a single time (there are many use-cases, but specifically this time I wanted to change the way the Python garbage collection worked when dealing with Qt and threads).

Anyways, usually I did that setting some flag so that the second time a function is called, it'd check that flag and would skip the function in the second time.

After tinkering a bit about it, I came with a shorter version changing the function code which I thought is pretty nice:

 def func():  
   print 'Calling func only this time'  
   func.func_code = (lambda:None).func_code

Now, calling it a second time will actually execute the lamda:None code.

Yes, I know you'll say it's cryptic, so, I'm including a decorator version below which does the same checking with a variable:

 @call_only_once  
 def func():  
   print 'Calling func only this time'  
 def call_only_once(func):  
   def new_func(*args, **kwargs):  
     if not new_func._called:  
       try:  
         return func(*args, **kwargs)  
       finally:  
         new_func._called = True  
   new_func._called = False  
   return new_func

Plugins in Eclipse to accompany PyDev

2012-12-06T04:11:00.001-08:00

Below are some plugins which I believe may be very helpful for people installing Eclipse/PyDev (those are the plugins I usually install on any new Eclipse install).

1. StartExplorer:

Details at https://github.com/basti1302/startexplorer

Mostly, I use it to open the current file in explorer or copy the current file path to the clipboard (but it has other niceties too).

2. Extended VS Presentation plugin for Eclipse

Details at: http://andrei.gmxhome.de/skins/index.html

I use it for skinning Eclipse (i.e.: providing a better look and feel), but note it's only for Eclipse 3.x (Eclipse 4.x has a skinning engine builtin).

3. Eclipse Color Themes

Details at: http://eclipsecolorthemes.org

Note that if you're using the Aptana Studio editors, you don't really need this plugin as Aptana Studio itself has a theming feature which supersedes this, but if you're only on Eclipse/PyDev and other editors, this may be a nice addition so that you can apply the same colors for several editor plugins at once.

4. Practically Macro

Details at: https://sourceforge.net/projects/practicalmacro

Yes, I know, Eclipse does not have macro record/playback by default and it's a major shortcoming. Well, this plugin does cover most of the use-cases I have for doing macro record/playback (although it does have some weirdness sometimes -- in which case I usually just start notepad++ editor just for this feature).

5. EGit

Details at: http://wiki.eclipse.org/EGit/User_Guide

Note that Aptana Studio has its own Git integration (so, when I'm in Aptana Studio I don't usually install EGit). Although it can do many things with Eclipse, I use the Eclipse integration (either from EGit or Aptana Studio) mostly to have a quick way to know which files I've changed and sometimes seeing the history for a given file. For most of my work I like to work on the shell, usually using a tool I wrote in Python: https://github.com/fabioz/mu-repo which helps managing many git repositories at once from a shell (and showing differences I can edit via WinMerge) -- and sometimes I even fire http://code.google.com/p/gitextensions for exploring some git repository.

I was thinking a bit about this one since in cvs and svn I did everything inside Eclipse, but when it comes to git this trend didn't continue. I think it's mostly because the major thing for me is that the synchronize view wasn't available to me on cvs/svn and the EGit synchronize view is not as streamlined/fast as the other integration -- as I have the habit of reviewing all the code I'm about to update, and as EGit made that process a bit slower (and git is pretty easy to extend with scripting on top of it), I just went a different way this time (but who knows, if EGit provides a more streamlined/faster synchronize view in the future for my modus-operandi I may switch back to only Eclipse here -- but it'd have to beat the time in which I do things in mu-repo which fetches/diffs lots of repositories in parallel in a pretty fast way, so, I'm not sure how feasible that is).

Debugging SQL in Django (with PyDev)

2012-11-20T02:50:00.003-08:00

I'm working on a Django project recently and one of the things that came up was debugging SQL statements a bit to see what was actually going on.

After researching a bit, I found: http://pypi.python.org/pypi/django-debug-toolbar, which when installed wraps around the calls that Django does to the DB and prints the actual SQL statements used.

I didn't really install it as a debug toolbar as it recommends, but I use it in the interactive console shell to see how things happen when I'm experimenting.

To do this, after installing it and adding 'debug_toolbar' to your INSTALLED_APPS in Django, one can start the interactive console shell with Django support (if you're in a Python editor just do: Ctrl+2 dj shell) and add write the import:

from debug_toolbar.management.commands import debugsqlshell

Now, when you do some operation that does an SQL operation, it'll be printed to the console, while having all the PyDev niceties on the console (such as code-completion). Pretty nice.

PyDev & Eclipse 4.2 (i.e.: stick with Eclipse 3.8)

2012-08-14T07:45:00.002-07:00

Ok, I've been giving a whirl on supporting Eclipse 4.2 on PyDev and everything seems to work as expected, but unfortunately, I still can't recommend the Eclipse 4.x series over the 3.x series because performance-wise Eclipse 4 is still not on par with Eclipse 3 (and for me the only gain was that floating docks are now better -- as for skinning, I don't really think it's much better for final users unless someone else already got to the point of doing a skin exactly as you'd want -- so I think that for now, Aptana Studio 3 skinning (themeing) is a better solution -- and targeted at Eclipse 3.x).

I have hopes that those issues will be addressed in a newer release of Eclipse 4, but the truth is that right now Eclipse 4 feels sluggish when compared to Eclipse 3 (even changing configurations on the Eclipse appearance preferences for having some performance gains).

So, if by any chance you've just gotten into PyDev and it doesn't seem as speedy as you'd like, please make sure you check it with Eclipse 3.8 (in the downloads page: http://download.eclipse.org/eclipse/downloads/ there's a link to Eclipse 3.x downloads available: http://download.eclipse.org/eclipse/downloads/eclipse3x.html).

Also, make sure that you have the latest version of java as it may make a difference.

Or, if you've gotten the Aptana Studio 3 download directly, you're already good to go -- as it's already based on Eclipse 3.x and has PyDev preinstalled (see http://pydev.org/download.html for links).

Git (multiple repos)

2012-07-02T10:18:00.001-07:00

Improving my Git workflow has been one of my current targets lately... Especially when working with multiple projects.

So, although I'm working all the time inside Eclipse, I still like to use the command line for many things (so, even with the Aptana Studio 3 Git integration and EGit having improved, I still use the command line a lot on common operations, resorting to other options mostly when investigating the history of a repository or some other thing which may be suboptimal in the command line).

To improve that situation, I ended up creating a tool... Yes, I know there are probably many other command line tools that try to accomplish that, but I did a good amount of research and didn't find anything that worked as I wanted -- and not nearly as fun (but I appreciate suggestions in this realm).

The tool I did is hosted at GitHub: https://github.com/fabioz/mu-repo, and it's freely available (GPL 3).

It's a Python program which interfaces with the git binary to execute whatever is needed. Also, for diffing operations it currently relies on WinMerge (meaning this part of the tool is Windows only for now, but if there's interest, it should be straightforward having it support other diff tools such as KDiff3).

My basic workflow is now:

cd /workspace (which has many .git subrepos)

Note: all the "mu" operations below will actually be applied to all the git subdirs (must be previously registered), so, a single command line is enough for dealing with all repositories at once -- yes, I know about git submodules, but I don't really like how it works -- and it's one of the main reasons this tool exists :)

To get changes:

mu up (fetches origin for the current branch updating the origin/current_branch reference -- I found it weird that trying to do "git fetch origin current_branch" ended up updating FETCH_HEAD instead of origin/current_branch -- also, this and most other operations work only in the current branch, leaving other branches or references untouched).

mu dd origin/current_branch (preview incoming changes: compare current branch HEAD to origin/current_branch).

mu rebase origin/current_branch (rebase incoming changes -- could also be: mu merge origin/current_branch).

To submit changes:

mu st (check status of changed files -- indexed or not)
mu dd (diff working copy with winmerge while editing changes and keeping the working dir updated even if no symlink support is available)
mu acp "message" (add all working copy changes, commit, push)

To start working on a feature:

mu co -b branch_name (create branch for all repos)
mu ac "message" (add / commit changes on that branch)
mu co main_branch (go back to main branch)
mu rebase branch_name (get the changes from the branch into the main branch)
mu branch -D branch_name (remove that local branch)

Note that for my workflow I work mostly without taking "advantage" of the git index -- usually I review things with "mu dd" and on "mu ac" I add all the changes to the index and commit in a single operation (before that I was almost always doing "git add -A" and "git commit -m "message"" anyways).

I also still configure many things as I posted previously: http://pydev.blogspot.com.br/2011/02/git-on-command-line.html, but a bunch of things are now better integrated with this tool :)

Also, the tool works for a directory that has a '.git' repository, so, operations such as the diff with WinMerge can be used even when working with a single git repository.

p.s.: The tool is still in beta, so, although I'm using it all the time, it's possible that some things are not working as they should -- but on the good side, I still haven't been able to corrupt any of my git repos :)

PyDev 2.6.0 released

2012-06-28T13:43:00.004-07:00

PyDev 2.6.0 is just out of the door.

Main in this release were changes in the interactive console -- and mostly by external contributors, so, it seems this feature has really picked up some traction :)

Hussain Bohra made it possible to have an interactive console bound to a debugger frame and James Blackburn did a bunch of fixes in it.

Aside from that, the default PyUnit runner has new options to include/exclude files from the test loading process and Jython has an annoying fix done (print can be used as a dotted notation) along with many other fixes (see: http://pydev.org for more details).

Also, the PyDev official issue tracker is now the same tracker used in Aptana Studio 3: https://jira.appcelerator.org/browse/APSTUD/ (there's a component related to PyDev there).

The issues currently on Sourceforge will be migrated to that tracker, but there's no way to make the current Sourceforge trackers read-only, so, I'm just redirecting people to the new tracker and asking not to post on the old one -- hope that works :)

It's interesting to note that now the official Eclipse release is focused on Eclipse 4.x (and not on 3.x)... the integration has already been tested, but not as much as the 3.x series, so, please report any issues found in the way.

PyDev 2.5.0 released

2012-04-10T04:16:00.002-07:00

Major things in the PyDev 2.5.0 release include:

Proper support for Django 1.4 (as it's layout changed, the project creation wizard wasn't working properly).

Better patching of Django 1.4 for the debugger and auto-reload (details at the end of the page: http://pydev.org/manual_adv_django.html). Note: there is an issue when using it on Linux, which is already fixed in the nightly build, so, if you are on Linux and want to experiment with it, please update to the current nightly build.

The interactive console can now be attached to the variables/expressions view (the preference at: window > preferences > PyDev > Interactive Console and check 'Connect console to Variables Debug View?' must be enabled). Also, there's drag and drop support for it now.

Aside from those, many bugs were fixed and minor improvements were done. More details can be seen at: http://pydev.org/

PyCon retrospect

2012-03-13T08:00:00.003-07:00

Ok, so, this was the first time I've been on PyCon and I must say it was pretty nice.

Initially, it started out kind of lousy for me: United seems to have 'misplaced' my bag and I only got it back in time to check it back to Brazil, so, that sucked. On the other hand, to balance things, I won a Kindle DX in raffle from http://www.truecar.com :)

Mainly, it was awesome talking to so many great names of the Python world... Meeting people face to face does make a great difference (some things are simply not as well expressed in an e-mail).

On the PyDev side, it was pretty nice seeing many people using it while hacking away during the conference and even having some people asking 'how can I help to make PyDev better?'.

I guess I wasn't properly prepared and mostly, my answer was: read the devs guide to grab the code/compile, find an itch to scratch and mail me about it so that I can help you find yourself there... Some things can be done in Java, some in Jython, depends a lot on what you want to do.

Aside from that, documentation-wise, I guess the major issue would be having some section explaining how to configure each major library on how to work with PyDev (most work out of the box, but some need minor adjustments such as adding a token to the forced builtins and some may even require more than that), so, if someone is willing to help working on the PyDev documentation in that topic, that would be a great help (and I guess that if someone doesn't have an itch to scratch and still wants to contribute in code, there are things I can point to that need to be done and are relatively simple to get started).

On the scientific computing side, I liked Numba (https://github.com/ContinuumIO/numba), which seems to be a pretty nice idea... it's still in its early stages, but I think it could be a nice substitute for Psyco (but with knowledge about numpy types and maybe a bit more strict -- if I did understand it properly)... I'd much rather use something as Numba for speedups than Cython (as the Cython code is only 'close' to Python, but doesn't really run on a Python interpreter) and Pypy is just not an option right now...

Although Pypy seemed strong in the conference, I must say that it'll probably be a long, long time until I'll be able to use it in production... I really use lots of code in C/C++ bindings, so, this is a killer CPython feature which I'm not sure Pypy will ever be able to provide (and the 'current' workaround of porting a library to RPython as is being done in Pypy/Numpy doesn't seem to be really feasible).

One thing I thought was a bit strange was the lack of projects with Python in mobile platforms, as I was guessing it should be straightforward to compile Python to Android/iPhone (so, I'm guessing the issue is probably the lack of proper multi-platform bindings on that area). In contrast, Python seems very strong on the server-side and scientific computing fields.

And I think Python 3 does deserve a special note here: I think Guido left it pretty clear (as much of his keynote talked about the subject) that Python 2 is really deprecated, so, although it's currently the production version that's most widely deployed/used, people have to really start getting used to the idea that a port to Python 3 is the way to go -- although in the real world, I still do expect this to take quite a few years (probably more than was initially expected by the Python-core folks). The fact that Guido made this topic such a big portion of his keynote, does make me feel like many of the Python lovers were against this move and the actual response from the community on this respect has been mixed, but regardless of that, the direction that Python-core is going is 100% on Python 3 (maybe doing something here or there to help in easing the port or having a shared Python 2-3 codebase).

And, there are still have some talks I'll see later at: http://pyvideo.org (because many interesting talks took place at the same time).

PyCon

2012-03-06T16:58:00.002-08:00

Tomorrow I'll be traveling to PyCon and I'll be staying during the 3 days of the main conference (Friday to Sunday).

So, anyone who wants to talk about PyDev, scientific programming, how to best structure a Python development environment -- or just wants to say hi, I'll be there this time :)

Just to note, currently the major projects I'm working on are:

PyDev (http://pydev.org), which I guess is pretty well known (especially if you're reading this blog), and I've been developing it for 9 years already :)

and Kraken (https://www.esss.com.br/kraken/), which is a software to do post-processing of reservoir simulations (this may be less known, but actually, I started working on PyDev to scratch my own itches while doing scientific programming in Python, and so far, Kraken is the most advanced/complex software I've worked on this area and I've been involved with its development since its beginning, around 5 years ago).

See you in Santa Clara :)

PyDev forums -> StackOverflow

2012-02-08T16:09:00.000-08:00

The PyDev forums at SourceForge are now officially deprecated :)

So, anyone having a doubt regarding PyDev should now ask at StackOverflow and add a 'PyDev' tag.

I think this will be a real improvement over the current status quo... Some reasons I see for that are:

1. Many PyDev users follow StackOverflow and do answer things there, whereas in the PyDev forum, many questions were asked, but I was almost the only one answering... (I think the real plus here are the 'gaming' features that StackOverflow has, so, more people are inclined to participate actively).

2. As people started asking there anyways, I really had to follow StackOverflow closely too, so, deprecating the PyDev forums means I'll be able to follow a single place again :)

3. Interacting with StackOverflow as a whole seem a nice improvement over the SourceForge forum (it's edition is nicer, accepts pictures, etc.)

And now on to something a bit unrelated... the PyDev homepage (http://pydev.org) is now being generated from a wiki (it's still a read-only wiki -- but hopefully that'll change soon -- but at least, I feel it'll be easier for me to edit things there and later have the homepage updated from it). So, if someone finds something strange in the homepage, please let me know :)

PyDev 2.4.0 released

2012-02-01T09:51:00.000-08:00

This release was mostly focused on performance and memory optimizations.

On the performance front, the major focus was on start up (i.e.: start up Eclipse, open an editor, request a code-completion and show the globals token browser (Ctrl+Shift+T)) -- which should've become pretty fast (tested only with the Eclipse runtime and PyDev -- as things in other plugins can't really be controlled -- the subversive plugin for subversion seems to be especially slow to startup).

Memory-wise, things have been improved too, with the AST taking up less memory and doing a 'pseudo-intern' for some rather large caches (it's a pseudo-intern because the String.intern() function is not used: a HashMap is done and strings are reused inside it for some processes -- and later that HashMap is thrown away), and the Jython plugin was fine tuned to make less plugins visible to save on memory (and startup time).

Just to note: the real memory used can be seen going to window > preferences > general > show heap status (the real size of the java process in the OS will probably be bigger as java will usually grow to the size specified by -Xmx, regardless of how memory it's really using at a given time). Personally, on large projects I allocate 300 Mb for the process, but this is mostly because the subversion plugin seems to be rather resource hungry -- migrating to git on some of those projects seems to be making things better :)

Aside from that, this time I spent some time migrating the PyDev homepage to a wiki ( https://wiki.appcelerator.org/display/tis/Python+Development ) -- right now it's not available for external edition, but that should happen soon (hopefully), and the idea is that the PyDev homepage will be generated mostly from that wiki.

And as usual, a bunch of bugs were fixed :P

Code-completion strategies in PyDev

2012-01-05T08:01:00.000-08:00

I believe one of the strong points in PyDev is its code-completion, so, I thought a bit about giving some details on it :)

The main preference page for code completion is: Window > Preferences > PyDev > Editor > Code Completion (my preferred configuration is setting the 'Request completions on all letter chars and '_'', so that completions appear automatically when typing, otherwise Ctrl+Space would need to be used to request the completions -- I was actually thinking about making that the default and decided against it to conform to other editors in Eclipse).

* Word completion (also called Hippie Completion):

This is probably the simplest one and is provided by Eclipse itself (through Alt+/). It provides a simple word-based completion which uses all the currently opened editors in Eclipse.

I've actually provided a patch for Eclipse to improve the speed of this completion (https://bugs.eclipse.org/bugs/show_bug.cgi?id=270385), which was added in Eclipse 3.6.

* Templates completion:

These are user-defined templates that may be configured at PyDev > Editor > Templates (most of the base for this completion is provided by Eclipse... PyDev uses a subclass: PyTemplateCompletionProcessor and some of the available variables may be defined in Jython code -- see: pytemplate_defaults.py for details).

* Common tokens completion:

When you start typing in PyDev, some common tokens (i.e.: keywords, self, etc) start appearing directly. Those can be configured in PyDev > Editor > Code Completion (ctx insensitive and common tokens).

It's implementation is pretty simple (may be seen at: KeywordsSimpleAssist)

* Context insensitive completion:

This completion goes through all the tokens available for a given project (which may need to consider project dependencies and which interpreter is being used) and shows those tokens as a completion (i.e.: top-level tokens such as classes or methods and the modules themselves).

If one of those is selected, the token will be completed and an import will be added for it too (if the preference in PyDev > Editor > Auto Imports > "Do auto import?" is marked as true -- in that same preferences page, the number of chars that need to be available in a word so that these completions start appearing may be specified).

Note that if the option was set not to do the auto-import, one could just add the token, let it be marked as an unrecognized variable by PyDev and later do an Organize Imports (Ctrl+O), or a Quick Fix in that line (Ctrl+1), to add the import.

The major issue in this completion isn't actually the completion per-se (implemented in ImportsCompletionParticipant and CtxParticipant), but the structure which needs to be kept to have it as a fast and efficient completion.

Mainly, PyDev has a concept called 'AdditionalInfo' (this was done when PyDev Extensions was separated from the PyDev Open Source, so, the name is a bit strange now, but the general idea is that it was additional information related to a given project or interpreter), which keeps the following information:

- Two TreeMaps (AbstractAdditionalTokensInfo.topLevelInitialsToInfo and AbstractAdditionalTokensInfo.innerInitialsToInfo) which map token names to information of the places where the token may be found (i.e.: module and structure inside that module). Those are all kept in memory and are pretty fast to access (AbstractAdditionalTokensInfo.getTokensStartingWith is what's interesting for a code-completion and AbstractAdditionalTokensInfo.getTokensEqualTo is interesting when doing a quick fix or organize imports). This structure is also used in the global tokens browser (Ctrl+Shift+T).

- Note that it also has a structure (AbstractAdditionalDependencyInfo.completeIndex) which maps a module to all the available tokens in it. This structure is kept in memory only as a SoftHashMap (so, it's only kept in memory while there's enough space for it) and persisted to the disk. It's also only lazily created on operations that need it (currently only a project-wide rename refactoring or a find references (Ctrl+Shift+G) would use it as it's basically a structure which is a bit faster for doing exact match searches than actually doing a search in Eclipse -- especially if the SoftHashMap is still in memory, so, if many find references are done in succession, if there's enough memory, from the 2nd attempt onwards, things should be fast).

On a project build, the tokens of the completeIndex are simply all removed (to be recalculated when some action that needs it is called). As for the maps, those are always kept up to date when a file is changed. The strategy for having it build fast is that the in-memory cache is directly updated (which is reasonably fast) and instead of saving the whole map it just saves the delta information and when restoring the info, those deltas are applied to have it in the last state (and from time to time it does dump the whole structure and removes the deltas). Also, it runs in a separate thread (not actually in the thread that's doing the build, and a singleton: RunnableAsJobsPoolThread, makes sure than only some of those, depending on the number of processors in your machine, are running at the same time, so, if you change 200 files at once, your computer won't come to a halt).

* Context sensitive completion:

This is by far the most complex completion available as it analyzes the context where you're requesting a completion and provides tokens based on it. Basically, PyDev has an internal type-inference engine to do that (which is also used by actions such as find definition or TDD actions such as create method).

Internally it uses an LRU structure which maps module names to the module AST (Abstract Syntax Tree) and in a pretty recursive algorithm finds out about the available tokens needed for a given context and provides completions based on that (thankfully it has a huge amount of unit tests holding it all together). That process starts in PyCodeCompletion.getCodeCompletionProposals(ITextViewer, CompletionRequest) and the type inference engine main classes are: ASTManager and ProjectModulesManager.

On some occasions some modules may be pretty hard to analyze, in which case PyDev resorts to launching a shell and querying it for the needed tokens (those are pre-specified as in window > preferences > PyDev > Interpreter > Forced Builtins, and the communication happens in the java side through the AbstractShell class) -- it's also probably one of the main reasons of problems when configuring PyDev, as it's common to have a firewall blocking that communication (in which case PyDev wouldn't even be able to get common builtins such as len, object, etc).

On the good side, this also makes it possible for PyDev to analyze .pyd modules (although if you're developing such a module as a part of your project, you have to remember to call Ctrl+2, kill so that PyDev will kill those shells before you actually build it, otherwise that module will be locked and you won't be able to link it -- and tokens wouldn't be updated).

PyDev 2.3.0 released

2011-12-15T12:24:00.000-08:00

PyDev 2.3.0 is out now.

See http://pydev.org for details :)

Preparing for PyDev 2.3.0

2011-12-13T17:54:00.001-08:00

The major changes in this version will be the introduction of a pep8 analysis (it must be enabled in window > preferences > PyDev > Editor > Code Analysis > pep8) and scope selection/deselection (Shift+Alt+Up/Down).

The internal Jython version that PyDev uses was bumped from 2.1 to 2.2.1 to support pep8.py -- Jython 2.5 was discarded as an option because it became too big for PyDev (the PyDev distribution with Jython 2.2.1 has 7MB, if it went to Jython 2.5, it'd be around 14MB, so, I think it's not worth making the download twice as big just for this upgrade).

Also, as a plus, as Jython 2.2.1 is still based on the same grammar structure (JavaCC) that PyDev uses, so, I was able to give it a boost and provide some performance improvements that had been done to the PyDev grammar (Jython 2.5 uses Antlr, which seems to be one of the main reasons it became so big).

As a nice side-effect of this change, the PyDev startup seems to have improved somewhat (Jython 2.2.1 seems to start up faster than Jython 2.1), and now the Jython sources are included in the PyDev project (and not only a jython.jar), so, I can fine tune Jython itself for some PyDev use-cases (as it seems PyDev won't go for Jython 2.5 anyways).

There are also some critical bug-fixes in this release (an out of memory error when dealing with PyOpenGL, one cache issue which could make the pythonpath become wrong in memory and one deadlock were fixed).

So, anyone interested should get the current nightly build to see if all is working properly as an official release should be issued later this week (the current version should be the one released unless some critical issue is found in the meanwhile).