paradoxxx zero

wdb 3.0 material style

Wed, 08 Jun 2016 00:00:00 +0000

The future is here! wdb 3.0 is out get it while it’s hot (and certainly bugged)

Changes

A whole new material design lite interface (a bit responsive)
Clickable icons instead of hotkeys only
Visual distinction between stepping, post_mortem and shell
An actually readable help (you can get by clicking on the help icons)
A code mirror prompt with syntaxic coloration and more classic completion
New commands (open current file in external editor, respawn current process…)
A far better post mortem interaction when using ext and wdb is disabled.
importmagic suggestions on NameError
Use of uncompyle6 to get python source when only byte-code is available

For more information about wdb, see previous post

Announcing butterfly 2.0

Fri, 30 Oct 2015 00:00:00 +0000

After 1 year and a half of waiting butterfly 2.0 is finally here!

For those who don’t know butterfly is an open source in-browser nix terminal written in coffeescript (yeah I know, maybe I will port it to ES7 but you know… braces) and backed up by a python tornado server.

Features

Its feature list keep on growing:

xterm compatible (support a lot of unused features!)
Native browser scroll and search
Theming in css / sass (18 preset themes) endless possibilities!
HTML in your terminal! cat images and use <table>
Multiple sessions support (à la screen -x) to simultaneously access a terminal from several places on the planet!
Secure authentication with X509 certificates!
16,777,216 colors support!
Keyboard text selection!
Desktop notifications on terminal output!
Geolocation from browser!
May work on firefox too!

Give it a shot

It’s even easier to try it now, just:

    $ pip install butterfly
    $ pip install libsass  # If you want to use themes
    $ butterfly

A new tab should appear in your browser. Then type

    $ butterfly help

To get an overview of butterfly features.

For everyday use, it’s better to use a server see below.

Moar

Go to the github page for more informations, like setting up a server using systemd (I know I know) or why not contributing!

Take a look at existing themes at butterfly-themes and current demos showing the butterfly potential at butterfly-demos.

End

We are in 2015, we can finally display images and html in a terminal and use 16M colors, what a world we are living in.

Cutter

Wed, 13 May 2015 15:04:42 +0000

Today I finally wrote a readme for my little python tool cutter.

This is something that I use daily in conjuction of wdb for debugging python code.

Here’s the readme, I hope you will find it as useful as I am:

Readme

Cutter is a small python module that add some sugar on top of list like objects to ease list traversal.

Usage

from cutter import cut

numbers = [
  ['one', 'two', 'three'],
  ['un', 'deux', 'trois'],
  ['uno', 'due', 'tre']
]

>>> cut(numbers)[1]  # Take the second element of every sublist
['two', 'deux', 'due']*

>>> cut(numbers)[1, 2]  # Take the third letter of every second element of every sublist
['o', 'u', 'e'].

>>> cut(numbers)[1][2]  # Idem
['o', 'u', 'e'].

>>> cut(numbers)[1, 2].upper()  # Upper case the third letter of every second element of every sublist
['O', 'U', 'E']

>>> cut(numbers)[...]  # Flatten the list for one level
['one', 'two', 'three', 'un', 'deux', 'trois', 'uno', 'due', 'tre']

>>> cut(numbers)[..., 0]  # First letter of every elements
['o', 't', 't', 'u', 'd', 't', 'u', 'd', 't'].

Syntaxic sugars

There are two syntaxic sugars for an even easier use:

The |_ syntax

from cutter import _

>>> numbers |_ [1]
['two', 'deux', 'due']*

>>> (numbers |_ [1] |_ [2] |_ .upper)()
['O', 'U', 'E']

>>> numbers |_ [...]
['one', 'two', 'three', 'un', 'deux', 'trois', 'uno', 'due', 'tre']

>>> numbers |_ [..., 0]
['o', 't', 't', 'u', 'd', 't', 'u', 'd', 't'].

The ! syntax

This syntax is meant for use in shells.

It is for example used in wdb and the bundled cut.py interpreter.

Cutter provide a bang_compile function which is a wrapper of the python builtin compile function.

# This code muste be compiled with cutter.utils.bang_compile

>>> numbers!1
['two', 'deux', 'due']*

>>> numbers!1!2!upper()
['O', 'U', 'E']

>>> numbers!*
['one', 'two', 'three', 'un', 'deux', 'trois', 'uno', 'due', 'tre']

>>> numbers!*!0
['o', 't', 't', 'u', 'd', 't', 'u', 'd', 't'].

This syntax use the python tokenizer and ast to make it work. This is really useful when debugging to inspect list content.

Cutter works with dictionaries too:

cut(dict)['key']

slices:

cut(list)[:5]

For more examples see the test files : test

Cutter is compatible with at least: python 2.6, 2.7, 3.2, 3.3, 3.4 and pypy and is licensed under lgpl v3

wdb 2.1 new features and a new clean theme

Thu, 07 May 2015 00:00:00 +0000

Hi everybody!

wdb 2.1 is here with a new more serious light theme:

and some new features:

New completion mechanism, should complete a lot better
Experimental object tree search (.f command) to look for a key or a value condition in an object
Media display in debugger (.i command)
New diff function to print differences between string / obj / files (.x command)
Timing for expression evaluation (remote and local)
Prompt exception inspection (click on it)
A new shell mode and an executable wdb
A new multiline prompt mode triggered by [ctrl] + [enter]
And a lot of bug fixes

For more features see the last wdb 2.0 post and the github readme page.

wdb 2.0 the python debugger in your web browser

Mon, 12 May 2014 00:00:00 +0000

I released wdb 2.0 this morning with a lot of fixes and features added.

wdb debugging itself (colors are awful due to gif compression)

Introduction

For those who don’t know the project, wdb is a python debugger like pdb, but with a slick web front-end and a lot of additional features, such as:

Source syntax highlighting
Visual breakpoints
Interactive code completion using jedi
Persistent breakpoints
Deep objects inspection using mouse
Multithreading / Multiprocessing support
Remote debugging
Watch expressions
In debugger code edition
Popular web servers integration to break on error
In exception breaking during trace (not post-mortem) in contrary to the werkzeug debugger for instance
Breaking in currently running programs through code injection (on supported systems)

This project started because I wanted to improve the werkzeug debugger and it ended up as a whole project.

wdb supports python 2.6, 2.7, 3.2, 3.3, 3.4 and pypy.

Try it

Enough with the introduction, let’s just try it.

    $ pip install wdb.server      # Install both debugging client and server
    $ wdb.server.py &             # Start the server as a background process
    $ python -m wdb your_file.py  # Start debugging your_file.py

With the last command, your default web browser should open with the debugger paused at the first executable line of your file. You can get some help by typing .h in the prompt. But for the basic commands:

.s or [Ctrl] + [↓] or [F11] : Step into
.n or [Ctrl] + [→] or [F10] : Step over (Next)
.r or [Ctrl] + [↑] or [F9] : Step out (Return)
.c or [Ctrl] + [←] or [F8] : Continue
.u or [F7] : Until (Next over loops)
.h : Get some help
.q : Quit

While the server is running, you can access the following url localhost:1984/ to have an overview of currently opened debugging sessions, global breakpoints and running python process (which you may pause see here for more information).

Wdb can also be used just like pdb, by putting:

    import wdb
    wdb.set_trace()

anywhere in your code, where you want it to pause.

Upgrade from v1.x

To upgrade to wdb 2.0 you need to run:

    $ pip install wdb.server
    $ pip install --upgrade wdb

This is because now the package is split between the server and the debugging client to avoid unneeded dependencies installed in virtualenvs or other python versions (once the server is installed and running you just have to install wdb in your virtualenvs).

For more detailed instructions and other features please read the README.md on the github page: github.com/Kozea/wdb

As always, you are more than welcome to contribute to the project (GPLv3), by adding new features, creating themes or supporting me financially see about page (Encouraged!)

butterfly now with ssl authentication

Fri, 21 Mar 2014 00:00:00 +0000

First of all thanks for all your feedback on butterfly, I never expected this to have such success (wow). Google Analytics says about 50k visits in 2 days scattered among 164 countries. That’s a lot and this led to a fair amount of constructive criticism (and obviously a huge amount of destructive criticism too).

Security

Security was a major topic among all discussions and therefore several security breaches were fixed. That’s why starting with the new 1.4.0 release, the SSL secure mode will now be enabled by default. This works by creating a certificate and a client authentication file (More info).

This is a boring process so I added some options in butterfly to make your life easier.

NB: Before going into the details, be aware that the previous behavior is still available using the --unsecure option.

Use butterfly > 1.4.0

As you launch butterfly.server.py you will see that it can’t start without generating certificates. So let’s make them!

Generating certificates files

In this example I suppose you run the server as root, you want to access it as user foo on a local network and your ip address on this network is 192.168.0.1:

$ sudo butterfly.server.py --generate-certs --host="192.168.0.1" # Generate the root certificate for running on local network
$ sudo butterfly.server.py --generate-user-pkcs=foo              # Generate PKCS#12 auth file for user foo

If you run as root the certificates will be placed in /etc/buttefly/ssl/, as user in ~/.butterfly/ssl.

Once generated, you will have to configure your browser to accept your self signed certificate and the PKCS#12 file (which contains a private key for your user to log in).

Importing the root certificate in Google Chrome

Under Google Chrome go to Settings > Show advanced settings… > HTTPS/SSL > Manage certificates… > Authorities > Import… and specify the butterfly_ca.crt file.

Importing the root certificate in Firefox

Under Firefox go to Edit > Preferences > Advanced > View Certificates > Authorities > Import… and specify the butterfly_ca.crt file.

Importing the user client authentication file

For the PKCS#12 you will have to change the file ownership before importing it since this file is a secured file:

$ sudo cp /etc/butterfly/ssl/foo.p12 /tmp # Copy it in /tmp
$ sudo chown foo /tmp/foo.p12             # Make it readable to the browser

Then go to the Your Certificates tab (for both Google Chrome and Firefox) and import it from /tmp. If you set a password during the creation you will have to type it otherwise leave it blank. Some browsers don’t seem to appreciate blank passwords, if this doesn’t work retry --generate-user-pkcs with a password.

$ sudo butterfly.server.py --host="192.168.0.1" # Run the server

Now point your browser at https://192.168.0.1:57575/ and you are done without typing any password (The authentication informations are contained in the PKCS#12 file).

You will have to run --generate-certs every time you want to use a different --host, the PKCS#12 will still work though since both host and user files comes from the same CA.

Other browsers

For Safari under Mac OS X, you will have to go to the Keychain Access application to import the certificate and the user PKCS#12 files. Sadly Safari seems to have some issues with WebSockets and self-signed certificates and I didn’t manage to make it work but I’m far from being an Apple tech and if anyone has succeeded in making it work I’m really interested in your procedure, otherwise file a bug to Apple. It works with the previous browsers in Mac OS though.

For other platforms, search for the browser / OS specific way to import certificates and pray for X509 client auth support. If it’s unavailable, use the --unsecure option and use ssh tunnels.

Changes

Since the previous post, a lot of effort has been put into making butterfly safer by removing the HTML escapes for now and by checking websocket origin but also into making it run on more operating systems (FreeBSD, Mac OS, Cygwin, …).

Incoming changes are listed on the github page. Please feel free to discuss them.

Have fun and if you want to support my free work go to the about page (Encouraged!)

butterfly your everyday terminal in your web browser

Fri, 28 Feb 2014 00:00:00 +0000

There’s something I’ve been working on recently, at start just for fun but it turned out I use it everyday, so here it is:

butterfly your everyday terminal right in your browser

Version 2.0 has been released please see here

Try it

To try it, it’s as simple as:

$ sudo pip install butterfly      # Install butterfly
$ butterfly.server.py --unsecure  # Launch the web server unsecure

Then all you have to do is go to : localhost:57575

Use it

Server side

Run at start-up with systemd

As it relies on a server running in background, you can download the systemd file from here butterfly.service put it in /etc/systemd/system/ or equivalent and:

$ sudo systemctl enable butterfly
$ sudo systemctl start butterfly

Now the server will always be running.

Running as root and multiple users

butterfly is made to work as the terminal user and as root. When running as root the privileges are dropped to the user that initiated the connection, i.e. the user that opened the web page. You can change the terminal user by going to localhost:57575/user/<user>.

Access from a remote machine

You can set the bind host with butterfly.server.py --host="0.0.0.0" which will allow other users to connect to your terminal. A password will be asked but IT IS NOT SECURE! So it’s recommended as of now to run this only on local network for testing purposes.

Run a specific shell

Use the shell command line option: butterfly.server.py --shell=/bin/fish

Client side

butterfly works like any other terminal but it has some specific features.

Enter browser shortcuts

When the terminal is focused all the hot-keys are caught (except from those the browser prevent to catch). If you want to access the address bar with CTRL + L it won’t work. That why the ALT + Z combination is here to tell butterfly that the next key combination must not be caught. In other word accessing the address bar is done by doing ALT + Z then CTRL + L. (If your browser support css3 filters the screen will turn sepia).

Quick selection of history text

This is a feature I’ve always wanted to find in a terminal, quick history selection. I’ve started to implement some ideas you can try by hitting CTRL + SHIFT + up and then navigating the selection with CTRL + SHIFT + up/down and the arrow keys. When you got it right, hit ENTER and butterfly will enter the selection right into the prompt:

Other features

The visual style is done using css so it’s entirely editable.
You can easily extend the in-browser terminal behavior by writing javascript (like it’s done for the quick selection).
As apparently it’s trendy to use a “modern” text editor based on a browser, so…

Missing features

The scroll-back is actually limited to 100 000 lines as of now in order to limit the memory footprint. It would be a good idea to store the scroll-back on the disk using the server.
Embedding html directly into the terminal is possible and edit: not supported anymore with the ESC]99;<html>BELL command for security reasons.
16.7 millions of color is not yet supported but it’s one hack away using the ESC[38;2;r;g;bm escape.
An in-terminal css/js editing options with why not a package repository for theme and extensions.

That’s all

The repository is hosted on github.

So let me know what do you think of it, I had a lot of fun writing it anyway.

wsreload 2.0 technical details

Mon, 02 Dec 2013 00:00:00 +0000

Following to the last post about wsreload, I will now explain how I implemented the reload mechanism.

Most existing browser reloaders use dirty X hacks like xdotool or xvkbd to send the [F5] key to the browser window. This doesn’t work reliably. No really. There’s also other solutions, like launching your browser with a debug backdoor (--remote-shell-port) but it kinda mess up everything else.

So to avoid these dirty hacks wsreload ships with a browser extension which role is to tell the browser to reload the page directly from the extension API with the added bonus of bypassing the cache.

But this does not solve the shell <-> browser communication problem, for that I used a python websocket server: the well known tornado server. The server is used to relay reload requests from the command line client to the listening extension in the browser all through websockets.

It looks like this:

As you can see the server is also used to watch over file modification using pyinotify.

For instance if you have told wsreload to watch over your /tmp directory and to reload all the http://localhost:1234/* pages with the following command:

$ wsreload --watch /tmp --url "http://localhost:1234/*"

a inotify watcher will be added to the /tmp directory with the url query attach to it.

When a file change in /tmp, inotify will tell the server and the server will send the corresponding query through all the connected websockets (opened by the browser extensions).

All the extensions will receive the query and try to match any open tab with it, reloading the one matching.

NB: The extension is also able to tell the server to watch for files when a url match the file:// prefix.

That’s all for now.

Ask me if you have any question :)

wsreload 2.0

Fri, 29 Nov 2013 00:00:00 +0000

Today I released wsreload 2.0 which is IMHO a neat tool to increase productivity.

NB: As it depends on inotify, it’s only available on linux (if you are interested in making it work on other OS feel free to make a pull request :D)

NB: As of now it is only compatible with google chrome / chromium. Nothing prevents it to work under other browsers though, except from the fact that someone has to port the chrome extension code to his preferred browser again : pull request :)

Reloading your tabs

wsreload has one aim: making you forget about the [F5] key.

file:// reloading

Every local file opened in your browser with a file:// scheme will be automatically synchronized with the file.

Selective tabs reloading

But that’s not all of it, wsreload comes with a handy executable: wsreload which can reload any tab matching a criteria when a file is modified.

For example, I’m currently using it while writing the draft of this blog post.

The following command:

$ wsreload --watch _site/ --url "http://localhost:4000/*"

tells the browser to reload all tabs that match the following url pattern : http://localhost:4000/* whenever a file in the _site/ directory is modified. (This is the directory in which jekyll generates this site.) With it my draft preview is automatically reloaded every time I save the markdown file (in conjunction with jekill serve -w which regenerates the site when a file change).

The wsreload command comes with several matching criteria:

$  wsreload --help

Reload all tabs matching query through websocket

optional arguments:
  -h, --help            show this help message and exit
  --active ACTIVE       Whether the tabs are active in their windows.
  --pinned PINNED       Whether the tabs are pinned.
  --highlighted HIGHLIGHTED
                        Whether the tabs are highlighted.
  --status {loading,complete}
                        Whether the tabs have completed loading.
  --title TITLE         Match page titles against a pattern.
  --url URL             Match tabs against a URL pattern.
  --windowId WINDOWID   The ID of the parent window, or
                        chrome.windows.WINDOW_ID_CURRENT for the current
                        window.
  --windowType {normal,popup,panel,app}
                        The type of window the tabs are in.
  --index INDEX         The position of the tabs within their windows.
  --delay DELAY         Delay to wait before refreshing.
  -H HOST, --host HOST
  -P PORT, --port PORT
  -E ENDPOINT, --endpoint ENDPOINT
  --watch WATCH [WATCH ...]
                        Tell the server to watch files and reload according to
                        the other options
  --unwatch UNWATCH [UNWATCH ...]
                        Tell the server to unwatch the following files

These rules follow the chrome extension api tabs query criteria

Installing it

To forget about the F5 key you have to install the wsreload library like any other python package:

$ pip install wsreload

Then start the server using:

$ wsreload-server&

or if you are using systemd:

$ wget https://github.com/paradoxxxzero/wsreload/raw/master/wsreload.service
$ sudo mv wsreload.service /etc/systemd/system
$ sudo systemctl enable wsreload
$ sudo systemctl start wsreload

And finally just install the wsreload chrome extension like any other chrome extension. That’s it !

In a following post I will write about the technical details behind all of this.

Prelude

Sun, 24 Nov 2013 15:24:42 +0000

Hello there and welcome to my brand new site paradoxxxzero.github.io.

I’ve decided to build up this site in order to write about my personal and professional projects and some less serious stuffs.

I hope that you will enjoy reading it.