Fresh from the plantation

Surprise features that you didn't ask for - Mike Haworth

Sun, 14 Sep 2014 17:18:40 +1200

Server Side Request Forgery

Could access services on localhost
Access other hosts in DMZ
Bypass host-based authentication systems

Vulnerable systems:

memcached
couchdb

Urllib: accepts file://

Pycurl accepts gopher://

Memcached: gopher://127.0.0.1:11211/2get

Preventing: blacklist is error prone.

XML External Entity (XXE):

Safer: http://pypi.python.org/pypi/diffusedxml

Signed cookies - session data is visible to user.

Also unless it’s got discount applied, just reapply discount. Can have a unique number that is issued each time we give discount.

Don’t use pickle as could execute shellcode but instead use JSON.

Pentesting challenges: http://tinyurl.com/h4ckit

Automated Deployment with Ansible - Juergen Brendel

Sun, 14 Sep 2014 16:29:26 +1200

Configuration - system packages, settings, users, apps

CM Tools - describe the desired state. E.g. ensure all system packages are updated.

Puppet, chef - feel bit like working with Java.

Salt, Ansible - simple.

Ansible - orchestration engine. Written in Python. Uses YAML to write “playbooks”. Can use it to configure Windows.

No central configuration server/client (on target).

Just needs SSH access and Python on target machine.

Modules - 100s of them. They know how to do stuff.

Deletes module code from target after running it on target.

Can define groups to operate against.

group_vars directory/ all, groupname1

Ansible ‘cloud’ modules: AWS, Docker.

Modules: all output to stdout + JSON.

https://github.com/jbrendel/ansible_cluster_setup

Lightning Talks

Sun, 14 Sep 2014 16:00:47 +1200

LCAuckland

LCA: 12-16 January 2015

Program online in next year or 2.

Singapore also put in proposal for future LCA.

6-700 attendees hoped.

Heritage Preserve #WeWantJam

Grabbed 30TBs of .co.nz internet.

2 days of catering, hacking.

If you’d like to do it again tweet #WeWantJam

Less Boilerplate

import argparse

Command line

http://github.com/aliles/begins

Command line programs for busy developers.

Show us the world

rbenv - isolate different

pyenv shell 3.4.1

pyenv virtualenv name

pyenv local dirname

MySQL performance schema

Debugging schema for the database.

Can configure at runtime.

5.5 or 6 dbs.

Julia

Very pythonic. Stolen lots of good ideas from Python.

Compiled just in time. Timing a file that does nothing it takes 2.25s.

The Python Promotion Pamphlet

International Python Promotion Pamphlet. Glossy & stylised.

Creating NZ-version. Will have high-quality printable version and web-version.

Make your company more visible to Python programmers rather than pay recruitment agents.

Blackbox

http://sheltered-forest-9460.herokuapp.com

http://github.com/tuxbert/blackbox

NZPUG

When you subscribe to the mailing list (607 members on mailing list) you aren’t a member of the incorporated society.

Jessica McKellar’s Kiwi PyCon73,000 views. Far more than Guido McV’s keynote at PyCon US.

Thomi is commandeering LT and announcing my leaving committee and asking if someone can join NZPUG committee. See committee members if would like to join.

Docker for Python

Will do to IT industry what container box did to shipping industry.

LXC (Linux Lightweight Containers) + UnionFS.

Docker file is essentially like a bash script file of commands.

You can inherit from other docker files.

Fig - Vagrant for Docker.

Ship deltas not images.

Shipyard - 3rd party ecosystem.

Gotchas

Use docker-osx, PyCharm

Object Factories

Sole object is to create objects for testing purposes.

Don’t care about what object is. Just assign random attributes to objects.

Code Obfuscation

Weird bugs: object() > object()

If doing tests against time it will fail at midnight depending on your timezone.

False = True

Arduino + Thermal Printer + Python

Arduino sketch, Python script, Git

http://github.com/nick-NZ/Arduino-pull-request-printer

Dynamically creating Python tutorials and presentations

Massey Computer Science moved to Python in 2011.

Emacs: an operating system disguised as an editor.

http://www.orgmode.org

Org-babel - generate code.

Org-slidy

Bad-ass Postgres Tricks

http://bit.ly/pgtrix1

Template Databases - clone dbs
Estimated Counts
Hstore - values have to be strings
Smart Indexes
Schema Change Locks
PL/Python

“One button” test and deploy on AWS

Use Vagrant+Ansible

Scripted creation of AMI and deployment.

Python Antipatterns

Better: list comprehension

Better: raise AssertionError

Better: if pattern in input_str

Don’t pass empty list as default are.

if 5 <= i < 10: print “something”

Better: return NotImplementedError

Map reduction using Python scripts

Script to create ArcMap maps.

Making weird maps with Python

Pulled data from census + meshblock boundaries (Koordinates.com) + map box (tile mill) + GDAL.

Should have used Fiona + Shapely.

Multimedia programming using Gstreamer (and, of course, Python) - Douglas Bagnall

Sun, 14 Sep 2014 13:58:31 +1200

Pipelines consist of linked elements.

Elements:

Link
Source
Sink (file, net, screen, etc)
Demux, tee

$ gst-launch-1.0 filesrc location=video.ogv ! oggdemux ! theoradec ! autovideosink

playbin - just play file = mplayer

Probably about 100 media players in world that are based on gstreamer

Muxer: combining 2 streams into 1 file.

Really good example - video wall.

Decode bin -> split into 4 streams (tee) -> queues (often freezes as all running in 1 thread and bump into eachother) -> crop (I.e. Remove 3/4s of video). And also play audio.

Can just use gstreamer cli syntax when calling gi calls in Python.

github: douglasbagnall

http://gstreamer.freedesktop.org

Understanding human language with Python - Alyona Medelyan

Sun, 14 Sep 2014 13:30:23 +1200

Android Auto: hands-free operation via voice commands.

WordLense: “augmented reality translation”

“Her” movie in reality: “Siri will you marry me? Siri doesn’t seem to be available.”

Segmentation complexities.

Disambiguation complexities. E.g. Flying planes can be dangerous.

Categories - taxonomy terms, entities - biological/chemical, sentiment.

NKTK - Python toolkit for NLP.

How to get to the core words? Remove stopwords.

from nltk.corpus import stopwords from nltk.tokenizie import word_tokenize

Keyword scoring: TFxIDF

Gensim:

from gensim import corpora, models

When ranking words can use score to discard them.

Text Categorisation with NLTK

Sentiment Analysis:

from textblob import TextBlob

http://deeplearning.net/software/theano

http://textblob.readthedocs.org

http://scikit-learn.org/stable

http://nltk.org

http://radimrehurek.com/gensim

http://github.com/zelandiya/KiwiPyCon-NLP-tutorial

PyPy.js: What? How? Why? - Ryan Kelly

Sun, 14 Sep 2014 11:51:47 +1200

@rfkelly

Mozilla - protecting & promoting the web.

The Web? Technology + Culture (open - don’t need to submit to ‘web pool’, ubiquitous, secure, trustworthy)

“For Mozilla, anything that the Web can’t do, or anything that the Web is not faster and better at than native technologies, is a bug. We should file it in our Bugzilla system, so we can start writing a patch to fix it” - Andreas Gal (Mozilla CEO)

var vm = new PyPyJS() import js js.eval()

An experiment in building a fast, compliant, in-browser Python environment.

PyPy.js is … Not so Fast. Incompatibilities between the JIT compilers.

Is … Humongous.

How? PyPy + Emscripten.

Emscripten: an LLVM backend that generates JavaScript.

“It’s awful all the way down”

Alts: http://GitHub.com/PythonJS

“Please, file bugs against the Web”

http://pypyjs.org

http://github.com/rfk/talk-pypyjs-what-how-why

Deploying a Django application using Juju - Tim Penhey

Sun, 14 Sep 2014 11:11:29 +1200

Or “How I used Juju to deploy my Django app up to AWS”

Used backbone.js

Juju is written in Go. Service orchestration in the cloud.

Charms:

Encapsulate application configurations
Define how services get deployed

Existing Charms

PostgreSQL
Python-Django
Gunicorn

Juju can help with scaling later.

Simple repeatable deployments.

Writing a subordinate charm.

Hooks are just scripts. Used Install hooks and Everything else hooks.

Promoted charms are only ones that are officially available.

“As a Juju core developer it was interesting that if I didn’t know the value that Juju would give me later, I would’ve thrown it out ages ago.”

Insufficient documentation around using the Python-Django charm, had to read the code.

Insufficient documentation around how to write a payload charm for Python-Django, very much work it out as you go.

Python-Django and gunicorn still needs nginx to be really useful.

Should have a full stack Django charm.

juju bootstrap|deploy|add-relation|deploy|expose

2nd Keynote: A Snake in Space - The rise of scientific Python in Astrodynamics and Astronomy - Francesco Biscani

Sun, 14 Sep 2014 10:37:00 +1200

Max Planck Institute for Astronomy, European Space Agency.

PaGMO and Astropy - will mainly talk about PaGMO as wrote this and more familiar with this.

PaGMO - Parallel Global Multiobjective Optimiser. Optimisation tool. Initially a trajectory optimisation tool, evolved as a general-purpose optimiser.

Focused on parallel and distributed computing. Can use via C++ or Python.

Astropy - community effort to develop a single core package for Astronomy.

Optimisation - a large area of applied mathematics. “The selection of a best element (with regard to some criteria) from some set of available alternatives.” E.g. Travelling salesman (TSP).

E.g. algorithms: gradient-based methods, evolutionary algorithms, stochastic algorithms.

Interplanetary trajectories - space mission trajectories are defined by sets of parameters: launch date, initial velocity vector, sequence of flybys, sequence of deep-space manoeuvres (DSM).

Usually we want to minimise fuel consumption.

The resulting optimisation problem is hard: multimodal objective function, highly nonlinear, highly dimensional.

Traditionally tackled by teams of human experts.

Bio-inspired algorithms:

genetic algorithms
differential evolution
ant-colony optimisation
artificial bee-colony optimisation

Island model - name inspired by Darwin’s trip to the Galápagos Islands.

History of PaGMO - pattern of scientific programming code: created as part of some research and then abandoned for many years; not useable by anyone else and then picked up later and made more consumable by others.

Initially created Python bindings to initially created C/C++ ‘research’ code (2008-2009) and followed 'eat own dog food’ approach by using it a lot for internal research. Been through 2 GSoCs and now 'fully-fledged’ open source project.

Pros of Scientific Computing

Emphasis on correctness & reproducibility
Powerful driver for innovations in HPC

Cons of Scientific Computing

Wheel re-invention
Code is often written with a one-paper-horizon mindset
Most scientists are not trained in software engineering

PaGMO framework:

The abstract island class includes a problem, an algorithm and a population of candidate solutions, and a virtual evolve() method that dispatches the optimisation (to a thread or a process on another machine)

Implemented via Boost.Python.

Challenging, non-trivial issues:

serialisation across language boundaries involving virtual classes, base pointers, etc
extension from Python of C++ base classes
working around some of Python’s limitation wrt parallel programming (GIL)

Leveraging Python’s strengths:

Scientific Python stack: NumPy (crunching results), SciPy (optimisation algorithms), matplotlib, IPython, etc

PaGMO uses: evolution of neural networks for autonomous Martian rovers, selection of Near Earth Asteroids for future human missions.

Astropy - “a community package for Astronomers”:

Handle practical needs of astronomers: units, coordinates, FITS files, cosmological calculations
“One of best community packages ever seen”
Not research package like PaGMO
Heavy reliance on NumPy

Java for Python Developers - Chris Neugebauer

Sat, 13 Sep 2014 17:01:53 +1200

One company who likes Python but in reality likes Java a lot more = …. Google!

.java -> [compiler] -> JVM ByteCode -> JVM

Jython - setuptools - make .jars

from java.Lang.system

What can do in Python 2.7 can also do in Jython.

Android devices outsold full-size computers 2x.

Dalvik Virtual Machine = Android Runtime (ART)

.java -> [compiler] -> .dex (Dalvik exe) -> DVM

No dynamic 3rd-party libraries. All statically compiled into Dalvik exe.

Java Native Interface (JNI).

Kivy

Python-For-Android: Cpython API for Java.

from jnius import autoclass

PyJnius runs on Android!

JNI method signatures are a nightmare.

If Java’s more important, use Jython! Support for C extensions is not complete.

If Python’s more important, use PyJnius!

Jython is corporate sponsored by Sun.

2.7 support on Jython is only recent.

Kivy doesn’t have a very native look-and-feel. They say they’re working on this but “they don’t know what they’re talking about”.

Toga aims to use the native APIs for theming but currently has no Android support.

Intro to flask-security - Beau Butler

Sat, 13 Sep 2014 16:25:15 +1200

Why Flask? Steep Django learning curve. Like to start simple and then become more complex.

App example code: http://github.com/oddy/bangingminimal

Don’t get Async mail sending, Pretty forms.

You get: sess auth + RBAC, base users db, all std workflows, passwd crypto, token/JSON auth.

pip install flask flask-security flask-SQLAlchemy pyBcrypt paste

Google: Prove Domain ownership - use button option.

Black box testing: Know lots of hackers.

Need motivate hackers - apply drink.

How hacked? ARP Redir (had PC on same subnet), SSLSplit, click forgot password, intercept reset URL, reset URL.

Actual hack - other PC pretending to be Google SMTP server (on same subnet) intercepted program call and passed call on to Google SMTP server and hacker’s app.

Py smtplib doesn’t check Google’s cert! 2.x smtplib can’t do it at all.

Admin Ronacher FTW.

Flask-Sec RBAC itself still not broken.

Have a go: pwnoddy.com.