tag:blogger.com,1999:blog-8401274931342779902013-01-27T06:39:43.364-02:00Scientific adventures with Python.Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.comBlogger66125PythonInSciencehttp://feedburner.google.comtag:blogger.com,1999:blog-840127493134277990.post-78970170205473955282012-12-26T20:15:00.003-02:002012-12-26T20:15:53.777-02:002012-12-26T20:15:53.777-02:00

GeoJSON can get quite big when you need represent complex maps with lots of polygons. Even though it compresses well, being text, it can still be quite a hassle to store and upload to a website in uncompressed form. Recently while working on a web app which requires the upload of maps in GeoJSON format, I stumbled upon Google App Engine's limitation of 32MB for POST requests. At that point I

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com0http://pyinsci.blogspot.com/2012/12/compressing-geojson-with-python.htmltag:blogger.com,1999:blog-840127493134277990.post-30360935610416832542012-04-01T10:55:00.000-03:002012-04-01T10:55:40.119-03:002012-04-01T10:55:40.119-03:00

Natural Language Processing (NLP). A marvelous world of possibilities! Fortunately it is also a great example of another domain of application for which Python is wonderfully well equipped. I have been playing with Python and NLP, for a couple of years now, integrating its tools on a reasonably large project. I hope to demo this project really soon, but it is not the topic of this post. The

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com4http://pyinsci.blogspot.com/2012/04/benchmarking-nltk-under-pypy.htmltag:blogger.com,1999:blog-840127493134277990.post-39044009372792508152011-10-10T10:08:00.000-03:002011-10-10T10:08:27.101-03:002011-10-10T10:08:27.101-03:00

I have been deeply involved in text mining problems lately, If you watched my presentation a EuroScipy this year (http://www.slideshare.net/fccoelho/mining-legal-texts-with-python) you have an idea of some of the things I am up to. Well part of problem of text mining is to get hold of the text you want to analyze in the first place. For many projects I am involved with, I already have mountains

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com7http://pyinsci.blogspot.com/2011/10/scrapy-google-scholar-and-mongodb.htmltag:blogger.com,1999:blog-840127493134277990.post-7282570629396406502011-09-05T07:05:00.000-03:002011-09-05T07:05:23.078-03:002011-09-05T07:05:23.078-03:00

Yesterday I had to scratch a little annoying itch. I had to embed a javascript library (MathJax) into a PyQt project of mine(Model-Builder). Embedding static files into Qt projects is done via resource files. I was surprised that Qt Designer resource editor has no way to recursively scan directories. You have to add files manually. That was not an option for me since MathJax is somewhat

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com1http://pyinsci.blogspot.com/2011/09/generating-qt-resource-files-from-whole.htmltag:blogger.com,1999:blog-840127493134277990.post-86639416212452976692010-12-28T07:06:00.005-02:002010-12-28T07:42:21.703-02:002010-12-28T07:42:21.703-02:00

In my previous post, some readers pointed out that the pure Python version of the code was slower than it should be. I checked and found out that the timing was wrong due to some bug in the %time flag in the sage notebook.Some other interested readers pointed out that using numpy's RNGs in the pure Python version would sure improve the performance. Again I went back and tested it.So without

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com4http://pyinsci.blogspot.com/2010/12/efficient-mcmc-in-python-errata-and.htmltag:blogger.com,1999:blog-840127493134277990.post-55998343570267958572010-12-25T19:59:00.014-02:002010-12-26T09:34:08.282-02:002010-12-26T09:34:08.282-02:00

I have recently come across a very interesting blog. That is, interesting for people like me, who are enthusiasts of Statistics, programming and their combination expressed preferably in the Python language. One recent post of that blog, caught my attention, as it offered a comparison between MCMC implementations of a simple Gibbs sampler in multiple languages, namely: R, Python, Java and

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com7http://pyinsci.blogspot.com/2010/12/efficcient-mcmc-in-python.htmltag:blogger.com,1999:blog-840127493134277990.post-82915125185921600682010-10-22T10:07:00.039-02:002010-10-27T11:19:43.836-02:002010-10-27T11:19:43.836-02:00

Model-Builder is a niche educational software and the first "largish" Python application I ever wrote, way back in 2001. Back then, WxPython seemed like the easiest way to develop a GUI application inPython, mainly due to the automation provided by the IDE Boa-constructor.A lot has changed since then and for many years now, my favorite GUI Toolkit has been Qt via it's Python binding, PyQt. For a

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com0http://pyinsci.blogspot.com/2010/10/model-builder-road-to-10.htmltag:blogger.com,1999:blog-840127493134277990.post-79526770280700603732010-08-22T17:21:00.009-03:002010-08-22T18:51:57.679-03:002010-08-22T18:51:57.679-03:00

Consegi 2010 is over. It was a great meeting! I was happily suprised to realize how far the adoption of free software has come in the last few years. Congratulations to the Free Software community for such a successful lobby!Big thanks are due to Giuseppe (@gsromag) , Luiz Guilherme (@aldabalde) and their colleagues from SERPRO, who where responsible for the or ganization of this amazing

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com0http://pyinsci.blogspot.com/2010/08/python-in-clouds.htmltag:blogger.com,1999:blog-840127493134277990.post-28525678787188172552010-07-15T09:09:00.007-03:002010-07-15T10:57:16.975-03:002010-07-15T10:57:16.975-03:00

I have been invited to give a talk on the 3rd International Congress on Free Software and E-Government -- Consegi, taking place in Brasilia on 18-20 of august. It is a massive event bringing together Leading professionals from all over the world. Last year's event had 6000 participants from 19 countries!This year's theme is cloud computing and I'll bee talking about Google AppEngine (GAE). Since

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com0http://pyinsci.blogspot.com/2010/07/consegi-2010.htmltag:blogger.com,1999:blog-840127493134277990.post-11636893741840754572010-06-03T05:38:00.001-03:002010-06-03T05:38:31.835-03:002010-06-03T05:38:31.835-03:00

Just came across this interesting python library for machine learning. People interested in this should also checkout Peach: http://code.google.com/p/peach/acerca de: http://pybrain.org/pages/features (ver no Google Sidewiki)

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com0http://pyinsci.blogspot.com/2010/06/pybrain.htmltag:blogger.com,1999:blog-840127493134277990.post-6022128314691659712010-04-19T14:04:00.004-03:002010-04-19T14:24:30.497-03:002010-04-19T14:24:30.497-03:00

I recently blogged about Liveplots. I have just released another version with a very important improvement. Before, even though plot server was running on a separate process, there was still a small overhead for the calling code since the plotting function had to be evaluated (and data piped to Gnuplot) before the execution returned to your simulation. To improve matters, I have implemeted

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com0http://pyinsci.blogspot.com/2010/04/new-liveplots-release.htmltag:blogger.com,1999:blog-840127493134277990.post-8368896554866564112010-04-15T10:40:00.008-03:002010-04-15T11:13:47.084-03:002010-04-15T11:13:47.084-03:00

I have recently released a package I had been using privately for a while. It's called Liveplots and is a package which provides the tools for monitoring computationally intensive, long running numerical simulations. I am aware that similar tools for Java, like Livegraph. But I needed something that was light, simple, and in Python.Liveplots provide a daemonized plot server (i.e. running as a

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com4http://pyinsci.blogspot.com/2010/04/liveplots-package.htmltag:blogger.com,1999:blog-840127493134277990.post-89871216839797534902010-04-05T12:25:00.008-03:002010-04-05T13:31:20.740-03:002010-04-05T13:31:20.740-03:00

I am working on a new version of Model-Builder, which will incorporate Uncertainty analysis features based on my BIP package.One key aspect of doing uncertainty analysis, is the ability to attribute prior distributions to the model's parameters, and programatically, this is easily accomplished through the use of scipy.stats random number generators which provide a large variety of both continuous

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com4http://pyinsci.blogspot.com/2010/04/automatic-parameterization-of-scipys.htmltag:blogger.com,1999:blog-840127493134277990.post-38619245882117932262010-03-18T12:30:00.004-03:002010-03-18T12:37:55.101-03:002010-03-18T12:37:55.101-03:00

This may sound like a weird thing to do, but actually, I have craved for something like that every time I have to read C code. Converting C code to Python, can not only help us understand code more easily, but also turn non perfomance-critical code easier to maintain. C can be easy enough to read if well-written and formatted (indented) adequately, however having to move back and forth between .h

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com8http://pyinsci.blogspot.com/2010/03/converting-c-code-to-python.htmltag:blogger.com,1999:blog-840127493134277990.post-75402223536437447312010-02-26T11:16:00.001-03:002010-02-26T11:16:43.280-03:002010-02-26T11:16:43.280-03:00

Interesting post, But hoping to invert those proportions, i.e., making the smarter guys the 80% is harder that having a society where everyone earns more money than they can spend... Pretty next to impossible...;-)in reference to: Coding Horror: The Two Types of Programmers (view on Google Sidewiki)

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com1http://pyinsci.blogspot.com/2010/02/utopia.htmltag:blogger.com,1999:blog-840127493134277990.post-11238282413501279682009-12-14T08:27:00.001-02:002009-12-14T08:27:42.594-02:002009-12-14T08:27:42.594-02:00

I am currently working on a presentation where I have to include a lot of mathematical graphs, so instead of plotting using matplotlib and inserting the result as a figure, I am considering using specialized LaTeX packages to create the graphs. While searching for documentation, I found this great repository of examples for the Tikz package.referente a: http://www.texample.net/tikz/examples/ (ver

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com5http://pyinsci.blogspot.com/2009/12/latex-tikz-examples.htmltag:blogger.com,1999:blog-840127493134277990.post-45476375435594041722009-11-19T06:10:00.004-02:002009-11-19T06:45:28.533-02:002009-11-19T06:45:28.533-02:00

I have been so busy lately that I almost forgot to write about my lightning venture into writing a Google Wave robot: Trendy.For those of you living under a rock in the last several months, Google Wave offers an API for the development of robot, which are... well, robots, which when added to waves, do automated tasks with its contents. There are robots for translating, do syntax highlighting on

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com0http://pyinsci.blogspot.com/2009/11/dveloping-wave-robot.htmltag:blogger.com,1999:blog-840127493134277990.post-34308924665125801522009-10-22T09:05:00.001-02:002009-10-22T09:05:55.295-02:002009-10-22T09:05:55.295-02:00

I have recently come across an interestng problem while working on my random variable implementation for BIP. Any type in Python is expected to have a __str__(self) method which returns an adequate and expressive string representation of the object. Well, as far as I could think, the most straightforward representation of a random variable is its probability distribution. Probability

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com8http://pyinsci.blogspot.com/2009/10/ascii-histograms.htmltag:blogger.com,1999:blog-840127493134277990.post-12758953385080743302009-09-29T10:11:00.001-03:002009-09-29T10:11:56.793-03:002009-09-29T10:11:56.793-03:00

This document is a must read (see link at the end).I want to add my own items to it:1. Net Neutrality is not only protecting Internet content provider corporations' profitsWe need to defend net neutrality in a way which goes beyond what is currently done: We need to establish fixed ip addresss for every private individual so that publishing rights don't have to be gatekeeped by large corporations

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com3http://pyinsci.blogspot.com/2009/09/internet-manifesto.htmltag:blogger.com,1999:blog-840127493134277990.post-24873396637085225362009-09-28T06:49:00.001-03:002009-09-28T06:49:09.725-03:002009-09-28T06:49:09.725-03:00

I recently came across, this interesting, opensource Python Scientific Distribution for Windows. I normally don't pay too much attention to windows tools, but it's good to have something to recommend to windows users when you want them to try out some Python code. For Linux users, it's not really relevant, because we all have powerfull package managers to help us get most Python packages

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com3http://pyinsci.blogspot.com/2009/09/pythonxy-scientific-python-distribution.htmltag:blogger.com,1999:blog-840127493134277990.post-74324761789699520202009-09-24T07:51:00.001-03:002009-09-24T07:51:55.879-03:002009-09-24T07:51:55.879-03:00

I have just created a Scientific Python Group at LinkedIn. I was actually surprised that there wasn't one already.Python is quickly becoming a major tool in scientific computing and we should all do our best to advertise its capabilities.referente a: Scientific Python Group | LinkedIn (ver no Google Sidewiki)

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com0http://pyinsci.blogspot.com/2009/09/scientific-python-group-at-linkedin.htmltag:blogger.com,1999:blog-840127493134277990.post-34325216219671144192009-09-17T11:41:00.008-03:002009-09-17T12:24:15.501-03:002009-09-17T12:24:15.501-03:00

One of the things I sorely missed from matplotlib for a very long time, was a violin plot implementation. Many a time, I thought about implementing one myself, but never found the time.Today, browsing through Matplotlib's documentation, I found the recently added fill_betweenx function. Finally it seemed to have become a piece of cake to implement a violin plot. I Googled for violin plot and

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com8http://pyinsci.blogspot.com/2009/09/violin-plot-with-matplotlib.htmltag:blogger.com,1999:blog-840127493134277990.post-19396017552985518302009-09-11T11:52:00.002-03:002009-09-11T11:55:40.267-03:002009-09-11T11:55:40.267-03:00

I recently came across a repository of jewels in the form of the unpublished manuscripts of E.W. Dijkstra. I just finished reading one entitled "Some Meditations on Advanced Programming". It is amazingly well written and still so relevant to present-day computer science, that I recommend anyone with at least a passing interest on the subject to read it. I was pleasantly surprised to find in the

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com1http://pyinsci.blogspot.com/2009/09/why-python.htmltag:blogger.com,1999:blog-840127493134277990.post-35671869647426705722009-09-02T10:27:00.003-03:002009-09-02T10:49:34.864-03:002009-09-02T10:49:34.864-03:00

The recommended new way (since 2.6) of doing string formating in Python is to use the format method of string objects instead of the % operator and %s (and variants) placemarks.I decided to check it out to generate some LaTeX tables programatically. Bad Idea. the method expects placemarks like this: "some string {key}".format(key=123) and that is a big problem when formatting LaTeX strings.

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com5http://pyinsci.blogspot.com/2009/09/new-style-string-formatting-and-latex.htmltag:blogger.com,1999:blog-840127493134277990.post-7805975184304197762009-07-10T16:28:00.002-03:002009-07-10T16:56:40.765-03:002009-07-10T16:56:40.765-03:00

The web platform is starting to get annoying in its attempt to re-invent and re-create everything that previously existed in the Desktop space.Finally, Desktop developers are beginning to wake up to this problem.For years, browser-developing companies have tried to turn the Browser into an operating-system-like autonomous system. While the rest of us were trying to figure out how to get our

Flavio Coelhohttps://plus.google.com/106488325672572942691noreply@blogger.com11http://pyinsci.blogspot.com/2009/07/chrome-os-and-google-world-domination.html