Andrey Zhukov's Tech Blog

I'm moving to Substack

Sat, 03 Aug 2024 09:30:00 +0000

You can subscribe here.

Technical posts with code examples stays on this site.

5 reasons I'm not buying the new iPad Pro M4 (2024)

Sat, 22 Jun 2024 11:30:00 +0000

I have iPad Pro 11" since 2020. I use it daily. Here is why I don’t upgrade to the new M4 version.

3 upstream errors in Nginx ingress with Gunicorn/Flask as backend in Kubernetes environment

Tue, 28 May 2024 10:00:00 +0000

Recently I faced 3 types of upsteam errors. It was like 2% of all requests, but it was quite annoying.

How to plan a day as Indie Developer

Tue, 14 May 2024 10:00:00 +0000

This year 1st May was Wednesday. I took another 2 days from my 9-5 job and 2 days from the weekend. The goal was to build a schedule as if I were an indie developer. The main project was to develop an idea for a new iOS app.

8 macOS productivity apps in 2024

Sun, 28 Apr 2024 10:00:00 +0000

Several apps that help to be more focused and organized.

12-Week Review - Q1/2024

Sun, 14 Apr 2024 10:00:00 +0000

For accountability, I decided to publish 12-week reviews. In a month there is not always something big happening, so I start with a year quarter. Maybe I’ll transform it into a newsletter later. I’ll publish it as a YouTube video too.

Protocol buffers organization in Python microservices

Wed, 20 Mar 2024 10:00:00 +0000

In this post one of the approaches to organizing Protobuf files in microservices architecture using gRPC.

How to move to a new Mac without Migration Assistant

Thu, 07 Mar 2024 10:00:00 +0000

Last summer I got a new MacBook Pro M2 and decided to go with a fresh install. With my previous MacBooks, I used Migration Assistant and it worked well. But this time it was a switch from Intel and I wanted to clean unnecessary files, applications, and settings.

Here is the list of files I moved and apps I installed.

How to Overcome Procrastination: A Mindset Shift for Productivity

Fri, 09 Feb 2024 06:30:00 +0000

As responsible individuals, we often find ourselves prioritizing others over our own goals. If you’ve ever struggled to make progress on a personal project, this mindset change might be the key to unlocking your productivity.

Essential Python tools for writing quality code

Sat, 27 Jan 2024 06:30:00 +0000

Writing quality code is essential for both personal and team projects. It helps to reduce the number of bugs, makes the code more readable and maintainable, and improves the overall quality of the project.

In this article some basic libraries that I use in everyday life.

2023 Achievements

Thu, 11 Jan 2024 06:30:00 +0000

This blog has been on hold since 2020, and every year I’m thinking of restarting it. This year is no exception.

The stopping factor was that it’s the technical blog. Yet, I write a lot during journaling and when preparing YouTube videos. I don’t know if I’ll be able to transform them into blog posts, but I’ll try.

How to replace azure-storage

Wed, 25 Jan 2023 11:30:00 +0000

Continue of previous task.


from azure.storage.blob import BlockBlobService, ContentSettings, PublicAccess
BlockBlobService(
                account_name=cluster_config["config"]["AZURE_STORAGE_ACCOUNT"],
                account_key=cluster_config["config"]["AZURE_STORAGE_KEY"],
            )
storage.create_container(container_name, public_access=PublicAccess.Blob)
        storage.create_blob_from_bytes(
            container_name, blob_name, content, content_settings=ContentSettings(content_type=content_type) 
block_blob_service.create_blob_from_stream(
                folder_name, filename, content, content_settings=ContentSettings(content_type=mimetype)
            )


from azure.storage.blob import ContainerClient, ContentSettings, PublicAccess
account_name = cluster_config["config"]["AZURE_STORAGE_ACCOUNT"]
        account_key = cluster_config["config"]["AZURE_STORAGE_KEY"]
        account_url = f"https://{account_name}.blob.core.windows.net"
        container_clients.append(
            ContainerClient(account_url=account_url, credential=account_key, container_name=container_name)
if not container_client.exists():
            container_client.create_container(public_access=PublicAccess.Blob)  # exception, or overwrite true
        container_client.get_blob_client(blob_name).upload_blob(
            content, content_settings=ContentSettings(content_type=content_type)            , overwrite=Tru

How to replace oauth2client with google-auth and google-auth-oauthlib

Tue, 17 Jan 2023 11:30:00 +0000

Recently I made an update for some legacy code and one of the tasks was to replace oauth2client. This library used to interact with OAuth2-protected resources related to Google API and deprecated since September 2018. The recomendation is to use google-auth and for OAuth 2 flow google-auth-oauthlib. The changes are simple.

How to use asyncio gRPC in AioHTTP microservices

Sun, 23 Jan 2022 11:30:00 +0000

Usually I have a lot of microservices and they communicate using REST API. For communication with frontend I still need REST, but between microservices gRPC looks more promising.

Some benefits of it:

is built on HTTP2, using multiplexed streams - better connection management
performance benefits because of Protobuf binary format
code generation for a multilingual environment

Here is my first attempt to use gRPC in AioHTTP applications with some explanations.

Celery queues monitoring in Datadog

Tue, 07 Sep 2021 11:30:00 +0000

If you have Redis integration in Datadog, there is an easy way to add monitoring for celery queues.

How to send celery task from async code

Mon, 16 Aug 2021 11:30:00 +0000

The more simple way is to add an endpoint to the service with celery. Here is an example of a low level way based on Message Protocol.

IQUNIX A80 mechanichal keyboard - first impressions

Tue, 06 Jul 2021 11:30:00 +0000

It’s my first mechanical keyboard and it’s about a year that I’m interesting to try it. I like my apple keyboard, but there are so many reviews for the mechanical ones, so finally I gave up and bought this IQUNIX A80.

My 5 use cases for iPad

Sun, 27 Jun 2021 11:30:00 +0000

I bought iPad 11" one year ago and I had a lot of doubts about it usefulness, it’s not one of the cheapest devices. If you are in the same position, maybe this post will help you.

How to replace celery banner with one log line

Thu, 17 Jun 2021 11:30:00 +0000

If you, like me, suffer from celery banner in your logs, but still want to see useful information from it.

Blog performance after a year of inactivity

Wed, 09 Jun 2021 06:30:00 +0000

It’s almost a year that I did not post anything to this blog and here is a chart from Google Analytics.

How to update a script with azcopy v7 to v10

Thu, 02 Jul 2020 11:30:00 +0000

Here is the previous version of the script to transfer containers between storages. AzCopy v10 has totally different parameters, in fact, it’s another utility that does the same thing. Instead of --source, --dest parameters, it now requires a SAS token. #!/bin/bash CONTAINER_NAME=$1 SOURCE_STORAGE_NAME=$2 SOURCE_STORAGE_KEY=$3 TARGET_STORAGE_NAME=$4 TARGET_STORAGE_KEY=$5 source_exists=$(az storage container exists --name $CONTAINER_NAME --account-name $SOURCE_STORAGE_NAME --account-key $SOURCE_STORAGE_KEY --output tsv) if [ $source_exists != "True" ]; then echo "Source container does not exist.

Calling python async function from sync code

Sat, 06 Jun 2020 10:00:00 +0000

This is an example of how to transform sequential requests to parallel in some legacy code with asyncio loop when the refactoring is not an option.

To show the idea I’m using a simplified version of microsoft graph api requests to get user manager by user ids.

VSCode settings and plugins

Sat, 30 May 2020 10:00:00 +0000

It’s almost two years when I switched to VSCode from IntelliJ IDEA. The last license expired in September 2018. Sometimes I switched back to IDEA when there were problems with Python Language Server or pytests discovering, but it did not last long. The IDEA is good, it works well, but it’s too heavy for me, even if I do not restart it often. With VSCode I still have sometimes problems with “Go to Definition”, but most of the time it solves with a simple reload, which takes just several seconds.

Settings

Mostly I’m using the settings by default, here are some changes:

How to set environment variables for pytest in VSCode

Sat, 23 May 2020 10:00:00 +0000

To run tests I usually use database with tmpfs volume, sometimes it makes them more than twice faster. To achieve this I run the database on another port and use env variable in application config, like this docker run -d --name=mongo_test -p 27018:27017 --tmpfs /data/db mongo:4.2.3 MONGODB_URI=mongodb://localhost:27018/test_db pytest --cov=app tests The configuration for env variables in VSCode is simple, you need to place .env file into your project and add "python.

RabbitMQ with AioHttp example

Sun, 17 May 2020 10:00:00 +0000

In this example I have two services:

auth which sends an event that user is authenticated
users which listen for this event and update user’s last online time.

These services use aio_pika library for async RabbitMQ connection and topic exchange for the message routing.

How to run several docker-compose projects with their traefik in each on the same server

Wed, 22 Apr 2020 11:00:00 +0000

The configuration is simple, for each project a docker-compose project name and port should be defined. It can be done with .env files. COMPOSE_PROJECT_NAME=project1 PORT=8081 Docker compose will use COMPOSE_PROJECT_NAME variable as a prefix for the names of the containers. Then in docker-compose.yml, these variables can be used to define the port mapping and the traefik constraints. services: traefik: image: "traefik:v2.1.7" command: ... - "--providers.docker.constraints=Label(`com.docker.compose.project`, `${COMPOSE_PROJECT_NAME}`)" ports: ... - "${PORT}:80" This can be used to run different projects or several instances of one project to test different branches.

Traefik dashboard on another port and with authentication

Tue, 14 Apr 2020 11:00:00 +0000

Here is an example for Traefik dashbord on port 9090 and with basic auth middleware. To generate password: echo $(htpasswd -nb user password) | sed -e s/\\$/\\$\\$/g For services routers it’s important to set entryPoints with correct port: "traefik.http.routers.ui.entryPoints=web". version: '3.3' services: ui: image: "ui:v1.0.0" restart: always labels: - "traefik.enable=true" - "traefik.http.routers.ui.entryPoints=web" - "traefik.http.routers.ui.rule=Host(`${HOST_NAME}`)" - "traefik.http.services.ui.loadbalancer.server.port=8080" - "traefik.http.services.ui.loadbalancer.server.scheme=http" traefik: image: "traefik:v2.1.7" restart: always command: - "--providers.docker=true" - "--providers.docker.exposedbydefault=false" - "--entrypoints.web.address=:80" - "--entrypoints.

How to clone MongoDB collection indexes from one server to another

Mon, 23 Mar 2020 11:00:00 +0000

from pymongo import MongoClient def clean_indexes(indexes): """Remove index for id and clean ns attribute.""" for index in indexes: if index['name'] == '_id_': continue index.pop('ns', None) yield index def get_indexes(db): """Get map collection name -> list of cleaned indexes.""" coll_indexes_map = {} for coll in db_source.list_collections(): coll_name = coll['name'] coll_indexes_map[coll_name] = list(clean_indexes(list(db[coll_name].list_indexes()))) return coll_indexes_map def create_indexes(db, coll_indexes_map): for coll_name, indexes in coll_indexes_map.items(): if indexes: db.command('createIndexes', coll_name, indexes=indexes) def clone_indexes(source_uri, dest_uri): source_db = MongoClient(source_uri).

A better Celery ping for Docker healthcheck

Fri, 13 Dec 2019 11:00:00 +0000

In the previous post the command is celery inspect ping --app app.celeryapp -d worker@$$HOSTNAME, but it initialize the app on each ping and can be expensive. The lightweight version is celery inspect ping -b "redis://redis:6379/0" -d worker@$$HOSTNAME.

AsyncIO REST example with aiohttp, motor and umongo

Thu, 15 Aug 2019 10:00:00 +0000

Not so long I started to use aiohttp to create small services, they are 2-5x times faster.
Here is my example of such an application with CRUD operations for items stored in MongoDB: https://github.com/sneawo/asyncio-rest-example. It’s not production-ready, but there is an example of how to run it with docker-compose.

Mock aiohttp request in unittests

Wed, 22 May 2019 10:00:00 +0000

Here is an example for the python 3.7.3 and the next versions of libs aiohttp==3.5.4 asynctest==0.12.3 from typing import Dict from aiohttp import ClientSession from aiohttp.web_exceptions import HTTPInternalServerError async def send_request() -> Dict: async with ClientSession() as session: async with session.get('http://localhost') as response: if response.status != 200: raise HTTPInternalServerError() return await response.json() from asynctest import patch, CoroutineMock from aiohttp import web from aiohttp.web_exceptions import HTTPInternalServerError from aiohttp.test_utils import AioHTTPTestCase, unittest_run_loop from .

A script to wait for a service running in docker swarm

Tue, 30 Apr 2019 10:00:00 +0000

For docker stack deploy there is no option to wait while all services are in running state. With docker service ps it’s possible to find desired state and current state for one service for each node. $ docker service ps redis ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 50qe8lfnxaxk redis.1 redis:3.0.6 manager1 Running Running 6 seconds ago ky2re9oz86r9 \_ redis.1 redis:3.0.5 manager1 Shutdown Shutdown 8 seconds ago 3s46te2nzl4i redis.

Alfred workflow step with bash script in background

Sun, 21 Apr 2019 08:00:00 +0000

I spent some time to find this answer, so I’ll share it also here. <path>/script.sh > /dev/null 2>&1 & Without the output redirections Alfred will wait for the background process to complete and the next steps will not happen.

Nginx proxy with prefix

Fri, 29 Mar 2019 10:00:00 +0000

It can be done this way. location /my_app { rewrite ^/also/?(.*)$ /$1 break; proxy_pass http://my_app_upstream; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; ... } Without rewrite it will pass /my_app to upstream, so prefix should be handled on my_app level. location /my_app { proxy_pass http://my_app_upstream; And with / in upstream it will add an additional /, so urls in proxied service will look like //some-url location /my_app { proxy_pass http://my_app_upstream/;

How to connect to a service in docker swarm with docker

Fri, 15 Mar 2019 10:00:00 +0000

It’s a rare case, suppose there is a myapp swarm cluster with myapp_mongo database without published port and there is a need to run a command from some docker image with connection to this database. By default docker stack deploy creates a non-attachable network, so docker run --network myapp_default will output Error response from daemon: Could not attach to network myapp_default: rpc error: code = PermissionDenied desc = network myapp_default not manually attachable.

Get blinker signal's receivers names

Thu, 07 Mar 2019 10:00:00 +0000

from unittest import TestCase from app.main import init_app from app.signals import my_signal class SignalTests(TestCase): def setUp(self): init_app() def test_signal_is_connected_to_my_receiver(self) receivers = [r().__name__ for r in signal.receivers.values()] self.assertIn('my_receiver', receivers)

MongoDB dump and restore with docker

Thu, 14 Feb 2019 10:00:00 +0000

Here are two commands to take a partial dump of the collection from production database and put it in dev mongo instance running through docker-compose. docker run -v `pwd`/:/dump mongo mongodump --gzip --archive=/dump/my_collection.agz --host <connection url> --ssl --username <username> --password <password> --authenticationDatabase admin --db <prod_db> --collection my_collection --query '{date: {$gte: ISODate("2019-02-01T00:00:00.000+0000")}}' docker-compose run -v `pwd`/my_collection.agz:/my_collection.agz mongo mongorestore --gzip --archive=/my_collection.agz --host mongo --nsFrom <prod_db>.my_collection --nsTo <dev_db>.my_collection

Easy charts in python app with plotly

Thu, 24 Jan 2019 10:00:00 +0000

An example how to add a chart to Flask app with plotly library: def get_chart_data(): return mongo_db['item'].aggregate([ {'$sort': {'date': 1}}, {'$group': { '_id': {'year': {'$year': '$date'}, 'month': {'$month': '$date'}}, 'num': {'$sum': 1} }}, {'$project': { 'date': {'$dateFromParts' : {'year': '$_id.year', 'month': '$_id.month'}}, 'num': '$num' }}, {'$sort': {'date': 1}} ]) def get_chart(): data = list(get_chart_data()) layout = plotly.graph_objs.Layout(title='Items by month') scatter_data = [ plotly.graph_objs.Scatter( x=[d['date'] for d in data], y=[d['num'] for d in data] ) ] fig = plotly.

Mongoengine as_pymongo performance

Fri, 18 Jan 2019 10:00:00 +0000

When you need to get only several fields from the list of complex objects, it works much faster with as_pymongo function. In my situation I have a 16x increase: %timeit list(SomeObject.objects.scalar('id')[0:500]) # 129 ms ± 11 ms per loop %timeit list(o['_id'] for o in SomeObject.objects.scalar('id')[0:500].as_pymongo()) # 7.98 ms ± 849 µs per loop

Celery Execution Pools

Fri, 11 Jan 2019 11:30:00 +0000

A good explanation is in this post Celery Execution Pools: What is it all about?.

Blog Results 2018

Thu, 03 Jan 2019 11:30:00 +0000

The goal was the same as in 2017 to write 52 posts, one per week. It’s not achived, the are 32, less than 45 in 2017.

How to find the document with maximum size in MongoDB collection

Fri, 28 Dec 2018 11:00:00 +0000

The command to get document size is Object.bsonsize. The next query is to find the document in a small collection, cause it can be slow: db.getCollection('my_collection').find({}).map(doc => { return {_id: doc._id, size: Object.bsonsize(doc)}; }).reduce((a, b) => a.size > b.size ? a : b) To do this faster with mongo mapReduce: db.getCollection('my_collection').mapReduce( function() { emit('size', {_id: this._id, size: Object.bsonsize(this)}); }, function(key, values) { return values.reduce((a, b) => a.size > b.size ? a : b); }, {out: {inline: 1}} )

How to find number of MongoDB connections

Tue, 18 Dec 2018 11:00:00 +0000

From the MongoDB side the current connections can be found with db.currentOp() command. Then they can be grouped by client ip, counted and sorted. var ips = db.currentOp(true).inprog.filter( d => d.client ).map( d => d.client.split(':')[0] ).reduce( (ips, ip) => { if(!ips[ip]) { ips[ip] = 0; } ips[ip]++; return ips; }, {} ); Object.keys(ips).map( key => { return {"ip": key, "num": ips[key]}; } ).sort( (a, b) => b.num - a.num ); The result will be like this:

Maximum number of client connections in Flask-SocketIO with Eventlet

Tue, 11 Dec 2018 11:00:00 +0000

It’s not mentioned in the docs for Flask-SocketIO that Eventlet has an option max_size which by default limits the maximum number of client connections opened at any time to 1024. There is no way to pass it through flask run command, so the application should be run with socketio.run, for example: ... if __name__ == '__main__': socketio.run(app, host='0.0.0.0', port='8080', max_size=int(os.environ.get('EVENTLET_MAX_SIZE', 1024)))

How to split Celery tasks file

Wed, 05 Dec 2018 11:00:00 +0000

Suppose there is a large tasks.py file, like this: @celery.task() def task1(): ... @celery.task() def task2(): ... task1.delay() ... ... A good idea is to split it on the smaller files, but Celery auto_discover by default search tasks in package.tasks module, so one way to do this is to create a package tasks and import tasks from other files in __init__.py. __init__.py from .task1 import task1 from .task2 import task2 __all__ = ['task1', 'task2'] task1.

CSRF exempt for Flask-RESTPlus API

Wed, 14 Nov 2018 11:00:00 +0000

The @csrf.exempt method does not work with Resource methods or decorators, it should be done on Api level. Here is an example how to exclude resources from CSRF protection based on class: def csrf_exempt_my_resource(view): if issubclass(view.view_class, MyResource): return csrf.exempt(view) return view api_blueprint = Blueprint('api', __name__) api = Api(api_blueprint, title='My API', decorators=[csrf_exempt_my_resource]) Or for all resources: api_blueprint = Blueprint('api', __name__) api = Api(api_blueprint, title='My Private API', decorators=[csrf.exempt])

Function as model column in Flask Admin

Thu, 18 Oct 2018 11:00:00 +0000

I like the idea about separation of business logic from models and views into services. There are more details in these slides from the EuroPython talk. Usually in Flask-Admin for a new column a new method with @property decorator is added and the model becomes fat. Also it’s not good to put something with queries in property. Another way is to put this function to the services and use column formatter in Flask-Admin.

Docker healthcheck for Flask app with Celery

Tue, 25 Sep 2018 11:00:00 +0000

With docker-compose it can be done the next way flask_app: ... healthcheck: test: wget --spider --quiet http://localhost:8080/-/health celery_worker: ... command: celery worker --app app.celeryapp -n worker@%h --loglevel INFO healthcheck: test: celery inspect ping --app app.celeryapp -d worker@$$HOSTNAME Where /-/health is just a simple route @app.route("/-/health") def health(): return 'ok' Update: A better Celery ping for Docker healthcheck

How to update docker service using a version from docker-compose

Tue, 18 Sep 2018 10:30:00 +0000

Here is a small script which compares the version of the running service and the version in the docker-compose file, if they are different it runs an update. REDIS_VERSION=$(docker service ls | grep "redis" | awk '{print $5}') REDIS_NEW_VERSION=$(grep -Po "image:\s*\Kredis:.*" docker-compose.yml) if [ "$REDIS_VERSION" != "$REDIS_NEW_VERSION" ]; then echo "update $REDIS_VERSION -> $REDIS_NEW_VERSION" docker service update --image $REDIS_NEW_VERSION app_redis fi

Gitlab CI configuration using Docker socket binding

Tue, 11 Sep 2018 10:30:00 +0000

To avoid rebuild of images on each run there is a possibility to use Docker socket binding. It’s covered in the documentation, here is a more detailed example.

EuroPython 2018

Wed, 08 Aug 2018 10:30:00 +0000

This year I finally visited EuroPython, thanks to my company. Here is a list of some interesting talks with the links to youtube: Die Threads – video Reliability in distributed systems – video Trio: a pythonic way to do async programming – video What’s new in Python 3.7 – video AsyncIO in production – video When to use machine learning: tips, tricks and warnings – video Django structure for scale and longevity – video Debugging your code with data visualization – video Getting started with mypy and type checking – video Citizen science with Python – video Good features beat algorithms – video Domain-driven design patterns in Python – video

Env variables for Vue.js in Nginx container

Fri, 27 Jul 2018 13:00:00 +0000

The default approach for Vue.js is to use .env files during build stage. But I like more this one with envsubst. There is no need to rebuild image for each environment or when the configuration is changed. Here is a small modification for entrypoint.sh, so it can replace all variables with VUE_APP_ prefix. #!/bin/sh function join_by { local IFS="$1"; shift; echo "$*"; } # Find vue env vars vars=$(env | grep VUE_APP_ | awk -F = '{print "$"$1}') vars=$(join_by ' ' $vars) echo "Found variables $vars" for file in /dist/js/app.

Auth0 authorization to a Python API with PyJWT

Tue, 17 Jul 2018 11:30:00 +0000

There is an example in the Auth0 quickstarts, but it uses python-jose-cryptodome and I already have PyJWT in the project.

So here is a small modification of this example.

A script to transfer azure storage containers to another storage

Thu, 05 Jul 2018 11:30:00 +0000

Here is an example of a script to copy azure storage containers from one storage to another and keep public access level. It uses Azure CLI and AzCopy tools. #!/bin/bash STORAGE_NAME_SOURCE="<source_account_name>" STORAGE_KEY_SOURCE="<source_account_key>" STORAGE_NAME_DEST="<dest_account_name>" STORAGE_KEY_DEST="<dest_account_key" CONNECTION_STRING_SOURCE="DefaultEndpointsProtocol=https;AccountName=$STORAGE_NAME_SOURCE;AccountKey=$STORAGE_KEY_SOURCE" CONNECTION_STRING_DEST="DefaultEndpointsProtocol=https;AccountName=$STORAGE_NAME_DEST;AccountKey=$STORAGE_KEY_DEST" containers=$(az storage container list --connection-string $CONNECTION_STRING_SOURCE --output tsv | awk '{print $2}') for i in $containers; do docker run --rm incendonet/azcopy azcopy --source https://$STORAGE_NAME_SOURCE.blob.core.windows.net/$i --source-key $STORAGE_KEY_SOURCE --destination https://$STORAGE_NAME_DEST.blob.core.windows.net/$i --dest-key $STORAGE_KEY_DEST --recursive access_level=$(az storage container show-permission -n $i --connection-string $CONNECTION_STRING_SOURCE --output tsv) az storage container set-permission -n $i --connection-string $CONNECTION_STRING_DEST --public-access $access_level done

Disable user loading for some routes for app using Flask-Security

Tue, 26 Jun 2018 11:30:00 +0000

An application with Flask-Security makes a request to the database for the authenticated user for each route, but sometimes it’s not necessary. Here is a way how to disable it with a decorator and custom UserDatastore.

MongoDB select fields after $lookup

Tue, 19 Jun 2018 11:30:00 +0000

When there is a $lookup stage to join a list of large documents, an error Total size of documents in ... matching ... exceeds maximum document size can arrive. It’s possible to avoid this with $unwind stage right after $lookup. More explanations in the documentation. And then the documents can be regrouped with the required fields. Order.objects.aggregate( { '$lookup': { 'from': 'item', 'localField': '_id', 'foreignField': 'order_id', 'as': 'items' } }, { "$unwind": "$items" }, { "$group": { "_id": "$_id", "date": {"$last": "$date"}, "items": { "$push": { "name": "$items.

How to find a change for a field with MongoDB aggregation

Tue, 12 Jun 2018 11:30:00 +0000

For example there is a collection device_status which stores the different states for the devices. The task is to find the devices which passed from off to on at least one time.

{ "device" : "device1", "state" : "on", "ts": ISODate("2018-06-07T17:05:29.340+0000") }
{ "device" : "device2", "state" : "off", "ts": ISODate("2018-06-08T17:05:29.340+0000") }
{ "device" : "device3", "state" : "on", "ts": ISODate("2018-06-09T17:05:29.340+0000")}
{ "device" : "device3", "state" : "shutdown", "ts": ISODate("2018-06-09T18:05:29.340+0000")}
{ "device" : "device2", "state" : "load", "ts": ISODate("2018-06-09T19:05:29.340+0000") }
{ "device" : "device2", "state" : "on", "ts": ISODate("2018-06-10T17:05:29.340+0000") }
{ "device" : "device3", "state" : "off", "ts": ISODate("2018-06-11T17:05:29.340+0000") }
{ "device" : "device1", "state" : "idle", "ts": ISODate("2018-06-11T18:05:29.340+0000") }
{ "device" : "device3", "state" : "on", "ts": ISODate("2018-06-12T17:05:29.340+0000") }
...

The first stage is to sort the data by device and date.

How to decode a Flask session or a CSRF token

Thu, 07 Jun 2018 11:30:00 +0000

I’m using this example. The only change I made is return_timestamp=True parameter to find when the session was generated.

Remote model in Flask-Admin

Wed, 30 May 2018 11:30:00 +0000

The task is to show a model from another application in Flask-Admin. A way to do it is to overwrite BaseModelView and get the data through HTTP request. Here is a simple example for a sortable list with a pagination.

Bullet Journaling vs Todoist

Mon, 30 Apr 2018 11:30:00 +0000

It’s about a year when I moved to Todoist from Wunderlist. And that was right, there are no new posts in the Wunderlist blog or in the Microsoft To-Do blog. On other side Todoist is updated regularly, my karma is growing and it works well. But there are some things, as for Wunderlist, that i’m missing: the week/month plan overview of previous week/month tracking for the repeating tasks So two month ago I decided to try Bullet Journal system and it works well for me.

Celery flower for several applications

Tue, 17 Apr 2018 11:30:00 +0000

If there is a need to monitor several applications, there are two ways: run celery flower instance for each application celery flower --app app1.celeryapp --port=5555 celery flower --app app2.celeryapp --port=5556 run celery flower with broker option, there will be less options to control tasks celery flower --broker=redis://redis/0

Celery checklist

Fri, 13 Apr 2018 11:30:00 +0000

There is a good checklist to build great celery async tasks. I want only to add how to autoreload celery worker in development mode. Before there was an --autoreload option, but now it’s removed. For this task I use watchdog. pip install watchdog watchmedo auto-restart --recursive -d app -p '*.py' -i '*.pyc' -- celery worker --app app.celeryapp --queues my_query -n my_queue@%h --loglevel INFO

Microsoft Graph Api UnknownError in batch request

Fri, 30 Mar 2018 11:30:00 +0000

It’s possible to optimize the usage of Microsoft Graph Api by batching requests. There is a limitation for 20 requests in one batch request.

Here is a simplified examle:

How to log user ip in Docker Swarm

Tue, 20 Mar 2018 11:30:00 +0000

There is a long standing issue for this task. The logs looks like: 10.255.0.2 - - [15/Mar/2018:15:32:15 +0000] “GET / HTTP/2.0” 200

One of the solutions, from the issue’s comments, is to run a service in the host mode, the swarm load balancing for this service will not work. But if there is a load balancer behind the swarm cluster, it seems not a big problem.

Google Adsense results

Tue, 06 Mar 2018 11:30:00 +0000

I added Adsense at the end of January. In February I got 1.6K users and €2.46 to my balance, a little more than expected.

Export and import for MongoEngine model in Flask-Admin

Fri, 16 Feb 2018 11:30:00 +0000

Another tip for Flask-Admin. The task’s requirements are:

possibility to choose some model’s objects and download them in JSON
possibility to upload them back

There is a ModelView property can_export, it adds an action to export in CSV or another format supported by tablib, but it does not allow to select records and there is no import. So for my task it’s not a solution.

The export is easy to do with an action decorathor.

How to disable field in MongoEngine subdocument in Flask-Admin

Thu, 08 Feb 2018 11:30:00 +0000

In the documentation there is an information about form_widget_args and form_subdocuments attributes of ModelView class. To do this task they can be used together:

An update for the productivity apps list

Tue, 06 Feb 2018 11:30:00 +0000

Several productivity apps I added to this list in the last month. HazeOver - automatically highlights the front window by fading out all the background windows. Hazel - watches whatever folders you tell it to, automatically organizing your files according to the rules you create. Bartender - lets you organize your menu bar apps. iStatMenus - adds system monitor to your menu bar. For iTerm2 I failed to setup Automatic Profile Switching, but I found a good solution here - Change terminal color when SSH from OS-X.

Notebook stand as a solution for the back pain

Thu, 01 Feb 2018 11:30:00 +0000

The health is one of the most important part of our life and the programming work affects it a lot. For the years I had a pain in my back, sometimes less, sometimes more, at the beginning it was a low back pain, now it’s upper back.

The reasons in my case are:

Google Adsense and blog on subdomain

Fri, 26 Jan 2018 11:30:00 +0000

This month I decided to try Google Adsense, just as an experiment. I expect no more 1-2$ in the next month. The process of adding adsense is easy, but the problem is that it requires a primary domain and on the primary domain sneawo.com I had a redirect to blog.sneawo.com. So the first attempt failed after 2 days of waiting. For a second attempt I put an html with a short text on the primary domain.

A bootstrap for a microservice based on Flask with MongoDB

Thu, 18 Jan 2018 11:30:00 +0000

Starting a new project is a common task in microservices architecture. To do this it’s better to have a some template. I put my version to the flask-mongoengine-bootstrap repository. The key point are: very basic, only flask, flask-mongoengine and structlog in requirements configuration through environment variables configured logging in JSON format marking log records with request_id possibility to run development version make dev and tests make test through docker a template for Makefile examples of model, api route and tests If you’ll use it, do not forget to change SECRET_KEY.

Subscription to the blog updates

Fri, 12 Jan 2018 11:30:00 +0000

This year I decided to spend some time on the blog marketing. I don’t know how to do it, but as a first step I’ve added a subscription to the monthly digest of updates. You can see it on the right. I used MailChimp because: it’s free up to 2,000 subscribers and 12,000 emails per month there is a subscription form builder there is an automatic campaign to send an email by schedule with the posts from RSS The only problem was to choose an physical address, it’s a requirement for FTC’s CAN-SPAM Act.

Blog Results 2017

Tue, 02 Jan 2018 11:30:00 +0000

One of the goals for 2017 year was to reanimate the blog and write 52 posts, one per week.

No-Cache headers in Flask

Wed, 20 Dec 2017 11:30:00 +0000

The second line of “The Zen of Python” (try import this) says “Explicit is better than implicit.”. By default Flask response does not return any cache header and you can suppose that browsers will not cache, but some of them can decide for you. So it’s better to provide them, especially for API and it’s easy: @app.after_request def set_response_headers(response): response.headers['Cache-Control'] = 'no-cache, no-store, must-revalidate' response.headers['Pragma'] = 'no-cache' response.headers['Expires'] = '0' return response

Flask-Admin authentication through oAuth

Wed, 13 Dec 2017 11:30:00 +0000

Flask-Admin provides the examples of authentication with Flask-Login and Flask-Security. Here is an example how to authenticate using Microsoft oAuth.

Create an application in Microsoft Application Registration Portal.

Problems with Gmail in default Postfix configuration

Wed, 06 Dec 2017 12:30:00 +0000

Recently I moved some of the sites to a new hosting. The previous server had Ubuntu 12.04 with Exim4 and I forgot how to configure them.

On a new server I installed Postfix with Ansible:

    - name: Set up Postfix to relay mail
      debconf: name=postfix
               question='{{ item.question }}'
               value='{{ item.value }}'
               vtype='{{ item.vtype }}'
      with_items:
        - { question: 'postfix/mailname', value: '{{ ansible_fqdn }}', vtype: 'string' }
        - { question: 'postfix/main_mailer_type', value: 'Internet Site', vtype: 'string' }

After that I started to receive the next delivery reports from Gmail:

Flask-BabelEx translations in a Celery task

Wed, 29 Nov 2017 12:30:00 +0000

The problem with translation in a Celery task is that there is no request context and flask_babelex.get_locale returns None. For Flask-Babel there is a force_locale context manager which solves this problem. But Flask-Security uses Flask-BabelEx and it does not work well with Flask-Babel.

Here is an example how to create a context manager which provides dummy request context.

Flask-RESTPlus similar resources for MongoEngine model

Tue, 21 Nov 2017 12:30:00 +0000

Suppose there are several similar MongoEngine models and there is a need to modify them through REST. Here is an example how to create a set of such similar resources.

Alfred Workflow to add an email from Mail.app to Todoist inbox

Thu, 16 Nov 2017 13:30:00 +0000

There is a way to add an email to Todoist by sending it to a special address, the only thing that annoing me, that it doesn’t keep a link to the message in Mail.app. I created a simple Alfred workflow, which takes the selected messages in Mail.app, creates a Markdown links as [email subject](message://message_id) and send it to Todoist Inbox using its API. You can find the details about installation and usage in sneawo/mailapp-to-todoist repository.

Socket.IO in Docker Swarm

Tue, 07 Nov 2017 13:30:00 +0000

Socket.IO requires the sticky sessions, so when it’s runned in Docker Swarm environment, the Swarm load balancer can forward the request to any node where the service is launched and there will be the 400 errors. The solution is to limit the number of nodes for Socket.IO service to one or to use another load balancer like Traefik or HAProxy.

Dump and restore commands for PostgreSQL in Docker

Fri, 03 Nov 2017 13:30:00 +0000

Sometimes during development it’s necessary to share the db dump with colleagues. In the previous article i provided the commands for MySQL, here is the same for PostgreSQL. DATE = $(shell date +%Y-%m-%d) path ?= . help: ## show this help @echo 'usage: make [target] ...' @echo '' @echo 'targets:' @egrep '^(.+)\:\ .*##\ (.+)' ${MAKEFILE_LIST} | sed 's/:.*##/#/' | column -t -c 2 -s '#' dump: ## dump db, usage 'make dump path=/path/to/dumps' docker-compose exec --user postgres db pg_dumpall --clean | gzip > $(path)/project-$(DATE).

Django in Docker

Tue, 24 Oct 2017 13:30:00 +0000

There are many instructions how to run Django in Docker. The base configuration of docker-compose usualy has the next services: nginx, web (django), db (mysql, postgresql), cache (memcached, redis). I’ll just add some notes.

Azure Cosmos DB

Tue, 03 Oct 2017 13:30:00 +0000

If you look for a MongoDB-as-a-service and want to try Azure Cosmos DB, be careful, because not all mongo’s functions are supported. For example, there is a problem with aggregation and group by. I didn’t found the list about what else can be different. Other solid solutions are MongoDB Atlas, it supports Azure from this summer, and mLab.

StackOverflow changes from 2013 to 2017

Mon, 25 Sep 2017 13:30:00 +0000

Here is an update for StackOverflow statistics for the users with a reputation more than 300. The last measure was made in January 2013.

I almost stopped to answer from the beginning of 2014. At that time I had a reputation around 1600, now it’s 2630.

The number of users growed more than 3 times, but the distribution by reputation is mostly the same.

EuroPython 2017

Tue, 12 Sep 2017 13:30:00 +0000

This year, as in previous, I missed the conference. This time I even bought the ticket, but had to return it. The refund process works well, but takes time. Fortunately, there are the live streams from all days, the only problem is to find a time to watch them.

How to install pillow, psycopg, pylibmc packages in python:alpine image

Thu, 07 Sep 2017 13:30:00 +0000

Here is the examples how to install some packages in python:alpine image. Some dependencies should be installed permanently and some only for package build and removed after. pillow RUN apk add --no-cache jpeg-dev zlib-dev psycopg RUN apk add --no-cache postgresql-dev pylibmc RUN apk add --no-cache libmemcached-dev zlib-dev Then to install: RUN apk add --no-cache --virtual .build-deps build-base linux-headers \ && pip3 install pip --upgrade \ && pip3 install <packages list> \ && apk del .

Mailgun for emails in docker

Tue, 08 Aug 2017 13:30:00 +0000

Usually, even a small web application should send some emails: errors, registration, restore password, invitations, etc.

There are 3 solutions:

use an existing account, like gmail: the main problem - you don’t know how many emails you can send before being marked as spam
use own mail server: need to install, configure and support
use email service, like sendgrid, mailgun: easy to configure, provide statistics and logs, can use default http port.

For some my small projects i choosed mailgun, it provides 10000 emails in month for free, for my needs it’s more than enough and after that it’s also not expensive. For django there is a email backend django-mailgun and it takes two minutes to switch to it.

Interesting articles 2

Tue, 01 Aug 2017 13:30:00 +0000

It’s good to know how things are done in the other companies. The Stack That Helped Medium Scale To 2.6 Millennia Of Reading Time How CircleCI Processes 4.5 Million Builds Per Month

JSON logging in Python

Fri, 28 Jul 2017 13:30:00 +0000

A lot of the log systems (loggly, logentries, sematext, kibana, etc.) can understand the JSON format. The benefits: it’s easier to search, filter and analize logs.

Here is an example of configuration for Python with structlog library.

Install

pip install structlog
pip install python-json-logger

How to run Docker Swarm localy

Thu, 20 Jul 2017 13:30:00 +0000

Suppose there is an application which uses docker-compose to run and you want to try it in swarm mode.

First of all you need to initialize swarm nodes.

Interesting articles

Tue, 11 Jul 2017 13:30:00 +0000

Several interesting articles I found recently. Some of them I think from Dan Bader twitter. Django vs Flask - a detailed comparison Django vs Flask. In most cases I preferer Django. It has better documentation There are better apps in core, with Flask you need to search for an app, compare alternatives and sometimes they are not updated for a long time You can choose which core apps to use I think it’s better developed and supported How the heck does async/await work in Python 3.

Flask-RESTful migration to Flask-RESTPlus

Wed, 05 Jul 2017 13:30:00 +0000

In a simple application with a several endpoints, it’s ok to use the default Flask routes.

@app.route('/api/v1.0/users', methods=['GET'])
def get_users():
    return jsonify({'objects': users})

But when an application becomes bigger, it can be useful to use a some REST framework or split it on the smaller apps. The REST framework is used to:

enforce best practices
simplify: versioning, serialization, documentation, authenication
provide browsable API

How to use Jupyter notebooks with Flask app

Tue, 27 Jun 2017 13:30:00 +0000

The strong part of Python and other interpreted languages is an interactive shell. But there is an even more powerful tool - Jupyter notebook.

In a simple case it allows to:

verify how some code works
check hypothesis
save these experiments and continue to work on them later
share them with the colleagues

Flask-Security 3.0.0 released

Tue, 20 Jun 2017 13:30:00 +0000

Finally it happened. The previous version 1.7.5 was released in December 2015. It’s not a big library but the useful one. There is a flask-security-fork, but it’s better to have a one official place for all issues and pull requests. Now it’s merged. Thanks to Jiri Kuncar for all the efforts. The full list of changes is here.

Mongoengine sanitized string field

Fri, 16 Jun 2017 13:30:00 +0000

One of the rules to prevent XSS attack is to sanitize HTML. Here is an example how to do this with Mongoengine custom field and bleach library.

Makefile help target

Tue, 13 Jun 2017 13:30:00 +0000

In many projects to run complex commands or group of commands I use Makefile. To document it I add a help target. Here is an exemple for dump and restore commands for PostgreSQL in docker-compose under alias db. DATE = $(shell date +%d%m%Y) path ?= . help: ## show this help @echo 'usage: make [target] ...' @echo '' @echo 'targets:' @egrep '^(.+)\:\ .*##\ (.+)' ${MAKEFILE_LIST} | sed 's/:.*##/#/' | column -t -c 2 -s '#' dump: ## dump db, usage 'make dump path=/path/to/dumps' docker-compose exec --user postgres db pg_dumpall --clean | gzip > $(path)/db_$(DATE).

How to debug AngularJs 1.5 in browser console

Thu, 08 Jun 2017 14:01:00 +0000

Sometimes it’s useful to run service, controller function or view some scope directly in console. Here are several examples how to get access to them.

Nginx SSL configuration in Docker Swarm

Fri, 02 Jun 2017 13:30:00 +0000

When you have several Nginx replicas in Docker Swarm and you need to share SSL certificate, you can use secrets.

Here is a short extract:

Docker clean commands

Tue, 30 May 2017 13:30:00 +0000

When you start to use Docker, at some point there will be a question - where is all my space. Most of it is occupied by containers, images and dangling volumes. An easy way to clean is to use Docker garbage collection by Spotify and docker volume rm commands. I have an alias for this. alias docker-gc='docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc -e GRACE_PERIOD_SECONDS=172800 spotify/docker-gc && docker volume rm $(docker volume ls -qf dangling=true)' On mac sometimes it’s easier to remove all, for this you need to remove Docker.

Nginx upstream problems in docker environment

Tue, 23 May 2017 13:30:00 +0000

I have nginx as a proxy for a python application. To launch them I have a docker-compose.

How to refresh access token for Microsoft Graph in Python

Thu, 18 May 2017 13:30:00 +0000

The connection to Microsoft Graph with Flask-OAuthlib described well in this tutorial.

One detail it’s missing - how to refresh the access token, because it expires in an hour. In Flask-OAuthlib there is no method for this but it can be done easily with a POST request.

AngularJs 1.5 controllers inheritance

Fri, 05 May 2017 13:30:00 +0000

In the first place there is a good style guide for Angular.

When you need to reuse the code of some controller and use controllerAs syntax, you can use inheritance with the call() method.

My current tools

Fri, 05 May 2017 13:30:00 +0000

Here is an update for the tools I’m using. The previous record was in 2013.

AppleScript to JavaScript

Fri, 28 Apr 2017 14:00:39 +0200

A small example how to rewrite make call from AppleScript to JavaScript. tell application "Microsoft Outlook" make new signature with properties {name: "Signature Name", content: "Best regards, ..."} end tell Application("Microsoft Outlook").Signature({name: "Signature Name", content: 'Best regards, ...'}).make();

Microsoft killed Wunderlist

Tue, 25 Apr 2017 13:21:39 +0200

Sadly, Microsoft is killing Wunderlist, more info in an official blog and on thenextweb.com. So it’s time to try something else, my requirements: should work on any device possibility to share lists with my family sub projects, minimum two levels possibility to review next and previous weeks possibility to review all tasks developer API Some of these does not exists in Wunderlist, but covered by wunderweek.com, which I created at the end of 2016.

Migrations for MongoEngine

Fri, 14 Apr 2017 13:39:48 +0100

A MongoDB collection does not have a strict schema, the documents in one collection can have different fields. But in MongoEngine the schema is provided on an application level and there is a validation. So if you’ll remove a field from the Document and it’s still exists in the database, there will be a mongoengine.errors.FieldDoesNotExist exception. If there is no need in the schema validation, a dynamic schema can be used.

So when there is a strict schema, it’s good to have a tool to make an updates for the documents in the collection when something changes.

How to use multiple databases with Flask-MongoEngine

Tue, 11 Apr 2017 13:39:48 +0100

Sometimes it’s a good idea to use different databases for different parts of the application, they can have different requirements for performance and backup strategy or as a step to split a monolith to microservices.

Here is an example of configuration for Flask and MongoEngine.

Sublime Text freeze

Thu, 30 Mar 2017 13:31:48 +0100

Recently my Sublime Text has stopped running with freeze on start. I don’t know the cause, but removing of the last session helped. cd "~/Library/Application Support/Sublime Text 3/Local" rm *.sublime_session

How to build a dynamic query with mongoengine

Sun, 26 Mar 2017 19:26:48 +0100

An example of the use of the Q class to build the query.

Migration from http to https for hugo blog

Fri, 17 Mar 2017 13:36:48 +0100

This week I moved the blog from http to https. The reason - everything is moving to https and maybe it’s good for SEO. There were several steps.

Google App Engine limits and vendor locks

Fri, 03 Mar 2017 13:36:48 +0100

At the beginning of the 2015 I worked on a python application in Google App Engine. After all, because of some limitations and vendor locks, the app was moved to docker environment. Here is the list of these limitations. I did not check since then, maybe something changed, there is App Engine Flexible Environment, but it’s still in beta.

Celery background task with notifications through socket.io

Tue, 28 Feb 2017 13:36:26 +0100

If you have some long background tasks, sometimes it’s useful to notify a user about the progress. In this article i put an example for flask, celery and socket.io. In short form it’s described in flask-socketio documentation.

How to create a test certificate to sign Electron application

Tue, 21 Feb 2017 13:36:26 +0100

If you use electron-builder and you need to sign your app to test the update process, but you don’t have the Apple Developer account.

Flask-Admin formatters examples

Fri, 10 Feb 2017 13:36:26 +0100

To customize how the model’s columns are displayed in the list of objects in Flask-Admin, it requires column formatters. Here are some examples.

Macbook Air 2012 upgrade to Macbook Pro 2016

Fri, 03 Feb 2017 00:00:00 +0000

The last 4 years I worked on Macbook Air Mid 2012 - 8 Gb, now I have Macbook Pro 13.3" 2016 - lowest model, no touchbar, but with 16 Gb. Actually there is nothing to compare, the Macbook Pro is better in every aspect.

The main result is that everything 1.5-2 times faster, some examples:

Fast tests with in-memory MongoDB

Thu, 26 Jan 2017 17:11:00 +0100

It’s always better to have a faster tests. For tests which use a database there are two ways to increase the speed:

a mock library, like mongomock
in-memory database

Achievments in 2016

Thu, 19 Jan 2017 17:11:00 +0100

A list of my most important achievments in 2016:

started a new job at MailInBlack & LetSignIt, like it.
used a lot of different technologies: flask, celery, angular, mondodb, scala, docker, azure & google clouds, google app engine

Migrate from octopress to hugo

Thu, 12 Jan 2017 13:55:34 +0100

As usual at the beginning of the year I decided to continue the blog, but I was on a new macbook with a fresh system and also I forgot how to use octopress. So, why not to try something new.

NGINX for static files for dev python server

Sat, 14 May 2016 00:00:00 +0000

When you work on the backend part of django or flask project and there are many static files, sometimes the development server becomes slow. In this case it’s possible to use nginx as reverse proxy to serve static. I’m using nginx in docker and the configuration is quite simple.

strftime for datetime before 1900 year

Wed, 08 Apr 2015 00:00:00 +0000

Recently I’ve got an error for birthdays before 1900 year ValueError: year=1890 is before 1900; the datetime strftime() methods require year >= 1900 for this code: import datetime datetime.date(1890,7,1).strftime('%d.%m.%Y') And it’s described in the documentation The full set of format codes supported varies across platforms, because Python calls the platform C library’s strftime() function, and platform variations are common. To see the full set of format codes supported on your platform, consult the strftime(3) documentation.

Test django view with cookies

Fri, 20 Mar 2015 00:00:00 +0000

To test some view which use cookies: from Cookie import SimpleCookie from django import test class SomeTest(test.TestCase): def test_some_view(self): self.client.cookies = SimpleCookie({'test_cookie': 'test_value'}) response = self.client.get('/some-url/') self.assertEqual(response.client.cookies['test_cookie'].value, 'test_value')

My current tools

Sat, 20 Jul 2013 00:00:00 +0000

MacBook Air 13" instead of iMac 24" 2007 It’s faster and provides me mobility, but I’m lacking in screen size, I plan to solve this with external monitor. Sublime Text3 with next packages: package control djaneiro (django support) django-docsearch git (checkout branch python3) jsformat modific (highlighting lines changed since the last commit) sidebar enchancement sublimelinter (checkout sublime-text-3 branch) theme phoenix Before this I used PyCharm, it’s very good, but slower.

Installing Django on Ubuntu memo

Tue, 14 May 2013 00:00:00 +0000

Here are a few notes and links after moving to the new Ubuntu server on linode.com.

Some steps for security: My First 5 Minutes On A Server

Install packages:

apt-get install apache2 libapache2-mod-wsgi
apt-get install postgresql postgresql-server-dev-9.1 python-dev
apt-get install mysql-server mysql-common mysql-client libmysqlclient-dev 
apt-get install git git-core

Setup virtualenv:

apt-get install python-setuptools
apt-get install python-pip
pip install virtualenv
virtualenv --no-site-packages /path/to/venv

Install PIL in virtualenv:

apt-get install libjpeg libjpeg-dev libfreetype6 libfreetype6-dev zlib1g-dev
ln -s /usr/lib/`uname -i`-linux-gnu/libfreetype.so /usr/lib/
ln -s /usr/lib/`uname -i`-linux-gnu/libjpeg.so /usr/lib/
ln -s /usr/lib/`uname -i`-linux-gnu/libz.so /usr/lib/
pip install PIL

Real time messaging in Django using Node.js and Redis

Fri, 08 Feb 2013 00:00:00 +0000

Here, just a small example of this task. Assume you have django comments system enabled.

Then you need to add publishing of comments into redis. It can be done with comment_was_posted signal.

How to update Django user profile from ForeignKey to OneToOneField

Thu, 31 Jan 2013 00:00:00 +0000

In one of my projects for the user profile model was used ForeignKey field.

class UserProfile(models.Model):
    user = models.ForeignKey(User, unique=True)

This leads to a lot of unnecessary queries when you want to access to related data, like:

{% raw %}
{% for item in items %}
    {{ item.user.get_profile.company }}
{% endfor %}
{% endraw %}

StackOverflow Charts

Sat, 19 Jan 2013 00:00:00 +0000

A month ago I began to participate in StackOverflow process. So I was interested to collect some statistics data and find where am I in this game. StackOverflow has a good API with limits for 300 requests/days for anonymous user and 10 000 requests/day for authorized. I created a simple bot on python, collected data to MongoDb and built these charts. With 10 000 requests and page size 100 I got users from highest reputation (524k) to 269. So I built these charts starting from reputation of 300.

The number of users with reputation > 300 is 96 654. And total number of users is about 1 375 000. So the percent of active users is about 7%.

My version of django boilerplate

Sun, 13 Jan 2013 00:00:00 +0000

django_boilerplate - it’s a template to fast start new Django project. Based on: django twitter bootstrap Used apps: south - data migrations django-crispy-forms - for forms rendering django-allauth - for local and social registration, authorization django-debug-toolbar django-extensions It also includes example app with List, Create, Update, Delete views. Install pip install -r requirements.txt python manage.py syncdb

A simple CRUD app with Django and Mongoengine

Sat, 05 Jan 2013 00:00:00 +0000

There are several possibilities to use MongoDB in Django:

Django MongoDB uses django-nonrel, which is a fork based on Django 1.3.
I don’t like this idea, because now Django 1.5 is ready to out. Beetween Mongoengine and MongoKit, I like more Mongoengine. There are several comparative articles:

So I created a simple CRUD app using Mongoengine.