Internet-developer workshop

Debugging in docker-compose

2021-05-19T08:33:58+03:00

It is often necessary to use the debugging tools inside the container, but for this you need to get a full-fledged tty with to interact with the debugger interface via stdin/stdout.

In the note below I will show you how to do this, using this method I run pudb , pdb, ipdb, etc.

You need to add stdin_open: true and tty: true to your service:

version: "3.9"
services:
  api:
    build: .
    stdin_open: true
    tty: true
    ports:
      - "8000:8000"
    volumes:
      - .:/app

Then run it as follows:

docker-compose run --service-ports api

If for some reason you cannot do this, then you can use the wdb

Sniffing HTTPS Traffic in Chromium with Wireshark

2020-12-02T22:33:04+03:00

For decrypt traffic from Chromium (or Chrome), you need to configure writing SSL-logs to a file, and then configure Wireshark so that it reads this file.

Chromium setup

/usr/local/bin/chromium --ssl-version-max=tls1.3 --ssl-key-log-file=/tmp/sslkeylog.log

(it will be more convenient to specify alias for your shell)

Wireshark setup

Go to Preferences > Protocols > TLS, specify the path to the file:

As a result, you will be able to see the decrypted data:

Simple healthcheck to Kafka for docker-compose

2020-11-25T12:36:26+03:00

version: "2.1"

services:
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
      - '9093:9093'
    environment:
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_CFG_LISTENERS=CLIENT://:9093,EXTERNAL://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9093,EXTERNAL://localhost:9092
      - KAFKA_INTER_BROKER_LISTENER_NAME=CLIENT
    depends_on:
      - zookeeper
    healthcheck:
      test: ["CMD-SHELL", "kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic <TOPIC_NAME> --describe"]
      interval: 2s
      timeout: 2s
      retries: 15

  someservice:
    image: ...
    depends_on:
      kafka:
        condition: service_healthy
    links:
      - kafka
    command: ['...']

Denial of cybercrime

2018-12-13T11:42:11+03:00

Photo from the wikipedia

Some time ago, my name began to be mentioned in the media, pointing to my involvement in illegal actions, to which I have no relation. I am always aware of what I'm doing. I have never done, not doing now and do not plan to do in the future anything illegal. This can be confirmed by everyone whom I have ever worked or communicated.

Disclaimer: The article that you see is an inaccurate translation of the original article and is for reference only. The translation was not performed by a professional and may not coincide in meaning with the original article.

Please refute in the media (such as BuzzFeed, Krebs on Security, Forbes) due to erroneous or unreasonable including me on the list of those accused of violating the law.

I ask the FBI to remove my name from the list of suspects, since I have never taken any illegal actions (described in the document or any others) did not commit.

In 2016, I already responded to such accusations by the Krebs on Security, gave an interview to The Republic magazine with a refutation of any illegal activity on my part. I attach to this article screenshots of correspondence with the journalist of the The Republic edition:

In addition to the above, I want to write a few more words about myself: I lead an open public lifestyle, play sports and skateboarding, work as an outsourcer programmer, my direction is developing web applications on Python, Erlang and JavaScript, data analysis and consulting services. In my free time I keep a technical blog. I am outgoing, positive and law-abiding person. Any illegal activity for me is rejected by nature of me.

I am sure that people and organizations that have access to all my personal, confidential information and are engaged in this investigation - can easily make sure that I am innocent.

P.S.: I have great respect for the profession of a reporter, but please do not disturb me on the above issue, because I can not add anything else to this.

Regards, Mikhail Andreev.

Mount Google Drive drive through gdfs on Ubuntu 17.04

2017-12-29T10:28:12+03:00

I use Google Drive to store backups on some of my servers, but I can invent many scenarios for using this cloud storage.

To do this, we will need to install the client library to work with the Google API, the gdfs driver, get the authorization code and configure the automatic mount when the server boots.

Install the Google Client API

The client is needed to authorize gdfs

sudo apt install -y python-pip
git clone https://github.com/google/google-api-python-client
cd google-api-python-client
sudo python setup.py install install_egg_info

Install Google Drive FS

Install gdrivefs and make a symlink for convenience

sudo pip install gdrivefs
cd /sbin
sudo ln -s `which gdfs` mount.gdfs

Customization

You need to authenticate to the Google API Client and create a directory to mount the drive

gdfstool auth -u
# Open the link in your browser, please log in
# and copy the received key as <code>
gdfstool auth -a /var/.gdfs.creds "<code>"
mkdir /mnt/gdrivefs

Mounting through /etc/fstab

You can add the following entry to /etc/fstab

echo "/var/.gdfs.creds/mnt/gdrivefs gdfs allow_other, big_writes 0 0" >> /etc/fstab
mount /mnt/gdrivefs

But the mount takes place before the network connection of the server, and without the network you can not connect Google Drive, so after rebooting the server you will have problems with the mount. You can add the option _netdev:

echo "/var/.gdfs.creds/mnt/gdrivefs gdfs allow_other,big_writes,_netdev 0 0" >> /etc/fstab
mount /mnt/gdrivefs

But I could not use this option:

fuse: unknown option "_netdev"

Mounting through /etc/network/interfaces

I mount the disk in the following way

echo "post-up /bin/mount/var/.gdfs.creds/mnt/gdrivefs -t gdfs -o allow_other,big_writes" >> /etc/network/interfaces

After rebooting the server, you will start post-up and mount the disk.

Checking

We will mount it manually and look at the files:

mount /var/.gdfs.creds /mnt/gdrivefs -t gdfs -o allow_other,big_writes
ls -la /mnt/gdrivefs

I recommend that you reboot and check that everything works.

Erlang. Fix run multiple copies of epmd

2013-02-11T11:50:50+04:00

Faced with the problem when you do not ping the erlang-node on some server that is running two epmd. And running two of them, because one runs the project (at the start erl), and the second starts ejabberd.

Symptoms resemble a problem with firewall:

(test2@127.0.0.1)1> net_adm:ping('test1@127.0.0.1').
pang

See ps ax:

ps ax | grep epmd
root     1212   ... /usr/local/lib/erlang/erts-5.9.3.1/bin/epmd -daemon
root     1515   ... /usr/local/bin/epmd -daemon

And if you execute kill 1212 1515 and run again erl, it will be pinged:

(test2@127.0.0.1)1> net_adm:ping('test1@127.0.0.1').
pong

Additional symptoms I described on Stack Overflow.

Solution

Open for editing /etc/rc.conf:

ejabberd_enable="YES"
epmd_enable="YES"

And in /usr/local/etc/rc.d/epmd adding:

start_precmd=epmd_prestart
epmd_prestart() {
    `ps ax | grep epmd | awk '{ print $1}' | xargs kill`
    return 0
}

We have to use a hack, because I could not find how to get ejabberd not to launch epmd.

Fast resize and caching of images with "django-nginx-image"

2012-11-10T09:26:01+04:00

For a project KinsburgTV I had to do resize images. At first I copied from one of my projects a template tag for resizing (taking roots from this snippet), but for production it is very slow solution. So I wrote another template tag, which built correct URL for Nginx, and this URL handled in ngx_http_image_filter_module and proxy_cache.

I have a chart for comparison, which I did through JMeter, but it was not informative for me. If anyone interested, here's the chart. At the left side is PIL, at the right side Nginx. Select for 1000 requests.

Obtained a faster, transparent for applications a resizer and cropper, with caching results. And command for convert images to supported format. So, meet django-nginx-image!

Installation and settings

Install from PyPI:

pip install django-nginx-image

Next, you should setup Nginx:

<STORAGE_ROOT> - a path to "media/statiс" directories (example "/storage/kinsburg_tv");
<CACHE_NAME> - arbitrarily name for cache (example "kinsburg_tv_thumbnails_cache").


http {

    # Specify a desired path to cache directory, name of cache and maximum size of cache
    proxy_cache_path <STORAGE_ROOT>/nginx/cache levels=1:2 keys_zone=<CACHE_NAME>:10m max_size=1G;

    # Now, you must setup a "server", which will cache results
    server {
        listen 80;
        server_name www.example.org;

        location ~* ^/(resize|crop)/ {
            proxy_pass http://image.example.org$request_uri;
            proxy_cache <CACHE_NAME>;
            proxy_cache_key "$host$document_uri";
            proxy_cache_valid 200 1d;
            proxy_cache_valid any 1m;
            proxy_cache_use_stale error timeout invalid_header updating;
        }
    }

    # And a second "server", which will be resize and crop
    server {
        listen 80;
        server_name image.example.org;

        location ~* ^/resize/([\d\-]+)/([\d\-]+)/(.+)$ {
            alias <STORAGE_ROOT>/$3;
            image_filter resize $1 $2;
            image_filter_buffer 2M;
            error_page 415 = /empty;
        }

        location ~* ^/crop/([\d\-]+)/([\d\-]+)/(.+)$ {
            alias <STORAGE_ROOT>/$3;
            image_filter crop $1 $2;
            image_filter_buffer 2M;
            error_page 415 = /empty;
        }

        location = /empty {
            empty_gif;
        }
    }
}

If you have no need for caching, then simply remove a location from the first "server" and move locations from the second section "server" in the the first. Caching is increased performance to 10-20%.

Then add to settings.py:


INSTALLED_APPS = (
    'nginx_image',
)

Scheme to illustrate the work

Usage

In any of your templates, plug in the module nginx_image with the template tag thumbnail, give to tag a path to image , width, height and optional flag crop to crop the image, not resize:


{% load nginx_image %}

Proportionally resize a image, based on the width and the height:
    <img src="{% thumbnail user.profile.avatar 130 130 %}" />

Proportionally resize a image, based on the width:
    <img src="{% thumbnail user.profile.avatar 130 '-' %}" />
    <img src="{% thumbnail user.profile.avatar 130 0 %}" />
    <img src="{% thumbnail user.profile.avatar 130 %}" />

Proportionally resize a image, based on the height:
    <img src="{% thumbnail user.profile.avatar '-' 130 %}" />
    <img src="{% thumbnail user.profile.avatar 0 130 %}" />

Crop a image:
    <img src="{% thumbnail user.profile.avatar 130 130 crop=1 %}" />
    <img src="{% thumbnail user.profile.avatar 130 0 crop=1 %}" />
    <img src="{% thumbnail user.profile.avatar 0 130 crop=1 %}" />

In general, this tag only generates URL like this:

/resize/130/-/media/users/avatars/12345.jpg
/resize/-/130/media/users/avatars/12345.jpg
/crop/130/-/media/users/avatars/12345.jpg

This tag can be used outside a template, for example:


class Film(models.Model):
    thumbnail = models.ImageField(upload_to='films/thumbnails', null=True)

    def thumbnail_preview_url(self):
        from nginx_image.templatetags.nginx_image import thumbnail
        return thumbnail(self.thumbnail, 160, 140, crop=True)

The command nginx_image_converter

Unfortunately (or fortunately), ngx_http_image_filter_module only supports JPEG, GIF and PNG, so I had to write a converter that converts the formats other than the above to JPEG.

./manage.py nginx_image_converter -i /storage/project/media -o /storage/project/newmedia

Additional options for this command you can see here.

The problem with Nginx resolver

If you found a bug in nginx/error.log:

... no resolver defined to resolve image.example.org ...

then setup resolver in nginx/nginx.conf, example:

resolver 127.0.0.1 8.8.8.8;

Nice to meet, KinsburgTV!

2012-09-29T18:00:43+04:00

Passed three years since the launch of the first version, and two years with the second.

The new major release is available on the new domain kinsburg.tv, so I decided not to disconnect the old site, and develop a new site as parallel.

The new project is written in Python and Django, also under the hood are Nginx+uwsgi, Scrapy, MySQL, SphinxSearch, Redis, Celery, and to deploy and support the project using virtualenv, supervisor, South and Fabric.

The project is rewritten, because I'm in the last 3 years have been programming using Python and Django. Maintain and develop features for PHP was too expensive, so I decided to rewrite everything in Python.

New features

Appeared English interface!
The new filter (with sorting and searching) without reloading the page. Looking well, in addition to text, there is a search for the mask.
New bookmarking service with the ability to create your own types of bookmarks:
- Also has the ability to move, delete and copy bookmarks;
- Architecture provides an opportunity to put in a bookmark any objects (thanks to "django.contrib.contenttypes"), and not only films (but now the bookmarks are only for films).
Film companies and countries are now clickable:
- Now, there are few published film companies, but they will be added gradually through Scrapy;
- Countries were not represented in the menu, but you can get it from the film page.
Blogs about movies, movie stars and film companies:
- Blogs yet taken only from the site www.spletnik.ru and only themes about movies, of course.
Film reviews:
- While reviews are taken from Kinopoisk, but also has a manual entry of reviews completed yet and will soon be published and user-friendly interface to writing reviews.
Appeared the authorization through social networks! I decided that now is relevant only Facebook, Vkontake and Twitter. I'm thinking about adding Google, Yandex and OpenID, but am not sure that this is true. Authorization is implemented using nice library django-social-auth.

With the project also helped me mouellke. She did a good design for the entire site! Thank her so much!

Visit KinsburgTV right now!

Released a pluggable application "django-sql-stacktrace"

2012-08-23T17:16:18+04:00

This application is puts python stack trace to a SQL-query as comment. It's useful when you is debugging your project, when you see many SQL-queries and do not understand where these queries were launched.

PyPI: django-sql-stacktrace

For example, you have come to a new project and you have been asked to deal with a lot of SQL-queries that hang at run time. You look at the "processlist" and see a lot of different SQL-queries:


...

SELECT COUNT(*) FROM `films_film` INNER JOIN `films_year` ON (`films_film`.`year_id` = 
`films_year`.`id`) WHERE (`films_film`.`status` = 'published' AND `films_year`.`id` IS NOT NULL);

SELECT `blogs_post`.`id`, `blogs_post`.`blog_id`, `blogs_post`.`title`, `blogs_post`.`slug`, 
`blogs_post`.`status`, `blogs_post`.`created`, `blogs_post`.`updated`, 
`blogs_post`.`published`, `blogs_post`.`description`, `blogs_post`.`thumbnail`, 
`blogs_post`.`content`, `blogs_post`.`count_of_views`, `blogs_post`.`source_content`, 
`blogs_post`.`source_published`, `blogs_post`.`source_count_views`, 
`blogs_post`.`source_author_of_content`, `blogs_post`.`source_author_of_photo`, 
`blogs_post`.`source_keywords`, `blogs_post`.`source_url` FROM `blogs_post` WHERE 
`blogs_post`.`status` = 'published' ORDER BY `blogs_post`.`published` DESC LIMIT 4;

SELECT `social_auth_usersocialauth`.`id`, `social_auth_usersocialauth`.`user_id`, 
`social_auth_usersocialauth`.`provider`, `social_auth_usersocialauth`.`uid`, 
`social_auth_usersocialauth`.`extra_data` FROM `social_auth_usersocialauth` WHERE 
`social_auth_usersocialauth`.`user_id` = 42;

SELECT `films_film`.`id`, `films_film`.`status`, `films_film`.`created`, `films_film`.`updated`, 
`films_film`.`published`, `films_film`.`title_original`, `films_film`.`title_ru`, `films_film`.`slug`, 
`films_film`.`thumbnail`, `films_film`.`slogan`, `films_film`.`description`, 
`films_film`.`date_premiere`, `films_film`.`year_id`, `films_film`.`rating_votes`, 
`films_film`.`rating_score`, `films_film`.`rating_score_with_vote_weight`, `films_year`.`id`, 
`films_year`.`name`, `films_year`.`count`, `films_year`.`size` FROM `films_film` LEFT OUTER 
JOIN `films_year` ON (`films_film`.`year_id` = `films_year`.`id`) WHERE (`films_film`.`status` 
= 'published'  AND `films_year`.`id` IS NOT NULL) ORDER BY `films_film`.`published` DESC 
LIMIT 30;

...

Where did they run you do not know, moreover, it is not only the queries from views, it may also be sql-queries from the django commands, something daemons, etc.

No problem! You take django-sql-stacktrace, run on a local copy of the project and see the following:


...

SELECT COUNT(*) FROM `films_film` INNER JOIN `films_year` ON (`films_film`.`year_id` = `films_year`.`id`)
WHERE (`films_film`.`status` = 'published' AND `films_year`.`id` IS NOT NULL)
/* STACKTRACE: 
> File "/home/adw0rd/work/kinsburg_tv/kinsburg_tv/common/views.py", line 27, in get_context_data
> context = super(MainPage, self).get_context_data(**kwargs)

STACKTRACE: 
> File "/home/adw0rd/work/kinsburg_tv/kinsburg_tv/films/mixins.py", line 14, in get_context_data
> context = super(FilmsMixin, self).get_context_data(**kwargs)

STACKTRACE: 
> File "/home/adw0rd/work/kinsburg_tv/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 351, in count
> return self.query.get_count(using=self.db)

STACKTRACE: 
> File "/home/adw0rd/work/kinsburg_tv/venv/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 418, in get_count
> number = obj.get_aggregation(using=using)[None]

STACKTRACE: 
> File "/home/adw0rd/work/kinsburg_tv/venv/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 384, in get_aggregation
> result = query.get_compiler(using).execute_sql(SINGLE)

STACKTRACE: 
> File "/home/adw0rd/work/kinsburg_tv/venv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 818, in execute_sql
> cursor.execute(sql, params)

STACKTRACE: 
> File "/home/adw0rd/work/kinsburg_tv/venv/local/lib/python2.7/site-packages/sqlstacktrace/stacktracecursor.py", line 16, in execute
> stacks = get_stacktrace()

STACKTRACE: 
> File "/home/adw0rd/work/kinsburg_tv/venv/local/lib/python2.7/site-packages/sqlstacktrace/stacktrace.py", line 93, in get_stacktrace
> stack = get_stack()
*/;

...

Good job! Now you know where these queries were launched and has something to do with this problem.

Installation and configuration

Install from PyPI:

pip install django-sql-stacktrace

Or install the dev-version from GitHub:

pip install -e git://github.com/adw0rd/django-sql-stacktrace.git#egg=sqlstacktrace

Now, add to "INSTALLED_APPS":

INSTALLED_APPS = (
    ...
    'sqlstacktrace',
    ...
)

And turn on the SQL_STACKTRACE:

SQL_STACKTRACE = True

Getting code

You can get django-sql-stacktrace from https://github.com/adw0rd/django-sql-stacktrace. The application is distributed by BSD license.

The "django-lictor" for Django Dash 2012

2012-08-21T17:16:58+04:00

Hello everybody, at weekend we participated in the Django Dash 2012, and we made an application to visualize your projects via stack trace.

What is "lictor"?

The lictor (possibly from Latin: ligare, "to bind") was a member of a special class of Roman civil servant, with special tasks of attending and guarding magistrates of the Roman Republic and Empire who held imperium, the right and power to command; essentially, a bodyguard. The origin of the tradition of lictors goes back to the time when Rome was a kingdom, perhaps acquired from their Etruscan neighbours.

http://en.wikipedia.org/wiki/Lictors

Team

Our team was called Tetronix and included:

Oleg Grigoriev (vasa-c)
Mikhail Andreev (adw0rd)
Boris Timokhin (mathete)

We did in amount 132 commits, but was not enough time to finish the project.

Demostration

You can see the demo on page http://lictor.tetronix.org/

Unfortunately, I can't create screencast in Linux (xUbuntu), please suggest a program for it!

It's open source and you can get a code from https://github.com/ussi/django-lictor.

What's next?

Since "django-lictor" is part of the idea of visualization Django-projects, then my next step will be the appearance of the application [django-visualization](https://github.com/adw0rd/django-visualization, which will render the whole project, not just the call stack.

While planning to use AST, source code of "django-lictor" will be used to visualize the stack trace built tree project.

Also the opportunity to create a task, review, etc.

jQuery. Wrote a plugin "jquery-cleverfocus"

2012-03-01T15:01:15+04:00

Today, I released a plugin for jQuery, which, when you enter a some text in "nowhere" intercepts it, and at the expiration of the limit entered symbols - write collected symbols to you specified "input" and puts a "focus".

A plugin is compability with the jQuery 1.7.1, because it has only been tested.

Use cases

Case #1

User comes to the site, seen the search form and typing a search words (herewith he looks at the keyboard). Looking up, he understand that the text wrote "to nowhere", he is irritated. Puts focus to search input, typing several characters, check what he does to type "where necessary" and calmer continues typing in search terms.

Then you say: "Hey! Need to set the focus after loading the DOM!".
But no, this is also a bad decision, I explain it in the Case #2.

Case #2

User comes to the site, he was not interested in the search form, he wants scrolling (Left/Right, Up/Down, PgUp/PgDown, Home/End, etc.) page using the keyboard (users who use laptops without mouse quite a lot), and since we have is the focus on the search form, then we cant scroll the page.

For decision this two cases has been created this plugin: jquery-cleverfocus.

Example

Suppose, you have a form with id="search_input":

<input id="search_input" />

Then, just use the plugin:

$("#search_input").cleverfocus({
    keypress_limit: 3
});

The plugin includes functional_test.html, which serves to testing form, as well as you can see onlone demo on adw0rd.github.com/jquery-cleverfocus.

Why do I named it that?

First, I named it as "jquery-autofocus", but then searched in Google and found this name plugin with the same name, which is designed to emulate the autofocus attribute of HTML5 in HTML earlier versions.

Later, I named it "jquery-smartfocus", but here waiting for me disappointed, because there is such a plugin.

In the end, I named it by "humanly" - cleverfocus.

Released a pluggable application "django-multi-sessions"

2012-02-27T16:11:44+04:00

This application provides interfaces for multi-session backend. Basically designed to transfer sessions from one machine to another, without stopping your sessions.

Current version is 0.0.3

Installation from PyPI:

pip install django-multi-sessions

Or manual installation:

Download tarball, extract and run:

python setup.py install

Settings

You need specify in settings.py a "multi_sessions.session" as SESSION_ENGINE and fill the SESSION_MULTISESSIONS_POOL, as in example:

SESSION_ENGINE = "multi_sessions.session"
SESSION_MULTISESSIONS_POOL = (
    {
        "backend": "redis_sessions.session",
        "modes": ["read", "write", "delete"]
    },
    {
        "backend": "django.contrib.sessions.backends.db",
        "modes": ["read", "delete"]
    },
)

In this example the "redis-session" is using as a priority storage (read, write/create, delete), and "database-backend" is using only for reads and delete a sessions, because we migrate to "redis", and is no meaning in store data in "database-backend".

Sources

This application is open source, and you can getting source code from https://github.com/adw0rd/django-multi-sessions