paulo.difficiliora - un po' più difficile ?

Convertitore mp3 video e altro

noreply@blogger.com (Paulo) — Tue, 06 Oct 2020 07:12:00 +0000

Monitor in tempo reale - Prometheus ed altri oggetti.

noreply@blogger.com (Paulo) — Wed, 18 Sep 2019 11:48:00 +0000

Prometheus - Monitoring Tool - Centos7

Installazione di Prometheus Alertmonitor Grafana Node-exporter con docker come servizio e cartelle montate localmente per mantenere le configurazioni ed i dati dei servizi. La configurazione comprende anche l'uso di Telegram per le notifiche

Azioni

Prerequisiti (Docker Docker-Composer)
Installazione immagini (Docker Pull)
Configurazione avvio immagini
Configurazione servizi
Configurazione avvio servizi
Test (configurazione ed architettura)

Prerequisiti

docker
docker-compose

Riferimenti

installazione e configurazione di sistema

https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-using-docker-on-centos-7
http://www.tutorialspoint.com/articles/how-to-install-and-configure-prometheus-using-docker-on-centos-7
https://www.shellhacks.com/prometheus-monitoring-install-docker-ubuntu-centos/

regole di alert e queryper Prometheus

https://awesome-prometheus-alerts.grep.to/rules.html
https://www.fabernovel.com/en/engineering/alerting-in-prometheus-or-how-i-can-sleep-well-at-night
https://www.robustperception.io/irate-graphs-are-better-graphs

integrazione con Telegram

https://sudonull.com/posts/194-Prometheus-Grafana-Node-Exporter-Docker-in-Azure-with-notifications-in-Telegram

Installazione immagini Docker

 $ sudo docker pull prom/prometheus
 $ sudo docker pull prom/alertmanager
 $ sudo docker pull prom/node-exporter
 $ sudo docker pull google/cadvisor # TODO
 $ sudo docker pull grafana/grafana 
 $ sudo docker pull metalmatze/alertmanager-bot

Configurazioni permessi sulle cartelle

Sono le cartelle che conterranno le configurazioni dei vari servizi in maniera permanente anche con un riavvio di Docker

# prometheus folder settings
$ mkdir /dati/prometheus/data
$ sudo chown 65534:65534 /dati/prometheus/data

# grafana folder setting
$ /dati/grafana
$ id # get the $uid number
$ sudo shown $uid:$uid  /dati/grafana/

# alertmanager-bot Telegram
$ mkdir /dati/prometheus/alertmanager-bot

Configurazione Prometheus

E' configurato per raccogliere i dati da diverse sorgenti (scrape_configs)
/dati/prometheus/conf/prometheus.yml

global:
  scrape_interval:     15s 
  external_labels:
    monitor: 'codelab-monitor'

# configurazione degli alert
rule_files:
  - 'alert.rules.yml'

alerting:
  alertmanagers:
  - scheme: http
    static_configs:
    - targets:
      - "HOST_ALERT_MANAGER:9093"

scrape_configs:
# sorgenti da cui recupera le metriche

  # controlla se stesso
  - job_name: 'prometheus-service'
    scrape_interval: 5s
    static_configs:
      - targets: ['IP_HOST_LOCALE:9090']

  # controlla una applicazione NodeJS in con swagger
  - job_name: 'swagger-SRS'
    scrape_interval: '10s'
    metrics_path: '/swagger-stats/metrics'
    static_configs:
      - targets: ['IP_NODE_SERVER:8010']

  # controlla l'host locale
  - job_name: 'localhost-stats'
    scrape_interval: '20s'
    static_configs:
      - targets: ['IP_HOST_LOCALE:9100']

  # controlla grafana TODO

Configurazione regole alerts in Prometheus

Sono configurati due alert standard. Il primo si aziona se qualsiasi sorgente non è up per più di 30 sec. Il secondo se il carico dell'host supera 0.5
/dati/prometheus/alert.rules.yml

groups:
- name: example
  rules:

  # Alert for any instance that is unreachable for >2 minutes.
  - alert: service_down
    expr: up == 0
    for: 30s
    labels:
      severity: page
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."

  - alert: high_load
    expr: node_load1 > 0.5
    for: 2m
    labels:
      severity: page
    annotations:
      summary: "Instance {{ $labels.instance }} under high load"
      description: "{{ $labels.instance }} of job {{ $labels.job }} is under high load."


# Alert too many 500 api error on rate
 - alert: SRSHttpErrors5xx
    expr: sum(rate(api_request_total{code=~"^5.."}[1m])) / sum(rate(api_request_total[1m])) * 100 > 5
    for: 30s
    labels:
      severity: error
    annotations:
      summary: "HTTP errors 5xx (instance {{ $labels.instance }}) for 30 sec."
      description: "Too many HTTP requests with status 5xx (> 5%)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

Configurazione AlertManager

Per tutti gli alert è configurato un unico destinatario via email o Telegram

global:
  smtp_smarthost: 'smtp.__.__.it:25'
  smtp_from: 'alertmanager@:__.__.it'
  smtp_require_tls: false

route:
  group_by: ['alertname','priority','example']
  # Send all notifications to me.
  receiver: email-me
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 10m

  routes:
    - receiver: "email-me"
      group_wait: 10s
      match_re:
        severity: error|warning
      continue: true

    - receiver: "alertmanager-bot"
      group_wait: 10s
      match_re:
        severity: error
      continue: true

receivers:
  - name: email-me
    email_configs:
    - to: alert.destination@__.__.it

 - name: alertmanager-bot
    webhook_configs:
    - send_resolved: true
      url: 'http://hostname.it:8088'

Configurazione Docker compose per avvio servizi

Viene configurato l'avvio dei vari container docker con il mount delle cartelle. Per Grafana sono impostati diversi parametri di avvio.
/dati/prometheus/docker-compose.yml

version: '3'
services:
   prometheus:
      image: prom/prometheus:latest
      container_name: prometheus
      volumes:
         - /dati/prometheus/conf:/etc/prometheus
         - /dati/prometheus/data:/prometheus
      command:
         - '--config.file=/etc/prometheus/prometheus.yml'
         - '--storage.tsdb.path=/prometheus'
      ports:
         - '9090:9090'
# restart: always
   node-exporter:
      container_name: prometheus-node-exporter
      image: prom/node-exporter
      ports:
         - '9100:9100'
   alertmanager:
      container_name: prometheus-alertmanager
      image: prom/alertmanager
      volumes:
         - /dati/prometheus/conf/alertmanager.yml:/alertmanager.yml
      command:
         - '--config.file=/alertmanager.yml'
      ports:
         - '9093:9093'
   grafana:
      container_name: grafana-service
      image: grafana/grafana
      user: # $uid es: '140'
      volumes:
         - /dati/grafana:/var/lib/grafana
      environment:
         - GF_SECURITY_ADMIN_PASSWORD=password
         - https_proxy=https://proxy.__.__.it:8080
         - http_proxy=http://proxy.__.__.it:8080
         - GF_SMTP_ENABLED=true
         - GF_SMTP_HOST=smtp.__.__.it:25
         - GF_SMTP_FROM_ADDRESS=grafana-alert@__.__.it
      ports:
         - '3000:3000'
# TODO google/cadvisor 
# telegram bot alert
   alertmanager-bot:
      image: metalmatze/alertmanager-bot
      environment:
         - ALERTMANAGER_URL=http://hostname:9093
         - https_proxy=http://proxy._._.it:8080
         - http_proxy=http://proxy._._.it:8080
         - LISTEN_ADDR=0.0.0.0:8088
         - BOLT_PATH=/data/bot.db
         - STORE=bolt
         - TELEGRAM_ADMIN=99999
         - TELEGRAM_TOKEN=999999:QQ__QQ
         - TEMPLATE_PATHS=/templates/default.tmpl
      volumes:
         - /dati/prometheus/alertmanager-bot:/data
      ports:
         - '8088:8088'

Per verificare che il file sia correttamente digitato eseguire:

$ docker-compose up -d

Configurazione come servizio di sistema

/etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus monitoring docker container
After=docker.service
BindsTo=docker.service

[Service]
Restart=always
WorkingDirectory=/dati/prometheus/

# Ubuntu
ExecStart=/usr/local/bin/docker-compose up
ExecStop=/usr/local/bin/docker-compose down

# CentOS
#ExecStart=/usr/bin/docker-compose up
#ExecStop=/usr/bin/docker-compose down

[Install]
WantedBy=multi-user.target

Impostazione e controllo funzionamento come servizio di sistema ed analisi avvio

# impostazione
$ sudo systemctl daemon-reload
$ sudo systemctl enable prometheus
# controllo servizi
$ sudo systemctl start prometheus
$ sudo systemctl stop prometheus
$ sudo systemctl restart prometheus

Analisi avvio dei container docker (esempio)

# lista dei container
$ sudo docker ps -a

CONTAINER ID        IMAGE                                COMMAND                  CREATED              STATUS                       PORTS                    NAMES
1abc1e837fa3        prom/node-exporter                   "/bin/node_exporter"     About a minute ago   Up About a minute            0.0.0.0:9100->9100/tcp   prometheus-node-exporter
becfde05313f        prom/prometheus:latest               "/bin/prometheus -..."   About a minute ago   Up About a minute            0.0.0.0:9090->9090/tcp   prometheus
53b3323e7d52        prom/alertmanager                    "/bin/alertmanager..."   About a minute ago   Up About a minute            0.0.0.0:9093->9093/tcp   prometheus-alertmanager
75e610620c07        prom/prometheus                      "/bin/prometheus -..."   25 hours ago         Exited (2) 25 hours ago    

# ATTENZIONE il container 75e610620c07 è in stato "Exited" i logs sono da verificare
$ sudo docker logs 75e610620c07

Configurazione Grafana

TODO - TODO

Ambiente a runtime

Console Prometheus

http://IP_HOST:9090

Console AlertManager

http://IP_HOST:9093

Console NodeExporter

http://IP_HOST:9100

Console Grafana

http://IP_HOST:3000

TODO - Configurazione Aler swagger stats / node-explorer / ecc. ecc.

Tutorial per PromQL per creazioni alert (note rate e irate)

Regola "too many 5xx error" legata ad una finestra temporale

# rapporto fra il "burst" delle chiamate con errore (5xx) e le chiamate totali
# l'alert si attiva se la % è maggiore di 5 (ovvero se gli numero di errori 
# sono più del 5% delle chiamate totali nell'ultimo minuto

sum(rate(api_request_total{code=~"^5.."}[1m])) / sum(rate(api_request_total[1m])) * 100 > 5

rate() ritorna per secondo la media del cambiamento sul tuo intervallo usando il primo e l'ultimo punto dell'intervallo qualsiasi esso sia e qualsiasi timestamp abbia, può nascondere degli spikes (picchi di attività)
irate() come rate() ma è calcolato sugli ultimi due data point

Suggerimento : intervallo minimo è doppio di quello si scrape ad esempio

scrape : 20sec interval : 1m

Sidebar: The minimum range interval versus the minimum query step Suppose that you have a continuously updating metric that Prometheus scrapes every fifteen seconds. To do a rate() or irate() of this, you need at least two metric points and thus a range interval of thirty seconds (at least; in practice you need a somewhat larger interval). However, you get a new metric value roughly every fifteen seconds and thus a potentially new rate of change every fifteen seconds as well (due to your new metric point). This means it's reasonable to look at a graph with a query step of fifteen seconds and an interval of thirty seconds.
FINE DOCUMENTO

Aumentare un set di immagini per Ssd Mobilenet

noreply@blogger.com (Paulo) — Sun, 28 Oct 2018 12:25:00 +0000

Insieme minimo
Imgaug
LabelImg
Script su github
Esempio su Codelab

Cryptocurrencies wallets for beginner [GUIDE] — Steemit

noreply@blogger.com (Paulo) — Sun, 31 Dec 2017 12:40:00 +0000

https://steemit.com/wallet/@mrshinzen/cryptocurrencies-wallets-for-beginner-guide

Cryptocurrencies wallets for beginner [GUIDE] — Steemit

noreply@blogger.com (Paulo) — Sat, 30 Dec 2017 21:27:00 +0000

https://steemit.com/wallet/@mrshinzen/cryptocurrencies-wallets-for-beginner-guide

Zero to Hero: Guide to Object Detection using Deep Learning: Faster R-CNN,YOLO,SSD – CV-Tricks.com

noreply@blogger.com (Paulo) — Sat, 30 Dec 2017 08:57:00 +0000

http://cv-tricks.com/object-detection/faster-r-cnn-yolo-ssd/

Deep Learning Achievements Over the Past Year – Stats and Bots

noreply@blogger.com (Paulo) — Sat, 30 Dec 2017 08:54:00 +0000

https://blog.statsbot.co/deep-learning-achievements-4c563e034257?mkt_tok=eyJpIjoiWVRVMk1qWXdPRGcxWkRkaSIsInQiOiJUdjZ5NGx1MmlcL05vekVjNWpmRVRUWVRobHhtQ0N0QmxkS2xZbHZPSGV1Y09GQ0pvZVR2NUh6SDFyTkNPT2xPMlwvZDNlN055RGN2RWRGQVRmZmxBQUZEcEo1aGppTEg2amU4K29oQ1NoejBtWHVoXC9XQzQ5N0JGYVwvSERUTDNXQWgifQ%3D%3D

Neural Networks with Google CoLaboratory | Artificial Intelligence Getting started

noreply@blogger.com (Paulo) — Thu, 28 Dec 2017 14:59:00 +0000

https://medium.com/@howal/neural-networks-with-google-colaboratory-artificial-intelligence-getting-started-713b5eb07f14

Deep Learning Trading and Hedge Funds | Toptal

noreply@blogger.com (Paulo) — Wed, 27 Dec 2017 21:39:00 +0000

https://www.toptal.com/deep-learning/deep-learning-trading-hedge-funds

Machine Learning Part 9: Regularization - Chun’s Machine Learning Page

noreply@blogger.com (Paulo) — Tue, 26 Dec 2017 20:37:00 +0000

https://chunml.github.io/ChunML.github.io/tutorial/Regularization/

Deep Learning Course - TOP 10

noreply@blogger.com (Paulo) — Thu, 07 Dec 2017 07:55:00 +0000

https://edgylabs.com/top-10-free-deep-learning-moocs

calendar heatmap

noreply@blogger.com (Paulo) — Thu, 07 Dec 2017 07:38:00 +0000

https://github.com/DKirwan/calendar-heatmap

How to train your own Object Detector with TensorFlow’s Object Detector API

noreply@blogger.com (Paulo) — Sat, 25 Nov 2017 13:54:00 +0000

https://towardsdatascience.com/how-to-train-your-own-object-detector-with-tensorflows-object-detector-api-bec72ecfe1d9

SSD-Tensorflow/pascalvoc_to_tfrecords.py at master · balancap/SSD-Tensorflow · GitHub

noreply@blogger.com (Paulo) — Sat, 25 Nov 2017 13:50:00 +0000

https://github.com/balancap/SSD-Tensorflow/blob/master/datasets/pascalvoc_to_tfrecords.py

https://towardsdatascience.com/building-a-toy-detector-with-tensorflow-object-detection-api-63c0fdf2ac95

noreply@blogger.com (Paulo) — Thu, 23 Nov 2017 14:20:00 +0000

Step by Step TensorFlow Object Detection API Tutorial — Part 3: Creating Your Own Dataset

noreply@blogger.com (Paulo) — Wed, 22 Nov 2017 19:05:00 +0000

https://medium.com/@WuStangDan/step-by-step-tensorflow-object-detection-api-tutorial-part-3-creating-your-own-dataset-6369a4d30dfd

Augmentor/README.md at master · mdbloice/Augmentor · GitHub

noreply@blogger.com (Paulo) — Sun, 19 Nov 2017 11:41:00 +0000

https://github.com/mdbloice/Augmentor/blob/master/README.md

imgaug/README.md at master · aleju/imgaug · GitHub

noreply@blogger.com (Paulo) — Sun, 19 Nov 2017 11:40:00 +0000

https://github.com/aleju/imgaug/blob/master/README.md

Paradise papers

noreply@blogger.com (Paulo) — Thu, 09 Nov 2017 19:55:00 +0000

Ciao,

hai seguito la vicenda dei Paradise Papers? Ho appena firmato una lettera aperta scritta da Gordon Brown, ex primo ministro inglese, per la chiusura dei paradisi fiscali. La consegnerà direttamente al G20! Firmala anche tu, la trovi qui: https://secure.avaaz.org/campaign/it/paradise_papers_loc/?kFmlKbb

Inviato dal mio smartphone LG