Digital (dis)content

I've moved :)

noreply@blogger.com (Julien) — Sat, 05 May 2018 17:06:00 +0000

This blog has now moved to http://medium.com/@julsimon.

Amazon AI services @ Javascript Meetup (Wellington)

noreply@blogger.com (Julien) — Fri, 17 Mar 2017 09:27:00 +0000

Building serverless apps with Node.js @ NZ.js conference (Wellington)

noreply@blogger.com (Julien) — Fri, 17 Mar 2017 09:24:00 +0000

Advanced Task Scheduling with Amazon ECS @ AWS User Group Singapore

noreply@blogger.com (Julien) — Fri, 17 Mar 2017 09:22:00 +0000

Exploring the GDELT data set with Amazon Athena

noreply@blogger.com (Julien) — Fri, 03 Mar 2017 10:54:00 +0000

The Global Database of Events, Language and Tone (GDELT) Project monitors the world's broadcast, print, and web news from nearly every corner of every country in over 100 languages and identifies the people, locations, organisations, counts, themes, sources, emotions, counts, quotes, images and events driving our global society every second of every day.

Data set v1.0 is publicly available in S3 and I figured that this would be as good an excuse as any to play with my new favorite analytics service, Amazon Athena. Yes, Redshift, I still love you, baby ;)

The data set contains (at the time to writing) 1,542 uncompressed CSV files: 58 columns, 440+ million lines, 140+ GB. It is updated daily.

Data is formatted according to a couple of specs (here and here, PDF). A few lookup tables are also available here (country codes, organisation names, etc.).

All right, let's get to business. As you may know, Athena is able to query data hosted in S3 : no infrastructure to launch or manage, no data preparation, no loading time. All we have to do is create a table using the Hive DDL.

This is how we do it. All scripts and queries are available on Github, including how to create lookup tables.

Quite a mouthful, but pretty straightforward: just read the doc, define as many columns as needed with the right type and point to the S3 bucket holding all files. One immediate benefit of this is that whenever we run queries, Athena will automatically use all available files, including the additional one that's delivered daily. Zero work!

OK, let's run some queries. I'm using SQL Workbench/J with the Athena JDBC driver.

None of these took more than 30 seconds and that's with uncompressed CSV, the least performing data format possible. Converting the data set columnar formats such as Parquet or Orc would yield a massive improvement, but it's extra work, so why bother? ;)

That's it for today. Thanks for reading!

Amazon Polly: "Hello World"... literally!

noreply@blogger.com (Julien) — Thu, 01 Dec 2016 02:05:00 +0000

Today, AWS announced a new text-to-speech service, called Polly.

Well... I had to try it :D

Here's a *very* basic example in Python. There is much more to Polly, but this should get you started. You can list all available voices with 'aws polly describe-voices'.

And yes, I'm sure there are more clever ways to play sound files in Python, but they're beyond my weak skills, so there ;)

Very fun service. I see a lot of chatty build servers on the horizon, yelling in German at careless developers. Oh yes, pure bliss :D

Till next time, keep rockin'.

A hands-on look at the Amazon Rekognition API

noreply@blogger.com (Julien) — Wed, 30 Nov 2016 17:16:00 +0000

Amazon Rekognition is a Deep Learning based image analysis service. Don't worry though, you won't have to wade through Machine Learning / Deep Learning mumbo jumbo to work with Recognition. Quite the contrary, as Rekognition provides a very easy-to-use API.

It allows developers to:

detect thousands of objects and scenes;
analyze faces;
compare two faces to measure similarity;
build face collections and match faces against these collections.

As usual, this service can be used with the AWS CLI (as in 'aws rekognition' ), or with one of our language SDKs. I'll show you some CLI examples first and then we'll use the popular Python SDK, aka boto3.

First things first: how do we send images for processing? Two options: send the image as a byte blob or put it in S3. I suspect the most of use will use the second option, so that's what I'll use. Time to play!

$ aws rekognition detect-faces --image "S3Object={Bucket="jsimon-public", Name="julien1.jpg"}"

{
    "FaceDetails": [
        {
            "BoundingBox": {
                "Width": 0.3883333206176758,
                "Top": 0.12222222238779068,
                "Left": 0.33666667342185974,
                "Height": 0.2588889002799988
            },
            "Landmarks": [
                {
                    "Y": 0.23426248133182526,
                    "X": 0.46131378412246704,
                    "Type": "eyeLeft"
                },
                {
                    "Y": 0.22791674733161926,
                    "X": 0.5936729311943054,
                    "Type": "eyeRight"
                },
                {
                    "Y": 0.27828338742256165,
                    "X": 0.5404868721961975,
                    "Type": "nose"
                },
                {
                    "Y": 0.3229646682739258,
                    "X": 0.48395034670829773,
                    "Type": "mouthLeft"
                },
                {
                    "Y": 0.31654009222984314,
                    "X": 0.5957114696502686,
                    "Type": "mouthRight"
                }
            ],
            "Pose": {
                "Yaw": 4.216298580169678,
                "Roll": -4.777482509613037,
                "Pitch": -2.406636953353882
            },
            "Quality": {
                "Sharpness": 70.0,
                "Brightness": 65.17163848876953
            },
            "Confidence": 99.99468231201172
        }
    ],
    "OrientationCorrection": "ROTATE_0"
}

JSON, the cornerstone of any nutritious service. So, what do we have here? A face has been found with 99.99+% confidence. It's delimited by the BoundingBox coordinates (top left corner, face width, face height): these are fractional values with respect to the total height and width of the image. Eyes, nose and mouth have been located too (that's reassuring).

Now, let's see what Rekognition can tell us about this second picture.

$ aws rekognition detect-labels --image '{"S3Object":{"Bucket":"jsimon-public","Name":"julien2.jpg"}}'

{
    "Labels": [
        {
            "Confidence": 99.29261779785156,
            "Name": "Human"
        },
        {
            "Confidence": 99.2958984375,
            "Name": "People"
        },
        {
            "Confidence": 99.2958984375,
            "Name": "Person"
        },
        {
            "Confidence": 99.2667007446289,
            "Name": "Book"
        },
        {
            "Confidence": 99.2667007446289,
            "Name": "Text"
        },
        {
            "Confidence": 71.22590637207031,
            "Name": "Bookcase"
        },
        {
            "Confidence": 71.22590637207031,
            "Name": "Furniture"
        },
        {
            "Confidence": 71.22590637207031,
            "Name": "Shelf"
        },
        {
            "Confidence": 52.00172805786133,
            "Name": "Portrait"
        },
        {
            "Confidence": 52.00172805786133,
            "Name": "Selfie"
        }
    ]
}

With a very good level of confidence, this is the picture of a human with books on a bookshelf, possibly a portrait. A pretty good summary.

Let's compare the two previous pictures. Is this truly the same person? Spoiler: yes, although I look 15 years older on the first one. Note to self: no more promo shots after 36 sleepless hours :D

$ aws rekognition compare-faces --source-image '{"S3Object":{"Bucket":"jsimon-public","Name":"julien1.jpg"}}' --target-image '{"S3Object":{"Bucket":"jsimon-public","Name":"julien2.jpg"}}

{
    "FaceMatches": [
        {
            "Face": {
                "BoundingBox": {
                    "Width": 0.5596370100975037,
                    "Top": 0.1318063884973526,
                    "Left": 0.3889369070529938,
                    "Height": 0.5596370100975037
                },
                "Confidence": 99.98912811279297
            },
            "Similarity": 98.0
        }
    ],
    "SourceImageFace": {
        "BoundingBox": {
            "Width": 0.3883333206176758,
            "Top": 0.12222222238779068,
            "Left": 0.33666667342185974,
            "Height": 0.2588889002799988
        },
        "Confidence": 99.99468231201172
    }
}

Similarity is 98%. Jet lag or not, I'm always the same me.

See how simple this service is? I don't see how they could have made it easier. How long would it take to design, build and *train* something like this on your own? I have really no idea and to I don't intend to find out!

Enough CLI, let's switch to Python and run more visual examples. For this purpose, I've written a couple of scripts (available here), using boto3 and the Pillow image processing library.

In a nutshell:

rekognitionDetect.py bucket_name image [copy | nocopy ] : try to detect faces inside an image. If faces are found, each of them will be highlighted by a box and an updated image will be saved. The script will also report image labels and face information (gender, beard, glasses, etc.). Maximum number of labels and default confidence are respectively set to 10 and 75% by default.
rekognitionCompare.py bucket_name sourceImage targetImage [copy | nocopy ]: try to match a reference face to another image. If the face is found, it will be highlighted by a box and an updated image will be saved.

All images must be present with the same name both locally and in S3 . The last parameter for both scripts allows you to skip the copy to S3 if the file is already there.

Hopefully, the code reads like well-written prose (hi Uncle Bob). If not, blame jet lag (yes, it's the root of all evil). Anyway, there's nothing complicated here, I'm sure you'll figure it out in no time.

Let's play some more!

$ rekognitionDetect.py jsimon-public booth1.jpg nocopy

output file


Label Human, confidence: 99.3180236816

Label People, confidence: 99.3190917969
Label Person, confidence: 99.3190917969
Label Clothing, confidence: 92.1037216187
Label Overcoat, confidence: 92.1037216187
Label Suit, confidence: 92.1037216187
Label Computer, confidence: 76.0058441162
Label Electronics, confidence: 76.0058441162
Label LCD Screen, confidence: 76.0058441162
Label Laptop, confidence: 76.0058441162

*** Face 0 detected, confidence: 99.999671936
Gender: Male
HAPPY 96.4477920532
CALM 8.28260231018
CONFUSED 1.53788328171

*** Face 1 detected, confidence: 99.9654922485
Gender: Male
Beard
Mustache
HAPPY 98.5274353027
ANGRY 5.03668212891
CONFUSED 2.61067152023

*** Face 2 detected, confidence: 99.9955444336
Gender: Male
Eyeglasses
HAPPY 97.6237945557
ANGRY 1.31589770317
CALM 0.939458608627

*** Face 3 detected, confidence: 99.9996109009
Gender: Male
Eyeglasses
HAPPY 98.9962310791
SAD 11.4119710922
CONFUSED 1.69576406479

Say hi to Romain, Cédric and Damian, my friendly AWS colleagues. Rekognition sees 4 males, 1 with a beard, 2 with eyeglasses, all of them very happy... and I'm the calmest of the bunch, how about that. Amazingly, Rekognition manages to catch my hardly visible laptop (left edge of the picture, on the table).

Here's a tougher one (Hallo to my German friends).

$ rekognitionDetect.py jsimon-public oktoberfest.jpg nocopy

output file

Label People, confidence: 99.0898742676
Label Person, confidence: 99.0898971558
Label Human, confidence: 99.0639343262
Label Alcohol, confidence: 88.8537063599
Label Beverage, confidence: 88.8537063599
Label Drink, confidence: 88.8537063599
Label Crowd, confidence: 84.0972671509
Label Female, confidence: 84.0796279907
Label Girl, confidence: 84.0796279907

*** Face 0 detected, confidence: 99.9854202271
Gender: Male
HAPPY 60.5386123657
ANGRY 12.2481765747
DISGUSTED 2.10083723068

*** Face 1 detected, confidence: 99.9825744629
Gender: Female
HAPPY 98.0062866211
SURPRISED 10.8561573029
SAD 0.810676813126

*** Face 2 detected, confidence: 99.9904937744
Gender: Female
HAPPY 84.5134887695
SURPRISED 8.68589305878
ANGRY 1.35719180107

*** Face 3 detected, confidence: 99.9073257446
Gender: Male
Beard
Mustache
HAPPY 80.5190963745
SURPRISED 23.9800624847
ANGRY 1.17569565773

*** Face 4 detected, confidence: 99.9972229004
Gender: Male
Mustache
HAPPY 75.2949371338
CONFUSED 10.9511556625
DISGUSTED 1.91761255264

*** Face 5 detected, confidence: 99.9999771118
Gender: Male
HAPPY 35.9886474609
SURPRISED 3.75992059708
ANGRY 2.48707532883

*** Face 6 detected, confidence: 99.9915084839
Gender: Female
HAPPY 99.4766082764
CALM 0.791561603546
ANGRY 0.620931386948

*** Face 7 detected, confidence: 99.9998931885
Gender: Female
HAPPY 99.8826293945
SAD 7.21873044968
DISGUSTED 5.48685789108

*** Face 8 detected, confidence: 83.6580963135
Gender: Male
Eyeglasses
SAD 94.9213943481
SURPRISED 76.9153442383
HAPPY 8.52976131439

*** Face 9 detected, confidence: 99.9944610596
Gender: Male
HAPPY 27.327457428
DISGUSTED 26.6790218353
ANGRY 12.1302127838

*** Face 10 detected, confidence: 99.9998855591
Gender: Male
SURPRISED 99.2624435425
HAPPY 22.0922241211
SAD 6.69546127319

*** Face 11 detected, confidence: 99.9861831665
Gender: Male
SURPRISED 60.7816810608
SAD 7.07310438156
HAPPY 3.66672611237

*** Face 12 detected, confidence: 99.9990692139
Gender: Male
HAPPY 48.0631027222
SURPRISED 2.61369943619
CONFUSED 2.40399837494

*** Face 13 detected, confidence: 87.6368408203
Gender: Male
HAPPY 16.2307357788
SAD 14.2565965652
ANGRY 12.3210906982

*** Face 14 detected, confidence: 99.9553375244
Gender: Male
HAPPY 54.3005943298
DISGUSTED 5.99133396149
SURPRISED 3.63597273827

Wow, 15 people, including partial faces. All genders are correct. Emotions are mostly ok, but we definitely need to add 'DRUNK' to the list ;) The labels are spot on: a crowd of men and women drinking alcohol.

Let's try another one. Low res, low quality.

$ rekognitionDetect.py jsimon-public maradona.jpg nocopy

output file

Label People, confidence: 99.2043991089
Label Person, confidence: 99.2043991089
Label Human, confidence: 99.1917037964
Label Football, confidence: 97.2220993042
Label Soccer, confidence: 97.2220993042
Label Sport, confidence: 97.2220993042
Label American Football, confidence: 83.3328475952
Label Athlete, confidence: 78.3234786987

*** Face 0 detected, confidence: 99.963470459
Gender: Male
Mustache
SURPRISED 21.8802871704
CALM 17.4065952301
SAD 11.6566238403

*** Face 1 detected, confidence: 99.9813308716
Gender: Male
Eyeglasses
HAPPY 38.6969680786
ANGRY 6.79734945297
SURPRISED 2.61010527611

*** Face 2 detected, confidence: 99.9385604858
Gender: Male
SURPRISED 36.6970825195
SAD 7.66330337524
ANGRY 6.10639476776

*** Face 3 detected, confidence: 99.9514923096
Gender: Male
SAD 32.6836242676
DISGUSTED 4.55095767975
HAPPY 4.19711828232

*** Face 4 detected, confidence: 99.8046951294
Gender: Male
Beard
Mustache
SAD 46.0139579773
HAPPY 4.15547084808
DISGUSTED 0.981283187866

*** Face 5 detected, confidence: 99.2888412476
Gender: Male
SAD 90.2270889282
CALM 5.9303817749
HAPPY 3.26179981232

Labels are fine, except for 'American Football'. 83%??? Gimme a break, the training set needs more Soccer images! In addition, I don't think number 4 is wearing eyeglasses, but again this is a low res picture. Apart from this, Rekognition correctly picked up all faces and funny enough, the expressions make sense too: "sad" and "surprised" are definitely how these guys must have felt against the legendary Diego!

A last one for the road: how about this complex abstract-ish nighttime picture of Shinjuku?

$ rekognitionDetect.py jsimon-public shinjuku.jpg nocopy

output file

Label City, confidence: 88.4259796143 Label Downtown, confidence: 88.4259796143
Label Metropolis, confidence: 84.8462677002
Label Urban, confidence: 84.8462677002
Label Night, confidence: 69.7816467285
Label Outdoors, confidence: 69.7816467285
Label Shop, confidence: 68.228477478
Label Flyer, confidence: 60.3522796631
Label Poster, confidence: 60.3522796631
Label Neighborhood, confidence: 55.3994293213
*** Face 0 detected, confidence: 97.9367828369
Gender: Female SAD 46.1420478821 ANGRY 7.63346576691 HAPPY 6.28939962387

Note that I lowered the confidence threshold from 75% to 50% get more labels. Still, Rekognition does a good job. It also gets the girl's face and yes, she does look quite sad. The Anime face isn't detected but I guess this is the desired behavior.

Alright, enough detection. Let's now try to match faces, using some of the previous pictures as well as some new ones.

$ rekognitionCompare.py jsimon-public julien1.jpg julien2.jpg nocopy

Face match, confidence=99.9891281128, similarity=98.0

$ rekognitionCompare.py jsimon-public julien1.jpg booth1.jpg nocopy

Face match, confidence=99.999671936, similarity=96.0

$ rekognitionCompare.py jsimon-public julien1.jpg booth2.jpg nocopy

Face match, confidence=99.9991455078, similarity=84.0

$ rekognitionCompare.py jsimon-public julien1.jpg keynote.jpg no copy
Face match, confidence=99.9932250977, similarity=82.0

Quite good! The last one is particularly nice, given the distance, the angle and the poor lighting (see actual picture above).

These are just a few examples and I'm sure you can't wait to try your own. Hopefully this post has given you a visual, hands-on overview of the Recognition service and how user-friendly it is. I didn't cover face collections, but the API is pretty much what you'd expect (create, delete, etc.).

Feel free to explore and experiment. Until we meet again, keep rockin'.

The Lost Tales of Platform Design

noreply@blogger.com (Julien) — Sun, 20 Nov 2016 20:26:00 +0000

Disclaimer: all opinions are my own (what did you expect?).

This is a presentation I've meant to build for a while. I guess I was just waiting for an excuse to spend enough time to do so. My talk at The Family's Lion program (http://joinlion.co) gave me this excuse and I want to thank them for the opportunity.

In a nutshell, I'm sick and tired of the current state of Software Engineering. On second thought, it's probably a key reason why I decided to step away from CTO roles and try something different. No more babysitting "engineers" who think they know it all and have no interest or respect for the vast body of knowledge that has been built over the last 60 years.

Fuck trends, hipsters and "rockstars". We need professionals. Software Engineering is Engineering. Computer Science is Science. There are Proven-Ways-To-Do-Things-Right.

Some would say that this is precisely what management is about. Teaching, training, coaxing, blah blah blah. Maybe, maybe not. What I know is that my time is my most precious asset. I will never waste it again on developers who are not willing to listen and learn from people who have been there before. Especially when top management doesn't have the guts to hire, fire or reward accordingly (and of course they'll say otherwise).

This presentation sums up some of my core technical beliefs. Feel free to read, learn, and ask questions: I'll be delighted to help. Or keep drowning in your shit trendy code, you deserve it.

Maybe one day I'll find a team of like-minded code warriors, or maybe I'll feel like building one again. I'm not holding by breath. AWS is home :)

Keep rockin', my brothers. You know who you are.

The Lost Tales of Platform Design (November 2016) from Julien SIMON

3 webinaires AWS en français :)

noreply@blogger.com (Julien) — Sat, 05 Nov 2016 09:07:00 +0000

Voici 3 webinaires que j'ai eu le plaisir d'animer récemment et dont les vidéos sont disponibles sur YouTube:

Keynote @ DevOps D-Day, 07/10/2016, Marseille

noreply@blogger.com (Julien) — Sat, 05 Nov 2016 09:02:00 +0000

Slides: http://www.slideshare.net/JulienSIMON5/devops-with-amazon-web-services

Talk @ Paris Container Day 2016

noreply@blogger.com (Julien) — Thu, 11 Aug 2016 15:54:00 +0000

Paris Container Day 2016 : Déployer et scaler des clusters Docker avec Amazon ECS. (Julien Simon - @Amazon Web Services) from Captavideo on Vimeo.

2 talks @ AWS Summit Tel Aviv, June 2016

noreply@blogger.com (Julien) — Tue, 02 Aug 2016 13:52:00 +0000

Using Amazon CloudWatch Events, Lambda & Spark to Process EC2 Events

DevOps on AWS: Deep Dive on Continuous Delivery

Scale, baby, scale - Talk @ The Family, April 2016

noreply@blogger.com (Julien) — Tue, 02 Aug 2016 13:46:00 +0000

This is pretty much the same talk as the one I did at Devoxx FR, but in English this time :)

Big Data Combo @ Toulouse Data Science Meetup

noreply@blogger.com (Julien) — Thu, 19 May 2016 09:51:00 +0000

Amazon Redshift, Amazon QuickSight, Amazon Machine Learning and Amazon DSSTNE :)

My session @ Devoxx France, 22/04/2016

noreply@blogger.com (Julien) — Mon, 09 May 2016 13:36:00 +0000

Hands-on with AWS IoT @ Paris Robotics, AI and IoT meetup, 14/03/2016

noreply@blogger.com (Julien) — Sat, 16 Apr 2016 17:25:00 +0000

Hands-on with AWS IoT from Julien SIMON

Clustering Docker on AWS with Amazon ECR & ECS

noreply@blogger.com (Julien) — Sat, 16 Apr 2016 17:24:00 +0000

Slides are here.

Webinar @ Salon du Big Data, 02/03/2016

noreply@blogger.com (Julien) — Sat, 16 Apr 2016 17:19:00 +0000

Slides are here.

Devops with AWS @ AWS User Group Turkey, Istanbul, 13/04/2016

noreply@blogger.com (Julien) — Sat, 16 Apr 2016 17:17:00 +0000

AWS CodeCommit, CodeDeploy & CodePipeline from Julien SIMON

Workshop @ IoT World, 23/03/2016

noreply@blogger.com (Julien) — Sat, 16 Apr 2016 17:15:00 +0000

Workshop AWS IoT @ IoT World Paris from Julien SIMON

Keynote @ IoT World, 23/03/2016

noreply@blogger.com (Julien) — Sat, 16 Apr 2016 17:14:00 +0000

Keynote @ IoT World Paris from Julien SIMON

Workshop @ Salon du Big Data, Paris, 08/03/2016

noreply@blogger.com (Julien) — Wed, 09 Mar 2016 09:37:00 +0000

Building a data warehouse with Amazon Redshift … and a quick look at Amazon Machine Learning from Julien SIMON

My sincere thanks to the 250+ people who attended, stayed until the end and asked many smart questions :)

Talk @ Meetup AWS User Group, Nantes, 03/03/2016

noreply@blogger.com (Julien) — Fri, 04 Mar 2016 05:29:00 +0000

A 60-mn tour of AWS compute (March 2016) from Julien SIMON

Talk @ Meetup Toulouse DevOps, 24/02/2016

noreply@blogger.com (Julien) — Sun, 28 Feb 2016 14:20:00 +0000

AWS CloudFormation (February 2016) from Julien SIMON

Talk @ Meetup Docker Paris, 22/02/2016

noreply@blogger.com (Julien) — Tue, 23 Feb 2016 10:12:00 +0000

Docker Paris #29 from Julien SIMON