My notes

WebRTC one to one without signaling server

2020-05-25T00:00:00+03:00

To make peer to peer connection users should exchange so called sdp which determines a way to exchange data directly between browsers without servers

Usually in all demos there is websocket server which is used to handle this exchange but for better understanding of how it works we are going to exchange this by hands

How WebRTC connection established

Note that this is kind of simplified (there is no exchange of ice candidates) but in general it is how it works

demo

So from initiator side we are:

const offer = await connection.createOffer()
await connection.setLocalDescription(offer)

Then sending this offer to peer and on his side we are:

await connection.setRemoteDescription(offer)
const answer = await connection.createAnswer()
await connection.setLocalDescription(answer)

Then sending this answer back to initiator and on his side we are:

await connection.setRemoteDescription(answer)

Note that I have missed ice candidates exchanges here to reduce example size and give better understanding of what is going on

And here is full code

<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>webrtc</title>
</head>
<body>
    <script>
        let channel = null

        const connection = new RTCPeerConnection({ iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] })

        connection.ondatachannel = event => {
            console.log('ondatachannel')
            channel = event.channel
            // channel.onopen = event => console.log('onopen', event);
            // channel.onmessage = event => console.log('onmessage', event);
            channel.onmessage = event => alert(event.data)
        }

        connection.onconnectionstatechange = event => document.getElementById("connectionState").innerText = connection.connectionState // console.log('onconnectionstatechange', connection.connectionState)
        connection.oniceconnectionstatechange = event => document.getElementById("iceConnectionState").innerText = connection.iceConnectionState // console.log('oniceconnectionstatechange', connection.iceConnectionState)

        async function step_1_initiator_create_offer() {
            channel = connection.createDataChannel('data')
            // channel.onopen = event => console.log('onopen', event)
            // channel.onmessage = event => console.log('onmessage', event)
            channel.onmessage = event => alert(event.data)

            connection.onicecandidate = event => {
                // console.log('onicecandidate', event)
                if (!event.candidate) {
                    document.getElementById("createdOffer").value = JSON.stringify(connection.localDescription)
                    document.getElementById("createdOffer").hidden = false
                }
            }

            const offer = await connection.createOffer()
            await connection.setLocalDescription(offer)
        }

        async function step_2_accept_remote_offer() {
            const offer = JSON.parse(document.getElementById("remoteOffer").value)
            await connection.setRemoteDescription(offer)
        }

        async function step_3_create_answer() {
            connection.onicecandidate = event => {
                // console.log('onicecandidate', event)
                if (!event.candidate) {
                    document.getElementById("createdAnswer").value = JSON.stringify(connection.localDescription)
                    document.getElementById("createdAnswer").hidden = false
                }
            }
            
            const answer = await connection.createAnswer()
            await connection.setLocalDescription(answer)
        }

        async function step_4_accept_answer() {
            const answer = JSON.parse(document.getElementById("remoteAnswer").value)
            await connection.setRemoteDescription(answer)
        }

        async function send_text() {
            const text = document.getElementById("text").value

            channel.send(text)
        }
    </script>

    <table width="100%" border="1">
        <tr>
            <th>#</th>
            <th>initiator</th>
            <th>peer</th>
        </tr>
        <tr>
            <td>step 1</td>
            <td>
                <input type="button" value="create offer" onclick="step_1_initiator_create_offer()">
                <input id="createdOffer" type="text" hidden>
            </td>
            <td></td>
        </tr>
        <tr>
            <td>step 2</td>
            <td></td>
            <td>
                <input id="remoteOffer" type="text" placeholder="offer from initiator">
                <input type="button" value="accept offer" onclick="step_2_accept_remote_offer()">
            </td>
        </tr>
        <tr>
            <td>step 3</td>
            <td></td>
            <td>
                <input type="button" value="create answer" onclick="step_3_create_answer()">
                <input id="createdAnswer" type="text" hidden>
            </td>
        </tr>
        <tr>
            <td>step 4</td>
            <td>
                <input id="remoteAnswer" type="text" placeholder="answer from peer">
                <input type="button" value="accept answer" onclick="step_4_accept_answer()">
            </td>
            <td></td>
        </tr>
    </table>
    <hr>
    <input id="text" type="text">
    <input type="button" value="send" onclick="send_text()">
    <hr>
    <table border="1">
        <tr>
            <th colspan="2">connection</th>
        </tr>
        <tr>
            <th>connectionState</th>
            <td id="connectionState">unknown</td>
        </tr>
        <tr>
            <th>iceConnectionState</th>
            <td id="iceConnectionState">unknown</td>
        </tr>
    </table>
</body>
</html>

custom bash autocomplete

2020-01-07T00:00:00+02:00

Often I do need to check which containers in a given kubernetes deployment in which status, to retrieve such info following command might be used:

kubectl get po -l app=my-awesome-app -o json | jq "[.items[0].status.containerStatuses[] | {name: .name, ready: .ready, state: .state | keys[0]}]"

which will return something like:

[
  {
    "name": "exporter",
    "ready": true,
    "state": "running"
  },
  {
    "name": "nginx",
    "ready": true,
    "state": "running"
  },
  {
    "name": "proxy",
    "ready": true,
    "state": "running"
  }
]

as you can imagine remembering such command is not possible, so we are going to create a function in ~/.bash_profile:

po-kube() {
    kubectl get po -l app=$1 -o json | jq "[.items[0].status.containerStatuses[] | {name: .name, ready: .ready, state: .state | keys[0]}]"
}

and from now on we cat instead run:

po-kube my-awesome-app

but what if you have many deploymnets with long names?

here are pretty nice documentation of how bash autocompletion works and how to create custom bash autocomplete

In my case I added /usr/local/etc/bash_completion.d/po-kube:

_po-kube() {
    local cur prev opts
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    opts=$(kubectl get deployment --template '{{range .items}}{{.metadata.name}}{{" "}}{{end}}')

    if [[ ${cur} == * ]] ; then
        COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
        return 0
    fi
}
complete -F _po-kube po-kube

and from now on I can do something like:

po-kube s[TAB]
sms-api spam-api

kafka command line tools

2020-01-03T00:00:00+02:00

First of all we gonna need kafka itself, e.g.:

docker run -it --rm --name=kafka -e SAMPLEDATA=0 -e RUNNING_SAMPLEDATA=0 -e RUNTESTS=0 -e FORWARDLOGS=0 -e ADV_HOST=127.0.0.1 -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 -p 9092:9092 -p 9581-9585:9581-9585 lensesio/fast-data-dev:2.3.0

wget https://raw.githubusercontent.com/confluentinc/examples/5.3.1-post/cp-all-in-one/docker-compose.yml
docker-compose up -d

run everything by hands on your own like described in quickstart

get confluent.cloud

Kafka topics

The most basic and needed

List topics

kafka-topics --bootstrap-server localhost:9092 --list

Create topic

kafka-topics --bootstrap-server localhost:9092 --create --topic demo2 --partitions 3 --replication-factor 1

Delete topic

kafka-topics --bootstrap-server localhost:9092 --delete --topic demo2

Console producer and consumers

Here are examples for following use cases:

simple without key
simple with string key
simple with integer key
json without key
json with string key
json with ingeteger key
json with json key
avro without key
avro with string key
avro with integer key
avro with avro key

By deafult in all following examples messages delimited by new line, e.g. start producer, type something, press enter.

All follogin examples are run agains

docker run -it --rm --name=kafka -e SAMPLEDATA=0 -e RUNNING_SAMPLEDATA=0 -e RUNTESTS=0 -e FORWARDLOGS=0 -e ADV_HOST=127.0.0.1 -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 -p 9092:9092 -p 9581-9585:9581-9585 lensesio/fast-data-dev:2.3.0

String producer and consumer

Simple without key

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic SimpleWithoutKey --partitions 3 --replication-factor 1

Start kafka-console-producer which will produce simple string messages

docker exec -it kafka kafka-console-producer --broker-list localhost:9092 --topic SimpleWithoutKey

Start kafka-console-consumer to consume simple string messages

docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic SimpleWithoutKey --from-beginning

Produce simple messages like:

hello
world

And you should see them in consumer as:

hello
world

Simple with string key

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic SimpleWithStringKey --partitions 3 --replication-factor 1

Start kafka-console-producer which will produce simple string messages with string key

docker exec -it kafka kafka-console-producer --broker-list localhost:9092 --topic SimpleWithStringKey --property parse.key=true --property key.separator=:

Notes:

--property parse.key=true our consumer will expect us to enter key along side value
--property key.separator=: is optional and by default is space

Start kafka-console-consumer to consume simple string messages with string key

docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic SimpleWithStringKey --property print.key=true --from-beginning

Notes:

--property print.key=true will print key

Produce messages like:

1:one
2:two

And you should see them in consumer as:

1	one
2	two

If you try to produce message without key you should see an error:

>message without key
org.apache.kafka.common.KafkaException: No key found on line 3: acme
	at kafka.tools.ConsoleProducer$LineMessageReader.readMessage(ConsoleProducer.scala:265)
	at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:54)
	at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)

Simple with integer key

Not fully possible at moment, here are some links:

The problem is that no matter what you will pass to console producer it still will send bytes

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic SimpleWithIntKey --partitions 3 --replication-factor 1

Start kafka-console-producer which will produce simple string messages with integer key

docker exec -it kafka kafka-console-producer --broker-list localhost:9092 --topic SimpleWithIntKey --property parse.key=true --property key.serializer=org.apache.kafka.common.serialization.IntegerDeserializer --property value.serializer=org.apache.kafka.common.serialization.StringDeserializer --property key.separator=:

Notes:

--property key.serializer=org.apache.kafka.common.serialization.IntegerDeserializer defines which deserializer should be used for key
--property value.serializer=org.apache.kafka.common.serialization.StringDeserializer defines which deserializer should be user for value
both not being applied

Start kafka-console-consumer to consume simple string messages with integer key

docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic SimpleWithIntKey --property print.key=true --from-beginning --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer --property value.deserializer=org.apache.kafka.common.serialization.StringDeserializer --skip-message-on-error

Notes:

key.deserializer=org.apache.kafka.common.serialization.StringDeserializer we are forced to use string instead of integer deserializer here, otherwise will receive an error ERROR Error processing message, skipping this message: (kafka.tools.ConsoleConsumer$) org.apache.kafka.common.errors.SerializationException: Size of data received by IntegerDeserializer is not 4
--skip-message-on-error do not crash on bad message, just skip it

Produce messages like:

1:one
2:two

And you should see them in consumer as:

1	one
2	two

Json producer and consumer

Json without key

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic JsonWithoutKey --partitions 3 --replication-factor 1

Start kafka-console-producer which will produce json messages

docker exec -it kafka kafka-console-producer --broker-list localhost:9092 --topic JsonWithoutKey

Note that like in previous example with integer key, kafka-console-producer does not respect given serializers so we will just put string which looks like json but still sent as bytes

Start kafka-console-consumer to consume json messages

docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic JsonWithoutKey --property value.deserializer=org.apache.kafka.connect.json.JsonDeserializer --skip-message-on-error --from-beginning --property print.timestamp=true

Produce json messages like:

{"foo": "bar"}
{"acme": 42}

And you should see them like:

CreateTime:1578081298745        {"foo":"bar"}
CreateTime:1578081304001        {"acme":42}

There is not checks in producer but if you send something wrong you will see an error in consumer

CreateTime:1578081353956        [2020-01-03 19:55:54,970] ERROR Error processing message, skipping this message:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.SerializationException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'foo': was expecting 'null', 'true', 'false' or NaN
 at [Source: (byte[])"foo"; line: 1, column: 7]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'foo': was expecting 'null', 'true', 'false' or NaN
 at [Source: (byte[])"foo"; line: 1, column: 7]

Json with string key

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic JsonWithStringKey --partitions 3 --replication-factor 1

Start kafka-console-producer which will produce json messages with string key

docker exec -it kafka kafka-console-producer --broker-list localhost:9092 --topic JsonWithStringKey --property parse.key=true --property key.separator=:

Start kafka-console-consumer to consume json messages with string key

docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic JsonWithStringKey --property print.key=true --from-beginning --property value.deserializer=org.apache.kafka.connect.json.JsonDeserializer

Produce messages:

1:{"foo":"bar"}
2:{"acme":42}

And you should see:

1       {"foo":"bar"}
2       {"acme":42}

Note that there is the same problem with keys as in previous examples, and you can not force integer key.

Json with json key

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic JsonWithJsonKey --partitions 3 --replication-factor 1

Start kafka-console-producer which will produce json messages with json keys

docker exec -it kafka kafka-console-producer --broker-list localhost:9092 --topic JsonWithJsonKey --property parse.key=true --property key.separator="|"

Start kafka-console-consumer to consume json messages

docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic JsonWithJsonKey  --property value.deserializer=org.apache.kafka.connect.json.JsonDeserializer --property key.deserializer=org.apache.kafka.connect.json.JsonDeserializer --skip-message-on-error --from-beginning --property print.key=true

Produce messages:

{"id":1}|{"foo":"bar"}
{"id":2}|{"acme":42}

And you should see:

{"id":1}        {"foo":"bar"}
{"id":2}        {"acme":42}

If you will produce bad key or value you will get:

ERROR Error processing message, skipping this message:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.SerializationException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'foo': was expecting 'null', 'true', 'false' or NaN
 at [Source: (byte[])"foo"; line: 1, column: 7]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'foo': was expecting 'null', 'true', 'false' or NaN
 at [Source: (byte[])"foo"; line: 1, column: 7]

Avro producer and consumer

Avro without key

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic AvroWithoutKey --partitions 3 --replication-factor 1

Start kafka-avro-console-producer to produce avro messages

docker exec -it kafka kafka-avro-console-producer --broker-list localhost:9092 --topic AvroWithoutKey --property value.schema='{"type":"record","name":"AvroWithoutKey","fields":[{"name":"foo","type":"string"}]}'

Note that from now on we are using kafka-avro-console-producer instead of kafka-console-producer which has few additional properties like --property value.schema='{"type":"record","name":"AvroWithoutKey","fields":[{"name":"foo","type":"string"}]}' messages published via this consumer will be validated against given schema. Also note that this producer does not show > symbol, so do not wait for it.

Start kafka-avro-console-consumer to consume avro messages

docker exec -it kafka kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic AvroWithoutKey --from-beginning

Try sending something like:

{"foo":"hello"}
{"foo":"world"}

and you should see exactly the same output in consumer.

If you will try send something wrong you will receive an error:

{"acme":42}
org.apache.kafka.common.errors.SerializationException: Error deserializing json {"acme":42} to Avro of schema {"type":"record","name":"AvroWithoutKey","fields":[{"name":"foo","type":"string"}]}
Caused by: org.apache.avro.AvroTypeException: Expected field name not found: foo
        at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
        at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)

but still if something you are sending is schema compatible everything should be ok, try sending {"foo":"bar","acme":42} and you will receive {"foo":"bar"} in your consumer

Avro with string key

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic AvroWithStringKey --partitions 3 --replication-factor 1

Start kafka-avro-console-producer to produce avro messages with primitive string key

docker exec -it kafka kafka-avro-console-producer --broker-list localhost:9092 --topic AvroWithStringKey --property value.schema='{"type":"record","name":"AvroWithStringKey","fields":[{"name":"foo","type":"string"}]}' --property parse.key=true --property key.schema='{"type":"string"}' --property key.separator=" "

Not that we have added --property key.schema='{"type":"string"}' which allow us to use primitives as key and they still will be validated.

Start kafka-avro-console-consumer to consume avro messages with string keys

docker exec -it kafka kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic AvroWithStringKey --from-beginning --property print.key=true --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer

Try send something like:

"one" {"foo":"1"}
"two" {"foo":"2"}

and you should get:

one     {"foo":"1"}
two     {"foo":"2"}

Do not forget to wrap key with double quotes otherwise you will get an error:

org.apache.kafka.common.errors.SerializationException: Error deserializing json one to Avro of schema "string"
Caused by: org.codehaus.jackson.JsonParseException: Unexpected character ('o' (code 111)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: java.io.StringReader@3feb2dda; line: 1, column: 2]

Avro with int key

Does not work, in example below after trying to send 1|{"foo":"bar"} receiving an error:

org.apache.kafka.common.errors.SerializationException: Error deserializing json 1|{"foo":"hello"} to Avro of schema {"type":"record","name":"AvroWithIntKey","fields":[{"name":"foo","type":"int"}]}
Caused by: org.apache.avro.AvroTypeException: Expected record-start. Got VALUE_NUMBER_INT

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic AvroWithIntKey --partitions 3 --replication-factor 1

Start kafka-avro-console-producer to produce avro messages with integer keys

docker exec -it kafka kafka-avro-console-producer --broker-list localhost:9092 --topic AvroWithIntKey --property value.schema='{"type":"record","name":"AvroWithIntKey","fields":[{"name":"foo","type":"int"}]}' --property key.separator="|"

Start kafka-avro-console-consumer to consume avro messages

docker exec -it kafka kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic AvroWithIntKey --from-beginning

Avro with avro key

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic AvroWithAvroKey --partitions 3 --replication-factor 1

Start kafka-avro-console-producer which will produce avro messages with avro keys

docker exec -it kafka kafka-avro-console-producer --broker-list localhost:9092 --topic AvroWithAvroKey --property value.schema='{"type":"record", "name": "AvroWithAvroKey", "fields":[{"name":"foo","type":"string"}]}' --property parse.key=true --property key.schema='{"type":"record","name": "key", "fields":[{"name":"id","type":"int"}]}' --property key.separator=" "

Start kafka-avro-console-consumer to consume avro messages with avro keys

docker exec -it kafka kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic AvroWithAvroKey --from-beginning --property print.key=true

Try send

{"id":1} {"foo":"hello"}
{"id":2} {"foo":"world"}

and you should receive

{"id":1}        {"foo":"hello"}
{"id":2}        {"foo":"world"}

if you will try send wrong key like {"id":"guid"} you will receive an error

org.apache.kafka.common.errors.SerializationException: Error deserializing json {"id":"guid"} to Avro of schema {"type":"record","name":"key","fields":[{"name":"id","type":"int"}]}
Caused by: org.apache.avro.AvroTypeException: Expected int. Got VALUE_STRING

Confluent Cloud

If you are using confluent.cloud from confluent.io you still able to do all this with few more params added for commands

More examples can be found here

Topics

You gonna need properties file which you can retrieve from https://confluent.cloud/environments/*****/clusters/***-*****/integrations/clients#java by navigating cluster then “CLI & client configuration”

cloud.properties

bootstrap.servers=xxx-xxxxx.us-east1.gcp.confluent.cloud:9092
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="xxxxxxx" password\="xxxxxxx";

docker run -it --rm -v $PWD/cloud.properties:/cloud.properties confluentinc/cp-kafka:5.3.2 kafka-topics \
  --bootstrap-server xxx-xxxxx.us-east1.gcp.confluent.cloud:9092 \
  --command-config /cloud.properties \
  --list

all other commands will work as expected

Produce & consume simple messages

If you are going to run simple producer without avro and schema registry then properties file from previous example should be enough

Create topic

docker run -it --rm -v $PWD/cloud.properties:/cloud.properties confluentinc/cp-kafka:5.3.2 kafka-topics \
  --bootstrap-server xxx-xxxxx.us-east1.gcp.confluent.cloud:9092 \
  --command-config /cloud.properties \
   --create --topic simple1 --partitions 3 --replication-factor 3

Start producer

docker run -it --rm -v $PWD/cloud.properties:/cloud.properties confluentinc/cp-kafka:5.3.2 kafka-console-producer \
  --broker-list xxx-xxxxx.us-east1.gcp.confluent.cloud:9092 \
  --producer.config /cloud.properties \
  --topic simple1

Start consumer

docker run -it --rm -v $PWD/cloud.properties:/cloud.properties confluentinc/cp-kafka:5.3.2 kafka-console-consumer \
  --bootstrap-server xxx-xxxxx.us-east1.gcp.confluent.cloud:9092 \
  --consumer.config /cloud.properties \
  --topic simple1

Cleanup

docker run -it --rm -v $PWD/cloud.properties:/cloud.properties confluentinc/cp-kafka:5.3.2 kafka-topics \
  --bootstrap-server xxx-xxxxx.us-east1.gcp.confluent.cloud:9092 \
  --command-config /cloud.properties \
   --delete --topic simple1

Note that you are not restricted to strings only, you can also use all previous examples with different keys and json

Produce & consume AVRO messages in confluent.cloud

Create topic

docker run -it --rm -v $PWD/cloud.properties:/cloud.properties confluentinc/cp-kafka:5.3.2 kafka-topics \
  --bootstrap-server xxx-xxxxx.us-east1.gcp.confluent.cloud:9092 \
  --command-config /cloud.properties \
   --create --topic avro1 --partitions 3 --replication-factor 3

Start producer

docker run -it --rm -v $PWD/cloud.properties:/cloud.properties confluentinc/cp-schema-registry:5.3.2 kafka-avro-console-producer \
    --broker-list xxx-xxxxx.us-east1.gcp.confluent.cloud:9092 \
    --topic avro1 \
    --property value.schema='{"type":"record","name":"AvroWithoutKey","fields":[{"name":"foo","type":"string"}]}' \
    --producer.config /cloud.properties \
    --property schema.registry.url="https://xxxx-xxxxx.us-east1.gcp.confluent.cloud" \
    --property schema.registry.basic.auth.user.info="xxxxxxx:xxxxxxx" \
    --property basic.auth.credentials.source=USER_INFO

Start consumer

docker run -it --rm -v $PWD/cloud.properties:/cloud.properties confluentinc/cp-schema-registry:5.3.2 kafka-avro-console-consumer \
    --bootstrap-server xxx-xxxxx.us-east1.gcp.confluent.cloud:9092 \
    --topic avro1 \
    --from-beginning \
    --value-deserializer io.confluent.kafka.serializers.KafkaAvroDeserializer \
    --key-deserializer org.apache.kafka.common.serialization.StringDeserializer \
    --consumer.config /cloud.properties \
    --property schema.registry.url="https://xxxx-xxxxx.us-east1.gcp.confluent.cloud" \
    --property schema.registry.basic.auth.user.info="xxxxxxx:xxxxxxx" \
    --property basic.auth.credentials.source=USER_INFO

Notes:

we are using another docker image confluentinc/cp-schema-registry:5.3.2 because of kafka avro console consumer and producer
cloud.properties is still enough but schema registry settings should be passed via command line arguments

Kafka connect

We are going to use kafka connect to:

produce predefined messages from a file to replay some sequence of events
produce generated messages to get millions of them for test purposes
have sample sink connector to save messages to a file

All example will be made as standalone worker which should not be used in production and used here only because of its easy to use

At very end worker command looks liks like this: connect-standalone worker.properties task1.properties task2.properties where worker.properties contains configuration for worker itself and some defaults for tasks, taskX.properties is task configuration, you can have many of them, for example your worker might have few tasks which will produce messages from different files and one task to consume them into elasticsearch.

Tasks producing data into kafka called source, tasks consuming data from kafka called sink.

Be aware of advertised hosts and rest ports, if you are connecting to dockerized kafka which have localhost as advertised host from your worker which is also run in container nothing will work, use --net=host for such scenarios, but then you gonna need to change rest.port to avoid conflict with already taken port.

Local kafka connect

Start your kafka

docker run -it --rm --name=kafka -e SAMPLEDATA=0 -e RUNNING_SAMPLEDATA=0 -e RUNTESTS=0 -e FORWARDLOGS=0 -e ADV_HOST=127.0.0.1 -p 2181:2181 -p 3030:3030 -p 8081-8082:8081-8082 -p 9092:9092 -p 9581-9585:9581-9585 lensesio/fast-data-dev:2.3.0

Note that I’m not exposing 8083 which is used by kafka connect rest api to avoid conflicts, otherwise do not forget to change rest.port in worker.properties

Simple messages

worker.properties

bootstrap.servers=localhost:9092

# do not forget to change me to avoid conflicts
rest.port=8083

# required for standalone workers
offset.storage.file.filename=/tmp/standalone.offsets

# where to look for additional plugins
plugin.path=/usr/share/java,/usr/share/confluent-hub-components

# optional, defaults for tasks
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter

Notes:

key and value converters are optional and can be overriden in tasks
most used converters are: org.apache.kafka.connect.storage.StringConverter, org.apache.kafka.connect.json.JsonConverter, io.confluent.connect.avro.AvroConverter
avro converter requires schema registry
for json converter do not forget to add value.converter.schemas.enable=false if you wish not to receive schema, e.g. by sending {"foo":"bar"} you will receive {"schema":{"type":"string","optional":false},"payload":"{\"foo\": \"bar\"}"}

Kafka Connect Source Text File

Notes on task configuration properties:

do not forget that each task should have unique name it will be used to watch for offsets and for distributed wrokers it will be used for topic names
connector.class is a kind of plugin, you can choose from hub.confluent.io
tasks.max control parallelism, for sink tasks can not be bigger that number of topic partitions

source-text-file.properties

name=source-text-file
connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector
# optional, override worker defaults
value.converter=org.apache.kafka.connect.storage.StringConverter
topic=DemoTextFile
file=demo-text-file.txt

demo-text-file.txt

hello
world
mac
was
here

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic DemoTextFile --partitions 3 --replication-factor 1

Start consumer

docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic DemoTextFile --from-beginning

Start worker

docker run -it --rm \
    --name=standalone \
    --net=host \
    -v $PWD:/data \
    -w /data \
    confluentinc/cp-kafka-connect:5.3.2 connect-standalone worker.properties source-text-file.properties

Note that we are bypassing our current directory into container so worker has access to all configuration files

If everything is ok after some while you will see your messages from a source file in your consumer

Kafka Connect Source JSON File

This one will work same way as previous

source-json-file.properties

name=source-json-file
connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector
# optional, override worker defaults
# value.converter=org.apache.kafka.connect.json.JsonConverter
# value.converter.schemas.enable=false
# if your will use JsonConverter here you will receive string with escaped json
value.converter=org.apache.kafka.connect.storage.StringConverter
topic=DemoJsonFile
file=demo-json-file.ndjson

demo-json-file.ndjson

{"foo": "hello"}
{"foo": "world"}
{"foo": "bar"}
{"acme": 42}

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic DemoJsonFile --partitions 3 --replication-factor 1

Start consumer

docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic DemoJsonFile --property value.deserializer=org.apache.kafka.connect.json.JsonDeserializer --skip-message-on-error --from-beginning

Start worker

docker run -it --rm \
    --name=standalone \
    --net=host \
    -v $PWD:/data \
    -w /data \
    confluentinc/cp-kafka-connect:5.3.2 connect-standalone worker.properties source-json-file.properties

While everything running, try add more records to a source file and save it, you should immediatelly see them in consumer.

Also try to add non json line to a source file, you will get an error:

[2020-01-04 10:09:00,896] ERROR Error processing message, skipping this message:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.SerializationException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'non': was expecting 'null', 'true', 'false' or NaN
 at [Source: (byte[])"non json"; line: 1, column: 5]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'non': was expecting 'null', 'true', 'false' or NaN
 at [Source: (byte[])"non json"; line: 1, column: 5]

but because we are running consumer with a --skip-message-on-error flag it should not die and continue listening to new records

unfortunatelly there is no way to produce messages with keys from simple files, if you will look at sources you will see that null is passed as key

If you wish to have keys you should run configured console producer and pipe file contents into it

Replaying Avro Messages With Key Value

This particular example does not use Kafka Connect but still might be used to replay some sequence of messages

Lets suppose that our source.txt file will look like:

source.txt

{"id":1}|{"foo":"hello"}
{"id":2}|{"foo":"world"}

where each line is an message with key and value separated by pipe

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic AvroFromFile --partitions 3 --replication-factor 1

Start kafka-avro-console-consumer to consume avro messages from file

docker run -it --rm --net=host confluentinc/cp-schema-registry:5.3.2 kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic AvroFromFile --from-beginning --property print.key=true

Start kafka-avro-console-producer which will produce avro messages from a file

docker run -it --rm --net=host -v $PWD:/data -w /data confluentinc/cp-schema-registry:5.3.2 sh -c "kafka-avro-console-producer --broker-list localhost:9092 --topic AvroFromFile --property value.schema='{\"type\":\"record\", \"name\": \"AvroFromFile\", \"fields\":[{\"name\":\"foo\",\"type\":\"string\"}]}' --property parse.key=true --property key.schema='{\"type\":\"record\",\"name\": \"key\", \"fields\":[{\"name\":\"id\",\"type\":\"int\"}]}' --property key.separator=\"|\" < source.txt"

And you should see your desired messages in consumer:

{"id":1}	{"foo":"hello"}
{"id":2}	{"foo":"world"}

Note that I have used sh -c "...." here because of bash can not understand whether last < source.txt should be ran inside docker or not

Kafka Connect Source DataGen Avro

In following example we are going to generate tousand of recods based on given avro schema

source.properties

name=source
connector.class=io.confluent.kafka.connect.datagen.DatagenConnector
kafka.topic=AvroDatagen
# override worker.properties
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://localhost:8081
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
# number of messages to generate
iterations=1000
tasks.max=1
# avro schema
schema.filename=/data/AvroDatagen.avsc

Some additional properties can be found here

Note that by default auto.register.schemas is set to true so you do not need to register schemas upfront everything will be done automatically. Also note that both key.subject.name.strategy and value.subject.name.strategy are set to io.confluent.kafka.serializers.subject.SubjectNameStrategy so schema names will be AvroDatagen-key and AvroDatagen-value retrospectively.

AvroDatagen.avsc

{
  "type": "record",
  "name": "AvroDatagen",
  "namespace": "ua.rabota.topics",
  "fields": [
    {
      "name": "userId",
      "type": {
        "type": "int",
        "arg.properties": {
          "range": {
            "min": 1,
            "max": 100
          }
        }
      }
    },
    {
      "name": "vacancyId",
      "type": {
        "type": "long",
        "arg.properties": {
          "range": {
            "min": 7710732,
            "max": 7711732
          }
        }
      }
    },
    {
      "name": "platform",
      "type": ["null", {
        "type": "string",
        "arg.properties": {
          "options": ["desktop", "mobile", "ios", "android"]
        }
      }],
      "default": null
    }
  ]
}

Note that usually in avro schema you defining properties like {"name": "foo", "type": "string"} where type is usually primitive string with type name, for datagen we are describing type as object with additional arg.properties

Crate topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic AvroDatagen --partitions 3 --replication-factor 1

Start consumer

docker exec -it kafka kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic AvroDatagen --from-beginning

Start avro datagen producer

docker run -it --rm \
    --name=standalone \
    --net=host \
    -v $PWD:/data \
    -w /data \
    confluentinc/cp-kafka-connect:5.3.2 bash -c "confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:0.2.0 && connect-standalone worker.properties source.properties"

Note how we are installing kafka-connect-datagen before starting connect-standalone it does not shipped by deafult

After a while, when everything will boot up you should see incomming messages in consumer

When datagen will produce desired 1000 messages it will die and you will see something like:

[2020-01-04 11:22:37,984] ERROR WorkerSourceTask{id=source-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179)
org.apache.kafka.connect.errors.ConnectException: Stopping connector: generated the configured 1000 number of messages

Unfortunatelly datagen is quite limited about keys only way you can have keys is to provide schema.keyfield which will use one of generated properties as message key, and according to sources it still will be simple string key.

Kafka Connect Simple Sink To Text File

This might be used for debug and log

sink.properties

name=sink
connector.class=org.apache.kafka.connect.file.FileStreamSinkConnector
tasks.max=1
topics=SinkDemo
file=/data/data.txt

Create topic

docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --create --topic SinkDemo --partitions 3 --replication-factor 1

Start Kafka Connect Sink to save messages to a text file

docker run -it --rm \
    --name=standalone \
    --net=host \
    -v $PWD:/data \
    -w /data \
    confluentinc/cp-kafka-connect:5.3.2 connect-standalone worker.properties sink.properties

Start console producer

docker exec -it kafka kafka-console-producer --broker-list localhost:9092 --topic SinkDemo

and start typing messages into it, you should immediatelly see them in text file

Do not forget that you can run some tricky setups like connect-standalone worker.properties source.properties sink.properties which might generate data into topic and immediatelly sink them into source

Standalone connect worker with confluent.cloud

All previous examples should work well with confluent.cloud if you will provide required configuration options

What you gonna need

cloud.properties

bootstrap.servers=xxx-xxxxx.us-east1.gcp.confluent.cloud:9092
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="xxxxxxx" password\="xxxxxxx";

this file will be used by kafka-topics to create topic

worker.properties

bootstrap.servers=xxx-xxxxx.us-east1.gcp.confluent.cloud:9092
plugin.path=/usr/share/java,/usr/share/confluent-hub-components

offset.storage.file.filename=/tmp/standalone.offsets

# TODO: check whether this is a deafults
# default 60000
offset.flush.interval.ms=10000
# default 40000
request.timeout.ms=20000
# 100
retry.backoff.ms=500
consumer.request.timeout.ms=20000
consumer.retry.backoff.ms=500
producer.request.timeout.ms=20000
producer.retry.backoff.ms=500

# deafult https
ssl.endpoint.identification.algorithm=https
# default PLAINTEXT
security.protocol=SASL_SSL
# default GSSAPI
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="xxxxxxx" password\="xxxxxxx";

# Connect producer and consumer specific configuration
producer.ssl.endpoint.identification.algorithm=https
producer.confluent.monitoring.interceptor.ssl.endpoint.identification.algorithm=https
consumer.ssl.endpoint.identification.algorithm=https
consumer.confluent.monitoring.interceptor.ssl.endpoint.identification.algorithm=https
producer.security.protocol=SASL_SSL
producer.confluent.monitoring.interceptor.security.protocol=SASL_SSL
consumer.security.protocol=SASL_SSL
consumer.confluent.monitoring.interceptor.security.protocol=SASL_SSL
producer.sasl.mechanism=PLAIN
producer.confluent.monitoring.interceptor.sasl.mechanism=PLAIN
consumer.sasl.mechanism=PLAIN
consumer.confluent.monitoring.interceptor.sasl.mechanism=PLAIN
producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="xxxxxxx" password\="xxxxxxx";
producer.confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="xxxxxxx" password\="xxxxxxx";
consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="xxxxxxx" password\="xxxxxxx";
consumer.confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="xxxxxxx" password\="xxxxxxx";

# Confluent Schema Registry for Kafka Connect
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.basic.auth.credentials.source=USER_INFO
value.converter.schema.registry.basic.auth.user.info=xxxxxxx:xxxxxxx
value.converter.schema.registry.url=https://xxxx-xxxxx.us-east1.gcp.confluent.cloud

key.converter=io.confluent.connect.avro.AvroConverter
key.converter.basic.auth.credentials.source=USER_INFO
key.converter.schema.registry.basic.auth.user.info=xxxxxxx:xxxxxxx
key.converter.schema.registry.url=https://xxxx-xxxxx.us-east1.gcp.confluent.cloud


# additions - https://docs.confluent.io/current/cloud/connect/connect-cloud-config.html

confluent.topic.bootstrap.servers=xxx-xxxxx.us-east1.gcp.confluent.cloud:9092
confluent.topic.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="xxxxxxx" password\="xxxxxxx";
confluent.topic.security.protocol=SASL_SSL
confluent.topic.sasl.mechanism=PLAIN

reporter.admin.bootstrap.servers=xxx-xxxxx.us-east1.gcp.confluent.cloud:9092
reporter.admin.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="xxxxxxx" password\="xxxxxxx";
reporter.admin.security.protocol=SASL_SSL
reporter.admin.sasl.mechanism=PLAIN

reporter.producer.bootstrap.servers=xxx-xxxxx.us-east1.gcp.confluent.cloud:9092
reporter.producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="xxxxxxx" password\="xxxxxxx";
reporter.producer.security.protocol=SASL_SSL
reporter.producer.sasl.mechanism=PLAIN

this one is for worker to be able to comminicate with confluent cloud

source.properties

name=source
tasks.max=1

connector.class=io.confluent.kafka.connect.datagen.DatagenConnector
kafka.topic=demo1
iterations=1000
schema.filename=/data/demo1.avsc

this one will be used by datagen connector to generate random data into given topic

sink.properties

name=sink
connector.class=org.apache.kafka.connect.file.FileStreamSinkConnector
topics=demo1
file=/data/data.txt

sing generated messages back from cloud to local file

demo1.avsc

{
  "type": "record",
  "name": "demo1",
  "namespace": "ua.rabota.topics",
  "fields": [
    {
      "name": "userId",
      "type": {
        "type": "int",
        "arg.properties": {
          "range": {
            "min": 1,
            "max": 100
          }
        }
      }
    },
    {
      "name": "vacancyId",
      "type": {
        "type": "long",
        "arg.properties": {
          "range": {
            "min": 7710732,
            "max": 7711732
          }
        }
      }
    },
    {
      "name": "platform",
      "type": ["null", {
        "type": "string",
        "arg.properties": {
          "options": ["desktop", "mobile", "ios", "android"]
        }
      }],
      "default": null
    }
  ]
}

schema for messages to be generated

create topic

docker run -it --rm -v $PWD/cloud.properties:/cloud.properties confluentinc/cp-kafka:5.3.2 kafka-topics \
  --bootstrap-server xxx-xxxxx.us-east1.gcp.confluent.cloud:9092 \
  --command-config /cloud.properties \
  --create --topic demo1  --partitions 3 --replication-factor 3

start worker

docker run -it --rm \
    --name=standalone \
    -v $PWD:/data \
    -w /data \
    confluentinc/cp-kafka-connect:5.3.2 bash -c "confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:0.2.0 && connect-standalone worker.properties source.properties sink.properties"

after a while you will see that your data.txt file becomes full of random generated messages

So now you can quickly send batch of messages both generated and predefined not only to local kafka but also to your confluent cloud one - profit

Distributed Worker

Confluent cloud not giving you distributed workers for some reasons. Seems like it is because they do not know how much of them you gonna need. To start your own connect cluster you will need worker.properties from previous example just remove offset.storage.file.filename from it and add

group.id=mac1
offset.storage.topic=mac1-offsets
config.storage.topic=mac1-configs
status.storage.topic=mac1-status

offset.storage.partitions=3
replication.factor=3
config.storage.replication.factor=3
offset.storage.replication.factor=3
status.storage.replication.factor=3

take a closer look to first four settings, make sure they are unique

The difference between standalone and distributed worker is that from now you going to add and remove your tasks via rest api

In most of the cases everything will look the same as in previous examples, except that now you are going to post json instead of property files like in example from docs:

POST /connectors HTTP/1.1
Host: connect.example.com
Content-Type: application/json
Accept: application/json

{
    "name": "hdfs-sink-connector",
    "config": {
        "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector",
        "tasks.max": "10",
        "topics": "test-topic",
        "hdfs.url": "hdfs://fakehost:9000",
        "hadoop.conf.dir": "/opt/hadoop/conf",
        "hadoop.home": "/opt/hadoop",
        "flush.size": "100",
        "rotate.interval.ms": "1000"
    }
}

Here is an example of docker run which is a good starting point to run your connect cluster in kubernetes

docker run -it --rm \
    --name=mac1 \
    -p 8083:8083 \
    -e CONNECT_BOOTSTRAP_SERVERS=xxx-xxxxx.us-east1.gcp.confluent.cloud:9092 \
    -e CONNECT_GROUP_ID=mac1 \
    -e CONNECT_OFFSET_STORAGE_TOPIC=mac1-offsets \
    -e CONNECT_CONFIG_STORAGE_TOPIC=mac1-configs \
    -e CONNECT_STATUS_STORAGE_TOPIC=mac1-status \
    -e CONNECT_OFFSET_STORAGE_PARTITIONS=3 \
    -e CONNECT_REPLICATION_FACTOR=3 \
    -e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=3 \
    -e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=3 \
    -e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=3 \
    -e CONNECT_OFFSET_FLUSH_INTERVAL_MS=10000 \
    -e CONNECT_REQUEST_TIMEOUT_MS=20000 \
    -e CONNECT_RETRY_BACKOFF_MS=500 \
    -e CONNECT_CONSUMER_REQUEST_TIMEOUT_MS=20000 \
    -e CONNECT_CONSUMER_RETRY_BACKOFF_MS=500 \
    -e CONNECT_PRODUCER_REQUEST_TIMEOUT_MS=20000 \
    -e CONNECT_PRODUCER_RETRY_BACKOFF_MS=500 \
    -e CONNECT_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=https \
    -e CONNECT_SECURITY_PROTOCOL=SASL_SSL \
    -e CONNECT_SASL_MECHANISM=PLAIN \
    -e CONNECT_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xxxxxxx\" password=\"xxxxxxx\";" \
    -e CONNECT_PRODUCER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=https \
    -e CONNECT_PRODUCER_CONFLUENT_MONITORING_INTERCEPTOR_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=https \
    -e CONNECT_CONSUMER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=https \
    -e CONNECT_CONSUMER_CONFLUENT_MONITORING_INTERCEPTOR_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=https \
    -e CONNECT_PRODUCER_SECURITY_PROTOCOL=SASL_SSL \
    -e CONNECT_PRODUCER_CONFLUENT_MONITORING_INTERCEPTOR_SECURITY_PROTOCOL=SASL_SSL \
    -e CONNECT_CONSUMER_SECURITY_PROTOCOL=SASL_SSL \
    -e CONNECT_CONSUMER_CONFLUENT_MONITORING_INTERCEPTOR_SECURITY_PROTOCOL=SASL_SSL \
    -e CONNECT_PRODUCER_SASL_MECHANISM=PLAIN \
    -e CONNECT_PRODUCER_CONFLUENT_MONITORING_INTERCEPTOR_SASL_MECHANISM=PLAIN \
    -e CONNECT_CONSUMER_SASL_MECHANISM=PLAIN \
    -e CONNECT_CONSUMER_CONFLUENT_MONITORING_INTERCEPTOR_SASL_MECHANISM=PLAIN \
    -e CONNECT_PRODUCER_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xxxxxxx\" password=\"xxxxxxx\";" \
    -e CONNECT_PRODUCER_CONFLUENT_MONITORING_INTERCEPTOR_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xxxxxxx\" password=\"xxxxxxx\";" \
    -e CONNECT_CONSUMER_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xxxxxxx\" password=\"xxxxxxx\";" \
    -e CONNECT_CONSUMER_CONFLUENT_MONITORING_INTERCEPTOR_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xxxxxxx\" password=\"xxxxxxx\";" \
    -e CONNECT_VALUE_CONVERTER=io.confluent.connect.avro.AvroConverter \
    -e CONNECT_VALUE_CONVERTER_BASIC_AUTH_CREDENTIALS_SOURCE=USER_INFO \
    -e CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=xxxxxxx:xxxxxxx \
    -e CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL=https://xxxx-xxxxx.us-east1.gcp.confluent.cloud \
    -e CONNECT_KEY_CONVERTER=io.confluent.connect.avro.AvroConverter \
    -e CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE=true \
    -e CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE=true \
    -e CONNECT_REST_POST=8083 \
    -e CONNECT_REST_ADVERTISED_HOST_NAME=localhost \
    -e CONNECT_KEY_CONVERTER_BASIC_AUTH_CREDENTIALS_SOURCE=USER_INFO \
    -e CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=xxxxxxx:xxxxxxx \
    -e CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL=https://xxxx-xxxxx.us-east1.gcp.confluent.cloud
    confluentinc/cp-kafka-connect:5.3.2

Bash aliases

Even simple operations like creating topic becomes not easy to remember especially if you will have local, dev, prod kafka clusters

If you are using ccloud command line tool you already should have ~/.ccloud/ which to me seems a good place to save my cloud.properties files in my case it will be dev.properties and prod.peroperties

Here are few starting point examples

Local kafka bash aliases

alias local-topic="docker run -it --rm --net=host confluentinc/cp-kafka:5.3.2 kafka-topics --bootstrap-server localhost:9092"

alias local-topic-list="local-topic --list"

alias local-topic-delete="local-topic --delete --topic"

alias local-topic-describe="local-topic --describe --topic"

alias local-topic-create="local-topic --create --replication-factor 1 --topic"

alias local-topic-create1="local-topic --create --replication-factor 1 --partitions 1 --topic"

alias local-topic-create2="local-topic --create --replication-factor 1 --partitions 2 --topic"

alias local-console-consumer="docker run -it --rm --net=host confluentinc/cp-kafka:5.3.2 kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic"

alias local-console-producer="docker run -it --rm --net=host confluentinc/cp-kafka:5.3.2 kafka-console-producer --broker-list localhost:9092 --topic"

Confluent cloud kafka bash aliases

alias dev-topic="docker run -it --rm -v /Users/mac/.ccloud/dev.properties:/dev.properties confluentinc/cp-kafka:5.3.2 kafka-topics --bootstrap-server $(grep bootstrap.server ~/.ccloud/dev.properties | tail -1 | cut -d'=' -f2) --command-config dev.properties"

alias dev-topic-list="dev-topic --list"

alias dev-topic-delete="dev-topic --delete --topic"

alias dev-topic-describe="dev-topic --describe --topic"

alias dev-topic-create="dev-topic --create  --replication-factor 3 --topic"

alias dev-console-consumer="docker run -it --rm -v /Users/mac/.ccloud/dev.properties:/dev.properties confluentinc/cp-kafka:5.3.2 kafka-console-consumer --bootstrap-server $(grep bootstrap.server ~/.ccloud/dev.properties | tail -1 | cut -d'=' -f2) --consumer.config dev.properties --topic"

alias dev-console-producer="docker run -it --rm -v /Users/mac/.ccloud/dev.properties:/dev.properties confluentinc/cp-kafka:5.3.2 kafka-console-producer --broker-list $(grep bootstrap.server ~/.ccloud/dev.properties | tail -1 | cut -d'=' -f2) --producer.config dev.properties --topic"

docker localhost ssl

2019-06-12T00:00:00+03:00

Two ways of having trusted self signed certs for local development needs

First one with mkcert which will add root ca to system so it will trust it

Installation and generating certificates is as easy as:

brew install mkcert
mkcert -install
mkcert -cert-file vcap.me.crt -key-file vcap.me.key "*.vcap.me"
cp "$(mkcert -CAROOT)/rootCA.pem" ca.crt

Note that I’m using vcap.me which is resolving to 127.0.0.1

And here is docker-compose.yml

version: "3.5"
services:
  nginx-proxy:
    container_name: proxy
    image: jwilder/nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - ./certs:/etc/nginx/certs
    networks:
      default:
        ipv4_address: 172.20.0.200

  whoami:
    container_name: whoami
    image: jwilder/whoami
    volumes:
      - ./certs/ca.crt:/usr/local/share/ca-certificates/ca.crt
    environment:
      - VIRTUAL_HOST=whoami.vcap.me
      - VIRTUAL_PORT=8000
    extra_hosts:
      - "nginx.vcap.me:172.20.0.200"
      - "whoami.vcap.me:172.20.0.200"

  nginx:
    container_name: nginx
    image: nginx:alpine
    volumes:
      - ./certs/ca.crt:/usr/local/share/ca-certificates/ca.crt
    environment:
      - VIRTUAL_HOST=nginx.vcap.me
      - VIRTUAL_PORT=80
    extra_hosts:
      - "nginx.vcap.me:172.20.0.200"
      - "whoami.vcap.me:172.20.0.200"

networks:
  default:
    ipam:
      config:
        - subnet: 172.20.0.0/16

With such setup ssl will work not only from outside docker but between containers alsto.

There is localhost.tools guys registered domain and configured lets encrypt wild card certificates which allows to achieve the same result without installing anything on system like this:

version: "3.5"
services:
  proxy:
    container_name: proxy
    image: tarampampam/localhost
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      default:
        ipv4_address: 172.20.0.200

  whoami:
    container_name: whoami
    image: jwilder/whoami
    labels:
      traefik.frontend.rule: Host:whoami.localhost.tools
      traefik.protocol: http
      traefik.port: 8000
    extra_hosts:
      - "whoami.localhost.tools:172.20.0.200"
      - "api.localhost.tools:172.20.0.200"

  api:
    container_name: api
    build:
      context: api
      dockerfile: Dockerfile
    volumes:
      - ./api:/app
      - /api/bin
      - /api/obj
    labels:
      traefik.frontend.rule: Host:api.localhost.tools
      traefik.protocol: http
      traefik.port: 5000
    extra_hosts:
      - "whoami.localhost.tools:172.20.0.200"
      - "api.localhost.tools:172.20.0.200"

networks:
  default:
    ipam:
      config:
        - subnet: 172.20.0.0/16

Node.js HTTP rest server side by side with UDP on same port

2018-11-18T00:00:00+02:00

We are going to build service which will be able to respond to both HTTP requests and UDP messages comming to the same port.

UDP might be used to broadcast messages which might be used to service discovery like things if all services are running in same network.

server.js

const http = require("http");
const url = require("url");
const dgram = require("dgram");
const { StringDecoder } = require("string_decoder");

const PORT = process.env.PORT || 3000;

// http/tcp server
const server = http.createServer((req, res) => {
  const method = req.method.toLowerCase();
  const parsed = url.parse(req.url, true);
  const path = parsed.pathname.replace(/^\/+|\/+$/g, "");
  const query = parsed.query;
  const headers = req.headers;

  const decoder = new StringDecoder("utf-8");
  let body = "";
  req.on("data", data => (body += decoder.write(data)));
  req.on("end", () => {
    body += decoder.end();

    console.log({
      kind: "HTTP_REQUEST",
      method,
      path,
      query,
      headers,
      body
    });

    // TODO: process request

    res.setHeader("Content-Type", "application/json; charset=utf-8");
    res.writeHead(200);
    res.end(JSON.stringify({ success: true, message: "ok" }));
  });
});

// udp server
const socket = dgram.createSocket(
  {
    type: "udp4",
    reuseAddr: true // <- NOTE: we are asking OS to let us reuse port
  },
  (buffer, sender) => {
    const message = buffer.toString();
    console.log({
      kind: "UDP_MESSAGE",
      message,
      sender
    });

    // demo: respond to sender
    socket.send(message.toUpperCase(), sender.port, sender.address, error => {
      if (error) {
        console.error(error);
      } else {
        console.log({
          kind: "RESPOND",
          message: message.toUpperCase(),
          sender
        });
      }
    });
  }
);

// POI: bind two servers to same port
server.listen(PORT);
socket.bind(PORT);

And here is simple client which will send broadcast udp message to whole network and print response if any, after that it closes socket and exits.

client.js

const dgram = require("dgram");
const socket = dgram.createSocket("udp4");

const PORT = process.env.PORT || 3000;

socket.bind();
socket.on("listening", () => {
  socket.setBroadcast(true);

  // 255.255.255.255 - boradcast for local network - RFC922
  socket.send("hi", PORT, "255.255.255.255", err => {
    console.log(err ? err : "Sended");
    // socket.close();
  });

  socket.on("message", (buffer, sender) => {
    const message = buffer.toString();
    console.log(`Received message: ${message} from ${sender.address}`);
    socket.close();
  });
});

Now is you start server and try to run following:

$ node client.js
Sended
Received message: HI from 192.168.105.108
$ curl -X POST http://localhost:6000/foo?acme=42 -d '{"foo":"bar"}'
{"success":true,"message":"ok"}

On a server console you will see:

{ kind: 'UDP_MESSAGE',
  message: 'hi',
  sender:
   { address: '192.168.105.108',
     family: 'IPv4',
     port: 56939,
     size: 2 } }
{ kind: 'RESPOND',
  message: 'HI',
  sender:
   { address: '192.168.105.108',
     family: 'IPv4',
     port: 56939,
     size: 2 } }
{ kind: 'HTTP_REQUEST',
  method: 'post',
  path: 'foo',
  query: { acme: '42' },
  headers:
   { host: 'localhost:3000',
     'user-agent': 'curl/7.54.0',
     accept: '*/*',
     'content-length': '13',
     'content-type': 'application/x-www-form-urlencoded' },
  body: '{"foo":"bar"}' }

Nginx Proxy saceld docker-compose services

2018-11-18T00:00:00+02:00

Easiest way to proxy scaled containers

docker-compose.yml

version: "3"

services:
  proxy:
    image: jwilder/nginx-proxy:alpine
    container_name: proxy
    ports:
      - 80:80
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
    logging:
      driver: none

  whoami:
    image: jwilder/whoami
    ports:
      - 8000
    environment:
      VIRTUAL_HOST: localhost

Now if you run:

docker-compose up
docker-compose scale whoami=5

And open localhost in browser, each time you refresh the page you will see different container, and you can still access concrete containers by their own ports

$ curl http://localhost
I'm a9998f9cefbd

$ curl http://localhost
I'm 66d5e9d72b61

$ curl http://localhost
I'm 6bab17a8d614

$ curl http://localhost
I'm 103bac359b56

$ docker ps --format ": " --filter ancestor=jwilder/whoami
a9998f9cefbd: 0.0.0.0:32791->8000/tcp
66d5e9d72b61: 0.0.0.0:32790->8000/tcp
6bab17a8d614: 0.0.0.0:32789->8000/tcp
103bac359b56: 0.0.0.0:32788->8000/tcp
de52272cac61: 0.0.0.0:32787->8000/tcp

$ curl http://localhost:32790
I'm 66d5e9d72b61

AmbiguousActionException

2018-11-04T00:00:00+02:00

Here is challange, I wish to have following API:

GET /persons/1

PATCH /persons/1
{
    "name": "Alex"
}

PATCH /persons/1
{
    "age": 33
}

where PATCH is used to partially update related entity.

The problem is that it is not so easy to implement in dotnet.

TLDR: look for code at end of page

If you decide to make two models PatchPersonName and PatchPersonAge and two appropriate controller actions, something like:

public class PatchPersonName
{
    [Required]
    public string Name { get; set; }
}

public class PatchPersonAge
{
    [Required]
    [Range(1, 99)]
    public int Age { get; set; }
}

[ApiController]
public class DemoController : ControllerBase
{
    [HttpPatch]
    [Route("person/{id:int:min(1)}")]
    public void PatchPersonName(int id, [FromBody]PatchPersonName model) => {};

    [HttpPatch]
    [Route("person/{id:int:min(1)}")]
    public void PatchPersonAge(int id, [FromBody]PatchPersonAge model) => {};
}

you immediatelly will catch AmbiguousActionException telling you that multiple actions matched so dotnet can not choose the right one.

Next step you might try to do is create some kind of base class class and inherit your models from it. e.g.

public class BasePatchModel {}

public class PatchPersonName : BasePatchModel {...}
public class PatchPersonAge : BasePatchModel {...}

[ApiController]
public class DemoController : ControllerBase
{
    [HttpPatch]
    [Route("person/{id:int:min(1)}")]
    public void PatchPerson(int id, [FromBody]BasePatchModel model) => {};
}

But immediatelly you will get following problem - it wont be deserialized as you expect and will be always empty.

But still you can go further and so something like this:

[HttpPatch]
[Route("person/{id:int:min(1)}")]
public object PatchPerson(int id, [FromBody]JObject model)
{...}

and now you can use ToObject<T> of JObject something like this:

var patchName = model.ToObject<PatchName>();
if (patchName != null)
{
    ...
}

var patchAge = model.ToObject<PatchAge>();
if (patchAge != null)
{
    ...
}
else
{
    return BadRequest();
}

but it becomes ugly and long, also there few more problems you will realize pretty soon.

So first of all - you loose annotation validation :( which can be fixed like this:

if (patchName != null && Validator.TryValidateObject(patchName, new ValidationContext(patchName), new List<ValidationResult>(), true))

Next, we wish to have separate action methods for further Swashbuckle API documentation generation, but if you try something like:

[ApiController]
public class DemoController : ControllerBase
{
    [HttpPatch]
    [Route("person/{id:int:min(1)}")]
    public object PatchPerson(int id, [FromBody]JObject model)
    {...}

    [HttpPatch]
    [Route("person/{id:int:min(1)}")]
    public void PatchPersonName(int id, [FromBody]PatchPersonName model)
    {...}

    [HttpPatch]
    [Route("person/{id:int:min(1)}")]
    public void PatchPersonAge(int id, [FromBody]PatchPersonAge model)
    {...};
}

And once again you will get ambiguous error, so what we are going to do is to tell dotnet to always skip our concrete actions with action constraint so dotnet will always use our generic action. And we will hide it from swashbuckle, e.g.

public class DoNotActivateAttribute : ActionMethodSelectorAttribute
{
    public override bool IsValidForRequest(RouteContext routeContext, ActionDescriptor action)
    {
        return false;
    }
}

[ApiController]
public class DemoController : ControllerBase
{
    [HttpPatch]
    [ApiExplorerSettings(IgnoreApi = true)] // will hide me from Swashbuckle
    [Route("person/{id:int:min(1)}")]
    public object PatchPerson(int id, [FromBody]JObject model)
    {
        var patchName = model.ToObject<PatchName>();
        if (patchName != null && Validator.TryValidateObject(patchName, new ValidationContext(patchName), new List<ValidationResult>(), true))
        {
            return PatchPersonName(id, patchName);
        }

        var patchAge = model.ToObject<PatchAge>();
        if (patchAge != null && Validator.TryValidateObject(patchAge, new ValidationContext(patchAge), new List<ValidationResult>(), true))
        {
            return PatchPersonAge(id, patchAge);
        }
        else
        {
            return BadRequest();
        }
    }

    [HttpPatch]
    [DoNotActivate] // will skip me when choosing action
    [Route("person/{id:int:min(1)}")]
    public void PatchPersonName(int id, [FromBody]PatchPersonName model)
    {...}

    [HttpPatch]
    [DoNotActivate] // will skip me when choosing action
    [Route("person/{id:int:min(1)}")]
    public void PatchPersonAge(int id, [FromBody]PatchPersonAge model)
    {...};
}

So we kind of done, technically it does what we want but the challenge is can it be done other way?!

Action Method Selector

What we are going to do is to remove generic action, and will try to make our action method selector in a such way that it will try to deserialize incomming request body into desired model and if it will - we are going to select action.

The tricky part here is that we need to enable rewind of request body which is stream. and do not forget to move stream position back to zero when we done - otherwise everything will be broken.

Here is what I end up with:

public class PatchForAttribute : ActionMethodSelectorAttribute
{
    public Type Type { get; }

    public PatchForAttribute(Type type)
    {
        Type = type;
    }

    public override bool IsValidForRequest(RouteContext routeContext, ActionDescriptor action)
    {
        // IMPORTANT: required to rewind stream
        routeContext.HttpContext.Request.EnableRewind();
        // kind of copy body stream
        var body = new StreamReader(routeContext.HttpContext.Request.Body).ReadToEnd();
        try
        {
            // we are trying to deserialize incommingn body into desired type
            JsonConvert.DeserializeObject(body, Type, new JsonSerializerSettings { MissingMemberHandling = MissingMemberHandling.Error });
            return true; // yes we can choose this action
        }
        catch (Exception)
        {
            return false; // no we wont choose this action, go further
        }
        finally
        {
            // IMPORTANT: do not forget to rewind stream
            routeContext.HttpContext.Request.Body.Position = 0;
        }
    }
}

and now our controller will look like this:

[ApiController]
public class DemoController : ControllerBase
{
    [Route("person/{id:int:min(1)}")]
    [PatchFor(typeof(PatchPersonName))]
    public object PatchPersonName(int id, [FromBody]PatchPersonName model)
    {...}

    [HttpPatch]
    [Route("person/{id:int:min(1)}")]
    [PatchFor(typeof(PatchPersonAge))]
    public object PatchPersonAge(int id, [FromBody]PatchPersonAge model)
    {...}
}

Pros: validation is working, no need for third action, will work with swashbuckle

Cons: for concrete this actions we are reading and deserializing body twice

Single route with multiple models

And here is final result, everything combined for a note:

Program.cs

using System;
using System.ComponentModel.DataAnnotations;
using System.IO;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http.Internal;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.Abstractions;
using Microsoft.AspNetCore.Mvc.ActionConstraints;
using Microsoft.AspNetCore.Routing;
using Microsoft.Extensions.DependencyInjection;
using Newtonsoft.Json;

public static class Program
{
    public static void Main(string[] args) => WebHost.CreateDefaultBuilder(args).UseStartup<Startup>().Build().Run();
}

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        app.UseMvc();
    }
}

public class Person
{
    public int Id { get; set; }
    public string Name { get; set; }
    public int Age { get; set; }
}

public class PatchPersonName
{
    [Required]
    public string Name { get; set; }
}

public class PatchPersonAge
{
    [Required]
    [Range(1, 99)]
    public int Age { get; set; }
}

public class PatchForAttribute : ActionMethodSelectorAttribute
{
    public Type Type { get; }

    public PatchForAttribute(Type type)
    {
        Type = type;
    }

    public override bool IsValidForRequest(RouteContext routeContext, ActionDescriptor action)
    {
        routeContext.HttpContext.Request.EnableRewind();
        var body = new StreamReader(routeContext.HttpContext.Request.Body).ReadToEnd();
        try
        {
            JsonConvert.DeserializeObject(body, Type, new JsonSerializerSettings { MissingMemberHandling = MissingMemberHandling.Error });
            return true;
        }
        catch (Exception)
        {
            return false;
        }
        finally
        {
            routeContext.HttpContext.Request.Body.Position = 0;
        }
    }
}

[ApiController]
public class DemoController : ControllerBase
{
    // curl -s http://localhost:5000/person/1 | jq
    // {
    //     "id": 1,
    //     "name": "Alex",
    //     "age": 33
    // }
    [HttpGet]
    [Route("person/{id:int:min(1)}")]
    public Person Person(int id) => new Person
    {
        Id = id,
        Name = "Alex",
        Age = 33
    };

    // curl -s -X PATCH -H 'Content-Type: application/json' http://localhost:5000/person/1 -d '{"name": "Maria"}'
    [HttpPatch]
    [Route("person/{id:int:min(1)}")]
    [PatchFor(typeof(PatchPersonName))]
    public object PatchPersonName(int id, [FromBody]PatchPersonName model) => new
    {
        Id = id,
        Kind = "PATCH",
        Prop = "Name",
        Value = model.Name
    };

    // curl -s -X PATCH -H 'Content-Type: application/json' http://localhost:5000/person/1 -d '{"age": 30}' | jq
    // {
    //     "id": 1,
    //     "kind": "PATCH",
    //     "prop": "Age",
    //     "value": 30
    // }
    // # still have working validation
    // curl -s -X PATCH -H 'Content-Type: application/json' http://localhost:5000/person/1 -d '{"age": 30000}' | jq
    // {
    //     "Age": [
    //         "The field Age must be between 1 and 99."
    //     ]
    // }
    [HttpPatch]
    [Route("person/{id:int:min(1)}")]
    [PatchFor(typeof(PatchPersonAge))]
    public object PatchPersonAge(int id, [FromBody]PatchPersonAge model) => new
    {
        Id = id,
        Kind = "PATCH",
        Prop = "Age",
        Value = model.Age
    };
}

Dotnet Core Content Negotiation

2018-11-03T00:00:00+02:00

Having fun with content negotiation and xslt transformations.

Content Negotiation is an ability of backend to respond to the clients request with appropriate data format (e.g. html, json, xml, etc).

Each client sends Accept header with requests which is used to determine appropriate serializer.

For example Chrome, by default uses text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 this is why in older .Net WebApi you saw XML instead of desired JSON - reason is content negotiation in action.

The worse thing you might have done is to turn in off, remove all serializers and add only json, just to see json in browser.

In modern dotnet core 2.1 guys turned of content negotiation completelly, so if you have something like this:

public class Post
{
    public int Id { get; set; }
    public string Title { get; set; }
    public string Body { get; set; }
}

[ApiController]
public class DemoController : ControllerBase
{
    [HttpGet]
    [Route(nameof(Posts))]
    public IEnumerable<Post> Posts() => new[] {
        new Post {
            Id = 1,
            Title = "Hello World",
            Body = "Lorem ipsum dot color"
        },
        new Post {
            Id = 2,
            Title = "Post 2",
            Body = "Lorem ipsum dot color"
        }
    };
}

And try to run:

curl -k -i -s -H 'Accept: text/xml' http://localhost:5000/posts
curl -k -i -s -H 'Accept: application/json' http://localhost:5000/posts

Both will return:

[{ "id": 1, "title": "Hello World", "body": "Lorem ipsum dot color" }, { "id": 2, "title": "Post 2", "body": "Lorem ipsum dot color" }]

To get content negotiation working back you need to make following ajustments in your Startup.cs:

public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc(options =>
    {
        options.RespectBrowserAcceptHeader = true; // default is false
    })
    .AddXmlSerializerFormatters() // does not added by default
    .SetCompatibilityVersion(CompatibilityVersion.Version_2_1);
}

and now suddenly

curl -k -i -s -H 'Accept: text/xml' http://localhost:5000/posts

will return:

<ArrayOfPost xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Post>
    <Id>1</Id>
    <Title>Hello World</Title>
    <Body>Lorem ipsum dot color</Body>
  </Post>
  <Post>
    <Id>2</Id>
    <Title>Post 2</Title>
    <Body>Lorem ipsum dot color</Body>
  </Post>
</ArrayOfPost>

which is desired behavior.

XSLT

Now fun part, in Chrome we see good old XML, but how about make it more human friendly while still leave as clean XML

public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc(options =>
    {
        options.RespectBrowserAcceptHeader = true; // default is false
        // options.OutputFormatters.Add(new XmlSerializerOutputFormatter()); // not enoug
        options.OutputFormatters.Add(new MyXmlSerializerOutputFormatter());
    })
    // .AddXmlSerializerFormatters() // does not added by default - not enoug
    .SetCompatibilityVersion(CompatibilityVersion.Version_2_1);
}

So, we are going to customize XML serilizer a little bit to inject XSLT link to the output.

public class MyXmlSerializerOutputFormatter : XmlSerializerOutputFormatter
{
    protected override void Serialize(XmlSerializer xmlSerializer, XmlWriter xmlWriter, object value)
    {
        // TODO: add me only if controller has some kind of custom attribute with XSLT file name
        xmlWriter.WriteProcessingInstruction("xml-stylesheet", "type=\"text/xsl\" href=\"template.xsl\"");
        base.Serialize(xmlSerializer, xmlWriter, value);
    }
}

With this small addition your output now will be something like this:

<?xml-stylesheet type="text/xsl" href="template.xsl"?>
<ArrayOfPost xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Post>
    <Id>1</Id>
    <Title>Hello World</Title>
    <Body>Lorem ipsum dot color</Body>
  </Post>
  <Post>
    <Id>2</Id>
    <Title>Post 2</Title>
    <Body>Lorem ipsum dot color</Body>
  </Post>
</ArrayOfPost>

and with small xsl like this:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="html" encoding="utf-8" indent="yes" />

  <xsl:template match="/">
    <xsl:text disable-output-escaping='yes'>&lt;!DOCTYPE html&gt;</xsl:text>
    <html>
    <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Demo</title>
    <style>
        html, body {
            padding: 0;
            margin: 0;
            font: normal 18px/1.5 sans-serif;
            background: #ebebeb;
        }

        body {
            margin: 2em auto;
            max-width: 75vw;
        }

        h1, h3 {
            font-weight: normal;
        }

        summary h3 {
            display: inline;
        }
    </style>
    </head>
    <body>
    <h1>Hello World</h1>
    <xsl:for-each select="//Post">
        <details>
            <summary><h3><xsl:value-of select="Title"/></h3></summary>
            <p><xsl:value-of select="Body" /></p>
        </details>
    </xsl:for-each>
    </body>
    </html>
  </xsl:template>

</xsl:stylesheet>

you will have nice rendered page:

Whole code:

Program.cs

using System.Collections.Generic;
using System.Threading.Tasks;
using System.Xml;
using System.Xml.Serialization;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.Formatters;
using Microsoft.Extensions.DependencyInjection;

namespace ContentNegotiation
{
    public class Program
    {
        public static void Main(string[] args) => CreateWebHostBuilder(args).Build().Run();

        public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
                .UseStartup<Startup>();
    }

    public class MyXmlSerializerOutputFormatter : XmlSerializerOutputFormatter
    {
        protected override void Serialize(XmlSerializer xmlSerializer, XmlWriter xmlWriter, object value)
        {
            // TODO: add me only if controller has some kind of custom attribute with XSLT file name
            xmlWriter.WriteProcessingInstruction("xml-stylesheet", "type=\"text/xsl\" href=\"template.xsl\"");
            base.Serialize(xmlSerializer, xmlWriter, value);
        }
    }

    public class Startup
    {
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddMvc(options =>
            {
                options.RespectBrowserAcceptHeader = true; // default is false
                // options.OutputFormatters.Add(new XmlSerializerOutputFormatter()); // not enoug
                options.OutputFormatters.Add(new MyXmlSerializerOutputFormatter());
            })
            // .AddXmlSerializerFormatters() // does not added by default, but not enoug
            .SetCompatibilityVersion(CompatibilityVersion.Version_2_1);
        }

        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            app.UseStaticFiles();
            app.UseMvc();
        }
    }

    public class Post
    {
        public int Id { get; set; }
        public string Title { get; set; }
        public string Body { get; set; }
    }

    [ApiController]
    public class DemoController : ControllerBase
    {
        // curl -k -i -s -H 'Accept: text/xml' http://localhost:5000/posts
        // curl -k -i -s -H 'Accept: application/json' http://localhost:5000/posts
        [HttpGet]
        [Route(nameof(Posts))]
        public IEnumerable<Post> Posts() => new[] {
            new Post {
                Id = 1,
                Title = "Hello World",
                Body = "Lorem ipsum dot color"
            },
            new Post {
                Id = 2,
                Title = "Post 2",
                Body = "Lorem ipsum dot color"
            }
        };
    }
}

BigQuery Window Function Calculate User Sessions

2018-10-26T00:00:00+03:00

Combine stream of user actions with a timestamps into a windowed sessions example:

WITH data AS (
  -- user 1, session 1, morning
  SELECT TIMESTAMP('2001-01-01 07:01:00') AS timestamp,
                                        1 AS user,
                                   'view' AS action
  UNION ALL SELECT TIMESTAMP('2001-01-01 07:02:00'), 1, 'view'
  UNION ALL SELECT TIMESTAMP('2001-01-01 07:03:00'), 1, 'view'
  UNION ALL SELECT TIMESTAMP('2001-01-01 07:04:00'), 1, 'apply'

  -- user 1, session 2, midday
  UNION ALL SELECT TIMESTAMP('2001-01-01 12:01:00'), 1, 'view'
  UNION ALL SELECT TIMESTAMP('2001-01-01 12:02:00'), 1, 'view'
  UNION ALL SELECT TIMESTAMP('2001-01-01 12:03:00'), 1, 'apply'

  -- user 1, session 3, evening
  UNION ALL SELECT TIMESTAMP('2001-01-01 19:01:00'), 1, 'view'



  -- user 2, session 1, evening
  UNION ALL SELECT TIMESTAMP('2001-01-01 07:01:00'), 2, 'view'

  -- user 2, session 2, evening
  UNION ALL SELECT TIMESTAMP('2001-01-01 18:01:00'), 2, 'view'
  UNION ALL SELECT TIMESTAMP('2001-01-01 18:01:00'), 2, 'apply'

/*+---------------------+------+--------+
  |      timestamp      | user | action |
  +---------------------+------+--------+
  | 2001-01-01 07:01:00 |    1 | view   |
  | 2001-01-01 07:02:00 |    1 | view   |
  | 2001-01-01 07:03:00 |    1 | view   |
  | 2001-01-01 12:01:00 |    1 | view   |
  | 2001-01-01 12:02:00 |    1 | view   |
  | 2001-01-01 19:01:00 |    1 | view   |
  | 2001-01-01 07:04:00 |    1 | apply  |
  | 2001-01-01 12:03:00 |    1 | apply  |
  | 2001-01-01 07:01:00 |    2 | view   |
  | 2001-01-01 18:01:00 |    2 | view   |
  | 2001-01-01 18:01:00 |    2 | apply  |
  +---------------------+------+--------+*/
), data_with_previous_timestamp AS (
  SELECT
  -- POI: previous timestamp if any
  LAG(timestamp, 1) OVER (PARTITION BY user ORDER BY timestamp) AS previous_timestamp,
  *
  FROM data

/*+---------------------+---------------------+------+--------+
  | previous_timestamp  |      timestamp      | user | action |
  +---------------------+---------------------+------+--------+
  |                NULL | 2001-01-01 07:01:00 |    1 | view   | <- I have NO previous action
  | 2001-01-01 07:01:00 | 2001-01-01 07:02:00 |    1 | view   | <- I HAVE previous timestamp
  | 2001-01-01 07:02:00 | 2001-01-01 07:03:00 |    1 | view   |
  | 2001-01-01 07:03:00 | 2001-01-01 07:04:00 |    1 | apply  |
  | 2001-01-01 07:04:00 | 2001-01-01 12:01:00 |    1 | view   |
  | 2001-01-01 12:01:00 | 2001-01-01 12:02:00 |    1 | view   |
  | 2001-01-01 12:02:00 | 2001-01-01 12:03:00 |    1 | apply  |
  | 2001-01-01 12:03:00 | 2001-01-01 19:01:00 |    1 | view   |
  |                NULL | 2001-01-01 07:01:00 |    2 | view   | <- I have no previous action
  | 2001-01-01 07:01:00 | 2001-01-01 18:01:00 |    2 | view   |
  | 2001-01-01 18:01:00 | 2001-01-01 18:01:00 |    2 | apply  |
  +---------------------+---------------------+------+--------+*/
), data_with_is_new_session AS (
  SELECT
  -- POI: 30 min window
  CASE WHEN TIMESTAMP_DIFF(timestamp, previous_timestamp, MINUTE) >= 30 OR previous_timestamp IS NULL THEN 1 ELSE 0 END AS is_new_session,
  *
  FROM data_with_previous_timestamp

/*+----------------+---------------------+---------------------+------+--------+
  | is_new_session | previous_timestamp  |      timestamp      | user | action |
  +----------------+---------------------+---------------------+------+--------+
  |              1 |                NULL | 2001-01-01 07:01:00 |    1 | view   | <- user 1, session 1
  |              0 | 2001-01-01 07:01:00 | 2001-01-01 07:02:00 |    1 | view   |
  |              0 | 2001-01-01 07:02:00 | 2001-01-01 07:03:00 |    1 | view   |
  |              0 | 2001-01-01 07:03:00 | 2001-01-01 07:04:00 |    1 | apply  |
  |              1 | 2001-01-01 07:04:00 | 2001-01-01 12:01:00 |    1 | view   | <- user 1, session 2
  |              0 | 2001-01-01 12:01:00 | 2001-01-01 12:02:00 |    1 | view   |
  |              0 | 2001-01-01 12:02:00 | 2001-01-01 12:03:00 |    1 | apply  |
  |              1 | 2001-01-01 12:03:00 | 2001-01-01 19:01:00 |    1 | view   | <- user 1, session 3
  |              1 |                NULL | 2001-01-01 07:01:00 |    2 | view   | <- user 2, session 1
  |              1 | 2001-01-01 07:01:00 | 2001-01-01 18:01:00 |    2 | view   | <- user 2, session 2
  |              0 | 2001-01-01 18:01:00 | 2001-01-01 18:01:00 |    2 | apply  |
  +----------------+---------------------+---------------------+------+--------+*/
), data_with_sessions AS (
  SELECT
  -- POI: will be 1, 2, 3, 4, 5 - e.g. overall sessions
  SUM(is_new_session) OVER (ORDER BY user, timestamp) AS global_session_id,
  -- POI: will be 1, 2, 3 for user 1 and 1, 2 for user 2 - e.g. sessions per user
  SUM(is_new_session) OVER (PARTITION BY user ORDER BY timestamp) AS user_session_id,
  *
  FROM data_with_is_new_session

/*+-------------------+-----------------+----------------+---------------------+---------------------+------+--------+
  | global_session_id | user_session_id | is_new_session | previous_timestamp  |      timestamp      | user | action |
  +-------------------+-----------------+----------------+---------------------+---------------------+------+--------+
  |                 1 |               1 |              1 |                NULL | 2001-01-01 07:01:00 |    1 | view   |
  |                 1 |               1 |              0 | 2001-01-01 07:01:00 | 2001-01-01 07:02:00 |    1 | view   |
  |                 1 |               1 |              0 | 2001-01-01 07:02:00 | 2001-01-01 07:03:00 |    1 | view   |
  |                 1 |               1 |              0 | 2001-01-01 07:03:00 | 2001-01-01 07:04:00 |    1 | apply  |
  |                 2 |               2 |              1 | 2001-01-01 07:04:00 | 2001-01-01 12:01:00 |    1 | view   |
  |                 2 |               2 |              0 | 2001-01-01 12:01:00 | 2001-01-01 12:02:00 |    1 | view   |
  |                 2 |               2 |              0 | 2001-01-01 12:02:00 | 2001-01-01 12:03:00 |    1 | apply  |
  |                 3 |               3 |              1 | 2001-01-01 12:03:00 | 2001-01-01 19:01:00 |    1 | view   |
  |                 4 |               1 |              1 |                NULL | 2001-01-01 07:01:00 |    2 | view   |
  |                 5 |               2 |              1 | 2001-01-01 07:01:00 | 2001-01-01 18:01:00 |    2 | view   |
  |                 5 |               2 |              0 | 2001-01-01 18:01:00 | 2001-01-01 18:01:00 |    2 | apply  |
  +-------------------+-----------------+----------------+---------------------+---------------------+------+--------+*/
)
SELECT AVG(views) AS avg_views_per_session FROM (
  SELECT
  -- user,
  -- user_session_id,
  COUNT(*) AS views
  FROM data_with_sessions
  WHERE action = 'view'
  GROUP BY user, user_session_id
)
/*+-----------------------+
  | avg_views_per_session |
  +-----------------------+
  |                   1.6 |
  +-----------------------+*/

Main source

Note: to get ascii art use bq query --nouse_legacy_sql "select * from acme"

Dotnet Core Swashbuckle 3 Bearer Auth

2018-10-24T00:00:00+03:00

Suppose you have your API already, it is requires bearer jwt auth and you already added and configured swashbuckle.

From dotnet point of view you gonna need configure services to have JWT bearer authentication something like:

services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(options =>
    {
        options.TokenValidationParameters = new TokenValidationParameters
        {
            ValidateIssuer = false,
            ValidateAudience = false,
            ValidateLifetime = true,
            IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(
                "PASSPHRASE_TO_ENSURE_GIVEN_JWT_IS_SIGNED_BY_YOUR_AUTH_SERVER"
            ))
        };
    });

Code above is just for demo purposes, in real world you should validate everything and use something like JWKS) instead of knowing secret.

The problem with new swagger ui 3 is that it has limited auth capabilities

e.g. you gonna need to go to your auth server, authenticate there, grab your token, paste it here, and whenever you refresh window with swagger - start over :(

Customize Swagger UI 3

The right way will be to inject into react redux internals of swagger ui 3 but it is hard to achive, so we are going to do it in easier way by customizing index.html

There is two ways you can use this file:

Embed into assembly

pros - configuration from server side

cons - every change requires recompile

Save index.html in root directory of your project (e.g. side by side with Startup.cs)

In your csproj file:

<ItemGroup>
  <None Remove="index.html" />
  <EmbeddedResource Include="index.html" />
</ItemGrop>

Which will embed file into assembly, then in Startup:

app.UseSwaggerUI(c =>
{
    c.SwaggerEndpoint("/swagger/v1/swagger.json", "My API V1");

    var assembly = GetType().GetTypeInfo().Assembly;
    var ns = assembly.GetName().Name;
    c.IndexStream = () => assembly.GetManifestResourceStream($"{ns}.index.html");
});

Which will override standard index.html with one you have provided.

Notice that you need to prepend embeded file name with your assembly namespace prefix.

Static file

pros - does not require compilation for changes to take effect

cons - configuration hand coded

Just save index.html to wwwroot and add static and default files to your Startup.cs like so:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    app.UseStaticFiles();
    app.UseDefaultFiles();
    // ...
}

Few things left for this file to work:

Replace paths for script sources from relative to absolute, e.g. from ./swagger-ui-bundle.js to /swagger/swagger-ui-bundle.js.

Replace %(DocumentTitle) with whatever you want.

Remove %(HeadContent).

Replace JSON.parse('%(ConfigObject)') with something like: {"urls":[{"url":"/swagger/v1/swagger.json","name":"My API V1"}],"validatorUrl":null}, make sure to change url to your swagger.json if needed.

Replace JSON.parse('%(OAuthConfigObject)') with {}.

You can look for all this stuf from file generated by deafult by opening your http://localhost:5000/swagger/index.html.

Custom auth

Now, when we have our file in place we are going to hook inside swagger ui and make our own bearer auth.

First of all we need some styles (somewhere in head tag):

.swagger-ui.swagger-container .topbar.rua {
  background: #3b4151;
  text-align: center;
  color: white;
}

.swagger-ui.swagger-container .topbar.rua input[type="submit"] {
  margin: 5px 0;
  padding: 8px 10px;
  border: 1px solid #89bf04;
  border-radius: 4px;
  background: #89bf04;
  color: white;
}

And markup (right above <div id="swagger-ui"></div>):

<div class="swagger-ui swagger-container">
    <div class="topbar rua">
        <div class="wrapper">
            <form id="signin" hidden>
                <input id="username" type="email" placeholder="username@contoso.com" pattern=".+@contoso.com" required>
                <input id="password" type="password" placeholder="*********" required>
                <input type="submit" value="signin">
            </form>
            <form id="signout" hidden>
                <span id="hello"></span>
                <input type="submit" value="signout">
            </form>
        </div>
    </div>
</div>

Do not forget to change username pattern

Now all is left is to wireup everything, we are going to change configObject and add some new properties:

var configObject = {
  urls: [{ url: "/swagger/v1/swagger.json", name: "My API V1" }],
  validatorUrl: null,
  dom_id: "#swagger-ui",
  presets: [SwaggerUIBundle.presets.apis, SwaggerUIStandalonePreset],
  layout: "StandaloneLayout",
  requestInterceptor: req => {
    const token = localStorage.getItem("token");
    if (token) {
      req.headers.authorization = `Bearer ${token}`;
    }
    return req;
  },

  onComplete: () => {
    const signin = document.getElementById("signin");
    const signout = document.getElementById("signout");
    const hello = document.getElementById("hello");

    const update = () => {
      const token = localStorage.getItem("token");

      if (token) {
        const { email } = JSON.parse(
          window.atob(
            token
              .split(".")
              .splice(1, 1)
              .shift()
              .replace("-", "+")
              .replace("_", "/")
          )
        );
        hello.innerText = email;
        signin.hidden = true;
        signout.hidden = false;
      } else {
        signin.hidden = false;
        signout.hidden = true;
      }
    };

    signin.addEventListener("submit", event => {
      event.preventDefault();
      const options = {
        method: "POST",
        headers: {
          "Content-Type": "application/json"
        },
        body: JSON.stringify({
          username: document.getElementById("username").value,
          password: document.getElementById("password").value
        })
      };

      fetch("https://account.contoso.com/token", options)
        .then(res => res.text())
        .then(token => localStorage.setItem("token", token))
        .then(update);
    });

    signout.addEventListener("submit", event => {
      event.preventDefault();
      localStorage.removeItem("token");
      update();
    });

    update();
  }
};

More details might be found here

We have added requestInterceptor which will inject bearer token if any to requests being send to server.

Also we added onComplete which add event listeners to our forms, and performs UI update, nothing fancy here, but be sure to take a closer look at sign in handler and modify it for your needs.

Server Side Rendering with Dotnet Core

2017-11-18T00:00:00+02:00

dotnet core spa services allow you to render any frontend library on backend

detailed instructions can be found here

the simples possible example:

mkdir acme
cd acme
dotnet new mvc
dotnet add package Microsoft.AspNetCore.SpaServices
npm init -y -f
npm i -S aspnet-prerendering

few things left:

add following line to /Views/_ViewImports.cshtml

@addTagHelper *, Microsoft.AspNetCore.SpaServices

it will allow you to use asp-prerender-module tag helpers in your views

replace app.UseMvc(...) in Startup.cs with this one:

app.UseMvc(routes =>
{
    routes.MapRoute(
        name: "default",
        template: "{controller=Home}/{action=Index}/{id?}");

    routes.MapSpaFallbackRoute(
        name: "spa-fallback",
        defaults: new { controller = "Home", action = "Index" });
});

it is actually same only thing is added is MapSpaFallbackRoute wich will cach all routes to index view of home controller

now, somewhere ins your Views/Home/Index.cshtml place following:

<div id="app" asp-prerender-module="src/boot"></div>

and add src/boot.js with contents like this:

const prerendering = require('aspnet-prerendering');

module.exports = prerendering.createServerRenderer(params => {
    return new Promise((resolve, reject) => {
        const result = `
            <h1>Hello from JS</h1>
            <p>Current time in Node is: ${new Date()}</p>
            <p>Request path is: ${params.location.path}</p>
            <p>Absolute URL is: ${params.absoluteUrl}</p>
        `;
        resolve({html: result})
    });
});

and you are ready to go, try to run your app with dotnet run

you will see your mvc app as usual but take a look at our demo div which contains content generated on a server side

dotnet core itself has predefined templates for react and angular, but virtually can render anything

as a side note - we have checked and running such projects on dotnet is 3 times faster than node and seems to be more stable under load

MaxMind BigQuery GeoIP

2017-07-26T00:00:00+03:00

Idea here is to have maxmind ip lookup table in bigquery for further logs analysis

So, first of all we need clean everything up

bq rm -f -t rualogs:data.maxmind
rm *.txt *.csv *.zip

Download and unzip fresh data

wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country-CSV.zip
wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-ASN-CSV.zip

unzip -jo GeoLite2-Country-CSV.zip
unzip -jo GeoLite2-ASN-CSV.zip

Load data into BigQuery

bq load --skip_leading_rows=1 rualogs:data.tmp_country_ip GeoLite2-Country-Blocks-IPv4.csv "network:string,geoname_id:integer,registered_country_geoname_id:integer,represented_country_geoname_id:integer,is_anonymous_proxy:integer,is_satellite_provider:integer"

bq load --skip_leading_rows=1 rualogs:data.tmp_country_labels GeoLite2-Country-Locations-en.csv "geoname_id:integer,locale_code:string,continent_code:string,continent_name:string,country_iso_code:string,country_name:string"

bq load --skip_leading_rows=1 rualogs:data.tmp_asn GeoLite2-ASN-Blocks-IPv4.csv "network:string,autonomous_system_number:integer,autonomous_system_organization:string"

Now is most important stuff - we are going to process uploaded data and save result as new table

bq query --use_legacy_sql=false --allow_large_results --destination_table=rualogs:data.maxmind "$(cat maxmind.sql)"

After this step is done we may cleanup temp tables

bq rm -f -t rualogs:data.tmp_country_ip
bq rm -f -t rualogs:data.tmp_country_labels
bq rm -f -t rualogs:data.tmp_asn

From now on anywhere needed we may join our maxmind table like so

SELECT l.*, m.country_name, m.autonomous_system_organization
FROM data.log AS l
LEFT JOIN data.maxmind AS m ON CAST(NET.IPV4_TO_INT64(NET.IP_FROM_STRING(l.ip))/(256*256*256) AS INT64) = class_a AND NET.IPV4_TO_INT64(NET.IP_FROM_STRING(l.ip)) BETWEEN start_num AND end_num
WHERE (_PARTITIONTIME = TIMESTAMP(CURRENT_DATE()) OR _PARTITIONTIME IS NULL)
LIMIT 100

maxmind.sql

WITH raw AS (

SELECT
REGEXP_REPLACE(network, r'/\d+$', '') as net,
CAST(REGEXP_REPLACE(network, r'^\d+\.\d+\.\d+\.\d+/', '') AS int64) as mask,
CAST(CASE WHEN (-2 + POW(2, 32 - CAST(REGEXP_REPLACE(network, r'^\d+\.\d+\.\d+\.\d+/', '') AS int64))) < 1 THEN 1 ELSE (-2 + POW(2, 32 - CAST(REGEXP_REPLACE(network, r'^\d+\.\d+\.\d+\.\d+/', '') AS int64))) END AS INT64) as hosts,
* FROM `rualogs.data.tmp_country_ip`

), num AS (

select
NET.IPV4_TO_INT64(NET.IP_FROM_STRING(net)) + 1 as start_num,
NET.IPV4_TO_INT64(NET.IP_FROM_STRING(net)) + hosts as end_num,
* from raw

), ip AS (

select
cast(start_num/(256*256*256) as int64) as class_a,
NET.IP_TO_STRING(NET.IPV4_FROM_INT64(start_num)) as start_ip,
NET.IP_TO_STRING(NET.IPV4_FROM_INT64(end_num)) as end_ip,
* from num

), maxmind AS (

select
ip.*,
l.locale_code, l.continent_code, l.continent_name, l.country_iso_code, l.country_name,
a.autonomous_system_number, a.autonomous_system_organization
from ip
left join rualogs.data.tmp_country_labels l on ip.geoname_id = l.geoname_id
left join rualogs.data.tmp_asn a on ip.network = a.network

)
select * from maxmind

Generate custom JWT token for Firebase from dotnet core

2017-05-07T00:00:00+03:00

In my case I were trying to marry our office Active Directory with Firebase Custom Authentication

That is preatty easy task if you are using anything except C# :)

In dotnet core at moment even talking to ldap is made via 3rd party library

After googling have found nice article: Create JWT with a Private RSA Key

And after few attempts at least got working sample:

using System;
using System.Collections.Generic;
using System.IO;
using System.Security.Cryptography;
using Jose;
using Org.BouncyCastle.Crypto.Parameters;
using Org.BouncyCastle.OpenSsl;

namespace WebApplication1.Services
{
	/// <summary>
	/// Firebase Custom Token Generator
	///
	/// Authenticate with Firebase in JavaScript Using a Custom Authentication System
	/// Docs: https://firebase.google.com/docs/auth/web/custom-auth
	/// Client (firebase service account) email and private key can be retrieved here:
	/// https://console.firebase.google.com/project/_/settings/serviceaccounts/adminsdk
	///
	/// Required packages: BouncyCastle.NetCore, jose-jwt
	///
	/// Usage example:
	/// var firebase = new FirabaseCustomToken(clientEmail, privateKey);
	/// var token = firebase.CreateToken("alexandrm@rabota.ua");
	/// </summary>
	public class FirabaseCustomToken
	{
		private readonly string _clientEmail;
		private readonly string _privateKey;

		public FirabaseCustomToken(string clientEmail, string privateKey)
		{
			_privateKey = privateKey;
			_clientEmail = clientEmail;
		}

		public string CreateToken(string uid, Dictionary<string, object> claims = null)
		{
			var now = DateTimeOffset.Now.ToUnixTimeSeconds();

			var payload = new Dictionary<string, object>
			{
				{ "alg", "RS256" },
				{ "iss", _clientEmail },
				{ "sub", _clientEmail },
				{ "aud", "https://identitytoolkit.googleapis.com/google.identity.identitytoolkit.v1.IdentityToolkit" },
				{ "iat", now },
				{ "exp", now + 3600 },
				{ "uid", uid },
				{ "claims", claims ?? new Dictionary<string, object>() }
			};

			return SignToken(payload);
		}

		private string SignToken(Dictionary<string, object> payload)
		{
			string jwt;
			RsaPrivateCrtKeyParameters key;
			using (var stringReader = new StringReader(_privateKey))
			{
				var pemReader = new PemReader(stringReader);
				key = (RsaPrivateCrtKeyParameters)pemReader.ReadObject();
			}
			using (var rsa = new RSACryptoServiceProvider())
			{
				rsa.ImportParameters(ToRsaParameters(key));
				jwt = JWT.Encode(payload, rsa, JwsAlgorithm.RS256);
			}
			return jwt;
		}

		/// <summary>
		/// https://github.com/neoeinstein/bouncycastle/blob/master/crypto/src/security/DotNetUtilities.cs
		/// </summary>
		/// <param name="privKey">string</param>
		/// <returns></returns>
		private static RSAParameters ToRsaParameters(RsaPrivateCrtKeyParameters privKey) => new RSAParameters
		{
			Modulus = privKey.Modulus.ToByteArrayUnsigned(),
			Exponent = privKey.PublicExponent.ToByteArrayUnsigned(),
			D = privKey.Exponent.ToByteArrayUnsigned(),
			P = privKey.P.ToByteArrayUnsigned(),
			Q = privKey.Q.ToByteArrayUnsigned(),
			DP = privKey.DP.ToByteArrayUnsigned(),
			DQ = privKey.DQ.ToByteArrayUnsigned(),
			InverseQ = privKey.QInv.ToByteArrayUnsigned()
		};
	}
}

Further improvements may be accepting claims instead of dictionary, but unfortunatelly it seam that firebase itself does not respect claims from cutom tokens, so it is not an issue

As for Active Directory in .Net Core working sample is:

using ActiveDirectoryJsonWebToken.Models;
using Microsoft.Extensions.Options;
using Novell.Directory.Ldap;
using System.Collections.Generic;
using System.Linq;
using System.Security.Claims;
using System.Security.Principal;

namespace ActiveDirectoryJsonWebToken.Services
{
	public class IdentityService
	{
		private readonly LdapConfig _config;
		public IdentityService(IOptions<LdapConfig> config)
		{
			_config = config.Value;
		}

		public ClaimsIdentity GetIdentity(LoginModel login)
		{
			var cn = new LdapConnection();
			try
			{
				cn.Connect(_config.Hostname, LdapConnection.DEFAULT_PORT);
				cn.Bind($"RABOTA\\{login.Username.Split('@').FirstOrDefault()}", login.Password);

				var claims = new List<Claim>();
				var results = cn.Search(_config.BaseDn, LdapConnection.SCOPE_ONE, $"(mail={login.Username})", null, false);
				var entry = results.next();

				claims.Add(new Claim(ClaimTypes.Email, login.Username));
				claims.Add(new Claim(ClaimTypes.GivenName, entry.getAttribute("name").StringValue));

				var groups = entry.getAttribute("MemberOf").StringValues;
				while (groups.MoveNext())
				{
					var group = groups.Current.ToString().Split(',').FirstOrDefault()?.Replace("CN=", "");
					claims.Add(new Claim(ClaimTypes.Role, group));
				}
				cn.Disconnect();

				return new ClaimsIdentity(new GenericIdentity(login.Username.ToLower()), claims);
			}
			catch (LdapException e)
			{
				if (e.ResultCode == LdapException.INVALID_CREDENTIALS)
				{
					return null;
				}

				throw e;
			}
		}
	}
}

It does require Novell.Directory.Ldap.NETStandard package to be installed to work

All is left is just to marry all this, but after playing around I have decided to run stuff in node windows container so having best from both worlds, leaving this just for future possible reuse

Bitbucket mass hook addition

2017-04-15T00:00:00+03:00

Suppose you have many repositories and wish to ensure that each has slack hook with desired url

Here is how you can massively update all your repositories with powershell:

$account = 'rabotaua'
$slackDesiredUrl = 'https://hooks.slack.com/services/*********/*********/************************'
$slackHookName = 'Slack'
$username = 'mac2000'
$password = '*******************'

$authorization = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("$($username):$($password)"))
$headers = @{ Authorization = "Basic $authorization" }

$hookBody = @{
	description = $slackHookName
	url = $slackDesiredUrl
	active = $true
	events = @('repo:push')
}

$repositories = @()
$response = @{ next = "https://api.bitbucket.org/2.0/repositories/$account" }
do {
	$response = Invoke-RestMethod -Headers $headers -Uri $response.next
	$repositories += $response.values
} while($response.next)
Write-Host "Retrieved $($repositories.Count) repositories from `"$account`" account" -ForegroundColor Cyan

foreach($repository in $repositories) {
	Write-Host $repository.name -NoNewline
	$hooks = Invoke-RestMethod -Headers $headers -Uri $repository.links.hooks.href
	$slackHook = $hooks.values |? description -EQ $slackHookName | select -First 1
	if ($slackHook) {
		if ($slackHook.url -ne $slackDesiredUrl) {
			Invoke-RestMethod -Method Put -Headers $headers -Uri $slackHook.links.self.href -ContentType 'application/json' -Body ($hookBody | ConvertTo-Json) | Out-Null
			Write-Host " existing hook updated" -ForegroundColor Yellow
		} else {
			Write-Host " nothing to do" -ForegroundColor Cyan
		}
	} else {
		Invoke-RestMethod -Method Post -Headers $headers -Uri $repository.links.hooks.href -ContentType 'application/json' -Body ($hookBody | ConvertTo-Json) | Out-Null
		Write-Host " new hook created" -ForegroundColor Green
	}
}

SqlDataReader Mapper

2016-07-07T00:00:00+03:00

SqlDataReader simple mapper extension with XML support

In my case I have following setup:

USE master
GO
IF EXISTS(SELECT * FROM sys.databases WHERE name='DataMapperTests')
BEGIN
	ALTER DATABASE DataMapperTests SET SINGLE_USER WITH ROLLBACK IMMEDIATE
	DROP DATABASE DataMapperTests
END
CREATE DATABASE DataMapperTests
GO
USE DataMapperTests
GO

PRINT 'Creating tables'
GO

CREATE TABLE Tag (
	Id INT IDENTITY(1, 1) PRIMARY KEY,
	Name NVARCHAR(50) NOT NULL
)

CREATE TABLE Post (
	Id INT IDENTITY(1, 1) PRIMARY KEY,
	Title NVARCHAR(50) NOT NULL,
	Published BIT NOT NULL DEFAULT 0,
	CreatedAt DATETIME2 NOT NULL DEFAULT CONVERT(DATE, GETDATE()),
	PublishedAt DATETIME2
)

CREATE TABLE PostTags (
	PostId INT NOT NULL,
	TagId INT NOT NULL,
	PRIMARY KEY (PostId, TagId),
	CONSTRAINT FK_Post FOREIGN KEY (PostId) REFERENCES Post(Id) ON DELETE CASCADE,
	CONSTRAINT FK_Tag FOREIGN KEY (TagId) REFERENCES Tag(Id) ON DELETE CASCADE
)
GO

PRINT 'Seed tables'
GO

SET NOCOUNT ON

SET IDENTITY_INSERT Tag ON
INSERT INTO Tag (Id, Name) VALUES
	(1, 'Tag 1'),
	(2, 'Tag 2');
SET IDENTITY_INSERT Tag OFF
GO

SET IDENTITY_INSERT Post ON
INSERT INTO Post (Id, Title, Published, PublishedAt) VALUES
	(1, 'Post 1', 1, CONVERT(DATE, GETDATE())),
	(2, 'Post 2', 0, NULL),
	(3, 'Post 3', 1, CONVERT(DATE, GETDATE()));
SET IDENTITY_INSERT Post OFF
GO

INSERT INTO PostTags (PostId, TagId) VALUES
	(1, 1),
	(2, 1), (2, 2);
GO

PRINT 'Create view'
GO

CREATE VIEW SampleView AS
SELECT
	Id,
	Title,
	CreatedAt,
	Published,
	PublishedAt,

	ISNULL((SELECT DISTINCT
		LTRIM(RTRIM(T.Name))
		FROM PostTags AS PT
		JOIN Tag AS T ON PT.TagId = T.Id AND PT.PostId = P.Id
		FOR XML PATH ('string'), ROOT('ArrayOfString'), TYPE), '<ArrayOfString></ArrayOfString>') AS ListString,

	ISNULL((SELECT DISTINCT
		LTRIM(RTRIM(T.Id))
		FROM PostTags AS PT
		JOIN Tag AS T ON PT.TagId = T.Id AND PT.PostId = P.Id
		FOR XML PATH ('int'), ROOT('ArrayOfInt'), TYPE), '<ArrayOfInt></ArrayOfInt>') AS ListInt,

	ISNULL((SELECT DISTINCT
		T.Id AS Id,
		LTRIM(RTRIM(T.Name)) AS Name
		FROM PostTags AS PT
		JOIN Tag AS T ON PT.TagId = T.Id AND PT.PostId = P.Id
		FOR XML PATH ('Tag'), ROOT('ArrayOfTag'), TYPE), '<ArrayOfTag></ArrayOfTag>') AS ListTag

FROM Post AS P
GO

Sample view is returning following data:

| Name        | Type               |
| ----------- | ------------------ |
| Id          | int                |
| Title       | string             |
| CreatedAt   | DateTime           |
| Published   | bool               |
| PublishedAt | DateTime           |
| ListString  | List&lt;string&gt; |
| ListInt     | List&lt;int&gt;    |
| ListTag     | List&lt;Tag&gt;    |

Our models are:

public class Tag
{
	public int Id { get; set; }
	public string Name { get; set; }
}

public class Post
{
	public int Id { get; set; }
	public string Title { get; set; }
	public bool Published { get; set; }
	public DateTime PublishedAt { get; set; }
	public DateTime CreatedAt { get; set; }

	public List<int> ListInt { get; set; }
	public List<string> ListString { get; set; }
	public List<Tag> ListTag { get; set; }
}

Here is SqlDataReaderMapperExtension:

public static class SqlDataReaderExtensions
{
	private static readonly MemoryCache Cache = MemoryCache.Default;

	public static T Map<T>(this SqlDataReader reader) where T : new()
	{
		var properties = GetProperties(typeof(T));

		var item = new T();

		for (var i = 0; i < reader.FieldCount; i++)
		{
			if (reader.IsDBNull(i)) continue;

			var property = properties[NormalizeKey(reader.GetName(i))].FirstOrDefault();

			if (property == null) continue;

			if (reader.GetFieldType(i) == property.PropertyType)
				property.SetValue(item, reader[i]);
			else if (reader.GetProviderSpecificFieldType(i) == typeof(SqlXml))
				property.SetValue(item, new XmlSerializer(property.PropertyType).Deserialize(reader.GetXmlReader(i)));
		}

		return item;
	}

	private static ILookup<string, PropertyInfo> GetProperties(Type type)
	{
		var cacheKey = $"{nameof(SqlDataReaderExtensions)}.{type.FullName}";
		var result = Cache.Get(cacheKey) as ILookup<string, PropertyInfo>;

		if (result != null) return result;

		result = type.GetProperties(BindingFlags.Public | BindingFlags.Instance).Where(p => p.CanWrite).ToLookup(p => NormalizeKey(p.Name));
		Cache.Add(cacheKey, result, null);

		return result;
	}

	private static string NormalizeKey(string name)
	{
		return name.Replace("_", "").ToLower().Trim();
	}
}

which will allow you to get objects like this:

using (var command = new SqlCommand(Query, GetSqlConnection()))
{
	command.Connection.Open();
	using (var reader = command.ExecuteReader())
	{
		if (reader.HasRows)
		{
			while (reader.Read())
			{
				var post = reader.Map<Post>();
			}
		}
	}
}

and here is event shorten way with command extension

public static class SqlCommandExtensions
{
	public static IEnumerable<T> ExecuteReader<T>(this SqlCommand command) where T : new()
	{
		command.Connection.Open();
		using (var reader = command.ExecuteReader())
		{
			if (!reader.HasRows) yield break;

			while (reader.Read())
			{
				yield return reader.Map<T>();
			}
		}
		command.Connection.Close();
	}
}

and now you can do something like this:

using (var command = new SqlCommand(Query, GetSqlConnection()))
{
	var posts = command.ExecuteReader<Post>();
}

Extension were written to handle xml in first place and for demo purposes

Merge Console Application into Single Executable

2016-05-23T00:00:00+03:00

Goal is to have nice and clean output for release build.

At minimum you may:

Turn off vhost.exe by unchecking “Enable the Visual Studio hosting process” under “Debug” tab of your project proerties. Make sure to do this only for release configuration.

You can also do it by hands, by adding <UseVSHostingProcess>false</UseVSHostingProcess> to the <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' "> of your csproj file.

Next step may be removing pdb and xml files that are copied with referenced libraries, to do so, you may want to add afterbuild target to your cproj file like this:

<Target Name="AfterBuild" Condition="'$(Configuration)' == 'Release'">
	<Delete Files="$(OutputPath)Elasticsearch.Net.pdb" />
	<Delete Files="$(OutputPath)Nest.pdb" />
	<Delete Files="$(OutputPath)Elasticsearch.Net.xml" />
	<Delete Files="$(OutputPath)Nest.xml" />
</Target>

Or if you want to delete everything at once you may do something like this:

<Target Name="AfterBuild" Condition="'$(Configuration)' == 'Release'">
	<ItemGroup>
		<FilesToDelete Include="$(OutputPath)*.pdb"/>
		<FilesToDelete Include="$(OutputPath)*.xml"/>
	</ItemGroup>
	<Message Text="Delete: @(FilesToDelete, ', ')" Importance="High" />
	<Delete Files="@(FilesToDelete)" />
</Target>

ItemGroup is a way to list files, that able to use asteriks.

@(FilesToDelete, ', ') is equivalent to string.Join(", ", FilesToDelete)

By default Visual Studio displays minimal output from MSBuild, so to see your own, do not forget to add Importance="High"

There is option to not create any pdb files in advanced compilation options, but it seems that it is works only for your own class libraries

Merge dll into exe

First of all we need ILMerge tool, you may install it via nuget: Install-Package ILMerge

<Target Name="AfterBuild" Condition="'$(Configuration)' == 'Release'">
	<ConvertToAbsolutePath Paths="$(OutputPath)">
		<Output TaskParameter="AbsolutePaths" PropertyName="OutputFullPath" />
	</ConvertToAbsolutePath>
	<ItemGroup>
		<MergeAssemblies Include="$(OutputPath)$(MSBuildProjectName).exe" />
		<MergeAssemblies Include="$(OutputPath)Newtonsoft.Json.dll" />
	</ItemGroup>
	<PropertyGroup>
		<OutputAssembly>$(OutputFullPath)$(MSBuildProjectName).Standalone.exe</OutputAssembly>
		<Merger>$(SolutionDir)packages\ILMerge.2.14.1208\tools\ILMerge.exe</Merger>
	</PropertyGroup>
	<Message Text="Merge -&gt; $(OutputAssembly)" Importance="High" />
	<Exec Command="&quot;$(Merger)&quot; /target:exe /out:&quot;$(OutputAssembly)&quot; @(MergeAssemblies->'&quot;%(FullPath)&quot;', ' ')" />
	<ItemGroup>
		<FilesToDelete Include="$(OutputPath)$(MSBuildProjectName).exe" />
		<FilesToDelete Include="$(OutputPath)*.pdb" />
		<FilesToDelete Include="$(OutputPath)*.xml" />
		<FilesToDelete Include="$(OutputPath)*.dll" />
	</ItemGroup>
	<Message Text="Cleanup -&gt; @(FilesToDelete, ', ')" Importance="High" />
	<Delete Files="@(FilesToDelete)" />
	<Copy SourceFiles="$(OutputAssembly)" DestinationFiles="$(OutputPath)$(MSBuildProjectName).exe" />
	<Delete Files="$(OutputAssembly)" />
</Target>

ConvertToAbsolutePath is used to get full output path.

MergeAssemblies item group should contain list of referenced libraries (dll) and executable itself

PropertyGroup is used just to define some variables for later use

Note that later on, ILMerge package version may change, you should not forget to fix verion number

The main job is going to be in: Exec which is going to run something like:

ILMerge.exe /target:exe /out:App.Standalone.exe App.exe Newtonsoft.Json.dll

/target may be exe or winexe depending of project you are building

Notice that we can not write to file we are building, so by default we are mering into something like App.Standalone.exe and then replacing original one with out. It need to be done to not brake ConfigurationManager if one is used.

Transform Web.config after build with computername

2016-01-28T00:00:00+02:00

The goal is to run Web.config transformation after build if there is Web.COMPUTERNAME.config file.

Here is sample web.config file

Web.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<appSettings>
		<add key="Sample" value="Hello World"/> <!-- We are going change this -->
	</appSettings>
	<connectionStrings>
		<add name="Default" connectionString="Data Source=localhost;Initial Catalog=Northwind;Integrated Security=True" providerName="System.Data.SqlClient"/> <!-- And this -->
	</connectionStrings>
	<system.web>
		<compilation debug="true" targetFramework="4.5.2"/>
		<httpRuntime targetFramework="4.5.2"/>
	</system.web>
	<system.serviceModel>
		<bindings>
			<basicHttpBinding>
				<binding name="JobsearcherWSSoap" />
			</basicHttpBinding>
			<customBinding>
				<binding name="JobsearcherWSSoap12">
					<textMessageEncoding messageVersion="Soap12" />
					<httpTransport />
				</binding>
			</customBinding>
		</bindings>
		<client>
			<endpoint address="http://example.com/acme.asmx"
				binding="basicHttpBinding" bindingConfiguration="AcmeWSSoap"
				contract="ServiceReferenceSample.AcmeWSSoap" name="AcmeWSSoap" /> <!-- And this -->
		</client>
	</system.serviceModel>
</configuration>

Web.Mac.config

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
	<connectionStrings>
		<add name="Default" connectionString="Data Source=127.0.0.1;Initial Catalog=Northwind;Integrated Security=True" xdt:Transform="SetAttributes" xdt:Locator="Match(name)"/>
	</connectionStrings>
	<appSettings>
		<add key="Sample" value="MAC was here" xdt:Transform="SetAttributes" xdt:Locator="Match(key)"/>
	</appSettings>
	<system.serviceModel>
		<client>
			<endpoint name="AcmeWSSoap" address="http://127.0.0.1/acme.asmx" xdt:Transform="SetAttributes" xdt:Locator="Match(name)" />
		</client>
	</system.serviceModel>
</configuration>

Transformation rules are simple enought and can be copy pasted as much as needed.

To make your configuration file appear under Web.config like Release and Debug change your csproj like this:

<ItemGroup>
	...
	<None Include="Web.Debug.config">
		<DependentUpon>Web.config</DependentUpon>
	</None>
	<!--
	Was:
	<None Include="Web.Mac.config" / >
	Now:
	-->
	<None Include="Web.Mac.config">
		<DependentUpon>Web.config</DependentUpon>
	</None>
	<None Include="Web.Release.config">
		<DependentUpon>Web.config</DependentUpon>
	</None>
</ItemGroup>

Other notes:

If you want to use messages for debuging and do not see them in output window - increase verbosity of msbuild

Tools \ Options \ Projects and Solutions \ Build and Run: And set MSBUild verbosity from minimal to normal.

You can use environment variables like so:

<Message Text="HELLO WORLD $(COMPUTERNAME)" />

It seems that there can be only one before/after build target

Powershell Invoke-WebRequest against Azure Storage Tables

2015-12-11T00:00:00+02:00

To perform rest requests to azure storage tables you must sign your request, here is sample piece of code demonstrating how can it be done in Powershell:

$accountName = 'contoso'
$accountKey = '**************************************************************************************=='
$tableName = 'mytable'

$uri = "http://$accountName.table.core.windows.net/$tableName(PartitionKey='tasksSeattle',RowKey='1')"

$date = [datetime]::UtcNow.ToString('R', [System.Globalization.CultureInfo]::InvariantCulture)

$resource = [uri] $uri | select -ExpandProperty AbsolutePath # Path without query string

$stringToSign = $date + "`n/" + $accountName + $resource

$hasher = New-Object System.Security.Cryptography.HMACSHA256
$hasher.Key = [Convert]::FromBase64String($accountKey)
$signedSignature = [Convert]::ToBase64String($hasher.ComputeHash([Text.Encoding]::UTF8.GetBytes($stringToSign)))
$authorizationHeader = "SharedKeyLite " + $accountName + ":" + $signedSignature

$headers = @{
	Authorization = $authorizationHeader
	Date = $date
}

Write-Host 'Perform first request' -ForegroundColor Cyan
$response = Invoke-WebRequest -Uri $uri -Headers $headers
$response | select StatusCode, StatusDescription, @{n='ETag';e={ $_.Headers.ETag }}
$xml = [xml]$response.Content
$xml.entry.content.properties | fl

That code will return something like:

PartitionKey : tasksSeattle
RowKey       : 1
Timestamp    : Timestamp
Description  : Take out the trash.
DueDate      : DueDate
Location     : Home

Make sure to provide correct values, otherwise you will likely get Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signa ture. error.

Azure tables and conditional headers

It seems that Azure tables do not support conditional headers like If-Modified-Since like blobs do, but there is still one trick.

If you will make Invoke-WebRequest instead of Invoke-RestMethod you will also get response headers. And there will be ETag header - which is entity timestamp.

$response = Invoke-WebRequest -Uri $uri -Headers $headers
$response | select StatusCode, StatusDescription, @{n='ETag';e={ $_.Headers.ETag }} | fl
$xml = [xml]$response.Content
$xml.entry.content.properties | fl

Will return:

StatusCode        : 200
StatusDescription : OK
ETag              : W/"datetime'2015-12-11T10%3A53%3A52.6115577Z'"

PartitionKey : tasksSeattle
RowKey       : 1
Timestamp    : Timestamp
Description  : Take out the trash.
DueDate      : DueDate
Location     : Home

And now you can perform requests like:

$headers = @{
	Authorization = $authorizationHeader
	Date = $date
	'If-None-Match' = $response.Headers.ETag
}
Invoke-RestMethod -Uri $uri -Headers $headers

Which will give you desired (304) Not Modified.

Bootstraping example table from php

composer.json

{
	"repositories": [
		{
			"type": "pear",
			"url": "http://pear.php.net"
		}
	],
	"require": {
		"pear-pear.php.net/mail_mime" : "*",
		"pear-pear.php.net/http_request2" : "*",
		"pear-pear.php.net/mail_mimedecode" : "*",
		"microsoft/windowsazure": "*"
	}
}

sandbox.php

<?php
require_once 'vendor\autoload.php';
use WindowsAzure\Common\ServiceException;
use WindowsAzure\Common\ServicesBuilder;
use WindowsAzure\Table\Internal\ITable;
use WindowsAzure\Table\Models\EdmType;
use WindowsAzure\Table\Models\Entity;

$accountName = "contoso";
$accountKey = "**************************************************************************************==";
$connectionString = "DefaultEndpointsProtocol=https;AccountName=$accountName;AccountKey=$accountKey";

/** @var ITable $tableRestProxy */
$tableRestProxy = ServicesBuilder::getInstance()->createTableService($connectionString);

try {
	$createTableResult = $tableRestProxy->createTable("mytable");
} catch(ServiceException $ex) {
	if($ex->getCode() !== 409) throw $ex;
}

$entity = new Entity();
$entity->setPartitionKey("tasksSeattle");
$entity->setRowKey("1");
$entity->addProperty("Description", null, "Take out the trash.");
$entity->addProperty("DueDate", EdmType::DATETIME, new DateTime("2012-11-05T08:15:00-08:00"));
$entity->addProperty("Location", EdmType::STRING, "Home");

try {
	$tableRestProxy->insertEntity("mytable", $entity);
} catch(ServiceException $ex) {
	if($ex->getCode() !== 409) throw $ex;
}

Some links

ETag

Specifying Conditional Headers for Blob Service Operations

Authenticating against Azure Table Storage

Authentication for the Azure Storage Services

How to use table storage from PHP

TortoiseSVN Force UTF-8

2015-11-19T00:00:00+02:00

To prevent commits with not valid UTF-8 files in it you may create pre_commit_hook in your TortoiseSVN and from now on do not worry about encodings.

The trick is to check is file is valid utf8 one. In linux there is isutf8 utility which checks file in pretty nice way - it compares raw bytes in file with bytes its retrieve while reading file as utf8 one - if they are same - then file is valid.

Setup pre commit hook

Right click on your repository, navigate to TortoiseSVN \ Settings \ Hook Scripts and press Add…

Hook Type: Pre-Commit Hook

Working Copy Path: Path to your repository, e.g.: C:\Users\Alexandr\Desktop\Sample

Command Line To Execute: powershell -executionpolicy bypass -file C:\Users\Alexandr\Desktop\ForceUTF8.ps1

Check Wait for the script to finish and Hide the script while running checkboxes

Force UTF8

Here is ForceUTF8.ps1:

$errors = 0

Get-Content -Path $args[0] | %{
	if(Test-Path $_) {
		$bytes1 = Get-Content -Path $_ -Encoding Byte -Raw
		$bytes2 = [System.Text.Encoding]::UTF8.GetBytes((Get-Content -Path $_ -Encoding UTF8 -Raw))

		if(Compare-Object $bytes1 $bytes2) {
			Write-Error $_
			$errors += 1
		}
	}
}

exit $errors

As first argument $args[0] to our script we retrieve path to temporary file that contains file paths that are going to be commited.

We are checking each file to be valid UTF-8, and if its not, we are writing then to stderr and returning non zero exit code.

From now one if you will try to commit non utf8 files you will get something like this:

Powershell check is there non UTF-8 files

Get-ChildItem -Path C:\Rabota.UA\trunk\Proj1, C:\Rabota.UA\trunk\Proj2 -Recurse -Include *.master, *.ascx, *.aspx, *.cs, *.ashx -ErrorAction SilentlyContinue | %{
	$Bytes1 = Get-Content -Path $_.FullName -Encoding Byte -Raw
	$Bytes2 = [System.Text.Encoding]::UTF8.GetBytes((Get-Content -Path $_.FullName -Encoding UTF8 -Raw))

	# UTF-8 with BOM
	if($Bytes1[0] -eq 239 -and $Bytes1[1] -eq 187 -and $Bytes1[2] -eq 191) {
		$Bytes1 = $Bytes1 | select -Skip 3
	}

	# No valid UTF-8
	if(Compare-Object $Bytes1 $Bytes2) {
		Write-Host $Path
	}
}

Automate JIRA version management

2015-11-19T00:00:00+02:00

In our workflow we do use versions in Jira, but there is never ending story, we are forgetting to add new versions, release old and so on.

What do we have is knowledge which version is current.

So here are some ththoughts about automating that.

Example of REST API request from PowerShell

$username = 'your_user_name'
$password = '********************************'
$authorization = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("${username}:${password}"))
$headers = @{ Authorization = "Basic $authorization" }

Invoke-RestMethod -Uri http://jira.example.com:85/rest/api/2/project/RUA/versions -Headers $headers | Format-Table projectId, id, name, archived, released, releaseDate -AutoSize

Retrieve versions

https://docs.atlassian.com/jira/REST/6.1.7/#d2e231

Invoke-RestMethod -Uri http://jira.example.com:85/rest/api/2/project/RUA/versions -Headers $headers

Create version

https://docs.atlassian.com/jira/REST/6.1.7/#d2e537

$body = @{
	name = '73.8'
	projectId = 10040
}
Invoke-RestMethod -Uri http://jira.example.com:85/rest/api/2/version -Headers $headers -Method Post -ContentType 'application/json' -Body ($body | ConvertTo-Json)

Release version

https://docs.atlassian.com/jira/REST/6.1.7/#d2e607

$versionId = 13710
$body = @{
	archived = $false
	released = $true
	releaseDate = (Get-Date -Format 'yyyy-MM-dd')
}
Invoke-RestMethod -Uri "http://jira.example.com:85/rest/api/2/version/$versionId" -Headers $headers-Method Put -ContentType 'application/json' -Body ($body | ConvertTo-Json)

Version comprasion

Powershell actually have special type for versions and can compare them

[version]'71.3' -lt [version]'71.5'

Create new version if required

$ErrorActionPreference = 'Stop'
$CurrentVersion = [version]((Invoke-WebRequest http://test.example.com/version.aspx | Select-Object -ExpandProperty AllElements | Where-Object id -EQ data | Select-Object -ExpandProperty innerHTML) -split ' ' | Select-Object -First 1)
$NextVersionsCount = Invoke-RestMethod -Uri http://jira.example.com:85/rest/api/2/project/RUA/versions -Headers @{ Authorization = "Basic ***************************==" } | Select-Object -ExpandProperty SyncRoot | Where-Object name -Like "$($CurrentVersion.Major).*" | Select-Object *, @{n='version';e={ [version]$_.name }} | Where-Object version -gt $CurrentVersion | Measure-Object | Select-Object -ExpandProperty Count

if($NextVersionsCount -eq 0) {
	$Body = @{
		name = "$($CurrentVersion.Major).$($CurrentVersion.Minor + 1)"
		projectId = 10040
	}
	Invoke-RestMethod -Uri http://jira.rabota.ua:85/rest/api/2/version -Headers @{ Authorization = "Basic ***************************==" } -Method Post -ContentType 'application/json' -Body ($Body | ConvertTo-Json)
}

So, from now on responsibility to create new versions in jira is on your machine not by you.