Running Commentary

A Dive in Data Lake

noreply@blogger.com (Waterfox) — Tue, 24 Dec 2024 20:04:00 +0000

Contract Testing

noreply@blogger.com (Waterfox) — Wed, 07 Jun 2023 17:59:00 +0000

Why and what is contract testing?

The distributed nature of microservices, combined with the sheer number that live within an application, make it much harder for developers to perform the integration tests that were a straightforward, routine part of monolithic app development.

Contract testing is a technique for testing an integration point by checking each application in isolation to ensure the messages it sends or receives conform to a shared understanding that is documented in a "contract".

Consumer-driven contract testing is a type of contract testing that ensures that a provider is compatible with the expectations that the consumer has of it. This involves checking that the provider accepts the expected requests, and that it returns the expected responses.

Consumer contract testing provides the following benefits:

Contract tests provide a mechanism to explicitly verify that a provider microservice meets a consumer’s contract/needs.
These HTTP requests and responses are used in the mock HTTP server to mock the service provider. The interactions are then used to generate the contract between a service consumer and a service provider.
If the provider returns something unexpected, Pact marks the interaction as a failure, and the contract test fails.
A consumer’s contract test is made available to the provider to verify. By testing each side of an integration point using a simulated version of the other microservice, we get two sets of tests, one to define the interaction required and the other (playback) to verify the provider is meeting the needs defined in the consumer contract.

Contract testing for microservices is not without challenges.

The main challenge is that you'll need to run many individual tests to evaluate all microservices within your application. In this sense, contract testing is trickier to orchestrate than end-to-end integration testing. Integration testing only requires one testing environment and one set of tests. Contract testing requires teams to run a long series of tests, each for a different pair of microservices.

Another major limitation of contract testing revolves around the use of mocks. Although a mock is a reliable way of mimicking microservice behavior without running a microservice, there's no guarantee that the actual microservice will behave in the same way when in production. Due to the nature of the instances spun up, integration testing isn't subject to this same limitation.

Finally, teams must be sure to keep contracts up to date. The contract testing process needs to loop in each development team and its members so that they can help to define new contract requirements whenever they make changes to a microservice's functionality or behavior. Otherwise, tests will become based on contracts that no longer accurately reflect the desired behavior of a microservice.

A very high level introduction to Docker

noreply@blogger.com (Waterfox) — Tue, 14 Jun 2022 04:47:00 +0000

This is the powerpoint presentation I used when giving a session to my team about Docker. Tried to start with fundamental concepts and go up to using Docker Compose.

Example of Azure Event Trigger Function implemented in Java

noreply@blogger.com (Waterfox) — Mon, 17 Jan 2022 04:34:00 +0000

There are so few examples of Azure Functions implemented in Java and even official documentation is a little confusing, so I decided to write my own post explaining some of the points.

So here is my function class:

public class EventHubTriggerFunction {
    /**
     * This function will be invoked when an event is received from Event Hub.
     */
    @FunctionName("EventHubTrigger-Java")
    public void run(
        @EventHubTrigger(name = "message",
            eventHubName = "my-event-hubs",
            connection = "EventHubConnectionString",
            consumerGroup = "$Default"
            ) String message,
        final ExecutionContext context
    ) {
        context.getLogger().info("Java Event Hub trigger function executed.");
        context.getLogger().info("Message:" + message);
    }
}

The noteworthy thing in this snippet are the annotation attributes.

name: Any name that can be given to the trigger.

eventHubName: Name of the event hubs instance that will be used to receieve messages.

connection: Name of the property which stores Event Hubs Namespace (not the Event Hubs instance) connection string. What does this mean? It means that we don't have to give the connection string here, but define the conection string in local.settings.json (when running locally) or in Configuration page in the portal.

Here is the local.settings.json file. 'EventHubConnectionString' is the key that has been given in the connection attribute above.

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "",
    "FUNCTIONS_WORKER_RUNTIME": "java",
    "EventHubConnectionString": "Endpoint=sb://eventhubs.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=accesskeystring"
  }

This same key/value has to be added in the function app configuration on the portal:



By default the EventHub Trigger assumes that the message is in JSON format and it will give following error:

Exception while executing function: Functions.EventHubExample. Microsoft.Azure.WebJobs.Host: Exception binding parameter message.

Microsoft.Azure.WebJobs.Host: Binding parameters to complex objects (such as Object) uses JSON.NET serialization.

To get rid of this, either the message should be a valid JSON or we need to specify the 'dataType' parameter for the trigger:

public void run(
    @EventHubTrigger(name = "message",
        eventHubName = "my-event-hubs",
        connection = "EventHubConnectionString",
        consumerGroup = "$Default",
        dataType = "string"
        ) String message

The behavior of trigger is controlled by certain attributes. The various runtime attributes and their default values are mentioned here. We can override these values in our host.json file:
{
  "version": "2.0",
  "extensions": {
    "eventHubs": {
      "batchCheckpointFrequency": 5,
      "eventProcessorOptions": {
        "maxBatchSize": 1,
        "prefetchCount": 512
      },
      "initialOffsetOptions": {
        "type": "fromStart",
        "enqueuedTimeUtc": ""
      }
    }
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[2.*, 3.0.0)"
  }
}
I think this is all there is to implement and run a simple event hubs trigger function in Java.
Source: Github

How I converted a table in an email to a graph on a website? Hint: Azure Functions and Logic App

noreply@blogger.com (Waterfox) — Sun, 05 Dec 2021 18:35:00 +0000

This mini-project started as a necessity.

I get an email like this everyday, which gives daily numbers for different metrics. The problem with such an email is that it is difficult to keep track of or gauge the rate of growth of these metrics. If you want to know how what the previous day's numbers were you need to go to the previous day's email.

The better option any day is to have a graph that shows the values of the metrics every day. Something like this:

With this goal I set out to make this system but with as less coding and expenses as possible.

First obvious step in the process was to get hold of the email and extract its text. The easiest way to do this is to use Azure Logic App.

Here is what my logic app looks like:

When any new email arrives in my Outlook 365 account, with a subject that contains "System usage Stats as on", it passes the body of the email to an Azure function named 'EmailTextCleanerFunction' (code below), whose job is to clean any HTML text.

The Azure function returns cleaned-up text, which is then written to a file in the "attachments" folder. The name of the file is "<sender email id>.txt".

Congratulations! We have crossed our first hurdle. We have got our source data neatly written in a text file.

Before we proceed further, here is our EmailTextCleanerFunction.

Now that we have the email body written in a file, we need to start our next step: actually reading the text and extracting the data to keep it in a form that we can use to create the graph.

And this has to be done every time a file is written in the 'attachments' folder so we create a new Blob Trigger function, which gets triggered whenever a new file drops in this folder.

This blob trigger function, named very simply BlobTriggerFunction, lives in EmailBodyProcessorFunctionApp. There is another function named 'HttpTriggerFunction' as well, we will talk about it later.

Here is the function definition. In essence, this function is the heart of the whole system. It reads the text file, extracts the dates, metrics, and values, and writes all of this in a file named "<metrics>.txt" in a folder named "data-files".

Following values need to be set up in the application insights to ensure the function runs properly. Just Azure things!

Instead of writing the data in files, we could have used a database. But obviously, files are much cheaper so I used the files.

Here are all the different files in the "data-files" folder:

As I said earlier, we need the data in a format that can be used to plot the date-wise chart. So we have to write the values along with the dates.

Now we have our data stored in a datewise format in different files so anytime we want the data for a specific metric, we need to just read the file. And this is where the HttpTriggerFunction comes into the picture.

This is, as the name suggests, an HttpTriggerFunction which, when gets a GET request for a specific metric, returns the data read from the file. Our HTML file actually makes use of these GET requests to render the data as a chart.

This brings us to the last piece of the puzzle, the front end. A simple JS library (CanvasJS) in a static HTML file is hosted as a 'static website' in Azure Blob Storage.

Every time we select a metric in the dropdown in the HTML file, it sends an API call to /api/HttpTriggerFunction?name=<metric> and the data is returned to be displayed by the JS library. Actually, I set the format of the data in the files according to what was needed by the library.

So this is all that there is to it!

See and compare Microsoft's new Default Font candidates

noreply@blogger.com (Waterfox) — Mon, 17 May 2021 09:38:00 +0000

If like me you wanted to see Microsoft's five new default font candidates, one of which is supposed to replace Calibri, but don't have access to Microsoft 365 (which is the only place these fonts are currently available), you have come to the right place.

Browse through the following screenshots and you will have a decent idea of what each font will look and feel like in real-life conditions.

How do these fonts look in Powerpoint slides?

Now that you have seen everything, what do you think? It's ok if you are unable to decide at once. Take your time and once you are done and have selected a winner, head over to this Twitter post and tell Microsoft which one you like. And tell me also in the comments.

CyclicBarrier Example in Java

noreply@blogger.com (Waterfox) — Tue, 09 Feb 2021 09:25:00 +0000

Here is a simple example of how and when to use CyclicBarrier in Java.

CountDownLatch Example in Java

noreply@blogger.com (Waterfox) — Tue, 09 Feb 2021 08:39:00 +0000

Here is a simple example of how and when to use CountDownLatch in Java.

Azure Data Lake Analytics: In short

noreply@blogger.com (Waterfox) — Thu, 05 Nov 2020 10:46:00 +0000

Azure is a great cloud platform but its services have mostly terrible names and confusing, often heavily overlapping features between them.
And so another one in this series of confusing services is Azure Data Lake. I have tried to make sense of this service and what it does!

Azure Data Lake

Azure Data Lake has two components, which are very tightly integrated:

a. Azure Data Lake Analytics (ADLA)

b. Azure Data Lake Storage (ADLS)

ADLA gives us the ability to run analytical jobs (query, extract, aggregate, transform output etc.) on data, which is stored in ADLS, in form of files.

Below ‘temp’ is the name of ADLA account. As we can see ‘tempdl’ is the name of associated ADLS account, which acts as the data source for this ADLA account.

As we can see below, the ‘tempdl’ has got following data stored in form of files.

ADLA lets us query, transform and do other operations on this data using USQL (properietary language, a combination of SQL and C#) and save the result data again in form of files in the ADLS account.

How does ADLA do that?

In form of jobs. Below we can see a job named ‘Query a TSV file’, which is a U-SQL type job.

Below we see details of the above-mentioned job.

Below is the U-SQL script used to do this task of querying the file named ‘SearchLog.tsv’.

As mentioned in the script above, the output of the query is written to SearchLog_output.tsv file.

Here is what the output file looks like:

For any job there are four stages:

a. Preparing: the script is compiled, and execution plan is created

b. Queued: Resources are collected

c. Running: Actual execution happens.

d. Done: Any cleanup tasks are executed

More U-SQL concepts

Following script does data filtering on a file:

@searchlog =

EXTRACT UserId int

, Start DateTime

, Region string

, Query string

, Duration int

, Urls string

, ClickedUrls string

FROM "/Samples/Data/SearchLog.tsv"

USING Extractors.Tsv();

@filter = SELECT UserId, Region, Query, Duration, Urls

FROM @searchlog WHERE Duration > 500;

OUTPUT @filter

TO "/Samples/Output/LongDurationSearches.tsv"

USING Outputters.Tsv();

Note that the language is case sensitive, unlike SQL.

Catalog, Databases and Tables

Whenever we create a new ADLS account, we get a U-SQL Catalog, which has got one or more databases. It has one Master database by default and many user-created databases.

Data is stored in Tables. There is also concept of Views, Stored Procedures etc.

Below script creates a new database and table:

//Create Database SampleDBTutorials

CREATE DATABASE IF NOT EXISTS SampleDBTutorials;

//Create Table OlympicAthletes

CREATE TABLE IF NOT EXISTS SampleDBTutorials.dbo.SearchLog

(

//Define schema of table

UserId int,

Start DateTime,

Region string,

Query string,

Duration int,

Urls string,

ClickedUrls string,

INDEX idx1 //Name of index

CLUSTERED (Region ASC) //Column to cluster by

DISTRIBUTED BY HASH (Region) //Column to partition by

);

As we can see below, running a job with above script produces no data files.

Following script populates this newly created table using the data from SearchLog.tsv file:

//Read some data

@searchlog =

EXTRACT UserId int,

Start DateTime,

Region string,

Query string,

Duration int,

Urls string,

ClickedUrls string

FROM @"/Samples/Data/SearchLog.tsv"

USING Extractors.Tsv();

//Insert it into a previously created table

INSERT INTO SampleDBTutorials.dbo.SearchLog

SELECT *

FROM @searchlog

WHERE Duration > 500;

Below is the Job Graph, along with Heat Map on the basis of ‘Written’ data. Heat Map highlights the step in the graph which has maximum value for given parameter.

Note that output in this scenario is not a file, but a table.

But this table can’t be queried by just clicking on it. We need another job with a U-SQL query to view the data.

Following query queries the table and writes data to a file:

//Read from SearchLog table

@athletes =

SELECT *

FROM SampleDBTutorials.dbo.SearchLog;

//Write it to a file so we can look at it

OUTPUT @athletes

TO @"/Samples/Output/SearchLog_output.tsv"

USING Outputters.Tsv();

Here is the output file:

As we can see the analytics works very closely with the storage account. Great thing is that the data lake storage can store data files of any size and any number of files can be stored. Also, as we saw, structured (tables) and unstructured or semi-structured data can be stored in the ADLS and ADLA has capability to work on it.

Azure Data Lake Storage Gen 2

ADLS Gen 2 is not a separate service but is built on top of Blob Storage only.

Blob Storage stores data in blobs and doesn’t really have folders. ADLS stores data hierarchically in folders (which are first class citizens). ADLS Gen2 assimilates hierachical storage as well as other features of ADLS (made for analytical/Big Data services, large amount of any type of data etc.) with Blob Storage (massive availability, partner eco-system etc.). For any data/file stored in Gen 2, we can get two endpoints, one as an object store and other as a file store.

ADLA doesn’t work with Gen 2 yet. A list of Gen 1 vs Gen 2 capabilities.

AZ-900 Microsoft Azure Fundamentals : Study Notes

noreply@blogger.com (Waterfox) — Sun, 11 Oct 2020 12:40:00 +0000

List of resources to prepare for AZ 900 certification:

1. Tim Warner's video playlist

2. Andrew Brown's 3-hour video (broken into individual videos for every topic)

3. My collection of Microsoft documentation links

And below are my own study notes, created from all of the above. Happy preparing.

Availability Options

An Azure geography is a discrete market, typically containing at least one or more regions, that preserves data residency and compliance boundaries.

An Azure region is a set of datacenters, deployed close by and connected through a dedicated regional low-latency network. Each Azure region is paired with another region within the same geography (such as US, Europe, or Asia) at least 300 miles away, which together make a region pair

Examples of geographies and corresponding regions:

Geography	Regions (Location of datacenters)
India	Central India (Pune), South India (Chennai), West India (Mumbai)
Europe	North Europe (Ireland), West Europe (Netherlands)
UK	UK South (London), UK West (Cardiff)

Azure Availability Zones are unique physical locations (different datacenter buildings) within an Azure region. Each zone is made up of one or more datacenters equipped with independent power, cooling, and networking.

Service types:

Foundational – Available in all recommended and alternate regions when the region.
Mainstream – Available in all recommended regions, demand-driven in alternate regions.
Specialized – Targeted service offerings, often industry-focused or backed by customized/specialized hardware. Demand-driven availability across regions.

Availability sets

Availability sets are a way for you to ensure your application remains online if a high-impact maintenance event is required, or if a hardware failure occurs.

Availability sets are made up of Update domains (UD) and Fault domains (FD).

Update domains. When a maintenance event occurs (such as a performance update or critical security patch applied to the host), the update is sequenced through update domains.

Sequencing updates using update domains ensures that the entire datacenter isn't unavailable during platform updates and patching.

Update domains are a logical section of the datacenter, and they are implemented with software and logic.

It could be that when any update is started, it is like U0->U1->U2->….->Un. When one update domain is being updated then others are working normally.

Fault domains. Fault domains provide for the physical separation of your workload across different hardware in the datacenter.

Virtual machines in the same fault domain share a common power source and physical network switch.

In the event the hardware that supports a server rack becomes unavailable, only that rack of servers would be affected by the outage.

Maximum Fault Domains are 3, Maximum Update domains can be 20.

Azure Management Tools

Azure Portal

Azure CLI – Cross-platform command-line tool. Can be installed on MacOS using Homebrew
brew update && brew install azure-cli

Azure Powershell – Can be installed on Windows Powershell or Powershell Core (cross-platform version of Powershell, can be installed on Linux and MacOS)

Azure Cloud Shell

Azure REST API

Azure Policy

Azure Policy is a tool to create, manage, assign policies that control and enforce the properties of resources.

Imagine we allow anyone in our organization to create virtual machines (VMs) with not more than 4 CPUs.

Once the policy is implemented, Azure Policy will

1. stop anyone from creating a new VM outside the list of allowed stock-keeping units (SKUs).

2. If you try to update an existing VM, it will be checked against policy.

3. Finally, Azure Policy will audit all the existing VMs in our organization to ensure our policy is enforced. It can audit non-compliant resources, alter the resource properties, or stop the resource from being created.

Initiatives: An initiative definition is a set or group of policy definitions to help track your compliance state for a larger goal.

Azure RBAC

Azure role-based access control (Azure RBAC) helps you manage who has access to Azure resources, what they can do with those resources, and what areas they have access to.

Key points:

- Security Principal

- Role Definition

- Scope (Management Groups, Subscriptions, Resource Groups)

- Role Assignments/Deny Assignments

Azure Blueprints

Blueprints are a declarative way to orchestrate the deployment of various resource templates and other artifacts such as:

Role Assignments
Policy Assignments
Azure Resource Manager templates (ARM templates)
Resource Groups

Resource Locks/Management Locks

Delete: Users can read/update resource but can’t delete it.

Read Only: Read Only

Need Owner or User Access Admin to create or delete these locks

Locks are inherited, meaning if you set a lock at resource group, it is applied to all the resources of the group.

Azure Network Security Group (NSG)

Once inside a virtual network (VNet), it's crucial that you limit communication between resources to only what is required.

For communication between virtual machines, Network Security Groups (NSGs) are a critical piece to restrict unnecessary communication.

Network Security Groups allow you to filter network traffic to and from Azure resources in an Azure virtual network. An NSG can contain multiple inbound and outbound security rules that enable you to filter traffic to and from resources by source and destination IP address, port, and protocol.

They provide a list of allowed and denied communication to and from network interfaces and subnets, and are fully customizable.

CIA Security Principle

Confidentiality: Least privilege principle

Integrity: prevent unauthorized changes

Availability: services always available to authorized users

Azure Advisor vs Azure Monitor

Advisor suggests best practices to optimize costs, improve reliability, operational excellence, security and performance.

Monitor is a one stop shop for searching and viewing logs and analysing metrics.

Service Health

Dashboard that tells about health of services in regions across the world. Also mentions planned maintenance events and gives RCA for any issues.

Also health of individual resources can be checked.

Service Trust Portal vs Trust Center vs Compliance Manager

STP: Where compliance managers/officers go to check Azure’s audit reports.

Trust Center: Where one can read Microsoft’s privacy policies, Data collection policies etc.

Security Center vs Sentinel

Azure Security Center plays a vital role in “Collect” and “Detect” roles. While Azure Sentinel in addition to the first two roles also designed to perform “Investigate” and “Respond” roles.

ASC helps to setup azure and hybrid environment as securely as possible.

Sentinel has lot many connectors so it can get data from ASC and other 3^rd party security tools also. It is a tool for infosec professionals.

Azure DDoS protection, combined with application design best practices, provides defense against DDoS attacks.

Every property in Azure is protected by Azure's infrastructure DDoS (Basic) Protection. Azure DDoS protection Standard provides additional mitigation capabilities over the Basic service tier that are tuned specifically to Azure Virtual Network resources.

Network Resources

VNets can be connected to each other using vNet Peering or VPN Gateways.

VPN Gateways can be used to connect onprem resources to Azure network or two different azure networks. Only one VPN Gateway per vnet, so obviously one VPN Gateway can have multiple connections.

Load Balancer: Can be used to distribute traffic in, out and within a vnet. Public LB can be used to let the traffic coming from internet. Private LB manages traffic between resources in the vnet.

Application Gateway does traffic distribution for internet applications only.

Traffic Manager uses DNS closest to user.

Points to remember

- Tags are not inherited

- Permissions are inherited

Example of Using SimpleHttpOperator to make POST call

noreply@blogger.com (Waterfox) — Wed, 08 Apr 2020 05:20:00 +0000

Airflow has SimpleHttpOperator which can be used to invoke REST APIs. However using this operator is not exactly straightforward.

Airflow needs to be told about the connection parameters and all the other information that is needed to connect to external system. For this we need to create Connections. Open 'Connections' page through Admin->Connections link.

Expand the dropdown to see the various types of connection options available. For a REST call, create an HTTP connection. Give the host URL and any other details if required.

Now when we write our task using SimpleHttpOperator we will need to refer to the connection that was just created. The task below is making a post call to https://reqres.in/api/users API and passing it some data in JSON format.

myHttpTask = SimpleHttpOperator(
task_id='get_op',
method='POST',
http_conn_id='dcro',
data=json.dumps({
"name":"Morpheus",
"job":"Leader"
}),
endpoint='api/users',
headers={"Content-Type": "application/json"},
dag=dag)

Learn more about connections here: https://airflow.apache.org/docs/stable/howto/connection/index.html

How to set Trace Sampling rate for Jaeger in Istio?

noreply@blogger.com (Waterfox) — Thu, 02 Jan 2020 05:48:00 +0000

The default trace sampling rate for Jaeger in Istio is 1%, meaning, to get a single trace in Jaeger you need to send at least 100 requests to your application. This is good for a production environment but in dev or QA you would want a better sampling rate.

Unfortunately, Istio doesn't give any UI or dashboard to view/edit lots of settings and options which it uses. The documentation related to this is also hard to find.

If you are installing using istioctl then keep an eye for 'values.pilot.traceSampling' option. See more details about this and other options here.

If istio is already installed and running, we need to edit the 'istio-pilot' deployment and change the PILOT_TRACE_SAMPLING environment variable by running below command:

$ kubectl -n istio-system edit deploy istio-pilot

This opens vim (or whatever your default text editor) window with the deployment config file and lets you edit the value (which ranges from 0.0 to 100.0)

Just make the change and save the file and the changes take effect almost immediately.

How to deploy a Web App on Kubernetes cluster using Azure Kubernetes Service

noreply@blogger.com (Waterfox) — Sat, 18 May 2019 19:22:00 +0000

I deployed my webapp on Azure and found that the ease and simplicity of deploying and managing an Azure Webapp Service does not come cheap!

So next best option was to deploy it on Kubernetes using Azure Kubernetes Service. It involves a few more steps than webapp service, but then it is cheaper and if I want to move it to Google Container Engine, I don't have to break into sweat.

So here are the steps I followed:

Created a new Resource Group named firstResourceGroup.

az group create --name firstResourceGroup --location eastus

Created new ACR in this new resource group. Let's call it 'firstContainerRegistry'.

az acr create --resource-group firstResourceGroup --name firstContainerRegistry --sku Basic

On successful completion, the output is like this:

{

"adminUserEnabled": false,

"creationDate": "2019-04-24T05:03:32.564208+00:00",

"id": "/subscriptions/1900743b-c1ab-48cd-9951-eb03f5c2378d/resourceGroups/firstResourceGroup/providers/Microsoft.ContainerRegistry/registries/firstContainerRegistry",

"location": "southindia",

"loginServer": "firstcontainerregistry.azurecr.io",

"name": "firstContainerRegistry",

"networkRuleSet": null,

"provisioningState": "Succeeded",

"resourceGroup": "firstResourceGroup",

"sku": {

"name": "Basic",

"tier": "Basic"

"status": null,

"storageAccount": null,

"tags": {},

"type": "Microsoft.ContainerRegistry/registries"

}

Note the 'loginServer' from above output: firstcontainerregistry.azurecr.io. We will need it in next step.

Created docker image of the app by running this command:

docker build --tag waterfox83/personastore-flask-webapp:v1.0.0 .

(We have already pushed this image to docker hub using this command:

docker push waterfox83/personastore-flask-webapp:v1.0.0)

To use the personastore-flask-webapp container image with ACR, the image needs to be tagged with the login server address of our container registry. This tag is used for routing when pushing container images to an image registry.

docker tag

docker tag waterfox83/personastore-flask-webapp:v1.0.0 firstcontainerregistry.azurecr.io/personastore-flask-webapp:v1

Pushed this image to ACR (the ACR url helps understand the destination registry)

docker push firstcontainerregistry.azurecr.io/personastore-flask-webapp:v1

So till now, we have created a new Container Registry and put our image in the registry by tagging it with ACR URL and pushing it.

Now we need to actually create a Kubernetes cluster. This is not a one step process unfortunately and we need to take care of the access rights first.

To access images stored in ACR, we have to grant the AKS service principal the correct rights to pull images from ACR. Below command gives resourceID of the ACR:

az acr show --resource-group firstResourceGroup --name firstContainerRegistry --query "id" --output tsv

and it looks like this: /subscriptions/1900743b-c1ab-48cd-9951-eb03f5c2378d/resourceGroups/firstResourceGroup/providers/Microsoft.ContainerRegistry/registries/firstContainerRegistry

Now we create a service principal:

az ad sp create-for-rbac --skip-assignment

{

"appId": "3e6c413a-4893-437d-9535-ce1d288de5ff",

"displayName": "azure-cli-2019-04-24-05-43-54",

"name": "http://azure-cli-2019-04-24-05-43-54",

"password": "dbc48ad3-ee7c-4e82-b89e-c6d29be23643",

"tenant": "83faf872-71d4-4def-ba45-29da8b57c092"

}

To grant the correct access for the AKS cluster to pull images stored in ACR, assign the 'AcrPull' role to this service principal for this ACR (notice we are using the resourceId of the firstContainerRegistry we found above):

az role assignment create --assignee 3e6c413a-4893-437d-9535-ce1d288de5ff --scope /subscriptions/1900743b-c1ab-48cd-9951-eb03f5c2378d/resourceGroups/firstResourceGroup/providers/Microsoft.ContainerRegistry/registries/firstContainerRegistry --role acrpull

and output should be like this:

{

"canDelegate": null,

"id": "/subscriptions/1900743b-c1ab-48cd-9951-eb03f5c2378d/resourceGroups/firstResourceGroup/providers/Microsoft.ContainerRegistry/registries/firstContainerRegistry/providers/Microsoft.Authorization/roleAssignments/193c6b05-bf4e-4bf5-9180-0f0bec0432be",

"name": "193c6b05-bf4e-4bf5-9180-0f0bec0432be",

"principalId": "a87797f5-924e-418a-b668-b3ee5ab6e119",

"resourceGroup": "firstResourceGroup",

"roleDefinitionId": "/subscriptions/1900743b-c1ab-48cd-9951-eb03f5c2378d/providers/Microsoft.Authorization/roleDefinitions/7f951dda-4ed3-4680-a7ca-43fe172d538d",

"scope": "/subscriptions/1900743b-c1ab-48cd-9951-eb03f5c2378d/resourceGroups/firstResourceGroup/providers/Microsoft.ContainerRegistry/registries/firstContainerRegistry",

"type": "Microsoft.Authorization/roleAssignments"

}

Finally we create actual cluster:

az aks create \

--resource-group firstResourceGroup \

--name firstCluster \

--node-count 1 \

--service-principal \

--client-secret \

--generate-ssh-keys

(Replace appId and password with values received when creating service principal)

{

"aadProfile": null,

"addonProfiles": null,

"agentPoolProfiles": [

{

"count": 1,

"maxPods": 110,

"name": "nodepool1",

"osDiskSizeGb": 100,

"osType": "Linux",

"storageProfile": "ManagedDisks",

"vmSize": "Standard_DS2_v2",

"vnetSubnetId": null

}

"dnsPrefix": "firstCluste-firstResourceGroup-190074",

"enableRbac": true,

"fqdn": "firstcluste-firstResourceGroup-190074-2c0deb43.hcp.southindia.azmk8s.io",

"id": "/subscriptions/1900743b-c1ab-48cd-9951-eb03f5c2378d/resourcegroups/firstResourceGroup/providers/Microsoft.ContainerService/managedClusters/firstCluster",

"kubernetesVersion": "1.11.9",

"linuxProfile": {

"adminUsername": "azureuser",

"ssh": {

"publicKeys": [

{

"keyData": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXgw/bZ5spvfVtXGIXHYO8Bm751vQEym+mMyZ2WIco27nY6vSD3oYnZtTBXlaWz87M9IKMhhM/US53f2sdebvqzhkRGM/e8PfkujDh4vAe6h1EPkBrTjuQp/+NJsLoeai4reKBsuSdneV0LKQ2kluJBavCk1xz3NI+jQfqDEIAMZVIB8k+Zt6EmsdujMcg66H+MMKV5zovkeWKalOUPhBGT4bYEH8zs94/k2cPLJstOJfzswzI2VJrgkcG50beSwSWKwU9x1AXaMbcj6p+ZAi27359OF1/wUULhZeRdKxeLJu"

}

]

}

"location": "southindia",

"name": "firstCluster",

"networkProfile": {

"dnsServiceIp": "10.0.0.10",

"dockerBridgeCidr": "172.17.0.1/16",

"networkPlugin": "kubenet",

"networkPolicy": null,

"podCidr": "10.244.0.0/16",

"serviceCidr": "10.0.0.0/16"

"nodeResourceGroup": "MC_firstResourceGroup_firstCluster_southindia",

"provisioningState": "Succeeded",

"resourceGroup": "firstResourceGroup",

"servicePrincipalProfile": {

"clientId": "3e6c786a-4893-437d-9535-ce1d288de5ff",

"secret": null

"tags": null,

"type": "Microsoft.ContainerService/ManagedClusters"

}

Cluster has got created. Now we need to be able to connect to it using kubectl.

az aks get-credentials --resource-group firstResourceGroup --name firstCluster

Added kubernetes deployment yaml and created deployment.

kubectl apply -f kubernetes-deployment.yaml

And that's it. The app will be deployed and be available on the external IP!

Hadoop for beginners, by a beginner (Part 2) - Hadoop Architecture explained easy

noreply@blogger.com (Waterfox) — Fri, 22 Feb 2019 09:28:00 +0000

If you have gone through part 1 of this series, you have used Cloudera's Hadoop Quickstart VM to setup a working instance of Hadoop and have running instance of various Hadoop services. Now it is a good time to go back to a little theory and see how different pieces fit with each other.

HortonWorks, another Hadoop distributor, has got an excellent tutorial for Hadoop and each of its accompanying services, which you see in below image (taken from the above URL). You can go to this page and read about Hadoop in detail.

However I am going to summarize and simplify some of the content and definitions which make it easy for a beginner to quickly understand and proceed.

OK, so this is the definition of Hadoop on HortonWorks site: Apache Hadoop® is an open source framework for distributed storage and processing of large sets of data on commodity hardware.

You will agree that biggest challenges in any computing are very basic: storage and processing. Processing could be any operation that you do on the data. It could be merely counting the words in text files or finding more complex patterns in them. And since you can't do all the processing with your data kept in RAM all the time, you need to store it on HDD.

Hadoop uses 'commodity hardware' for both these purposes. The usual machines used as 'commodity hardware' have generally better configuration than your average laptop which has 8 GB RAM, 5-7 core processors and 500 GB HDD. But the non-commodity hardware which Hadoop clusters tend to compete with, are something like this: Oracle Exadata and Exalogic machines having TBs of RAM and hundreds of processors. These are extremely well engineered machines with software and hardware carefully designed to complement each other, and they cost a bomb!

Such machines are an example of what is called 'vertical scaling': increasing RAM, processing power and storage. Hadoop clusters which simply add more and more machines are examples of 'horizontal scaling' and since there is virtually no limit of how many machines you can add to a cluster, there is no limit to the processing power and storage capacity you can achieve!

(Theoretically even Exadata machines can be clustered to create a super powerful cluster but it will be so costly that it won't be economically viable. Just imagine having a cluster where each machine costs a million dollars!)

Let's go back to the architecture diagram shown above.

As is very clear Data Management and Data Access modules form the core of Hadoop and are responsible for storage and processing of the data. HDFS (Hadoop Distributed File System) and YARN (Yet Another Resource Negotiator) form the crux of it. HortonWorks site actually explains these technologies quite comprehensively so read it.

OK if you are reading this now you probably know how does HDFS work. Let's just see how it looks like. Go to Hue web UI. Click on 'File Browser'. What you see is the directory structure of the HDFS, which looks and behaves much like your local file system. You can create and delete directories and files, change directories and alter permissions for them, just like files on your local files.

You can do all of this on terminal using shell commands as well.

hadoop fs -mkdir newDirectory //create a new directory on HDFS

hadoop fs -put /home/someFile /user/cloudera/newDirectory //copy a file from your local file system to HDFS directory

hadoop fs -get /user/cloudera/newDirectory /home/someFile //download a file from HDFS directory to local file system.

There are lot many commands which you can check here.

How to delete all records from a Google Cloud Datastore Entity?

noreply@blogger.com (Waterfox) — Fri, 22 Feb 2019 09:27:00 +0000

Google has recently given a new feature known as 'Dataflow Templates' which have got a number of tasks that you can do without writing any code.

Click the link on top of Dataflow page:

You can select task from the list of available options:

Select 'Bulk Delete Entities in Cloud Datastore' from the list.

Fill in the required parameters. Instead of table name, it asks for GQL query. For selecting all the entities in a table give "select * from ".

IMPORTANT: In the optional parameters, you should specify the namespace/tenant for which you want to delete the entities. Not specifying namespace will delete ALL the records from the selected table across the tenants.

Once done, press 'Run Job'. Dataflow will launch and do the cleanup.

You can read documentation here. Source code of all templates is also on github.

How to get heapdump for any microservice running in Docker container in Google App Engine

noreply@blogger.com (Waterfox) — Fri, 22 Feb 2019 09:16:00 +0000

Getting heapdump for our services is very important for doing any memory profiling to debug any leaks or performance issues. Unfortunately it is not that straight forward. Fortunately, it is not impossible either!

We need to SSH in the container in which our service is running. Go to App Engine → Instances page and select your service from the 'services' dropdown.

SSH into any of the instances:

Ignore this and SSH into VM.

Once you SSH, you should be able to see cloud shell as below. If you are logging in for the first time, there may be some questions asking for SSH key. You can leave the key blank and continue.

This shell is the Linux VM on which our docker image is running. We need to find out more details about it before we can do anything with it.

There are a bunch of images running. We are interested in the first one ('us.gcr.io/.....') running in a container named 'gaeapp'. Now we need to go inside this docker container running our service. Below command executes '/bin/bash' command in the container and we get a bash shell inside container. (Call it VMception!)

Create a directory here to be able to store the dump in and give it all access.

mkdir dump_dir.

chmod 777 dump_dir

We will use it later.

Now in this VM we can check the processes running and the users running those. Give 'top' command on the shell and you see this:

As we can see 'jetty' user is running a java process. This is what we are interested in. We would now try to get heapdump of this process. For this we need to connect as the user 'jetty'. Currently we are logged in as user 'root' and we can't take the dump. Press Ctrl+C to exit this screen and get back to the shell.

su - jetty

/bin/bash

We get to the shell now as 'jetty'.

Goto the directory we created earlier.

cd jda/scripts/dump_dir

All set to take the dump! Here we need to give the PID of the java process we found above (50 in this example)

jmap -dump:live,file=dump_dir/ 50

If all is well, you should be able to get the hprof file created in this directory. Stage 1 is now over. Stage 2 is getting this file to your VM.

Write 'exit' on the bash shell to exit from the bash shell of jetty user.

Write 'exit' again on the prompt to exit from the jetty user.

Write 'exit' again on the bash shell to exit from the root user and from the docker.

Now you are on the VM where you logged first. Copy the file to this vm file system from the docker file system and give it all access.

docker cp gaeapp:/jda/scripts/dump_dir/ .

chmod 777 filename.prof

Stage 2 is now over. Stage 3 is now getting this file to your local machine. (I know, this seems like a never ending process but please stay with me!)

On a shell on your local machine give following commands. Second command is optional if you have already a default project configured.

gcloud auth login

gcloud init

gcloud app instances scp --service  --version  :/<filename>

For example, in my case above command looks like this:

gcloud app instances scp --service metadata-service --version v171 aef-metadata--service-v171-qhl0:/home/abhishek_asthana/dump_hprof.bin dump.hprof

And with this Stage 3 is also over. You have finally got the hprof file on your machine and can now use any memory profiler for analysis.

Running Apache Beam pipeline using Spark Runner on a local standalone Spark Cluster

noreply@blogger.com (Waterfox) — Sun, 01 Jul 2018 18:00:00 +0000

The best thing about Apache Beam (Batch + Stream) is that multiple runners can be plugged in and same pipeline can be run using Spark, Flink or Google Cloud Dataflow.

If you are a beginner like me and want to run a simple pipeline using Spark Runner then whole setup may be tad daunting.

Start with Beam's WordCount examples which help you quickstart with running pipelines using different types of runners. There are code snippets for running the same pipeline using different types of runners but here the code is running on your local system using Spark libraries which is good for testing and debugging pipeline. If you want to run the pipeline on a Spark cluster you need to do a little more work!

Let's start by setting up a simple standalone single-node cluster on our local machine. Extending the cluster is as easy as running a command on another machine, which you want to add to cluster.

Start with the obvious: install spark on your machine! (Remember to have Java and Scala downloaded and installed beforehand.)

To setup master and slave nodes, follow these steps:

1. Go to “SPARK_HOME/libexec/conf/” directory.
(SPARK_HOME is the complete path to root directory of Apache Spark in your machine.)

2. Edit the spark-env.sh.template file and save it as spark-env.sh.
Add following line: SPARK_MASTER_HOST=''

3. Goto SPARK_HOME/libexec/sbin and execute the following command.
$ ./start-master.sh

4. The output of above command will be a log file. Go to the log file and check the port at which spark master webUI is running. For me it is 8080.

Don't confuse this with 7077. That is the port at which master will listen for slave nodes.

5. Now we need to setup slave node for above master. On the machine where you want to setup the slave repeat the steps #1 and #2 above. If you are setting the slave on same machine, then no need to do these.

6. Goto SPARK_HOME/libexec/sbin and execute the following command.
$ ./start-slave.sh spark://:7077

7. Once again verify the logs. Now if you go to the master's webUI, you will see your slave listed under 'Workers'.

Our cluster is setup and we need to now run the same Beam WordCount examples on this cluster. We need to create a jar (which will be submitted to spark cluster) which bundles our code with required spark libraries. Without these libraries Beam will not accept spark-runner as an eligible runner for the pipeline and will throw an error for this when you run the pipeline.

mvn package -Pspark-runner

Now submit the job with same parameters as earlier.

spark-submit --class org.apache.beam.examples.WordCount --master spark://:7077 target/word-count-beam-bundled-0.1.jar --runner=SparkRunner --inputFile=pom.xml --output=counts

While the code is running you can now refresh the master's webUI and you will see an entry in running applications!

This should help you start the learning. Any questions or comments, please leave in comments section.

Running Jenkins inside Docker

noreply@blogger.com (Waterfox) — Sat, 30 Dec 2017 16:14:00 +0000

Whether you are a beginner to Jenkins and want to have your setup done quickly or you want to harness the full power of containers by running Jenkins in Docker, this simple step-by-step tutorial is for you.

Running Jenkins in Docker

We will pull Jenkins Docker image from Docker repository:

docker pull jenkins

Using below command we will map ‘jenkins’ directory in my working directory ($PWD/jenkins) to /var/jenkins_home directory in container and map port 49001 on host to 8080 on container.

docker run -d -p 49001:8080 -v $PWD/jenkins:/var/jenkins_home:z -t jenkins

Now to run jenkins, just give http://localhost:49001/. You can proceed with the setup and management of Jenkins now.

Your entire Jenkins setup has been created in the 'jenkins' directory which you mapped earlier. Go ahead and take a look!

Packaging app in Docker and pushing to repository

Sometimes you would want to package your app as a Docker image and publish to the Docker Hub (or any other container registry of your choice) as the final step of your build process. If so, follow this step.

We will use CloudBees Docker Build and Publish Plugin. Go ahead and install it using the Manage Jenkins -> Manage Plugins -> Available tab in your Jenkins UI.

To publish to Docker, our jenkins host machine needs to have docker client installed. For this we need to downloaded a different image which is based on Jenkins and also has docker client.

You can stop the docker container which you started in previous step.

To create this new image we will use following dockerfile:

FROM jenkins

USER root

RUN mkdir -p /tmp/download && \

 curl -L https://get.docker.com/builds/Linux/x86_64/docker-1.13.1.tgz | tar -xz -C /tmp/download && \

 rm -rf /tmp/download/docker/dockerd && \

 mv /tmp/download/docker/docker* /usr/local/bin/ && \

 rm -rf /tmp/download && \

 groupadd -g 999 docker && \

 usermod -aG staff,docker jenkins

user jenkins

We create an image named ‘jenkins-docker’ using the above dockerfile with following command. You can change the image name to whatever you want. The '.' implies that the dockerfile is present in this directory. If your dockerfile is in some other directory please give that path.

docker build -t jenkins-docker .

We now run a container with this image:

sudo docker run -d -p 49001:8080 -v $PWD/jenkins:/var/jenkins_home:z -v /var/run/docker.sock:/var/run/docker.sock -t jenkins-docker

In this command we are mapping the docker socket for docker on our machine with the docker socket in the image.

Once this is done, we add ‘Docker Build and Publish’ step in the build steps. We give the repository name (of Docker Hub) to which we want to publish, and its credentials. Now when the build runs, it packages the app to a Docker image and publishes to Docker Hub.

Jenkins Job DSL

DSL: Domain Specific Language

Writing a Jenkins job in Groovy instead of doing it from UI. Writing a simple job of getting a Node JS app from github and doing ‘npm install’ using DSL:

job(‘NodeJS App’) {

	scm {

		git(‘git://github.com/') { node ->

			node / gitConfigName(‘UserName’)

			node / gitConfigEmail(‘usermail’)

}

}

	triggers {

		scm(‘H/5 * * * *’) //How many times to run? Every 5 mins

}

	wrappers {

		nodejs(‘nodejs’) //name of NodeJS install in Jenkins

}

	steps {

		shell(“npm install”)

}

}

To setup job DSL, we create a new Freestyle project and add a new build step ‘Process Job DSLs’. There either we can copy the script or if we have specified any repo in Source Code Management, give the path of the Job DSL groovy script.

If we run the build now, it will fail with error: ‘script not yet approved for use’. To approve, go to ‘In-process Script Approval’ and approve the script. Once this build succeeds, we will see a new Node JS project created (as we specified in the DSL script)

Jenkins Pipeline DSL

For creating a pipeline which pulls code from SCM, runs tests and pushes to Docker, we have forked a project ‘docker-demo’ and altered its Jenkinsfile:

node {

   def commit_id

   stage('Preparation') {

     checkout scm

     sh "git rev-parse --short HEAD > .git/commit-id"                        

     commit_id = readFile('.git/commit-id').trim()

}

   stage('test') {

     nodejs(nodeJSInstallationName: 'nodejs') {

       sh 'npm install --only=dev'

       sh 'npm test'

}

}

   stage('docker build/push') {

     docker.withRegistry('https://index.docker.io/v1/', 'dockerhub') {

       def app = docker.build("waterfox83/docker-nodejs-demo:${commit_id}", '.').push()

}

}

}

Creating a new ‘Pipeline’ type project with this Jenkinsfile, creates a new project with three stages and pushes the build to docker.

Running Build/Test in Docker Containers

If you want that build/test stages of your build pipeline are run in their own containers, you can do that as well. In this example, there is a stage named 'test with a DB' which requires mySql to be running. We will change our dockerfile so that a new container with mySql image is created and test is run within that container.

It is possible to run everything in containers as part of build pipeline. In below jenkinsfile, we have modified the file from previous example and are running everything in containers.

   stage('test') {

     def myTestContainer = docker.image('node:4.6')

     myTestContainer.pull()

     myTestContainer.inside {

       sh 'npm install --only=dev'

       sh 'npm test'

}

}

   stage('test with a DB') {

     def mysql = docker.image('mysql').run("-e MYSQL_ALLOW_EMPTY_PASSWORD=yes") 

     def myTestContainer = docker.image('node:4.6')

     myTestContainer.pull()

     myTestContainer.inside("--link ${mysql.id}:mysql") { // using linking, mysql will be available at host: mysql, port: 3306

          sh 'npm install --only=dev' 

          sh 'npm test'                     

}

     mysql.stop()

}

I hope this post has helped you in starting with running Jenkins on Docker. If you have any questions/comments please post them in comments. Happy learning!

Introduction to Mutation Testing

noreply@blogger.com (Waterfox) — Thu, 10 Aug 2017 10:37:00 +0000

You have written unit tests, integration tests, acceptance tests and all sorts of tests and code coverage is 99% but still there are bugs! How is that possible? It is possible if your testcases are incorrect and quality of test code is low. With Mutation testing, find out how good are your tests. Know more about this in this presentation!

How to query data using GQL if your Kind name has a dot in it?

noreply@blogger.com (Waterfox) — Mon, 22 May 2017 06:52:00 +0000

There is a big chance that your kind (table) name is of the format . because there are various tables in your module.

e.g. I had StoreService.Store and StoreService.Employee as two of the kinds in my service named StoreService.

In Google Cloud Console, if you write a GQL to query to fetch/filter the entities you will get an error:

SELECT * from StoreService.Store
GQL query error:

Encountered "." at line 1, column 27. Was expecting one of: "group", "limit", "offset", "order", "where"

Enclosing the Kind name in single quotes doesn't help either:

SELECT * from 'StoreService.Store'
GQL query error:

Encountered "'StoreService.Store'" at line 1, column 15. Was expecting one of: ,

To do it right, you need to enclose the name in backquotes (that key below ~ in the keyboard)!!!
SELECT * from `StoreService.Store`

This information is hidden in Datastore documentation here. Try finding it!

How to transfer data from Cloud Datastore to Big Query in Google Cloud Platform

noreply@blogger.com (Waterfox) — Thu, 23 Mar 2017 11:36:00 +0000

If you are here I am assuming that you are looking to migrate the data from Cloud Datastore to Big Query because you want to do some analysis and are frustrated by limitations imposed by GQL (Google Query Language).

First of all you need to create a backup of the data in datastore. Use the Datastore Admin tool provided by Google to take a backup and store it automatically in the Cloud Storage bucket.

Select all the entities and press 'Backup Entities'. Give a backup name, select Google Cloud Storage as backup storage destination and specify a bucket name.

Once the backup job is completed, you will see the backup listed. You can select a backup and press 'Info' and see the details (Entities are masked in the screenshot below).

Go to the bucket mentioned in 'Handle' and you will see the file mentioned above. You will also see many more files with similar names, ending with .backup_info (e.g. ahRzfmpkYS1wZC1zbG8tc2FuZGJveHJBCxIcX0FFX0RhdGFzdG9yZUFkbWluX09wZXJhdGlvbhix_-4DDAsSFl9BRV9CYWNrdXBfSW5mb3JtYXRpb24YAQw.JobDetailsEntity.backup_info)

This is the backup file for a specific entity which you will need to specify when creating a table in Big Query.

Head over to Big Query and create a new dataset.

In Location field select 'Google Cloud Storage' and give the location of backup file for the specific entity. File format is 'Cloud Datastore Backup'.

Like the one we found earlier: gs://jda_so__78700310-e2f9-4cf2-8f20-dd325de09a4d_data_bkup/ahRzfmpkYS1wZC1zbG8tc2FuZGJveHJBCxIcX0FFX0RhdGFzdG9yZUFkbWluX09wZXJhdGlvbhix_-4DDAsSFl9BRV9CYWNrdXBfSW5mb3JtYXRpb24YAQw.JobDetailsEntity.backup_info.

Here the bucket name: jda_so__78700310-e2f9-4cf2-8f20-dd325de09a4d_data_bkup is coming from the 'Handle' field in the backup information in the datastore admin and the file name you got in the previous step!

Specify the name of the table you want to create in Big Query in 'Destination' field. Press 'Create Table' and if everything is correct, the job will complete successfully. Select a table from the left panel and click on Preview to see the data populated. And you are done!

Let me know in comments if you have any questions.

How to upload to Google Cloud Storage buckets using CURL

noreply@blogger.com (Waterfox) — Thu, 23 Mar 2017 10:25:00 +0000

Signed URLs are pretty nifty feature given by Google Cloud Platform to let anyone access your cloud storage (bucket or any file in the bucket) without need to sign in.

Official documentation gives step by step details as to how to read/write to the bucket using gsutil or through a program. This article will tell you how to upload a file to the bucket using curl so that any client which doesn't have cloud SDK installed can do this using a simple script.

This command creates a signed PUT URL for your bucket.

 gsutil signurl -c 'text/plain' -m PUT serviceAccount.json gs://test_bucket_location

Here is my URL:

 https://storage.googleapis.com/test_sl?GoogleAccessId=my-project-id@appspot.gserviceaccount.com&Expires=1490266627&Signature=UfKBNHWtjLKSBEcUQUKDeQtSQV6YCleE9hGG%2BCxVEjDOmkDxwkC%2BPtEg63pjDBHyKhVOnhspP1%2FAVSr%2B%2Fty8Ps7MSQ0lM2YHkbPeqjTiUcAfsbdcuXUMbe3p8FysRUFMe2dSikehBJWtbYtjb%2BNCw3L09c7fLFyAoJafIcnoIz7iJGP%2Br6gAUkSnZXgbVjr6wjN%2FIaudXIqAxyz4O6VU%2FIob8RHmfYBD3rYw8oRmZer0g%2BMTDw5Bu3%2BdjbxCriCOkOLWkvlpBGvd71hRP89lxpD%2FwX1N1pzAiRBpmKAhpKrYrq8h4kNLAtTHXYSz5j%2Bzb28UD4YMG52n8UpfyPgFIg%3D%3D

Now this command will upload a text file to the bucket using the signed URL we created:

 $ curl -X PUT -H 'content-type: text/plain' -d @test_upload.txt https://storage.googleapis.com/test_sl?GoogleAccessId=my-project-id@appspot.gserviceaccount.com&Expires=1490266627&Signature=UfKBNHWtjLKSBEcUQUKDeQtSQV6YCleE9hGG%2BCxVEjDOmkDxwkC%2BPtEg63pjDBHyKhVOnhspP1%2FAVSr%2B%2Fty8Ps7MSQ0lM2YHkbPeqjTiUcAfsbdcuXUMbe3p8FysRUFMe2dSikehBJWtbYtjb%2BNCw3L09c7fLFyAoJafIcnoIz7iJGP%2Br6gAUkSnZXgbVjr6wjN%2FIaudXIqAxyz4O6VU%2FIob8RHmfYBD3rYw8oRmZer0g%2BMTDw5Bu3%2BdjbxCriCOkOLWkvlpBGvd71hRP89lxpD%2FwX1N1pzAiRBpmKAhpKrYrq8h4kNLAtTHXYSz5j%2Bzb28UD4YMG52n8UpfyPgFIg%3D%3D

On similar lines, you can create a RESUMABLE URL and then upload file using following command:

 $ curl -v -X 'POST' -H 'content-type: text/plain' -H 'x-goog-resumable:start' '[signed url]' -d ''

This command gives 'session URI' which is similar to signed url, but longer!

 $ curl -v -X PUT -d @test_upload.txt -H 'content-type: text/plain' '[session URI]'

Simple isn't it!

Diving Deeper in Angular JS 1.5 : Services and Routing in depth

noreply@blogger.com (Waterfox) — Tue, 27 Sep 2016 18:17:00 +0000

How to update git remote origin URL

noreply@blogger.com (Waterfox) — Sat, 24 Sep 2016 03:26:00 +0000

OK, whatever way you are passionate about when it comes to updating your remote origin URLs, I have got you covered!

Option 1: Go to config file in /.git folder.

change url field from

url = ssh://git@<old-url> to url = ssh://git@<new-url>

Option 2: Go to your repo location, on cmd give git remote set-url origin ssh://git@<new-url>.

Option 3: Go to your repository in source tree, click on settings, on Remotes tab in repository settings box, you can select the origin and edit.

What is life without options!!

Angular JS basics for beginners (with code!)

noreply@blogger.com (Waterfox) — Thu, 16 Jun 2016 12:21:00 +0000