mchampaneri.in

Agent Orchestration with VS Code and GitHub Copilot (Local Agents Only)

noreply@blogger.com (Manish Champaneri) — Sat, 4 Apr 2026 22:55:00 +0530

Agent Orchestration with VS Code and GitHub Copilot (Local Agents Only)

Agent orchestration is becoming a key pattern in AI-assisted development. Instead of relying on a single assistant, developers can coordinate multiple specialized agents that collaborate to complete complex workflows.

This article focuses strictly on local agents within GitHub Copilot in Visual Studio Code. It does not cover background agents or cloud agents.

Scope of this article:

Local agent capabilities in VS Code
Agent handoff for orchestration
Use of skills
Reference implementation

Focus on Local Agents

Local agents operate entirely within the VS Code environment.

Characteristics

Execute inside the editor session
Provide fast, interactive responses
Work with local files and context
No dependency on external execution pipelines

Why Local Agents

Immediate feedback loop
Better developer control
Ideal for iterative workflows
Lower complexity compared to distributed systems

Key idea:

Local agents are sufficient to build effective multi-agent orchestration without involving external systems

😐 Prefer executable knowledge over written explanations?

You might want to begin here:

https://github.com/groupzer0/vs-code-agents

[ This is not my repo, Nor I'm contributer, I'm one of who gave a star 🌟 for good agents. 😁 ]

VS Code GitHub Copilot Local Agent Model

Within the scope of local agents, Copilot acts as a multi-role system rather than a single assistant.

Role-based Local Agents

You can structure agents based on responsibilities:

Planning Agent
- Breaks down tasks
- Defines approach and steps
Implementation Agent
- Writes and modifies code
- Executes instructions from planning
Review Agent
- Validates code
- Suggests improvements
Debug Agent
- Identifies issues
- Proposes fixes

Key Concept

Multiple logical agents can operate within the same local environment
Separation is defined by prompts, roles, and instructions rather than infrastructure

Agent Handoff for Local Orchestration

Agent handoff enables coordination between local agents.

Definition

Transfer context and responsibility from one agent to another within the same environment

How It Works (Local Context)

One agent completes a step
The next agent continues using the same context
No external execution required

Example Workflow

Planning Agent
- Defines feature structure
Handoff to Implementation Agent
- Generates code
Handoff to Review Agent
- Validates output
Handoff to Debug Agent (if needed)
- Fixes issues

Benefits

Clear separation of concerns
Improved output quality
Easier iteration and refinement

Use of Skills with Local Agents

Skills enhance local agents by providing reusable capabilities.

Definition

Predefined instruction sets or modules
Automatically used when relevant

Role in Local Orchestration

Reduce repetitive prompting
Standardize execution
Improve consistency across agents

Examples

Generate API endpoints
Create test cases
Refactor code
Enforce coding standards

Key Insight

Skills act as a reusable capability layer within local agent workflows

Orchestration Patterns Using Local Agents

Sequential Flow

Planning → Implementation → Review → Debug

Iterative Loop

Implementation → Review → Fix → Review

Parallel Thinking (Conceptual)

Multiple approaches generated, then evaluated

Skill-driven Execution

Agents invoke predefined skills instead of relying only on prompts

Reference Repository

A practical implementation can be found here:

https://github.com/groupzer0/vs-code-agents

What to Explore

Agent role definitions
Structured workflows
Reusable configurations

Key Takeaways

Local agents can be organized effectively using simple configuration patterns
Complex orchestration does not require distributed systems

How to Build a Local Agent System

Step 1: Define Roles

Assign clear responsibilities to each agent
Keep roles focused and minimal

Step 2: Structure Prompts

Use consistent instructions
Maintain clarity between agents

Step 3: Design Handoffs

Define when and how control shifts between agents

Step 4: Add Skills

Identify repeatable tasks
Convert them into reusable modules

Step 5: Iterate

Continuously refine roles, prompts, and flows

Conclusion

Local agents in VS Code with GitHub Copilot provide a powerful foundation for agent orchestration.

Key points:

Entire workflow runs within the editor
No dependency on background or cloud agents
Handoff enables coordination between roles
Skills improve efficiency and consistency

This approach allows developers to build structured, multi-agent systems while keeping execution simple and fully local.

Three ways to use user defined vscode snippets

noreply@blogger.com (Manish Champaneri) — Mon, 30 May 2022 21:35:00 +0530

What are code snippets?

Code snippets are templates that make entering repeated code patterns, such as loops or conditional statements, easier.

Types of snippets

Built-in snippets

Vscode has built-in snippets for number of languages like PHP, Javascript, Typescript etc.

Snippets available on market place

There custom snippets averrable on marketplace for specific frameworks / languages and platforms as well. For example snippets for VUE, Laravel etc.

User defined snippets

You can defined your own snippets, It's called user defined snippets. You can create snippets according to your need using the snippet templates.

Ways you can use user defined snippets

You can use snippets in several way. Like generating fixed static code or generating code with placeholders according to your need.

Snippet for fixed static code

You can create a snippet to insert the fixed static code, which is fixed but you need to add that in several places. You can use snippet in this case.

For example, You need to add copyright in every page footer.

<p> Copyright @ 2020 </p>

You can create a snippet for it like,

"Add copyright text": {
  "prefix": "copyright",
  "body": [
    "<p> Copyright @ 2020 </p>"
  ],
  "description": "Add copyright text"
}

Snippet as template with placeholders

You can create a snippet for the repetitive code block have changes in some fixed attributes.

For example, Form Input of of bootstrap. In this case you just need to change the Label and type of text.

<div class="mb-3"> 
    <label for="exampleFormControlInput1" 
          class="form-label"
    >
          Email address
    </label> 
    <input type="email" 
     class="form-control" 
     id="exampleFormControlInput1" 
     placeholder="name@example.com"
    > 
</div>
In the above code block, Things that will be changed per block is,
Label value
Label for value
Input type
Id of input
and Placeholder value

We can create template for form-input like this,

"Form input block": {
  "prefix": "form-input",
  "body": [
    "<div class=\"mb-3\"> ",
    "    <label for=\"$2\" ",
    "          class=\"form-label\"",
    "    >",
    "          $1",
    "    </label> ",
    "    <input type=\"$3\" ",
    "     class=\"form-control\" ",
    "     id=\"$2\" ",
    "     placeholder=\"$4\"",
    "    > ",
    "</div>$0"
  ],
  "description": "Form input block"
}

Here, $1, $2, $3, $4 and $0 are tab-indexed placeholders. Number after $ shows the tab index. Like $1 means first tab. $2 means second tab and so on. $0 is alway last.

Note, I'm using $2 for label for and id of the input both, as both will be sharing the same value. So It will write on the both places using multi cursor. Cool, Right!

Snippet as Modifiers / Generator

Suppose you have a javascript code where you are following code convention like this,

For any [variable Name] error checking code is invalid[variable Name].

E.g For Name, validation function is invalidVariable()

var Name=""
invalidName() {
return _.get(this.errors, "Name[0]");
}

In this case, We can create a snippet we can generate invalid[Variable Name] function for us. This kind of snippet can be created using a special built-in variable provided by vscode. ${TM_SELECTED_TEXT}

Snippet will look like this,

"validation error check function": {
  "prefix": "invalid-feedback-function",
  "body": [
    "invalid${TM_SELECTED_TEXT}() {",
    "      return _.get(this.errors, \"${TM_SELECTED_TEXT}[0]\");",
    "    },"
  ],
  "description": "validation error check function"
}

${TM_SELECTED_TEXT} takes the code currently selected as input.

You can create more complex and useful snippets according to your need. Check the official link to the vscode user-defined snippets to find out what other built-in variables are available to create snippets.

Here is the link to official documentation for vscode user defined snippes:
https://code.visualstudio.com/docs/editor/userdefinedsnippets

Use google cloud log-sink to log incidents to your project management system

noreply@blogger.com (Manish Champaneri) — Sun, 23 May 2021 19:55:00 +0530

Small team working on multi service product, With carrying more then one responsibilities. It's a big challenge to keep everything on track.

Most importantly, When it comes to incident reporting. We can not automate every thing but, at least we can automate incident reporting to our project management service eg Gitlab / GitHub / Slack.

( Slack 😅 , for small team like us slack also work like project management tool. )

Google cloud log-sink

Here comes google cloud log-sink handy to you.

We can direct all of your logs with severity level of

Error
Crtitical
Alert
Emergency

to a logs ink which log a incident on your project & issue tracking service ( For example Gitlab ) using cloud pub/sub topics. [ You can use firebase function for such pub/sub topics ]

Inclusion filter for error logs severity='(ERROR OR CRITICAL OR ALERT OR EMERGENCY)'

Next time when ever error log will be generated on your cloud logs. You will get an incident created on your gitlab.

So Don't wait for your tester or your user to report the issue, get it right away to your project management service with adequate information in it using log sink.

Transpile your mithril jsx using babel's online transpiler

noreply@blogger.com (Manish Champaneri) — Sat, 7 Nov 2020 08:43:00 +0530

You can transpile your JSX based mithril component to mithril component online with help of babeljs.

Goto https://babeljs.io/repl

Copy your JSX component over there.

Select React and ES2015 in preset for compilation.

You will be getting compiled code on the right side.

If you notice this compiled code is for React, and not for the mithril.
You have to go one step more, Replace all the React.createElement with m and you got the code for your mithril.

Cheers!

Gitlab CI/CD for firebase project

noreply@blogger.com (Manish Champaneri) — Sun, 1 Nov 2020 18:58:00 +0530

CI/CD stands for continues integration, continuous deployment.

Normally, I am hosting my private projects on Gitlab. One of my project is implemented on google's firebase platform. It's using firebase functions, hosting, pub-sub and other google service heavily.

Why I needed CI/CD ? 🤔

Initially, there was hosting and very few firebase functions to handle. It was giving me option to deploy that particle service.

Like, I want to deploy only hosting, I will say
$ firebase deploy --only hosting

Or, If I want to deploy only function
$ firebase deploy --only functions

Or more precisely, I want to deploy only a particular function.
$ firebase deploy --only functions:[function-name]

But, as the project started to grow, this approach started showing issues in integration.

Local version were working fine while deployment were showing issues. Why ? . Because, the approach of deploying only services that has been touched was depending on the list of deployment. If that list misses something, depending services will generate false results and in worst case, It might brake other dependent services as well.

Though due to workflow of multiple tests before moving on, we were able to rectify the missing service, it was stretching the process of deployments to several hours before we can confirm success of the deployment.

CI/CD as Solution 👊

As a solution for the problem of integration, we moved to GitLab CI/CD. Meaning, now we were making full deployment of all services which were used while development phase.

Our full deployment is taking 9-15 minutes, which is higher then specific deployment ( 3-5 minutes ), But less then time was spent to confirmation the validation of deployment (15 minutes to several hours if test results were false positive)

It saved us from the loop of
test -> rectify missing service -> deploy service -> test

It also saved me from making manual deployments, we are using merge event on specific branch to trigger CI/CD. ✌

Google blogger Ideas panel

noreply@blogger.com (Manish Champaneri) — Mon, 12 Oct 2020 12:00:00 +0530

Google blogger Ideas

I opened by blogger today, and.. I got this.

Google blogger Ideas

A panel suggesting a topic on which I can write my next blog. It's fetching unanswered question from web according to your previous post and topics.

It was something, I was really looking for, after all it takes time to finding subject on which to write next and still being in the same niche.

Awesome feature Blogger!

Immutable.js with firebase

noreply@blogger.com (Manish Champaneri) — Sat, 10 Oct 2020 11:18:00 +0530

To store data we can use either object or we can use array in javascript. Both have their own advantage and disadvantage.

Array : Array has advantage of maintaining order, but negative side is that If you want to update any element inside array you have to find the index of the element. May be via looping through the whole array if indexes are not predictable.

Object : You can store element as object property. It's like key-value pair, So If you want to update any property (element) of object, you can update it easily using key. So, It's easy to update the elements in object, but object uses hash for property, So order is not guaranteed.

Immutable.js with firebase

Firebase provides live updates. Suppose you are having a program where you are listening for live updates on some last x elements, while retaining others for display only, but order of display matters.

You are listening for last x elements live, So you have to update the data of those elements when you receive update from firestore, but also have to maintain order of all the elements.

Here comes immutable.js, to get advantage of both Array and Object. Using OrderedMap from immutable.js which has ability to update with key and which guarantees the order as well, solves the problem.

[Source code]

In orderedMap element stored in order of they are set, But for existing key it updates the existing element with that key. So, you get your update with key problem solved here.

Now next issue is display in order.

You can convert OrderdMap to an array using toArray method on it. So you will alway get orderd output.

[Source code]

Real world uses case is pagination. How ??? Try to implement a pagination with infinite scroll which fetches old message when scrolled up while also listens live to the last X messages while on the bottom.

Ahmedabad Shopping festival

noreply@blogger.com (Manish Champaneri) — Sun, 9 Aug 2020 22:24:00 +0530

Ahmedabad Shopping festival is solution created by Tuzzin Infotech Pvt. Ltd.
I worked as part of the development team

My journey started as a part-time web developer. A MySQL database, Php for backend, VueJs and jQuery frontend, scss for css and I'm done. So easy!

I was never thinking about scaling, rate limiting, timeout on those days, and normal site even does not need to care about such things unless they reaches that threshold.

Luckily, I got chance to work on a product which was requiring all of this. It was the first time when I got exposure to real scalable application with spike in traffic, Ahmedabad Shopping festival 2019.

Brief Intro of Ahmedabad Shopping festival

Ahmedabad Shopping festival was first digitally managed shopping festival of India hosted in Ahmedabad, Gujarat. Every customer were getting rewards and coupons for chance to win prize on making shopping during shopping hours.

Technical requirements

Ahmedabad Shopping festival was consisting a web portal for retailers, a web portal for administrators of shopping festival and mobile application for shoppers.

All of this modules were needing access to database, sms module and push notification module and requests to each service could scale from 0 to massive amount at any time.

It was also required to be able to make deployment during shopping hours without having any downtime or any glitch in any of the module.

Issues and Resolutions

Ability to make deployments during shopping hours.

Our first and for most issue was ability to make a deployment during shopping hour without introducing any glitch. Glitch was not acceptable at all as It could affect result of lucky winner selection based on coupons issued.

This issue was solved using versioning with app engine services and traffic migration in conjunction with warmup request.

Rate Limiting

We were using sms and push notifications with all the modules. Sms service was capped by a rate limit per time period (X / sec ). Reaching that limit would have resulted in delivery failure of sms.

Sms were used for notifications to retailers and admins but, most importantly sms were also used for OTP for signup/sign-in in shopper facing app. So neither late not failure of sms delivery was acceptable at all.

Google cloud task was used as solution. All of the sms from different backend services were written to that queue which was optimised in way that it was sending messages at highest possible rate without exceeding limit.

Taking daily backup, loading data and processing to find lucky winner

This tasks were supposed to run daily on specified time. It was handled using cron jobs.

Giving as fast response as possible to every request

Shoppers were in count of thousands. During shopping hour request could making spikes that it reach the read limit per second of database. Therefor reading from the database was not a feasible one.

As solution memcache was used. For every incoming request we were first looking at memcache, if memcache does not have data then data was loaded to memcache and same time it was returned as response as well.

Here we found that data being sent to shopper were not going to be changed by any event or action of shopper other then his/her coupon counts. Most of the data was going to be static for the day,

There was only one request the first one of the day has to go to read the database, but we know that reading memcache is faster then reading database. So to make first request to read memcache as well, we started refreshing and reloading memcache before the first request of the day hits api.

So, technically were serving thorough only memcache through out the day without going to database for reading more then once in a day in normal scenario.

Data sent to shoppers were static and going to be unchanged for day long. As result we were caching data to the devices local storage for day that was reducing request to the api and loading data faster at same time.

So there were two caches used,

on cloud in form of memcache
second local storage at device level which significantly improved response time.

Conclusion

Architecture was designed to generate faster response with minimum usage of resource and without exceeding constrains imposed by third party service providers and cloud services (database & big-query ) itself.

Ahmedabad Shopping festival was the project which made me think as a person for whom things working is not enough, but the way it works is also matters.

Creating 2d games with golang and gontuz/prototype

noreply@blogger.com (Manish Champaneri) — Sun, 7 Jun 2020 16:43:00 +0530

Golang was created as server language. But with the time and curiosity of people, golang has covered GUI application and game development as well.

I am sharing here one of the available golang library, on top of which you can create 2d games. Its gonutz/prototype.

Sample video

Gonutz/prototype is very small and easy to learn library. It's cross-platform, that means the game your are creating in one platform will be able to run on other platforms without any change in code ( in normal case ).

This package currently supports windows , mac and linux.

gonutz/prototype features

Support for keyboard input
Support for mouse input
Support for 2d drawing
Support for image and sprite-sheets
Support for sound

Want to learn more about this package.

Follow my youtube video playlist "2d game development in golang"

Video tutorial playlist:

https://www.youtube.com/playlist?list=PLub5C2vM5SjJIF0f561mAON7nFG2nKHD9

Go-boiler : smaller and simplest starter kit

noreply@blogger.com (Manish Champaneri) — Mon, 29 Apr 2019 11:59:00 +0530

Project Name
Go-boiler

Project Descrtiption
Simple, small & easy to get started boiler-plate(starter-kit) written in golang for creating website/webapps.

External libraries

Cloudykit/Jet ( HTML templating engine)
gorilla/mux ( HTTP Routing )
gorilla/csrf ( CSRF )

Source code
https://github.com/mchampaneri/go-boiler

Documentation
https://mchampaneri.github.io/go-boiler

Customize Scrollbar look

noreply@blogger.com (Manish Champaneri) — Mon, 25 Mar 2019 16:27:00 +0530

You can customize the look of scrollbar using CSS.
Here is a snippet of sample custom scrollbar UI,

( I have introduced :hover because I want to show scrollbar on an element only when user hover on that element )

*::-webkit-scrollbar {
  width: 5px;
 display: none;
  background-color: #666666;
}

*:hover::-webkit-scrollbar {
  width: 5px;
  display: block;
  background-color: #666666;
}
*::-webkit-scrollbar-track {
  background: #ddd;
}

*::-webkit-scrollbar-thumb {
  background: #666;
  border-radius: 20px;
}

YTDownload ( Server )

noreply@blogger.com (Manish Champaneri) — Sat, 2 Mar 2019 10:59:00 +0530

Project Name
YTDownload

Project Descrtiption
YTDownload ( Server ) is a server-based application. Its aim is to download video from youtube in all format and qualities provided by youtube.
It uses a web browser for its UI. The core logic of the server is written in goLang.

External libraries & dependcies

VueJS ( JavaScript FrameWork )
Spectre ( Css FrameWork )
YTDL ( GoLang Youtube download Library )
Gorilla Mux ( GoLang HTTP Routing Library )

Pros

It's a server-based application. So you just have to start the server and you will be able to use it from any device having a browser and connected to the same network using local address of the server.
No need to install

Cons

Downloads with a single thread, so downloading speed will be less than those other downloaders provides.
Can not use without the local server running.
No mobile platform for the server is supported

Source code
https://sourceforge.net/projects/ytdownloader-xyz/

Download link
https://ytdownloader-xyz.sourceforge.io/

Datastore : Handling heavy insertions

noreply@blogger.com (Manish Champaneri) — Thu, 24 Jan 2019 12:52:00 +0530

Note: This is a case study to share what I faced, this might not be new to you and may have a better solution then I find. (You can share you solution here 😊) .

At first sight, datastore looks very easy to work with. And it is if insertions have to be done at a low rate. But when insertions are done at a high rate you come to find out certain issues which are not because your program is faulty but because you don't read the ( long detailed ) documentation of datastore, that itself has certain restrictions you might never hear before.

Timeout Problems

There are two kinds of timeouts
1. Call error 11: Deadline exceeded
2. API error 5 (datastore_v3: TIMEOUT)

Call error 11: Deadline exceeded

This kind of time out happens when your datastore operation takes time more than 60 sec to finish the operation.
For example, you are inserting too much data with a putMulti function. That takes time more then 60s insert.

API error 5 (datastore_v3: TIMEOUT)

This mostly occurs due to write contention.

When you attempt to write to a single entity group too quickly. Writes to a single entity group are serialized by the App Engine datastore, and thus there's a limit on how quickly you can update one entity group. In general, this works out to somewhere between 1 and 5 updates per second.

My Case

In my case, it was actually worst. Because I was doing which was recommended not to do by the GAE datastore documentation.

The most common example of this occurs when an app rapidly inserts a large number of entities of the same kind with sequential IDs. In this case, most inserts hit the same range of the same tablet, and the single tablet server is overwhelmed with writes. Most apps never have to worry about this: it only becomes a problem at write rates of several hundred queries per second and above ( chances are high when you use gorotines to speed up appliaction). If this does affect your app, the easiest solution is to use more evenly distributed IDs instead of the auto-allocated ones.

( So, I got both kinds of the error are the system, and as a solution, I have to slow down insertions, because have no choice to move ids to auto-allocated ones )

Transactions Related Issues

A transaction is a set of Google Cloud Datastore operations on one or more entities in up to 25 entity groups. Each transaction is guaranteed to be atomic, which means that transactions are never partially applied. Either all of the operations in the transaction are applied, or none of them is applied.

I have highlighted 25 here. Because it is the digit drive crazy two times in 5 days.

For the first time I was using a multiPut of 25+ entity X + 2 related entities and I got an error of exceeding entity counts. So, I fixed it by batching entity X in the batch of 25. With that everything is fine (As per my assumption)😅.

After 48hrs I got the same error again.

Then I realised that I have to take those 2 related entities also in the count and was actually trying to process 27 entities. But luckily it was the first incident of processing 23+ entity X in production env. So, It was quickly fixed before further such incidents.

Counting entities to be processed as a group in the single transaction example.

Problematic code

// entityX :=  array of 25 entityX elements
// entityXKeys :=  array of 25 datastore keys for entityX elements

_, err := client.RunInTransaction(ctx, func(tx *datastore.Transaction) error {
 
        _, err := tx.PutMulti(entityXKeys, entityX) // 25 entites
 tx.Put(key1, &ele1) // 1 entity
 tx.Put(key2, &ele2) // 1 entity

 // So total here we are processing 25 + 1 + 1 = 27 elements in one entity group 
        // which is not allowe by datastore 
        return err
})

Solution

// instead of inserting 25 element at once use batch of approriate count of elements ( in

// this case 22 ) so inside transction count elements remains below or most equal to 25

for {

// entityX :=  array of 22 entityX elements
// entityXKeys :=  array of 22 datastore keys for entityX elements

_, err := client.RunInTransaction(ctx, func(tx *datastore.Transaction) error {
 
        _, err := tx.PutMulti(entityXKeys22, entityX22) // 25 entites
 tx.Put(key1, &ele1) // 1 entity
 tx.Put(key2, &ele2) // 1 entity

 // So total here we are processing 25 + 1 + 1 = 27 elements in one entity group 
        // which is not allowe by datastore 
        return err
})

Conclusion,

When you are going to do heavy insertion in datastore, you should keep in mind the insertion limit of transaction and timeout criteria provided by GAE datastore API.

Cursor based navigation for datatable

noreply@blogger.com (Manish Champaneri) — Wed, 2 Jan 2019 18:30:00 +0530

I was working on a module where data was comming from datastore and paginated using cursor. If you are not familier with datastore, then you can refer to this link.

In case of datastore we just have cursor to get next set of data. Though there is offset availabe in datastore its costly. So to be cost effective we have only option available is cursor.

Datastore decodes cursor and in response returns data associated with that cursor.

Now, If you are using bootstrap data-table you might find it cofusing that how to pass the cursor to your queryparameter.

Even I still don't get answer of how I can change data-queryparamter dynamically to update cursor on the url.

But instead I got another simple solution.

HTML Snippet

<table id="table" data-toggle="table" >
    <thead>
      <tr>
       // your data table fields
      </tr>
    </thead>
    <tbody></tbody>
</table>
<nav aria-label="Page navigation example" style="margin-top: 2%;">
    <ul class="pagination">
      <li class="page-item"><a class="page-link" href="#"  onclick="fetchPrevdata()">Previous</a></li>
      <li class="page-item"><a class="page-link" href="#" id="nextData" onclick="fetchNextdata()">Next</a></li>
    </ul>
</nav>

Below is Javascript

$(document).ready(function () {
  fetchFirst();
});


var cursors = [""];
var index = 0;

function fetchFirst(){

  alert("fetchFirst called")
  var url = "/winners-data?c=" + cursors[index]
  $.get(url)
    .done(function (data)
    {
      if(index == cursors.length-1 ){
        cursors.push(data.Cursor);     
      }             
    })
}

function fetchNextdata()
{  
  index++
  var url = "/url?c=" + cursors[index] 
  $.get(url)
    .done(function (data)
    {
      if(index == cursors.length-1 ){
        cursors.push(data.Cursor);          
      }   
      if (data.rows != null) {       
        $table.bootstrapTable('load', data.rows)
      }  
    })
}

function fetchPrevdata()
{
  index--;
  var url = "/url?c=" +  cursors[index]
  $.get(url)
    .done(function (data)
    {
      if (data.rows != null) {
        $table.bootstrapTable('load', data.rows)     
      }        
    })
}

Hopefully, You find it useful. If you any otherway to do this please share it via comments.

datastore to bigquery data export

noreply@blogger.com (Manish Champaneri) — Fri, 7 Dec 2018 11:11:00 +0530

Google Cloud Provides a ready made script to run scheduled export from datastore.
I am using that export data as input to BigQuery.

To get it done, I had created two crons,

First for calling data export operation
Second for loading that exported data to BigQuery

Problem I was facing was that export was taking some time more time than I have assumed, So Bigquery loading cron was getting initiated before data export opertion get completed, and result bigquery loading was failing.

Solution was to make those two cron job as a single one which works as chain. On successful completion of task one second gets called.

So, I just modified the schedule export script provided by google, which exports files, and then checks if operation is successfully done or not fixed time intervals in loop, and if operation is found to be done successfully, it calls BigQuery load task.

Here is gist

partial search with firestore

noreply@blogger.com (Manish Champaneri) — Wed, 28 Nov 2018 16:00:00 +0530

You can provide partial search [ type ahead facility ] with firestore. It's not that much streight forward and also not feasible in case of large amount of data that has to support the partial search.

Here is just a sample of how you provide partial search with firestore.

Logic behind the search is a custom kind of indexing that enables partial search for us. You need to create a collection of documents called indexes which contains indexes generated for the fields you want to search over.

First reqirement is to create every possible partial search strings.

Below function returns array of the possible indexes generate after chopping out input string character by character.

/* storing searchabel fields */

function _chopped_object(iString) {

  var searchables = {}
  for (i = 0; i < iString.length; i++) {
    var string_part = iString.substr(0, iString.length - i)
    searchables[string_part] = true
  }

  return searchables

}

Output of _chopped_object(manish) will be,
{
manish :true
manis :true
mani :true
man :true
ma :true
m :true
}
That's what we want, Right !
We want tha if we write "m..." it should know we may want to write "manish"

Next question is how to store this index and use it with firebase project.

Create a sperate collection called searchables ( I am saying it searachables 😀, you are free to say anything man ), and store these index documents inside it.

 db.collection("searchables")
   .doc([some-randome-string])
   .set({
   // searchable_index: _chopped_object(sender_profile[key]),
   searchables_index: _chopped_object(sender_profile[key]),

   mapping_value: [mapping_value]

   }).then(() => {

    console.log("Document successfully updated!");

   }).catch((err) => {

    console.log(err)

   });

It may be that it's some confusing, how to use this code.

Example

We have dictonary of Persons, It includes details like name,phone,email,address

{
    name:"manish",
    phone:[not-going-to-share],
    email:m.champaneri.20@gmail.com,
    address:"something something",
},
{
    name:"mchampnaeri",
    phone:[not-going-to-share],
    email:m.champaneri.20@gmail.com,
    address:"something something",
},
{
    name:"rohan",
    phone:[not-going-to-share],
    email:m.champaneri.20@gmail.com,
    address:"something something",
},

This is our document we are storing in collection of persons

We want to create searchable index of field of name.
Means, If, I type "m" I should get suggetion of "manish" and "mchampaneri" and after "ma" it should just suggest "manish" 👍.

In this case,

when you store the person details in document you can create a searchable index in same process.

( I am using nodejs exmpale is in javascript )

function storePersonDeatails(){

var some_radom_key = [ -value- ]  // To store person document 
var some_radom_key2 = [ -value-2 ]  // To store person document search index
 
db.collection("persons")
   .doc( some_radom_key )
   .set({
   personObject
   }).then(() => {   

 // Incase of success full insertion you should create a searchable
        // index on name field for that person
  db.collection("searchables")
    .doc( some_radom_key2 )
    .set({
      searcables_index: _chopped_object(personObject.name),
     mapping_value: some_radom_key  // Same value used as key for document, 
                                                  // so index get attached with the document via key 

    }).then(() => {
     console.log("Document successfully updated!");

    }).catch((err) => {
     console.log(err)

    });

   }).catch((err) => {

    console.log(err)

   });

}

If you see your serachables documents you will find data look like somthing this

searchable Index

UpTo This we just make searchable index now just use it in your search box

here is example code for search box ..

function search(value) {
 db.collection("searchables")
  .where("searchables_index."+ value,"==",true)                 
  .get()
  .then((searchbles)=>{
    searchbles.forEach((item)=>{
    consle.log(item.data())
  })
 }) 
}

Upper code will just try to read the property [value] of the object searchables_index and will get true if it exist.

Hope this was easy to understand.

If it's not clear, let me know!

Have a great day !

Creating nested documents in firestore

noreply@blogger.com (Manish Champaneri) — Tue, 20 Nov 2018 14:39:00 +0530

Firestore returning no records yet, I see them in console
Collection not retuning any element.
... ...

You might facing issues written above. Then it might be this case.

You might getting empty response in this kind of query though there are elements in firestore.

db.collection(collection1).doc(doc1)
.collection(collection2).doc(doc2)
.collection(collection3).get()
.then(
 function (querySnapshot) {
            querySnapshot.forEach(function (doc) {   
              console.log(doc.id, " = ", doc.data());
 }
)

I face the same problem. Though data was stored sucessfully ( I was guessing, which was wrong actully ). To verify that your data is not stored as you expceted you can do check this.

Key in italics font means element is deleted

If key of your data is in italic font then your data is not stored or is marked as deleted [ that was happening in my case ]. See in image.

Now you might thinking, that I have created element and not deleted it then why it's showing as deleted.

Truth is that if you want to create nested

collection ( parent collection )

document ( parent docuemnt )
collection ( child collection )

document ( child document )

you have to create first parent document and after using its referance you have to create child document.

You should find reson behind this behaviour your self , but for now, perfect way to handle this would be is make sure you code creating documents at every level explicitly.

For example,

var collectoinRef= db.collection(collection1).doc((doc1))
    .collection(collection2).doc(doc2)
    .collection(collection3)

  collectoinRef.doc(doc1)
    .get()
    .then((docSnapshot) => {

      // If document we are going to use as parent document is exist then 
      // we just have to set the child doc inside collectin of that document
      if (docSnapshot.exists) {
         collectoinRef.doc(doc1)
          .collection(collection2)
          .doc({customeid-for-doc}).set(doc)
          .then(() => {
         // Something ....            
          })
     
      // But if document we want to use as parent not exist, Then first we have to 
      // create it.
      // If that document is just used for arragment of elements then just pass empty
      // object set({}) and it will creat empty document for you.
      // After creation of empty parent object you can create collection of child docs 
      // inside it.
     
      } else {
  // Creating empty parent object
         collectoinRef.doc(doc1).set({}).then(() => {
          // creating child object inside parent object
          // after succefull insertion of parent object
          collectoinRef.doc(doc1).collection(collection2)
            .doc({customeid-for-doc}).set(doc)
            .then(() => {
             // Something ....        
            })
        })     
      }
    })

Upload Files to Firebase Storage with dropzone.js

noreply@blogger.com (Manish Champaneri) — Tue, 6 Nov 2018 15:33:00 +0530

Almost every file uploader I found on web are expecting url to where we want to upload file. If you are using firebase storage then you will find this awkward as you don't have any rest url where you have to upload file.

Instead, file upload handled by firestore library firebase-storage.

After certain effort I found a way to use dropzone.js with firebase storage. Here is some snippet of code which you may find useful, If you are in the same situation as I am.

This both function are in same file

Dropzone configuration.. ( in index.js )

   var myDropzone = new Dropzone("form#dropzone", { // Make the whole body a dropzone
          url: '/',  
          addRemoveLinks: true,  
          method: 'put',  
          chunking:true,
          forceChunking:true,      
          autoQueue: false,
          autoProcessQueue: false         
        });
        myDropzone.autoDiscover = false;
        
        myDropzone.on("addedfile", function(file) {
          var reader = new FileReader();
          if (myDropzone.files.length < 4 ) {
              reader.onload = function(event) {
                 // event.target.result contains base64 encoded image
                 console.log("file being uploaded ")
                  storage_upload("image",event.target.result, file, myDropzone ,(r)=>{
                    console.log("Storage upload response")
                    console.log(r)
                  })
              };      
              reader.readAsDataURL(file); 
          }else {
            console.log('skipping file as we are excceding the upload limit')
          }
        });        

        myDropzone.on("removedfile", function(file){
          var storageRef = firebase.storage().ref("/");
          storageRef.child(file.fullPath).delete().then(
            function(){          
             console.log(file.fullPath  + " deleted succefuly")
            }).catch(function(error) {
              console.log("Something is wrong")
              console.log(error)
            });
          })    
        });

Firebase upload function ( in index.js not in function.js )

function storage_upload(filedata, filehandle, DropzoneHandle, cb) {

  // Getting Handle of the progressbar element of current file // 
  var progressBar = filehandle.previewElement.children[2]  

  // Firestore storage task 
  var task
  // uuid for file being uploaded
  var uuid_string = uuid()

  // Getting Storeage referance for file 
  var storageRef = firebase.storage().ref(User.client_id + "/" + uuid_string );

  // Because I am uploading file in base64 data_url // 
  task  = storageRef.putString(filedata, 'data_url');
  
  // Making progress bar of current file preview element visible  
  progressBar.opacity = 1

  task
    .on(firebase.storage.TaskEvent.STATE_CHANGED,

      // Trakcing progress of upload
      function progress(snapshot){
        var progress = (snapshot.bytesTransferred / snapshot.totalBytes) * 100;
    
        // Updating progressbar width - to make it look like filling
        progressBar.children[0].style.width = progress+'%'
       
      },

      // Firebase storeage upload error handling 
      function(error) {
        // Hanlde your way
        // A full list of error codes is available at
        // https://firebase.google.com/docs/storage/web/handle-errors
      },

      // Finishing process and returing data
      // Returning file-meta and  url
      // and hiding progress bar  
      function() {

        // Upload completed successfully, now we can get the meta data of file
        task.snapshot.ref.getMetadata().then(function(meta) {
   // Getting download url of file
          task.snapshot.ref.getDownloadURL().then(function(downloadUrl){

              // storing meta data and download url in object and returning 
              // it to callign function ... 
              var response = {
                object_info: {
                  publicURL: downloadUrl,
                  metainfo: meta
                },
                type_of_object: type
              }
              filehandle.fullPath = meta.fullPath
              return cb(response)
            })
        })
        
        // Hiding progress bar for current file 
        progressBar.style.opacity = 0
      }
    )
  
}

Firebase - update a spacific fields of single element of object of array in firestore

noreply@blogger.com (Manish Champaneri) — Fri, 5 Oct 2018 11:43:00 +0530

Firebase - update a spacific fields of single element of object of array in firestore

Its actully advisable to use map instead of array when ever it is possible. But, there are cetain cases where you don't have option to do so.

For example, you are directly saving the response from some outer source without any modification and they send you an array. In this case you will have array to work with.

Firestore does not support array here is why
"bad things can happen if you have multiple clients all trying to update or delete array elements at specific indexes. In the past, Cloud Firestore addressed these issues by limiting what you can do with arrays "
For more details information you can refer to Kato Richardson post Best Practices: Arrays in Firebase.

Firestore document having array [ used from stackoverflow question ]

Suppose you have array of object something like shown in array. Now you want to update endTime field of the object on the index [1].

There is no direct way to update it. You might get some solution like this on internet.

db.collection(something).doc(something)
.collection(something").doc(something)
.update({
"data.[1].startTime": [ value]
})

db.collection(something).doc(something)
.collection(something").doc(something)
.update({
"data.${1}.startTime": [ value]
})

But none of this is working actully [ at least not for me].

The only way to perform update on existin array element is to fetch complete array upate the desired element and update complete array.

// First data of the desired document
db.collection(something).doc(something)
.collection(something).doc(something)
.get().then((doc) => {

// Assign array to local javascript variable
var objects = doc.data().dates

// Assing desired element of object to local javascript variable
var objectToupdate = objects [1]

// Update field of the element assigned to local javascript variable
objectToupdate.startDate = [new Date]

// reassign object to local array variable
objects [1] = objectToupdate

// Update complete array with update copy of element we have
// created in local javascript variable.
db.collection(something).doc(something)
.collection(something).doc(something)
.update({
dates: dates
})
})

mailgun forward incoming mail

noreply@blogger.com (Manish Champaneri) — Mon, 10 Sep 2018 18:14:00 +0530

Mailgun is email service provider. For detail infomration about mailgun you can visit its site mailgun.com. Untill my last project( blogger-editor.xyz) i was just using mailgun sending mails. There was no need to receive emails on that domain.

But in blogger-editor.xyz, I need to have a domain based email address on which i should send and receive email. Its quite straight forward setup to receive email through mailgun.

Just you have to do these two things :

Make sure your mx record points to mailgun. [ It should be ticked verfied on mailgun dns verification ]
Setup route on mailgun

MX Record setup with Domain Registrar

In domain verification and Dns section of mailgun you will find that you have to setup two mx record with your domain service provider.

mxb.mailgun.org ( priority 10 )
mxb.mailgun.org ( priority 10 )

During creation of mx record you will be asked for host. Host value for your record is the @ if you are using root domain with mailgun , and it will be subdomain value if you are using subdomain with mailgun like mg.samplesite.com

create mx record with appropriate host [ i have used mg subdomain so have used mg as host (godday is my registrar)]

Make sure your mx records are verified by maingun. It should be right inside green circle. Other wise some thing is wrong with your mx record.

mx records verified by mailgun

Route Setup on Mailgun

What mailgun says about routes is " Mailgun Routes filter messages and trigger defined actions based on those filters. "

Mailgun does not provide faciltity like inbox where you can directly see mails you have received. But they provieds an api for fetching emails. They also provied facility to forward email.

Here you have two option

Use api and create your own inbox app which will read mails from mailgun storage
Use mailguns routes forward faciltity and forward that mail to your email address ( abc@gmail.com etc .. .)

Setup route for mail forwarding on mailgun

Find count of day differance between two dates

noreply@blogger.com (Manish Champaneri) — Fri, 7 Sep 2018 14:32:00 +0530

In golang there is no direct way [ atleast i don't know ] to find count of differance in days between two days. So, I do it the way might every one does [ except who has already faced this ].

Partial Solution

What I does is just find out duration differance of two dates using sub function.

differance := time.Now().Sub( startTime )
daysCount := int(differance.Hours() / 24.0)

It might seem ok. But, there is a problem with this solution.

Let assume, startTime is 2018-02-02 23:59:59.

In this case daysCount will be 0 on next day [ 2018-02-03 ] until time is 2018-02-03 23:59:59.

Perfect Solution

Simple solution is just normalize startTime to starting of that date.

normalizedTime := time.Date(startTime .Year(),startTime .Month(),startTime .Day(), 0, 0, 0, 0, time.Local)
differance := time.Now().Sub( normalizedTime )
daysCount := int(differance.Hours() / 24.0)

normalizedTime is startTime but at starting of date.

So, Now we are substracting time from 2018-02-02 00:00:00 instead of 2018-02-02 23:59:59.

Setup forward proxy in apache2

noreply@blogger.com (Manish Champaneri) — Wed, 22 Aug 2018 18:09:00 +0530

Proxy word is noramally used for forward proxy. It forwards requst to other server and pass downs the response to the client.

This post is not supposed to describe the differneace between forward and revsers proxy.

But if you want to find out exact differance you can refer this stackoverflow post.

https://stackoverflow.com/questions/224664/difference-between-proxy-server-and-reverse-proxy-server

To setup forward proxy on your apahce2. you have to install [ if you are on linux ] / enable [ if you are using windows ] modules

mod_proxy.so
mod_proxy_ajp.so
mod_proxy_balancer.so
mod_proxy_connect.so
mod_proxy_http.so
mod_slotmem_shm.so
mod_ssl.so

[ If I am includeing any extra please corrent me. But I had to eanble all this modules. ]

You have to make virtual host for proxy forward

< VirtualHost *: XX >

ProxyRequests On

ProxyVia On

< Proxy * >

Require all granted

< /Proxy >

</VirtualHost>

You can restrict forward proxy for the request from certain hosts only. Just have change required greatend

ProxyRequests On

ProxyVia On

// Allow certain host only to

// use forward proxy

// For other hosts it will block

// Requests

Require xx . xxx . xxx . xxx granted

</Proxy >

</VirtualHost>

Packfolder : archive your resource of sciter based application

noreply@blogger.com (Manish Champaneri) — Tue, 7 Aug 2018 17:11:00 +0530

Packfolder

packfolder is utility provided as part sciter-sdk. It compresses resources folder under a single file of the format of your choice ( should support by packfolder ).

Currently, the supported output formats are

Syntex,

[path]\packfolder.exe [resource-folder] [resource-file].[ext] -V [name-of-variable] -[output-format]

For example,

Windows PowerShell

Source Code

08-packfolderIntro

Use packfolder to archive your resource

noreply@blogger.com (Manish Champaneri) — Tue, 7 Aug 2018 16:56:00 +0530

In commit 25d8ed42e0bb4c4acb742de9f148f6902d32ff41 go-sciter SDK has added one new facility which automatically resolves file path for archived data . Now you can use archived data from the archive and it will resolve import by itself, as it does for c & c++.

Packfolder

packfolder is utility provided as part sciter-sdk. It compresses resources folder under a single file of the format of your choice ( should support by packfolder ). Currently, supported output format files are bin ( normal binary file ) an array of bytes in c, go, d and rust language.

Syntex is ,
[path]\packfolder.exe [resource-folder] [resource-file].[ext] -V [name-of-variable] -[output-format]

path = full / relative path to packfolder.exe to execute command
resource-folder = full/relative path to the folder inside which your resouce is contained
resource-file = name of the file you want as your output file
name-of-variable = it will the name of the varilble that will iniated with the array of bytes of your resource folder
output-format = format of your output file

Example

[ ps stands for PowerShell - I'm using windows ]

Command
PS:.\packfolder.exe res res.go -V resource -go

here I've put packfolder.exe in the same folder as my project. [ ya, you can do that, after all, packfolder is standalone utitily . ]

res is the folder which contians resource used in my app

res.go is the file I am expecting to be generated on compress of the resource folder.

-V stands for the variable name I want the variale name as resoruce so I "-V resource"

and at last, i want output for the golang file. so -go

It will generate a syntactilly valid file of golang contiaing bytes of our archive. Upto this we have compressed our res folder in a single resource file.

Load archived resource in your App using SetResourceArchive

SetResouceArchive function is responsible for loading the archived file at their relative paths.

func main() {
// make rect for window
rect := sciter.NewRect(100, 100, 300, 500)

// create a window using upper rect
win, _ := window.New(sciter.SW_MAIN|sciter.SW_CONTROLS|sciter.SW_RESIZEABLE, rect)

// SetResourceArchive expects array of bytes as input
// It does open the archive and also auto resolvs file
// file path
// Here resoure is array of bytes that has been generated
// by packfolder.exe
win.SetResourceArchive(resources)
win.LoadFile("this://app/htdocs/notepad.htm")

win.SetTitle("Notepad+-")

// registering methods
win.DefineFunction("closeWindow", closeApplication)
win.DefineFunction("save", save)
win.DefineFunction("open", open)

win.Show()
win.Run()
win.CloseArchive()
}

So now your resource are part of your gocode. There is no need to ship the resource folder any more. It also make your resource safe and hidden from outside.

For example, visit 08-packfolderIntro

Remove console screen for your gui application written in go

noreply@blogger.com (Manish Champaneri) — Fri, 20 Jul 2018 11:24:00 +0530

Remove console screen for your gui application written in go

Images of appliaction used as example is notepad plus minus. I writen as example. To get details of this program you can visit post " Simple document based application with golang "

You write a GUI application and it runs with a console screen behind. It looks pretty weird, right! I mean what is the meaning of a console when console actually not going to interact with the user.

GoLang Gui Application with Console Screen

Don't know how this works on Mac or Linux as I have not done this on either Mac or Linux. But for windows, I find a way to compile console-less GUI application for golang.

Just build the application with this argument.

go build -ldflags "-H=windowsgui"

ldflags used to set build time variable. Now. in the upper statement we are saying golang that current application we are building is windowsgui application so don't create the console with it. Just create a GUI of application.

Go Gui Application without console