Google Developers Blog

Developer updates from Coral

2021-06-25T10:08:00.000-07:00

Posted by The Coral Team

We're always excited to share updates to our Coral platform for building edge ML applications. In this post, we have some interesting demos, interfaces, and tutorials to share, and we'll start by pointing you to an important software update for the Coral Dev Board.

Important update for the Dev Board / SoM

If you have a Coral Dev Board or Coral SoM, please install our latest Mendel update as soon as possible to receive a critical fix to part of the SoC power configuration. To get it, just log onto your board and install the update as follows:

This will install a patch from NXP for the Dev Board / SoM's SoC, without which it's possible the SoC will overstress and the lifetime of the device could be reduced. If you recently flashed your board with the latest system image, you might already have this fix (we also updated the flashable image today), but it never hurts to fetch all updates, as shown above.

Note: This update does not apply to the Dev Board Mini.

Manufacturing demo

We recently published the Coral Manufacturing Demo, which demonstrates how to use a single Coral Edge TPU to simultaneously accomplish two common manufacturing use-cases: worker safety and visual inspection.

The demo is designed for two specific videos and tasks (worker keepout detection and apple quality grading) but it is designed to be easily customized with different inputs and tasks. The demo, written in C++, requires OpenGL and is primarily targeted at x86 systems which are prevalent in manufacturing gateways – although ARM Cortex-A systems, like the Coral Dev Board, are also supported.

Web Coral

We've been working hard to make ML acceleration with the Coral Edge TPU available for most popular systems. So we're proud to announce support for WebUSB, allowing you to use the Coral USB Accelerator directly from Chrome. To get started, check out our WebCoral demo, which builds a webpage where you can select a model and run an inference accelerated by the Edge TPU.

New models repository

We recently released a new models repository that makes it easier to explore the various trained models available for the Coral platform, including image classification, object detection, semantic segmentation, pose estimation, and speech recognition. Each family page lists the various models, including details about training dataset, input size, latency, accuracy, model size, and other parameters, making it easier to select the best fit for the application at hand. Lastly, each family page includes links to training scripts and example code to help you get started. Or for an overview of all our models, you can see them all on one page.

Transfer learning tutorials

Even with our collection of pre-trained models, it can sometimes be tricky to create a task-specific model that's compatible with our Edge TPU accelerator. To make this easier, we've released some new Google Colab tutorials that allow you to perform transfer learning for object detection, using MobileDet and EfficientDet-Lite models. You can find these and other Colabs in our GitHub Tutorials repo.

We are excited to share all that Coral has to offer as we continue to evolve our platform. Keep an eye out for more software and platform related news coming this summer. To discover more about our edge ML platform, please visit Coral.ai and share your feedback at coral-support@google.com.

Apply now for Google for Startups Accelerator: Women Founders

2021-06-24T09:00:00.005-07:00

Posted by Ashley Francisco

The Google for Startups Accelerator: Women Founders program is now open for applications until July 19.

Last year, we launched the inaugural Google for Startups Accelerator: Women Founders program in North America to help women-led startups identify and solve technical challenges while scaling their companies. The inaugural cohort also received tailored programming to address some of the longstanding barriers that women founders face.

Women founders remain underrepresented in the tech startup ecosystem because they often lack access to the resources needed to start, build, and grow their businesses. The COVID-19 pandemic exacerbated these structural barriers by disproportionately impacting women in the workforce—and research shows women were more vulnerable to the economic effects of COVID-19 because of existing gender disparities.

For women founders, access to capital is one of the major challenges to launching their business. A recent report showed women-led startups received a mere 2.3% of global venture capitalist funding in 2020, falling from 2.8% the year before.

The Google for Startups Accelerator: Women Founders program aims to help bridge the gap and create opportunities for women founders to succeed. Beyond mentorship and technical project support, the accelerator also includes deep dives and workshops focused on product design, customer acquisition, and leadership development for founders. Participants will also hear from a roster of speakers and facilitators who deliver both technical and nontechnical programming for women-led startups.

Applications for the second Google for Startups Accelerator: Women Founders program are now open until July 19, for North American applicants. Approximately 10-12 startups with at least one woman founder will be selected from across North America. The accelerator runs from September through to December 2021.

To learn more about the program and to apply, visit the website.

Introducing "Serverless Migration Station" Learning Modules

2021-06-23T10:12:00.013-07:00

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

Helping users modernize their serverless apps

Earlier this year, the Google Cloud team introduced a series of codelabs (free, online, self-paced, hands-on tutorials) designed for technical practitioners modernizing their serverless applications. Today, we're excited to announce companion videos, forming a set of "learning modules" made up of these videos and their corresponding codelab tutorials. Modernizing your applications allows you to access continuing product innovation and experience a more open Google Cloud. The initial content is designed with App Engine developers in mind, our earliest users, to help you take advantage of the latest features in Google Cloud. Here are some of the key migrations and why they benefit you:

Migrate to Cloud NDB: App Engine's legacy ndb library used to access Datastore is tied to Python 2 (which has been sunset by its community). Cloud NDB gives developers the same NDB-style Datastore access but is Python 2-3 compatible and allows Datastore to be used outside of App Engine.
Migrate to Cloud Run: There has been a continuing shift towards containerization, an app modernization process making apps more portable and deployments more easily reproducible. If you appreciate App Engine's easy deployment and autoscaling capabilities, you can get the same by containerizing your App Engine apps for Cloud Run.
Migrate to Cloud Tasks: while the legacy App Engine taskqueue service is still available, new features and continuing innovation are going into Cloud Tasks, its standalone equivalent letting users create and execute App Engine and non-App Engine tasks.

The "Serverless Migration Station" videos are part of the long-running Serverless Expeditions series you may already be familiar with. In each video, Google engineer Martin Omander and I explore a variety of different modernization techniques. Viewers will be given an overview of the task at hand, a deeper-dive screencast takes a closer look at the code or configuration files, and most importantly, illustrates to developers the migration steps necessary to transform the same sample app across each migration.

Sample app

The baseline sample app is a simple Python 2 App Engine NDB and webapp2 application. It registers every web page visit (saving visiting IP address and browser/client type) and displays the most recent queries. The entire application is shown below, featuring Visit as the data Kind, the store_visit() and fetch_visits() functions, and the main application handler, MainHandler.


import os
import webapp2
from google.appengine.ext import ndb
from google.appengine.ext.webapp import template
 
class Visit(ndb.Model):
    'Visit entity registers visitor IP address & timestamp'
    visitor   = ndb.StringProperty()
    timestamp = ndb.DateTimeProperty(auto_now_add=True)
 
def store_visit(remote_addr, user_agent):
    'create new Visit entity in Datastore'
    Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put()
 
def fetch_visits(limit):
    'get most recent visits'
    return (v.to_dict() for v in Visit.query().order(
            -Visit.timestamp).fetch(limit))
 
class MainHandler(webapp2.RequestHandler):
    'main application (GET) handler'
    def get(self):
        store_visit(self.request.remote_addr, self.request.user_agent)
        visits = fetch_visits(10)
        tmpl = os.path.join(os.path.dirname(__file__), 'index.html')
        self.response.out.write(template.render(tmpl, {'visits': visits}))
 
app = webapp2.WSGIApplication([
    ('/', MainHandler),
], debug=True)

Baseline sample application code

Upon deploying this application to App Engine, users will get output similar to the following:

VisitMe application sample output

This application is the subject of today's launch video, and the main.py file above along with other application and configuration files can be found in the Module 0 repo folder.

Next steps

Each migration learning module covers one modernization technique. A video outlines the migration while the codelab leads developers through it. Developers will always get a starting codebase ("START") and learn how to do a specific migration, resulting in a completed codebase ("FINISH"). Developers can hit the reset button (back to START) if something goes wrong or compare their solutions to ours (FINISH). The hands-on experience helps users build muscle-memory for when they're ready to do their own migrations.

All of the migration learning modules, corresponding Serverless Migration Station videos (when published), codelab tutorials, START and FINISH code, etc., can all be found in the migration repo. While there's an initial focus on Python 2 and App Engine, you'll also find content for Python 3 users as well as non-App Engine users. We're looking into similar content for other legacy languages as well so stay tuned. We hope you find all these resources helpful in your quest to modernize your serverless apps!

Google Pay plugin for Magento 2

2021-06-22T09:00:00.000-07:00

Posted by Soc Sieng, Developer Advocate

Google Pay plugin for Magento 2

Update June 15, 2021: The Google Pay extension v1.1.1 for Magento now supports the Cybersource payment gateway allowing merchants who prefer Cybersource to easily integrate Google Pay into their checkout flow.

Other key updates include support for Magento 2.4 as well as Magento 2.3. Easy to download and install without any coding, Google Pay for Magento provides fast, frictionless checkout and allows consumers to use already-stored payment credentials. This reduces data entry errors and translates to higher conversion rates and larger average order size for merchants.

We are pleased to announce the launch of the official Google Pay plugin for Magento 2. The Google Pay plugin can help increase conversions by enabling a simpler and more secure checkout experience in your Magento website. When you integrate with Google Pay, your customers can complete their purchases quickly using the payment methods they’ve securely saved to their Google Accounts.

Google Pay in action.

The Google Pay plugin is built in collaboration with Unbound Commerce, is free to use, and integrates with popular payment service providers including Adyen, BlueSnap, Braintree, FirstData - Payeezy & Ucom, Moneris, Stripe, and Vantiv.

Installation

The Google Pay plugin can be installed from the Magento Marketplace using this link or by searching the Magento Marketplace for “Google Pay”.

Refer to the Magento Marketplace User Guide for more installation instructions.

Getting started

To get started with the Google Pay plugin, you will need your Google Pay merchant identifier which can be found in the Google Pay Business Console.

Your Merchant ID can be found in the Google Pay Business Console.

Configuring the Google Pay plugin

Once installed, you can configure the plugin in your site’s Magento administration console by navigating to Stores > Configuration > Sales > Payment Methods and selecting the Configure button next to Google Pay.

Click on the Configure button to start the setup process.

Testing out Google Pay can be achieved in three easy steps:

Google Pay credentials: enter your Google Pay merchant ID (available from the Google Pay Business Console) and merchant name.
Payment gateway credentials: select your payment gateway from the list of payment gateways supported by the Google Pay plugin.
1. Choose the Sandbox environment for testing purposes.
2. Enter your payment gateway’s credentials into their respective form fields.
Google Pay settings: enable Google Pay and choose the card networks that you would like to accept.

You can optionally try out some of the advanced settings that provide the ability to customize the color and type of Google Pay button as well as enabling Minicart integration, which is recommended.

Checkout the Advanced Settings to further customize how and where the Google Pay button is presented in your store.

If your payment provider isn’t listed as an option in the payment gateway list, check to see if your payment provider’s plugin has built-in support for Google Pay.

Launching Google Pay for your website

When you’ve completed your testing, submit your website integration in the Google Pay Business Console. You will need to provide your website’s URL and screenshots to complete the submission.

Summing it up

Integrating Google Pay into your website is a great way to increase conversions and to improve the purchasing experience for your customers.

Find out more about Google Pay and the Google Pay plugin for Magento.

What do you think?

Do you have any questions? Let us know in the comments below or tweet using #AskGooglePayDev.

Add dialogs and slash commands to your Google Workspace Chat bots

2021-06-17T10:00:00.001-07:00

Posted by Charles Maxson, Developer Advocate

Developing your own custom Google Chat bot is a great way for users and teams to interact with your solutions and services both directly and within context as they collaborate in Chat. More specifically, Chat bots can be used in group conversations to streamline workflows, assist with activities in the context of discussions, and provide information and notifications in real time. Chat bots can also be used in direct messages, offering a new way to optimize workflows and personal productivity, such as managing project tasks or reporting time activity. Because use cases for bots are varied, you can consistently reach a growing audience of Chat users over time, directly where they work and uh-hum, chat.

Once you’ve identified your specific use case for your custom Chat bot, how you design the bot itself is super important. Bots that are intuitive and easy to use see better adoption and develop a more loyal following. Those that are not as fluid or approachable, or come across as confusing and complicated to use, will likely miss the mark of becoming an essential “sticky” tool even if your back end is compelling. To help you build an engaging, must-have Google Chat bot, we recently added a one-two feature punch to the Chat bot framework that allows you to build a much richer bot experience than ever before!

(Re)Introducing slash commands for Google Chat bots

The first new(er) feature that you can leverage to enhance the usability of your Chat bots are slash commands. Released a few months back, slash commands simplify the way users interact with your Chat bot, offering them a visual leading way to discover and execute your bot’s primary features. Unlike bots created prior to slash commands, where users had to learn what features a bot offered and then invoke the bot and type the command correctly to execute them, slash commands make Chat bot usage faster and help users get the most out of your bot.

Users can now simply type “/” in the message line to reveal a list of all the functions offered by the bots available to the room or direct message, and select the one to their liking to execute it. Slash commands can be invoked standalone (e.g. /help) or include user added text as parameters (e.g. /new_task review project doc ) that the developer can handle when invoked. To help make bot command discovery even simpler, the slash commands list filters matching commands once the user starts typing beyond the / (e.g. “/h” shows all commands beginning with H). This is super helpful as more and more bots are added to a room, and as more bots with slash commands are introduced by developers. Also included directly in the Slash Command UI is a description of what each command does (up to 50 characters), easing the guesswork out of learning.

As a developer, slash commands are straightforward to implement, and daresay essential in offering a better bot experience. In fact, if you have an existing Google Chat bot you’ve built and deployed, it’s likely more than worthwhile to revise your bot to include slash commands in an updated release.

To add slash commands to any Chat bot, you will need to register your commands in the Hangouts Chat API configuration page. (e.g. https://console.cloud.google.com/apis/api/chat.googleapis.com/hangouts-chat?project=<?yourprojectname?>) There is a section for slash commands that allows you to provide the /name and the description the user will see, along with the important Command Id unique identifier (a number between 1-1000) that you will later need to handle these events in your code.

When a user invokes your bot via a Slash Command, there is a slashCommand field attached to the message sent to the bot that indicates the call was initiated from a Slash Command. Remember users can still @mention your bot to call it directly by name without a / command and this helps you distinguish the difference. The message also includes the corresponding commandId for the invoked command based on what you set up in the bot configuration page, allowing you to identify the user’s requested command to execute. Finally, the message also offers additional annotations about the event and includes any argumentText supplied by the user already parsed from the command text itself.

{
  ...
  "message": {
    "slashCommand": {
      "commandId": 4
    },
    "annotations": [
      {
        "length": 6,
        "slashCommand": {
          "type": "INVOKE",
          "commandId": 4,
          "bot": {
            "type": "BOT",
            "displayName": "Slashbot"
          },
          "commandName": "/debug"
        },
        "type": "SLASH_COMMAND"
      }
    ],
    ...
    "argumentText": " show code",
    "text": "/debug show code",
    ...
}

Here is a simple example used to determine if a Slash Command was invoked by the user, and if so, runs the requested command identified by its Command Id.

function onMessage(event) {

  if (event.message.slashCommand) {

    switch (event.message.slashCommand.commandId) {
      case 1: // Command Id 1
        return { 'text': 'You called commandId 1' }

      case 2: // Command Id 2
        return { 'text': 'You called commandId 2' }

      case 3: // Help
        return { 'text': 'You asked for help'  }

    }
  }
}

Introducing dialogs for Google Chat bots

The second part of the one-two punch of new Google Chat bots features are dialogs. This is a brand new capability being introduced to the Chat bot framework that allows developers to build user interfaces to capture inputs and parameters in a structured, reliable way. This is a tremendous step forward for bot usability because it will simplify and streamline the process of users interacting with bot commands. Now with dialogs, users can be led visually to supply inputs via prompts, versus having to rely on wrapping bot commands with natural language inputs -- and hoping they correctly executed syntax the bot could decipher.

For developers, you can design UIs that are targeted to work precisely with the inputs you need users to supply your commands, without having to parse out arguments and logically infer the intent of users. In the end, dialogs will greatly expand the type of solution patterns and use cases that Chat bots can handle, as well as making the experience truly richer and more rewarding for users and developers alike.

Technically, Chat bot dialogs leverage the aforementioned slash commands combined with the existing Google Workspace Add-on Card framework to support the creation and handling of dialogs. To get started, you create a Slash Command that will invoke your dialog by designating it’s Slash command triggers a dialog setting to true in the Slash Command configuration process, as seen below:

Once you have configured a Slash Command to trigger a dialog, it will send an onMessage event when it’s invoked as it would before, but now it includes new details that indicate it is representing a dialog request. To handle this event you can use the example above with non-dialog Slash Command, using the commandId you can use a switch to determine what the user requested.

Designing the actual elements that the dialog renders is where you draw from the Google Workspace Add-on Card-based framework. If you’ve built a new generation of Google Workspace Add-on, this part will be familiar where you construct widgets, add headers and sections, create events, etc. In fact, you can even reuse or share some of your Add-on UIs within your Chat bots, but do note there currently is a lighter subset of elements available for bots. The benefits of using Cards allows you to build modern, consistently-styled user interfaces for your bots that doesn’t require that you get bogged down in low level details like managing tags or CSS. You can learn more about working with Cards starting here. To make building your Cards-based interfaces for Add-ons and Chat bots even easier, we have also just introduced the GWAO Card Builder tool, which employs a drag-n-drop visual designer to boost your development efforts.

Once you’ve assembled your Card’s widgets, to make it render as a dialog when invoked you must specify that its a DIALOG type within the action_response as seen stubbed out here below:

{
  "action_response": {
  "type": "DIALOG",
  "dialog_action": {
    "dialog": {
      "body": {
        "sections": [
          {
           "widgets": [
             {
               "textInput": {
                 "label": "Email",
                 "type": "SINGLE_LINE",
                 "name": "fieldEmail",
                 "hintText": "Add others using a comma separator",
...

Now with a working dialog, all there is left to do is handle user events once it's displayed. Again this is similar to how you would handle events working with Cards within Add-ons. Your bot will receive an event that is type CARD_CLICKED with a DialogEventType set to SUBMIT_DIALOG. The actionMethodName value will let you know what element the user clicked to process the request, e.g. ‘assign’ as depicted below. The response includes the formInputs details which are the user provided inputs returned from the dialog, which you can process as your solution needs to.

{ dialogEventType: 'SUBMIT_DIALOG',
  type: 'CARD_CLICKED',
  action: { actionMethodName: 'assign' },
  ...
  common: 
   { hostApp: 'CHAT',
     formInputs: 
      { 'whotochoose-dropdown': [Object],
        whotochoose: [Object],
        email: [Object] },
     invokedFunction: 'assign' },
  isDialogEvent: true }

Once your bot is finished processing its task, it can respond back to the user in one of two ways. The first is with a simple acknowledgement (aka OK) response letting them know their action was handled correctly and close out the dialog.

{
    "action_response": {
      "type": "DIALOG",
      "dialog_action": {
        "action_status": "OK",
...

The other option is to respond with another dialog, allowing you to follow-up with a new or revised dialog useful for complex or conditional input scenarios. This is accomplished as it was originally when you called a dialog using a dialog card within an ActionResponse to get started.

{
  "action_response": {
  "type": "DIALOG",
  "dialog_action": {
    "dialog": {
...

Next Steps

To get started building Google Workspace Chat bots, or to add slash commands and dialogs to your existing Chat bots, please explore the following resources:

Our latest updates on Fully Homomorphic Encryption

2021-06-14T10:00:00.003-07:00

Posted by Miguel Guevara, Product Manager, Privacy and Data Protection Office.

As developers, it’s our responsibility to help keep our users safe online and protect their data. This starts with building products that are secure by default, private by design, and put users in control. Everything we make at Google is underpinned by these principles, and we’re proud to be an industry leader in developing, deploying, and scaling new privacy-preserving technologies that make it possible to learn valuable insights and create helpful experiences while protecting our users’ privacy.

That’s why today, we are excited to announce that we’re open-sourcing a first-of-its-kind, general-purpose transpiler for Fully Homomorphic Encryption (FHE), which will enable developers to compute on encrypted data without being able to access any personally identifiable information.

A deeper look at the technology

With FHE, encrypted data can travel across the Internet to a server, where it can be processed without being decrypted. Google’s transpiler will enable developers to write code for any type of basic computation such as simple string processing or math, and run it on encrypted data. The transpiler will transform that code into a version that can run on encrypted data. This then allows developers to create new programming applications that don’t need unencrypted data. FHE can also be used to train machine learning models on sensitive data in a private manner.

For example, imagine you’re building an application for people with diabetes. This app might collect sensitive information from its users, and you need a way to keep this data private and protected while also sharing it with medical experts to learn valuable insights that could lead to important medical advancements. With Google’s transpiler for FHE, you can encrypt the data you collect and share it with medical experts who, in turn, can analyze the data without decrypting it - providing helpful information to the medical community, all while ensuring that no one can access the data’s underlying information.

In the next 10 years, FHE could even help researchers find associations between specific gene mutations by analyzing genetic information across thousands of encrypted samples and testing different hypotheses to identify the genes most strongly associated with the diseases they’re studying.

Making more products private by design

Our principle to make our products private by design drives us to build ground-breaking computing technologies that enable personalized experiences while protecting your private information. Privacy-preserving technologies are on the cutting-edge of Google’s innovations, and they have already shown great potential to help shape a more private internet.

In 2016, Google researchers invented Federated Learning, a technique that helps preserve privacy by keeping as much personal information on your device as possible. And in 2019, Google made its differential privacy library freely available to any organization or developer, an advanced anonymization technology that enables developers to learn from their data privately. No one has scaled the use of Differential Privacy more than we have.

We’ve been thrilled to see these technologies put to use across the globe; in France, for example, a startup called Arkhn has been able to accelerate scientific discovery using differential privacy to share data across hospitals.

We still have a ways to go before most computations happen with FHE -- but much as it took some time for HTTPS to take off and be widely adopted, today’s announcement is an important step towards bringing users helpful products that preserve their privacy and keep their data safe.

At Google, we know that open-sourcing our technologies with the developer community for feedback and use helps make them better. We will continue to invest and lead the privacy-preserving technology field by publishing new work, and open-sourcing it for everyone to use at scale - and we're excited to continue this practice by sharing this latest advancement with developers everywhere. We can't wait to see what you’ll build, and we look forward to collaborating on the journey towards a safer Internet.

#IamaGDE: Josue Gutierrez

2021-06-10T10:05:00.001-07:00

Posted by Alicja Heisig

#IamaGDE series presents: Google Maps

The Google Developers Experts program is a global network of highly experienced technology experts, influencers, and thought leaders who actively support developers, companies, and tech communities by speaking at events and publishing content.

Meet Josue Gutierrez — Maps, Web, Identity and Angular Google Developer Expert.

Josue currently works at the German company Boehringer Ingelheim and lives near Frankfurt. Before moving to Germany, Josue was working as a software engineer in Mexico, and before that, he spent almost a year in San Francisco as a senior front-end developer at Sutter Health.

Josue Gutierrez

Josue studied computer science and engineering as an undergraduate and learned algorithms and programming. His first language was C++, and he learned C and Python, but was drawn to web technologies.

“When I saw a web browser for the first time, it stuck with me,” he says, “It was changing in real time as you’re developing. That feeling is really cool. That’s why I went into frontend development.”

Josue has worked on multiple ecommerce projects focused on improving customers’ trade experience. He sees his role as creating something from scratch to help people improve lives.

“These opportunities we have as developers are great — to travel, work for many verticals, and learn many businesses,” he says. “In my previous job, I developed tech-oriented trade tools for research companies, to manipulate strings or formulas. I was on the team involved in writing these kinds of tools, so it was more about the trade experience for doctors.”

Getting involved in the developer community

Josue’s first trip outside Mexico, to San Francisco, exposed him to the many developer communities in the area, and he appreciated the supportive communities of people trying to learn together. Several of the people he met suggested he start his own meetup in Mexico City, to get more involved in Google technologies, so he launched an Angular community there. As he hunted for speakers to come to his Angular meetup, Josue found himself giving talks, too.

Then, the GDG Mexico leader invited Josue to give talks on Google for startups.

“That helped me get involved in the ecosystem,” Josue says. “I met a lot of people, and now many of them are good friends. It’s really exciting because you get connected with people with the same interests as you, and you all learn together.”

“I’m really happy to be part of the Google Maps ecosystem,” Josue says. “It’s super connected, with kind people, and now I know more colleagues in my area, who work for different companies and have different challenges. Seeing how they solve them is a good part of being connected to the product. I try to share my knowledge with other people and exchange points of view.”

Josue says 2020 provided interesting opportunities.

“This year was weird, but we also discovered more tools that are evolving with us, more functionalities in Hangouts and Meetup,” Josue says. “It’s interesting how people are curious to get connected. If I speak from Germany, I get comments from countries like Bolivia and Argentina. We are disconnected but increasing the number of people we engage with.”

He notes that the one missing piece is the face-to-face, spontaneous interactions of in-person workshops, but that there are still positives to video workshops.

“I think as communities, we are always trying to get information to our members, and having videos is also cool for posterity,” he says.

He is starting a Maps developer community in Germany.

“I have colleagues interested in trying to get a community here with a solid foundation,” he says. “We hope we can engage people to get connected in the same place, if all goes well.”

Favorite Maps features and current projects

As a frontend developer, Josue regards Google Maps Platform as an indispensable tool for brands, ecommerce companies, and even trucking companies.

“Once you start learning how to plant coordinates inside a map, how to convert information and utilize it inside a map, it’s easy to implement,” he says.

In 2021, Josue is working on some experiments with Maps, trying to make more real-time actualization, using currently available tools.

“Many of the projects I’ve been working on aren’t connected with ecommerce,” he says. “Many customers want to see products inside a map, like trucking products. I’ve been working in directories, where you can see the places related to categories — like food in Mexico. You can use Google Maps functionalities and extend the diversification of maps and map whatever you want.”

“Submission ID is really cool,” he adds. “You can do it reading the documentation, a key part of the product, with examples, references, and a live demo in the browser.”

Future plans

Josue says his goal going forward is to be as successful as he can at his current role.

“Also, sharing is super important,” he says. “My company encourages developer communities. It’s important to work in a place that matches your interests.”

Follow Josue on Twitter at @eusoj |Check out Josue’s projects on GitHub.

For more information on Google Maps Platform, visit our website or learn more about our GDE program.

Starting your Google Career in IT | Kate Grant

2021-06-07T10:00:00.005-07:00

Posted by Max Saltonstall

A little over 10 years ago, we launched the IT Residency Program (ITRP) at Google with a twofold mission: to provide exceptional tech support for Googlers and to empower the next generation of IT pioneers.

ITRP’s founding principle is learning and development. In addition to formal on-the-job IT support training, the program takes its residents through a focused career development path, including a hands-on rotation in a specific Google function in their chosen specialty. For their part, the residents, who come from a wide spectrum of often non-traditional backgrounds, bring with them a passion for learning. ITRP converts that passion into real-world experience and equips them for a lifelong career in tech.

Today, hundreds of Googlers are ITRP alums, working in disciplines ranging from site reliability engineering to security and privacy to program management and all points in between. Looking back on the program’s 10-plus years, we wanted to share some of their stories, their experiences and their triumphs. Look for more installments in this series in the weeks to come.

Kate Grant

While studying anthropology in college, Kate Grant had a part time help desk job, helping solve people's computer problems (and sometimes the computer's people problems). And it was through that job that she realized she really liked the satisfaction she got from helping people so directly. At school, Kate could help students, faculty and staff — both technical folks and non-technical, and it was great to save the day for them, teach them, and constantly learn about new technologies.

Joining the Pride March in NYC as part of Google's presence

As graduation loomed close, Kate needed something to do afterwards, and she stumbled upon the ITRP posting. It looked like a great way to continue doing what she loved, and in a much bigger company, with more to learn. She applied, thinking "there's no way in hell" she would get this job.

Turns out she was wrong!

Growth in ITRP

Starting at Google in August 2012, Kate worked in Mountain View, CA helping Googlers of all types with a broad range of tech challenges. She got to spend some time with the Search team at Google working on technical documentation, helping engineers at Google as well as web designers outside Google better understand Google search features. This experience gave Kate something cool she could show her family too, a very tangible "I made this" moment, which can be hard to come by in IT operations work.

During her time in the program, Kate also had the chance to spend some time working in the New York office, which felt good because it brought her closer to New Jersey, where she grew up. It was the first step towards coming back to the Northeast, a welcome return to a comfortable place. Eventually she'd move to New York to work in Google's NYC office full time, helping with IT operations and later managing junior IT help desk folks as the team expanded.

Neon welcome sign in the lobby of the Google NYC office

Working across teams and projects and help desks in ITRP helped Kate develop all kinds of skills, from improving proficiency with Linux, Windows, MacOS and Chrome to also developing better judgment and analysis skills when solving novel problems that walked into the support desk. And as she began helping to train new employees in orientation, covering IT, Security and Technology topics, she got practice improving her public speaking; by the end of it she was speaking to over 100 people at a time on Mondays in Mountain View as new employees learned the essentials of their work at Google.

After ITRP

While she had been considering a tech writing career path before, Kate realized that the work in ITRP helping people day-to-day let her wear many hats. She was writing documentation, that was one key component. But she also wrote code, managed programs and projects, mentored junior members of the team, and analyzed data. The breadth of the job matched her growth goals, and Kate ended up continuing her career in Techstop, transitioning into a role as a Corporate Operations Engineer in 2014.

The work remained in the front-line support team and gave Kate tremendous exposure to the wide array of people in the Google headquarters. She focused more on projects to improve the onboarding and daily operations of the support teams, helping to keep systems and services healthy. This involved scheduling, mentoring, training and helping the newer members of the team, mostly later cohorts of IT Residents.

Kate's experience training new employees, and mentoring new IT Residents, made her a great fit for a new opportunity on the onboarding team, where she worked to make the new employee experience more consistent and reliable between offices across Google globally. This began a traversal of different parts of Google's IT org, where she learned about how the company, and its many teams, operate. Kate's work focused on helping to create better, smoother, more automated processes for the teams in IT, and help to scale the successes they had already achieved.

Making the jump to Operations Manager

Driving improvements to Google’s onboarding infrastructure was fun and satisfying, but Kate missed working with junior techs day to day, and mentoring them on their own career growth. Luckily in 2016 a new manager position opened up in NYC, and Kate jumped at it. She started leading a team of newer support technicians, including IT Residents. Now she could focus on making a really great support experience for Googlers, and give back to the program she had enjoyed by helping the folks on her team grow.

In 2019 Kate had the opportunity to move to Austin, TX to build out a new ITRP Hub location from scratch, and continues to manage IT Residences and Corporate Operations Engineers there today.

Reflections

Coming full circle now Kate shared some great advice with us for those thinking about ITRP:

"Don't screen yourself out of the opportunity. Folks come into the program from all walks of life. You don't have to have a super fancy computer science degree. If you’re excited about technology and helping people, ITRP could be a great next step for you."

She reinforced that you get out of the program what you put into it: “If you take the time to be curious, to ask questions, to investigate, you will learn so much. It's really an endless amount of material you could absorb, and nobody can get through it all. But when you put in the work, it pays you back”.

And as people go through the ITR Program, they end up in all sorts of places. "There's no cookie cutter outcome… It doesn’t mean you’re doing something wrong if your path doesn’t look like someone else’s. The right way is what's right for you and your goals."

Getting Started with Smart Home Notifications and Follow-up Responses

2021-06-04T10:00:00.001-07:00

Posted by Toni Klopfenstein, Developer Advocate

Alerts for important device events, such as a delivery person arriving or the back door failing to lock, create a more beneficial and reassuring experience for your smart home device users.

As we announced at I/O, you can now add proactive notifications and follow-up responses to your Smart Home Action to alert users to events in a timely, relevant and helpful fashion and better engage with your end users.

Notifications can either alert a user to an event that has occurred without them proactively issuing a request through the Assistant, or as a follow-up to verify that the user's request has been fulfilled. Each device event that triggers one of these notifications has a unique event id, which helps the Assistant route it to the appropriate Home Graph users and Google Home Smart Speakers or Nest Smart Displays, depending on the notification type and priority. Notifications and follow-up responses can also provide users with additional information, such as error and exception codes, or timestamps for the event.

You can enable notifications on your existing devices once users opt-in to receive alerts by updating the device definition and requesting a SYNC intent. You can then send device notifications along with any applicable device state changes using the Home Graph API.

We are adding support for traits where asynchronous requirements are a core use case.The following device traits now support follow-up responses to user queries:

Additionally, we are launching proactive notification alerts for the following traits:

For more information, check out the developer guides and samples, or check out the Notifications video.

We want to hear from you, so continue sharing your feedback with us through the issue tracker, and engage with other smart home developers in the /r/GoogleAssistantDev community. Follow @ActionsOnGoogle on Twitter for more of our team's updates, and tweet using #AoGDevs to share what you’re working on. We can’t wait to see what you build!

Machine Learning GDEs: Q1 2021 highlights, projects and achievements

2021-06-03T10:00:00.005-07:00

Posted by HyeJung Lee and MJ You, Google ML Ecosystem Community Managers. Reviewed by Soonson Kwon, Developer Relations Program Manager.

Google Developers Experts is a community of passionate developers who love to share their knowledge with others. Many of them specialize in Machine Learning (ML). Despite many unexpected changes over the last months and reduced opportunities for various in person activities during the ongoing pandemic, their enthusiasm did not stop.

Here are some highlights of the ML GDE’s hard work during the Q1 2021 which contributed to the global ML ecosystem.

ML GDE YouTube channel

With the initiative and lead of US-based GDE Margaret Maynard-Reid, we launched the ML GDEs YouTube channel. It is a great way for GDEs to reach global audiences, collaborate as a community, create unique content and promote each other's work. It will contain all kinds of ML related topics: talks on technical topics, tutorials, interviews with another (ML) GDE, a Googler or anyone in the ML community etc. Many videos have already been uploaded, including: ML GDE’s intro from all over the world, tips for TensorFlow & GCP Certification and how to use Google Cloud Platform etc. Subscribe to the channel now!!

TensorFlow Everywhere

17 ML GDEs presented at TensorFlow Everywhere (a global community-led event series for TensorFlow and Machine Learning enthusiasts and developers around the world) hosted by local TensorFlow user groups. You can watch the recorded sessions in the TensorFlow Everywhere playlist on the ML GDE Youtube channel. Most of the sessions cover new features in Tensorflow.

International Women’s Day

Many ML GDEs participated in activities to celebrate International Women’s Day (March 8th). GDE Ruqiya Bin Safi (based in Saudi Arabia) cooperated with WTM Saudi Arabia to organize “Socialthon” - social development hackathons and gave a talk “Successful Experiences in Social Development", which reached 77K viervers live and hit 10K replays. India-based GDE Charmi Chokshi participated in GirlScript's International Women's Day event and gave a talk: “Women In Tech and How we can help the underrepresented in the challenging world”. If you’re looking for more inspiring materials, check out the “Women in AI” playlist on our ML GDE YouTube channel!

Mentoring

ML GDEs are also very active in mentoring community developers, students in the Google Developer Student Clubs and startups in the Google for Startups Accelerator program. Among many, GDE Arnaldo Gualberto (Brazil) conducted mentorship sessions for startups in the Google Fast Track program, discussing how to solve challanges using Machine Learning/Deep Learning with TensorFlow.

TensorFlow

Meanwhile in Europe, GDEs Alexia Audevart (based in France) and Luca Massaron (based in Italy) released “Machine Learning using TensorFlow Cookbook”. It provides simple and effective ideas to successfully use TensorFlow 2.x in computer vision, NLP and tabular data projects. Additionally, Luca published the second edition of the Machine Learning For Dummies book, first published in 2015. Her latest edition is enhanced with product updates and the principal is a larger share of pages devoted to discussion of Deep Learning and TensorFlow / Keras usage.

On top of her women-in-tech related activities, Ruqiya Bin Safi is also running a “Welcome to Deep Learning Course and Orientation” monthly workshop throughout 2021. The course aims to help participants gain foundational knowledge of deep learning algorithms and get practical experience in building neural networks in TensorFlow.

Nepal-based GDE Kshitiz Rimal gave a talk “TensorFlow Project Showcase: Cash Recognition for Visually Impaired" on his project which uses TensorFlow, Google Cloud AutoML and edge computing technologies to create a solution for the visually impaired community in Nepal.

On the other side of the world, in Canada, GDE Tanmay Bakshi presented a talk “Machine Learning-powered Pipelines to Augment Human Specialists” during TensorFlow Everywhere NA. It covered the world of NLP through Deep Learning, how it's historically been done, the Transformer revolution, and how using the TensorFlow & Keras to implement use cases ranging from small-scale name generation to large-scale Amazon review quality ranking.

Google Cloud Platform

We have been equally busy on the GCP side as well. In the US, GDE Srivatsan Srinivasan created a series of videos called “Artificial Intelligence on Google Cloud Platform”, with one of the episodes, "Google Cloud Products and Professional Machine Learning Engineer Certification Deep Dive", getting over 3,000 views.

Korean GDE Chansung Park contributed to TensorFlow User Group Korea with his “Machine Learning Pipeline (CI/CD for ML Products in GCP)” analysis, focused on about machine learning pipeline in Google Cloud Platform.

Last but not least, GDE Gad Benram based in Israel wrote an article on “Seven Tips for Forecasting Cloud Costs”, where he explains how to build and deploy ML models for time series forecasting with Google Cloud Run. It is linked with his solution of building a cloud-spend control system that helps users more-easily analyze their cloud costs.

If you want to know more about the Google Experts community and all their global open-source ML contributions, visit the GDE Directory and connect with GDEs on Twitter and LinkedIn. You can also meet them virtually on the ML GDE’s YouTube Channel!

Grow your indie game with help from Google Play

2021-06-01T09:00:00.001-07:00

Posted by Patricia Correa, Director, Global Developer Marketing

At Google Play we’re committed to helping all developers thrive, whether these are large multinational companies or small startups and indie game studios. They are all critical to providing the services and experiences that people around the world look for on their Android devices. The indie game developer community, in particular, constantly pushes the boundaries with their creativity and passion, and bring unique and diverse content to players everywhere.

To continue supporting indies, today we’re opening submissions for two of our annual developer programs - the Indie Games Accelerator and the Indie Games Festival. These programs are designed to help small games studios grow on Google Play, no matter what stage they are in:

If you are a small games studio looking for help to launch a new title, apply for the Accelerator to get mentorship and education;
Or, if you have already created and launched a high quality game that is ready for the spotlight, enter the Festival for a chance to win promotions.

This year the programs come with some changes, including more eligible markets and fully digital event experiences. Learn more below and apply by July 1st.

Accelerator: Get education and mentorship to supercharge your growth

If you’re an indie developer, early in your journey - either close to launching a new game or recently launched a title, this is the program for you. We’ll provide education and mentorship that will help you build, launch and grow successfully.

This year we have nearly doubled the eligible markets, with developers from over 70 countries being eligible to apply for the 2021 program.

Selected participants will be invited to take part in a 12-week online acceleration program. During this time you’ll get exclusive access to a community of Google and industry experts, as well as a network of other passionate developers from around the world looking to supercharge their growth.

Festival: win promotions that put your game in the spotlight

If you're an indie game developer who has recently launched a high quality game, this is your chance to have your game discovered by industry experts and players worldwide.

This year we will, again, host three competitions for developers from Japan, South Korea, and selected European countries.

Prizes include featuring on Google Play store, promotional campaigns worth 100,000 EUR, and more.

How useful did you find this blog post?

★ ★ ★ ★ ★

Tech Camp introduces Georgia high schoolers to technology careers

2021-05-27T16:05:00.006-07:00

Posted by Posted by Erica Hanson, Senior Program Manager, Google Developer Student Clubs

Tamta Kapanadze wishes that she had learned sooner about careers in technology. By the time that the Georgian citizen learned about them, she was already a university student.

As Kapanadze continued her studies and her interest in technology grew, she wanted to spread the word about the growing field to high-school students in Georgia, a country where the industry is still small.

To do this, Kapanadze called in the support of Google Developer Student Clubs (GDSCs), community groups for college and university students interested in Google's developer technology. After Kapanadze graduated from university, she continued her work by organizing a chapter of Google Developer Groups (GDGs) for Kutaisi.

Google Developer Groups are the largest community network of professional developers in the world. The program consists of local chapters that provide inclusive environments open to everybody interested in tech. The chapters let members learn new skills, and meet other developers with similar interests through online and in-person events.

However, even after all that, Kapanadze still wanted to do more. She brought together friends and colleagues to partner with Mariam, GDSC Georgia American University Lead; Iliko, GDSC Georgia American University core team member; Giorgi, GDSC Tbilisi State University Lead; and Bakar, GDSC San Diego State University Lead. With this team assembled, they planned Tech Camp, a virtual technological learning experience that teaches high schoolers about tech fields and how to start careers in web development, game development, artificial intelligence, machine learning, and more.

Tamta Kapanadze.

While it's difficult enough to plan and execute a new event, Kapanadze and her partners didn't let the additional challenges of the last year stop their plans to launch Tech Camp. They wanted to publicize the event by mid-January, so they made a to-do list and set deadlines for themselves. After a few weeks of intense planning, they:

Chose the session topics
Started looking for speakers
Chose dates and created a timetable for the camp
Created an application form
And created logos and other designs

Kapanadze and her partners accepted applications for Tech Camp from Jan. 20 to Feb. 10 and announced their speakers to the public to keep the buzz about the event going. They originally hoped to receive 30 applications, but instead received 500 across their events. In particular, 300 students attended the speaker sessions and 50 students attended the coding sessions, where they would teach them about algorithms and the basics of C++.

Finally, the first day of Tech Camp arrived on Feb. 15. They began each session with fun icebreakers to help everybody feel comfortable, including themselves. Here's a timeline of what each day covered:

Day 1:
- Digital professions
- Hardware and software
Day 2:
- Mobile development
- Web development
Day 3:
- Cybersecurity
- Game development
- Data engineering
Day 4:
- UI/UX design
- Embedded systems
Day 5:
- Cloud
- Test automation
Day 6:
- Artificial intelligence and machine learning
- Career development
Day 7:
- Importance of technology
- Freelance jobs
- Award ceremony

A screenshot of the virtual session about game development.

Everybody defines success differently, but for Kapanadze and her team it meant impacting at least one person. By this measure, Tech Camp succeeded because many of those who attended decided to pursue careers in tech. As for Kapanadze, she can’t wait to see what the future holds for Georgia's high schoolers and the country's growing tech industry.

To watch recordings from Tech Camp, please visit the playlist on YouTube.

The recordings available from Tech Camp.

For more information, find a Google Developers community group near you.

Behind the scenes: How the Google I/O photo booth was made

2021-05-27T10:00:00.011-07:00

Posted by the Google Developers team

A closer look at building a Flutter web app with Google Developer products

If you attended Google I/O this year, you probably stopped by the Google I/O photo booth for a selfie with our Google Developer mascots: Flutter’s Dash, Android Jetpack, Chrome’s Dino, and Firebase’s Sparky. If you didn’t, it’s not too late to jump in, take a selfie, and share it on social media! We loved seeing all of the pictures you posted and your favorite props! Want to learn more about building a camera plugin, layouts, and gestures used in a photo booth for Flutter on the web?

It took a combination of Google developer products to make the photo booth successful. The Flutter and Firebase teams joined forces to build a best in class example of Flutter on the web that used Firebase for hosting, auth, performance monitoring, and social sharing. Take a closer look at how the photo booth was built here and then grab the open source code on Github!

Flutter team members having fun in the photo booth!

Prosthesis control via Mirru App using MediaPipe hand tracking

2021-05-26T10:00:00.002-07:00

Guest post by the Engineering teams at Mirru and Tweag

What is the Mirru App?

Mirru is a free and open source Android app under development with which one can control robotic prosthetic hands via hand tracking. With our app, a user can instantly mirror grips from their sound hand onto a robotic one, which could be 3D-printed and self-assembled at low cost. With Mirru, we want to provide a cheap, intuitive and open end-to-end alternative to existing, costly, cumbersome and proprietary technology.

Figure 1: A demonstration of using MediaPipe hand tracking to move a robotic hand’s fingers with the Mirru app.

The Mirru team is a collaboration between Violeta López and Vladimir Hermand, two independent designers and technologists currently based in Paris. To kickstart the project, the team took part in Tweag’s Open Source Fellowship program which provided funding, mentorship and data engineering expertise from one of their engineers, Dorran Howell. The fellowship helped get Mirru launched from the ground-up.

Our goal for the 3-month fellowship was to develop an initial version of the Android app that can control any bluetooth open source hand using computer vision techniques, and make the app available for free on the Google Play store so anyone can print their own hand, assemble it, and download the app. With the help of MediaPipe, we were able to quickly prototype our app without having to build our own machine learning model, as we didn’t have the resources or training data to do so.

Why use hand tracking?

Using your phone and a front-facing camera with hand tracking opens up a new, affordable, accessible, and versatile way to control prosthetics.

Let's say I'm a left hand amputee who owns a robotic prosthesis. Every day, I need my prosthetic hand to actuate a lot of different grip patterns. For example, I need to use a pinch or tripod grip to pick up small objects, or a fist grip to pick up objects like a fruit or a cup. I change and execute these grip patterns via myoelectric muscle sensors that allow me to, for example, open and close a grip by flexing and unflexing my upper limb muscles. These myoelectric muscle sensors are the main interface between my body and the prosthesis.

However, living with them is not as easy as it seems. Controlling the myoelectric sensors can take a lot of time to get used to, and many never do. It can also be quite expensive to get these sensors fitted by a prosthetist, especially for people in developing countries or anyone without health insurance. Finally, the number of grips on many devices currently on the market is limited to less than ten, and only few models come with ways to create custom grips, which are often cumbersome.

Mirru provides an alternative interface. Using just their phone, a tool many have access to, a user can digitally mirror their sound hand in real-time and communicate with their prosthesis in an intuitive way. This removes the expensive need to be fitted by a prosthetist and enables the user to quickly program an unlimited amount of grips. For now, Mirru stays away from electromyography altogether as reliable muscle sensors are expensive. The programmed grips therefore need to be triggered via the android phone, which is why this first version of our app is more suited for activities like sweeping, holding a book while reading it, or holding a cup or shopping bag. In the future we hope to combine myoelectric sensors with hand tracking to get the benefits of both.

Programming a grip with the Mirru app looks like the following: Let’s say that I want to grab an object with my robotic hand. I bring my prosthesis near the object and I then form the desired grip with my sound hand in front of my android phone and Mirru mirrors it in real-time to the prosthesis. I then lock my prosthesis into this new grip and free up my vsound hand. Finally I might save this grip for later use and add it to my library of grips.

Figure 2: A user tester using hand-tracking on their phone to program their prosthesis’s grip to pick up a measuring tape and measure with the other hand.

The Brunel Hand and the Mirru Arduino Sketch

In order to accomplish our goal of allowing as many people as possible to print, assemble, and control their own hand, we designed the Mirru android app to be compatible with any robotic hand that is controlled by a bluetooth-enabled Arduino board and servo motors.

For our project, we printed and assembled an open source robotic hand called the Brunel Hand made by Open Bionics. First, we 3D printed the Brunel Hand’s 3D printable files that are made available under the CC Attribution-Sharealike 4.0 International License. We then bought the necessary servos, springs, and screws to assemble the hand. In combination with printing and buying the servos, the hand costs around €500 to purchase and assemble.

The Brunel Hand comes with myoelectric-based firmware and a PCB board developed by Open Bionics, but since the hand is in essence just 4 servo motors, any microcontroller could be used. We ended up using an Adafruit ESP32 feather board for its bluetooth capabilities and created an Arduino sketch that can be downloaded, customized, and uploaded for anyone who is printing and assembling their own hand. They could then download the Mirru app to use as the control-interface for their printed hand.

Hand-tracking with MediaPipe

There are many computer vision solutions available for hand tracking that could be used for this project, but we needed a fast, open source solution that didn’t require us to train our own model, and that could be used reliably on a portable device such as a phone.

MediaPipe provides great out of the box support for hand tracking, and since we didn’t have the training data or resources available to create a model from scratch, it was perfect for our team. We were able to build the Android example apps easily and were excited to find that the performance was promising. Even better, no tweaking on the ready-made hand tracking model or the graphs was necessary, as the hand landmark model provided all the necessary outputs for our prototype.

When testing the prosthesis on real users, we were happy to hear that many of them were impressed with how fast the app was able to translate their movements, and that nothing else exists on the market that allows you to make custom grips as fast and on-the-fly.

Figure 3: A user tester demonstrates how quickly the MediaPipe hand-tracking can translate her moving fingers to the movement of her prosthesis’s fingers.

Translating 3D MediaPipe points into inputs for Robotics

To achieve the goals of the Mirru app, we need to use hand tracking to independently control each finger of the Brunel Hand in real-time. In the Brunel Hand, the index, middle, and ring fingers are actuated using servos that move at an angle from 0 to 180 degrees; 0 means the finger is fully upright and 180 means the finger is fully flexed down. As we lacked adequate training data to create a model from scratch that could calculate these servo angles for us, we opted to use a heuristic to relate the default hand tracking landmark outputs to the inputs required by our hardware for an initial version of our prototype.

Figure 4: In the lab testing the translation of the outputs to inputs with the app and the prototype.

We were initially unsure whether the estimated depth (Z) coordinate in the 3D landmarks would be accurate enough for the translation of inputs or if we would be limited to working in 2D. As an initial step, we recorded an example dataset and spun up a visualization of the points in a Jupyter Notebook with Plotly. We were immediately impressed by the quality and accuracy of the coordinates, considering that the technology only uses a single camera without any depth sensors. As noted in the MediaPipe documentation, The Z coordinates have a slightly different scale than the X/Y coordinates, but this didn’t seem to pose a significant challenge for our prototype.

Figure 5: A data visualization of the hand made up of 21 3D hand landmarks provided by MediaPipe.

Given the accuracy of the 3D landmarks, we opted to use a calculation in 3D for relating landmark outputs to the inputs required by the prosthesis. In our approach, we calculate the acute angles of the fingers in relation to the palm by calculating the angle between the finger direction and the normal of the plane defined by the palm. An angle of 0° corresponds to maximum closure of the finger, and an angle of 180° indicates a fully extended finger. We were able to calculate the finger direction by calculating the vector from the landmark at the base of the fingers to the landmark on the tip of the fingers.

Figure 6: Diagram showing the 3D landmarks and which ones we used to calculate the finger direction vector, the palm normal, and the angle that both form.

We calculate the palm normal by selecting three points in the plane of the palm. Using Landmark 0 as the reference point, we calculate the vectors for side 1 and side 2, and compute the cross product of those vectors to give us the palm normal. Finally, we compute the angle of the finger direction and the palm normal. This returns an angle in radians that we use to calculate degrees.

We had to do some extra processing to match the degrees of freedom for the thumb on our prosthetic hand. The thumb moves in more complex ways than the rest of the fingers. In order to get our app to work with the thumb, we did similar calculations for thumb direction and the palm normal, but we used different landmarks.

Once we do the calculation of the servo angles on the android phone, we send those values via bluetooth to the Arduino board, and the Arduino board moves the servos to the proper position. Due to some noise in the model outputs, we add a smoothing step to the pipeline, which is important so that the movements of the robotic fingers aren’t too jittery for precise grips.

Figure 7: A user tester makes a pinch grip on her prosthesis with the Mirru app.

Summary

The Mirru app and Mirru Arduino Sketch are designed to allow anyone to control an open source prosthesis with their sound hand and an Android phone. This is a novel and frugal alternative to muscle sensors, and MediaPipe has proven that it is the right tool for the essential hand tracking component of the full application. The Mirru team was able to get started quickly with MediaPipe’s out of the box solutions without having to gather any training data or having to design a model from scratch. The speed of the real-time translation from hand tracking points to the robotic hand has especially excited all of our users in our testing sessions and opens up many possibilities for the future of prostheses.

We see exciting potential for combining the MediaPipe hand tracking features with existing myoelectric prostheses which could open powerful and advanced ways to create and save custom prosthesis grips in real-time. Also, with the help of MediaPipe, we have been able to provide an open source alternative to proprietary prostheses without the need for myoelectric sensors or a visit to a prosthetist, at a cost that is much lower than what is already on the market, and whose source code can be customized and built-upon by other developers. Our team is excited to see what other ideas the open source community might come up with, and to see what hand tracking can bring to users and manufacturers of prostheses.

As for the current state of the Mirru application, we have yet to implement the possibility of recording and saving moving gestures that are longer sequences compared to the static grip positions. For example, imagine being able to record the movement of the fingers to play a bass line on a piano, like a loopable animated gif. There is a realm of possibilities for prostheses that is waiting to be explored, and we’re really happy that MediaPipe gives us access to it.

We are looking for contributors. If you have ideas or comments about this application, please reach out to mirruapp@gmail.com, or visit our GitHub.

This blog post is curated by Igor Kibalchich, ML Research Product Manager at Google AI.

A conversation with Hebe He, a developer from Guangzhou

2021-05-24T11:43:00.000-07:00

Posted by Brian Shen, Program Manager, Google Developers

Google Developer Groups are one of the largest community networks of developers in the world. Every group has an organizer that helps curate events based on the interests of their local developer community.

As we continue to explore how different Google Developer Groups build their communities, we interviewed Hebe He, an organizer of Google Developer Group Guangzhou in China. Learn more about how she is building the developer scene in China, thinking up new events for her community, and more below.

Hebe He, an organizer of Google Developer Group Guangzhou in China.

Tell us about yourself.

I am Hebe from China and I'm a native of Guangzhou. I'm the organizer of GDG Guangzhou, as well as an ambassador for Women Techmakers (WTM). I work at one of China's new electric-vehicle brands, where I'm responsible for the intelligent business operation of the Internet of Vehicles. I'm relatively outgoing and active, so I really like to deal with different people, whether it's at work or in other activities.

How did you learn about Google Developer Groups?

In 2014, I participated in GDG Guangzhou DevFest for the first time by coincidence and met the founder of GDG Guangzhou. Afterward, I joined the founder's company and volunteered at many GDG programs. In 2017, I officially became an organizer after the existing organizers recognized my ability and desire to contribute more to the GDG Guangzhou community.

Tell us more about Guangzhou and the developer community there.

Our community members are talented, passionate, and amazing. I see all kinds of possibilities in them. They're always excited for every event we hold, keep a fanatical attitude toward Google's technological innovation, and are particularly interested in Android, Kotlin, and Flutter.

What are events like in your community?

We highly value feedback from event participants, who are interested in a wide range of topics. For this reason, we generally use 15% of every event to cover non-technical topics, such as entrepreneurship, business management, and careers. For more comprehensive activities, such as DevFest, we increase the amount of non-technical content to roughly 30%.

What is your Google Developer Group focused on right now?

We devote most of our energy to improving the quality of activities. We try to add more elements to the event to strengthen the interaction of participants in hopes of improving the feedback mechanism and gaining more valuable suggestions for future event optimization. We also try to improve the quality of guests and themes, and pay more attention to event details, such as event announcements, registration, and check-in.

What’s your favorite community memory from a Google Developer Group event?

The memory that touches me the most is the construction of WTM Guangzhou. From the first event with only 80 developers to the audience of more than 500 people in recent years, it represents the recognition of, and support for, our events. There are many people who come to participate every year; some are actively encouraging their friends to participate and others are even urging us to hold events. They feel honored to be invited to our events and their enthusiasm endured during the pandemic.

What's next for you and your Google Developer Group?

There's still lots of room to grow in our community. We hope that we can continue to develop a Google Developer Group that reflects the best of Guangzhou. We also hope to find better ways to accumulate the experience shared by speakers and the value of community users.

If you want to grow your career and coding knowledge with people like Hebe He, join a Google Developer Group near you.

Join us for Google for Games Developer Summit 2021

2021-05-24T09:00:00.005-07:00

Posted by Greg Hartrell, Head of Product Management, Games on Android & Google Play

With a surge of new gamers and an increase in time spent playing games in the last year, it’s more important than ever for game developers to delight and engage players. To help developers with this opportunity, the games teams at Google are back to announce the return of the Google for Games Developer Summit 2021 on July 12th-13th.

Hear from experts across Google about new game solutions they’re building to make it easier for you to continue creating great games, connecting with players, and scaling your business. Registration is free and open to all game developers.

Register for the free online event at g.co/gamedevsummit to get more details in the coming weeks. We can’t wait to share our latest innovations with the developer community.

What's new for Android developers at Google I/O

2021-05-20T16:53:00.002-07:00

Cross-posted on the Android Developers blog by Karen Ng, Director, Product Management & Jacob Lehrbaum, Director of Developer Relations, Android & Play

As Android developers, we are all driven by building experiences that delight people around the world. And with people depending on your apps more than ever, expectations are higher and your jobs as developers aren’t getting easier. Today, at Google I/O, we covered a few ways that we’re trying to help out, whether it be through Android 12 - one of the biggest design changes ever, Jetpack, Jetpack Compose, Android Studio, and Kotlin to help you build beautiful high quality apps. We’re also helping when it comes to extending your apps wherever your users go, like through wearables and larger-screened devices. You can watch the full Developer Keynote, but here are a few highlights:

Android 12: one of the biggest design updates ever.

The first Beta of Android 12 just started rolling out, and it’s packed with lots of cool stuff. From new user safety features like permissions for bluetooth and approximate location, enhancements to performance like expedited jobs and start up animations, to delightful experiences with more interactive widgets and stretch overscrolling, this release is one of the biggest design updates to Android ever. You can read more about what’s in Android 12 Beta 1 here, so you can start preparing your apps for the consumer release coming out later this year. Download the Beta and try it with your apps today!

Jetpack Compose: get ready for 1.0 in July!

For the last few years, we’ve been hard at work modernizing the Android development experience, listening to your feedback to keep the openness–a hallmark of Android, but becoming more opinionated about the right way to do things. You can see this throughout, from Android Studio, a performant IDE that can keep up with you, to Kotlin, a programming language that enables you to do more with less code, to Jetpack libraries that solve the hardest problems on mobile with backward compatibility.

The next step in this offering is Jetpack Compose - our modern UI toolkit to easily build beautiful apps for all Android devices. We announced Compose here at Google I/O two years ago and since then have been building it in the open, listening to your feedback to make sure we got it right. With the Compose Beta earlier this year, developers around the world have created some truly beautiful, innovative experiences in half the time, and the response to the #AndroidDevChallenge blew our socks off!

With the forthcoming update of Material You (which you can read more about here), we’ll be adding new Material components as well as further support for building for large screens, making it fast and easy to build a gorgeous UI. We’re pressure testing the final bits in Compose and will release 1.0 Stable in July—so get ready!

Android Studio Arctic Fox: Design, Devices, & Developer Productivity!

Android Studio Arctic Fox (2020.3.1) Beta, the latest release of the official powerful Android IDE, is out today to help you build quality apps easier and faster. We have delivered and updated the suite of tools to empower three major themes: accelerate your UI design, extend your app to new devices, and boost your developer productivity. With this latest release you can create modern UIs with Compose tooling, see test results across multiple devices, and optimize debugging databases and background tasks with the App Inspector. We’re also making your apps more accessible with the Accessibility Scanner and more performant with Memory Profiler. And for faster build speeds, we have the Android Gradle plugin 7.0, new DSL, and variant APIs. You can learn more about the Android Studio updates here.

Kotlin: the most used language by professional Android devs

Kotlin is now the most used primary language by professional Android developers according to our recent surveys; in fact, over 1.2M apps in the Play Store use Kotlin, including 80% of the top 1000 apps. And here at Google, we love it too: 70+ Google apps like Drive, Home, Maps and Play use Kotlin. And with a brand-new native solution to annotation processing for Kotlin built from the ground up, Kotlin Symbol Processing is available today, a powerful and yet simple API for parsing Kotlin code directly, showing speeds up to 2x faster with libraries like Room.

Android Jetpack: write features, not boilerplate

With Android Jetpack, we built a suite of libraries to help reduce boilerplate code so you can focus on the code you care about. Over 84% of the top 10,000 apps are now using a Jetpack library. And today, we’re unpacking some new releases for Jetpack, including Jetpack Macrobenchmark (Alpha) to capture large interactions that affect your app startup and jank before your app is released, as well as a new Kotlin Coroutines API for persisting data more efficiently via Jetpack DataStore (Beta). You can read about all the updates in Android Jetpack here.

Now is the time: a big step for Wear

The best thing about modern Android development is that these tools have been purpose built to help make it easy for you to build for the next era of Android, which is all about enabling devices connected to your phone–TVs, cars, watches, tablets–to work better together.

Starting today, we take a huge step forward with wearables. First, we introduced a unified platform built jointly with Samsung, combining the best of Wear and Tizen. Second, we shared a new consumer experience with revamped Google apps. And third, a world-class health and fitness service from Fitbit is coming to the platform. As an Android developer, it means you’ll have more reach, and you’ll be able to use all of your existing skills, tools, and APIs that make your mobile apps great, to build for a single wearables platform used by people all over the world.

Whether it’s new Jetpack APIs for Wear tailored for small screens and designed to optimize battery life, to the Jetpack Tiles API, so you can create a custom Tile for all the devices in the Wear ecosystem, there are a number of new features to help you build on Wear. And with a new set of APIs for Health and Fitness, created in collaboration with Samsung, data collection from sensors and metrics computation is streamlined, consistent, and accurate–like heart rate to calories to daily distance–from one trusted source. All this comes together in new tooling, with the release of Android Studio Arctic Fox Beta, like easier pairing to test apps, and even a virtual heart rate sensor in the emulator. And when your app is ready, users will have a much easier time discovering the world of Wear apps on Google Play, with some big updates to discoverability. You can read more about all of the Wear updates here.

Tapping the momentum of larger screens, like tablets, Chrome OS and foldables

When it comes to larger screens -- tablets, foldables, and Chrome OS laptops-- there is huge momentum. People are increasingly relying on large screen devices to stay connected with family and friends, go to school, or work remotely. In fact, there are over 250 million active large screen Android devices. Last year, Chrome OS grew +92% year over year–5 times the rate of the PC market, making Chrome OS the fastest growing and the second-most popular desktop OS. To help you take advantage of this momentum, we’re giving you APIs and tools to make optimizing that experience easier: like having your content resize automatically to more space by using SlidingpaneLayout 1.2.0 and a new vertical navigation rail component, Max widths on components to avoid stretched UIs, as well as updates to the platform, Chrome OS, and Jetpack windowmanager, so apps work better by default. You can learn more here.

Google Duo's optimized experience for foldable devices

This is just a taste of some of the new ways we’re making it easier for you to build high quality Android apps. Later today, we’ll be releasing more than 20 technical sessions on Android and Play, covering a wide range of topics such as background tasks, privacy, and Machine Learning on Android, or the top 12 tips to get you ready for Android 12. If building for cars, TVs, and wearables is your thing, we got that covered, too. You can find all these sessions - and more - on the I/O website. Beyond the sessions and news, there’s a number of fun ways to virtually connect with Googlers and other developers at this year’s Google I/O. You can check out the Android dome in I/O Adventure, where you can see new blog posts, videos, codelabs, and more. Maybe even test out your Jetpack Compose skills or take a virtual tour of the cars inside our dome!

Google Pay introduces a Flutter plugin for payments

2021-05-19T11:00:00.001-07:00

Posted by Jose Ugia, Developer Programs Engineer, Google Pay and Anthony Panissidi, Technical Writer, Google Developer Studio

We made it easier than ever to integrate Google Pay in Flutter apps!

Our open source Flutter plugin simplifies the addition of payments to Flutter apps on iOS and Android.

The plugin gives you the ability to add functionality to your apps across platforms with a single and familiar codebase written in Dart.

It adapts common steps required to facilitate payments that adhere to how Flutter constructs components, works with the user interface of the app, and exchanges information between the native and Dart ends.

Now, as a Flutter developer, you can easily reap the benefits of Google Pay, which lets you provide users with a secure and fast checkout experience that increases conversions, and frees you from the need to manage credit cards and payments.

How it works

To use the plugin, add pay as a dependency in your pubspec.yaml file. For more information, see Adding a package dependency to an app.

To configure a payment, load a payment profile with the desired configuration, either with a local file or one retrieved from a remote server. For a complete list of all configuration options, see the PaymentDataRequest object.

Here's an example of a JSON file that defines payment options:

sample_payment_configuration.json

{
  "provider": "google_pay",
  "data": {
    "environment": "TEST",
    "apiVersion": 2,
    "apiVersionMinor": 0,
    "allowedPaymentMethods": [{
      "type": "CARD",
      "tokenizationSpecification": {
        "type": "PAYMENT_GATEWAY",
        "parameters": {
          "gateway": "example",
          "gatewayMerchantId": "gatewayMerchantId"
        }
      },
      "parameters": {
        "allowedCardNetworks": ["VISA", "MASTERCARD"],
        "allowedAuthMethods": ["PAN_ONLY", "CRYPTOGRAM_3DS"],
        "billingAddressRequired": true,
        "billingAddressParameters": {
          "format": "FULL",
          "phoneNumberRequired": true
        }
      }
    }],
    "merchantInfo": {
      "merchantId": "01234567890123456789",
      "merchantName": "Example Merchant Name"
    },
    "transactionInfo": {
      "countryCode": "US",
      "currencyCode": "USD"
    }
  }
}

For more examples of JSON files that define payment options, take a look at the example/assets/ folder.

Now you can use this configuration to add the Google Pay button to your app and forward the payment method selected by your users.

Here's an example of a Dart file:

import 'package:pay/pay.dart';
 
const _paymentItems = [
  PaymentItem(
    label: 'Total',
    amount: '99.99',
    status: PaymentItemStatus.final_price,
  )
];
 
// In your Widget build() method
GooglePayButton(
  paymentConfigurationAsset: 'sample_payment_configuration.json',
  paymentItems: _paymentItems,
  style: GooglePayButtonStyle.black,
  type: GooglePayButtonType.pay,
  onPaymentResult: onGooglePayResult,
),
 
 
// In your Stateless Widget class or State
void onGooglePayResult(paymentResult) {
  // Send the resulting Google Pay token to your server or PSP
}

How to use it

The best part of this news is that you can use the plugin today. To get started with it, check out the pay package on pub.dev. We also want to hear your thoughts and feature requests, and look forward to your contributions on GitHub.

Learn more

Want to learn more about Google Pay? Here's what you can do:

To get started with Google Pay, see our developer documentation.
To integrate with Google Pay, visit Google Pay's Business Console.
Follow @GooglePayDevs on Twitter for future updates. If you have questions, tag @GooglePayDevs and include #AskGooglePayDevs in your tweets.

A new open source content library from Google

2021-05-19T10:00:00.002-07:00

Posted by Sebastian Trzcinski-Clément, Program Manager, Developer Relations

Developers around the world are constantly creating open source tools and tutorials but have a hard time getting them discovered. The content published often spanned many different sites - from GitHub to Medium. Therefore we decided to create a space where we can highlight the best projects related to Google technologies in one place - introducing the Developer Library.

The platform showcases blog posts and open source tools with easy-to-use navigation. Content is categorized by product areas; Machine Learning, Flutter, Firebase, Angular, Cloud, Android, with more to come.

What makes the Developer Library unique is that each piece featured on the site is reviewed, in detail, by a team of Google experts for accuracy and relevancy, so you know when you view the content on the site it has the stamp of approval from Google.

To demonstrate the breadth of content on the site here are some examples of published content pieces and video interviews with the developers who authored these posts:

A blood cell detection code sample using TensorFlow's Object Detection API and interview with author Sayak Paul
Blog posts in Spanish on how to use Angular and interview with author Vanessa Aristizabal
A Javascript package built on TensorFlow.js that provides easy-to-use data structures for manipulating and processing data and interview with author Rising Odegua
Coding tips on how to make efficient API calls in Flutter using a test-driven approach and interview with author Waleed Arshad

There are two ways you can help us grow the Developer Library.

Firstly, If you have great content that you would like to see published on the Developer Library, please submit it for review here.

Secondly, the team welcomes feedback, so if you have anything you’d like to see added or changed on the Developer Library site, do complete this short feedback form or just file an issue on GitHub.

We can't wait to see what you build together!

New for I/O: Assistant tools and features for Android apps and Smart Displays

2021-05-18T13:00:00.008-07:00

Posted by Rebecca Nathenson, Director of Product for the Google Assistant Developer Platform

Today at I/O, we shared some exciting new product announcements to help you more easily bring Google Assistant to your Android apps and create more engaging content on smart displays.

Assistant development made easy with new Android APIs

App Actions helps you easily bring Google Assistant to your Android app and complete user queries of all kinds, from booking a ride to posting a message on social media. Companies such as MyFitnessPal and Twitter are already using App Actions to help their users get things done, just by using their voice. You can enable App Actions in Android Studio by mapping built-in intents to specific features and experiences within your apps. Here are new ways you can help users easily navigate your content through voice queries and proactive suggestions.

Better support for Assistant built-in intents with Capabilities

Capabilities is a new framework API available in beta today that lets you declare support for common tasks defined by built-in intents. By leveraging pre-built requests from our catalog of intents, you can offer users ways to jump to specific activities within your app.

For example, the Yahoo Finance app uses Capabilities to let users jump directly to the Verizon stock page just by saying “Hey Google, show me Verizon’s stock on Yahoo Finance.” Similarly, Snapchat users can use their voice to add filters and send them to friends: “Hey Google, send a snap with my Curry sneakers.”

Improved user discoverability with Shortcuts in Android 12

App shortcuts are already a popular way to automate most common tasks on Android. Thanks to the new APIs for Shortcuts in Android 12, it’s now easier to find all the Assistant queries that are supported with apps. If you build an Android Shortcut, it will automatically show up in the Assistant Shortcuts gallery, so users can choose to set up a personal voice command in your app, when they say “Hey Google, shortcuts.”

Google Assistant can also suggest relevant shortcuts to help drive traffic to your app. For example, when using the eBay app, people will see a suggested Google Assistant Shortcut appear on the screen and have the option to create a shortcut for "show my bids."

We also introduced the Google Shortcuts Integration library, which identifies shortcuts pushed by Shortcuts Jetpack Module and makes them available to Assistant for use in managing related voice queries. By doing so, Google Assistant can suggest relevant shortcuts to users and help drive traffic to your app.

Get immediate answers and updates right from Assistant using Widgets, coming soon

Improvements to Android 12 also makes it easier to discover glanceable content with Widgets by mapping them to specific built-in intents using the Capabilities API. We're also looking at how to easily bring driving optimized widgets to Android Auto in the future. The integration with Assistant will enable one shot answers, quick updates and multi-step interactions with the same widget.

For example, with Dunkin’s widget implementation, you can say “Hey Google, reorder from Dunkin’ to select from previous drinks and place the order. Strava’s widget helps a user track how many miles they ran in a week by saying “Hey Google, check my miles on Strava”, and it will show up right on the lock screen.

Build high quality Conversational Actions for smart displays

Last year, we introduced a number of improvements to the Assistant platform for smart displays, such as Actions Builder, Actions SDK and new built-in intents to improve the experience for both developers and users. Here are more improvements rolling out soon to make building conversational actions on smart displays even better.

New features to improve the developer experience

Interactive Canvas helps you build touch- and voice-controlled games and storytelling experiences for the Assistant using web technologies like HTML, CSS, and JavaScript. Companies such as CoolGames, Zynga, and GC Turbo have already used Canvas to build games for smart displays.

Since launch, we've gotten great feedback from developers that it would be simpler and faster to implement core logic in web code. To enable this, the Interactive Canvas API will soon provide access to text-to-speech (TTS), natural language understanding (NLU), and storage APIs that will allow developers to trigger these capabilities from client-side code. These APIs will provide experienced web developers with a familiar development flow and enable more responsive Canvas actions.

We’re also giving you a wider set of options around how to release your actions. Coming soon, in the Actions Console, you will be able to manage your releases by launching in stages. For example, you can launch to one country first and then expand to more later, or you can launch to just a smaller percentage and gradually roll out over time.

Improving the user experience on smart displays

You'll also see improvements that will enhance visual experiences on the smart display. For example, you can now remove the persistent header, which allows you to utilize full real estate of the device and provide users with fully immersive experiences.

Before Interactive Canvas brought customized touch interfaces to the Smart Display, we provided a simple way to stop TTS from playing by tapping anywhere on the screen of the device. However, with more multi-modal experiences being released on Smart Displays, there are use cases where it is important to continue playing TTS while the user touches the display. Developers will soon have the option to enable persistent TTS for their actions.

We’ve also added support for long-form media sessions with updates to the Media API so you can start playback from a specific moment, resume where a previous session stopped, and adapt conversational responses based on media playback context.

Easier transactions for your voice experiences

We know how important it is to have the tools you need to build a successful business on our platform. In October of last year, we made a commitment to make it easier for you to add seamless voice-based and display-based monetization capabilities to your experience. On-device CVC and credit card entry will soon be available on smart displays. Both of these features make on-device transactions much easier reducing the need to redirect users to their mobile devices.

We hope you are able to leverage all these new features to build engaging experiences and reach your users easily, both on mobile and at home. Check out our technical sessions, workshops and more from Google I/O on YouTube and get started with App Actions and Conversational Actions today!

Unlock new use cases and increase developer velocity with the latest ARCore updates

2021-05-18T13:00:00.007-07:00

Posted by Ian Zhang, Product Manager, AR & Zeina Oweis, Product Manager, AR

ARCore was created to provide developers with simple yet powerful tools to seamlessly blend the digital and physical worlds. Over the last few years, we’ve seen developers create apps that entertain, engage, and help people in different ways–from letting fans interact with their favorite characters, to placing virtual electronics and furniture for the perfect home setup and beyond.

At I/O this year, we continue on the mission of improving and building AR developer tools. With the launch of ARCore 1.24, we’re introducing the Raw Depth API and the Recording and Playback API. These new APIs will enable developers to create new types of AR experiences and speed up their development cycles.

Increase AR realism and precision with depth

When we launched the Depth API last year, hundreds of millions of Android devices gained the ability to generate depth maps in real time without needing specialized depth sensors. Data in these depth maps was smoothed, filling in any gaps that would otherwise occur due to missing visual information, making it easy for developers to create depth effects like occlusion.

The new ARCore Raw Depth API provides more detailed representations of the geometry of objects in the scene by generating “raw” depth maps with corresponding confidence images. These raw depth maps include unsmoothed data points, and the confidence images provide the confidence of the depth estimate for each pixel in the raw depth map.

Improved geometry from the Raw Depth API enables more accurate depth measurements and spatial awareness. In the ARConnect app, these more accurate measurements give users a deeper understanding of their physical surroundings. The AR Doodads app utilizes raw depth’s spatial awareness to allow users to build realistic virtual Rube Goldberg machines.

ARConnect by PHORIA (left) and AR Doodads by Jam3 (right) use the improved geometry from the Raw Depth API

The confidence image in the Raw Depth API allows developers to filter depth data in real time. For example, TikTok’s newest effect enables users to upload an image and wrap it onto real world objects. The image conforms to surfaces where there is high confidence in the underlying depth estimate. The ability for developers to filter for high confidence depth data is also essential for 3D object and scene reconstruction. This can be seen in the 3D Live Scanner app, which enables users to scan their space and create, edit, and share 3D models.

TikTok by TikTok Pte. Ltd. (left) and 3D Live Scanner by Lubos Vonasek Programmierung (right) use confidence images from the ARCore Raw Depth API

We’re also introducing a new type of hit-test that uses the geometry from the depth map to provide more hit-test results, even in low-texture and non-planar areas. Previously, hit-test worked best on surfaces with lots of visual features.

Hit Results with Planes (left)

Works best on horizontal, planar surfaces with

good texture

Hit Results with Depth (right)

Gives more results, even on non-planar or
low-texture areas

The lifeAR app uses this improved hit-test to bring AR to video calls. Users see accurate virtual annotations on the real-world objects as they tap into the expertise of their social circle for instant help to tackle everyday problems.

lifeAR by TeamViewer uses the improved depth hit-test

As with the previous Depth API, these updates leverage depth from motion, making them available on hundreds of millions of Android devices without relying on specialized sensors. Although depth sensors such as time-of-flight (ToF) sensors are not required, having them will further improve the quality of your experiences.

In addition to these apps, the ARCore Depth Lab has been updated with examples of both the Raw Depth API and the depth hit-test. You can find those and more on the Depth API documentation page and start building with Android and Unity today.

Increase developer velocity and post-capture AR

A recurring pain point for AR developers is the need to continually test in specific places and scenarios. Developers may not always have access to the location, lighting will change, and sensors won’t catch the exact same information during every live camera session.

The new ARCore Recording and Playback API addresses this by enabling developers to record not just video footage, but also IMU and depth sensor data. On playback, this same data can be accessed, enabling developers to duplicate the exact same scenario and test the experience from the comfort of their workspace.

DiDi used the Recording and Playback API to build and test AR directions in their DiDi-Rider app. They were able to save 25% on R&D and testing costs, 60% on travel costs, and accelerated their development cycle by 6 months.

DiDi-Rider by Didi Chuxing saves on development resources with the Recording and Playback API

In addition to increasing developer velocity, recording and playback unlocks opportunities for new AR experiences, such as post-capture AR. Using videos enables asynchronous AR experiences that remove time and place constraints. For instance, when visualizing AR furniture, users no longer have to be in their home. They can instead pull up a video of their home and accurately place AR assets, enabling them to “take AR anywhere”.

Jump AR by SK Telecom uses the Recording and Playback API to transport scenes from South Korea right into users’ homes to augment with culturally relevant volumetric and 3D AR content.

JumpAR by SKT uses Recording and Playback to bring SouthKorea to your home

VoxPlop! by Nexus Studios is experimenting with the notion of Spatial Video co-creation, where users can reach in and interact with a recorded space rather than simply placing content on top of a video. The Recording and Playback API enables users to record videos, drop in 3D characters and messages, and share them with family and friends.

VoxPlop! by Nexus Studios uses the Recording and Playback API to experiment with Spatial Video co-creation

Learn more and get started with the Recording and Playback API docs.

Get started with ARCore today

These latest ARCore updates round out a robust set of powerful developer tools for creating engaging and realistic AR experiences. With over a billion lifetime installs and 850 million compatible devices, ARCore makes augmented reality accessible to nearly everyone with a smartphone. We're looking forward to seeing how you innovate and reach more users with ARCore. To learn more and get started with the new APIs, visit the ARCore developer website.

Google Pay integration patterns that drive conversions on Android

2021-05-18T11:30:00.010-07:00

Posted by Jose Ugia, Developer Relations Engineer, Google Pay & Anthony Panissidi, Technical Writer, Google Developer Studio

What do Gilt, MTS, Panera Bread, and SpotHero have in common?

At first glance, you probably only see four totally different businesses:

Gilt is an online shopping and lifestyle website.
MTS is a mobile network operator with 80 million users in Armenia, Belarus, and Russia.
Panera Bread is a chain of more than 2,000 fast-casual bakery-cafe restaurants in the US and Canada.
SpotHero is a digital parking marketplace that lets drivers reserve and pay for parking spots in more than 300 cities in the US and Canada.

However, all four businesses partnered with us to identify and adopt integration patterns that drive the most conversions on Google Pay for Android. In this blog post, we share these proven integration practices so that you can get the most out of Google Pay in your Android apps, as well as additional security tips that you can use to further secure your payment flows.

UI and UX patterns

Take a look at the following strategies to improve user experience in your app:

Payment-method selection
Express checkout
Guest checkout
Payment notifications

Payment-method selection

If you set Google Pay as a default payment option for ready-to-pay users, your users only need to click or tap twice to complete their transactions, so they enjoy a more-seamless payment experience and they're less likely to abandon their carts.

Our partners who implemented this pattern reported a significant increase in their success metrics. For example, at Gilt, 34% of total Google Pay checkouts were net-new Gilt member conversions and 57% of total Google Pay checkouts were reactivations of lapsed Gilt members.

Express checkout

This feature lets your users purchase an item directly from the item's detail page without adding it to a cart, which shortens their path to purchase completion.

For example, Gilt integrated this feature into their checkout process so its users can complete the checkout process with only a few clicks or taps. The Google Pay button on their product page lets users move directly to checkout with Google Pay set as a default payment option.

Guest checkout

This feature makes it easier for your users to complete purchases and convert, and more likely to create an account and engage again later.

To enable guest checkout, add Google Pay as an option to continue with the payment process alongside your account-creation elements.

For example, Panera Bread enabled guest checkout, and found a 7% increase in order value and 30% increase in wallet share.

As another example, SpotHero enabled guest checkout, and found that its sales funnel increased by 20 times while 87% of total checkouts were completed with Google Pay.

Payment notifications

This feature lets your users pay directly from notifications, which reduces friction in the payment process and further increases conversions.

Users sometimes receive payment notifications that they expect, such as after they abandon carts, make donations, or need to add credit to a prepaid card. They typically find these transactions simple and familiar, so they're ready to pay quickly with a little nudge.

MTS adopted this pattern to let their customers add credit to their accounts directly from notifications and experienced a 80% increase in conversions.

Learn more

For more information about how to implement these UI and UX patterns, see our sample open source app and developer documentation.

Security tips

Before we go, we also want to share these security tips to further secure your payment flows:

Use SSL for all connections between your apps and backend services over the public internet.
Do not collect or store payment data, or any other sensitive information in the clear within your app.
Order price can be calculated on the client side to show it in your UI and keep the user informed, but only allow for payments with calculations applied in your backend services.

Learn more

Want to learn more about Google Pay? Here's what you can do:

To get started with Google Pay, see our developer documentation.
To integrate with Google Pay, visit Google Pay's Business Console.
Follow @GooglePayDevs on Twitter for future updates. If you have questions, tag @GooglePayDevs and include #AskGooglePayDevs in your tweets.

Updated Google Pay app offers more consumer touchpoints

2021-05-18T10:30:00.003-07:00

Posted by Soc Sieng, Developer Advocate, Payments & Ola Ben Har, Payments DevRel Lead

We redesigned the Google Pay app to boost user engagement with your business.

The redesigned app makes it easy for users to find your business and provides you with a branded surface that lets you build relationships with your customers at scale.

The app is available in the App Store and Google Play Store in the US, India, and Singapore with availability in more markets on the way. In this blog post, we focus on features available in the US version of the app.

New in Google Pay

The Google Pay app focuses on users' relationships with people, businesses, and other everyday essentials.

The app lets users send money, save money, and see spending insights.

It makes it easy for users to save money at their favorite businesses and discover new ones.

It also provides your brand with another surface to initiate meaningful reengagement with your customers. The branded experience is automatically created when customers check out with Google Pay or a Google Pay-enrolled card in the app, in stores, or online. This dedicated space for your business is also where customers can redeem offers, sign up for loyalty rewards, and view their transaction histories.

How it works

Google Pay's new features are only part of the story.

Behind the scenes, we worked on the Google Pay APIs and developer tools to enable those experiences, help you acquire new customers, and better serve existing ones.

Google Pay APIs for Web and Android

Google Pay APIs for Web and Android enable your transaction history within your branded experience on Google Pay in addition to contactless payments in store. After a user makes a purchase with Google Pay or a Google Pay-enrolled card, they can search for your brand and view their transaction history in Google Pay.

When you integrate with the Google Pay APIs, you're not only providing a convenient and secure checkout option in your app or on your website, but you also let your users track their transactions, independent of the channel, in one central place. Your brand becomes searchable for millions of active Google Pay users, which provides you with more reengagement opportunities.

Loyalty Enrollment and Sign-in API

The Loyalty Enrollment and Sign-in API lets users discover, and sign up or sign in to your loyalty program from your branded experience with a few taps in Google Pay.

When users sign up, they provide their consent and Google Pay securely shares sign-up details with your loyalty program’s sign-up process. They can use information that they already saved to their Google Accounts, which makes the sign-up process a snap. Afterward, users can easily access their loyalty passes at checkout.

That does it for now, but these updates are only the beginning, so stay tuned for more news in this space!

Learn more

Want to learn more about Google Pay? Here's what you can do:

To get started with Google Pay, see our developer documentation.
To integrate with Google Pay, visit Google Pay's Business Console.
Follow @GooglePayDevs on Twitter for future updates. If you have questions, tag @GooglePayDevs and include #AskGooglePayDevs in your tweets.

New Dashboard in Google Developer Profiles

2021-05-04T09:00:00.005-07:00

Posted by Chris Demeke, Product Manager; Amani Newton, Technical Writer

The last time you signed into your Google Developer Profile, you may have noticed a change. Now, after signing-in, you’re invited to view your personal dashboard.

This feature provides a personalized view of your activity and learning content across Google Developers, Firebase, and Android (more to come soon). With the Developer Dashboard, you can see a history of your earned badges and other activity, continue any in-progress codelabs or pathways, or start a new pathway.

This feature is just the latest step in Google’s ongoing plan to maintain a thriving ecosystem for developers and provide more tailored experiences. Here’s what’s new:

Create and organize a personalized list of developer pages that you care about

Quickly access the pages you visit frequently using the Saved Pages feature. By selecting the bookmark icon next to any developer reference page, you can add it to your Saved Pages, and open them here at a later date.

Keep track of what you learn across Google Developers, Firebase, and Android

Your dashboard will keep track of your in-progress codelabs and pathways for you, eliminating the trouble of forgetting to bookmark. You can view a history of all of your developer achievements and activities, across Google Developers, Firebase, and Android.

Collect badges and share your achievements

Show off your familiarity with the latest technologies by passing short assessments, and earn badges that you can share on LinkedIn, Twitter, or Facebook.

Sign in today to view your personal dashboard, or to get started, head to developer.google.com/profile and click Create profile.

Updated Google Pay button increases click-through rates

2021-05-03T09:00:00.006-07:00

Posted by Soc Sieng, Developer Advocate, Google Pay

An improved Google Pay button works wonders for click-through rates and the checkout experience.

The updated Google Pay button displays a user's card information, which makes the user 30% more likely to use it and increases conversions by 3.6%.

The display of the card's type and last four digits reminds the user that they already saved a payment card to their Google Account, which makes them more likely to opt for the quick and easy checkout process that Google Pay provides.

How it works

If a user configured an eligible payment method in their Google Account at the time of purchase, the Google Pay button displays the type and last four digits of their most-recently used card.

Figure 1. An example of the Google Pay button with the additional information.

Figure 2. An example of the Google Pay button without the additional information.

How to enable card information

If you use the createButton API with default button options, your Google Pay button is automatically updated to include the user's card network and last four digits.

If you customized the createButton API and set buttonType to plain or short, set it to buy to make your Google Pay button display the user's card information.

If you haven’t integrated with the createButton API yet, consider doing so now so that the user knows that their payment details are a click away.

See it in action

To test the Google Pay button with other button options, check out this button-customization tool:

Next steps

To get started with Google Pay, visit Google Pay's Business Console. Make sure to use the createButton API to benefit from the new features. If you have any questions, tweet @GooglePayDevs on Twitter and use #AskGooglePayDevs.