Our ShopRunner Data Science team allows all members to have a quarterly hack week. It is important for data science teams to keep innovating so once per quarter team members are allowed to spend a week working on more speculative projects of their choice. For my 2019 Q3 hack week I decided to build a series of generator models to attempt to create fake products. Generator models are models commonly trained to create realistic images or text based on real world examples. This project may seem fairly outlandish, which it is, but my general idea is that if we can create strong generator models that can capture the diversity of our product catalog then we could use these generators to augment low frequency classes within our catalog for other deep learning projects such as taxonomy classification or attribute tagging.

The two networks I decided to use were OpenAI’s GPT-2 117M parameter small model for text generation and Nvidia’s StyleGAN for image generation. I then fine tuned these to internal ShopRunner datasets. For both models I found using the original Tensorflow implementations to be the best path forward since ports to other frameworks either didn’t have features that I needed or were not as well built out.

Both models were trained on a Nvidia 2080 TI graphics card.

GPT-2

In February of 2019 OpenAI announced its newest language model GPT-2. GPT-2 is trained on 40GB of internet text but OpenAI restricted the release of the model down to much smaller versions due to concerns of malicious behavior. A large part of building that high quality dataset was taking higher quality reddit. In its raw state GPT-2 is excellent at generating realistic sounding text but the text tends to fall into either reddit style dialogue or wikipedia style description. So to get the most use out of GPT-2 as a generator for fake products we need to tune it to our specific use case, in this case fashion.

The repo provided by nshepperd for fine tuning provides a series of scripts and instructions for fine tuning. To perform fine tuning we really just need to format a text dataset for consumption by GPT-2. For this I ended up writing 100K product descriptions to their own line in a .txt file with a GPT-2 specific <endoftext> token appended to the end so that GPT-2 would learn how to end the product descriptions and hopefully learn how to structure them in a more realistic manner.

GPT-2 trained fairly quickly producing good results after 15K batches/steps which took a few hours.

ShopRunner GPT-2 (SR GPT-2)

We fine tuned GPT-2 on ShopRunner data for 15K steps, leading to what we call SR GPT-2. After fine tuning SR GPT-2 is able to generate fairly realistic looking product descriptions including line breaks and formatting. These are also fairly entertaining to read.

======================================== SAMPLE 1 ========================================
WATERDOG | PINK HALSTON COLLECTION. HALSTON'S HOODED SILK FIT IS SO AWESOME HONG KONG\'s black wool-blend hooded cowl jacket is handmade from lightweight wool sourced from two countries located in the Arctic Circle and Wye Hydroelectric Power Supply.- HALSTON WOOL CLIMB JERRY WOOL FIT 
- Fabric: 90% wool, 10% wool crepe; 12% viscose, 8% polyester and 5% nylon 
- Washable, dry clean recommended 
- ImportedSKU: HU18-17FBCFABRIC & CAREMachine wash cold with similar colorsWarm down with whiteWash in cold water and hang dry  
======================================== SAMPLE 2 ========================================
Wool-Cotton Cropped T-Shirt Dress Dress with Boho Floral Print - Black This T-Shirt Dress features a Boho Floral printed front, a short sleeve hemline, and an asymmetrical ruffled hemline. A limited edition piece from our New York Fashion Show capsule.New York Fashion is celebrating the 20th anniversary of their popular T-Shirt Dress in 2011. The T-Shirt Dress is a loose-fitting bodice cut out cut in black silk crepe de chine with a fitted waistband. The dress has a V neckline, long sleeves, two front split pockets, and a draped waist.100% SilkDry Clean OnlyImportedSI38S-F18Size & FitModel is wearing a size 6Model Measurements: Height: 5ft 9in; Bust-Chest: 33in; Waist: 23.6in; Hips: 34.5inAbout SI38S-F18Sieviast designer and renowned expert James Simone launched his versatile collection of womens ready-to-wear and accessories in 2008. The label\'s garments have been named the year\'s top fashion showpieces, and the lines are sure to become a summer staple.

However using GPT-2 we can also generate context based examples.

Here is a sample generated with the context “sleeveless red asymmetric dress”

DetailsThis asymmetric fit-and-flare fit is cut in a slim fit. Cut for a flattering, relaxed fit.- Round neckline. 
- Sleeveless. 
- One center front, two center back. 
- 100% polyester. 
- Machine wash. 
- Imported. 
- Model is 5\'10""/178 cm and has a 32"" bust. She wears a US size 2.

This example seems fairly realistic and reasonable. One thing I was happy about is that besides just generating a realistic looking description was that words like “asymmetric” and “sleeveless” which were used in the input phrase also appear in the generated description.

On a less serious note, sometimes with less fashion sounding context we can still get some of the old GPT-2 training and structure bleeding through.

For context “meat dress” inspired by Lady Gaga

With the same context meat dress below are two generated examples. When SR GPT-2 generates a sample it takes into consideration words with the highest probability of following the given context. However there is some randomness involved in the process so outputs can come out quite differently. The first example generates a fairly reasonable looking product description. The second SR GPT-2 falls back a bit into the more definition type training of GPT-2

Meat dress: our Italian twist, crafted of a crepe fabric with a stretchy, crinkle finish. Features hand-woven details, an embroidered floral pattern throughout.- Adjustable, pull-on, belt 
- Side slit 
- Adjustable, belt with cut from a relaxed fit 
- Fabric has been softened by hand washing 
- 95% rayon, 5% spandex blend; lining: 100% polyester crepe de chine 
- Washable 
- Imported
dress made of meat, bone, and vegetable gabardine. In honor of the American Heart Foundation.

StyleGAN

Generative Adversarial Networks (GAN) are an interesting area of deep learning where the training process involves two networks a generator and a discriminator. The generator model starts to create images on its own, it starts from random noise while the discriminator gives feedback by looking at training examples and generator output and predicts if they are “real” or “fake”. Overtime this feedback helps the generator create more realistic images.

StyleGAN is a model that was released by Nvidia near the end of 2018. It is an improvement over a previous model from Nvidia called ProGAN. ProGAN was trained to generate high quality images 1024×1024 and did so by implementing a progressive training cycle where it starts training images at low-resolution (4×4)and increases that resolution over time by adding additional layers. Training the low resolution images helped make training faster and increased the quality of final images as the networks were able to learn important lower level characteristics. However ProGAN has limited ability to control the generated images.

StyleGAN improves on ProGAN by giving the ability to control the “style” of outputs by allowing users to manipulate the latent space vectors of a generated image. Every image that StyleGAN generates is represented by a vector that exists within StyleGAN’s latent space. So if you modify that vector you can adjust the characteristics of the image within StyleGAN’s latent space to create a new image with desired characteristics.

This is just a brief description of StyleGAN for more information check out the paper or other write-ups on online.

ShopRunner StyleGAN (SR StyleGAN)

I ended up training SR StyleGAN for around 4 days and generated around 2 million 512×512 images in the process. As a starting point for weight initialization I actually used another anime trained StyleGAN. I used this anime StyleGAN as a starting point because the original Nvidia StyleGAN was trained to generate 1024×1024 images which are great, but also harder to work with because they require more computational firepower. The anime StyleGAN in comparison was trained to generate 512×512 images so it is more manageable.

The dataset for SR StyleGAN was around 9000 mostly dress product images which I pruned down based on a few criteria. Step 1 I did with a few lines of code, but the last three steps were manual.

1. Size: I threw out images with a width or height below 300. If you leave low quality images in the dataset you end up with pixelated looking final generated images.
2. composition: for simplicity I tried to keep images where the model/product was located in the center of the image
3. background: removed overly complex backgrounds since it would mostly just mean lots of additional effort on the model’s part to begin to generate them well
4. removed non product shots: certain images were either blank placeholder images or zoomed in shots of pattern/fabric. Leaving shots like this in the dataset I found gives StyleGAN an easy way to cheat and generate “realistic” looking images to fool the discriminator. However, this is not really the most desirable behavior so I did my best to remove them.

Generating Low Frequency Examples

Now that we have walked through some of the training details of SR StyleGAN we can start talking about how to generate those low frequency products. In the following video you see a few seconds where jumpsuits are generated even though this dataset was mostly dresses. So for this hack week I used jumpsuits as an example low frequency class.

One quick method to generate additional jumpsuit samples would be to generate a large number of SR StyleGAN images unconditionally and search through those to find examples of the low frequency class we care about.

Here are some examples that I manually pulled out of a few hundred generated StyleGAN images. This is fine? BUT if we can figure out where exactly jumpsuits exist in the SR StyleGAN latent space we could generate them as we see fit.

Style Mixing with SR StyleGAN

Each image is represented by a feature vector in SR StyleGAN’s latent space. So if we combine different vectors together we are able to start to get at “style mixing”. In the two sets of videos below what we are seeing the top right image get mapped onto the bottom left image. The resulting mixture is in the bottom right image which should be dominated by the characteristics of the top right image. In both videos you see the characteristics of the top right image shift in response to changes in the bottom left image.

This is cool, it sort of gives us a way to combine two images by combining their feature vectors in SR StyleGAN’s latent space.

Combining Image and Text Generators?

Since the goal was to generate realistic looking fake products here are two examples of generated images with contextually generated text. As of now the text context is manually generated, but a future project could be to build a captioning model or simply use tags generated by internal attribute and taxonomy models which the team has been working on. These generated tags can be used as a sort of stand in for product title. For example potential attributes of the following dress could be “sleeveless red asymmetric dress” and could be fed into GPT-2 to get contextually generated product descriptions.

Context for SR GPT-2: sleeveless red asymmetric dress

DetailsThis asymmetric fit-and-flare fit is cut in a slim fit. Cut for a flattering, relaxed fit.- Round neckline. 
- Sleeveless. 
- One center front, two center back. 
- 100% polyester. 
- Machine wash. 
- Imported. 
- Model is 5\'10""/178 cm and has a 32"" bust. She wears a US size 2.

A second example using a jumpsuit generated by SR StyleGAN with potential tags being “black short sleeve jumpsuit” which feels like a reasonable description or boring title.

SR GPT-2 context: black short sleeve jumpsuit.

A loose, fluid silhouette lends a comfortable wear to any look. The buttonless back features a keyhole on the chest.Material and CareMaterial information: 100% Cotton, Lining: 100% Viscose, Lining: 100% Polyester

Then as a fun follow up I fed these two examples through our internal taxonomy classification service which uses images and text input and found that the taxonomy service successfully categorizes the two images as a “women’s dress” and a “jumpsuit”.

Wrapping Things Up

Over the course of this hack week I spent a lot of time training models and looking at generated image and text outputs. I still think that while hard to utilize these generator style models could potentially be very useful for adding interesting business value. I mentioned initially things like synthetic data augmentation for low frequency classes, but other ideas that came from the team could be letting users generate items and manipulate the items if we can figure out how to successfully locate and manipulate different features in the GAN’s latent space. If users can generate items they would like then we can do more standard visual searches of our catalog and so on.

As for notes on the models.

GPT-2 seems fine and learns quickly, potentially overfits quickly as well… by feeding it appropriate context it can generate reasonable text. What we use as context is really the question. Thoughts would be a captioning model based on a real image or a GAN image. A simpler way would just be to feed in all available attributes and taxonomy information as plain text and see how it does.

StyleGAN is decently trained and to get better results I would likely need to train it from scratch or at least from a much earlier point in its training. I intentionally had StyleGAN start at a point where it was generating fairly large images. all while using those anime weights.

Something that I experimented with but did not find much success with was mapping images in and out of StyleGAN’s latent space. The general idea is to use a pretrained network to learn find the the closest approximation of an image in StyleGAN’s latent space by generating StyleGAN vectors and comparing how close the image is to the original. If we can successfully map items into StyleGAN’s latent space then we can combine those vectors to have a bit more control of what we are modifying. For example we could map a bunch of jumpsuits into StyleGAN’s latent space and mix those jumpsuits together to make new samples. Another related step is finding where certain attributes or patterns exist in the latent space then we could potentially

For a full training run Nvidia does list a training time of 42 days with a single GPU. You could likely get good results in a full week or two of training since other folks who have tried training from scratch report the last few weeks are really just about getting clean minor details.

If I end up continuing on this hack week idea a lot of future work will likely be around manipulating SR StyleGAN outputs and locating where things like dress/sleeve length are located or colors and patterns in order to allow for more fine grained control over manipulating different aspects of the generated images.

I’m not an underrepresented candidate or employee in most organizations. I am a white, cisgender, heterosexual, upper-middle-class man. I have experienced tremendous privilege in my life, which aided in my growth and trajectory. Having privilege doesn’t mean I haven’t worked hard to get where I am today, both in my career and in my life, but by acknowledging my privilege, I also acknowledge that the systems we all live in in this society are rigged in my favor. I don’t get defensive about it because I didn’t build these systems, but I accept that it helps me to evolve so that I operate mindfully in the greater world despite it. 

I am an ally thanks to the help of my family, friends, and colleagues. An ally is someone who is not a member of an underrepresented group but who takes action to support that group. As a recruiter, I have so many opportunities to be an ally and make an impact on my company and society as a whole. What started as a curiosity to understand lived experiences outside of my own (remember, I’m a recruiter!) has grown into a perpetual search to show up as the best coworker and person possible – asking the right questions and admitting when I don’t know something. Growing up (and even in my early professional years), I was uneducated on the subject of equity and inclusion in the workplace and the world at large. And, believe it or not, I still am. I’m a work in progress, as are all humans. I am always learning more about how I can improve as an ally and do my part to help level the playing field for everyone. Many conversations around DEI (diversity, equity, and inclusion) start from the perspective of people who fall into underrepresented categories and that is super valuable and important. Allies need to work to keep the voices of underrepresented people centered in the discussion of how to affect change. It will require people like me stepping up into our roles as allies to make real dents in the system.

ShopRunner’s focus as a Talent Team is always, above all, to hire the best candidate for the role, but we do put an emphasis on building diverse talent pools. This can take some extra effort at times but building a diverse and inclusive environment is well worth it. It’s a verifiable fact that diverse teams drive better results for businesses as a whole. So far this year, We have hired about 40 employees, and in addition to filling these roles one by one, we’ve been nurturing a talent community. These evolving talent communities are used to fill the pipelines for our open roles now and in the future. I make a conscious effort to get to know underrepresented candidates and get them in the mix with hiring managers to review and include in the interview process. We’re looking for a workplace that is reflective of the communities where we operate, as well as our consumers. This way the employees involved in building our products reflect that broader world too. 

Let me reiterate: We hire the best candidate for the role, regardless of race, gender, age, religion, gender identity, or sexual orientation. I advocate for equality throughout every step of the hiring process. Even when the offer is in negotiation we use a simple powerful tool that come in the form of a set of salary bands to help ensure we’re making the candidate a compelling and fair offer. We use these set salary bands to guide us through the offer process to make sure the entire company is aligned in regard to compensation.

And I’ve saved my most important skill for last – listening. How can I be an ally to various oppressed groups if I’m not communicating and listening to what they have to say? I come with a humble heart and a curious mind, and I’ve grown as a listener these past few years both in my personal and professional life. I’ve learned that part of being a good ally is allowing the space for others to show up authentically and wholly. To allow them to show up wholly, I listen wholly. I work to actually hear how the other perspectives are different than mine. From there, I find space to allow for that perspective to shine in real-time collaboration. 

I am writing this to share my personal story as an ally. My hope is that others see it and in turn open their hearts and minds to learning. That’s the only way we can improve as a community and beyond.

 Name: Junior Galvez
Pronouns: he/him/his
Role/Title: Customer Operations Lead

How long have you been at ShopRunner?
Since October 2018

What do you do all day?
Help educate our customers what ShopRunner is and get them excited about it!

Astrological sign. Do you believe in astrology?
Scorpio. Yes, 100%. Everything I’ve ever read about Scorpios is absolutely true.

What should people know about ShopRunner?
ShopRunner should be a stand out company known for hiring the most kind, genuine and hardworking people.

When you’re not at ShopRunner, what do you like to do?
Wander aimlessly around Manhattan. Walk for blocks and blocks until I end up back at my apartment. There’s always something to see, appreciate, and love about living in New York.

Who is your favorite SR retailer?
Yves Saint Laurent. Best boots I’ve ever owned!

What’s the last thing you bought online?
A pair of Givenchy slides. Another favorite designer of mine!

Tell us about your favorite slack channel at SR:
#chicago I find it very entertaining. Matt cracks me up.

What’s your favorite way to shop?
In person for vintage and thrift finds. Online for new staple pieces. 

Have you ever worked in retail?
I have! Selling women’s designer shoes at Nordstrom in one of the busiest outdoor shopping malls in LA, The Grove! 

Give us your most inspiring bit of advice: 
I feel like I’ve accomplished a lot in my 31 years of life. Saving up for my own apartment in Manhattan, landing a great job, and the ultimate move to New York from LA on my own not knowing a single soul or having anything lined up. In all of that I learned to shut out the noise from other people and just focus on my dreams and goals. If you believe in something enough, it will 100% come true with hard work and being good to people. I’m very happy to say that I’m proof of that. 

Name: Bartosz Leper
Pronouns: he/him
How long have you been at ShopRunner? A year. Or three and a half, if we count my tenure at Spring.

What do you do all day?
I binge. That’s probably the only suitable definition that can unambiguously cover a longer period. I can literally spend entire months doing one thing. When I get fed up, I drop it on the spot and start doing something else. I used to binge on crocheting, reading about totalitarianism, learning how to make computer games, watching all seasons of “Gilmore Girls”, writing an Atari emulator with my son… Now I’m waiting for another quarter-long writing binge to finish off that novel that I started some three years ago.

Astrological sign. Do you believe in astrology?
Scorpio. And no, I don’t. When I need a prophecy, I ask my wife; her accuracy is unparalleled. I actually like my late grandmother’s attitude towards it: she didn’t care what sign she was born under. She picked Leo, because she identified with it, and if someone didn’t like it, they had to find a problem somewhere else.

What should people know about ShopRunner?
At ShopRunner, your voice matters, whether you joined a month ago, or you have been here from the start. For example, people on my team sometimes have more expertise in online shopping than I do, and I say this without any guilt or shame; listening to them is a part of my job. We all bring our unique qualities — that’s how it works.

What’s your favorite SR retailer?
Doesn’t apply for geographical reasons.

What’s the last thing you bought online?
An SD card — to play with Linux firmware on a Lego Mindstorms set. I wanted to build a robot that would use an infrared sensor as a radar to avoid obstacles. I actually got quite far… before my robot building binge ended.

Tell us about your favorite slack channel at SR:
There’s no other channel like #catsofshoprunner.

What’s your favorite way to shop?
It really depends on what I’m trying to buy and what my mood is. Online shopping is actually a second choice for me. I go online for very specific items like electronics or books, where it really makes a difference whether you buy “Twenty Lessons from the Twentieth Century”, or “21 Lessons for the 21st Century”. If all I need are a good pair of shoes, I’d rather go to an old good brick-and-mortar store. In my opinion, e-commerce will never replace offline shopping, just like the TV didn’t replace radio, and radio didn’t replace press. Harley Finkelstein, the COO of Shopify, recently said that the brick-and-mortar shops are transforming, not dying. I trust that he knows it better than I do.

Have you ever worked in retail?
No.

Give us your most inspiring bit of advice:
Learn history, embrace politics, listen to scientists. It’s far more important whether you understand your world than how good you perform as an employee.

• What are User Defined Functions (UDF)
• Python UDF in Redshift
• JavaScript UDF in Snowflake
• Parsing user-agent string using JavaScript UDFs
• Bundle UDF in a single file using webpack
• Conclusion

What are User Defined Functions

User Defined Functions (UDF) allow us to extend the system to perform operations that are not available through the built-in functions provided by Snowflake. Snowflake currently supports two types of UDFs SQL and JavaScript.

Python UDFs in Redshift

At ShopRunner, before migrating to Snowflake, Redshift was our Data Warehouse.

AWS Redshift is a managed Data Warehouse on AWS. Redshift supports UDF in Python 2.7. In addition to Python standard library modules like Pandas, NumPy, Scikit etc are part of Redshift implementation. It also allows us to import our own modules.

Parsing User Agent String in Redshift

CREATE OR REPLACE FUNCTION f_ua( ua_string varchar(2000) )
    -- type: returns User Agent Device by parsing given UA string
    -- return: varchar(30)
RETURNS varchar(1000)
IMMUTABLE
AS $$
from ua_parser import user_agent_parser
try:
	if ua_string:
		 uas = user_agent_parser.Parse(ua_string)
		 return "%s,%s,%s" % (
		 	uas.get('device').get('family'),
		 	uas.get('os').get('family'),
		 	uas.get('user_agent').get('family')
		 )
except exception as e:
	return e.message + ':dd:cc'
return 'UNKNOWN:UNKNOWN:UNKNOWN'

$$ LANGUAGE plpythonu;

Python has built-in ua_parser library which parses the user agent string. Parsing User Agent in Python is one line code, and everything else is a boilerplate.

What is user agent string

A browser’s user agent string (UA) helps identify which browser is being used, what version, and on which operating system. The browser sends this information in the User-Agent HTTP header every time it makes a request to any site.

JavaScript UDFs in Snowflake

Here is our first JavaScript UDF, which converts strings of various format into money. We receive order amount in order events, and different retailers send us a different formats. Sometimes they are also undefined JavaScript values.

CREATE OR REPLACE FUNCTION "CONVERT_TO_MONEY"(obj VARCHAR)
RETURNS FLOAT
LANGUAGE JAVASCRIPT
AS '
 if(obj === "" || typeof obj === ''undefined'') return null;

    // parseFloat will stop when it sees a comma.  For example "1,000" becomes `1`.
    // Remove all commas to work arround this.
    const cleanObj = typeof obj === ''string'' ? obj.replace(/,/g, '''') : obj;

    var number = parseFloat(cleanObj);
    if(!isNaN(number)) return number;
    else return null;
';

UDF to parse user-agent string in Snowflake

In order to parse user-agent string in Snowflake first we need a JavaScript library which can do the job. Finding a JavaScript library was easy, but there are more challenges

• Snowflake UDF doesn’t allow importing code from an external module.
• Snowflake UDF doesn’t allow reading from external file.
• All code for a UDF must live in a single file.

Here is the library which meets all the above requirements:

• https://github.com/faisalman/ua-parser-js

JavaScript Library to identify browser, engine, OS, CPU and device type/model from user agent String.

Our first approach, simple and straight forward. Just replace placeholder in the below Snowflake Create Function statment with the JavaScript Code in https://raw.githubusercontent.com/faisalman/ua-parser-js/master/src/ua-parser.js

create or replace function temp.ua_parser("data" string)
  returns variant
  language javascript
  as $$
<<<< JAVASCRIPT CODE TO PARSE USER AGENT STRING &gt;&gt;&gt;&gt;
return UAParser(data)
$$;

Why “$$” symbol?

This is how snowflake escape the special character(s) inside the javascript code. See here for details – https://support.snowflake.net/s/question/0D50Z00007PkpgcSAB/regex-written-inside-javascript-udf-behaves-differently

Bundle UDF in a single file using webpack

What is webpack now?

Webpack is a static module bundler for modern JavaScript applications. https://webpack.js.org/concepts

What tools we need

• Node Package Manager (npm)
  • How to install – https://nodejs.org/en/download/package-manager/

Setting up the environment

Create a folder named: snowflake-useragent-parser, with the following files:

• package.json
• tsconfig.json
• webpack.config.js
• ./lib/user_agent_udf.ts
• ./bin/create_udf.js

package.json

{
  "name": "snowflake-ua-udf",
  "version": "0.0.0",
  "private": true,
  "scripts": {
    "build": "webpack --progress"
  },
  "dependencies": {
  },
  "devDependencies": {
  }
}

Install the following required npm packages

# Libraries required to build webpack.
npm install --save-dev typescript
npm install --save-dev webpack
npm install --save-dev ts-loader
npm install --save-dev webpack-cli
 
# Libraries required to parse useragent string.
npm install --save ua-parser-js
npm install --save @types/ua-parser-js

./lib/user_agent_udf.ts

import {  UAParser } from "ua-parser-js";
 
/**
 * Parse a User Agent String using the ua-parser-js library
 * @param {string} data the user agent string to parse
 * @returns {IUAParser.IResult} the parsed object
 */
export function execute(data: string): IUAParser.IResult {
    const uaParser = new UAParser(data);
 
    return uaParser.getResult();
}

tsconfig.json

{
    "include": [
        "lib/**/*"
    ],
    "compilerOptions": {
        "noImplicitAny": true,
        "target": "es5",
        "allowJs": true
    }
}

webpack.config.js

"use strict";
 
const path = require("path");
const webpack = require("webpack");
 
module.exports = {
 
    // List one for each UDF
    entry: {
        user_agent_udf: "./lib/user_agent_udf.ts"
    },
 
    module: {
        rules: [
            // For typescript UDFs
            {
                test: /\.ts$/,
                loader: "ts-loader"
            }
        ]
    },
    devtool: "source-map",
    output: {
        path: path.resolve("./dist"),
        filename: "[name].js", // This resolves to the name found in entry above.
       // Load the results of the bundle into a variable called udf
        libraryTarget: 'var',
        library: 'udf'
    }
};

webpack.config.js bundles the UDF together. It also defines an entry point in JavaScript UDF from SQL to parse the user agent string. The entry section specifies which file contains the function and in the output section it assigns the code for complete UDF to a variable called udf.

/bin/create_udf.js

const fs = require('fs');
 
let udfSchema = "dw";
let udfName = "user_agent_udf";
 
// Read from file
let udfBundle = fs.readFileSync("dist/"+udfName+".js");
 
// Create SQL
let createUdfSql = `
create or replace function ` + udfSchema + "." + udfName + `("data" string)
returns variant
language javascript
as $$
` + udfBundle + `
 
  return udf.execute(data)
$$;
`;
 
// Write SQL to file
fs.writeFileSync("dist/" + udfName + ".sql", createUdfSql);
 
console.log("Generated UDF query for "+udfName);

/bin/create_udf.js will generate the SQL for creating the Snowflake UDF and write it in ‘dist/user_agent_udf.sql’

How to build and generate the UDF SQL

./node_modules/.bin/webpack
node ./bin/create_udf.js

How to deploy UDF in snowflake

The above steps will generate the SQL file name dist/user_agent_udf.sql, which on executing on Snowflake console will creates the dw.user_agent_udf in Snowflake.

How to test UDF in snowflake

SELECT
      eaa.event_id,
      eaa.user_agent,
      dw.user_agent_udf(eaa.user_agent) AS user_agent
FROM sra_event_attrs_axis eaa
limit 10;

Conclusion

When we could have created the UDF by just doing a copy and paste, then what did we gain with extra work?

Everything that an organized software engineer wants.

• Github Repo
• Ability to repeat and recreate
• Ability to extend
• Unit Testing
• CI/CD

Name: Ali Vanderveld
Pronouns: she/her
Role/Title: Director of Data Science
How long have you been at ShopRunner? Almost a year and a half

What do you do all day?
I spend most of the day communicating in one way or another — spending time with my team members, recruiting, keeping our priorities aligned and on track, and basically making sure that all of the data science at ShopRunner is done efficiently and by a fantastic team of happy, engaged people. I also give a few conference talks per year, so I find the time to work on those. I also try to spend a little time here and there making sure that I remember what code looks like, mostly by doing code reviews for the new stuff my team builds, but I also try to do smaller technical tasks whenever I can. I have a lot of meetings but staying technically engaged keeps me sane.

Astrological sign. Do you believe in astrology? 
I am a libra, but I don’t believe that it means anything. Studying astrophysics for 12 years made sure of that!

What should people know about ShopRunner?
That we are not a “shipping company”! We are a tech company that is working on so much more to help retailers compete in the new, fast-paced e-commerce world—like building tools for our partners such as express checkout and sophisticated recommendation systems.

When you’re not at ShopRunner, what do you like to do?
On evenings and weekends I train at a Chicago circus school called Aloft. I’ve been doing aerial there for about 9 years now. I occasionally perform on aerial silks or on my personal apparatus, double ropes, usually to some sort of rock / metal music. I’m also studying Spanish, and I’m a devoted cat mom.

What’s your favorite SR retailer?
New York and Company

What’s the last thing you bought online?
Indian food, via Grubhub

Tell us about your favorite slack channel at SR:
#catsofshoprunner, no question. Pretty self-explanatory.

What’s your favorite way to shop?
I actually hate shopping! I only ever buy things when I know exactly what I need and I can find it easily on the site of one of my go-to retailers. This leads to some unintentional frugality. I still wear a hoodie that I bought 15 years ago (the strings fell apart and the tab of the zipper broke off, but it’s still good).

Have you ever worked in retail?
Yep, my first job was at a farm stand and then my second job was as a cashier at PetSmart.

Give us your most inspiring bit of advice:
Uh, wow, this one is a lot harder than asking my favorite Slack channel. I guess I would say to anyone out there who is starting out in their data science career — always keep learning new things. If you ever find yourself getting into a rut and doing the same thing over and over, then it’s probably time to move to a new project or team that’s doing stuff that’s new to you. It can be a bit scary to not know stuff (and nice and comfortable to just know stuff), but that’s how you develop the skills that you need to be successful for the rest of your career. And tech is constantly changing, so the ability to stay up-to-date is always going to be a key to success.

Name: Nate Walker
Pronouns: He/Him/His

How long have you been at ShopRunner?
I started October 22, 2018 on contract and was brought on full time in February.

What do you do all day?
As an Integration Support Specialist I get to work with a wide variety of teams and partners on issues that range from broken feeds and token validation to partner asset updates and OOS messaging for Cole Haan and Neiman Marcus.

Astrological sign. Do you believe in astrology?
I don’t necessarily believe in astrology but if I did I would be a Virgo!

What should people know about ShopRunner?
Beyond offering cool products from a variety of retailers, ShopRunner also provides employees with unique opportunities and culture that is hard to find elsewhere. Also, on Wednesday there’s bagels! Everything Bagels, Plain Bagels, Onion Bagels, Sesame Bagels.

When I’m not at ShopRunner I enjoy:
When I’m not at ShopRunner I can usually be found somewhere in Philadelphia painting. Over the past few years I’ve had the opportunity to paint several murals in the City of Brotherly Love and display paintings at various restaurants, cafes, community centers and venues in the area. More recently I’ve been painting parking bollards (polls sticking out of the sidewalk) as pencils, which has been popular amongst the neighborhood and featured on NBC Philadelphia!

What’s your favorite SR retailer?
Adorama because I can buy both musical instrument supplies and a scanner/printer for my paintings.

What’s the last thing you bought online?
Acoustic Guitar strings (light/medium) and amp cord (thanks Adorama!).

Tell us about your favorite slack channel at SR:
My two favorite channels are #catsofshoprunner and #musicloversonly because I love both cats and music.

What’s your favorite way to shop?
Since starting at ShopRunner I’ve become more inclined to shop online because of how easy it can be but previously thrift stores were my favorite way to shop. Everything I bought felt like a treasure!

Have you ever worked in retail?
When I was in high school I worked in Target as a Starbucks Barista making Americanos, Frappuccinos, and Macchiatos.

Give us your most inspiring bit of advice:
Failure is the first step to success.

In my previous lives, I had the good fortune** to work on DEI initiatives as part of my role (in addition to all of the more traditional tasks an HR person is expected to perform). Although I was thrilled to work on something that I consider a crucial mission, I was always disappointed by the fact that the problem of rampant inequity had sprouted from seeds that we, ourselves, had planted. In these roles and at those companies, we hadn’t talked about the importance of diversity and equity early in the game, and years later we were doing our best to triage the situation we had set in our foundational stages.

I’ll be the first to admit our journey has been far from perfect, but I believe it’s important to call out the positive signals we’re sending and the progress we are making. Progress in equity doesn’t come through huge announcements, but rather by adjusting the million small ways that systemic issues come together to create inequity at work.

When I came to ShopRunner, the thesis was that caring about DEI and not shying away from having real conversations about it would make a difference. We’d plant different seeds this time, starting with a cultural assertion that inclusion is an integral part of who we are.

In short two years, here are just some of the ways we’re showing up to champion inclusion while we continue to build:

1. We celebrate our women leaders.
2. We acknowledge and celebrate the identities of our non-binary employees.
3. We make what has been implicit, explicit, and through our actions, we put an end to secret parenting that can be so rampant in workplaces today.
4. We look at our policies and work hard to make them progress our business but also become table stakes offerings to the best and brightest employees.
5. We use gender-neutral language throughout our employee handbook.
6. We have used salary bands from the beginning of our hyper-growth to ensure equity in compensation across employees and departments.
7. We are committed to seeing a diverse pipeline for each and every role we hire for and always hire the best person therein.
8. We have a call out for equality at the bottom of each and every role we post.
9. We have INCLUSION as a value amongst a strong set of values we hold dear across the organization.
10. We expanded our parental leave policy to be inclusive of any way a family comes together to expand with fully paid leave for both primary and secondary caregivers.
11. We’re always on the lookout for ways we can set the stage for better equality and inclusion which means the above list is only the start.
12. We create lactation rooms where employees can feel comfortable with peace of mind that they’ll get the privacy and space they need to transition back from time away.
13. We nurture wellness through mandated PTO and a wellness room that creates a calming environment and allows employees to meditate or pray as needed.
14. We bring in harassment training to ensure that all employees are ambassadors for harassment-free workplaces. We set the stage for zero tolerance on bad behavior.

We are owning who we are and that inclusion and equity matters to us. We yearn to serve a diverse set of customers and help our entire communities thrive in each place we have an office. “You can’t be what you can’t see” and we try to give a window in wherever we can. We hope we become what people see and inspire other workplaces to champion diversity, equity and inclusion at each and every time and place possible.

We’re proud of what we’ve built and it’s only the beginning. The foundation is strong and we’re continuing to be transparent about who we are, proud of showing up authentically and leading with our values out in front, fueling our business’ success and creating a workplace we love.

*it isn’t that radical

**this is sarcasm as it shouldn’t be “good fortune” but rather the way things are, everywhere

Written by: Lindsay Verstegen, Chief People Officer

Gender balance is essential for economies and communities to thrive.”

Celebrating everyone’s accomplishments is a daily occurrence at ShopRunner, but on March 8th, we highlighted accomplishments and encouraged shoutouts on our brag boards across all 3 offices.