MATLAB ... and more ...

my-alpine and docker-compose.yml

2023-06-03T21:41:00.003-07:00

```

version: '1'

services:
    man:
      build: .
      image: my-alpine:latest

```

Dockerfile:

```

FROM alpine:latest
ENV PYTHONUNBUFFERED=1
RUN apk add --update --no-cache python3 && ln -sf python3 /usr/bin/python
RUN python3 -m ensurepip
RUN pip3 install --no-cache --upgrade pip setuptools
RUN apk add git
```

A Kotlin recursive function

2023-05-23T13:29:00.000-07:00

import kotlin.reflect.full.memberProperties

data class Abc(

val a: String,

val b: String,

val c: String,

val d: D,

val e: E

) {

data class D(

val e: String,

val f: String,

val g: String

)

data class E(

val h: List<String>

)

}

data class Xyz(

val a: String,

val b: String,

val c: String,

val d: D,

val e: E

) {

data class D(

val e: String,

val f: String,

val g: G

){

data class G(

val h: String,

val i: String

)

}

data class E(

val h: List<String>

)

}

fun hasContent(inVal: Any): Boolean {

if (inVal::class.simpleName == "String") {

if (inVal != "") {

return true

}

} else {

inVal::class.memberProperties.forEach { ele ->

if (ele.returnType.classifier == String::class) {

if (ele.call(inVal) != "") {

return true

}

else if (ele.returnType.classifier == List::class) {

val list = ele.call(inVal) as List<*>

val listCheck = list.map { hasContent(it!!) }

if (listCheck.contains(true)) {

return true

}

else {

if (hasContent(ele.call(inVal)!!)) {

return true

}

return false

}

return false

}

fun main() {

// aa has content, should print true

val aa = Abc("a1", "b2", "c3", Abc.D("e4", "f5", "g6"), Abc.E(listOf("h7", "i8", "j9")))

println(" aa has content -> ${hasContent(aa)}")

// bb has no content, should print false

val bb = Abc("", "", "", Abc.D("", "", ""), Abc.E(listOf("", "", "")))

println(" bb has content -> ${hasContent(bb)}")

// cc has content, should print true

val cc = Abc("", "", "", Abc.D("e4", "", ""), Abc.E(listOf("", "", "j9")))

println(" cc has content -> ${hasContent(cc)}")

// dd has content, should print true

val dd = Abc("", "", "", Abc.D("", "", ""), Abc.E(listOf("", "", "j9")))

println(" dd has content -> ${hasContent(dd)}")

// xx has no content, should print false.

val xx = Xyz("", "", "", Xyz.D("", "", Xyz.D.G("", "")), Xyz.E(listOf("", "", "")))

println(" xx has content -> ${hasContent(xx)}")

// yy has content, should print true.

val yy = Xyz("", "", "", Xyz.D("", "", Xyz.D.G("", "")), Xyz.E(listOf("", "", "blahblah")))

println(" yy has content -> ${hasContent(yy)}")

// zz has content, should print true.

val zz = Xyz("", "", "", Xyz.D("", "", Xyz.D.G("", "blahblah")), Xyz.E(listOf("", "", "")))

println(" zz has content -> ${hasContent(zz)}")

}

A nice alias to check the resources used be each user on a Linux server

2022-06-29T10:28:00.002-07:00

Sometimes on a shared Linux server or AWS EC2, we found out the resources are limited because some other persons are using the machine, and we want to rank order the users by resources used from high to low. This alias does exactly this.

```

alias wtf="ps hax -o rss,user | awk '{a[\$2]+=\$1;}END{for(i in a)print i\" \"int(a[i]/1024+0.5);}' | sort -rnk2"

```

AWK: count which customer has the most unique accounts

2021-10-22T14:19:00.003-07:00

Content of the count_uniq.awk script:

BEGIN {
    FS=",";
}
{
    if ($0 in seen == 0){arr[$2]++} else {seen[$0]=1}
    seen[$0]=1;
}
END {
    for (a in arr) print a, arr[a]
}

What it does?

count how many accounts does each customer has, ignoring duplicated rows. Details as in image below:

Customer "a" has 3 different accounts: [1, 2, 3]

Customer "b" has 1 account: [2]

Customer "c" has 2 accounts: [1, 2]

Concatenate multiple files with same headers (and only keep one header line in the output file) - Version2

2021-09-30T09:04:00.000-07:00

Put this into an AWK script and call it merge.awk:

BEGIN{

h=ARGV[1];

ARGV[1]="";

}

{

if(FNR==1){print}

else if($1!~h){print}

}

and use an argument:

cat *.csv | awk -f merge.awk "<header>" > output.csv

git: find string in a certain branch

2021-09-08T12:56:00.001-07:00

So, today I cloned a repo and it has many branches. I know one of the branches has a file that is updated and contains the word "token". This is how I find out which branch it is:

git branch -a | cut -c3- | cut -d' ' -f 1 | xargs git grep "token"

cron.allow

2021-08-12T08:17:00.006-07:00

If a user on Linux can't run "crontab -l" or "corntab -e", add the user name to /etc/cron.allow, and then it should work.

WOE

2021-08-11T23:01:00.006-07:00

#http://lazynight.me/4048.html

import math

import numpy as np
from scipy import stats
from sklearn.utils.multiclass import type_of_target


class WOE(object):
    def __init__(self):
        self._WOE_MIN = -20
        self._WOE_MAX = 20

    def processing(self, X, y, event=1):
        self.check_target_binary(y)
        X1 = self.feature_discretion(X)

        res_woe = []
        res_iv = []
        for i in range(0, X1.shape[-1]):
            x = X1[:, i]
            woe_dict, iv1 = self.woe_single_x(x, y, event)
            res_woe.append(woe_dict)
            res_iv.append(iv1)
        return np.array(res_woe), np.array(res_iv)

    def woe_single_x(self, x, y, event=1):
        self.check_target_binary(y)

        event_total, non_event_total = self.count_binary(y, event=event)
        x_labels = np.unique(x)
        woe_dict = {}
        iv = 0
        for x1 in x_labels:
            y1 = y[np.where(x == x1)[0]]
            event_count, non_event_count = self.count_binary(y1, event=event)
            rate_event = 1.0 * event_count / event_total
            rate_non_event = 1.0 * non_event_count / non_event_total
            if rate_event == 0:
                woe1 = self._WOE_MIN
            elif rate_non_event == 0:
                woe1 = self._WOE_MAX
            else:
                woe1 = math.log(rate_event / rate_non_event)
            woe_dict[x1] = woe1
            iv += (rate_event - rate_non_event) * woe1
        return woe_dict, iv

    def woe_replace(self, X, woe_arr):
        if X.shape[-1] != woe_arr.shape[-1]:
            raise ValueError('WOE dict array length must be equal with features length')

        res = np.copy(X).astype(float)
        idx = 0
        for woe_dict in woe_arr:
            for k in woe_dict.keys():
                woe = woe_dict[k]
                res[:, idx][np.where(res[:, idx] == k)[0]] = woe * 1.0
            idx += 1

        return res

    def combined_iv(self, X, y, masks, event=1):
        if masks.shape[-1] != X.shape[-1]:
            raise ValueError('Masks array length must be equal with features length')

        x = X[:, np.where(masks == 1)[0]]
        tmp = []
        for i in range(x.shape[0]):
            tmp.append(self.combine(x[i, :]))

        dumy = np.array(tmp)
        # dumy_labels = np.unique(dumy)
        woe, iv = self.woe_single_x(dumy, y, event)
        return woe, iv

    def combine(self, list):
        res = ''
        for item in list:
            res += str(item)
        return res

    def count_binary(self, a, event=1):
        event_count = (a == event).sum()
        non_event_count = a.shape[-1] - event_count
        return event_count, non_event_count

    def check_target_binary(self, y):
        y_type = type_of_target(y)
        if y_type not in ['binary']:
            raise ValueError('Label type must be binary')

    def feature_discretion(self, X):
        temp = []
        for i in range(0, X.shape[-1]):
            x = X[:, i]
            x_type = type_of_target(x)
            if x_type == 'continuous':
                x1 = self.discrete(x)
                temp.append(x1)
            else:
                temp.append(x)
        return np.array(temp).T

    def discrete(self, x):
        res = np.array([0] * x.shape[-1], dtype=int)
        for i in range(5):
            point1 = stats.scoreatpercentile(x, i * 20)
            point2 = stats.scoreatpercentile(x, (i + 1) * 20)
            x1 = x[np.where((x >= point1) & (x <= point2))]
            mask = np.in1d(x, x1)
            res[mask] = (i + 1)
        return res

    @property
    def WOE_MIN(self):
        return self._WOE_MIN

    @WOE_MIN.setter
    def WOE_MIN(self, woe_min):
        self._WOE_MIN = woe_min

    @property
    def WOE_MAX(self):
        return self._WOE_MAX

    @WOE_MAX.setter
    def WOE_MAX(self, woe_max):
        self._WOE_MAX = woe_max

Street suffix abbreviations

2021-08-11T22:37:00.004-07:00

# Street suffix abbreviations

# https://pe.usps.com/text/pub28/28apc_002.htm

addressStandardMap = {'ALLEE': 'ALLEY',

'ALLY' : 'ALLEY',

'ALY': 'ALLEY',

'ANEX': 'ANNEX',

'ANNX': 'ANNEX',

'ANX': 'ANNEX',

'ARC': 'ARCADE',

'AV': 'AVENUE',

'AVE': 'AVENUE',

'AVEN': 'AVENUE',

'AVENU': 'AVENUE',

'AVN': 'AVENUE',

'AVNUE': 'AVENUE',

'BYU': 'BAYOU',

'BAYOO': 'BAYOU',

'BND': 'BEND',

'BLF': 'BLUFF',

'BLUF': 'BLUFF',

'BLFS': 'BLUFFS',

'BTM': 'BOTTOM',

'BOT': 'BOTTOM',

'BOTTM': 'BOTTOM',

'BLVD': 'BOULEVARD',

'BOUL': 'BOULEVARD',

'BOULV': 'BOULEVARD',

'BR': 'BRANCH',

'BRNCH': 'BRANCH',

'BRDGE': 'BRIDGE',

'BRG': 'BRIDGE',

'BRK': 'BROOK',

'BRKS': 'BROOKS',

'BG': 'BURG',

'BGS': 'BURGS',

'BYP': 'BYPASS',

'BYPA': 'BYPASS',

'BYPAS': 'BYPASS',

'BYPS': 'BYPASS',

'CP': 'CAMP',

'CMP': 'CAMP',

'CANYN': 'CANYON',

'CNYN': 'CANYON',

'CPE': 'CAPE',

'CSWY': 'CAUSEWAY',

'CAUSWA': 'CAUSEWAY',

'CTR': 'CENTER',

'CEN': 'CENTER',

'CENT': 'CENTER',

'CENTR': 'CENTER',

'CENTRE': 'CENTER',

'CNTER': 'CENTER',

'CNTR': 'CENTER',

'CTRS': 'CENTERS',

'CIR': 'CIRCLE',

'CIRC': 'CIRCLE',

'CIRCL': 'CIRCLE',

'CRCL': 'CIRCLE',

'CIRS': 'CIRCLES',

'CLF': 'CLIFF',

'CLFS': 'CLIFFS',

'CLB': 'CLB',

'CMN': 'COMMMON',

'CMNS': 'COMMONS',

'COR': 'CORNER',

'CORS': 'CORNERS',

'CRSE': 'COURSE',

'CT': 'COURT',

'CTS': 'COURTS',

'CV': 'COVE',

'CVS': 'COVES',

'CRK': 'CREEK',

'XING': 'CROSSING',

'CRSSNG': 'CROSSING',

'XRD': 'CROSSROAD',

'XRDS': 'CROSSROADS',

'CURV': 'CURVE',

'DL': 'DALE',

'DM': 'DAM',

'DV': 'DIVIDE',

'DIV': 'DIVIDE',

'DVD': 'DIVIDE',

'DR': 'DRIVE',

'DRIV': 'DRIVE',

'DRV': 'DRIVE',

'DRS': 'DRIVES',

'EST': 'ESTATE',

'ESTS': 'ESTATES',

'EXPY': 'EXPRESSWAY',

'EXP': 'EXPRESSWAY',

'EXPR': 'EXPRESSWAY',

'EXPW': 'EXPRESSWAY',

'EXPRESS': 'EXPRESSWAY',

'EXT': 'EXTENSION',

'EXTN': 'EXTENSION',

'EXTNSN': 'EXTENSION',

'EXTS': 'EXTENSIONS',

'FLS': 'FALLS',

'FRY': 'FERRY',

'FRRY': 'FERRY',

'FLD': 'FIELD',

'FLDS': 'FIELDS',

'FLT': 'FLAT',

'FLTS': 'FLATS',

'FRD': 'FORD',

'FRDS': 'FORDS',

'FRST': 'FORESTS',

'FOEREST': 'FORESTS',

'FRG': 'FORGE',

'FORG': 'FORGE',

'FRGS': 'FORGES',

'FRK': 'FORK',

'FRKS': 'FORKS',

'FT': 'FORT',

'FRT': 'FORT',

'FWY': 'FREEWAY',

'FREEWY': 'FREEWAY',

'FRWY': 'FREEWAY',

'FRWAY': 'FREEWAY',

'GDN': 'GARDEN',

'GARDN': 'GARDEN',

'GRDN': 'GARDEN',

'GDNS': 'GARDENS',

'GRDNS': 'GARDENS',

'GATEWY': 'GATEWAY',

'GATWAY': 'GATEWAY',

'GTEWAY': 'GATEWAY',

'GTWY': 'GATEWAY',

'GLN': 'GLEN',

'GLNS': 'GLENS',

'GRN': 'GREEN',

'GRNS': 'GREENS',

'GRV': 'GROVE',

'GROV': 'GROVE',

'GRVS': 'GROVES',

'HBR': 'HARBOR',

'HARBR': 'HARBOR',

'HARB': 'HARBOR',

'HRBOR': 'HARBOR',

'HBRS': 'HARBORS',

'HVN': 'HAVEN',

'HTS': 'HEIGHTS',

'HT': 'HEIGHTS',

'HWY': 'HIGHWAY',

'HIGHWY': 'HIGHWAY',

'HIWAY': 'HIGHWAY',

'HWAY': 'HIGHWAY',

'HIWY': 'HIGHWAY',

'HL': 'HILL',

'HLS': 'HILLS',

'HOLW': 'HOLLOWS',

'HLLW': 'HOLLOWS',

'HOLLOW': 'HOLLOWS',

'HOLWS': 'HOLLOWS',

'INLT': 'INLET',

'IS': 'ISLAND',

'ISLND': 'ISLAND',

'ISS': 'ISLANDS',

'ISLNDS': 'ISLANDS',

'ISLE': 'ISLES',

'JCT': 'JUNCTION',

'JCTION': 'JUNCTION',

'JCTN': 'JUNCTION',

'JUNCTN': 'JUNCTION',

'JUNCTON': 'JUNCTION',

'JCTS': 'JUNCTIONS',

'JCTNS': 'JUNCTIONS',

'KY': 'KEY',

'KYS': 'KEYS',

'KNL': 'KNOLL',

'KNOL': 'KNOLL',

'KNLS': 'KNOLLS',

'LK': 'LAKE',

'LKS': 'LAKES',

'LNDG': 'LANDING',

'LNDNG': 'LANDING',

'LN': 'LANE',

'LGT': 'LIGHT',

'LGTS': 'LIGHTS',

'LF': 'LOAF',

'LCK': 'LOCK',

'LCKS': 'LOCKS',

'LDG': 'LODGE',

'LDGE': 'LODGE',

'LODG': 'LODGE',

'LOOP': 'LOOPS',

'MNR': 'MANOR',

'MNRS': 'MANORS',

'MDW': 'MEADOW',

'MDWS': 'MEADOWS',

'MEDOWS': 'MEADOWS',

'MIL': 'MILL',

'MLS': 'MILLS',

'MSN': 'MISSION',

'MSSN': 'MISSION',

'MISSN': 'MISSION',

'MTWY': 'MOTORWAY',

'MT': 'MOUNT',

'MNT': 'MOUNT',

'NCK': 'NECK',

'ORCH': 'ORCHARD',

'ORCHRD': 'ORCHARD',

'OVL': 'OVAL',

'OPAS': 'OVERPASS',

'PRK': 'PARKS',

'PARK': 'PARKS',

'PKWY': 'PARKWAYS',

'PARKWY': 'PARKWAYS',

'PKWAY': 'PARKWAYS',

'PKY': 'PARKWAYS',

'PKWYS': 'PARKWAYS',

'PSGE': 'PASSAGE',

'PATH': 'PATHS',

'PIKE': 'PIKES',

'PNE': 'PINE',

'PNES': 'PINES',

'PL': 'PLACE',

'PLN': 'PLAIN',

'PLNS': 'PLAINS',

'PLZ': 'PLAZA',

'PLZA': 'PLAZA',

'PT': 'POINT',

'PTS': 'POINTS',

'PRTS': 'PORTS',

'PR': 'PRAIRIE',

'RADL': 'RADIAL',

'RAD': 'RADIAL',

'RADIEL': 'RADIAL',

'RNCH': 'RANCH',

'RANCHES': 'RANCH',

'RNCHS': 'RANCH',

'RPD': 'RAPID',

'RPDS': 'RAPIDS',

'RST': 'REST',

'RDG': 'RIDGE',

'RDGE': 'RIDGE',

'RDGS': 'RIDGES',

'RIV': 'RIVER',

'RVR': 'RIVER',

'RIVR': 'RIVER',

'RD': 'ROAD',

'RDS': 'ROADS',

'RTE': 'ROUTE',

'SHL': 'SHOAL',

'SHLS': 'SHOALS',

'SHR': 'SHORE',

'SHOAR': 'SHORE',

'SHRS': 'SHORES',

'SHOARS': 'SHORES',

'SKWY': 'SKYWAY',

'SPG': 'SPRING',

'SPNG': 'SPRING',

'SPRNG': 'SPRING',

'SPGS': 'SPRINGS',

'SPNGS': 'SPRINGS',

'SPRNGS': 'SPRINGS',

'SPURS': 'SPUR',

'SQ': 'SQUARE',

'SQR': 'SQUARE',

'SQRE': 'SQUARE',

'SQU': 'SQUARE',

'SQS': 'SQUARES',

'STA': 'STATION',

'STATN': 'STATION',

'STN': 'STATION',

'STRA': 'STRAVENUE',

'STRAV': 'STRAVENUE',

'STRAVEN': 'STRAVENUE',

'STRAVN': 'STRAVENUE',

'STRVN': 'STRAVENUE',

'STRVNUE': 'STRAVENUE',

'STRM': 'STREAM',

'STREME': 'STREAM',

'ST': 'STREET',

'STRT': 'STREET',

'STR': 'STREET',

'STS': 'STREETS',

'SMT': 'SUMMIT',

'SUMIT': 'SUMMIT',

'SUMITT': 'SUMMIT',

'TER': 'TERRACE',

'TERR': 'TERRACE',

'TRWY': 'THROUGHWAY',

'TRCE': 'TRACE',

'TRACES': 'TRACE',

'TRAK': 'TRACKS',

'TRACK': 'TRACKS',

'TRKS': 'TRACKS',

'TRK': 'TRACKS',

'TRFY': 'TRAFFICWAY',

'TRL': 'TRAILS',

'TRAIL': 'TRAILS',

'TRLS': 'TRAILS',

'TRLR': 'TRAILER',

'TRLRS': 'TRAILER',

'TUNL': 'TUNNUEL',

'TUNEL': 'TUNNUEL',

'TUNLS': 'TUNNUEL',

'TUNNELS': 'TUNNUEL',

'TUNNL': 'TUNNUEL',

'TPKE': 'TURNPIKE',

'TRNPK': 'TURNPIKE',

'UPAS': 'UNDERPASS',

'UN': 'UNION',

'UNS': 'UNIONS',

'VLY': 'VALLEY',

'VALLY': 'VALLEY',

'VLLY': 'VALLEY',

'VLYS': 'LAVVELYS',

'VIA': 'VIADUCT',

'VDCT': 'VIADUCT',

'VW': 'VIEW',

'VWS': 'VIEWS',

'VLG': 'VILLAGE',

'VILL': 'VILLAGE',

'VILLAG': 'VILLAGE',

'VILLG': 'VILLAGE',

'VILLIAGE': 'VILLAGE',

'VLGS': 'VILLAGES',

'VL': 'VILLE',

'VIS': 'VISTA',

'VIST': 'VISTA',

'VST': 'VISTA',

'VSTA': 'VISTA',

'WALK': 'WALKS',

'WY': 'WAY',

'WL': 'WELL',

'WLS': 'WELLS'}

some bash alias I found very useful

2021-08-09T10:41:00.003-07:00

# User specific aliases and functions

alias showheader='head -1|tr "," "\n" | tr "|" "\n" |tr "^" "\n" | nl -ba | less'

alias pnfc='awk -F"," "{print NF}"'

alias pnfp='awk -F"|" "{print NF}"'

alias mytop='top -u <username>'

alias cl='clear'

alias head5='head -5'

alias head2='head -2'

alias head1='head -1'

alias lsht='ls -lsh --sort=time'

alias lshs='ls -lsh --sort=size'

alias savehist='history | tail -2 | head -1 >> NOTES'

alias x='chmod +x'

alias ssactivate='source activate <env name>'

alias cls='clear'

alias lsl='ls -alh --color=auto'

alias du='du -h'

alias df='df -h'

alias jnotebook='jupyter notebook --port 8890 --no-browser'

alias screenkillall='screen -ls | grep pts | cut -d. -f1 | awk "{print $1}" | xargs kill'

alias screenkilldetached='screen -ls | grep Detached | cut -d. -f1 | awk "{print $1}" | xargs kill'

alias savedate='echo ~~~~~~~~~~~~~~~~~~~~$(date)~~~~~~~~~~~~~~~~~~~~ >> NOTES'

dec(){

openssl des3 -in "$1" -d -pass pass:<password>

}

export -f dec

rename files within a directory one by one, using for loop and mv

2021-03-16T23:42:00.005-07:00

for f in dir/*; do
    mv -- "$f" "$fSuffix" 
done

So all the files in the dir/ will be renamed:

file1.txt --> file1.txtSuffix

file2.png --> file2.pngSuffix

Copy s3 files selected by dates

2021-03-16T14:39:00.007-07:00

$ aws s3 ls s3://<bucket-name> --recursive | awk '/^2018-11-06/{system("aws s3 cp s3://<bucket-name>/$4 .") }'

Building a XGBoost model

2020-03-03T13:54:00.003-08:00

1. pre-process :

1.1 Boruta to clean data column-wise
1.2 TomekLinks to clean data row-wise

2. training (use a random sample to do this step if the data size is too big):

2.1 create a basic with fixed learning rate and n_estimators:
XGBRFClassifier(objective='binary:logistic',
n_estimator=X,
learning_rate=0.1,
n_jobs=-1)

2.2 grid search for optimal 'max_depth' and 'min_child_weight'.
-- use scoring='roc_auc'
-- use cv = RepeatedStratifiedKFold(n_splits=3, n_repeats=3)

2.3 get 'current_best' by using gridsearch.best_estimator, then fit current_best again.

2.4 then grid search for 'gamma' with current_best, when it's done, get 'current_best' by using gridsearch.best_estimator, then fit current_best again.

2.5 then grid search for 'subsample' and 'colsample_bytree', when it's done, get 'current_best' by using gridsearch.best_estimator, then fit current_best again.

2.6 then grid search for 'learning_rate' , when it's done, get 'current_best' by using gridsearch.best_estimator, then fit current_best again.

3. final training: use all data to fit the mdl with all the optimized params.

4. Evaluatiion.

Assign bin labels to new values during model inference

2020-02-28T16:04:00.000-08:00

In model development:

import pandas as pd
import numpy as np
np.random.seed(42)

bins = [0, 10, 15, 20, 25, 30, np.inf]
labels = bins[1:]
ages = list(range(5, 90, 5))
df = pd.DataFrame({"user_age": ages})
df["user_age_bin"] = pd.cut(df["user_age"], bins=bins, labels=False)

# sort by age
print(df.sort_values('user_age'))

In production, I will need to put individual age values to its corresponding bins. Here's how to do it:

# a new age value
new_age=30

# use this right=True and '-1' trick to make the bins match
print(np.digitize(new_age, bins=bins, right=True) -1)

speed up loading local csv file into AWS RDS MySQL database

2019-06-14T17:27:00.000-07:00

tricks I learned today:
1. use 'LOAD LOCAL INFILE'
2. 'SET AUTOCOMMIT=0' - and manually commit at the end.

Python function: check if an object is an float

2019-03-15T11:57:00.002-07:00

def isfloat(value):
  try:
    float(value)
    return True
  except ValueError:
    return False

Python function: format dollars

2019-03-14T17:15:00.000-07:00

def format_dollar(s):
      """takes in a str or a number and format it as dollar format
      i.e. u'24567.0' --> u'$24,567'
     """

     s = str(s) # in case input is not string

     try:
     i = int(s.split('.')[0])
     output = "$" + "{:,}".format(i)
     except:
     output = s

     return output

AWK: single quote eche line and add comma in the end

2019-02-05T10:40:00.003-08:00

File.csv looks like:

line1
line2
line3

Use:

cat file.csv | awk -v a="'" '{print a$0a ","}'

to make it look like:

'line1',
'line2',
'line3',

Python: Notes on Fluent Python

2019-01-10T15:54:00.000-08:00

1.

2. List comprehension

a = [['-'] * 3 for i in range(3)]

b = [['-']*3] *3

What is the difference between a and b?

3. Inplace method

Inplace method returns None and does not create a new object. For example:

lst = [5,4,3,2,1]
lst.sort() # return None

4. Sort a list of strings by length

fruits = ['apple', 'grape', 'orange', 'banaba', 'dragon fruit']
sorted(fruits, key=len)

5. recursion

def factorial(n):
return 1 if n<2 else="" factorial="" n-1="" n="" p="">print(factorial(5))

6. from operator import itemgetter, attrgetter, methodcaller

Pandas: groupby and find the most frequent item

2018-12-31T09:07:00.001-08:00

Say I have this dataframe:

order_id | class
1 | furniture
2 | book
2 | furniture
2 | book
3 | auto
3 | auto
3 | electronics
3 | pet

and to get the most frequent class of each order:

df.groupby('order_id').agg({'order_id': lambda x: x.value_counts().index[0]})

Make a simple heatmap in R with ggplot2

2018-11-14T14:22:00.001-08:00

So today I got a file that look like this:

And here's the end result of the heat map:

(Notice that the order of the Name in the chart is not the same as that in the dataframe.)

Here's the code I used to make this plot:

rm(list=ls())
library(ggplot2)

df = read.csv('fakedata.csv')

# reshape the dataframe
df.m = melt(df, id.vars = 'Name')

ggplot(df.m, aes(variable, Name)) +
geom_tile(aes(fill = value),
colour = "white") +
scale_fill_gradient(low = 'white',
high = 'blue4')

This is how the df.m look like:

convert a sqlite3 query result table into a pandas dataframe

2018-10-30T14:49:00.001-07:00

The sqlite3 query returns a list of tuples. Here's a good way to convert the list of tuples into a pandas data frame.

def q():
sql_c = " some query commands here"
df = pd.DataFrame(dbc.execute(sql_c).fetchall())

# get all column names
cns = [d[0] for d in dbc.description]
df.columns = cns

return df

DataFrame create multiple new columns by applying a function that returns multiple

2018-10-27T23:08:00.003-07:00

df[['sum', 'difference']] = df.apply(
    lambda row: add_subtract_list(row['a'], row['b']), axis=1)

vundle for vim plugin, use this one!

2018-10-09T14:04:00.001-07:00

https://github.com/vim-scripts/vundle

5 available pane layouts when using tmuxp

2018-09-17T09:18:00.000-07:00

tmuxp pane layout options:

1. even-horizontal

2. even-vertical

3. main-horizontal

4. main-vertical

5. tiled