Hanxue and IT

TCM Qigong Research Areas

2025-03-21T22:21:00.002+08:00

TCM Traditional Qigong is linked with other research/scientific areas

Physiology and Metabolism Research

Focal Point: Impact of Qigong on Autonomic Nervous System as well as metabolic indicators.

Research Examples:

2021 research published in Frontiers of Physiology shows that long term Taijiquan practice reduces sympathetic (nervous system) activity, increasing parasympathetic nervous activities (heart rate variability, HRV), reducing metabolic syndromes resulting from chronic stress (e.g. insulin resistance)
Research technique: combine wearable devices (e.g. Empatica E4) for real-time heart rate and galvanic skin response (electric conductivity)

Neural Plasticity and Brain Function

Research Breakthrough: How qigong affect's the brain's Default Mode Network (DMN)

Research Examples:

In 2022, Beijing University uses fMRI proves that Baduanjin practitioners show increased functional connectivity during the brain's resting state, prefrontal cortex increases limbic system coordination. These are related to emotional regulation functions.
Technology used: Deep Learning MRI

Immune System Regulation and Epigenetics

New development: Qigong's impact on immune cell epigenesis (e.g. DNA methylation)

America's NIH research grant NCT03925311 shows that after 6 months of Qigong practice by breast cancer patients, CD4+ T cell's FOXP3 gene methylation level significantly reduced, resulting in immunosuppresion relief.
Methodology: Whole Gene Bisulfite Sequencing (WGBS) combined flow cytometry

Relationship between Biological Science and Energy Medicine

Controversy and Exploration: Exploring the physical aspects of Qigong's "External Qi"

Research Cases

Shanghai Jiao Tong University research team using SQUID (Superconducting quantum interference) probing a Qigong master releasing Qi showing that there is an increase of low-frequency electromagnetic field (0.1-30Hz) by 10-100 times compared to a regular person. The frequency characteristics is similar to physiological rhythm.
Challenge: need to exclude environmental interference (for example: geomagnetic interference), there is some dispute on research reproductibility

How to quickly extract all photos at the original quality from powerpoint slides

2025-02-27T20:57:00.001+08:00

I want to extract a lot of useful imaging photos from a Powerpoint slide deck, and the usual way to select each photo and save them has two distinct disadvantages:

time consuming and repetitive
photos are not saved as the original quality

Modern Powerpoint files with the .pptx extension are actually zip archives. Rename the .pptx file into .zip file, and look into the media folder.

If it's an older .ppt Powerpoint file, first open up the file in Powerpoint, then save as .pptx

Using Windows Explorer, browse to the location of the Powerpoint file, and rename the .pptx file to .zip

Using Windows Explorer or a File archiver utility such as PeaZip, extract the content of the newly named .zip file

You should now see a ppt folder, with a media subfolder containing all the photos in the slides.

Resignation Letter - May 2020

2019-09-24T23:23:00.002+08:00

Dear Manager,

I hereby tender my resignation for OrgX effective on 17th May 2020.

It has been my great pleasure to serve in OrgX for a year. Since day 1, I realized that OrgX is different from other companies. This is a product-focused organization that values engineering, rather than trying to please Wall Street at all cost every quarter. There is great camaraderie and belief that we will continue to lead the industry.

However, when I look back at my own contribution to OrgX and to the tech industry at the whole, I realize that I have failed to make any positive contributions. Whilst I am a valued team member and perform well according to our KPI, I have not been able to make any significant contribution to increase our effectiveness and productivity by pushing the boundary of what is possible.

In short, I have become a cog in the wheel treating symptoms rather than using my ability to think and solve problems faced both by OrgX and others in my industry.

To carry on means I am settling. That I am in Day Two. I am no longer the highly motivated engineer who wants to create the best products for the world, I no longer care about taking away friction in my colleagues' day-to-day work. I no longer dare to crest the next wave and clamber the next mountain, and would rather cross my arms and look back in satisfaction.

A Ship in Harbor Is Safe, But that Is Not What Ships Are Built For.

I sincerely hope that whomever is replacing my role will have the vision and drive to bring OrgX to greater heights.

Best regards,
Lee Hanxue

I write this resignation letter not because I plan to quit from my current organization in one year's time. Neither am I writing this letter to get some negotiating leverage. I write it because a couple of years ago, I realized that life is finite. Anything broken can be fixed, but time wasted cannot be regained. I have never heard of anyone on their death bed regretting not doubling their net worth. But I have always heard of regrets of not taking the shot, giving up their childhood or lifelong dreams, not being kinder, not spending more time with their loved ones.

I do not want to work just to pay bills, or to keep up with the Jones-es. There are many options in life to pay bills, to feed the family and so on. And I do not want to squander the limited time I have to please my bosses and wait for my monthly paychecks.

Even if I fail, I would have the satisfaction of putting all my heart and soul to solve problems in my industry, and make the world just that little bit better.

These do not need to be novel creations but could be as mundane as:

shortening the product life-cycle by 30%
implementing a continuous delivery pipeline for AI inferencing solutions
preventing certain types of failures from occurring
creating a higher level of abstraction for engineering teams, resulting in >10% increase in productivity
implementing AI inferencing framework at hardware level (e.g. Tensorflow on FPGA)
implementing TFX
higher AI workflow utilization (e.g. slurm)

I had the opportunity to work in one of the best company in the world. My regret was not about quitting my job, but not quitting early because as good as the company was overall, business changes means there are no longer opportunities for me to push the boundary of technology to increase more business value. I do not wish to repeat the same mistake twice.

Comprehensive Guide for proxy settings for Chrome

2019-08-12T12:53:00.001+08:00

If you work in a large enterprise or use chrome in a school, chances are Google Chrome access the Internet through a proxy server.

What is a proxy server? It is simply an intermediary between two machines that want to communicate with each other: Google Chrome on your laptop and the web server hosting websites on the Internet. The more important question is, why use a proxy server? Some of the reasons are:

Security: only safe websites are allowed and unsafe or inappropriate websites are blocked
Limit access: only devices that are allowed to have Internet access can connect to the proxy server
Accountability: know who is accessing what resources
Speed: proxy servers used in conjunction with caching server can speed up access

Chrome Proxy Settings

Once upon a time, setting proxy server settings is straightforward. For example, in Internet explorer, go to Internet Options and insert the proxy address.

This used to be the case for Chrome, until it changed to using the Operating Systems' proxy settings

So Chrome does not automatically set the proxy, but you use your Operating System's proxy settings.

Windows Proxy Settings

Windows 10 has a new proxy Settings.

But the legacy Internet Options settings is still valid - check that nothing is set there.

Windows also reads the HTTP_PROXY and HTTPS_PROXY environment variable. If you find that you have no proxy settings enabled but Chrome is still acting weird, check for these environment variables.

macOS Proxy Settings

You can set proxy server details in System Preferences

You can set the proxy settings from command line too


networksetup -setwebproxy "Wi-fi" 127.0.0.1 8080
networksetup -setwebproxystate "Wi-fi" on
networksetup -getwebproxy "Wi-Fi"

And for SOCKS proxy


networksetup -setsocksfirewallproxy wi-fi localhost 1080
networksetup -setsocksfirewallproxystate "Wi-Fi" on
networksetup -getsocksfirewallproxystate "Wi-Fi"

Linux Proxy Settings

The best way to set proxy is to use environment variables, so that the settings is applied to all applications, including those started from the command line


http_proxy=http://myproxy.server.com:8080/
https_proxy=http://myproxy.server.com:8080/
all_proxy=http://myproxy.server.com:8080/
no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"

GNOME Proxy Settings

If you are using a Linux distribution that is running GNOME (for example: Ubuntu, CentOS and RHEL), the proxy settings is controlled from gsettings. You will need to use these commands


gsettings set org.gnome.system.proxy.http host "myproxy.server.com"
gsettings set org.gnome.system.proxy.http port "3128"

SOCKS vs HTTP

HTTP proxies are clear enough: you want to access websites over http/https, and HTTP proxy servers simply relay the requests across. But it also means you need to make changes to firewall rules to open up a new port for the proxy server.

The SOCKS protocol is designed to work at a lower level, without opening additional ports. For example, ssh natively support SOCKS for port forwarding, i.e. SSH tunneling.

The tricky bit is how to set SOCKS proxy. Instead of


http_proxy=http://myproxy.server.com:8080/

append socks5 in front of the SOCKS server address


http_proxy=socks://127.0.0.1:1080

Likewise in Windows 10 proxy settings, add the prefix socks=

Proxychains

What if the application you are using does not work with SOCKS protocol? You can use Proxychains to redirect the remote SOCKS/HTTP proxy to your application. For example


$ proxychains4 brew install zsh
[proxychains] config file found: /home/hanxuel.linuxbrew/etc/proxychains.conf

Proxy Chaining

Perhaps the remote proxy uses PAC / WPAD to configure a list of URLS. Or the remote proxy server is very simple, and you want to have fine-grained control over what URLs to go through the proxy server, and what URLs to access directly. pacproxy is the perfect tool.

Start Chrome with Command Line Options

You can also explicitly tell Chrome to use proxy settings by starting Chrome from the command line. For example:

google-chrome --proxy-server=socks://127.0.0.1:9999

Note that Chrome ignores any local host inclusion for using proxy for security reasons. Explained in more detail here. If you must include local addresses, use the <-loopback> option


google-chrome --proxy-server=socks://10.10.110.28:8080 --proxy-bypass-list=<-loopback>

How to convert a PDF document into Powerpoint slides

2019-08-10T13:29:00.001+08:00

We want to convert PDF to Powerpoint slides

But there is no direct way to convert between the two formats. So let's break it down into an intermediary step: convert each page in the PDF document into JPG, then import the JPGs into Powerpoint.

Convert PDF to JPG using Ghostscript

Ghostscript is the bet tool for converting from PDF or PS into other formats. ImageMagick is the more popular tool, but for PDF conversion it uses Ghostscript in the background.

Before conversion, you need to know a couple of things:

The actual density or dpi for the PDF document
Quality vs size for the generated image

This StackOverflow answer, especially the top voted comment has a good explanation on density and quality. Here is a straightforward Ghostscript command for the conversion assuming 200dpi and 100% JPEG quality:


gs -dNOPAUSE -sDEVICE=jpeg -r200 -dJPEGQ=100 -sOutputFile=document-%02d.jpg "The_Artificial_Intelligence_Crush_2018.pdf" -dBATCH

I wrote about ImageMagick image quality when converting from PDF.

Convert PDF to JPG using ImageMagick

Use ImageMagick (with Ghostscript installed) for a more user-friendly format, especially for tuning the quality


magick -density 200 -colorspace sRGB "The_Artificial_Intelligence_Crush_2018.pdf" -flatten output-%02d.jpg

Use convert instead of magick on Linux and macOS.

Import Images Into Powerpoint

Now that you have the images ready, let's open a new Presentation in Powerpoint, and Insert → Photo Album

Select all the images and click Create

Tada! Your PDF document is now a Powerpoint presentation.

How About macOS?

If you are using macOS, there is good news and bad news. The good news is exporting images from PDF is a breeze using Preview. View Thumbnails in Preview, or press ⌘ + ⌥ + 1. Then select all the images and File → Export. The Preview User Guide has a section titled Extract a PDF page as an image.

The bad news is Powerpoint for Mac still does not have the Insert Photo Album functionality. You can either use Automator, or create the slides in Keynote, then export to PPT.

Vuepress: How to embed video in markdown

2019-07-23T08:00:00.000+08:00

With Vuepress is you can start creating content in Markdown out of the box. And Markdown has tons of useful plugin. When I was updating the Wujiquan website, I wanted to embed videos into the content. While it is possible to add html tags directly within markdown, I want to keep the content clean and simple.

markdown-it-html5-embed

Vuepress supports markdown-it plugins, and https://github.com/cmrd-senya/markdown-it-html5-embed comes to the rescue! Adding


![6 healing sounds video](https://wujiquan.sgp1.digitaloceanspaces.com/Qigong/Wujiquan-six-healing-sounds.mp4)

to the page will render the following in HTML


&lt;source type=&quot;video/mp4&quot; src=&quot;https://wujiquan.sgp1.digitaloceanspaces.com/Qigong/Wujiquan-six-healing-sounds.mp4&quot;&gt;
Your browser does not support playing HTML5 video. You can &lt;a href=&quot;https://wujiquan.sgp1.digitaloceanspaces.com/Qigong/Wujiquan-six-healing-sounds.mp4&quot; download=&quot;&quot;&gt;download a copy of the video file&lt;/a&gt; instead.
Here is a description of the content: 6 healing sounds video

Adding Markdown Plugin to Vuepress

To add the markdown-it-html5-embed plugin, first add the package to your vuepress project


yarn add markdown-it-html5-embed

Then edit your config.js and add the following code


module.exports = {
  markdown: {
    extendMarkdown: md => {
      md.use(require('markdown-it-html5-embed'), {
        html5embed: {
          useImageSyntax: true,
          useLinkSyntax: false
        }
      })
    }
  },

For more information on the syntax, refer to the markdown-it-html5-embed documentation.

How To Change Visual Studio Code's Default Shell to Cmder

2019-07-22T16:00:00.000+08:00

Visual Studio Code's default Shell in Windows is PowerShell

There are better alternatives to PowerShell or cmd.exe in Windows. My favourite is Cmder.

Presumably you already have Cmder installed in your system, in a path without space, for example C:\Tools\Cmder. Chocolatey's Cmder installation works perfectly fine.

Change Default Shell

To change Visual Studio Code's default shell, go to Preferences and edit settings.json. Add the following lines, changing the path where Cmder is installed

    "terminal.integrated.shell.windows": "cmd.exe",
    "terminal.integrated.env.windows": {
    "CMDER_ROOT": "C:\\tools\\Cmder"
    },
    "terminal.integrated.shellArgs.windows": [
      "/k C:\\tools\\Cmder\\vendor\\init.bat"
    ],

Important: you need to set cmd.exe. Setting Cmder.exe will not work

Now you can have all the goodness of Cmder within vscode, such as built-in git integration.

PowerShell in Cmder

You can also change your default shell from the cmd.exe/bash default to PowerShell within Cmder. Use the following settings.json in vscode

    "terminal.integrated.shell.windows": "C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
    "terminal.integrated.env.windows": {
    "CMDER_ROOT": "C:\\tools\\Cmder"
    },
    "terminal.integrated.shellArgs.windows": [
      "-ExecutionPolicy", "Bypass", 
      "-NoLogo", "-NoProfile", "-NoExit", 
      "-Command", ". 'C:\\tools\\cmder\\vendor\\conemu-maximus5\\\\..\\\\profile.ps1'"
    ],

Automated Security Fixes on Github

2019-07-21T19:41:00.002+08:00

It's been a while since I have worked on open source projects. When I logged on to my project's Github page, I was pleasantly surprised to see this warning:

A bit of Googling showed that Github has started to automatically scan for outdated dependencies with security vulnerabilities.Clicking on the alert will show which packages that are impacted.

Github also helpfully show exactly what are the vulnerabilities

Automated Security Fix

What's even better is the automated security fix feature. You will need to turn it on for your repository though.

If you choose not to have automated fixes, you still can manually trigger the fix by using the Create automated security fix functionality.

A pull request will be automatically generated.

While it is nice for Github to generate security fixes, we should not be depending on 3rd parties for our project's security. Use a tool such as yarn audit to keep your project constantly up to date and secure.

How to check the number of CPUs and cores

2019-07-18T18:38:00.001+08:00

How do you check for the number of CPUs and cores? From /proc/cpuinfo?


$ grep -c ^processor /proc/cpuinfo
56
$ cat /proc/cpuinfo | awk '/^processor/{print $3}' | wc -l
56

They will return twice the number if hyperthreading is enabled. How about using nproc?


$ nproc --all
56

Still the same. Not accurate if hyperthreading is enabled.

getconf

The getconf command works great because it is available on both Linux and MacOS.


$ getconf _NPROCESSORS_ONLN
56

Unfortunately it only returns the number of virtual cores.

lscpu and sysctl

Another problem is the output from /proc/cpuinfo is meant to be read by humans, and hence may change across version. Not suitable for parsing and using it in scripts.

You should be using lscpu instead. Or sysctl on MacOS. This script from Stackoverflow by the user mklement0 counts the number of logical and physical CPU


#!/bin/sh

# macOS:           Use `sysctl -n hw.*cpu_max`, which returns the values of 
#                  interest directly.
#                  CAVEAT: Using the "_max" key suffixes means that the *maximum*
#                          available number of CPUs is reported, whereas the
#                          current power-management mode could make *fewer* CPUs 
#                          available; dropping the "_max" suffix would report the
#                          number of *currently* available ones; see [1] below

#
# Linux:           Parse output from `lscpu -p`, where each output line represents
#                  a distinct (logical) CPU.
#                  Note: Newer versions of `lscpu` support more flexible output
#                        formats, but we stick with the parseable legacy format 

#                        generated by `-p` to support older distros, too.
#                        `-p` reports *online* CPUs only - i.e., on hot-pluggable 
#                        systems, currently disabled (offline) CPUs are NOT
#                        reported.

# Number of LOGICAL CPUs (includes those reported by hyper-threading cores)
  # Linux: Simply count the number of (non-comment) output lines from `lscpu -p`, 
  # which tells us the number of *logical* CPUs.
logicalCpuCount=$([ $(uname) = 'Darwin' ] && 
                       sysctl -n hw.logicalcpu_max || 
                       lscpu -p | egrep -v '^#' | wc -l)


# Number of PHYSICAL CPUs (cores).
  # Linux: The 2nd column contains the core ID, with each core ID having 1 or
  #        - in the case of hyperthreading - more logical CPUs.
  #        Counting the *unique* cores across lines tells us the
  #        number of *physical* CPUs (cores).
physicalCpuCount=$([ $(uname) = 'Darwin' ] && 
                       sysctl -n hw.physicalcpu_max ||
                       lscpu -p | egrep -v '^#' | sort -u -t, -k 2,4 | wc -l)

# Print the values.
cat <<EOF
# of logical CPUs:  $logicalCpuCount
# of physical CPUS: $physicalCpuCount
EOF

How to allow a non-root process to listen to a port below 1024

2019-07-17T17:05:00.000+08:00

Linux does not allow for user processes to listen to a port below 1024. This is by design. Specifically in the file net/ipv4/af_inet.c , these lines perform the check


if (snum && snum < PROT_SOCK && !capable(CAP_NET_BIND_SERVICE))
              goto out;

The security reasoning is misguided in 2019. Malware writers and other bad actors cannot care less what port your process is running on: as long as they can take control of the process and your process has a lot of privileges, then your system will be compromised. But I digress.

setcap

The best method to overcome this limitation is to use the setcap command


sudo setcap 'cap_net_bind_service=+ep' /scratch/spiped

Note that sudo or root permission is required when running setcap or else you will see this message


$ setcap 'cap_net_bind_service=+ep' /scratch/spiped
unable to set CAP_SETFCAP effective capability: Operation not permitted

Due to setcap's limitation, the command can only work on a local filesystem. When I run the command on my NFS-mounted home directory, I see this error


$ sudo setcap 'cap_net_bind_service=+ep' ~/bin/spiped
Password: 
Failed to set capabilities on file `/home/hanxue/bin/spiped' (Operation not supported)

For more information, refer to the setcap manual.

iptables

You can re-route local connections from one port to another port. Note that the following will NOT work for local connections


iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 25 -j REDIRECT --to-port 8025

You need PREROUTING for local connections


iptables -A PREROUTING -t nat -p tcp --dport 80 -j REDIRECT --to-port 8080
iptables -A OUTPUT -t nat -p tcp --dport 80 -j REDIRECT --to-port 8080

SSH Tunnel

This sounds a bit silly, but you can open an SSH connection to the same machine, routing the port


sudo ssh localhost -L 25:localhost:8025 -N

authbind

Alternatively, use authbind. Configure port 25 for access to all users


sudo touch /etc/authbind/byport/25
sudo chmod 777 /etc/authbind/byport/25

Then run the "authbind" command


authbind --deep /scratch/spiped --with --your --arguments

Hack the Linux Kernel

That's the advantage of working with open source software; when all easy options fails, you can edit the source code.

Remove the 1024 port restriction and change it to 0


# /usr/src/linux-[version_number]/include/net/sock.h:
/* Sockets 0-1023 can't be bound to unless you are superuser */
#define PROT_SOCK       1024

Or in the aforementioned net/ipv4/af_inet.c file


#if (snum && snum < PROT_SOCK && !capable(CAP_NET_BIND_SERVICE))
#              goto out;

This really should be the last resort: who knows if there are any sloppy developers that count on listening to ports below 1024 to fail for their application to work ;-)

By the way, I was trying to relay mail delivery from one server to another server using spiped

Chrome's History and Usage Trends

2019-07-03T13:22:00.000+08:00

Google used to show the days and times when I most frequently use Search. I like to be able to see when I am most active and what are my usage trends over time. Unfortunately the feature has been taken away.

History Trends Unlimited

Thankfully there is History Trends Unlimited Chrome extension. It will automatically pull your full Chrome history and present them in an informative manner.

I can see roughly when are the times when I am at work, and what are the top sites that I visit.

Unlimited History

Chrome's history of the sites you have visited will only be stored for 90 days. Beyond that the older data is deleted. What's great with this extension is it will keep your history beyond 90 days - henec the name "Unlimited".

Export History

In case you need to export your complete browser history, the extension let's you do that. You can also automatically back up your history in a fixed interval.

History Unlimited options

Feel free to explore the export options - fantastic to perform analysis on your web browsing behaviour.

Github OpenSearch from Chrome's Omnibox

2019-07-01T14:14:00.000+08:00

You already know it is possible to directly search from Chrome's Omnibox (address bar). For some frequently visited website, you might have noticed Chrome automatically picks up the search feature:

OpenSearch

The magic that makes this feature work is called OpenSearch. This is StackOverflow's Open Search definition https://stackoverflow.com/opensearch.xml:


<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/" xmlns:moz="http://www.mozilla.org/2006/browser/search/">
<ShortName>GitHub</ShortName>
<Description>Search GitHub</Description>
<InputEncoding>UTF-8</InputEncoding>
<Image width="16" height="16" type="image/x-icon">https://github.com/favicon.ico</Image>
<Url type="text/html" method="get" template="https://github.com/search?q={searchTerms}&ref=opensearch"/>
<moz:SearchForm>https://github.com/search</moz:SearchForm>
</OpenSearchDescription>

Github OpenSearch

Github too has https://github.com/opensearch.xml . But for some strange reasons Chrome is not picking it up.

You can manually add any site's search in Chrome's settings. First, go to chrome://settings and click on Manage Search Engine

Chrome Manage Search Engine

Click on Add

and type in the following values:

Search Engine: Github
Keyword: gh
URL: https://github.com/search?q=%s&ref=opensearch

You can use any keyword you preferred. I used g for Google, so I am using gh for Github.

Searching

After adding the shortcut, you can enter gh in the omnibox and directly search Github

How to convert multi-page PDF to images in command line

2019-06-25T12:27:00.001+08:00

Ghostscript is your best bet to convert PDF (or PS files) into image formats. But the command is not as user-friendly:


gswin64c.exe -dNOPAUSE -sDEVICE=jpeg -r200 -dJPEGQ=60 -sOutputFile=document-%02d.jpg document.pdf -dBATCH

Compare to the equivalent in ImageMagick


magick -quality 60 -density 200 document.pdf document-%02d.jpg

You do need to have Ghostscript installed, or else you will see this error message.

Image Quality

Note that even if you use -quality 100 you may still get low quality image output. Unlike images, PDF files do not have a native resolution or scale. After all, they are documents meant for printing, so dots per inch (dpi) is the unit of measurement for resolution. This density and other settings such as colorspace will impact the quality of output. You can specify them in the input:


magick -density 300 -colorspace sRGB input.pdf output.jpg

Specifically if you need to convert CMYK PDF into RGB PNG or JPEG, you will need to edit delegates.xml inside ImageMagick's directory and add the -dUseCIEColor=false option for delegate decode="ps:cmyk", so it will look like:


<delegate decode="ps:cmyk" restrain="True" command="&quot;@PSDelegate@&quot; -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dEPSCrop -dAlignToPixels=0 -dGridFitTT=2 &quot;-sDEVICE=pamcmyk32&quot; -dTextAlphaBits=%u -dGraphicsAlphaBits=%u &quot;-r%s&quot; %s &quot;-sOutputFile=%s&quot; -dUseCIEColor=false &quot;-f%s&quot; &quot;-f%s&quot;"/>

Answer on StackOverflow by user2846289

If you are sure about the output size, you can also -scale it


magick -density 300 -scale 800 input.pdf output.jpg

Some PDF files also have transparency - and the transparent region will turn into black background. You can -flatten the image so that it has a white background, like what you see on screen.


magick -density 300 -flatten input.pdf output.jpg

You can also limit the JPEG output size


magick -density 200 input.pdf -define jpeg:extent=300kb output-%02d.jpg

Combine Into Single Image

By default, each PDF page will be output into individual image file. You can combine all the images into a single file with the -append option


magick -density 200 document.pdf -append single-document.jpg

Note: the magick.exe executable is usually installed as convert in Linux and MacOS.

When to use xargs and exec

2019-04-10T23:38:00.001+08:00

In a recent post, I have written about searching for files using find and opening them using Visual Studio Code. But what is the difference between xargs and find . [args] -exec [cmd] {} \;? Why can't we just pipe it?

Using Pipes

Using pipes is one of the most natural thing to do in UNIX. So why can't we simply pipe the output to code?


$ find . -name xcode_requirement* | code
Run with 'code -' to read from stdin (e.g. 'ps aux | grep code | code -').

The command above does not work because we are passing the filenames as STDIN to the code command. We are not passing arguments to the code command line, but we are simply passing a chunk of text.

Note that pipe simply transfer the standard output STDOUT of the preceding command (to the left of the pipe) as the standard input STDIN of the following (to the right of the pipe) command.

This article covers a simple example, searching for a list of files and opening those files with another application. But if you are chaining pipes and running more complicated commands, note that

UNIX pipes does not guarantee the execution order of processes.

Difference between xargs and exec

OK, so we cannot rely on STDIN, and need to pass a list of arguments argv to the command. But what is the difference between


find . -name "*.rb" -exec code {} \+

and


find . -name "*.rb" | xargs code

?

The difference is that xargs will stop execution if the code sub-command fails, whereas even if code fails on one of the files, find will continue passing the next file. In short, find -exec returns the exit code of itself, but xargs will return the exit code of the sub-command.

Performance

This article went into the detail of the performance of find -exec vs xargs. This is the summary of the result

In summary

use xargs if you have to pipe the input, otherwise
use find . [args] -exec [sub-command] {} \+

How to Find and Open Files on Visual Studio Code

2019-04-10T18:16:00.001+08:00

Visual Studio Code has an excellent interface and File Explorer. Use Quick Open with ⌘ + P or Ctrl + P and type in the file name. But what if you want to search through a complex project and find files with a specific pattern? Sometimes it is easier to just use simple bash commands with regex.

Opening A Single File

Search for the filename and then pipe it through code


$ find . -name "xcode_requirement*" | xargs code

Technically, a more efficient way to open a single file is to run it through use exec. It's just more keystrokes.


$ find . -name "xcode_requirement.*" -exec code {} \;

Or you could just create a sub-shell


$ code "$(find . -name xcode_requirement.rb)"

Opening Multiple Files

Search and opening multiple files is a bit different. Do you want to open all the files in a new window? If so, use a sub-shell


$ code "$(find . -name *.rb)"

Opening a list of searched files in a new window

But if you want to open all the files in an existing window, use xargs


$ find . -name "*.rb" | xargs code

You can also use find, but use "\+" so that all the results are concatenated in one line, and code is only invoked once, instead of every time for each result.


$ find . -name "*.rb" -exec code {} \+

Opening a list of searched files in the existing window

Opening with Other Applications

The commands above also can be used with other applications. If you want to open the file(s) with the default registered application (open in MacOS / xdg-open in Linux / start in Windows), simply substitute the command


$ find . -name "xcode_requirement*" | xargs open

Migrating Docker Machine to external hard disk

2019-01-20T10:00:00.000+08:00

This article will show you how to move Docker Machine VM image to another location. I have a small SSD disk as my built in hard disk, and had to move my Docker Machine's VM to an external USB hard disk.

Move the Docker Machine image

Before making any changes, stop any instances of docker machine.


$ docker-machine stop
Stopping "default"...
Machine "default" is already stopped.

Move the entire ~/.docker/machine/ directory. You will need to tell Docker Machine the new path to the VM. Consider adding the MACHINE_STORAGE_PATH environment variable to your ./bashrc.


$ mv ~/.docker/machine /Volumes/USB/docker/
$ export MACHINE_STORAGE_PATH=/Volumes/USB/docker/machine/

Update VirtualBox VM location

Docker Machine knows where to look for the files, but we also need to tell the VM hypervisor of the location. If you are using the default VirtualBox VM driver, open up the VirtualBox application

VirtualBox showing the VM is now inaccessible

VirtualBox will complain that the VM is inaccessible and you will see an error message similar to this


$ docker-machine -s /Volumes/USB/docker/machine start
Starting "default"...
(default) Check network to re-create if needed...
Unable to start the VM: /usr/local/bin/VBoxManage startvm default --type headless failed:
VBoxManage: error: Could not open the medium '/Users/hanxue/.docker/machine/machines/default/disk.vmdk'.
VBoxManage: error: VD: error VERR_FILE_NOT_FOUND opening image file '/Users/hanxue/.docker/machine/machines/default/disk.vmdk' (VERR_FILE_NOT_FOUND)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MediumWrap, interface IMedium

Remove the Virtual Machine. Then go to Machine ➜ Add... and browse to the new location and open default.vbox.

We are not done yet - we need to tell VirtualBox about the updated disk images. Open up the Settings of the default VM, and click on Storage.

VirtualBox VM with invalid storage devices

You will need to remove both the disk images and re-add them. boot2docker.iso is an Optical drive and disk.vmdk is a disk image.

Disk images re-added

Remember to set the shell environment variable MACHINE_STORAGE_PATH or else docker-machine will complain the VM is missing.


$ docker-machine start
Error: No machine name(s) specified and no "default" machine exists

Alternatively, you can use -s to specify the storage for the docker machine.


$ docker-machine -s /Volumes/USB/docker/machine start

Docker logo

How to Get Facebook to Show Featured Image from Blogger Homepage

2019-01-19T08:00:00.000+08:00

In my earlier post, I have shown how to set the featured image when sharing your blog's homepage on Twitter.

Twitter image preview of blog homepage

When I share this blog's homepage on Facebook, I get to see this irrelevant photo instead

Facebook share without featured image

Featured Image

First, decide on the featured image that you want to use. I will be using this image, the same as Twitter's image for this blog's homepage


https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwtKF4VIiNfjAZxlxL-Z-b7x1XgzhzzehAZH0_YwwSBIUqUi3xg2N7D63ry356dSPHQ2QxyHZXRrkSX_y_atOoCU9yJkpgyC_qvRBYOvM6RDIcOyc9D7VmJhfYpX6ZAGlimtJC4A/s1600/Screen+Shot+2014-06-25+at+2.14.03+PM.png

Facebook recommends the image dimension to be 1200x630, so let's get that size from Blogger / Google Photo's responsive image. Change the s1600 parameter to w1200-h630-p-nu which is set to 1200 x 630, with smart crop and no upscale


https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwtKF4VIiNfjAZxlxL-Z-b7x1XgzhzzehAZH0_YwwSBIUqUi3xg2N7D63ry356dSPHQ2QxyHZXRrkSX_y_atOoCU9yJkpgyC_qvRBYOvM6RDIcOyc9D7VmJhfYpX6ZAGlimtJC4A/w1200-h630-p-nu/Screen+Shot+2014-06-25+at+2.14.03+PM.png

To set the Featured Image, open up your Blogger's template and Edit HTML, search for


<b:include data='blog' name='all-head-content'/>

My previous post has covered showing the preview images in a Twitter card when sharing the blog home page. Look for the data:blog.postImageUrl condition, and add image URL for og:title. This is how it looks


       <b:if cond='data:blog.postImageUrl'>
          <meta expr:content='data:blog.postImageUrl' name='twitter:image'/>
          <meta expr:content='data:blog.postImageUrl' name='og:image'/>
       <b:else/>
          <meta content='https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwtKF4VIiNfjAZxlxL-Z-b7x1XgzhzzehAZH0_YwwSBIUqUi3xg2N7D63ry356dSPHQ2QxyHZXRrkSX_y_atOoCU9yJkpgyC_qvRBYOvM6RDIcOyc9D7VmJhfYpX6ZAGlimtJC4A/s1600/Screen+Shot+2014-06-25+at+2.14.03+PM.png' name='twitter:image'/>
          <meta content='https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwtKF4VIiNfjAZxlxL-Z-b7x1XgzhzzehAZH0_YwwSBIUqUi3xg2N7D63ry356dSPHQ2QxyHZXRrkSX_y_atOoCU9yJkpgyC_qvRBYOvM6RDIcOyc9D7VmJhfYpX6ZAGlimtJC4A/s1600/Screen+Shot+2014-06-25+at+2.14.03+PM.png' name='og:image'/>
       </b:if>

Adding the Blogger featured image for Facebook using the "og:image" Open Graph tag

You can check if sharing on Facebook works correctly using the Facebook Open Graph Object Debugger.

Preview of your Blogger homepage when shared on Facebook

Show an Image Preview When Sharing a Blogger Post to Facebook

2019-01-18T22:30:00.000+08:00

When you share a link from a popular website on Facebook, you will see a nice image preview, and some description of the article.

Sharing a Foreign Policy article on Facebook

But when you share your blog article on Facebook, it will appear as a simple link. No images, no title and description. Hmm... very compelling to click and read the article right?

Sharing Blogger link on Facebook without preview

Open Graph Sharing

Facebook created Open Graph sharing to facilitate sharing rich media content from around the Internet. You can read more about it and see examples here.

We need to tell Blogger to generate these Open Graph tags for all the articles that we publish. Open up your Blogger settings and navigate to Theme → Edit HTML and scroll up to the beginning with the html tag where the namespaces are defined. Add the Open Graph namespace at end of the HTML tag xmlns:og='http://ogp.me/ns#'


<?xml version="1.0" encoding="UTF-8" ?>

<!DOCTYPE html>
<html b:render='false' b:version='2' class='v2' expr:dir='data:blog.languageDirection' xmlns='http://www.w3.org/1999/xhtml' xmlns:b='http://www.google.com/2005/gml/b' xmlns:data='http://www.google.com/2005/gml/data' xmlns:expr='http://www.google.com/2005/gml/expr' xmlns:og='http://ogp.me/ns#'>

Next, scroll down to just before the closing tag of </head> and add in these open graph tags


<!-- Open Graph Meta Tags BEGIN -->
<meta expr:content='data:blog.pageName' property='og:title'/>
<b:if cond='data:blog.postImageThumbnailUrl'>
  <meta expr:content='data:blog.postImageThumbnailUrl' property='og:image'/>
</b:if>
<meta expr:content='data:blog.title' property='og:title'/>
<meta expr:content='data:blog.canonicalUrl' property='og:url'/>
<b:if cond='data:blog.metaDescription'>
  <meta expr:content='data:blog.metaDescription' property='og:description'/>
</b:if>
<!-- Open Graph Meta Tags END -->
</head>

When you share your link, Facebook will look for the article's title and get a description from your article, and include the image found in your article.

Now Facebook will show a preview from your Blogger link

Display Image in Twitter Card When Sharing Blogger Links

2019-01-18T07:00:00.000+08:00

When you share a Blogger link on Twitter, it will appear as a simple link.

Tweets with simple links

No image previews, short description or anything at all. How boring.

I am sure you have seen tweets sharing content with websites that has a small image preview, article title and a short description. Wouldn't it be nice to have a preview when you share a link from your blog?

Links in Twitter now appear in card format, with image preview, title, description and web address

Open Graph

The Open Graph protocol is invented for precisely this reason. Instead of flat HTML pages with links to each other, the Open Graph is designed to turn the web pages into a rich object, and content can be connected in rich multimedia.

Twitter has a well documented set of Twitter card tags. The information that you want to be displayed in your tweet are:

Your twitter username
Your blog's domain
Your blog post's URL
Blog post title
Blog post image
Summary of your blog post

Blogger Template for Twitter Open Graph Sharing

First, open up your Blogger settings and navigate to Theme → Edit HTML and scroll up to the beginning with the html tag where the namespaces are defined. Add the Open Graph namespace at end of the HTML tag xmlns:og='http://ogp.me/ns#'


&lt;?xml version="1.0" encoding="UTF-8" ?&gt;

<!DOCTYPE html>
<html b:render='false' b:version='2' class='v2' expr:dir='data:blog.languageDirection' xmlns='http://www.w3.org/1999/xhtml' xmlns:b='http://www.google.com/2005/gml/b' xmlns:data='http://www.google.com/2005/gml/data' xmlns:expr='http://www.google.com/2005/gml/expr' xmlns:og='http://ogp.me/ns#'>

Next, scroll down to just before the closing tag of </head> and add in these Twitter open graph tags


<!--START Twitter Card -->
    <meta content='summary' name='twitter:card'/>
    <meta content='@hanxue' name='twitter:site'/>
    <meta content='@hanxue' name='twitter:creator'/>
    <meta expr:content='data:blog.homepageUrl' name='twitter:domain'/>
       <b:if cond='data:blog.pageType == &quot;item&quot;'>
          <meta expr:content='data:blog.canonicalUrl' name='twitter:url'/>
          <meta expr:content='data:blog.pageName' name='twitter:title'/>
       <b:else/>
          <meta expr:content='data:blog.homepageUrl' name='twitter:url'/>
          <meta expr:content='data:blog.pageTitle' name='twitter:title'/>
       </b:if>
       <b:if cond='data:blog.postImageUrl'>
          <meta expr:content='data:blog.postImageUrl' name='twitter:image'/>
       <b:else/>
          <meta content='http://lh5.googleusercontent.com/-ArkTRuQev2U/AAAAAAAAAAI/AAAAAAAAGMk/Ml_6kHFDjDE/s512-c/photo.jpg' name='twitter:image'/>
       </b:if>
       <b:if cond='data:blog.metaDescription'>
          <meta expr:content='data:blog.metaDescription' name='twitter:description'/>
       </b:if>
    <!--END Twitter Card -->
</head>

The code above will associate all links from your blog share with your Twitter username. The code also check if the link is a blog article. If it is, the complete article link is shown on Twitter, together with the article's title. Otherwise if you are simply sharing your blog's address, it will show your blog's home address and title.

Likewise, your Twitter Card will only a preview of the first image on your article. If there are no image found, for example if you are sharing your blog's home page or tags or categories, then it will use the blog profile image that you specify.

And finally if you are sharing your blog's home page or categories, Twitter will display your blog's description. Otherwise, it will get the description from the article content directly.

Copy and paste the code from above, and remember to change these 3 values

twitter:site - Your twitter account. Mine is @hanxue
twitter:creator - Your twitter account. Mine is @hanxue
twitter:image - The profile image of your blog

If you don't change the 3 values, especially twitter:image, the Twitter link to your blog will end up like this

Hanxue's blog, shared on Twitter

Validating Twitter Cards

Before you start sharing, you can validate the links to your blog posts at https://cards-dev.twitter.com/validator For example, this is what I see

Twitter Card Validator in action

Bash: Show execution time for each command in the shell

2019-01-17T06:30:00.000+08:00

We can measure the execution time of a command with time


$ time yarn audit
yarn audit v1.12.3
--- output truncated ---
6 vulnerabilities found - Packages audited: 34977
Severity: 1 Low | 4 High | 1 Critical
✨  Done in 6.58s.

real 0m7.083s
user 0m2.149s
sys 0m0.199s

But having to pre-pend time with each command is a pain. Wouldn't it be nice to have the bash prompt display the time taken for the last command?


[last: 6s] $ yarn audit

Elapsed Time for Last Command

We cannot use the time command to set the shell prompt. Fortunately bash has a special SECONDS internal variable that tracks the number of seconds the script has been running. There is also PROMPT_COMMAND, a variable that holds the command to be executed just before the prompt is displayed. Finally, we can start the timer by using the DEBUG trap, then stop the timer just before the shell prompt is shown.

Putting it all together:


function timer_start {
  timer=${timer:-$SECONDS}
}

function timer_stop {
  timer_show=$(($SECONDS - $timer))
  unset timer
}

trap 'timer_start' DEBUG
PROMPT_COMMAND=timer_stop

PS1='[last: ${timer_show}s][\W]$ '

Colored Bash Prompt

Having a shell prompt in white is boring. Why not colorize the output? Use this website to help you to generate the shell prompt syntax. I have added different colors for the time and the current working directory with this:


PS1='[\[\033[36m\]last:\[\033[m\] \[\033[32m\]${timer_show}s\[\033[m\]] \[\033[33;1m\]\W\[\033[m\]\$ '

This is how it looks like

Note that the elapsed time include the idle time from the previously completed command.

How to tidy up code using Visual Studio Code

2019-01-16T23:30:00.000+08:00

Have you come across jumbled up code that you wish you can format? This article will show you how to format code using Visual Studio Code. Install the Prettier vscode extension and reload Visual Studio Code. You will now see a Format Document command.

Visual Studio Code Format Document command

The keyboard shortcuts are:

Windows Shift + Alt + F
Mac Shift + Option + F
Linux Shift + Ctrl + F

Beautify Javascript Code

Modern published Javascript code is uglified to reduce the file size and increase performance. Take for example this snippet from a Chrome extension


var h=function(a,b){function c(){}c.prototype=b.prototype;a.f=b.prototype;a.prototype=new c;a.e=function(a,c,g){return b.prototype[c].apply(a,Array.prototype.slice.call(arguments,2))}};

Running the Format Document command will result in


var h = function(a, b) {
  function c() {}
  c.prototype = b.prototype
  a.f = b.prototype
  a.prototype = new c()
  a.e = function(a, c, g) {
    return b.prototype[c].apply(a, Array.prototype.slice.call(arguments, 2))
  }
}

Formatting XML Code

The second example I have here is a Blogger Dynamic View Template which contains a very long line.

Jumbled up code from Blogger template

Cannot make out head or tail of the code? Open up a new tab in Visual Studio Code, and paste the line. Then run Format Document

No document formatter for plaintext file

Oops, Visual Studio Code is complaining that there is no document formatter installed. Our code snippet is loaded on an unsaved buffer, and is detected as plain text by default. Simply change the type to HTML and run Format Document again. The result should be similar to this


<b:if cond="data:top.showEmailButton"
  ><a
    class="goog-inline-block share-button sb-email"
    expr:href="data:post.sharePostUrl + "&target=email""
    expr:title="data:top.emailThisMsg"
    target="_blank"
    ><span class="share-button-link-text"
      ><data:top.emailThisMsg /></span></a></b:if
><b:if cond="data:top.showBlogThisButton"
  ><a
    class="goog-inline-block share-button sb-blog"
    expr:href="data:post.sharePostUrl + "&target=blog""
    expr:onclick=""window.open(this.href, \"_blank\", \"height=270,width=475\"); return false;""
    expr:title="data:top.blogThisMsg"
    target="_blank"
    ><span class="share-button-link-text"
      ><data:top.blogThisMsg /></span></a></b:if
>

Prettier supports a few popular programming languages out of the box. You will need to install other language extensions in order for Prettier to work. A useful workaround is to save a copy of your source code, say Main.groovy to Main.js. You can now apply the Format Document command.

Visual Studio Code logo

How to speed up building of npm packages on Windows

2019-01-16T07:00:00.000+08:00

Building node.js projects on Windows can be significantly slower than on Linux or macOS. Assuming that you have a fast SSD and plenty of RAM, building a large node.js project is often slowed by Antivirus.

Testing by keawade shows that Yarn build time is 54% slower when Windows Defender is enabled, and disk indexing for the project directory is enabled.

A separate test by vbfox shows that with similar hardware on Windows and Antivirus disabled, the performance is comparable to macOS.

The test shows that simply copying the node_modules folder can be much faster when the folders are excluded by the antivirus software.

Exclude Indexing for node_modules

Since there is no use for the content of "node_modules" to appear in Windows search results, exclude the "node_modules" folder. I exclude the folder containing all my projects. My editor (Visual Studio Code) already has excellent search functionality. On the command line, I use the platinum searcher.

If you are feeling adventurous, you can edit the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Search\CrawlScopeManager\Windows\SystemIndex\ to exclude by Wildcard

For those interested, you can modify these defaults and add your own by turning off the search service, modifying the registry and restarting the service. The applicable keys are located here:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Search\CrawlScopeManager\Windows\SystemIndex\

Any overrides you specify show up in the WorkingSetRules key. This supports wildcards so you can exclude, say, node_modules, .git .svn, etc. from search indexing. tswaters 8/1/2015

Exclude Indexing for Yarn Cache

Likewise, exclude the %LocalAppData%\Yarn folder from Windows Indexing. This folder is updated every time you run an add, update or install command in Yarn.

Antivirus Whitelist

You should also whitelist your project folder (or at least the node_modules folder) and the Yarn Cache in your antivirus application. Instructions to whitelist in Windows Security

Also exclude the node.exe process from antivirus scanning - this is the process that has very high I/O.

In terms of application choices, certain corporate antivirus products can cause an order of a magnitude slowdown. Yarn is also easily 5x faster than npm on average.

Summary

Here are the steps you can take on Windows to increase node.js performance

Use yarn instead of npm
Have adequate hardware - SSD and sufficient RAM
Whitelist node.exe, node_modules or the project folder and Yarn / NPM cache in your antivirus application
Exclude indexing of your project folder and the Yarn / NPM cache

Linux: Move swap to RAM

2019-01-15T13:04:00.000+08:00

Linux will use swap memory if a swap partition is available. But swapping memory into RAM is much slower than simply using physical RAM. What if your machine has plenty of physical RAM and you want Linux to avoid using swap?

This script will check if the amount of free RAM is more than the swap memory that is being used. If there is sufficient free RAM, then swap memory will be turned off, flushing all swapped memory into RAM. Finally, swap memory is turned on again in case it is needed.

Create a file called /usr/local/sbin/swap2ram.sh with the following content


#!/bin/sh

mem=$(LC_ALL=C free  | awk '/Mem:/ {print $4}')
swap=$(LC_ALL=C free | awk '/Swap:/ {print $3}')

if [ $mem -lt $swap ]; then
    echo "ERROR: not enough RAM to write swap back, nothing done" >&2
    exit 1
fi

swapoff -a &&
swapon -a

Add executable permission to the script


chmod +x /usr/local/sbin/swap2ram.sh

And finally add this script into root's cronjob


sudo crontab -e

Set it to run every hour on the 3rd minute. This is the crontab entry


3 * * * * /usr/local/sbin/swap2ram.sh

Configuring yarn cache

2019-01-13T16:49:00.000+08:00

yarn, similar to npm, utilizes a cache. When you run yarn add package or yarn install, yarn does not simply fetch the package and save it directly into your project's "node_modules/package". It will at least have to fetch the compressed tarball, verify the integrity of the package and then unpack it.

The yarn cache avoids repeatedly downloaded the same package.

Cache Directory

You can find out the cache location with the command


$ yarn cache dir
/Users/hanxue/Library/Caches/Yarn/v4

Temporary Cache

If you do not want to keep the cached packages, you can temporarily set a different cache location with the --cache-folder flag


$ yarn install --prod --cache-folder /tmp/ycache; rm -Rf /tmp/ycache

Docker and Temporary Yarn Cache

When creating a Docker image, you almost certainly do not want the files in the yarn cache to be saved. The contents of node_modules/ are sufficient for your application to work.

One workaround is to change the Yarn Cache to use Docker's shm temporary file system, with this command in your Dockerfile


RUN YARN_CACHE_FOLDER=/dev/shm/yarn_cache yarn --production

The default shm filesystem is only 64MB, and you may need to increase it to an appropriate size, for example


$ docker build -t my-image --shm-size 512M .

Clearing the Yarn Cache

You can remove the contents of the yarn cache with


$ yarn cache clean

Alternatively, if you just wish to remove a single package from the cache, you can specify the package name


$ yarn cache clean lodash

Search for Package in the Cache

You can search for packages in the yarn cache


yarn cache list --pattern gulp
yarn cache list --pattern "gulp|grunt"
yarn cache list --pattern "gulp-(match|newer)"

Leaving out --pattern will simply list the complete content of your yarn cache


yarn cache list

Difference with NPM Cache

One major difference that you may need to be aware, npm uses the cache key to set the cache folder, while yarn uses cache-folder


yarn config set cache-folder /mnt/vol2/yarn-cache

You can run just about any yarn command while specifying the alternative cache location


yarn add package --cache-folder /mnt/vol2/yarn-cache

For more information, refer to the Yarn cache command documentation.

Cannot Set Cache Outside of Application Directory

If you cannot write to any other directory such as /tmp or /dev/shm, you can trick yarn to use your current directory by setting the HOME environment variable


HOME=$(pwd) yarn install

Disabling the Cache

Currently there is no way to avoid the use of the yarn cache. There is a proposal to add this option, so please vote to show your support.

Git Branch Dependency Does Not Update

If you run


yarn add org/package#branch_name

and branch_name is updated, yarn will simply use the cached content instead of fetching the new commits.

Use Git Commit

Instead of using the branch name, use the commit id instead


yarn add org/package#3ea9fe7984b9005f3cc62b14ee4018013ae0db0f

Use Tarball

If you are adding from Github, you can also specify the tarball URL


yarn add package https://github.com/org/package/tarball/3ea9fe7984b9005f3cc62b14ee4018013ae0db0f/package.tar.gz

Concurrently Updating The Yarn Cache

If there are multiple instances of yarn running on the same machine as the same user, for example a CI tool such as Jenkins running yarn, you might run into errors such as


23:33:24 +0000 verbose 1.293 Error: https://registry.yarnpkg.com/file-type/-/file-type-3.9.0.tgz: ENOENT: no such file or directory, chmod '/var/lib/jenkins/.cache/yarn/v1/npm-file-type-3.9.0-257a078384d1db8087bc449d107d52a52672b9e9/package.json'
23:33:24 +0000 error An unexpected error occurred: "https://registry.yarnpkg.com/file-type/-/file-type-3.9.0.tgz: ENOENT: no such file or directory, chmod '/var/lib/jenkins/.cache/yarn/v1/npm-file-type-3.9.0-257a078384d1db8087bc449d107d52a52672b9e9/package.json'"

The solution is to use the --mutex flag. Either one of the commands below will work


yarn command --mutex file
yarn command --mutex file:/tmp/.yarn-mutex
yarn command --mutex network
yarn command --mutex network:30330

Debian: no network connection even after restarting network manager

2019-01-11T20:37:00.001+08:00

My colleague had an issue with a Debian machine that fails to connect to the Internet even though the Network Manager is restarted. This machine has a script that detects network connection, and if the connection is down. When the connection is down, the Network Manager service is restarted. Presumably the command is


sudo /etc/init.d/network-manager restart

Restart the Network Stack

Note that network-manager is a client GUI service and restarting it should also restarting the network stack. But really what we want is to restart the network stack directly


sudo /etc/init.d/networking restart

Using systemd, run


sudo systemctl restart networking

You can restart the network stack directly without using the System V init or SystemD systems


sudo ifdown --exclude=lo -a && sudo ifup --exclude=lo -a

Flush DNS Cache

Although the problem was eventually solved with a reboot, the actual cause was not known. When we restart the networking service, we tend to think everything that is related to network connectivity is restarted, including the DNS cache. From my experience providing support for large Linux environments, a common problem with networking is corrupted or outdated DNS cache.

We often face this issue on our laptop when there is internet connectivity, but some websites do not work on the web browser. Pinging the IP address works, but nslookup or dig fails. Simply restarting the networking services will not solve the problem.

On a newer Debian machine or Ubuntu 17.04 and above, you can flush the DNS cache with the command


sudo systemd-resolve --flush-caches

On other versions of Debian and Ubuntu, dnsmasq is used for DNS caching. You can restart it with


sudo /etc/init.d/dnsmasq restart


sudo kill -HUP $(pgrep dnsmasq)

For Ubuntu 11.10 and older Debian machines, DNS records are not cached. Chances are, the system administrator might have installed a DNS caching service such as nscd. Restart nscd with


sudo /etc/init.d/nscd reload

On a side note, nscd is known to be buggy. I used experience a lot of OpenLDAP failures that is traced back to nscd. Use another DNS caching service such as pdnsd instead.

Another popular DNS caching server is BIND


sudo /etc/init.d/named restart

There could be other DNS servers or caching services that is running. Check /etc/resolv.conf or your Network Manager settings to see if any DNS service is used, and restart them appropriately.