Apollo 13 - Tao Liu's blog

Bringing the RISC-V Simulator into VS Code

2026-08-04T14:10:42.192-04:00

One of the goals of my RiscV toolchain project has always been more than simply generating RISC-V assembly.

I want to build a complete toolchain that makes writing, compiling, and debugging RISC-V programs feel natural. This week, I reached another milestone: my RISC-V simulator can now debug programs directly inside Visual Studio Code.

Seeing breakpoints, registers, and single-step execution inside VS Code is surprisingly satisfying. It makes the project feel much closer to a real development environment.

Why a VS Code Debugger?

When developing a compiler, debugging generated assembly is just as important as generating it.

Previously, my workflow looked something like this:

Compile C source code.
Generate RISC-V assembly.
Run the simulator.
Print register values.
Repeat.

That works for small programs, but it quickly becomes painful as the compiler grows.

Having an interactive debugger completely changes the experience.

Instead of inserting print statements everywhere, I can now:

set breakpoints
single-step through instructions
inspect registers
follow the execution flow
verify generated assembly interactively

It dramatically shortens the feedback loop when tracking down compiler bugs.

Using the Debug Adapter Protocol

Rather than inventing my own debugger UI, I decided to integrate with Visual Studio Code's Debug Adapter Protocol (DAP).

This means the extension can take advantage of the debugger interface that developers already know.

Currently the debugger supports:

launching a RISC-V program
breakpoints
stepping through instructions
viewing all integer registers
inspecting the current execution point
displaying the call stack

Because the simulator acts as the debugging backend, every instruction executed is under complete control of the debugger.

Connecting the Pieces

This project has gradually evolved into several independent components that now work together:

C Source

│

▼

C Compiler

│

▼

RISC-V Assembly

│

▼

Simulator

│

▼

VS Code Debugger

Each component was originally developed independently.

Connecting them together has been one of the most rewarding parts of the project.

When I hit F5 in VS Code, the debugger launches the simulator, loads the executable, and immediately allows me to step through the generated instructions.

That is a much nicer workflow than running individual command-line tools.

The Screenshot

The screenshot below shows the debugger paused on an instruction.

On the left, VS Code displays the complete register set.

On the right, the current instruction is highlighted while breakpoints are shown directly in the editor.

Watching the register values change after each instruction has already helped me identify several code-generation issues that would have been much harder to diagnose otherwise.

RisvV C Compiler Dev Journal - 20260804

2026-08-04T07:20:54.775-04:00

72 Test Cases, `typedef`, and Function Pointers

Today I reached another milestone on my RISC-V C compiler project with Rust. The regression suite has now grown to 72 executable test cases, and this milestone adds support for two important C language features: typedef and function pointers.

I've been following a simple rule throughout this project: every new language feature must come with at least one regression test. This gives me confidence to continue improving the compiler without worrying about breaking functionality that was implemented earlier.

The compiler has gradually evolved from handling simple expressions into supporting a substantial subset of the C language. So far it includes:

variables and arithmetic expressions
if, while, and for
functions and recursion
pointers and pointer arithmetic
arrays
structures
union
enum
typedef
function pointers

Adding `typedef`

Although typedef does not introduce a new type, it is an important C language feature that allows programmers to create aliases for existing types.

Supporting typedef required extending the compiler's symbol table and type system so that type aliases could be correctly distinguished from ordinary identifiers and resolved during semantic analysis.

With typedef in place, complex declarations become much easier to read. This is especially important for function pointers, where the original C syntax can quickly become difficult to understand.

For example:

typedef int (*BinOp)(int, int);

is much cleaner than repeatedly writing the full function pointer declaration throughout a program.

Why Function Pointers?

Function pointers are one of the features that make C both flexible and powerful. They enable callbacks, dispatch tables, state machines, and many programming patterns commonly found in embedded software, operating systems, and low-level libraries.

Supporting function pointers required more than just parsing a complicated declaration. The compiler now needs to correctly handle:

function pointer type resolution
assignments between functions and function pointer variables
indirect function calls
passing function pointers as parameters
returning function pointers from functions
function pointers stored inside arrays and structures

Test Case #72

To validate both typedef and function pointer support, I created a regression test that combines several real-world usage patterns into a single program.

The test covers:

a typedef for a function pointer type
multiple arithmetic callback functions
an array of function pointers
passing function pointers as parameters
returning a function pointer from another function
storing function pointers inside a structure
indirect function calls through variables, arrays, and structure members

The test program defines several operations (add, sub, mul, and negate_via) and exercises different ways of invoking them through function pointers.

The expected final result is 42, which becomes the regression check for the entire feature set.

This type of test is especially valuable because it validates the interaction between multiple compiler components:

parsing
type resolution
semantic analysis
function call generation
memory layout handling
RISC-V code generation

A More Stable Foundation

One thing I've noticed during recent development is that segmentation faults have become much less common.

Earlier in the project, adding a new language feature often exposed issues deep inside the compiler. These problems usually came from assumptions that were valid when the compiler was smaller but became incorrect as more language features were introduced.

Recently, the development process has changed. Most of my effort is now spent adding new capabilities and expanding test coverage rather than debugging compiler crashes.

I believe this is a sign that the compiler framework itself is becoming more stable. The internal architecture, type system, and regression framework are mature enough to support continued growth.

There are still many bugs to discover, but it is encouraging to see the project moving from "building the foundation" toward "expanding capabilities."

Growing the Regression Suite

The compiler now includes 72 executable regression tests, and this number has become more meaningful than simply counting implemented features.

Every bug I fix becomes a permanent regression test.

Every new language feature adds another executable example.

Over time, the regression suite is becoming a living specification of what the compiler supports.

This approach also makes future optimization work safer because I can quickly verify that improvements do not break existing functionality.

What's Next?

With many core C language features now implemented, including struct, union, enum, typedef, and function pointers, the next phase of development will focus on improving compiler maturity in two areas.

The first area is expanding floating-point support. The compiler already has basic floating-point functionality, and the next step is to add more comprehensive regression tests covering more complex scenarios, including floating-point expressions, conversions, function calls, and interactions between integer and floating-point types.

The goal is not only to make floating-point programs compile, but also to build confidence that the generated RISC-V code behaves correctly across a wider range of real-world cases.

The second area is optimization. Up to this point, the primary focus has been correctness, language coverage, and compiler stability. With the framework becoming more mature and the regression suite growing, the next challenge is improving the quality of generated code.

Future optimization work will explore areas such as:

improving generated instruction sequences
reducing unnecessary operations
peephole optimization
intermediate representation improvements
exploring more advanced compiler optimization techniques

This marks an important transition for the project: moving from:

"Can the compiler correctly compile C programs?"

toward:

"Can the compiler generate efficient RISC-V code?"

I'm looking forward to this next stage of the journey.

SelfNote: Register Spillover and Replacement

2026-05-07T10:26:36.894-04:00

This is post is to record some notes when I refine my register allocation study/design when dealing with RiscV processors.

First i need a function to decide how to move register SetB from SetA where available register set is present as "available". The return value are tuple. The 1st element is Vec<(register name0, register name1)>. The spillover are register set.

The replacement is used to perform replacement operation to move register to conflicted status to free register. register name0 is being replaced by register name1. The spillover are registers required to push to stack.

MoveAway(setA, setB, available) -> (replacement, spillover)
if setA ∩ setB = φ { (φ, φ ) }
else
overlap = setA ∩ setB
replacement = overlap zip available //this will generate vec<(reg name0, reg name1)>
spillover = overlap - available

And this function can be simplifed to

MoveAway(setA, setB, available) -> (replacement, spillover)
overlap = setA ∩ setB
replacement = overlap zip available //this will generate vec<(reg name0, reg name1)>
spillover = overlap - available

Building My First RISC-V VSCode Extension (and Everything That Went Wrong)

2026-04-20T06:52:00.003-04:00

I set out to build a simple VSCode extension for RISC-V assembly. The idea sounded straightforward: define a new language, hook up a Rust-based parser through LSP, and get syntax highlighting plus diagnostics. I’ve built compilers before—how hard could this be? It turns out, the hard part wasn’t the compiler at all. It was everything around it.

The Setup

On paper, the architecture was clean. A TypeScript extension runs inside VSCode, launches a Rust-based language server, and that server parses .rv.s files, reports errors, and sends back semantic tokens for highlighting. In development mode, everything worked beautifully. I could open a file and immediately see highlighting, introduce an error and watch diagnostics appear exactly where I expected. It felt solid, and at that point I thought I was basically done. I wasn’t.

The VSIX Reality Check

The moment I packaged everything into a .vsix and installed it, things broke—quietly. There was no highlighting, no diagnostics, and no logs. The extension just sat there stuck on “Activating…”. That’s when it really clicked: dev mode hides problems, but packaging reveals them.

The First Wall: Missing Dependencies

The first real clue came from an error saying it couldn’t find vscode-languageclient/node. That was confusing because it was clearly listed in my package.json. The issue, as it turned out, was entirely self-inflicted. VSCode extensions don’t bundle dependencies automatically, and I had explicitly excluded node_modules in .vscodeignore. In other words, I built an extension and then removed part of it before shipping. Once I included node_modules again, that problem disappeared. Simple in hindsight, but not obvious when you’re in the middle of it.

The Silent Failure: LSP Not Starting

After fixing dependencies, the extension still didn’t work. This time there wasn’t even an error—just silence. That silence made it harder to debug than an actual crash. The issue turned out to be the path to the Rust binary. In development, I was using a relative path like server/target/release/rust_keyword_lsp_server.exe, which worked because everything ran inside my workspace. But once installed, VSCode runs the extension from a completely different location, so that path no longer pointed to anything valid. The fix was to resolve paths using context.extensionPath. Once I did that, the server finally started.

The Subtle Killer: stdout

Then came one of the most subtle bugs in the entire process. My parser used println! to print errors, which is perfectly normal in Rust. But in an LSP setup, stdout is not for logging—it’s the protocol itself. Every time I printed something, I was corrupting the JSON stream between the server and VSCode. The client would silently disconnect, making it look like the server never started. There was no obvious error pointing to this. The fix was simply to use eprintln! instead, sending logs to stderr. One small change, but it made a massive difference.

The Debugging Breakthrough: Developer Tools

At one point, I realized I was essentially debugging blind. There were no logs, no clear signals, just a stuck extension. That changed when I discovered the developer tools in VSCode through “Help → Toggle Developer Tools”. This was a turning point. Suddenly I could see activation errors, inspect console logs, catch missing modules immediately, and verify runtime paths. Before this, I was guessing. After this, I was actually debugging. If you’re building a VSCode extension and not using this, you’re making things much harder than they need to be.

Packaging Is a Minefield

Even after getting everything working, packaging still had its own set of traps. I had to carefully tune .vscodeignore to exclude Rust source code and large build folders, include only the release binary, and still keep all required Node dependencies. A single mistake here could break the extension again in ways that looked completely unrelated. It became clear that packaging isn’t just cleanup—it’s part of the system itself.

What I Learned

The biggest surprise in all of this was where the complexity actually lives. It’s not in parsing, not in Rust, and not even in the LSP protocol. It’s in the boundaries: the differences between development and packaged environments, stdout versus stderr, relative paths versus resolved ones, and what exists locally versus what actually gets shipped. Each of these seems small on its own, but together they create a system where things can fail silently in ways that are hard to reason about.

Where It Ended Up

After working through all of that, I ended up with a VSIX that installs cleanly, a Rust LSP that starts reliably, working diagnostics, and semantic highlighting for RISC-V assembly. More importantly, I now understand the real challenges of building tooling inside VSCode, and they’re not where I initially expected them to be.

Closing Thought

If you’re building an LSP-based extension, expect the bugs to come from the edges, not the core. Your parser will probably work, and your design will probably make sense. But the thing that breaks everything might be something as small as a single println!, and you won’t see it coming.

Eventually, I got instruction and registers highlight works.

Rust FlameGraph

2026-03-25T16:09:00.003-04:00

cargo install Flamegraph

cargo flamegraph --bin <my binary>

Please note that <my binary> does not need to add .exe on windows.

The call stack is up side down. The top most element is the funnction at lower level. Now let's see if i can get more time on debuggin rather than waiting for the code finish.

Left & Right Asscociate in Parsing

2026-03-21T11:09:52.648-04:00

I made the same mistake again when using Pratt parsing. I almost messed up the left asscociate and right asscociate. If an operator is left asscociate, the generated AST will be a tree lean-to-left. For struct, union, and other composite types, left asscoiate can expose the member as the right node. This can make the code generation much easier, because I know the type for the dot operator.

For example, StructA.structB.a, the lean-to-left tree can expose the "a" as the right node and I can easily find out the whole expression StructA.structB.a is what type. I did not set assignment as an operator, the assignment can be lean-to-right tree.

Hopefully this self-note can let me remember this rule and won't miss this case in the future.

RiscV GCC Dynamic GOT layout

2026-03-07T12:14:00.003-05:00

I've been struggling with the Linux ELF's dynamic link format for days. All GPT's provided wrong answer and they're wrong in the same way.

The correct GOT layout for Linux ELF for RiscV are like the following if there are two external funtions. These two functions are funct0 and funct1.

ffffffff ffffffff is reserved data
00000000 00000000 is reserved data
GOT[funct0]
GOT[funct1]
.dynamic virutal address

#1 and #2 are linker map and resolver address. These fields will be set by dynamic linker (or called dynamic loader).

Unlike GPT's info, .dynamic virtual address is set to the first slot. The .dynamic virtual address is set as the last element.

Foray into LLVM - Intercept RiscV Assembly Code Generation

2026-02-22T20:29:00.003-05:00

This weekend, I started my adventure to hook my assembler / linker to LLVM. The first step is to see what's the right position to intercept the asm code generation. I highly suggest anyone who is interested in LLVM to start with the offical web site forum, especially Beginner Resources + Documentation - Beginners - LLVM Discussion Forums. Personally, I did not want to bother anyone on forum, but that document is a good start.

And I did use ChatGPT and Google Gemini. The following is the comparison between these two systems. Gemini seems confident; however, its solution does not fit and I was struggle with endless compile error, missing include files, and link errors. In the end, I gave up on Gemini. ChatGPT's solution is more accurate on the architect level and the code is more accurate. Although GPT tried to lead me, it did not succeed. 😅 That's because those beginner resource on LLVM web site and some coding experience.

What I did is to add "TALIU" label before all the instructions. The final assembly generated is like the following:

TALIU: li a0, 42
TALIU: ret

GPT's solution is to replace return new RISCVInstPrinter(MAI, MII, MRI); in the createRISCVMCInstPrinter.

The createRISCVMCInstPrinter is located in llvm-project\llvm\lib\Target\RISCV\MCTargetDesc\RISCVMCTargetDesc.cpp. I use my own InstPrinter replace the existing RISCVInstPrinter. The .h definition is

namespace llvm {

  class MyRISCVInstPrinter : public RISCVInstPrinter {

  public:

    using RISCVInstPrinter::RISCVInstPrinter;

    void printInst(const MCInst *MI, uint64_t Address, StringRef Annot,

                   const MCSubtargetInfo &STI, raw_ostream &O) override;

};

} // namespace llvm

And the .c file is like the following.

#include "MyRISCVInstPrinter.h"

namespace llvm {

    void MyRISCVInstPrinter::printInst(

        const MCInst *MI,

        uint64_t Address,

        StringRef Annot,

        const MCSubtargetInfo &STI,

        raw_ostream &OS) {

        OS << "TALIU: ";

        RISCVInstPrinter::printInst(MI, Address, Annot, STI, OS);

}

}

After this, make sure the build command (under build folder) is "ninja LLVMRISCVCodeGen LLVMAsmPrinter" with configuration command "cmake -G Ninja -DLLVM_ENABLE_PROJECTS="" -DLLVM_TARGETS_TO_BUILD="RISCV" -DCMAKE_BUILD_TYPE=Debug -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ../llvm"

Gemini told me to do "cmake --build . --target llvm-mc" and does not work. Given my limited experience with llvm, I don't fully understand why.

Dynamic ELF Structure - Endless Segment Fault

2026-02-14T07:09:00.001-05:00

I created a RiscV assembler and linker. One of the challenges I was facing is to understand the ELF dynamic structure and make it work on a new chip (RiscV). The first challenge is the segment and section and how the segment header and section header work together. I want to post a few posts about the ELF and dynamic ELF from scratch. The previosu post (GOT/PLT) is part of my assembler / linker work.

Before you start the work, make sure you have WSL ready with RiscV support ready. Although the RiscV support is not mandatory, but you will find it very helpful if your WSL can support RiscV binary execution.

Ok, let me start from ELF structure with a diagram. ELF standard is big and I struggled a lot to put this thing into my head. Frankly, I still do not have something to learn about this ELF.

My assembler/linker generate ELF binary and let libc modify the PLT/GOT data, and eventually invoke the "printf" function. Even the target instruction is RiscV, but the ELF format is same for RiscV and other CPU/chips.

The following digram has two columns. The 1st column is binary data in th file. The layout of these binary files can be in a different order, but the concept is there. The 2nd column is the segment which is runtime memory structure. The linkage from two columns shows how the data from file mapped into the runtime memory.

Another view to read these two columns are. Left column contains ELF header, program headers, sections, and section headers. The data on the right is how segment is created in the memory and section data is loaded into those segment.

There are tons of attributes about these sections and segments. Let's leave it for the later posts. The important thing are: what are data, how the data mapped to runtime memory.

use read-elf command from linux, it shows there are program headers. Those program headers describe this runtime memory contains what data (from which section). I use the following diagram shows its origin. Please note that these memory ranges can overlap. For example, the PHDR loads all program headers data and the PHDR segment is located inside the 1st load segment. The first loaded segment contains PHDR and also it also load the ELF header. If you do the dynamic ELF, this first section is mandatory; otherwise, segment fault will be the result.

Those green color blocks are same case. They're programers but overlaps with other segments.

Also, the note section is loaded to memory as well. I do not think it's mandatory, but note section is the last thing I added before I concluded the dynamic ELF generation work. The Interp segment is another case where the INTERP overlap with another segment.

RiscV Assembly Got/PLT index Calculation

2026-02-04T20:13:00.004-05:00

I searched many online document about RiscV PLT table and how the relocation works. Eventually I got my own version working. This is my notes in case one day I will forget.

Based on the PLT/GOT RISC V code, the lazy binding code is like the following two tables. The first one is the PLT[0] code stub where the _dl_runtime_resolver code is being invoked. The t3 stores the _dl_runtime_resolver address and the jr t3 is to call that function. That function will requires two parameters: t0 holds the link map address, and t1 holds the function (e.g. printf) offset in the .got.plt. In the code, t2 is used as temporary register.

The summary is below

·       t3 is the _dl_runtime_resolver function address stored in GOT
·       t0 is the link map address in GOT
·       t1 is the function (e.g. printf) offset in .got.plt
·       t2 is temporary register

The hdr_size is the PLT[0] size which is two 16 bytes (32 bytes) size section. It has 8 instructions. In the following code, PTRSIZE = 4 if 32bits and PTRSIZE = 8 if 64bits

Table 1 PLT[0] Code Stub

1: auipc t2, %pcrel_hi(.got.plt)

sub t1, t1, t3 # shifted .got.plt offset + hdr size + 12

l[w|d] t3, %pcrel_lo(1b)(t2) # _dl_runtime_resolve

addi t1, t1, -(hdr size + 12) # shifted .got.plt offset, hdr_size is PLT0 size (32 bytes)

addi t0, t2, %pcrel_lo(1b) # &.got.plt

srli t1, t1, log2(16/PTRSIZE) # .got.plt offset

l[w|d] t0, PTRSIZE(t0) # link map

jr t3

And the above code is invoked by the following code. The following code is a function (e.g. printf) stub in plt section.

Table 2 PLT[N] Code Stub

1: auipc t3, %pcrel_hi(function@.got.plt)

l[w|d] t3, %pcrel_lo(1b)(t3)

jalr t1, t3

nop

How to get offset in t1 register

The t1 is used to compute the got.plt offset from the plt code stub.

When the code called from Table 2 PLT[N] Code Stub to Table 1 PLT[0] Code Stub. The first time the Table 2 PLT[N] Code Stub is called, the function@.got.plt is points to PLT[0]. The PLT[0]’s job is to update the function@.got.plt to actually function address.

Table 2 PLT[N] Code Stub set t1 and t3 to the following value

· t1 = &nop = &PLT[N] + 12,

o 12 is from 3 instruction and each instruction is 4 bytes

· t3 = &PLT[0]

In the Table 1 PLT[0] Code Stub, the t1 was computed

t1 = [t1 – t3 – (hdr_size + 12)] >>

= [&PLT[N] + 12 - &PLT[0] – hdr_size – 12] >>

= [&PLT[N] - &PLT[0] – hdr_size] >> (where hdr_size is the PLT[0] size, it is 32bytes, refer to the diagram below)

= [(N-1)*16] >>

= =

= (N-1)PTRSIZE

N starts from 1 because PLT[0] is reserved for dynamic resolver. So the 1^st function is at 0 and 2^nd function is at PTRSIZE. The PTRSIZE is the address data size.

· If 32bit address, the PTRSIZE is 4, which is 4 bytes equals 32 bits.

· If 64bit address, the PTRSIZE is 8, which is 8 bytes equals 64 bits.

Please note the GOT.PLT has 1^st and 2^nd element to be reserved value. The 1^st element is PLT[0] address and the 2^nd element is link map address.

Link map into t0

The link map is a reserve entry in got.plt section. The link_map is a data structure used internally by the Linux dynamic linker (ld.so) to keep track of all shared libraries (shared objects, .so files) loaded into a running process.

Executable (.got.plt)

│

├── [0] => address of resolver entry (__dl_runtime_resolve)

├── [1] => pointer to link_map (used by resolver)

├── [2] => address of function (e.g., printf)

Please note this is not the end. If you want to call printf, you will need to implement the gnu version table and hash algorithm. I tried both old hash algorithm and new hash algorithm, both works. My environemnt is QEMU and StarFive2 Ubuntu Linux.

Rust Macro

2026-01-13T22:15:00.001-05:00

Rust macro is an interesting topic and it provides a way to generate code on the fly. One of the problem for Rust programmer is the code is not easy to understand, especially for compiler and low level programming.

The code is located at ttliu2000/rust_macro: Rust macro project to make rust programming easier. which making my assembler / linker project code simpler. My background is C# and I will keep everything private and use accessor methods to access the code, which is heavy boilerplate code. Using the macro make the code more readable.

Thanks to ChatGPT's help, I quickly start the project and get the feature in. So if you start to create macro, ask GPT first.

Tao's Fragments: RiscV Linux lib ELF Generation

2026-01-01T05:14:00.001-05:00

Today is Jan 1st, 2026. And I will be back with what is meaningful. Years ago, I decide to start a new journey. After working on Rust, Linux OS, ELF file format, RiscV chip/ABI, and assembler and linker, and started C language compiler. Let's see how compiler tech and latest AI can change how technology can make programmer's life easier.

I feel I can start blog again and share some fragments of my work and ready for open source.

I've done the ELF executable with dynamic link support, it means that I can use my assembler+linker to generate ELF binary to invoke printf in libc. I've to say the RISCV's PLT/GOT RELA is really a headache, which can be topic for another fragemnt of mine. 😉

Today I finished the .so file generation and it can generate a library can be linked with gcc. Now I know why F# choose to support library after executable is done. And I know why F# library has its own printf. But anyway.

The .so ELF generation is simple, my test .s file is listed below. Please note that i added my own pseudo instruction for RiscV assembly language.

.data
.extern printf

tt_fmt: .string "%d\n"

const_float_or_string_value_104:
.string "Array index access test, expected value 42 and actual value: %d\n"

.text
.globl add2
.type add2, @function
add2:
add a0, a0, a1
ret
.size add2, .-add2

exit 42

The .extern will trigger dynamic symbol structure generation. The key here is the add2 function related instruction which is in bold font.

globl add2 is to add "add2" to the dynamic symbol (.dynsym) table.
add2 shows the starting point of add2 function and it will be used to compute the size of the add2 function.
.size show the add2's size is equal to current (the small dot) minus add2.

After these info being added to dynamic symbol table, the readelf -a shows the following:

the riscv64 gcc link can link this shared library with c code and command is listed below:

#include <stdio.h>

int add2(int a, int b);

int main() {
printf("%d\n", add2(20, 22));
return 0;
}

and the exeuction result shows 42 which is 20 + 22 from add2(20, 22).

Thanks to ChatGPT's which helps me to generate Linux command. Being a long time windows guy, I am still struggle to use correct Linux language, but let's see if this can change when I add more feature.

Compiler AST & Mermaid

2025-12-19T14:01:00.002-05:00

I'm interested in RISC compiler for a while. One of the challenge of a compiler is to process AST. Being a visual person, I am interested in seeing the structure in an image. So I generate the mermaid state diagram and can visualize the tree.

SelfNote: Enable Python in Visual Studio Interactive

2020-10-18T10:40:00.005-04:00

This note is to document how to configure Python interactive window in Visual Studio (not VS Code)

Reference a local library

When I added a new .py file in the project, I cannot add it to the python interact window. The reason is the default path for python interact is under "program files". The following command is to change the current working folder to a new folder:

import os
os.chdir(r"c:\mycode\xyz\")

Make sure the string is started with "r".

Reload a library

After change a library, the new content is not loaded to the python interactive session by default. The following command is to reload the library into the session.

import myModule #import my module

#the following is to reload myModule

import importlib

importlib.reload(myModule )

Please note: myModule does not need quote or double quote

Matrix Multiplication & Graph Reachable

2020-08-07T14:54:00.001-04:00

I have been struggling to find a good way to represent graph structure. There are number of graph library where Node and Edge are defined. I found that matrix can be used to represent a graph.

The row and column of the matrix represent the node in a graph. The element in the matrix is connection between two nodes. In a graph, this connection is called Edge. The following diagram represents a graph with 3 nodes and 3 edges.

In a matrix representation, it can be written as a matrix. Row and column are named A, B, and C. If there is an edge between two nodes, the corresponding cell is set to 1. For example, edge AB shows 1 in cell (A, B) in the grid. We can denote this matrix as M

Matrix "multiplication" to itself can reveal if a certain node can reach the other nodes in two steps. The "multiplication" operation is different from the linear algebra matrix multiplication.

K = M ⊗ M

Each element b in N can be calculated by the following formula, where N is the number of the nodes (or the column number of the matrix M).

Two nodes i and j. If there j is reachable from i, either of the following two conditions are met:

there is a direct edge between i and j
there is a node p which is reachable from i and p can reach j

Written as math language in the matrix M

M(i, j) = 1
彐p, M(i, p) = 1 and M(p, j) = 1

Repeat the "multiplication" operation logN times, it can calculate any node in a graph can be reachable from other nodes.

Finally, I can move away from a complicated node/edge definition into a formal math world. Hopefully the existing linear algebra can help me down the road.

SelfNote: VSCode setup

2020-02-19T18:11:00.001-05:00

Recently I started to use the Azure cloud machine to do my coding work. I can still do my coding in my laptop using full Visual Studio but an Azure VM with VSCode will be much convenient. Low tech but works. :)

The following VSCode setup script helps me set up a new machine:

open PowerShell
copy extension names to the clipboard (yes, Ctrl+C)
run get-clipboard | % { code --install-extension $_ }

DavidAnson.vscode-markdownlint

eamodio.gitlens

ms-azuretools.vscode-docker

ms-python.python

ms-vscode-remote.remote-wsl

ms-vscode.cpptools

ms-vscode.powershell

npclaudiu.vscode-gn

yzhang.markdown-all-in-one

Docker Image build clean up using PowerShell on local PC

2020-02-03T22:56:00.000-05:00

When I run the docker image to do the debug on PC, I was struggling to keep the hard disk space consumption low and running the recent binary in "Docker on Windows Desktop". This development stage, so I won't consider the Azure cloud for hosting.

I need a script to reclaim hard disk, remove the currently running container, build and run the new image in docker on windows desktop.

The following code first removes unused images and then gets the image running on the port 5555. I will have to play the string replace tricks using PowerShell. Please check "-replace" part for details.

#prune unused image

docker image prune -f

#remove existing running docker image

$a = docker container ls --filter expose=5555

$a2 = $a | foreach-object {$_ -replace '\s{3,}', ' '} | foreach-object {$_ -replace ' ', "`t"}

$obj = ConvertFrom-Csv -InputObject $a2 -Delimiter "`t"

if ($obj -ne $null)

{

docker rm -f $obj.names

}

else

{

Write-Verbose "cannot find running container image so won't remove"

}

#build flask app and set its version to 1

docker build -t flaskapp:1 .

docker run -d -p 5555:5555 flaskapp:1

#test running instance

sleep -s 4

invoke-restmethod "http://localhost:5555/api/health"

SelfNote: Setup Spacy Module in Visual Studio

2020-01-19T10:26:00.002-05:00

I guess not so many people use Visual Studio doing NLP with Python. I have been searching for a solution on how to add a Spacy module in Visual Studio. I kept getting cannot find en_core_web_sm module error and the only solution I could find is to run python -m spacy download en_core_web_sm. I was struggling to find where in the UI to run the Python command. And finally, the solution came.

you noticed the "Admin" icon? yes, that is the key to solve the problem. With admin permission, you can open PowerShell and run python -m spacy download en_core_web_sm.

I installed en_core_web_sm, but it shows a warning message. And the en_core_web_lg module can eliminate the warning.

Standford NLP Quick Setup on Win10 with WSL

2020-01-06T14:53:00.001-05:00

After setting up the Stanford NLP on my PC, I was struggling with how to run it faster. Then I ran into Windows Subsystem for Linux (WSL). I realize that the modification of Stanford NLP is not an option for me. I can use WSL to quickly start an NLP web service and start my work.

I ordered a more powerful VM from Azure and use PowerShell to set up the NLP environment.

Enable WSL

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

It will trigger a reboot if WSL is not enabled
Add Ubuntu disco to WSL

# download ubuntu 18.04 as save it as Ubuntu.appx at local directory
Invoke-WebRequest -Uri https://aka.ms/wsl-ubuntu-1804 -OutFile Ubuntu.appx -UseBasicParsing

# add Ubuntu.appx to WSL
Add-package Ubuntu.appx
Download Standard NLP zip file and unzip it to the current folder

# download the Stanford NLP and save the zip file locally as "corenlp.zip"
Invoke-WebRequest -uri http://nlp.stanford.edu/software/stanford-corenlp-full-2018-10-05.zip -outfile corenlp.zip -UseBasicParsing

# unzip the corenlp.zip
Expand-Archive corenlp.zip -DestinationPath .\CoreNlp\
install Java in WSL. Since Stanford NLP does not require Oracle Java, so I use Open Java to make the command shorter

wsl sudo apt-get update
wsl sudo apt-get install default-jdk

I go into WSL from PowerShell and launch Stanford NLP from WSL.

go to WSL from Powershell by using "wsl"
go to the folder which stores the unzipped Stanford NLP files in step 3
run java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer

Open Edge and go to http://localhost:9000/, it will show similar UI like http://corenlp.run.

The PowerShell script used to access the localhost 9000 ports are listed below:

$data = "The quick brown fox jumped over the lazy dog."
$url2 = 'http://localhost:9000/?properties={"annotators":"tokenize,ssplit,pos,lemma,ner, entitymentions,depparse,parse,relation,openie,dcoref,kbp","outputFormat":"json"}'
$r = Invoke-RestMethod -Uri $url2 -Method post -Body $data

The annotators are listed here in case you need it.

6-Year-Old Rule?

2019-09-02T15:48:00.000-04:00

The story begins recently I am indulging in the quantum computing programming whose foundation involves a lot of linear algebra. The direct consequence is I want to solve almost all problems using the complex number, vector, and matrix. The Mars Rover problem got my attention.

The rover on the Mars surface only takes L, R, and M instruction. L means turn left. R means turn right. M means move 1 step forward. If the initial position is (0, 0) and facing North, what will be the location and facing direction after receiving couples of instructions?

This is a perfect opportunity to practice my math, I thought. The direction and location are all complex numbers. I can manipulate numbers to make the code concise! I convert the L, R, and M instruction to complex number op:

turn left L is complex (0, 1) where the real part is 0 and the imaginary part is 1
turn right R is complex (0, -1)
move forward M is complex number (1, 0)

The new location and direction can be computed like the following:

new location += op.Real * Direction //L and R's real part is 0, so it does not move location
new direction *= op //M is 1, so it has no effect on the direction

It looks cool and code is concise. I avoided those unpleasant switch statements.

Everything looks satisfying until I realize those expensive multiply operations might be a performance problem. I wrote a simple solution using those switch statements. The simple solution doubled the code. However, the performance is 30+% faster than my "elegant" one!

I told my wife what was going on when I encountered her in the kitchen, with a little bit of frustration. She smiled and pointed my 6-year-old son. "If the solution can be understand by him, it must be faster." she said, "It requires a smaller brain so it runs faster."

Well, both algorithms are O(n). Maybe next time, I should think on the big O level.

Old stuff: my first IEEE paper on Computational Intelligence

2019-03-03T11:27:00.000-05:00

Before I finished my college, I managed to publish a paper on evolutionary computation (EC). The paper is on the encoding part. My next task is to use quantum to rewrite my algorithm. Well, there is a quantum genetic algorithm (QGA) there. However, I think with the latest decision on Q# and quantum algorithms. The paper link is here.

Microsoft Q# Day 1: Qubit Collapsed

2019-01-19T16:19:00.000-05:00

After going through the quantum linear algebra, I am going to see some quantum concept by using Q#.

The first one I want to prove is measurement will collapse the qubit. I will use the AssertProb to verify the value with probability. The pseudo code is listed below:

use H gate to set the qubit to superposition
check the probability to 1 equals 0.5 (50%)
measure the qubit
check the qubit equals the measured value equals to 1 (100%). The qubit collapsed.

the code listed below:

namespace Quantum.QSharpApplication2
{
    open Microsoft.Quantum.Primitive;
    open Microsoft.Quantum.Canon;
 open Microsoft.Quantum.Extensions.Convert;

    operation Operation1 () : Unit
    {
        body(...)
        {
            using (ancilla = Qubit())
   {
    H(ancilla);

    AssertProb([PauliZ], [ancilla], Zero, 0.5, "error", 1e-5);
    AssertProb([PauliZ], [ancilla], One, 0.5, "error", 1e-5);

    mutable gate = PauliZ;
    let mX = Measure([gate], [ancilla]);
    AssertProb([gate], [ancilla], mX, 1.0, "error", 1e-5);

    Message(ToStringD(mX == Zero ? 0. | 1.));

    Reset(ancilla);
   }
        }
    }
}

Microsoft Q# is my choice

2018-12-26T21:59:00.000-05:00

Today I am thinking to dig into the real quantum programming after learning the foundation part. There are three big players I was looking at on quantum programming framework/language. I hate to waste my time learning the framework or language itself. I want to focus on so I can focus on solving the problem. I have three choices:

IBM, IBM Q
Google Cirq
Microsoft Q#

It seems IBM Q and Cirq are reusing the Python to provide a framework while Q# is a brand new language. It makes sense IBM and Google use Python as base as it has a good math library. However, I doubt this choice is the best choice. Like the GPU, quantum is special hardware. Cuda is a like C language but it has its own features. For quantum computing, it has a totally different way to compute. Therefore, I think the best way to use a new language, maybe it is just a DSL. C++ can be used to write a functional program, but the best way is still to use a functional programming language.

Also, the examples from those three companies show a huge difference. Q# provides the best sample pack so far. It has a quantum foundation, matrix computation, and programming language. It will help me a lot to adopt this new stuff.

The setup and debugging is the last thought. I had tried to step away from Microsoft stack; however, the debugging and the setup needs more time I expected. I was spoiled by double click and just working.

Anyway, the current Q# shows good progress and the watch, star, and pull request number is bigger than other players. I will continue to focus on the quantum basics, like matrix and math part, anyway.

SelfNotes: Quantum Algorithms

2018-12-23T21:27:00.000-05:00

I believe the quantum computing is the key to enable real AI. No matter how complicated the AI algorithm can be, the fundamental computing power will make a real difference.

The huge difference between a classical computer and a quantum computer is so intriguing! It is the first time that math can come back to computer science and guide its evolution. Also, the quantum algorithm is the first time it introduces the internal parallelism. Those existing algorithms barely considered parallelism when it is designed. Now the quantum can change this situation!

The following are some notes about quantum algorithms. They are not related to any language implementation, it is just pure math. The only tool I want to use is "that piece of meat between my ears". :)

Automation with NodeJS-Puppeteer & Visual Studio Code

2018-12-08T13:58:00.000-05:00

I saw many attempts to automate the processing using commercial software. However, I do not see this is necessary if this task is not that huge. When more and more application is moving to the web-based. Automation of the Chrome browser will automate most of the scenarios. Therefore, I feel 90% of those cases won't need to pay and still get the job done.

First, we need to set up IDE. There are three files needs to be updated and placed in the place. Please make sure you put launch.json and tasks.json are under .vscode folder.

launch.json: it is used to launch the application with predefined tasks such as Typescript build.
tsconfig.json is to used to build the application using Puppeteer
tasks.json is to define the Typescript build process

The content of these three files

launch.json

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "launch",
            "name": "Launch Program",
            "sourceMaps": true,
            "preLaunchTask": "TSC",
            "program": "${workspaceFolder}/app.ts",
            "outFiles": [
                "${workspaceFolder}/**/*.js"
            ]
        }
    ]
}'

tasks.json

{
    // See https://go.microsoft.com/fwlink/?LinkId=733558
    // for the documentation about the tasks.json format
    "version": "2.0.0",
    "tasks": [
        {
            "label": "TSC",
            "type": "typescript",
            "tsconfig": "tsconfig.json",
            "problemMatcher": [
                "$tsc"
            ],
            "group": {
                "kind": "build",
                "isDefault": true
            }
        }
    ]
}

tsconfig.json

{
    "compilerOptions": {
        "target": "es5",
        "module": "commonjs",        
        "lib": ["es2015", "dom"],
        "sourceMap": true
    },
    "include": [
        "src",
        "node_modules/@types/puppeteer/index.d.ts"
      ]
}

The node_modules folder is created from the npm command provided by the NodeJS framework. The command to run is listed the following. The first line is to install TypeScript compiler which is not installed by default with VS Code. The second line is to install Puppeteer. The last line is to install the library for Typescript to work with Puppeteer.

1
2
3

npm i -g typescript
npm i --save puppeteer
npm i --save-dev @types/puppeteer

The last file is the Puppeteer file:

import * as puppeteer from 'puppeteer'
 
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('http://www.google.com');
  await page.screenshot({path: 'example.png'});
 
  await browser.close();
})();