my experiments with so called "truth"

How a language model learns: pretraining and fine-tuning — Part 2

2025-10-01T21:36:00.000-07:00

A language model can get a training target from the same text it is reading. Move the text over by one token: each input position now has a next token to predict. No one has to write a separate answer for every prefix. The catch is that the model must not be allowed to peek at that answer.

The text already contains a training target

Consider the sentence “The kettle is on the stove.” As before, we will pretend each word is one token and leave out punctuation. A real tokenizer may split it differently.

After “The”, the next token in this sentence is “kettle”. After “The kettle”, it is “is”. Each position gives us a prediction to compare with the token that actually follows.

The input and target sequences are offset by one position. Word-sized tokens are a simplification for this example.

This is why we can train a language model from ordinary text without manually writing an answer for every prefix. The text supplies the target.

However, later tokens are present in the training sequence. If the position predicting “kettle” could read “kettle” from a later position, the exercise would give away the answer. The causal-attention rule from part one prevents this: each prediction can use only its own and earlier input positions. We can therefore calculate predictions at several positions together without allowing them to read ahead.

“Stove” is the target for this particular sentence. “Table” might also be a reasonable continuation in another sentence. The training target records what came next in the example; it does not establish the only acceptable continuation in the language.

Measure the prediction, then update the parameters

Suppose the model assigns “stove” a probability of 0.1 after “The kettle is on the”. Another set of parameters assigns it 0.5. For this example, the second prediction fits the observed next token better.

We need a number that expresses that difference. For next-token training, cross-entropy loss penalizes a low probability for the target token. At a single position, the loss is the negative logarithm of that probability. We do not need to work through the logarithm here: the useful relationship is that assigning more probability to the target gives a lower loss.

Those probabilities are illustrative, not measurements from a training run. In practice, we combine losses across the training positions we are evaluating, usually over a batch of several sequences.

The model’s parameters include the embedding values and the weights in its layers. Backpropagation calculates gradients: the local sensitivity of the loss to changes in each parameter. An optimizer uses those gradients to choose an update. In a simple gradient-descent update, it moves parameters a small step in the direction that locally reduces the loss; the learning rate controls the size of that step. Then we process another batch and repeat.

Training changes parameters. Evaluation measures a fixed version of the model on separate data.

One update does not guarantee a better answer to every prompt. Parameters are shared across many examples, and training involves repeatedly adjusting them using many batches. A lower loss on one batch can also come with worse predictions on other examples.

What pretraining gives us—and what to check

When we train on a broad collection of text in this way, we call the stage pretraining. Starting from initialized parameters, the model learns patterns that help it predict tokens across those examples.

Training loss tells us how well it fits the examples used to update it. We also need separate, held-out data that does not contribute to those updates. Otherwise, a model that remembers training examples can look better than it is at handling unfamiliar text. If training loss keeps falling while held-out loss rises, that is one sign of overfitting: the improvements on the training examples are not carrying over. When we use a validation set to choose settings or when to stop training, we should keep a separate test set for the final assessment.

Even a good held-out language-model loss does not answer every question about the model. If we want reliable summaries, we must check summaries. If we want a classifier, we must evaluate its labels. Predicting text well and meeting the requirements of a particular application are related, but different, things to measure.

Data quality matters throughout. Training on a poor target does not make that target correct; the loss still rewards predicting it.

Fine-tuning gives the learning a more specific purpose

After pretraining, we can start with the learned parameters and train further on examples suited to a task. That is fine-tuning. Two examples make the distinction useful.

For classification, suppose we want to sort messages into categories. The message “Your parcel arrives tomorrow” could have the label “delivery update”. One approach adapts the model with an output layer that produces scores for the available labels, then trains on message-and-label examples. The desired output is a category. This is the classifier-head approach; it is not the only way to use a language model for classification.

For instruction fine-tuning, the examples contain requests and desired text responses. A request might be “Rewrite ‘Send the file’ more politely”, paired with “Could you please send the file?” The model still predicts text tokens. During training, a prediction for a response token can use the request and the earlier response tokens provided by the example. At generation time, it must continue using the response tokens selected so far. The training examples teach it patterns for responding to requests, rather than simply continuing whatever kind of text happened to come before.

These are two adaptation paths, not successive requirements. The messages and responses are invented teaching examples.

The difference is more than the amount of data. The starting parameters, examples, and desired outputs determine what we are training the model to do. A classifier head produces label scores; an instruction-following language model produces token scores used to generate a response.

Fine-tuning does not guarantee correct facts or perfect instruction following. We still need evaluation that matches the intended use. Other post-training methods, including learning from preferences, also exist; their mechanics are beyond this introduction.

What does building “from scratch” mean?

There are several pieces of work we should distinguish. We can implement an architecture so that it performs the calculations described in part one. We can train newly initialized parameters using the loop in this article. Or we can load pretrained parameters and adapt them, avoiding the need to repeat all of pretraining.

A small teaching model can make these pieces visible. We can inspect the inputs, predictions, loss, and updates without needing to understand a large production training system first. Training a capable large model adds much greater demands on data, compute and evaluation. Writing the architecture is only one part of that effort.

The same sentence plays two roles. During generation it is context for a prediction. During training, its following tokens also provide the answers used to calculate loss and change the weights. The update step is what turns prediction into learning.

Sources & references

This explanation of training and fine-tuning draws on Sebastian Raschka’s Developing an LLM: Building, Training, Finetuning. Credit for the teaching material belongs to him; the examples and diagrams here are my own explanations.

Inside a language model: from text to the next token — Part 1

2025-09-24T10:46:00.000-07:00

A language model can use a sentence you just typed without learning new weights from it. The prompt changes the calculation's input. During ordinary generation, the model's learned numbers stay fixed. To see how that works, give it the words “The kettle is on the” and follow one prediction.

We will follow a model in the GPT family, short for Generative Pre-trained Transformer. It generates text using already-learned parameters: the numbers its calculations use. We will trace one prediction through that model, then see how repeating the step produces more text. The second article explains how training learns those numbers.

Start with one prediction

After “The kettle is on the”, “stove” is a possible continuation. So is “table”. The model assigns scores to the possible next tokens, which we can turn into probabilities. Choosing a token from those probabilities is a separate step.

A token is a unit of text defined by a tokenizer. It might be a word, part of a word, or punctuation. To keep our example readable, we will pretend each word is one token and omit punctuation. This is a teaching simplification, not the output of a real tokenizer.

The immediate task is small: predict one more token using the text available so far. Repeating that task is how we get a longer answer.

Text becomes numbers

A tokenizer converts text into token IDs. These integers identify entries in a vocabulary. Their numerical order does not tell us anything about meaning: token 400 is not twice as meaningful as token 200.

The model uses each ID to look up an embedding, a vector, or ordered list, of learned numbers. We can think of the embedding table as one row per vocabulary entry. Looking up a token selects its row; it does not yet account for the surrounding sentence.

The IDs and vectors here are made up for illustration. A real tokenizer may split the words differently.

The model also needs information about order. In the GPT-style architecture we are following, it adds a learned position vector to each token’s embedding. Other architectures can represent position differently. Here, the result is a sequence of vectors carrying both token and position information.

Those vectors are the input to the transformer blocks. The text has become something we can calculate with.

Let each position use context

The embedding for “the” starts from the same vocabulary entry whenever that token occurs. But “the” after “The kettle is on” should contribute to a different prediction than “the” in another sentence. Its representation needs context.

Attention provides a way to combine information from different positions. For each position, the model calculates weights for the positions it can use, then combines information from them according to those weights. The weights depend on the input and learned parameters. For our final “the”, attention can bring in information from “kettle”, “is”, and “on” before the model predicts what comes next. That is a description of the available context, not a claim about measured weights in a real model. We do not hand-code a rule saying that a kettle must attend to a stove.

There is a restriction: each position can use itself and earlier positions, but cannot use later ones. This is causal attention.

Filled cells mean access is allowed. They do not show measured attention weights or how much a token matters.

For example, the third position, “is”, can use “The”, “kettle”, and itself. It cannot use the later “on” or “the”. The final position can use the whole prefix. This visibility rule also matters during training, which we will see in part two.

Multiple attention heads perform several learned combinations of context. Their results are combined before the model continues. We do not need to assign a fixed job such as “grammar head” or “meaning head” to each one.

Attention is only part of a transformer block, a group of operations repeated through the model. A feedforward network applies a learned transformation to the representation at each position. Attention has mixed information across positions; this network processes the resulting vectors one position at a time.

Residual connections add a block component’s input back to its output, so the next operation receives both the carried-forward representation and a learned change to it. Layer normalization centers and rescales values within a representation, with learned scale and offset. Stacking blocks lets later calculations work with representations already shaped by earlier ones.

Turn the result into a next token

At the end of the model, an output layer maps the final position’s representation to one score for every token in the vocabulary. Those scores are often called logits. A function called softmax converts them into nonnegative probabilities that sum to one.

The generation program then chooses a token. It could choose the highest-probability token, or sample from the distribution. Sampling gives lower-probability choices a chance too, so the same prompt can produce different continuations.

A probability here expresses the model’s prediction of a continuation. It is not a probability that a statement is factually true. A fluent continuation can still be wrong.

Repeat the step to get more text

Suppose the selected token is “stove”. The program appends it to the context, then asks for the next token using the extended sequence. It keeps doing this until a stopping condition, such as an end token or a length limit, is reached.

This is the conceptual data flow. Implementations can reuse intermediate calculations instead of recomputing everything on each step.

When “stove” is appended, the next prediction has a different input even though the model's parameters are unchanged. This is how a conversation can influence the answer without retraining the model. If we remove that context on the next call, we cannot assume the model will retain what was said.

The remaining question is where the parameters came from. A model with random numbers can perform these calculations too, but its predictions will not yet be useful. Part two follows the training process that changes those numbers.

Sources & references

This two-part introduction draws on Sebastian Raschka’s Developing an LLM: Building, Training, Finetuning. Credit for the teaching material belongs to him; the examples and diagrams here are my own explanations.

How an AI agent works

2025-01-11T17:27:00.000-08:00

Suppose you have a folder of notes from a side project. Somewhere in those notes is the reason you chose SQLite over Postgres. You remember discussing it, but you cannot remember which file contains the decision.

You could search the folder yourself, read a few files, and put the answer together. What would it take to let a language model do those steps?

The model can write a request to search. That request is still just output from the model; it has not searched a single file. Our application must run the search and send the results into another model call. Following that handoff is a useful way to understand an agent.

For this example, watch what changes after each tool result: the model has new information for choosing its next action.

Give the model a way to look

For this example, we will design two tools:

search_notes(query)
    Returns matching file names and short excerpts.

read_note(path)
    Returns the contents of one file in the notes folder.

A tool is a capability exposed to the model through an interface. We describe what it does, what arguments it accepts, and what it returns. The implementation might be an ordinary function that searches text files.

For a tool run by our application, the model produces a structured request. Our code checks it, executes the function, and sends the result back. The model does not gain access to the filesystem merely because we described a tool. The application supplies that access.

We can give our notes agent a short instruction:

Answer questions using the project notes. Read the relevant files before drawing a conclusion. Name the files that support your answer. If the notes do not establish an answer, say what is missing.

That tells the model how we want it to work. The tools give it the means to investigate.

Follow one possible run

Here is an illustrative sequence using fictional notes, rather than a recorded model run.

The user asks:

Why did we choose SQLite for the first version?

The model might request a search. In a simplified representation, its response could look like this:

{
  "tool": "search_notes",
  "arguments": {"query": "SQLite"}
}

Our search function returns two matches:

storage-options.md
    "SQLite and Postgres are both candidates ..."

prototype-decision.md
    "For the first version, use SQLite ..."

The model now has something it did not have before: file names and excerpts from the actual search result. It can request read_note("prototype-decision.md") to get the surrounding explanation.

Suppose that fictional file says:

The prototype runs on one laptop. Use SQLite so setup does not require a separate database service. Revisit this choice if we need several machines writing to the same shared database.

The answer can now explain both the choice and the condition for reconsidering it. It should identify prototype-decision.md as the source.

Notice how narrow that conclusion is. These notes support a decision about this prototype. They do not establish that SQLite is always preferable, or that Postgres would have been slow. The result gives the agent evidence to work with; it still has to interpret that evidence correctly.

The program keeps the loop moving

The next model call receives both the earlier conversation and the tool result. There is no separate step in which the model reaches into our folder: every file read happens through the application.

The application repeats the exchange until it receives an answer or reaches a stopping condition:

Send the question, instructions, and available tools to the model.

Repeat within a fixed step budget:
    Get the model's response.
    If it requests a tool:
        Validate the request and check access.
        Run the tool.
        Add the request and its result to the conversation.
    If it returns a final answer:
        Return that answer and stop.
    If the response cannot be used:
        Report the failure and stop.

If the budget runs out, report that the task is incomplete.

This is pseudocode showing one tool request at a time, not a runnable implementation. The step budget limits how many model responses we process. An implementation also needs timeouts for individual calls and handling for rejected or failed tool requests. Returning an answer ends this loop; it does not prove that the answer is supported by the notes.

The loop is what lets the next action depend on what just happened. An empty search result might lead to a different query. Two conflicting notes might lead to another file being read. A file that answers the question might be enough to stop.

This repeated use of actions and observations also appears in the ReAct research, published in 2022. Our example focuses on the visible requests and results needed to understand the execution.

What makes this an agent?

People use the word “agent” for different kinds of systems. A useful distinction is who chooses the next step.

We could write a fixed workflow that always searches once, reads the first three results, and asks for a summary. In our agent example, the model chooses the query and which files to read, using the results it receives. The application still determines which actions are allowed.

For a predictable task, the fixed workflow may be easier to test and sufficient for the job. The extra flexibility becomes useful when the next step depends on information we have not seen yet.

What does it remember?

In this design, our program keeps a conversation containing the question, tool requests, and results. It supplies that context on later model calls. That is how the model can use an earlier search result when choosing a file to read.

If the notes are large, we need to decide how much to return. Short excerpts help with finding files; full contents help with understanding a decision. Sending every file on every turn would make the search tool rather pointless.

Remembering information across separate sessions would require another choice about what to save and retrieve. We do not need that feature to explain this first example.

A small loop still needs boundaries

For this notes agent, I would start with access to one folder and no ability to modify it. The application should resolve each requested path, including symbolic links, and reject anything outside that folder. A prompt asking the model to stay within it is not a filesystem access check.

The notes themselves are evidence to read. A line in a file telling the agent to ignore the user or run a different tool should not grant it new authority. Keeping the tools read-only limits what this first version can change, though it does not guarantee a correct answer.

I would also make failures visible. If a file cannot be read, return an error that identifies the problem. If the step budget is exhausted, return an incomplete result rather than treating the last generated sentence as success.

Even a normal final answer needs scrutiny. Did it use the decision note, or only the list of candidates? Did it mention the condition for revisiting the choice? Can the reader open the file it cites?

Try removing prototype-decision.md from the test folder while leaving storage-options.md. The agent can still find both database names, but it no longer has the decision. Does it report the missing evidence, or turn a list of candidates into a confident answer? That small test checks whether the loop is helping it investigate.

Sources & references

OpenAI’s June 2023 function-calling announcement explains the request-and-result exchange for application-executed tools.
Shunyu Yao and colleagues’ ReAct paper explores interleaving reasoning and actions.

Understanding AWS CDK, one stack at a time

2022-06-16T20:41:00.000-07:00

In one sentence: With AWS Cloud Development Kit (CDK), we write a program that produces a CloudFormation template. CloudFormation reads that template and creates or updates the resources in AWS.

Keep that split in mind: the CDK application describes the infrastructure; CloudFormation builds it. The CDK command-line tool connects the two.

1. The problem

Say we need a network, a load balancer, a Postgres database and an application running in Amazon Elastic Container Service (ECS). We have three ways to describe the setup.

Configure it in the AWS console. We can create the resources, but repeating the setup and reviewing changes means keeping track of the individual settings ourselves.
Write a CloudFormation template by hand. The setup becomes a file we can review and reuse. As we add roles, security groups, logs and connections, that file grows.
Write a program that produces the template. That is CDK. We get functions, loops and types, and the output is still a file CloudFormation understands.

For example, beta and production might use the same arrangement with different sizes. A program can express the shared arrangement once and take the size as an input.

2. Start with CloudFormation

Before looking at CDK, we need three ideas.

A template is a description in JSON or YAML: “I want a bucket, a queue and a role that can read the bucket.” Each resource has a type, properties and a logical ID identifying it within the template.

A stack is the group of resources CloudFormation manages using that template.

CloudFormation is the builder. We give it a template to create a stack. Later, we submit a revised template to update that stack.

Put the old and new templates beside each other:

Old template	New template	What the change means
Bucket A	Bucket A, same properties	No template change for A.
Queue B	Queue B, changed properties	Update B; the property rules determine whether replacement is needed.
Service C	No C	Remove C from the stack, following its deletion policy.
No D	Role D	Create D.

The important idea is that removing a resource from the description is itself a change. On the next deployment, CloudFormation acts on that removal. Depending on its policy, the resource may be deleted or retained.

The template does not continuously control AWS on its own. Editing a file on our laptop changes nothing in the running stack until we deploy it.

3. CDK draws the blueprint

Think of the CDK application as an architect's program that draws a blueprint. CloudFormation is the builder who receives it. Creating an object in that program is not the same as creating its resource in AWS.

CDK calls those objects constructs. They form a tree: an app contains stacks, and each stack contains constructs describing its infrastructure.

Each stack produces a template. An app with several stacks can produce several templates.

Constructs come at different levels:

Level	What we write	Example
L1	A direct description of a CloudFormation resource. Class names start with `Cfn`.	`CfnBucket`
L2	A resource with defaults and helper methods.	`Bucket`, with `grantRead(role)`
L3	A pattern combining several constructs.	A load-balanced Fargate service.

For example, bucket.grantRead(role) creates the Identity and Access Management (IAM) policy needed to give that role read access. The generated policy refers to the bucket's Amazon Resource Name (ARN). We do not have to write those references by hand.

That also explains why one line of CDK code can add more than one resource to the template. The construct may describe a whole arrangement, including its supporting resources.

4. Follow one deployment from start to finish

For a TypeScript app that has been compiled, cdk.json might contain:

{
  "app": "node dist/app.js"
}

Now follow cdk deploy UploadsDemo.

Step 1. The command-line tool finds the program. It reads the app command from cdk.json and starts it.

Step 2. The program runs. It creates the app, stacks and construct tree in memory. A bucket constructor adds a description of a bucket; it does not provision one. The program can still read files or call other services. CDK resource lookups may query AWS and cache their results in cdk.context.json.

Step 3. Synthesis writes the output. The application produces a cloud assembly, normally in cdk.out. It contains a template for each stack and a manifest.json describing the assembly. Deployable files or Docker image sources can also be included as assets.

Step 4. The tool publishes assets when needed. Lambda code may need to go into S3; a container image may need to go into an Elastic Container Registry (ECR) repository. Bootstrapping sets up supporting resources, including asset storage and deployment roles, in the target AWS account and region. Those supporting resources can be updated later.

Step 5. CloudFormation prepares a change set. This previews the proposed operations: add, modify, replace or remove resources. It does not execute those operations just by showing them.

Step 6. CloudFormation executes the changes. It works through resource dependencies. A role needed by a task definition must be available before that task definition can use it. The terminal shows progress and failures. If an operation fails, rollback can try to undo the stack changes, but rollback can fail too and cannot guarantee recovery of stored data.

So an ordinary cdk deploy includes synthesis. It runs the application before applying the resulting description. cdk synth stops after producing that description.

5. Map the commands to the steps

Command	What it does
`cdk synth`	Runs the app and writes the cloud assembly.
`cdk ls`	Runs the app and lists its stacks.
`cdk diff`	Compares the new template with the deployed template, or a template we supply.
`cdk deploy <stack>`	Synthesizes, publishes assets when needed, and deploys the stack and its dependencies.
`cdk deploy <stack> --exclusively`	Deploys only the selected stack, skipping its dependencies.
`cdk destroy <stack>`	Asks CloudFormation to delete the stack, subject to resource policies.
`cdk bootstrap`	Creates or updates deployment-support resources in an account and region.
`cdk deploy --hotswap`	For supported development changes, updates resources directly through service APIs. Otherwise, falls back to a normal deployment.

deploy, destroy and bootstrap can change AWS resources. synth and ls do not perform the ordinary stack deployment, but running the app can still involve lookups or other service calls. diff against a deployed stack reads its template.

Run diff before deploying. Use it to review what changed, then inspect any proposed replacement or removal. A CloudFormation change set provides another view of the proposed operations. Neither a template diff nor a change set proves that execution will succeed.

Hotswap is a development shortcut. It bypasses CloudFormation for supported changes and introduces drift, so it should not be used for production deployments.

6. The surprises to understand

Removing something from code can remove it from AWS. A condition might leave a service out of the new template. If the old template contained it, the next deployment removes it from the stack. For data-bearing resources, inspect deletion and retention policies before proceeding.

A rename can change resource identity. CloudFormation uses logical IDs to track resources. CDK derives those IDs from construct paths. A variable name and a construct identifier therefore have different consequences.

In the small bucket example shown here, renaming the TypeScript variable leaves the template unchanged. Renaming the construct from Uploads to Documents changes the bucket's logical ID. CloudFormation sees the old entry disappear and a new one appear. Retaining the old bucket keeps its files; it does not copy them to the new bucket.

Some property changes require replacement. Keeping the same logical ID does not guarantee an update in place. The rules for the resource and property determine whether CloudFormation can modify it or must replace it.

Console edits can drift from the template. Changing a security group in the console does not update the file in our repository. A later stack operation may overwrite that change or fail because of it. A template diff alone does not fully inspect live drift.

Stacks can depend on each other. When two stacks in the same account and region share a resource value, CDK can produce a CloudFormation export in one and an import in the other. That export cannot be removed or changed while another stack imports it. Splitting infrastructure into stacks gives us boundaries, but also dependencies to manage.

7. Keep the whole path in view

The path is:

TypeScript → construct tree → templates → CloudFormation → AWS resources.

The first three stages produce the description. Deployment applies it. This is why we can inspect a CDK application's output without deploying the infrastructure.

For a small exercise, use CDK v2.28.0 to synthesize a stack containing a bucket. Open its template and find the logical ID. Rename only the variable and synthesize again. Then rename the construct identifier. Compare the outputs before considering a deployment: which edit changed how CloudFormation identifies the bucket?

Sources & references

AWS CDK v2 announcement.
CDK v2.28.0 command-line documentation.
CDK v2.28.0 core documentation, including constructs and stack dependencies.
CDK v2.28.0 S3 documentation, including bucket removal policies.
CDK logical-ID implementation.

groovy scripts for Jenkins

2014-06-09T01:14:00.003-07:00

At some point, if you are working on managing Jenkins for your team, you will find yourself looking for a set of useful groovy scripts to control jenkins configuration for a large no of builds/jobs.

Following are some mostly frequently needed ones

Get all jobs that belong to a view

 
hudson.model.Hudson.instance.getView('my-view')?.items.each() { 
    println it.fullDisplayName 
}

Clear the build queue

import hudson.model.*
def queue = Hudson.instance.queue
println "Queue contains ${queue.items.length} items"
queue.clear()
println "Queue cleared"

Disable jobs in a view

hudson.model.Hudson.instance.getView('1.1.x-tests')?.items.each() { 
   println("JOB : "+it.name)
   it.disabled=true
   it.save()
}

Get all jobs that that define a TimerTrigger with the trigger spec

 
import hudson.model.* 
import hudson.triggers.*
  
for(item in Hudson.instance.items) {
  
  for(trigger in item.triggers.values()) { 
    
    if(trigger instanceof TimerTrigger) {    
       println("--- Timer trigger for " + item.name + " ---")    
        println(trigger.spec + '\n')   
    }  
  } 
}

Disconnect and re-connect to slaves


import hudson.model.* 

Hudson.instance.computers.each { c ->
  c.disconnect()
  c.connect(true)
}

specs 2, Scala and maven with IntelliJ 12.1 and getting things right

2013-04-07T13:02:00.002-07:00

Recently, to learn a test framework for Scala ( even for Java) I chose specs2 . Its still difficult to move to SBT, largely due to the fact that I am yet to be comfortable using it. So, for the time being I will stick to maven.
To configure the scala/java project edit the pom.xml to include

    
     <dependencies>
        <!-- specs2 version I used : 1.12.3-->
        <dependency>
            <groupid>org.specs2</groupid>
            <artifactid>specs2_${scala.version}</artifactid>
            <version>${specs2.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
    </dependencies>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                     <jvmArgs>
                        <jvmArg>-Xms128m</jvmArg>
                        <jvmArg>-Xmx1024m</jvmArg>
                    </jvmArgs>
                    <scalaVersion>${scala.version}</scalaVersion>
                </configuration>
            </plugin>
     </build>

Include tests under src/scala/test . An ubiquitous example

  
import org.specs2.mutable._
import org.junit.runner.RunWith
import org.specs2.runner.JUnitRunner


@RunWith(classOf[JUnitRunner])
class MyTest extends Specification{



  "The 'Hello world' string" should {
    "contain 11 characters" in {
      "Hello world" must have size (11)
    }
    "start with 'Hello'" in {
      "Hello world" must startWith("Hello")
    }
    "end with 'world'" in {
      "Hello world" must endWith("world")
    }
  }
  // Update : by default tests are run in parallel, to execute them sequentially
  // add `sequential` before the spec e.g.

  sequential
  "My sequential spec" should {
     var a = 5
    "add 1 to a" in {
      a = a+1
      a mustEqual(6)
    }

    "multiply current a by 2" in {
      a = a* 2
      a mustEqual(12)
    }
  }

 // try removing `sequential` from the spec, it would fail randomly.
}

Thats it!. For running tests from IntelliJ simply right-click and run, and from cmdline execute mvn test

me, akka and scala

2013-03-12T12:09:00.000-07:00

I've been learning Scala recently, including through a Coursera course. I'm gradually moving from writing Java-like Scala toward thinking in terms of transformations on data.

What I like so far is being able to express operations on collections without spelling out every intermediate step. Shorter code helps when it makes the flow easier to follow, but it does not automatically make a program easier to maintain.

Recently, I came across Akka. It lets us organize concurrent work as actors that communicate through messages. Each actor handles one message at a time and can keep its own state. This gives us a way to separate responsibilities without having several workers directly update the same state. We still need to design how those actors coordinate.

I tried a word-count program using Akka and Scala. A mapper splits sentences into words, a reducer combines their counts, and an aggregator maintains the totals. The useful part of this exercise is seeing how work moves between actors while the running totals belong to one actor.

The example waits half a second before requesting the result. That gives the workers time, but doesn't guarantee they have finished. A useful next improvement would be to track completion explicitly, so the program knows when all the counts have reached the aggregator.

Sources & references

My word-count example on GitHub, using Scala 2.9.2 and Akka 2.0.5.
Akka 2.0.5: actors, state and mailboxes.

How to make most out of Jira?

2013-01-26T13:23:00.002-08:00

It has been quite a while (still < 1 year), I am using Atlassian Jira for bug/issue tracking withing my team of roughly 15. Previously, we were using a version of Redmine, doped to suit company environment.

I had read, heard a lot about Jira's abilities regarding how it can be useful to ease the project management and accurately measure the project's progress. But, recently under peer pressure of my group, where I was required to file some appropriate bugs/improvements/tasks, along with doing myriad of other things, I learned importance of using Jira.

How we can make most out it?

Well, Jira is no magic, but just a tool; rather, it provides a platform for task management. So, all task management is left to you. Here's how I did and worked well for me

create a high-level abstract issue of big task you will be working on and put in the initial estimates if you already know or leave it to the sub-tasks
divide the task into concrete sub-tasks with concrete estimates
and work !

That's it! Its that really simple. Instead of adding a bunch of many issues, this way of filing issues into Jira really helped me to work efficiently and convey accurate project progress to my mangers. At times, I also had to update the estimates for the tasks but earlier you do it the better it is. That was the overall gist of it.

getting android emulator to work normally on 64 bit linux

2013-01-26T13:00:00.002-08:00

Recently I started working on a nfc android app but I was stuck with very basic thing of getting emulator to work normally on my home laptop running Linux Mint. I tried numerous ways to configure eclipse, java paths, AVDs (android virtual devices), checking adb logcat etc but nothing really helped. The emulator could start but always hanged on the boot screen. (believe me, I even waited a complete day for it to boot up).

Thanks to StackOverflow again! As described in the comments here and also here, the Android team still has some more work to do until they make the emulator ( and other stuff) work with 64-bit version of java. But, I could happily run my android project with emulator ( which works quite fast now :) ) by simply configuring eclipse's default jre to be a 32-bit version of java 7. While 64 bit jdk being default for other things.

unique id generation at high scale

2013-01-20T04:15:00.000-08:00

Moving away from relational db allows us to work with huge data, which would otherwise be not feasible. But, it also mandates us to look into the issues that we didn't have to consider much about.

One of such thing was generating unique ids. With mysql you get auto-incremented ids which work absolutely fine for most of the use cases. But, for other systems, in context of nosql/distributed systems, that job is on system designer's shoulder. Inherently one-size fits all solution is difficult to come with.

There are many solutions proposed for this ubiquitous problem.

Twitter has developed snowflake which solves it. They guarantee id generation with the following constraints :

minimum 10k ids per second per process
response rate 2ms (plus network latency

The other solution I found is flake by boundary. It achieves the goal of generating ids with good performance and adequate bit size 128 which snowflake lacked.

I will update more information on it later, as I confront with more of these.

Migrating svn repository to git

2013-01-19T13:20:00.001-08:00

After googling I found this StackOverflow link. First, I needed to install git-svn module.

$ sudo apt-get install git-svn

Now we also need to map svn users to git users. we can get all svn users ( skip if you already know them) with this command ( execute under svn project root )

$ svn log | grep '^r[0-9]' | awk '{print $3}' | sort | uniq

Once you get all users, create a users.txt file mapping svn users to git

user1 = first_name last_lame <email@address.com>
user2 = first_name last_lame <email@address.com>

Once you make sure all of your svn users are mapped to git, then execute

$ git svn clone --stdlayout --no-metadata -A users.txt svn://hostname/path dest_dir

It will start fetching svn repository into git. If you run into other problems, like

Author: (no author) not defined in users.txt file

then simply add following line to users.txt

 (no author) = first_name last_lame <email@address.com>

Note : If, due to this error, git svn command stops then simply update users.txt file and resume the migration

$ git svn fetch

Now you can transfer your existing git repository to remote git url e.g.

$ git push --mirror git@github.com:username/project.git

Setting up Own Maven Repository

2012-08-17T22:36:00.001-07:00

 
 <settings>  
  <mirrors>  
   <mirror>  
    <!--This sends everything else to /public -->  
    <id>nexus</id>  
    <mirrorof>*</mirrorof>  
    <url>http://myhost.com:8080/nexus/content/groups/public</url>  
   </mirror>  
  </mirrors>  
  <profiles>  
   <profile>  
    <id>rakuten</id>  
    <repositories>  
     <repository>  
      <id>myprofile</id>  
      <name>My Internal Repository</name>  
      <releases>  
       <enabled>true</enabled>  
       <updatepolicy>never</updatepolicy>  
      </releases>  
      <snapshots>  
       <enabled>true</enabled>  
       <updatepolicy>always</updatepolicy>  
      </snapshots>  
      <url>http://myhost.com:8080/nexus/content/groups/public</url>  
      <layout>default</layout>  
     </repository>  
    </repositories>  
   </profile>  
  </profiles>  
  <activeprofiles>  
   <activeprofile>myprofile</activeprofile>  
  </activeprofiles>  
 </settings>

Getting cross domain Ajax to work!

2011-11-09T20:25:00.001-08:00

Last week I was working on a project which required me to use ajax to fetch the results from 3 different servers and use the result together to have a comparison side-by-side.

I had a little knowledge of Ajax, so I had to strive a bit for getting things to work. Here, I share my findings.

Problem: Using Ajax to fetch cross domain data.

What does it mean?

Suppose my current domain is blahblah.com and I want something from phewphew.com then I need to do a cross domain fetch with Ajax.

By conventions Javascript doesn't allow to fetch such cross domain data. It involves security issues. Anyways, I didn't go much into that !!

Quick solution 1 :

One solution to this problem is to use a custom servlet ( or php, whatever you use, it will act as a proxy ) /JSP page that downloads the JSON ( or any other type ) and simply reproduces it. In that case, you will put your Ajax calls to a page on same domain and that will work perfectly fine.

Solution 2 :

Another solution, if using JQueries, using JSONP would help retrieving JSON data easily.

NOTE: These were really abstract solutions, I would try to illustrate it with example code soon .

How To: Successfully set up Amazon EC2 micro instance

2011-08-05T13:06:00.000-07:00

Amazon Web Services is a very rich arsenal of web services. Surprisingly, using them is really easy and simple. I tried my first hands on Amazon's Elastic Cloud, in layman's terms, it is a server
I tried to set up a AWS micro instance on Amazon EC2. The setup is very simple and we can set a system running in just few minutes. I could also quickly set up a tomcat server running on it.
I will quickly list down the steps that I followed.

(1) From EC2 console launch a new Instance. For quick start select any AMI ( Amazon Machine Image) that suits your need. It can later be changed or upgraded.

(2) Once an AMI is selected, fill in the details related to instances (Micro instance's usage falls under free tier for the first year), kernel, ram, monitoring details etc.

(3) Next, create a key-value (optional)

(4) Select proper security group. (This is also important to keep your instance secure. Once I got my instance hacked due to careless firewall settings ). Don't allow many ports to be open, only the few that you intend to use for your need.

(5) Review everything and click 'Launch' and done!! Your instance will be ready to use within few minutes.
(6) Create a new key-pair and download the private file.

(7) Once the status of the instance turns 'running' it can be used to connect.

$ ssh -i aws.pem ec2-user@ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com

(8) This is it!! Time to have fun!!

A note on dictionaries in Programming languages

2011-05-08T00:56:00.000-07:00

I recently started with learning python. The good thing about python is that everything in python is dynamic. Python is dynamically typed language. What does it means?
In C/C++ or Java we need to define the type of a variable before using it.

int a; 
float b;
// even Objects need to be defined in
Matrix m1;

But, python is different in this regard. Here, we can use a variable and its type is decided on first assignment.

list  # not needed but just for explanation

list = [] # creates an empty list

Thus, Python decides the type of a variable on its first assignment making it dynamically typed. But, another thing to be careful with in it is that Python is also strongly typed language. Means, once assigned we cant change the type of that variable. In, above python code once list varible is assigned an list object it cant be assigned any other data type. It remains list.

Below, is the summarization of different types of languages based on the property we just discussed.

statically typed language

A language in which types are fixed at compile time. Most statically typed languages enforce this by requiring you to declare all variables with their datatypes before using them. Java and C are statically typed languages.

dynamically typed language

A language in which types are discovered at execution time; the opposite of statically typed. VBScript and Python are dynamically typed, because they figure out what type a variable is when you first assign it a value.

strongly typed language

A language in which types are always enforced. Java and Python are strongly typed. If you have an integer, you can't treat it like a string without explicitly converting it.

weakly typed language

A language in which types may be ignored; the opposite of strongly typed. VBScript is weakly typed. In VBScript, you can concatenate the string '12' and the integer 3 to get the string '123', then treat that as the integer 123, all without any explicit conversion.

line-by-line processing : Linux Bash

2011-05-06T23:06:00.001-07:00

Often we require to process a file content line by line. Linux inherently supports this thorough its powerful shell commands. Lets walk through some of the ways to parse a file line-by-line.

First, we can parse a file by catting a file and piping the file output to a while read loop.
The following bash script produces it own code as output. Isn't it interesting?

code for dumping own file content

#!bin/bash

cat $0 | while read line
do
echo $line;
done

Secondly, using file descriptors we can play a lot with it. But, before we will learn about firl descriptors in brief.

Under unix/linux OS, files are referenced, copied, and moved by unique numbers known as file descriptors.

0	stdin	It is(devie) usually the keyboard or mouse from where the input comes.
1	stdout	It is where output is redirected e.g. screen or file
2	stderr	It is where the standard error messages are routed by commands, programs, and scripts.

$ some_command 2>&1

this means, the command sends all of the error messages to the same output device that the standard output goes to, which is normally the terminal.

Now, we will use this file descriptors to write the same shell script we wrote before.( which outputs its own code)

#!bin/bash

exec 3<&0  #redirect standard input to file descritpor 3

exec 0<$0 # redirect current file content to standard input

while read LINE   # simply read from standard input now
do 
echo $LINE;
done

exec 0<&3  # revert back stanrd input from 3 to 0

Upgrading Ubuntu to Natty Narwhal

2011-05-06T12:37:00.001-07:00

I recently upgraded Maverick Meerkat to Natty Narwhal. The new ubuntu is packed with Unity for default desktop session. The Unity brings a lot of improvements with it. Along with improved UI and perceivable improved speed we could see a new scrollbar which takes less space and looks nice. Real gift to Ubuntu lovers!! worth a try !!!

Removing packages from linux (ubuntu)

2011-02-08T06:37:00.000-08:00

I tried to remove the packages from maverick (Ubuntu 10.10), today..and found some good ways to do it

(A)First search the package :

The traditional way:

dpkg -l | grep partial-name-of-package-to-be-removed

Using dpkg command ....more conspicuous way

dpkg -l '*partial-name-of-package-to-be-removed*'

For example, you want to remove jdk then

dpkg -l '*jdk*'

(B)There are two ways to remove packages:

Using apt command : ...the better way...

After getting the list of packages choose the appropriate one and run following command

* To remove packages:

sudo apt-get remove package-name

* To remove packages and dependencies:

sudo apt-get autoremove package-name

* To remove packages, dependencies and configuration files:

sudo apt-get purge package-name

Using dpkg command : ...the crude way...

* To remove packages:

sudo dpkg -r package-name

* To remove packages,and configuration files:

sudo dpkg --purge package-name

* To force removal, if got dependency error(this command turns errors into warning..):

sudo dpkg --purge --force-depends package-name

What is an Open System?

2011-02-06T07:06:00.000-08:00

I used 'Open System' term many times but today I got some time to think, in detail, about the very deep meaning of it. So here it goes...

An open system is one for which the architecture is not a secret. Just take example of TCP/IP system, which is Open. The TCP/IP protocols are same for every system. Whether its your laptop-PC using TCP/IP or your android smart-phone downloading apps from market. The architecture is same irrespective of the systems that use them. So this has many advantages, UNIX is a very good example of Open system software.

Install Vista / Win 7 from Pen Drive

2010-03-11T13:40:00.000-08:00

Here is a cool trick(in fact, a built in utility in vista/win 7) to make a pen drive bootable and use it to install vista or win 7 from it. What you are required to have is a pen drive (4 GB+) and vista/win 7 iso/DVD and a laptop or a PC. :P

So, if all is set we proceed, as follows:

First, open cmd prompt as administrator. Then type following commands one by one.
(Note: comments(#) are shown for the purpose of understanding and they are definitely not part of commands.)

diskpart
# Opens DISKPART utility built-in in vista/win 7

list disk
# Lists the available disks for installation

select disk 1
# select the disk 1 or 2 or 3 or so on.. i.e. pen drive disk which is there in the list

clean
# cleans the disk

create partition primary
# creates the primary partition

select partition 1
# does as specified

active
#....

foramt fs=NTFS
# format it with NTFS file system. it will take some time! :(

assign
# mounts the disk

exit
# get exit from diskpart utility

Don’t close the cmd prompt as we need to execute one more command at the next step. Just minimize it.
Now mount the iso using virtual drive manager like Power iso or insert dvd. We assume D: be the drive letter of it.

go to D:\BOOT folder from minimized cmd promp window.

Type another command given below to update the USB drive with BOOTMGR compatible code.

bootsect.exe /nt60 g:
# here g: is the assumed drive letter of pen drive. Change it, if different.

Copy your Windows 7/Vista DVD contents to the USB flash drive.
Now your pen drive is bootable.

Only thing you need to change the boot priority at the BIOS to USB from the HDD or CD ROM drive. I won’t explain it as it’s just the matter the changing the boot priority or enabling the USB boot option in the BIOS.

Why Study Computer Science/Engineering?

2009-08-14T16:07:00.001-07:00

Before I took Computer Science as my major in BTech, I barely could have wondered the reasons behind studying it. That time, it was the most lucrative major with good salaries and perks, and a obvious choice for me to make. But over course of 4 years, my view towards this field has changed significantly.
Computer Science requires you to have creativity(ability to devise), brainwave the situations (get clear insights of complex situations), thorough knowledge of the field and to some extent flair in problem solving. With the need of each new application, problem solving technique or opportunity comes the new range of problem solving solution sets. Computer Engineer's job here is to chose the best possible solution from the set to the need of situation. It is like intellectual stimulation. As stated its not a simple job that anyone can accomplish. It requires thorough analysis and exploration of resources going thorough different levels of details and abstractions ultimately leading to a perfect solution(not always though :P) which consist of an algorithm or a system/application. Just creating enhanced applications is not our job, but to maintain its integrity. Days of individuals working on computation are long gone. Today we work in groups,teams. Work is well divided and scheduled to achieve the best possible outcome as early as possible though efforts of the teams. It is a wonderful combination of theory and practice and thats what is the computer engineering is all about : providing effective solutions to the needs of society or industry.(Definition of engineering :P)
Now we come to the society benefits of it. Since computers forms the integral part of contemporary society (e.g., in transportation, medicine, engineering, economics, entertainment, technical theater, record keeping, insurance—the list seems endless), work in the computing field provides the opportunity to contribute to many aspects of society. Computing applications have great potential to help address human needs and improve the quality of life.
Computer science is inherently interdisciplinary. Many computing applications connect computer scientists with professionals in many disciplines. Complex systems draw upon multiple subjects and perspectives, so most computing folk work with a diverse range of people.