Cloud Architecture and Technology Blog

MCP Gateway: The Missing Proxy for AI Tools

2025-06-08T09:50:00.003+03:00

AI agents and tool integration are exciting — until you actually try to connect them. Different authentication systems (or none), fragmented documentation, and incompatible protocols quickly turn what should be simple integrations into debugging nightmares.

I've recently released ContextForge MCP Gateway as Open Source to solve this problem by sitting between your AI clients and tool servers, giving you an open source, clean, secure endpoint for everything — and supports both REST and MCP upstream and downstream protocols. If you find it useful or interesting, leave a star on GitHub!

The MCP Integration Problem

The Model Context Protocol promises to standardize how AI models call external tools, but the reality is messy. The ecosystem now has over 15,000 MCP-compatible servers, but they’re anything but uniform:

Transport chaos and incomplete implementations: Some servers only support STDIO, others stream over Server-Sent Events, and a few expose Streamable HTTP endpoints. They can’t talk to each other without custom adapters. And while the MCP specification evolves (now deprecating SSE) — MCP clients and servers are slow to catch up.

Security gaps: Many test servers skip authentication entirely or use weak schemes. Depending on what the client supports, you’ll want to either use a stdio wrapper, SSE + Bearer JWT auth, or full flown OAuth.

Writing new MCP Servers: Your existing API endpoints aren’t yet available as MCP Servers. You have to write new servers, and test them.

Everything lives somewhere different: Your prompt library runs on Server A, the vector database on Server B, and your custom tools on Server C. Managing URLs, keys, and retry logic across all of them becomes a full-time job.

How MCP Gateway Fixes This

Instead of handling retry logic and managing multiple MCP servers directly in your agent or client application, MCP Gateway centralizes all that complexity. You can create multiple virtual servers for different clients or use cases, each with their own tool configurations and access controls.

The MCP Gateway Wrapper allows you to connect to the gateway securely, using a JWT token, while exposing a local STDIO server.

The gateway acts as a smart proxy that normalizes everything behind a single endpoint:

One Consistent Interface

The gateway converts STDIO, SSE, and HTTP into consistent HTTPS+JSON-RPC, so your clients only need to know one protocol.

And your MCP Clients — Agents such as Langchain, Autogen, Crew.AI — or Visual Studio code plugins (ex: Microsoft Copilot) can connect over SSE + JWT Auth, or via STDIO locally (using mcpgateway-wrapper to connect securely to the Gateway).

Complete Tool Discovery and Debugging

The gateway automatically discovers all connected servers and presents every tool, prompt, and resource in one catalog.

Tools can be enabled, disabled and tested — and JSON Schema and tool description easily viewed.

Non-MCP API Support

Wrap any REST API endpoint and expose it as a fully-typed MCP tool with automatic retries and schema validation.

Built-in Observability

Every API call is timed and logged, so you can track performance and debug issues without external monitoring tools. Both per-tool, per server and aggregated metrics are available.

Getting Started

The gateway ships as a single Docker container or pip package with no external dependencies-just a local SQLite database.

Docker / Podman

docker run -d --name mcpgateway \
  -p 4444:4444 \
  -e HOST=0.0.0.0 \
  -e JWT_SECRET_KEY=your-secret-key \
  -e BASIC_AUTH_USER=admin \
  -e BASIC_AUTH_PASSWORD=your-password \
  ghcr.io/ibm/mcp-context-forge:v0.1.0

Python Package

pip install mcp-contextforge-gateway
BASIC_AUTH_PASSWORD=password mcpgateway --host 127.0.0.1 --port 4444

Once running, you’ll have access to http://localhost:4444 with:

Admin Dashboard (/admin) - Manage servers and tools through a web interface.
API Documentation (/docs- and /redoc) Interactive Swagger documentation.
Version, Health and Configuration (/version and /health) — get version, configuration and debugging information. Requires auth (login to admin page first, or use as API to retrieve JSON).
JSON-RPC Endpoint (/rpc) - RPC Endpoint.
Metrics (/metrics) - Performance and usage statistics.

Detailed deployment across Containers, Docker, Compose, Kubernetes, OpenShift, Minikube, Helm, Code Engine, AWS and Azure is available through the project documentation page.

Integration with VS Code and GitHub Copilot

One of the most practical applications is connecting MCP Gateway to GitHub Copilot in VS Code. This gives Copilot access to all your tools through a single, secure connection.

Generate a JWT token and test it:

python -m mcpgateway.utils.create_jwt_token \
  --username admin --exp 0 --secret my-test-key

curl -s -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \
     http://localhost:4444/version | jq

Spin up a couple of MCP Servers:

pip install uvenv
npx -y supergateway --stdio "uvenv run mcp-server-git"

Add the MCP Servers to your gateway, under “Gateways”.

Now, create a Virtual Server under the Servers Catalog tab, adding just the tools you want to share with your MCP Clients.

Then, enable MCP support in VS Code by adding "chat.mcp.enabled": true to your GitHub Copilot Chat settings.json. Then add a mcp configuration block as described in VS Code the documentation:

{
  "servers": {
    "gateway": {
      "type": "sse",
      "url": "http://localhost:4444/servers/1/sse",
      "headers": {
        "Authorization": "Bearer YOUR_JWT_TOKEN"
      }
    }
  }
}

Press Ctrl + Alt + I to open Copilot Chat, and click on Tools. You’ll see all your gateway-managed tools available in the Tools panel:

Troubleshooting

You can use MCP Inspector to access the Global Tools catalog: http://localhost:4444/tools — then each of the individual virtual servers you’ve created. Ex: http://localhost:4444/servers/1/sse

npx @modelcontextprotocol/inspector

Real-World Impact

Instead of maintaining separate connections to a dozen different tool servers, each with its own authentication, error handling, and monitoring, you manage one gateway instance. When you need to add a new tool, you register it with the gateway rather than updating every client.

This architectural change becomes more valuable as your tool ecosystem grows. Whether you’re building a personal AI assistant or deploying enterprise-scale automation, MCP Gateway eliminates the integration overhead that typically consumes more time than the actual feature work.

The gateway is production-ready with Kubernetes support, Helm charts, and comprehensive monitoring. You can start with the simple Docker setup and scale up as needed.

Try It Out

Source code: GitHub repository
Documentation: Project docs
Package: PyPI listing

The Model Context Protocol represents an important step toward standardized AI tool integration, but the implementation reality is still fragmented and the protocol and MCP ecosystem is going to evolve over time. MCP Gateway helps bridge that gap, turning different MCP servers into a unified, secure, and observable system.

Give it a try-you’ll spend less time on plumbing and more time building the AI experiences that matter.

Understand GenAI Large Language Model limitations, and how Retrieval Augmented Generation can help

2023-09-09T09:49:00.006+03:00

Add context from private data and documents to GenAI LLMs to reduce hallucinations and increase performance through Retrieval Augmented Generation.

Use Cases for Large Language Models

Key use cases for Large Language Models include:

Generation: LLMs can be used to generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way. For example, LLMs can be used to generate realistic dialogue for chatbots, write news articles, or even create poems.
Summarization: LLMs can be used to summarize text, extract the main points of an article or document, and create a shorter version that is still accurate and informative. For example, LLMs can be used to summarize research papers, news articles, or even books.
Classification: LLMs can be used to classify text, identify the topic of a document, and determine whether it is positive or negative, factual or opinion, etc. For example, LLMs can be used to classify customer reviews, social media posts, or even medical records.
Extraction: LLMs can be used to extract information from text, identify specific entities or keywords, and create a table or list of the extracted information. For example, LLMs can be used to extract contact information from a business card, product information from a website, or even scientific data from a research paper.
Q&A: LLMs can be used to answer questions in an informative way, even if they are open ended, challenging, or strange. For example, LLMs can be used to answer questions about a particular topic, provide customer support, or even generate creative text formats of text content.

While Generative AI Large Language Models often seem like a panacea, they suffer from a number of key issues:

Hallucinations: models will ‘make stuff up’ if they don’t know an answer. They also suffer from a lack of contextual understanding. Techniques like few-shot prompting can help.
Inference Performance: even the faster models are slower than a dial-up modem, or a fast typist! They also suffer from latency or time to first token. For most queries, expect 10–20 second response times from most models, and even with streaming, you’ll end up waiting a few seconds for the first token to be generated!
Inference Cost: LLMs are expensive to run! Some of the top 180B parameter models may need as many as 5xA100 GPUs to run, while even quantized versions of 70B LLAMA would take up a whole GPU! That’s one query at a time. The costs add up. For example, a dedicated A100 might cost as much as $20K a month with a cloud provider! A brute force approach is going to be expensive.
Stale training data: even top models haven’t been trained on ‘recent’ data, and have a cut-off date. Remember, a model doesn’t ‘have access to the internet’. While certain ‘plugins’ do offer ‘internet search’, it’s just a form of RAG, where ‘top 10 internet search query results’ are fed into the prompt as context, for example.
Use with private data: LLMs haven’t been trained on *your* private data, and as such, cannot answer questions based on our dataset, unless that data is inject through fine tuning or prompt engineering.
Token limits / context window size: Models are limited by the TOKEN_LIMIT, and most models can process, at best, a few pages of total input/output. You can’t feed a model and entire document, and ask for a summary or extract facts from the document. You need to chunk documents into pages first, and perform multiple queries.
They only support text: while this sounds obvious (from the name), it also means you can’t just feed a PDF file or WORD document to a LLM. You first need to convert that data to text, and chunk it to fit in the token limit, alongside your prompt and some room for output. Conversions aren’t perfect. What happens to your images, or tables, or metadata? It also means models can only output text. Formatting the text to output HTML or DOCX or other rich text formats requires a lot of heavy lifting in our pipeline.
Lack of transparency / explainability: why did the model generate a particular answer? Techniques such as RAG can help, as you are able to point at the ‘context’ that generated a particular answer, and even display the context. While the LLM answer may not necessarily be correct, you can display the source content that helped generate that answer.
Potential bias, hate, abuse, harm, ethical concerns, etc: sometimes, answers generated by an LLM can be outright harmful. Using the RAG pattern, in addition to HARM filters can help mitigate some of these issues.
Training and fine tuning costs: to put it in perspective, a 70B model like LLAMA2 might need ~2048 A100 GPUs for a month to train, adding up to $20–40M training cost, not to mention what it takes to download and store the data. The: “Training Hardware & Carbon Footprint” section from the LLAMA2 paper suggests a total of 3311616 GPU hours was used to train LLAMA2 (7/13/34 and 70B)!

It helps to think of of Large Language Models (LLMs) like mathematical functions, or your phone’s autocomplete:

f(x) = x’

Where the input (x) and the output (x’) are strings. The model starts by looking at the input, then will ‘autocomplete’ the output.
For example, f(“What is Kubernetes”) = “Kubernetes, often abbreviated as K8s, is an open-source platform designed to automate deploying, scaling, and operating application containers.”
Most chat interfaces will also provide a default system prompt. For LLAMA2, this is: “You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information."
Depending on the model and interface, there may be ‘hidden’ inputs to your model. Many Chat interfaces will include a conversational memory, where they insert a moving window of your previous prompts into the current prompt, as context. It would look something like this: “Below are a series of dialogues between a user and a AI assistant…. [dialogues] [new content]“

The inputs to a model are a little more complex though:

f(training_data, model_parameters, input_string) = output_string

training_data represents the data it was trained (different models will provide different answers). While not an ‘input’ as such, the data the model was trained (and how it was trained) on plays a key factor in the output.
model_parameters represent things like “temperature”, “repetition penalty”, “min tokens” or “max tokens”, “top_p”, “top_k”, and other such values.
input_string is the combination of prompt and context you give to the model. Ex: “What is Kubernetes” or “Summarize the following document: ${DOCUMENT}”
the ‘prompt’ is usually an optional instruction like “summarize”, “extract”, “translate”, “classify” etc. but more complex prompts are usually used. “Be a helpful assistant that responds to my question.. etc.”
The function can process a maximum of TOKEN_LIMIT (total input and output), usually ~4096 tokens (~3000 words in English, fewer in say.. Japanese). Models with larger TOKEN_LIMITS exist, though they usually don’t perform as well above the 4096 token limit. This means, in practice, you can’t feed a whole whitepaper to an LLM and ask it to ‘summarize this document’, for example.

What Large Language Models DON’T DO

Learn: A model will not ‘learn’ from interactions (unless specifically trained/fine-tuned).

Remember: A model doesn’t remember previous prompts. In fact, it’s all done with prompt trickery: previous prompts are injected. The API does a LOT of of filtering and heavy lifting!

Reason: Think of LLMs like your phone’s autocomplete, it doesn’t reason, or do math.

Use your data: LLMs don’t provide responses based on YOUR data (databases or files), unless it’s include in the training dataset, or the prompt (ex: RAG).

Use the Internet: A LLM doesn’t have the capacity to ‘search the internet’, or make API calls.

In fact, a model does not perform any activity other than converting one string of text into another string of text.
Any 3rd party data not in the model will need to be injected into prompts (RAG)

Adding a LLM to your software architecture:

A LLM is much much slower than a Faxmodem!

Believe it or not, LLMs are much slower than even a faxmodem! At WPM = ((BPS / 10) / 5) * 60, a 9600 baud modem will generate 11520 words / minute.

At an average 30 tokens / second (20 words) for LLAMA-70B, you’re getting 1200 words / minute!

Large models (70B) such as LLAMA2 can be painfully slow. Smaller models (20B, 13B, 7B) are faster, and require less GPU to run. Quantized models are also faster, but provide lower quality responses.

Quantize your model for faster inference

You can load and quantize your model in 8, 4, 3 or even 2 bits, sacrificing quality for faster inference speed.

This is always a tradeoff, as you’re sacfificing model output quality for faster inferencing. Since a quantized model needs less GPU VRAM to run in, this helps you run large models on commodity hardware.

Reducing model hallucinations:

LLMs lack context from private data — leading to hallucinations when asked domain or company-specific questions. RAG can help reduce hallucinations by ‘injecting’ context into prompts. Papers:

Retrieval Augmented Generation and the importance of Vector Databases

A vector database is a specialized database designed to store and query vector embeddings efficiently. Vector embeddings are numerical representations of text, images, audio, or other data. They are used in a variety of machine learning applications, such as natural language processing, image recognition, and recommendation systems.

Near Vector search or how to Search for “Sky” and find “Blue”:

Finding the most similar documents to a given document
Finding documents that contain a specific keyword or phrase
Clustering documents together based on their similarity
Ranking documents for a search query

Popular vector databases include ChormaDB, Weaviate, Milvus.

Advantages of using a VectorDB with your LLM, in a Retrieval Augmented Generation Pattern:

Insert your data into prompts every time
Cheap, and can work with vast amounts of data
While LLMs are SLOW, Vector Databases are FAST!
Can help overcome model limitations (such as token limits) — as you’re only feeding ‘top search results’ to the LLM, instead of whole documents.
Reduce hallucinations by providing context.

Loading Documents into your Vector Databases:

Loading data into your vector database typically requires you to convert documents to text, split the text into chunks, then vectorize those chunks using an embedding model. SentenceTransformers offers a number of pre-trained models, such as all-mpnet-base-v2 or all-MiniLM-L12-v2 that perform well for English text.

Scaling factor for RAG: what to consider:

Vector Database: consider sharding and High Availability
Fine Tuning: collecting data to be used for fine tuning
Governance and Model Benchmarking: how are you testing your model performance over time, with different prompts, one-shot, and various parameters
Chain of Reasoning and Agents
Caching embeddings and responses
Personalization and Conversational Memory Database
Streaming Responses and optimizing performance. A fine tuned 13B model may perform better than a poor 70B one!
Calling 3rd party functions or APIs for reasoning or other type of data (ex: LLMs are terrible at reasoning and prediction, consider calling other models)
Fallback techniques: fallback to a different model, or default answers
API scaling techniques, rate limiting, etc.
Async, streaming and parallelization, multiprocessing, GPU acceleration (including embeddings), generating your API using OpenAPI, etc.
Retraining your embedding model

RAG Talk from Shipitcon can be found on GitHub and YouTube:

Social media

https://twitter.com/CrivetiMihai — follow for more LLM content
https://youtube.com/CrivetiMihai — more LLM videos to follow
https://www.linkedin.com/in/crivetimihai/

Data Science environment with Docker and Jupyter on the IBM Mainframe

2018-09-24T22:44:00.000+03:00

Guide to getting started with Docker, Python and Jupyter Notebook on zLinux.

Here, I'm using Red Hat Enterprise Linux 7.5 to build and deploy Jupyter notebook in an Ubuntu container. I will go over the steps used to build and run a Docker container.

Oh, and in case you're wondering: why would anyone do this - check out this snippet from the z14 announcement: "Microservices can be built on z14 with Node.js, Java, Go, Swift, Python, Scala, Groovy, Kotlin, Ruby, COBOL, PL/I, and more. They can be deployed in Docker containers where a single z14 can scale out to 2 million Docker containers".

A few basic commands:

Establish the OS release and version. We're running on RHEL 7.5 for s390x.

[cmihai@rh74s390x ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)

[cmihai@rh74s390x ~]$ uname -a
Linux rh74s390x.novalocal 3.10.0-693.17.1.el7.s390x #1 SMP Sun Jan 14 10:38:29 EST 2018 s390x s390x s390x GNU/Linux

[cmihai@rh74s390x ~]$ docker --version
Docker version 17.05.0-ce, build 89658be

Setup regular user access, sudo and SSH keys

Create a regular user account

useradd cmihai
passwd cmihai
usermod -aG wheel cmihai
su - cmihai

Add your SSH public key to authorized_hosts

mkdir -p ~/.ssh
echo "YOURKEYHERE" >> ~/.ssh/authorized_keys

Log in as your new user, and forward port 9000:

ssh -L 9000:https://www.linkedin.com/redir/invalid-link-page?url=127%2e0%2e0%2e1%3A9000 -i cmihai.pem cmihai@myzLinux

Setup docker

Create the Docker group

sudo groupadd docker
sudo usermod -aG docker cmihai

Start Docker

sudo systemctl enable docker
sudo systemctl restart docker.service
sudo systemctl status docker.service

Test docker

docker run s390x/hello-world

Let’s run a simple Ubuntu interactive shell:

docker run --name s390x-ubuntu --hostname s390x-ubuntu --interactive --tty s390x/ubuntu /bin/bash

Building a Docker container for Jupyter Notebook

Create a Dockerfile from the s390x/ubuntu base image.

FROM s390x/ubuntu
MAINTAINER Mihai Criveti

# ADD AND RUN
RUN apt-get update \
    && apt-get install -y python3 python3-pip \
    && pip3 install jupyter \
    && apt-get clean

# COMMAND and ENTRYPOINT:
CMD ["jupyter","notebook","--allow-root","--ip=0.0.0.0","--port=9000"]

# NETWORK
EXPOSE 9000

Build the container:

docker build . --tag "cmihai/jupyter-lite:v1" -f Dockerfile

Run your new container:

docker run --name jupyter --hostname jupyter -p 9000:9000 cmihai/jupyter-lite:v1

Connect to Jupyter Notebook

http://localhost:9000

You can now install depedencies directly from Jupyter:

!apt-get install --yes zlib1g-dev libjpeg-dev

Potential next steps:

Consider setting up persistence for your notebooks (ex: VOLUME ["/notebooks"] in Dockerfile)
Setup Docker Compose and build multi-tiered applications specifications - such as connecting your Jupyter Notebook to PostgreSQL, Redis, Spark, etc.
Set up other programming languages or kernels (Java, R) even Zeppelin Notebook

For an interactive tutorial of using Docker for Data Science, check out: https://github.com/crivetimihai/docker-data-science

To see the original article, check out https://www.linkedin.com/pulse/data-science-environment-docker-jupyter-ibm-mainframe-mihai-criveti/

Learn Cloud Native, Docker, K8s & Istio with free courses and IBM Badges

2018-08-13T10:29:00.000+03:00

Microservices are the building blocks for cloud native architecture. Learn how to create and use containers with Docker, manage and orchestrate Containers with Kubernetes and secure and connect microservices with Istio.

Prerequisites:

You should have a good understanding of Linux and some basic concepts such as Version Control (preferably using Git). If you need a refresher, check out these 2 free resources:

Optional: Learn Git Version Control on Katakoda if you've never used Version Control.
Optional: Take up Introduction to Linux by The Linux Foundation on edX.

Get started learning containers with Docker

Find out what containers are, how they differ from Virtual Machines and what are the benefits of using containers. Install Docker CE on your machine, search for and run container from the Docker Hub, build your first Docker container from a Dockerfile and publish it to a Docker Registry.

Learn Docker & Containers using Interactive Browser-Based Scenarios on Katakoda.
Test your knowledge and collect the Docker essentials: Extend your apps with containers badge on developerWorks.
Optional: Try my interactive Docker for Data Science course.

Learn container orchestration with Kubernetes

Kubernetes is a platform for managing containerized workloads and services. Learn about the key Kubernetes components and architecture:

Join Introduction to Kubernetes by The Linux Foundation on edX.
Collect the Container & Kubernetes Essentials with IBM Cloud badge.
Complete Kubernetes Introduction on Katakoda.

Cloud Native, Microservices, 12-factor and Istio

Learn about essential cloud-native technologies, the twelve-factor app methodology, microservices, and Istio : an intelligent service mesh for microservices. Istio helps you to connect, secure, control, and observe services.

Getting started with Microservices with Istio and IBM Cloud Kubernetes Service badge.
Complete Beyond the Basics: Istio and IBM Cloud Kubernetes Service badge and also collect the badge for completing the Containers, microservices, Kubernetes, and Istio on the CloudLearning Path.
Complete the Istio course on Katakoda - use the Istio service mesh to connect, manage, and secure microservices.
Optional: Pursue the IBM Cloud Garage Method Explorer and IBM Cloud Garage Method Advocate badges and learn about key practices such as IBM Design Thinking, Agile, DevOps used in developing and managing Cloud Native applications.

Advanced Courses & Next Steps:

Complete Debugging and Troubleshooting Containers on Katakoda.
Learn Docker in Production on Katakoda.
Learn Docker Security on Katakoda.
Learn Docker Orchestration / Swarm Mode on Katakoda.
Running Containers without Docker on Kotakoda.
Configuration Management for Containerized Delivery on edX.

For IBMers:

For IBM-ers who have access to YourLearning courses and Safari Books Online:

Complete the IBM Cloud Private Consultant Bootcamp, which includes 15+ hours of self-paced learning on Kubernetes, Helm, Docker, Microservices (IBM Garage Method), Cloud Foundry, and introduction to IBM Cloud Private.
Complete other courses and certifications from the ICp series: IBM Cloud Private - Continuous Integration/Continuous Delivery Pipelines.

Books:

Read up on Docker on Safari Books online: Docker: Up & Running , Using Docker
Check out: Kubernetes: Up & Running and Kubernetes Cookbook.

Next Steps:

Find and join a Docker or Kubernetes community at meetup.com. Attend a few presentations, talk to people!
Capstone project: find an application you like, create a Docker container for it and publish it on Docker Hub.

For additional Learning Paths, check out my Learn Cloud and AI with IBM Badges article.

This article was originally published on https://www.linkedin.com/pulse/cloud-native-docker-k8s-istio-mihai-criveti/

Learn Cloud and AI with IBM Badges

2018-08-01T21:49:00.000+03:00

Life-long learning doesn't have to cost a fortune. I've put together a free 'summer study plan' that covers Cloud Native, Docker / Kubernetes, Data Science, Big Data, Blockchain and IoT - and best of all - it's free. Completing the courses will earn you badges on Acclaim that can be easily shared on social media.

My favourite IBM sites for free courses and badges include:

developerWorks Courses for courses and badges on Blockchain, IBM Cloud / Bluemix, Node-Red, Chatbots, Raspberry Pi, API Connect, Streaming, Docker and Kubernetes. I've completed the entire set - and I can say by far ' Cloud App Developer V2' is my favourite. 21 hours of self-paced hands-on training. The labs are excellent - and highlight the twelve-factor app methodology.
Data Science and Cognitive Computing Courses on CognitiveClass.AI for learning paths on Containers, Kubernetes and IStio, Big Data, Python, R, Deep Learning, Analytics, Hadoop and Spark. I found "Applied Data Science with Python" to be a fun and exciting course. All the courses have hands-on labs, with tools deployed and available on the cloud through IBM Cloud, Data Science Experience or CogntiveClass Labs!
CognitiveClass Labs - comes in support of the courses available on CogntiveClass.AI - providing access to data science tools such as OpenRefine, Jupyter Notebook/Lab, RStudio, Zeppelin Notebook, SeaHorse and more.
IBM Watson Academy for courses and badges such as "Watson and Cloud Platform", "Watson Assistant", "Watson Discovery Service" or "Watson Knowledge Studio".
IBM Garage Courses for "Garage Method Explorer" and "Garage Method Advocate"- an excellent series on DevOps, Agile, CI/CD, practices and tools.
Watson IOT Academy - courses and badges on Watson IOT offerings.
IBM Analytics Product Skills Validation - various analytics badges and courses - that usually build on top of existing learning paths.

Additional free resources:

Katakoda provides interactive scenario driven labs and excellent courses on topics such as Docker, Kubernetes, Prometheus, Git, Jenkins, Istio, Machine Learning, Tensorflow and more. I highly recommend taking some of the courses here!
https://www.edx.org - you can 'audit' courses for free, but verified certificates usually cost money. Worth taking a look for courses such as Introduction to Kubernetes from the Linux Foundation or Essential Math for Machine Learning: Python Edition
http://www.fast.ai - free courses on Deep Learning, Computational Linear Algebra.

If you want to try something hands-on, respond to IBM's Call for Code - and build an application to tackle natural disasters (hurricanes, earthquakes, floods, extreme weather, etc).

This article was originally published on LinkedIn: https://www.linkedin.com/pulse/learn-cloud-ai-ibm-badges-mihai-criveti/

Microsoft released .NET Core as open source. Find out what's inside!

2014-11-13T12:38:00.002+02:00

Microsoft announced that .NET Core will be open source, including the runtime and the framework libraries.
There are several open source Microsoft .NET projects on GitHub and CodePlex:

.NET Core 5 (corefx) - foundational libraries that make up the .NET Core development stack;
.NET Core 5 buildtools - Build tools that are necessary for building the .NET Core project;
.NET Compiler Platform (Roslyn) - open-source C# and Visual Basic compilers with rich code analysis APIs;
EntityFramework - Microsoft's recommended data access technology for new applications in .NET;
ASP.NET vNext Home - The Home repository is the starting point for people to learn about ASP.NET vNext;

So what's in this stuff, anyway? Let's dissect it using sloccount and cloc (count lines of code) and find out!

Grab the code for Roslyn:

$ GIT_CURL_VERBOSE=1 git clone -v https://git01.codeplex.com/roslyn

Cloning into 'roslyn'...
error: RPC failed; result=56, HTTP code = 200
GnuTLS recv error (-9): A TLS packet with unexpected length was received.

Whatever - no time to rebuild git against OpenSSL. Let's use a workaround:

$ mkdir roslyn; cd roslyn; git init
$ git remote add origin https://git01.codeplex.com/roslyn
$ git checkout -B master origin/master

Great. Now let's grab the .NET Core dev stack, libraries and build tools (github clones fine):

$ git clone https://github.com/dotnet/corefx.git
$ git clone https://github.com/dotnet/buildtools.git

Analyze the .NET Core 5 (corefx) and buildtools

$ cloc --xsl=1 --xml corefx/ buildtools/

Language	Files	Blank	Comment	Code
C#	565	17312	24750	104192
XML	2	0	7	4456
MSBuild scripts	15	54	90	1820
SKILL	2	11	0	52
Visual Basic	1	17	1	43
ASP.Net	6	0	0	32
C/C++ Header	1	5	1	7
DOS Batch	1	0	0	5
Total	593	17399	24849	110607

Analyze Roslyn

$ cloc --xsl=1 --xml roslyn/Src

Language	Files	Blank	Comment	Code
C#	4138	186147	131956	1173792
Visual Basic	1767	178277	140738	981301
XML	4	1870	1920	12451
MSBuild scripts	74	0	216	10066
C/C++ Header	15	1607	428	7361
SKILL	33	888	0	4954
C++	12	322	186	1698
XSD	3	74	217	401
ASP.Net	44	4	0	351
XAML	3	3	1	128
DOS Batch	6	22	11	79
PowerShell	3	7	0	13
Total	6102	369221	275673	2192595

$ sloccount roslyn/Src

Total Physical Source Lines of Code (SLOC)                = 1,205,824
Development Effort Estimate, Person-Year (Person-Month) = 343.86 (4,126.29)
 (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
Schedule Estimate, Years (Months)                         = 4.93 (59.14)
 (Basic COCOMO model, Months = 2.5 * (person-months**0.38))
Estimated Average Number of Developers (Effort/Schedule)  = 69.78
Total Estimated Cost to Develop                           = $ 46,450,431
 (average salary = $56,286/year, overhead = 2.40).

So - what have we learned?

sloccount COCOMO estimates are just that - estimates;
We have over one million lines of C# to play around with;
Porting this thing will likely depend on Mono and will require quite a lot of work;
It still doesn't make up an ecosystem - there's work to be done here to turn this into a cross-platform .NET environment;
Microsoft is most likely pushing for the clouds - this open source seems consistent with their recent partnership with Docker.

Set up a highly available firewall and load balancer in the IBM cloud

2013-02-21T06:04:00.004+02:00

http://www.ibm.com/developerworks/cloud/library/cl-setupfirewalloncloud/index.html
Summary: Set up a highly available (Virtual Router Redundancy Protocol) firewall and load balancer (Linux Virtual Server IP Virtual Server) using Keepalived for Linux and the IBM SmartCloud Enterprise virtual IP and anti-collocation features.

QEMU and kpartx to perform bare metal recovery or forensics on cloud instances

2012-10-20T00:09:00.000+03:00

Using qemu or kpartx to perform "bare metal" instance recovery or forensics on KVM cloud instances:

Use QEMU on IBM SmartCloud Enterprise to recover images after system disasters

HP to Contribute webOS to Open Source

2011-12-11T17:24:00.001+02:00

Looks like after killing their TouchPad tablet, HP open sources WebOS:

http://www.hp.com/hpinfo/newsroom/press/2011/111209xa.html?mtxs=rss-corp-news
http://developer.palm.com/blog/2011/12/open-source/
I wonder.. https://github.com/hpwebos

Might mean the TouchPad will make a comback. We'll see.

Review: 1Q84

2011-12-11T14:26:00.001+02:00

1Q84 by Haruki Murakami

My rating: 5 of 5 stars

I can't quite put my finger on it, but this books keeps reminding me of things I've forgotten or ignored.

I must say I've enjoyed Book 3 the most.

Well, that's it. For now at least - I'm out of Murakami :-(.

View all my reviews

Review: Getting Things Done: The Art of Stress-Free Productivity

2011-10-25T12:15:00.002+03:00

Getting Things Done: The Art of Stress-Free Productivity by David Allen

My rating: 5 of 5 stars

I've been using the methods and techniques described in GTD along side mind mapping in practice for over 8 years (mostly due to tools like OneNote, Emacs ORG mode, XMind/Fremind, etc). Reading this book has definitely helped optimize my workflow and planning into something quite a bit more efficient.

Right now I'm still looking to overhaul my tooling, OneNote isn't available on UNIX-like system (though it mostly works with wine), and Emacs ORG mode isn't practical for pasting images, screenshots, video, audio, files, etc. and such for reference.

It's tried a bunch of Java tools (Thinking Rock and GTD Free) but they're not very good for reference, note taking and reminders / calendar stuff. So far it's proving hard to beat the OneNote + Outlook + Project combo.

One tool I've been using for a while now (probably an year or so) along side OneNote is Tracks . It's a RoR webapp that covers most things in the GTD workflow. Although import/export isn't yet available as of version 2.0, it gets the job done. It's also available from Bitnami as a pre-installed stack.

If you're into GTD or just like to organize yourself with TODOS, tasks, projects and such, Tracks is worth a try.

View all my reviews

Convert AWK to Perl using a2p

2011-05-26T21:27:00.003+03:00

Automatic conversion of AWK scripts to perl.

A little awk script that numbers lines:

ls -la /usr | ./linum.pl
  1 : total 240
  2 : drwxr-xr-x+  12 root root  4096 2010-06-21 06:47 .
  3 : drwxr-xr-x   22 root root  4096 2011-05-11 12:06 ..
  4 : drwxr-xr-x+   2 root root 69632 2011-05-24 14:07 bin
  5 : drwxr-xr-x+   2 root root  4096 2010-05-05 11:33 games
  6 : drwxr-xr-x+   5 root root  4096 2010-06-21 06:47 i586-suse-linux
  7 : drwxr-xr-x+  52 root root  4096 2011-05-13 04:17 include
  8 : drwxr-xr-x+ 156 root root 69632 2011-05-24 14:07 lib
  9 : drwxr-xr-x+  10 root root  4096 2010-06-21 06:47 local
 10 : drwxr-xr-x+   2 root root 12288 2011-05-24 14:07 sbin
 11 : drwxr-xr-x+ 213 root root  4096 2011-05-23 13:19 share

Let's convert this to perl using a2p.

echo '{ printf("%5d : %s\n", NR, $0) }' | a2p > linum.pl
chmod +x linum.pl

Let's try it out:

ls -la /usr | ./linum.pl
  1 : total 240
  2 : drwxr-xr-x+  12 root root  4096 2010-06-21 06:47 .
  3 : drwxr-xr-x   22 root root  4096 2011-05-11 12:06 ..
  4 : drwxr-xr-x+   2 root root 69632 2011-05-24 14:07 bin
  5 : drwxr-xr-x+   2 root root  4096 2010-05-05 11:33 games
  6 : drwxr-xr-x+   5 root root  4096 2010-06-21 06:47 i586-suse-linux
  7 : drwxr-xr-x+  52 root root  4096 2011-05-13 04:17 include
  8 : drwxr-xr-x+ 156 root root 69632 2011-05-24 14:07 lib
  9 : drwxr-xr-x+  10 root root  4096 2010-06-21 06:47 local
 10 : drwxr-xr-x+   2 root root 12288 2011-05-24 14:07 sbin
 11 : drwxr-xr-x+ 213 root root  4096 2011-05-23 13:19 share

The code block looks pretty good:

while (<>) {
chomp;      # strip record separator
printf "%5d : %s\n", $., $_;
}

NSA Operating Systems Security Configuration Guides

2011-05-26T13:35:00.002+03:00

NSA published some rather interesting OS security guides:
http://www.nsa.gov/ia/guidance/security_configuration_guides/operating_systems.shtml

The RHEL Security Guides are quite good.

I've used these along with the Security Guide and the Security-Enhanced Linux guides from RHEL.

IBM SmartCloud Enterprise tips: LVM and FTPS

2011-05-20T16:58:00.002+03:00

IBM SmartCloud Enterprise tips:

Learn how to set up a 64-bit Linux® instance (starting with a Bronze-level offering) with the Linux Logical Volume Manager (LVM), capture a private image and deploy a new instance as a different offering (a Platinum tier) and then grow the LVM volumes and file systems to accommodate the new physical volumes. This article also shows how to configure and manage LVM across physical volumes using Linux LVM-type partitions:

Configure the Linux LVM and file system to accommodate larger physical volume offerings

Set up a Microsoft® Windows™ Server 2008 R2 on the IBM Cloud so that it can act as a file server. This article details the steps to configure an FTPS and SMB file server using persistent storage; this enables the cloud user to upload and download files to and from instances in the cloud and facilitate data exchange:

Configure an FTPS/SMB file server on a Windows Server 2008 R2 using persistent storage

Microsoft Mathematics Add-In for Word and OneNote

2010-10-10T03:06:00.002+03:00

Heh, this one is pretty awesome. Calculus and plotting in Microsoft Word :-). Best of all, it's a free add-on.

Microsoft Mathematics Add-In for Word and OneNote

And while we're on the subject of educational software:

Biology Add-in for Word

Chemistry Add-in for Word

You can even grab the source code for this: http://chem4word.codeplex.com/

Mount hfsplus and hfs read write on Linux

2010-10-02T14:11:00.005+03:00

Mount HFS+ rw on Linux (at your own risk...)

# Install HFS+ support

aptitude install hfsplus hfsprogs hfsutils

# Keep an eye on your dmesg:

tail -f /var/log/messages

# Get an idea on what partition you'd want to mount

cat /proc/partitions

# See what filesystems are supported

cat /proc/filesystems

# Check for a hfsplus LKM

modprobe -l | grep hfsplus

# Load it

modprobe hfsplus

# Check to see if it's loaded

grep hfsplus /proc/partitions

# Mount the HFS+ partition. If it's journaled, we don't need -o force.

mount -t hfsplus -o force -o rw /dev/sda2 /mnt/test

# Verify it

/dev/sda2 on /mnt/test type hfsplus (rw,force) touch /mnt/test/selftest ls -la /mnt/test/selftest

# If we're not allowed to mount with the write option, check dmesg

# If required, fsck, then remount.

/sbin/fsck.hfsplus -f /dev/sda2

# As an alternative, you can disable the journal on a MacOS:

# GUI: hold option while clicking the menu in Disk Utility. CMD:

diskutil list

sudo diskutil disableJournal /Volumes/Yourdisk

ClusterSSH alternative, broadcasting to multiple terminals

2010-08-24T22:29:00.004+03:00

Gnome "Terminator" is basically similar to ClusterSSH, dsh, kanif or such but runs all terminals in a single window you can split and such somewhat similar to screen.

It's also a bit more configurable (eg: gnome-terminal).

http://freshmeat.net/projects/gnometerminator/?branch_id=70626&release_id=268543

ThinkPad fingerprint scanner - Linux PAM setup

2010-08-13T13:50:00.003+03:00

Using fingerprint scanners for login is cool. Sure, it's insecure, but who cares about that :-). It's awesome!

Do enroll more than one finger, and have a fallback mechanism though...

# Install fingerprint software.
# You can play around with fprint_demo to try it out...
sudo aptitude install fprint-demo libpam-fprint libfprint

# Enroll the index finder. See pam_fprint_enroll --help for details
pam_fprint_enroll --enroll-finger 7

# Edit your PAM configuration to enable fingerprint login
# Fallback to password if it fails.

# emacs /etc/pam.d/commond-auth
# Add something like:

# Fingerprint
auth sufficient pam_fprint.so

You could also require both, but that's just awkward :-). And if finger print scanning breaks, it's a PITA.

Emacs: Different fonts in different modes

2010-07-26T01:44:00.007+03:00

I love monospaced fonts (such as Terminus or Consolas / Inconsolata), but they can be harder to read when it comes to documentation or simple conversation.

So, let's make Emacs use different fonts (monospaced and variable) depending on the mode we're in (eg: Info and ERC should not be monospaced), and let's add some shortcuts to change font type /size easily. You can do something like this with a bit of elisp in your init.el / .emacs.

;; Insidious Black Magic Bits:

;; Use variable width font faces in current buffer

(defun my-buffer-face-mode-variable ()

"Set font to a variable width (proportional) fonts in current buffer"

(interactive)

(setq buffer-face-mode-face '(:family "DejaVu Sans" :height 100 :width semi-condensed))

(buffer-face-mode))

;; Use monospaced font faces in current buffer

(defun my-buffer-face-mode-fixed ()

"Sets a fixed width (monospace) font in current buffer"

(interactive)

(setq buffer-face-mode-face '(:family "Consolas" :height 100))

(buffer-face-mode))

;; Set default font faces for Info and ERC modes

(add-hook 'erc-mode-hook 'my-buffer-face-mode-variable)

(add-hook 'Info-mode-hook 'my-buffer-face-mode-variable)

;; Control + scroll to change font type

(global-set-key [C-mouse-4] 'my-buffer-face-mode-fixed)

(global-set-key [C-mouse-5] 'my-buffer-face-mode-variable)

;; Shift + scroll to change font size

(global-set-key [S-mouse-4] 'text-scale-increase)

(global-set-key [S-mouse-5] 'text-scale-decrease)

And the results:

Note that you can also Shift + Left Click on a window to set fonts / font sizes.

Converting Perforce depots to Mercurial repositories with hg convert

2010-07-18T22:56:00.004+03:00

Converting Perforce to Mercurial repositories is fast (4-5 seconds for a 500MB repository with 300 revisions).

You'll also want to start p4d and log into P4V before, so it caches credentials.

1. Setup your p4 client:

export P4PORT=localhost:1666

export P4CLIENT=your_client_here

2. Enable hg converter by adding to ~/.hgrc

[extensions]

hgext.convert=

3. Start p4d

./p4d &

4. Convert the depot:

You need to specify a revision. Look it up in the p4 client (depot-details-revision). @all might also work.

hg convert //depot/...@307 converted-perforce path/to/hg-repo

5. Look at the history

hg history

6. Look at diffs and history with a graphical tool. This should give you a better idea of branches and such.

hgview

7. Clone your repository

hg clone /path/to/hg-repo

8. Publish your repository via the web interface

hg serve

9. Browse around

http://localhost:8000/

Additional notes:

Keep an eye on:

http://mercurial.selenic.com/wiki/RepositoryConversion#Perforce

Memory Scanner and Editor for Linux

2010-07-11T07:26:00.003+03:00

If you're familiar with ArtMoney, MemEdit or Cheat Engine on Windows, then you know what it's used for :-).

Basically, it lets you search for variables in the memory of a running process, then refine the search further (eg: give higher, lower or fixed values). You can use this basically to cheat at games. Look for the health, money or mana values, refine the search, then freeze or set the value to whatever you want.

ScanMem is a simple commandline tool on Linux that lets you do the same. There are other uses, but this one is the most common...

Nautilus Location Bar - Fix Text Mode

2010-06-07T20:46:00.005+03:00

Gnome keeps getting dumber and dumber all the time. First the "Open in Terminal" right click option was gone, now the address bar.

Before at least you had a crayon button to toggle it. Now you need to type / and start typing or Ctrl-L to show it. Makes it a bit harder to copy/paste stuff to it.

Here's how you fix it:

gconftool-2 --type=Boolean --set /apps/nautilus/preferences/always_use_location_entry true

VIM: The Matrix Has You

2010-05-21T22:50:00.005+03:00

(click for gif animation)

This is pretty much the coolest thing I've seen all day.

A "Matrix" screensaver for VIM. It's amazingly realistic too.

http://www.vim.org/scripts/script.php?script_id=1189

Share GNU screen sessions with a different user

2010-04-06T13:27:00.004+03:00

Here's something cool you can do with screen. Share a screen session with multiple terminals. "screen -x". Of course, this only works if you're using the same user.

Let's say you want to do some cooperative debugging with another username. Simple:

1. Set screen setuid flag:

chmod +s screen

2. Start a new screen sessions:

screen -S mihai

3. Set multiuser

Ctrl-A

:multiuser on

4. Add the user to join your session to the ACL list:

Ctrl-A

:acladd jimbo

5. Now connect as jimbo and join cmihai's session (called mihai):

screen -x cmihai/mihai

Now you can share you screen session with that user :-).

Unpack functions and BSDProgress - progress bar for your archives.

2010-01-03T04:09:00.006+02:00

A colored unpack script using bar:

A little KSH93 function for your .profile. Now you can augomatically unpack most archives while watching the pretty progress bar.

You need bsdprogress (Solaris and Linux port of NetBSD's progress). With a bit work it also works on AIX.

I just re-wrote this for progress, so I didn't check the cpio and such yet.

# Mihai Criveti - Unpack function:
# Progress Unpack - unpack using GNU tar and bsdprogress.
function up {
for archive in $*; do
  print $archive;
  case $archive in
       (*.tar) progress -f $archive gtar xpif - ;;
       (*.tar.gz) progress -f $archive gtar -zpixf - ;;
       (*.tgz) progress -zf $archive gtar -ipxf - ;;
       (*.tar.bz2) progress -f $archive gtar -jipxf - ;;
       (*.tbz) progress -f $archive gtar -jipxf - ;;
       (*.gz) progress -f $archive gunzip -;;
       (*.bz2) progress -f $archive bunzip -;;
       (*.tar.Z) uncompress -c $archive | tar gxvf - ;;
       (*.cpio) cpio -icd < $archive ;;
       (*.Z) uncompress $archive ;;
       (*.rar) unrar x $archive ;;
       (*.zip) unzip $archive ;;
  esac
done
}

Let's say we want to unpack files that end in *bz2 (it will automagically detect .tar.bz2 or tbz2), *gz and a specific tar archive:


cmihai@phobos:/home/cmihai/build$ up g*bz2 c*gz Python\-3.0.1.tar
gnupg-1.4.10.tar.bz2
100% |**********************************************************************|  3330 KB  296.95 KB/s    00:00 ETA
gnupg-2.0.14.tar.bz2
100% |**********************************************************************|  3888 KB  340.20 KB/s    00:00 ETA
ccache-2.4.tar.gz
100% |**********************************************************************| 86363     201.51 KB/s    00:00 ETA
Python-3.0.1.tar
91% |********************************************************               | 43320 KB  787.63 KB/s    05:00 ETA

See:

http://netbsd.gw.com/cgi-bin/man-cgi?progress++NetBSD-current

http://mac.freshmeat.net/projects/bsdprogress/

It builds fine on AIX (Makefile tweaks), Solaris and Linux. Native on NetBSD.

Other versions:

http://clpbar.sourceforge.net/

http://freshmeat.net/projects/progress