Tech

JS library

2013-11-18T23:22:00.000-08:00

From wiki:

A JavaScript library is a library of pre-written JavaScript which allows for easier development of JavaScript-based applications, especially for AJAX and other web-centric technologies.

History

Main article: JavaScript: History

While JavaScript, as first developed by Netscape (and later Mozilla), has long had a presence on the Web for many websites, it gained a particular pitch with the rise of the Web 2.0 era of computing, in which JavaScript became increasingly used for the development of user interfaces for applications, both web-based and desktop-based.

JavaScript was also combined with CSS to create dynamic web pages, which have also become popular as a more efficient and accessible alternative to Flash-based websites.

Libraries

With the expanded demands for JavaScript, an easier means for programmers to develop such dynamic interfaces was needed. Thus,

- JavaScript libraries such as Prototype, script.aculo.us, Ext Core, jsPHP, MooTools and jQuery

and

- JavaScript widget libraries such as Ext JS, DHTMLX, and Dojo Toolkit were developed,

allowing for developers to concentrate more upon more distinctive applications of Ajax. This has led to other companies and groups, such as Microsoft and Yahoo! developing their own JavaScript-based user interface libraries, which find their way into the web applications developed by these companies.

Some JavaScript libraries allow for easier integration of JavaScript with other web development technologies, such as CSS, PHP, Ruby, and Java. Many libraries include code to detect differences between runtime environments, and remove the need for applications to allow for such inconsistencies.

Almost all JavaScript libraries are released under either a copycenter or copyleft license to ensure license-free distribution, usage, and modification.

Frameworks[edit]

Some JavaScript libraries, such as YUI, are classified as frameworks since they exhibit full-stack capabilities and properties not found in general JavaScript libraries.

Web based SSH - ajaxterm setup on ubuntu 13.10

2013-10-30T01:33:00.003-07:00

from Wiki: http://en.wikipedia.org/wiki/Web-based_SSH

Web-based SSH clients basically consist of the following parts:

Client-side: Typically JavaScript and dynamic HTML pages are used to capture keystrokes, transmit messages to/from the server and display the results in the user's web browser.
Server-side/Web application: Incoming requests are processed on the web application server. Keyboard events are forwarded to a secure shell client communicating with the connected SSH server. Terminal output is either passed to the client where it is converted into HTML via JavaScript or it is translated into HTML by the server before it is transmitted to the client.

Client-side terminal emulation

Web-based SSH servers that utilize client-side terminal emulation typically transmit the raw terminal output from the SSH server directly to the client. This has the advantage of offloading the process of translating terminal output into HTML on to the client. The disadvantage of this method is that it is limited by the capabilities of JavaScript and it will use up a non-trivial amount of the client's CPU and memory to process the incoming character stream. It also relies on the client to keep track of the terminal state and respond to escape sequences.

Client-side terminal emulator example: vt100.js (from Shell in a Box)

Server-side terminal emulation

Web-based SSH servers that utilize server-side terminal emulation typically keep track of the terminal screen and state in memory and convert it to HTML either when a screen update occurs or when the client expressly requests an update. The advantage of this method is that the state of the terminal remains persistent even if the user connects to their existing session(s) from a different web browser. It also enables the server to act upon terminal output even if the user is disconnected.^[1] The disadvantage of this method is that it uses up more CPU and memory on the server.

Server-side terminal emulator example: terminal.py (From Gate One).

Advantages

The main advantages of web-based SSH can be summarized as follows:

Accessibility: Web-based SSH as described in this article requires no local installation of client software. It is thus possible to access SSH servers through a web browser from anywhere. As communication is based on HTTP or HTTPS it is also possible to access SSH servers from behind a firewall or proxy that restricts Internet access to only ports 80 (HTTP) or 443 (HTTPS).
Anonymous Access: As SSH access is tunneled through an intermediary web application server it is this server which actually communicates with the SSH server. This means that the SSH server will only be aware of the IP address of the web application server, keeping the actual client's IP address hidden.
Auditability: Because all communication between the client and the SSH server must pass through the web application serverthis communication can be logged. This prevents a malicious client from deleting logs of their activities.
Resuming Sessions: Some web-based SSH implementations allow the user to resume their SSH sessions after being disconnected. This is not possible with a traditional SSH client.
Embeddable: Web-based SSH implementations can be embedded into any web page allowing them to be integrated into other web-based applications.
Unique Features: Many web-based SSH tools have unique features such as the ability to share terminals with other users, can display images within terminals, and other useful capabilities.

--- ajaxterm on Ubuntu setup

https://help.ubuntu.com/community/AjaxTerm

https://help.ubuntu.com/12.10/serverguide/certificates-and-security.html

https://help.ubuntu.com/12.10/serverguide/httpd.html#https-configuration

1, sudo apt-get install ajaxterm

You will need to enable SSH password authentication for AjaxTerm to work.

2, uncomment "PasswordAuthentication yes" in /etc/ssh/sshd_config.conf

3, sudo service ssh restart

4, configure Apache2 for HTTPS and generate a certificate

------------------------------------------------------------------------------------------------------------------------

Generating a Certificate Signing Request (CSR)

Whether you are getting a certificate from a CA or generating your own self-signed certificate, the first step is to generate a key.

If the certificate will be used by service daemons, such as Apache, Postfix, Dovecot, etc, a key without a passphrase is often appropriate. Not having a passphrase allows the services to start without manual intervention, usually the preferred way to start a daemon.

This section will cover generating a key with a passphrase, and one without. The non-passphrase key will then be used to generate a certificate that can be used with various service daemons.

Running your secure service without a passphrase is convenient because you will not need to enter the passphrase every time you start your secure service. But it is insecure and a compromise of the key means a compromise of the server as well.

To generate the keys for the Certificate Signing Request (CSR) run the following command from a terminal prompt:

openssl genrsa -des3 -out server.key 2048

Generating RSA private key, 2048 bit long modulus
..........................++++++
.......++++++
e is 65537 (0x10001)
Enter pass phrase for server.key:

You can now enter your passphrase. For best security, it should at least contain eight characters. The minimum length when specifying -des3 is four characters. It should include numbers and/or punctuation and not be a word in a dictionary. Also remember that your passphrase is case-sensitive.

Re-type the passphrase to verify. Once you have re-typed it correctly, the server key is generated and stored in the server.key file.

Now create the insecure key, the one without a passphrase, and shuffle the key names:

openssl rsa -in server.key -out server.key.insecure
mv server.key server.key.secure
mv server.key.insecure server.key

The insecure key is now named server.key, and you can use this file to generate the CSR without passphrase.

To create the CSR, run the following command at a terminal prompt:

openssl req -new -key server.key -out server.csr

It will prompt you enter the passphrase. If you enter the correct passphrase, it will prompt you to enter Company Name, Site Name, Email Id, etc. Once you enter all these details, your CSR will be created and it will be stored in the server.csr file.

You can now submit this CSR file to a CA for processing. The CA will use this CSR file and issue the certificate. On the other hand, you can create self-signed certificate using this CSR.

Creating a Self-Signed Certificate

To create the self-signed certificate, run the following command at a terminal prompt:

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

The above command will prompt you to enter the passphrase. Once you enter the correct passphrase, your certificate will be created and it will be stored in the server.crt file.

If your secure server is to be used in a production environment, you probably need a CA-signed certificate. It is not recommended to use self-signed certificate.

Installing the Certificate

You can install the key file server.key and certificate file server.crt, or the certificate file issued by your CA, by running following commands at a terminal prompt:

sudo cp server.crt /etc/ssl/certs
sudo cp server.key /etc/ssl/private

Now simply configure any applications, with the ability to use public-key cryptography, to use the certificate and key files. For example, Apache can provide HTTPS, Dovecot can provide IMAPS and POP3S, etc.

HTTPS Configuration

The mod_ssl module adds an important feature to the Apache2 server - the ability to encrypt communications. Thus, when your browser is communicating using SSL, the https:// prefix is used at the beginning of the Uniform Resource Locator (URL) in the browser navigation bar.

The mod_ssl module is available in apache2-common package. Execute the following command from a terminal prompt to enable the mod_sslmodule:

sudo a2enmod ssl

There is a default HTTPS configuration file in /etc/apache2/sites-available/default-ssl. In order for Apache2 to provide HTTPS, acertificate and key file are also needed. The default HTTPS configuration will use a certificate and key generated by the ssl-cert package. They are good for testing, but the auto-generated certificate and key should be replaced by a certificate specific to the site or server. For information on generating a key and obtaining a certificate see Certificates

To configure Apache2 for HTTPS, enter the following:

sudo a2ensite default-ssl

The directories /etc/ssl/certs and /etc/ssl/private are the default locations. If you install the certificate and key in another directory make sure to change SSLCertificateFile and SSLCertificateKeyFile appropriately.

With Apache2 now configured for HTTPS, restart the service to enable the new settings:

sudo service apache2 restart

Depending on how you obtained your certificate you may need to enter a passphrase when Apache2 starts.

You can access the secure server pages by typing https://your_hostname/url/ in your browser address bar.

------------------------------------------------------------------------------------------------------------------------

Now copy [then symlink] the SSL config from the Available sites to the Enabled sites

5, sudo cp /etc/apache2/sites-available/default /etc/apache2/sites-available/ssl

Make sure all the modules and sites for proxy and ssl have been enabled.

6, sudo a2enmod proxy_http
6, sudo a2enmod ssl
6, sudo a2ensite default-ssl

It is time to decide on an htaccess password and username. Remember to make this password strong.

7, sudo mkdir /srv/ajaxterm
8, sudo htpasswd -bc /srv/ajaxterm/.htpasswd [user] [pass]

You will now need to edit the default Apache SSL configuration.

9, sudo nano -w /etc/apache2/sites-enabled/default-ssl.conf

Insert just before </VirtualHost>:

        ProxyRequests Off
        <Proxy *>
                AuthUserFile /srv/ajaxterm/.htpasswd
                AuthName EnterPassword
                AuthType Basic
                require valid-user

                Order Deny,allow
                Allow from all
        </Proxy>
        ProxyPass / http://localhost:8022/
        ProxyPassReverse / http://localhost:8022/

Restart apache2.

10, sudo service apache2 restart

Open your firewall port 443 and try to browse to https://yourhost.foo where yourhost.foo is either your IP adress or your hostname/web address.

--- ajaxterm on Ubuntu

https://help.ubuntu.com/community/AjaxTerm

https://help.ubuntu.com/12.10/serverguide/certificates-and-security.html

https://help.ubuntu.com/12.10/serverguide/httpd.html#https-configuration

Routing

2013-10-23T19:41:00.000-07:00

- problem definition
In graph theory, the shortest path problem is the problem of finding a path between two vertices (or nodes) in a graph such that the sum of the weights of its constituent edges is minimized.

- algorithm

The most important algorithms for solving this problem are:

Dijkstra's algorithm solves the single-source shortest path problems.
Bellman–Ford algorithm solves the single-source problem if edge weights may be negative.
A* search algorithm solves for single pair shortest path using heuristics to try to speed up the search.
Floyd–Warshall algorithm solves all pairs shortest paths.
Johnson's algorithm solves all pairs shortest paths, and may be faster than Floyd–Warshall on sparse graphs.

Additional algorithms and associated evaluations may be found in Cherkassky et al.^[1]

===============================================================================

- routing

Routing schemes differ in their delivery semantics:

unicast delivers a message to a single specific node
broadcast delivers a message to all nodes in the network
multicast delivers a message to a group of nodes that have expressed interest in receiving the message
anycast delivers a message to anyone out of a group of nodes, typically the one nearest to the source
geocast delivers a message to a geographic area

Unicast is the dominant form of message delivery on the Internet. This article focuses on unicast routing algorithms.

- algorithm

Distance vector algorithms (Main article: Distance-vector routing protocol)

Distance vector algorithms use the Bellman–Ford algorithm. This approach assigns a cost number to each of the links between each node in the network. Nodes will send information from point A to point B via the path that results in the lowest total cost (i.e. the sum of the costs of the links between the nodes used).

The algorithm operates in a very simple manner. When a node first starts, it only knows of its immediate neighbours, and the direct cost involved in reaching them. (This information — the list of destinations, the total cost to each, and the next hop to send data to get there — makes up the routing table, or distance table.) Each node, on a regular basis, sends to each neighbour node its own current assessment of the total cost to get to all the destinations it knows of. The neighbouring nodes examine this information and compare it to what they already 'know'; anything that represents an improvement on what they already have, they insert in their own routing table(s). Over time, all the nodes in the network will discover the best next hop for all destinations, and the best total cost.

When one network nodes goes down, any nodes that used it as their next hop discard the entry, and create new routing-table information. These nodes convey the updated routing information to all adjacent nodes, which in turn repeat the process. Eventually all the nodes in the network receive the updates, and discover new paths to all the destinations they can still "reach". e.g. RIPV1,RIPV2

- The Routing Information Protocol (RIP) is one of the oldest distance-vector routing protocol, which employs the hop count as a routing metric. RIP prevents routing loops by implementing a limit on the number of hops allowed in a path from the source to a destination. The maximum number of hops allowed for RIP is 15. This hop limit, however, also limits the size of networks that RIP can support. A hop count of 16 is considered an infinite distance, in other words the route is considered unreachable.

RIP implements the split horizon, route poisoning and holddown mechanisms to prevent incorrect routing information from being propagated. These are some of the stability features of RIP. It is also possible to use the Routing Information Protocol with Metric-Based Topology (RMTI)^[1] algorithm to cope with the count-to-infinity problem. With RMTI, it is possible to detect every possible loop with a very small computation effort.

Originally, each RIP router transmitted full updates every 30 seconds. In the early deployments, routing tables were small enough that the traffic was not significant. As networks grew in size, however, it became evident there could be a massive traffic burst every 30 seconds, even if the routers had been initialized at random times. It was thought, as a result of random initialization, the routing updates would spread out in time, but this was not true in practice. Sally Floyd and Van Jacobson showed in 1994^[2] that, without slight randomization of the update timer, the timers synchronized over time. In most current networking environments, RIP is not the preferred choice for routing as its time to converge and scalability are poor compared to EIGRP, OSPF, or IS-IS (the latter two being link-state routing protocols), and (without RMTI) a hop limit severely limits the size of network it can be used in. However, it is easy to configure, because RIP does not require any parameters on a router unlike other protocols.

RIP uses the User Datagram Protocol (UDP) as its transport protocol, and is assigned the reserved port number 520.^[3]

Link-state algorithms (Main article: Link-state routing protocol)

When applying link-state algorithms, a graphical map of the network is the fundamental data used for each node. To produce its map, each node floods the entire network with information about the other nodes it can connect to. Each node then independently assembles this information into a map. Using this map, each router independently determines the least-cost path from itself to every other node using a standard shortest paths algorithm such as Dijkstra's algorithm. The result is a tree graph rooted at the current node, such that the path through the tree from the root to any other node is the least-cost path to that node. This tree then serves to construct the routing table, which specifies the best next hop to get from the current node to any other node.

- IS-IS (pronounced "i-s i-s") is an interior gateway protocol, designed for use within an administrative domain or network. This is in contrast to Exterior Gateway Protocols, primarily Border Gateway Protocol (BGP), which is used for routing between autonomous systems (RFC 1930). The protocol was defined in ISO/IEC 10589:2002 as an international standard within the Open Systems Interconnection (OSI) reference design. Though originally an ISO standard, the IETF republished the protocol as an Internet Standard in RFC 1142. IS-IS has been called "the de facto standard for large service provider network backbones."^[1]

IS-IS is a link-state routing protocol, operating by reliably flooding link state information throughout a network of routers. Each IS-IS router independently builds a database of the network's topology, aggregating the flooded network information. Like the OSPF protocol, IS-IS uses Dijkstra's algorithm for computing the best path through the network. Packets (datagrams) are then forwarded, based on the computed ideal path, through the network to the destination.

- Open Shortest Path First (OSPF) is a link-state routing protocol for Internet Protocol (IP) networks. It uses a link state routing algorithm and falls into the group of interior routing protocols, operating within a single autonomous system (AS). It is defined as OSPF Version 2 in RFC 2328 (1998) for IPv4.^[1] The updates for IPv6 are specified as OSPF Version 3 in RFC 5340 (2008).^[2]

OSPF is perhaps the most widely used interior gateway protocol (IGP) in large enterprise networks. IS-IS, another link-state dynamic routing protocol, is more common in large service provider networks. The most widely used exterior gateway protocol is the Border Gateway Protocol (BGP), the principal routing protocol between autonomous systems on the Internet.

-----------

Both IS-IS and OSPF are link state protocols, and both use the same Dijkstra algorithm for computing the best path through the network. As a result, they are conceptually similar. Both support variable length subnet masks, can use multicast to discover neighboring routers using hello packets, and can support authentication of routing updates.

While OSPF is natively built to route IP and is itself a Layer 3 protocol that runs on top of IP, IS-IS is natively an OSI network layer protocol (it is at the same layer as CLNS). The widespread adoption of IP worldwide may have contributed to OSPF's popularity. IS-IS does not use IP to carry routing information messages. IS-IS is neutral regarding the type of network addresses for which it can route. OSPF, on the other hand, was designed for IPv4. This allowed IS-IS to be easily used to support IPv6. To operate with IPv6 networks, the OSPF protocol was rewritten in OSPF v3 (as specificed in RFC 2740).

IS-IS routers build a topological representation of the network. This map indicates the subnets which each IS-IS router can reach, and the lowest-cost (shortest) path to a subnet is used to forward traffic.

IS-IS differs from OSPF in the way that "areas" are defined and routed between. IS-IS routers are designated as being: Level 1 (intra-area); Level 2 (inter area); or Level 1-2 (both). Level 2 routers are inter area routers that can only form relationships with other Level 2 routers. Routing information is exchanged between Level 1 routers and other Level 1 routers, and Level 2 routers only exchange information with other Level 2 routers. Level 1-2 routers exchange information with both levels and are used to connect the inter area routers with the intra area routers.

In OSPF, areas are delineated on the interface such that an area border router (ABR) is actually in two or more areas at once, effectively creating the borders between areas inside the ABR, whereas in IS-IS area borders are in between routers, designated as Level 2 or Level 1-2. The result is that an IS-IS router is only ever a part of a single area.

IS-IS also does not require Area 0 (Area Zero) to be the backbone area through which all inter-area traffic must pass. The logical view is that OSPF creates something of a spider web or star topology of many areas all attached directly to Area Zero and IS-IS by contrast creates a logical topology of a backbone of Level 2 routers with branches of Level 1-2 and Level 1 routers forming the individual areas.

IS-IS also differs from OSPF in the methods by which it reliably floods topology and topology change information through the network. However, the basic concepts are similar.^{[citation needed]}

OSPF has a larger set of extensions and optional features specified in the protocol standards. However IS-IS is more easy to expand: its use of type-length-value data allows engineers to implement support for new techniques without redesigning the protocol. For example, in order to support IPv6, the IS-IS protocol was extended to support a few additional TLVs, whereas OSPF required a new protocol draft (OSPFv3). In addition to that, IS-IS is less "chatty" and can scale to support larger networks. Given the same set of resources, IS-IS can support more routers in an area than OSPF. This has contributed to IS-IS as an ISP-scale protocol.^{[citation needed]}

The TCP/IP implementation, known as "Integrated IS-IS" or "Dual IS-IS", is described in RFC 1195.

================================================================================

D.V -> RIP \

+----> IGP

L.S -> IS-IS / OSPF /

BGP ------> EGP

Gitosis setup

2013-08-29T00:55:00.002-07:00

official link: https://github.com/res0nat0r/gitosis

usually "git" user is created with following command before this installation. but you can use any other name for the user. e.g. "git2".

sudo adduser \
   --system \
   --shell /bin/bash \
   --gecos ‘User for managing of git version control’ \
   --group \
   --disabled-password \
   --home /home/git \ 
   git

You can change the home directory.

1. gitosis requires python tools, so need install it first
$ apt-get install python-setuptools

2. then clone the gitosis repo
$ git clone https://github.com/res0nat0r/gitosis.git gitosis

3. install gitosis (installs a couple of executables that Gitosis will use.)
$ cd gitosis
$ sudo python setup.py install

4.Gitosis wants to put its repositories under /home/git, which is fine. But if you have already set

up your repositories in /opt/git, so instead of reconfiguring everything, you can create a symlink:

$ ln -s /opt/git /home/git/repositories

5.remove the current authorized_keys, re-add the keys later, and let Gitosis control the authorized_keys

file automatically.

$ mv /home/git/.ssh/authorized_keys /home/git/.ssh/ak.bak

6. Next you need to turn your shell back on for the 'git' user, if you changed it to the git-shell command. People still won't be able to log in, but Gitosis will control that for you. So, let's change this line in your /etc/passwd file

git:x:1000:1000::/home/git:/bin/bash

7.initialize gitosis: (if you want to use anoterh user id instead of git, you need change -u param). You do this by running the gitosis-init command with your personal public key. If your public key isn’t on the server,you’ll have to copy it there(/tmp/id_rsa.pub). This lets the user with that key modify the main Git repository that controls the Gitosis setup.
$ sudo -H -u git gitosis-init < /tmp/id_dsa.pub
Initialized empty Git repository in /opt/git/gitosis-admin.git/

Reinitialized existing Git repository in /opt/git/gitosis-admin.git/

8.verifying the ssh connection to the server with the user git(via your pub key).
$ ssh git@gitserver

ERROR:gitosis.serve.main:Need SSH_ORIGINAL_COMMAND in environment.

Connection to <server> closed.
this means Gitosis recognized you but shut you out because you’re not trying to do any Git commands.

9. from now on, you can manage users/repos by editing gitosis.conf and public key files in keydir folder.
- add user key files to keydir;
- update gitosis.conf to add team and team members and also permissions.

[group myteam]
members = uname1 uname2 
writable = myproject

---output of python setup.py install---

running install

Checking .pth file support in /usr/local/lib/python2.7/dist-packages/

/usr/bin/python -E -c pass

TEST PASSED: /usr/local/lib/python2.7/dist-packages/ appears to support .pth files

running bdist_egg

running egg_info

creating gitosis.egg-info

writing requirements to gitosis.egg-info/requires.txt

writing gitosis.egg-info/PKG-INFO

writing top-level names to gitosis.egg-info/top_level.txt

writing dependency_links to gitosis.egg-info/dependency_links.txt

writing entry points to gitosis.egg-info/entry_points.txt

writing manifest file 'gitosis.egg-info/SOURCES.txt'

reading manifest file 'gitosis.egg-info/SOURCES.txt'

reading manifest template 'MANIFEST.in'

writing manifest file 'gitosis.egg-info/SOURCES.txt'

installing library code to build/bdist.linux-x86_64/egg

running install_lib

running build_py

creating build

creating build/lib.linux-x86_64-2.7

creating build/lib.linux-x86_64-2.7/gitosis

copying gitosis/init.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/access.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/serve.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/util.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/gitweb.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/mirror.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/__init__.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/app.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/run_hook.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/ssh.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/gitdaemon.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/snagit.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/repository.py -> build/lib.linux-x86_64-2.7/gitosis

copying gitosis/group.py -> build/lib.linux-x86_64-2.7/gitosis

creating build/lib.linux-x86_64-2.7/gitosis/test

copying gitosis/test/test_mirror.py -> build/lib.linux-x86_64-2.7/gitosis/test

copying gitosis/test/test_run_hook.py -> build/lib.linux-x86_64-2.7/gitosis/test

copying gitosis/test/test_group.py -> build/lib.linux-x86_64-2.7/gitosis/test

copying gitosis/test/util.py -> build/lib.linux-x86_64-2.7/gitosis/test

copying gitosis/test/test_ssh.py -> build/lib.linux-x86_64-2.7/gitosis/test

copying gitosis/test/test_repository.py -> build/lib.linux-x86_64-2.7/gitosis/test

copying gitosis/test/__init__.py -> build/lib.linux-x86_64-2.7/gitosis/test

copying gitosis/test/test_init.py -> build/lib.linux-x86_64-2.7/gitosis/test

copying gitosis/test/test_serve.py -> build/lib.linux-x86_64-2.7/gitosis/test

copying gitosis/test/test_gitdaemon.py -> build/lib.linux-x86_64-2.7/gitosis/test

copying gitosis/test/test_access.py -> build/lib.linux-x86_64-2.7/gitosis/test

copying gitosis/test/test_gitweb.py -> build/lib.linux-x86_64-2.7/gitosis/test

creating build/lib.linux-x86_64-2.7/gitosis/templates

copying gitosis/templates/__init__.py -> build/lib.linux-x86_64-2.7/gitosis/templates

creating build/lib.linux-x86_64-2.7/gitosis/templates/admin

creating build/lib.linux-x86_64-2.7/gitosis/templates/admin/hooks

copying gitosis/templates/admin/hooks/post-update -> build/lib.linux-x86_64-2.7/gitosis/templates/admin/hooks

creating build/lib.linux-x86_64-2.7/gitosis/templates/default

creating build/lib.linux-x86_64-2.7/gitosis/templates/default/hooks

copying gitosis/templates/default/hooks/post-receive -> build/lib.linux-x86_64-2.7/gitosis/templates/default/hooks

creating build/bdist.linux-x86_64

creating build/bdist.linux-x86_64/egg

creating build/bdist.linux-x86_64/egg/gitosis

copying build/lib.linux-x86_64-2.7/gitosis/init.py -> build/bdist.linux-x86_64/egg/gitosis

copying build/lib.linux-x86_64-2.7/gitosis/access.py -> build/bdist.linux-x86_64/egg/gitosis

copying build/lib.linux-x86_64-2.7/gitosis/serve.py -> build/bdist.linux-x86_64/egg/gitosis

creating build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/test/test_mirror.py -> build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/test/test_run_hook.py -> build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/test/test_group.py -> build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/test/util.py -> build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/test/test_ssh.py -> build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/test/test_repository.py -> build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/test/__init__.py -> build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/test/test_init.py -> build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/test/test_serve.py -> build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/test/test_gitdaemon.py -> build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/test/test_access.py -> build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/test/test_gitweb.py -> build/bdist.linux-x86_64/egg/gitosis/test

copying build/lib.linux-x86_64-2.7/gitosis/util.py -> build/bdist.linux-x86_64/egg/gitosis

copying build/lib.linux-x86_64-2.7/gitosis/gitweb.py -> build/bdist.linux-x86_64/egg/gitosis

copying build/lib.linux-x86_64-2.7/gitosis/mirror.py -> build/bdist.linux-x86_64/egg/gitosis

copying build/lib.linux-x86_64-2.7/gitosis/__init__.py -> build/bdist.linux-x86_64/egg/gitosis

copying build/lib.linux-x86_64-2.7/gitosis/app.py -> build/bdist.linux-x86_64/egg/gitosis

copying build/lib.linux-x86_64-2.7/gitosis/run_hook.py -> build/bdist.linux-x86_64/egg/gitosis

copying build/lib.linux-x86_64-2.7/gitosis/ssh.py -> build/bdist.linux-x86_64/egg/gitosis

copying build/lib.linux-x86_64-2.7/gitosis/gitdaemon.py -> build/bdist.linux-x86_64/egg/gitosis

copying build/lib.linux-x86_64-2.7/gitosis/snagit.py -> build/bdist.linux-x86_64/egg/gitosis

creating build/bdist.linux-x86_64/egg/gitosis/templates

creating build/bdist.linux-x86_64/egg/gitosis/templates/admin

creating build/bdist.linux-x86_64/egg/gitosis/templates/admin/hooks

copying build/lib.linux-x86_64-2.7/gitosis/templates/admin/hooks/post-update -> build/bdist.linux-x86_64/egg/gitosis/templates/admin/hooks

copying build/lib.linux-x86_64-2.7/gitosis/templates/__init__.py -> build/bdist.linux-x86_64/egg/gitosis/templates

creating build/bdist.linux-x86_64/egg/gitosis/templates/default

creating build/bdist.linux-x86_64/egg/gitosis/templates/default/hooks

copying build/lib.linux-x86_64-2.7/gitosis/templates/default/hooks/post-receive -> build/bdist.linux-x86_64/egg/gitosis/templates/default/hooks

copying build/lib.linux-x86_64-2.7/gitosis/repository.py -> build/bdist.linux-x86_64/egg/gitosis

copying build/lib.linux-x86_64-2.7/gitosis/group.py -> build/bdist.linux-x86_64/egg/gitosis

byte-compiling build/bdist.linux-x86_64/egg/gitosis/init.py to init.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/access.py to access.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/serve.py to serve.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/test/test_mirror.py to test_mirror.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/test/test_run_hook.py to test_run_hook.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/test/test_group.py to test_group.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/test/util.py to util.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/test/test_ssh.py to test_ssh.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/test/test_repository.py to test_repository.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/test/__init__.py to __init__.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/test/test_init.py to test_init.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/test/test_serve.py to test_serve.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/test/test_gitdaemon.py to test_gitdaemon.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/test/test_access.py to test_access.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/test/test_gitweb.py to test_gitweb.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/util.py to util.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/gitweb.py to gitweb.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/mirror.py to mirror.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/__init__.py to __init__.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/app.py to app.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/run_hook.py to run_hook.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/ssh.py to ssh.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/gitdaemon.py to gitdaemon.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/snagit.py to snagit.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/templates/__init__.py to __init__.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/repository.py to repository.pyc

byte-compiling build/bdist.linux-x86_64/egg/gitosis/group.py to group.pyc

creating build/bdist.linux-x86_64/egg/EGG-INFO

copying gitosis.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO

copying gitosis.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO

copying gitosis.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO

copying gitosis.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO

copying gitosis.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO

copying gitosis.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO

copying gitosis.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO

creating dist

creating 'dist/gitosis-0.2-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it

removing 'build/bdist.linux-x86_64/egg' (and everything under it)

Processing gitosis-0.2-py2.7.egg

creating /usr/local/lib/python2.7/dist-packages/gitosis-0.2-py2.7.egg

Extracting gitosis-0.2-py2.7.egg to /usr/local/lib/python2.7/dist-packages

Adding gitosis 0.2 to easy-install.pth file

Installing gitosis-init script to /usr/local/bin

Installing gitosis-run-hook script to /usr/local/bin

Installing gitosis-serve script to /usr/local/bin

Installed /usr/local/lib/python2.7/dist-packages/gitosis-0.2-py2.7.egg

Processing dependencies for gitosis==0.2

Searching for distribute==0.6.28dev-r0

Best match: distribute 0.6.28dev-r0

Adding distribute 0.6.28dev-r0 to easy-install.pth file

Installing easy_install script to /usr/local/bin

Installing easy_install-2.7 script to /usr/local/bin

Using /usr/lib/python2.7/dist-packages

Finished processing dependencies for gitosis==0.2

-----

gitlab - access denied issue (not able to push after installation)

2013-08-27T01:22:00.004-07:00

[gitlab]
installation guide:
https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/installation.md

- if you have the following issue with gitlab, that means your gitlab-shell not configured correctly

$ ssh git@server

Welcome to GitLab, Anonymous!

Connection to server closed.

$ git push -u origin master

Access denied.

fatal: Could not read from remote repository.

then you need update /home/git/gitlab-shell/config.yml

change default one("http://localhost/") to "http://<server>/"

refer to following info/explanation as well:

https://github.com/gitlabhq/gitlabhq/issues/3384#issuecomment-15457287

---- excerpted from above page ----

When invoking the internal api likehttp://server/api/v3/internal/allowed?key_id=2&action=git-upload-pack&ref=_any&project=user/projectfrom other machines, the nginx config will give gitlab as default web so it works.

However when actually doing clone, push and pull. The internal api is invoked by gitlab-shell on the same machine of the gitlab server. Now since the gitlab config for nginx /etc/nginx/sites-enabled/gitlab is set as:

server {
  listen YOUR_SERVER_IP:80 default_server;         # e.g., listen 192.168.1.1:80;
  server_name YOUR_SERVER_FQDN;     # e.g., server_name source.example.com;
  root /home/git/gitlab/public;

user git on the server invoked gitlab-shell to access web api at YOUR_SERVER_FQDN, but now the server IP to user git is 127.0.0.1. So gitlab is not valid website now. But the default nginx site listens to any IP at port 80 so the

<html>
<head>
<title>Welcome to nginx!</title>
</head>
<body bgcolor="white" text="black">
<center><h1>Welcome to nginx!</h1></center>
</body>
</html>

is given out. Because it is the index page of default nginx site.

Solution is to set the gitlab nginx config file /etc/nginx/sites-enabled/gitlab as:

server {
  listen *:80 default_server;         # e.g., listen 192.168.1.1:80;
  server_name YOUR_SERVER_FQDN;     # e.g., server_name source.example.com;
  root /home/git/gitlab/public;

Hope this solves problems.

If don't use FQDN only use IP, then the mentioned problem would not appear.

---------

- couple of config files to look at:
/home/git/gitlab/config/gitlab.yml
/home/git/gitlab-shell/config.yml
/etc/nginx/sites-enabled/gitlab

- restart service
sudo service gitlab stop/restart
sudo service nginx stop/restart
sudo /etc/init.d/gitlab restart

- turn on DEBUG log for gitlab-shell if still have trouble.
check /home/git/gitlab-shell/gitlab-shell.log

- if you meet following error:
fatal:bad default revision "HEAD"

most likely your repo has some issue, check ./git/refs, something must be missed there. and also use git branch --list --list -a to see whether correct branches listed.(should be no)

you can re-clone the project, redo checkout, and then overwrite the original ".git" with the new one.

- check status:

sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production

sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production

/home/git/gitlab-shell/bin/check

- make sure the version of gitlab-shell is correct(with above check). and also hooks path is correct in /home/git/gitlab/config/gitlab.yml.

gitlab_shell:
# REPOS_PATH MUST NOT BE A SYMLINK!!!
repos_path: /home/git/repositories/

hooks_path: /home/git/gitlab-shell/hooks/

Version control - Git

2013-08-21T00:31:00.001-07:00

[Setup git in Windows]

1, download client (GUI client and binary realese) from official website: http://git-scm.com/;

for windows, it's msysgit.

You can also download it from official site for msysgit: http://msysgit.github.io/ (moved from google to github), detail downloads : https://code.google.com/p/msysgit/downloads/list?q=full+installer+official+git

2, install it.(remember to choose second option for PATH env in order to use context sensitive menu.

3, config the user info and client look and feel.
$git config --global user.name "xxx"
$git config --global user.email "yy@zz.com"
$git config --global color.ui true

4, create local directory and clone the project
assume your local dir is C:\dev\proj
$cd /c/dev/proj
$git clone user@host : project URL

5, now you can add/update/delete source code; or add/udpate/delete branch;

Git Basics:
+ distributed VCS;
+ everything is local including commit; you can work offline with no network connection;
+ push and pull(fetch & merge) for sync among different hosts. whenever you have network connection;
+ snapshots vs deltas
+ three states:
+ working directory(work tree); modified
+ stage area(index); staged
+ git directory(repo); committed
Committed means that the data is safely stored in your local database.
Modified means that you have changed the file but have not committed it to your database yet. Staged means that you have marked a modified file in its current version to go into your next commit snapshot.

The Git directory is where Git stores the metadata and object database for your project. This is the most important part of Git, and it is what is copied when you clone a repository from another computer.
The working directory is a single checkout of one version of the project. These files are pulled out of the compressed database in the Git directory and placed on disk for you to use or modify.
The staging area is a simple file, generally contained in your Git directory, that stores information about what will go into your next commit. It’s sometimes referred to as the index, but it’s becoming standard to refer to it as the staging area.

+ configuration has three levels:
• /etc/gitconfig file: Contains values for every user on the system and all their repositories. If you pass the option--system to git config, it reads and writes from this file specifically.
• ~/.gitconfig file: Specific to your user. You can make Git read and write to this file specifically by passing the --global option.
• config file in the git directory (that is, .git/config) of whatever repository you’re currently using: Specific to that single repository. Each level overrides values in the previous level, so values in .git/config trump those in /etc/gitconfig.

+ configure
1, user identity
$ git config --global user.name "XXX"

$ git config --global user.email "YY@ZZ.com"
2, editor
$ git config --global core.editor emacs
3, diff tool
$ git config --global merge.tool vimdiff
4, show configurations
git config --list

+ remote & local
+ git remote; or git remote -v to show where you clone is from.

[Conventions]
+ use bare repo if you want to set up a git server (conceptually there is no central server, through this we just want to keep one host as a primary host);

- server repo is bare mode; no working directory. you can't use it directly for dev;

- master branch is used only for formal release; don't checkout master branch for coding;
- dev branch is a collaborative branch;

[Tips]
- you can create new branch in server side(with bare repo), but you can not check it out as there is no working tree/directory there;

- when you do a fetch that brings down new remote branches, you don’t automatically have local, editable copies of them. In other words, you don’t have a new local serverbranch — you only have an origin/serverbranch pointer that you can’t modify.
To merge this work into your current working branch, you can run git merge origin/serverbranch. If you want your own serverbranch branch that you can work
on, you can base it off your remote branch: "git checkout -b serverbranch origin/serverbranch" (looks like, use "git checkout origin/serverbranch" also works, it will automatically created a local branch with same name)

- "git checkout <branch>" : If <branch> is not found but there does exist a tracking branch in exactly one remote (call it <remote>) with a matching name, treat as equivalent to "git checkout -b <branch> --track <remote>/<branch>"

"git checkout fix" is identical as "git checkout --track origin/fix", and identical as "git checkout -b fix origin/fix"; if there is no local branch with same name already. it will automatically create the new local branch.

- to push a local branch to server, you can use "git push origin localbranch" this can push the localbranch to server. another form is "git push origin localbranch:localbranch" with same name, or with a new branch name on server side "git push origin localbranch:newnamebranch"

- push a local branch to origin/server, it will not take the remote branch(you just pushed) as the upstream of the local branch automatically. use "git branch --verbose --verbose -a" to show upstream relationship.

- use "git fetch -p"(prune) if there is some remote branch already removed/deleted, but you can not get it removed in your local host through fetch.
- to remove a remote branch, use "git push origin :featA" = "git push [alias] [null]:[branch name]". try to remember this as "git push [remotename] [localbranch]:[remotebranch]" syntax.

- use "git reset HEAD <file>" to unstage a file
- use "git checkout -- <file>..." to discard changes in working directory
- to merge from remote branch or other local branch, you need make sure there is no modified file in working tree.
- "git reset --hard <commit>/HEAD", will reset the index and working tree. Any changes to tracked files in the working tree since <commit> are discarded.
- "git reset --soft <commit>/HEAD", Does not touch the index file nor the working tree at all (but only resets the head to <commit>, just like all modes do). This leaves all your changed files "Changes to be committed", as git status would put it.

- "git reset HEAD <file>" default is mixed mode, it will reset the index but not the working tree (i.e., the changed files are preserved but not marked for commit) and reports what has not been updated. This is the default action.

- after fix the conflicts via editor, you need run "git add" to mark it as resolved (staging the file mark it as resolved in Git) so that git knows the conflict is resolved. if you use mergetool, after you exit the tool git will ask you if the merge is successful, if yes, it will stage the file and mark it as resolved.

- for newly added file, you can not use "git commit -am 'blabla' ", you need first add it then commit;

- rebase is a little bit confusing in git. but it's useful to make history linear; when you rebase from another branch, git will find the common commit between two branches and reapply every commit you have that is not in the other branch. Each time when git reapply a commit, you may need to resolve the conflict, for any conflict resolved, you need use "git add" to tell git the conflict is resolved. and then "git rebase --continue"

- set "HOME" env in windows(HOME=c:\users\uid\) to let msysgit to store the .gticonfig and other config files under this folder; if you set the HOME env as C:\, then the ".gitconfig" will be stored in "C:\users\<uid>\AppData\Local\VirtualStore" this is because git is not permitted to store files at C:\, so it use the virtualStore.

- right click the shortcut of gitbash/gitgui to set the dir to start with ("start in" box);

- two ways to create a bare repo:

A: create a repo from existing project, and then clone a bare repo;

B: git init --bare directly with a empty bare repo, and then clone it from another host and push contents;

[git server setup]

1, suppose you have a local repository setup with "git init" under dir "/opt/projects/my_project". there will be a ".git" dir under /opt/projects/my_project". go to that directory and execute the following command": git clone --bare <repo> <dir>

$ cd /opt/projects

$ mkdir my_project

$ add some source dir/files

$ git init #this will create a repo for my_project (a ".git" created under /opt/projects/my_project dir)

$ cd /opt/projects

$ git clone --bare my_project my_project.git

Initialized empty Git repository in /opt/projects/my_project.git/

after this, there will be a bare repo in /opt/projects/my_project.git

This is roughly equivalent to something like

$ cp -Rf my_project/.git my_project.git

2, putting the bare repo on a server

scp: secure file transfer over network; -r means recursively; git.exmple.com is the server to put; /opt/git is the dir to put;

$ scp -r my_project.git user@git.example.com:/opt/git

3, at this point, anyone else can clone the repo via:(need input password of user named "user" )

$ git clone user@git.example.com:/opt/git/my_project.git

4, for every developer that need to access the server(via ssh), you need

either create a user id in the server for each of them;

or create a single ‘git’ user on the machine, ask every user who is to have write access to send you an SSH public key, and add that key to the ~/.ssh/authorized_keys file of your new ‘git’ user. At that point, everyone will be able to access that machine via the ‘git’ user. This doesn’t

affect the commit data in any way — the SSH user you connect as doesn’t affect the commits you’ve recorded.

5, gen SSH pub key and send to git server

$ ssh-keygen (run under ".ssh" dir under home dir C:\users\<uid>\.ssh)

leave passphrase as empty to avoid inputing password every time you interact with git server;

$ scp id-rsa.pub user@server:/path

6, add every developer's public key to the authorized_keys file for git user. (under /home/git/.ssh)

$ cat /tmp/id_rsa.john.pub >> ~/.ssh/authorized_keys

$ cat /tmp/id_rsa.josie.pub >> ~/.ssh/authorized_keys

$ cat /tmp/id_rsa.jessica.pub >> ~/.ssh/authorized_keys

[tips part 2]

- if you use gitbash(wiondows host) to clone a github repo, and get the following error:

$ git clone https://github.com/msysgit/msysgit.git syslab

Cloning into 'syslab'...

fatal: unable to access 'https://github.com/msysgit/msysgit.git/': Failed connect to github.com:443; No error

this means you need config a proxy for it.

you can use

$git config --global http.proxy <proxy server>:<port>

BTW, if you use

$sudo -u git -H clone https://github.com/msysgit/msysgit.git syslab

then you need first switch to user git to config the http proxy(via git config --global http.proxy)

Virtualization - Hypervisor

2013-08-12T20:51:00.000-07:00

Wiki: http://en.wikipedia.org/wiki/Hypervisor

Hypervisor = Virtual machine monitor
is a piece of computer software, firmware or hardware that creates and runs virtual machines.

A computer on which a hypervisor is running one or more virtual machines is defined as a host machine. Each virtual machine is called a guest machine. The hypervisor presents the guest operating systems with a virtual operating platform and manages the execution of the guest operating systems. Multiple instances of a variety of operating systems may share the virtualized hardware resources.

Two types:

Type 1 (or native, bare metal) hypervisors run directly on the host's hardware to control the hardware and to manage guest operating systems. A guest operating-system thus runs on another level above the hypervisor.

This model represents the classic implementation of virtual-machine architectures; IBM developed the original hypervisors as bare-metal tools in the 1960s: the test tool, SIMMON, and CP/CMS. CP/CMS was the ancestor of IBM's z/VM. Modern equivalents include Oracle VM Server for SPARC, Oracle VM Server for x86, the Citrix XenServer and VMware ESX/ESXi.

Type 2 (or hosted) hypervisors run within a conventional operating-system environment. With the hypervisor layer as a distinct second software level, guest operating-systems run at the third level above the hardware. VMware Workstation and VirtualBox exemplify Type 2 hypervisors.

The classification of specific hypervisor implementations as Type 1 or Type 2 is not always clear cut. For example:

Kernel-based Virtual Machine (KVM) is implemented as a kernel module for Linux 2.6.20 which, when loaded, allows the Linux kernel to operate as a bare-metal (i.e., Type 1) hypervisor.^[2] However, as Linux is an operating system in its own right, one can argue that KVM is a Type 2 hypervisor.^[3]
Microsoft Hyper-V (released in June 2008)^[4] has also been misidentified as a Type 2 hypervisor.^[5] Both the free stand-alone version and the version that is part of the commercial Windows Server 2008 product use a virtualized Windows Server 2008 parent partition to manage the Type 1 Hyper-V hypervisor. In both cases the Hyper-V hypervisor loads prior to the management operating-system, and any virtual environments created run directly on the hypervisor, not via the management operating-system.

Attempts have been made to introduce the term Type 0 (Zero) Hypervisor to differentiate specific hypervisor implementations.^[6]^[7] However, no consensus as to the validity of this term has been reached.^[8]

-----

Several factors led to a resurgence around 2005 in the use of virtualization technology among Unix and Linux server vendors:^[10]

Expanding hardware capabilities, allowing each single machine to do more simultaneous work
Efforts to control costs and to simplify management through consolidation of servers
The need to control large multiprocessor and cluster installations, for example in server farms and render farms
The improved security, reliability, and device independence possible from hypervisor architectures
The ability to run complex, OS-dependent applications in different hardware or OS environments

virtulization - Libvirt/KVM/Qemu

2013-08-09T02:50:00.000-07:00

1, Libvirt: http://libvirt.org/
Anatomy of the libvirt virtualization library (From IBM, 2010)

Libvirt provides a hypervisor-agnostic API to securely manage guest operating systems running on a host. Libvirt isn't a tool per se but an API to build tools to manage guest operating systems. Libvirt itself is built on the idea of abstraction. It provides a common API for common functionality that the supported hypervisors implement. Libvirt was originally designed as a management API for Xen, but it has since been extended to support a number of hypervisors.

developed by Redhat.

The two fundamental differences are that libvirt calls the physical host a node, and the guest operating system is called a domain. Note here that libvirt (and its application) runs in the domain of the host Linux operating system (domain 0).
-----------------------------

Control of remote hypervisors with libvirtd
-----------------------------

To support extensibility over a wide variety of hypervisors, libvirt implements a driver-based architecture, which allows a common API to service a large number of underlying hypervisors in a common fashion.

Hypervisors that libvirt supports

Hypervisor	Description
Xen	Hypervisor for IA-32, IA-64, and PowerPC 970 architectures
QEMU	Platform emulator for various architectures
Kernel-based Virtual Machine (KVM)	Linux platform emulator
Linux Containers (LXC)	Linux (lightweight) containers for operating system virtualization
OpenVZ	Operating system-level virtualization based on the Linux kernel
VirtualBox	Hypervisor for x86 virtualization
User Mode Linux	Linux platform emulator for various architectures
Test	Test driver for a fake hypervisor
Storage	Storage pool drivers (local disk, network disk, iSCSI volume)

libvirt introduction(chinese): http://smilejay.com/2013/03/libvirt-introduction/

Term & Goals from official website:

Terminology and goals

To avoid ambiguity about the terms used, here are the definitions for some of the specific concepts used in libvirt documentation:

a node is a single physical machine
an hypervisor is a layer of software allowing to virtualize a node in a set of virtual machines with possibly different configurations than the node itself
a domain is an instance of an operating system (or subsystem in the case of container virtualization) running on a virtualized machine provided by the hypervisor

Now we can define the goal of libvirt: to provide a common and stable layer sufficient to securely manage domains on a node, possibly remote.

As a result, libvirt should provide all APIs needed to do the management, such as: provision, create, modify, monitor, control, migrate and stop the domains - within the limits of the support of the hypervisor for those operations. Not all hypervisors provide the same operations; but if an operation is useful for domain management of even one specific hypervisor it is worth providing in libvirt. Multiple nodes may be accessed with libvirt simultaneously, but the APIs are limited to single node operations. Node resource operations which are needed for the management and provisioning of domains are also in the scope of the libvirt API, such as interface setup, firewall rules, storage management and general provisioning APIs. Libvirt will also provide the state monitoring APIs needed to implement management policies, obviously checking domain state but also exposing local node resource consumption.

This implies the following sub-goals:

All API can be carried remotely though secure APIs
While most API will be generic in term of hypervisor or Host OS, some API may be targeted to a single virtualization environment as long as the semantic for the operations from a domain management perspective is clear
the API should allow to do efficiently and cleanly all the operations needed to manage domains on a node, including resource provisioning and setup
the API will not try to provide high level virtualization policies or multi-nodes management features like load balancing, but the API should be sufficient so they can be implemented on top of libvirt
stability of the API is a big concern, libvirt should isolate applications from the frequent changes expected at the lower level of the virtualization framework
the node being managed may be on a different physical machine than the management program using libvirt, to this effect libvirt supports remote access, but should only do so by using secure protocols.
libvirt will provide APIs to enumerate, monitor and use the resources available on the managed node, including CPUs, memory, storage, networking, and NUMA partitions.

So libvirt is intended to be a building block for higher level management tools and for applications focusing on virtualization of a single node (the only exception being domain migration between node capabilities which involves more than one node).

2, relationship between QEMU & KVM （chinese)

http://blog.chinaunix.net/uid-23769728-id-3256677.html

QEMU is an independent virtualization tool/env, it can work without KVM. but if we have KVM and hardware(CPU) support (e.g. Intel VT), QEMU can use KVM to improve the performance in CPU virtualization.

KVM has two kernel modules: kvm.ko & kvm_intel.ko(kvm_amd.ko), these modules are used to provide CPU virtualization. for others, like IO virtuzalition, KVM based on QEMU's solution, all in all, it's qemu-kvm. KVM has dependency on QEMU.

difference between NFV and SDN

2013-08-02T01:38:00.001-07:00

Network Functions Virtualisation aims to transform the way that network operators architect
networks by evolving standard IT virtualisation technology to consolidate many network equipment
types onto industry standard high volume servers, switches and storage, which could be located in
Datacentres, Network Nodes and in the end user premises, as illustrated in Figure 1. It involves the
implementation of network functions in software that can run on a range of industry standard server
hardware, and that can be moved to, or instantiated in, various locations in the network as required,
without the need for installation of new equipment.

As shown in Figure 1, Network Functions Virtualisation is highly complementary to Software Defined Networking (SDN), but not dependent on it (or vice-versa). Network Functions Virtualisation can be implemented without a SDN being required, although the two concepts and solutions can be combined and potentially greater value accrued.

Figure 1. Network Functions Virtualisation Relationship with SDN

Network Functions Virtualisation goals can be achieved using non-SDN mechanisms, relying on the techniques currently in use in many datacentres. But approaches relying on the separation of the control and data forwarding planes as proposed by SDN can enhance performance, simplify compatibility with existing deployments, and facilitate operation and maintenance procedures. Network Functions Virtualisation is able to support SDN by providing the infrastructure upon which the SDN software can be run. Furthermore, Network Functions Virtualisation aligns closely with the SDN objectives to use commodity servers and switches.

NFV is about the virtualization of function box, like router, firewall, etc; Just put those functions in VMs, but control and data are still together in one box; SDN is about the separation of control plan and data plane of one node.

NFV covers all L2-L7 function boxes; while SDN focus on L2-L3 switches/routers;

SDN and NFV – Working Together?

Let’s look at an example of how SDN and NFV could work together. First, Figure 2 shows how a managed router service is implemented today, using a router at the customer site.

Figure 2: Managed Router Service Today

NFV would be applied to this situation by virtualizing the router function, as shown in Figure 3. All that is left at the customer site is a Network Interface Device (NID) for providing a point of demarcation as well as for measuring performance.

Figure 3: Managed Router Service Using NFV

Finally, SDN is introduced to separate the control and data, as shown in Figure 4. Now, the data packets are forwarded by an optimized data plane, while the routing (control plane) function is running in a virtual machine running in a rack mount server.

Figure 4: Managed Router Service Using NFV and SDN

The combination of SDN and NFV shown in Figure 4 provides an optimum solution:

An expensive and dedicated appliance is replaced by generic hardware and advanced software.
The software control plane is moved from an expensive location (in dedicated platform) to an optimized location (server in a data center or POP).
The control of the data plane has been abstracted and standardized, allowing for network and application evolution without the need for upgrades of network devices.

1, SDN & NFV concept is from SS;

2, map has a summary of key involvers wihtin;

SDN - OpenNetSummit 2013

2013-07-30T21:01:00.000-07:00

1, Value proposition
2, real-world use case
3,Market Segments
Data Center - Scalability Data Center Networks
Carriar - Service Provider Data Networks
Enterprise
- Streamlined and Dynamic, but Not a technology Nor a market
- Standardized platform for Development of Novel Services

why a new net architecture is needed:
not cost-effective;
not agile enough: time to market; complex/manually intensive provisioning/configuration/network management;
not oriented towards services/end users: static configuration/static traffic patterns and policy
hardware/infrastructure oriented: ports/bandwidth/ids/vlan tags etc

SDN architecture: 25'30"-27'50"
Programmability/contralized Intelligence/Abstraction
OpenFlow: 28'10"-30'20"
southbound protocol
break the layer boundaries(of OSI 7 layer);

value proposition: 33'00"
automation -> OpEx
optimization -> CapEx -> more intelligent processing and a better utilization of the capacity;(more efficient bandwidth - mutli-tenants, multi networks run on the same infrastructure)
Monetization -> New Revenues (Dynamics, intelligence pricing)

applications of Segments: 35'40"
[Data Center]
1, Virtualization: VM mobility/Domain isolation
2, Hybrid Clouds: span public and private DC
3, Multi-Tenancy: multiple customers share services
4, Disaster Recovery: Improved visibility and managememt
[Carrier network] more efficient usage of carrier bandwidth, especially mobile/wireless bandwidth;
1, Mobile Edge: seamless roaming for 3G/4G/Wifi
2, service migration to cloud: span public and private cloud
3, service monitoring: automated provisioning / centralized network view
4, traffic steering: content delivery services
[NFV] carrier lead ETSI

<Network Virtualization>

<Carrier> manage fabric instead fo manage boxes
1, Optimization
- network utilization: Performance-on-Demand (PoD) - e.g. NaaS / Efficient multi-layer traffic engineering/fixed access utilization
- Service performamce: netrwork aware load balancing/Mobile-WiFi offload(or n-casting)/traffic steering for Mobile QoE
- operation: largely centralized upgrades, features,services additions/high-fidelity simulation, straightforward push to production;
2, Monetization
- intellegence : better matching of network supply & user/application demand -- maximized economic utility : PoD- e.g. NaaS; QoS

- Colt: L3 CPE virtualization/ NFV

<Data Center>

<Enterprise>
challenge: multi-VLAN / VM mobility (IP addr mgmt, access control)
requirements: Live migration / Traffic prioritization / Policy enforcement/ Ease of operations and maintenance
SDN solution: (NEC+Hyper-V)
• Live VM migration without network reconfiguration
• Integrated virtual and physical network management
• Greater flexibility/faster deployment of network services

• Open Standard – more infrastructure options

<SP-SDN>
1, NTT: traffic on demand(by user) / VLAN-MPLS(Cloud)

2, Ericsson+Telstra: virtualization of the aggregation netowkr (NFV),
IP edge of the network & service chaining(1st SDN use case for SP)
where treat different data flows different ways & not route traffic in and out of boxes that don't need to.
so that Operator can
A) create service that use less network resources.(lower capex/opex)
B) Create service more quickly because we can route services more dynamically and quick TTM without traditional integrating to every network and every network box.
Using traffic steering as part of SDN allows to individualize products more quickly and more readily.

Telstra expects to move to field trials later in 2013 and to take the first steps toward operationalizing SDN in its production network in 2014.

Ericsson tech talk about Sp-SDn: cross domain (Radio/Core/DC)

blog to read: SP SDN gets real

Difference between SDN & NFV: SDNCentral article

- http://www.nojitter.com/post/240158913/whats-the-role-of-sdn-and-uc

IaaS/PaaS players to watch 2013

2013-07-17T02:33:00.000-07:00

http://shakayumi.wordpress.com/2013/03/04/thirteen-aws-cloud-alternatives-to-watch-in-2/
[IaaS]
- AWS

- Verizon Terremark : more secured solution

Verizon/Terremark has already launched cloud services compliant with with
the Health Insurance Portability and Accountability Act (HIPAA),
the Federal Information Security Management Act (FISMA), and
the National Institutes of Standards and Technology (NIST) 800-53 Security and Privacy Controls.
Their compliance cloud services include Collocation, Managed Hosting, Enterprise Cloud, Enterprise Cloud Express Edition and Enterprise Cloud Private Edition, for each which Verizon offers a business associate agreement (BAA), which guarantees that the data in question is secure, supports geolocation requirements, and is auditable by tenants.

- CenturyLink/Savvis : security and reliability
Enterprises with mission-critical workloads and/or sensitive data like the idea of cloud for its flexibility, cost savings and automation, but they remain wary of potential security breaches or data loss due to outages. Many of these enterprises shun Amazon Web Services and look to Infrastructure as a Service providers with a history in uptime and reliability, such as CenturyLink/Savvis.
Savvis is also addressing interoperability concerns while making a push for hybrid cloud — having the ability to work with private cloud and move workloads to public cloud when necessary.

- Goolge Compute Engine : the greatest threat to Amazon's dominance in IaaS;
Google’s Compute Engine (GCE) remains in limited preview, and it doesn’t yet support Windows, What makes Google even more dangerous is that it owns fiber-optic networks, unlike Amazon, which rely on ISPs. Google has been ahead of the software-defined networking (SDN) game as well, and can compete with Amazon on pricing and performance right out of the gate.
Amazon is still ahead of Google when reaching out to enterprise customers, but Google could change this dynamic if it chose to. With GCE expected to open up to general availability this year, Google could be a major disruptor in the Infrastructure as a Service market.

- Rackspace: fanatical customer support
Rackspace adjusted its managed cloud support strategy from a pooled model of resources to a more traditional dedicated support team model.

In addition to its managed services work, Rackspace cloud also appears to be inching toward the Platform as a Service (PaaS) realm with its Service Registry, a tool that enables cloud consumers to orchestrate the process of assigning a workload and having a cloud service respond to that workload automatically.

Rackspace has also played a significant role in shaping OpenStack. The company began offering public cloud services based on OpenStack in August 2012, setting the stage for other major cloud service providers to follow. If Rackspace continues to broaden to its IaaS managed cloud services offerings as well as evolve its PaaS products for the burgeoning DevOps market, it should have an exciting year.

- SoftLayer : Bare Metal Cloud
The Dallas-based company, one of the largest privately owned cloud infrastructure providers, believes its competitive advantage lies in its array of product options and automation offerings designed to more finely customize cloud environments.

One of those product options is the “bare metal cloud,” in which SoftLayer removes the hypervisor from the mix and offers customers a choice in customizing its hardware infrastructure, including one to 64 processors and access to solid-state drive (SSD) storage and a high-speed global network that can be provisioned in real time. SoftLayer’s technology doesn’t come with lengthy commitments, which minimizes users’ financial risk.

A lot of users like the bare metal cloud, where SoftLayer just give them the raw power for running ‘big data’ or large databases. SoftLayer focus on customers that have performance issues where they need high disk I/O and higher network speeds and AWS can't meet this need.

Its CloudLayer services can either be integrated with dedicated servers and services; they also operate in a standalone mode that can be delivered on demand and self-managed. It isn't as big as AWS from a scale perspective, but they have more options available to users that let them fine-tune their cloud.

- ProfitBricks : Speed/InfiniBand
By adopting the venerable InfiniBand protocol, a switched fabric communications link more traditionally used in high-performance computing (HPC), company officials believe ProfitBricks has a significant speed and performance advantage over its competitors.

We were the first to bring InfiniBand to IaaS market, which changes the dynamics in a couple of ways: through higher performance and greater vertical scalability — the whole up and down provisioning aspect; and we are better able to give customers more for their money by including load balancing, firewalls and redundant networks,

In November 2012, Microsoft announced it plans to provide big compute capabilities in Windows Azure when it debuts its first virtualized InfiniBand RDMA (remote direct memory access) network, for those organizations looking to do HPC analyses. Another headwind ProfitBricks could face is that InfiniBand is pricier than other IaaS options.

- Time Warner/NaviSite :
NaviSite is splitting the difference between colocation and managed services with pure cloud computing,

Over the past two years, NaviSite has shifted its focus from colocation to Infrastructure as a Service (IaaS),

- CloudSigma : flexible/give customer control
CloudSigma, an IaaS provider based in Zurich, Switzerland, was founded in 2009 as an alternative to public cloud giants, as a way to combine the flexibility and scalability of public cloud with the control customers were used to with managed hosting — a mission statement that could appeal to cloud-wary enterprises. Facilitate a more collaborative, flexible relationship between public cloud providers and customers.

CloudSigma's solution is really somewhere between managed services and pure cloud computing, Some enterprises will find that more desirable considering they are not giving up total control.”

[PaaS]
- EMC/VMware
Pivotal : combines cloud application development and big data analytics properties into a 1,400-person “virtual organization” within EMC, led by former VMware CEO Paul Maritz.

在今年3月份的VMware投资者大会上，VMware和EMC联合宣布将成立合资公司Pivotal，由EMC控股，EMC持有VMware约80%股份。Pivotal将由两家公司的数据分析和云应用资产合并而成，并且VMware公司前任CEO Paul Maritz出任Pivotal公司CEO，Pat Gelsinger接管VMware公司，David Goulden继续掌管EMC。北京时间2013年4月26日，Pivotal公司正式宣布成立，开始作为一个独立的实体运营。同时披露Pivotal One新一代PaaS计划，Pivotal One将是第一个集成新的分片式数据(Data Fabric)、现代编程框架、云便携性和遗留系统支持的平台。

EMC、VMware、Pivotal三家公司对详细业务进行了布局，其中EMC从事基础设施，VMware中间层，Pivotal则定位在顶端应用。Pivotal还宣布了Pivotal One计划，该计划旨在打造新一代企业PaaS平台。其中，Greenplum和Cloud Foundry是新公司的核心资产。Cloud Foundry是VMware推出的开源PaaS平台，Greenplum专注在大数据分析应用。Cetas作为大数据初创公司，也是Pivotal业务的补充。

这三家公司被业界称为“铁三角”，而且Pivotal之后，EMC系的企业级市场战略更加完善。三家公司有了明确分工，Pivotal定位在下一代云计算和大数据应用市场，VMware致力于软件定义数据中心，而EMC则专注在信息基础架构。

- CloudBees: Java PaaS and continuous delivery

Unlike other recently launched Platform as a Service vendors, CloudBees is focused on Java, JRails and Grails, rather than trying to be all things to all customers in terms of languages.

Instead, CloudBees offers vertical integration of the software process, pulling in continuous integration services through its Jenkins plugin, and it boasts partnerships with a number of ecosystem partners, including PaperTrail for log sequencing and New Relic for monitoring. Code is committed in CloudBees, tested by Jenkins and then deployed automatically to the CloudBees platform without the user having to worry about the care and feeding of each system along the way.

“Having Jenkins set up was a huge advantage for us, and allowed more time to work on core products,” said Mario Cruz, chief technology officer (CTO) and co-founder of Choose Digital, which switched to CloudBees from Heroku about a year ago.
“CloudBees is as much of a must-have to us as GitHub,” said Adrian Cole, founder of the jclouds open source project, which gets free space on CloudBees and has collaborated on the Jenkins plugin. The Jenkins plugin is also used by Google App Engine.

- Engine Yard : reduce vender lock-in concerns
Platform as a Service (PaaS) allows companies to quickly spin up applications to stay competitive in cutthroat verticals. But the ugly phrase “vendor lock-in” still looms over development teams’ heads. Engine Yard is hoping to change that.

By adding a new architecture to its Engine Yard Cloud, the San Francisco-based PaaS provider hopes to free developers from the need to re-architect apps for the platform. Its Infrastructure Abstraction layer gives developers the ability to deploy an application without worrying what other vendor(s) they are already running. And by partnering with various Infrastructure as a Service (IaaS) providers, Engine Yard hopes to minimize the dreaded fear of lock-in.

Part of its aim in minimizing developer strain might come from a growing emphasis on DevOps, which “represents the convergence and tighter integration of the build, plan, deploy and manage” cycle. makes the difference between the classic terms of ‘developer’ and ‘operations’ less meaningful.

- DotCloud : right price

dotCloud is another platform gunning for market leader Heroku, which launched in 2011, boasts tens of thousands of developer customers and is making inroads into enterprises through “guerilla” efforts — where developers at Fortune 1000 organizations lobby internally to use dotCloud for a specific project.

Designed for a new generation of cloud-native apps, with a focus on fast performance for Java development.

The Platform as a Service provider was also the only service available about a year ago that offered a stack that included Java, MongoDB, the Spring framework, the Dojo library and other specialized elements with an uptime service-level agreement (SLA). DotCloud’s SLA builds on Amazon Web Services IaaS’ 99.95% annual uptime guarantee.

- HeroKu: Market leader in PaaS

With a development community that has turned out well over 100,000 applications, along with the financial backing of Salesforce.com, which acquired Heroku in December 2010, Heroku seemed a lock to dominate the market among open source developers.

One reason for Heroku’s security is that it allows developers to build and deploy apps using not only Ruby, but also Node.js, Java, Python and Scala. Another reason is the array of management and fault-tolerant tools built into the base platform.

Late last year, Heroku delivered a couple of products that were received favorably, including Heroku Enterprise for Java, a service that allows IT shops to build and run applications in the cloud quickly. The product lets cloud providers gravitate to a continuous delivery model sooner rather than later. Additionally, the company, which introduced the idea of add-ons for PaaS providers, released version 2.0 of its Heroku Add-Ons Marketplace. The new release better enables business processes for cloud services providers, something that has gained more focus among customers.

Because Ruby has proven to be the popular programming language among developers creating social and mobile apps — the hottest app development segments at the moment – the Heroku cloud market position appears to be further strengthened.

Terremark- Acquired by Verizon

2013-07-16T23:26:00.003-07:00

Manny Medina born in Cuba setup Terremark in 1980's, a real estate company constructing office buildings. During the dot-com era more and more of his buildings were leased to computer data centers and the company morphed into an information technology services company itself starting with the NAP of the Americas,^[2] Terremark's flagship facility, in downtown Miami.

NAP of the Americas one of the most significant telecommunications projects in the world. The facility was the first purpose-built, carrier-neutral network access point (NAP) and is the only facility of its kind specifically designed to link Latin America with the rest of the world. The building is a 750,000 square foot, purpose-built datacenter. The equipment floors are 32 feet above sea level. The building is designed to withstand a Category 5 hurricane with approximately 19 million pounds of concrete roof ballast. It has 7 inch thick steel reinforced concrete exterior panels.^[3] NAP of the Americas carries 95% of the data traffic between North and South America.

Verizon Communications on January 27, 2011 announced it would buy Terremark Worldwide for $19 a share, in a deal valued at $1.4 billion. The deal for Verizon is a play on the rapid growth in cloud computing, an area where the company has been lagging behind competitors like AT&T.^[4] Terremark will now become a wholly owned subsidiary under Verizon, keeping its name, its employees and headquarters in Miami.

Terremark now operates 13 data centers throughout the world. Medina will receive about $83 million out of the Verizon deal from his 4.4 million shares, almost 7 percent of the company.^[5] On May 10, 2011 Manny Medina received the Jay Malina Award from the Beacon Council. They stated that Terremark was Miami's leading force in the technology industry.^[6] Verizon completed its acquisition of Terremark on April 12, 2011.^[7] Medina left the company at the time of the takeover and Terremark has gone through two presidents in one year. Currently three high ranking executives are running the business.

Wiki page: http://en.wikipedia.org/wiki/Terremark

Cloud Computing - Amazon Web Service

2013-07-07T21:05:00.005-07:00

1, From Wiki page: https://en.wikipedia.org/wiki/Cloud_computing

After the dot-com bubble, Amazon played a key role in all the development of cloud computing by modernizing their data centers, which, like most computer networks, were using as little as 10% of their capacity at any one time, just to leave room for occasional spikes.

Having found that the new cloud architecture resulted in significant internal efficiency improvements whereby small, fast-moving "two-pizza teams" (teams small enough to feed with two pizzas) could add new features faster and more easily, Amazon initiated a new product development effort to provide cloud computing to external customers, and launched Amazon Web Services (AWS) on a utility computing basis in 2006.^[11]^[12]

In early 2008, Eucalyptus became the first open-source, AWS API-compatible platform for deploying private clouds. In early 2008, OpenNebula, enhanced in the RESERVOIR European Commission-funded project, became the first open-source software for deploying private and hybrid clouds, and for the federation of clouds.^[13] In the same year, efforts were focused on providing quality of service guarantees (as required by real-time interactive applications) to cloud-based infrastructures, in the framework of the IRMOS European Commission-funded project, resulting to a real-time cloud environment.^[14]

2, From Wiki page: https://en.wikipedia.org/wiki/Amazon_Web_Services

Officially launched in 2006,^[3] Amazon Web Services provide online services for other web sites or client-side applications. Most of these services are not exposed directly to end users, but instead offer functionality that other developers can use in their applications. Amazon Web Services’ offerings are accessed over HTTP, using REST and SOAP protocols. All services are billed based on usage, but how usage is measured for billing varies from service to service.

In late 2003, Chris Pinkham and Benjamin Black presented a paper proposing the company could build and sell a set of services based on the experience of building and operating the infrastructure for Amazon.com.^[4] The first AWS service launched for public usage was Simple Queue Service in November 2004.^[5] Amazon EC2 was built by a team in Cape Town, South Africa under Pinkham and lead developer Chris Brown.^[6]

3, List of AWS products[edit]

Compute[edit]

Amazon Elastic Compute Cloud (EC2) provides scalable virtual private servers using Xen.
Amazon Elastic MapReduce (EMR) allows businesses, researchers, data analysts, and developers to easily and cheaply process vast amounts of data. It uses a hosted Hadoop framework running on the web-scale infrastructure ofEC2 and Amazon S3.

Networking[edit]

Amazon Route 53 provides a highly available and scalable Domain Name System (DNS) web service.
Amazon Virtual Private Cloud (VPC) creates a logically isolated set of Amazon EC2 instances which can be connected to an existing network using a VPN connection.
AWS Direct Connect provides dedicated network connections into AWS data centers, providing faster and cheaper data throughput.

Content Delivery[edit]

Amazon CloudFront, a content delivery network (CDN) for distributing objects to so-called "edge locations" near the requester.

Storage & Content Delivery[edit]

Amazon Simple Storage Service (S3) provides Web Service based storage.
Amazon Glacier, Provides a very low cost long-term storage option (when compared to its S3 service). High redundancy and availability, but low-frequent access times. Ideal for archiving data.
AWS Storage Gateway, an iSCSI block storage virtual appliance with cloud-based backup.
Amazon Elastic Block Store (EBS) provides persistent block-level storage volumes for EC2.
AWS Import/Export, accelerates moving large amounts of data into and out of AWS using portable storage devices for transport.

Database[edit]

Amazon DynamoDB provides a scalable, low-latency NoSQL online Database Service backed by SSDs.
Amazon ElastiCache provides in-memory caching for web applications. This is Amazon's implementation of Memcached.
Amazon Relational Database Service (RDS) provides a scalable database server with MySQL, Informix,^[15] Oracle, and SQL Server support.
Amazon Redshift provides petabyte-scale data warehousing with column-based storage and multi-node compute.
Amazon SimpleDB, allows developers to run queries on structured data. It operates in concert with EC2 and S3 to provide "the core functionality of a database."
AWS Data Pipeline, provides reliable service for data transfer between different AWS compute and storage services(e.g. Amazon S3, Amazon RDS, Amazon DynamoDB, Amazon EMR.). In other words this service is simply a data-driven workload management system, which provides a simple management APIs to managing and monitoring of data-driven workloads in cloud applications.

Deployment[edit]

Amazon CloudFormation provides a file based interface for provisioning other AWS resources.
AWS Elastic Beanstalk provides quick deployment and management of applications in the cloud.
AWS OpsWorks for configuration of EC2 services using Chef.

Management[edit]

Amazon Identity and Access Management (IAM), an implicit service, the authentication infrastructure used to authenticate access to the various services.
Amazon CloudWatch, provides monitoring for AWS cloud resources and applications, starting with EC2.
AWS Management Console (AWS Console), A web-based point and click interface to manage and monitor the Amazon infrastructure suite including (but not limited to) EC2, EBS, S3, SQS, Amazon Elastic MapReduce, andAmazon CloudFront. Amazon also makes available a mobile application for the Android which has support for some of the management features from the console.

App Services[edit]

Amazon CloudSearch provides basic full text search and indexing of textual content.
Amazon DevPay, currently in limited beta version, is a billing and account management system for applications that developers have built atop Amazon Web Services.
Amazon Elastic Transcoder (ETS) provides video transcoding of S3 hosted videos, marketed primarily as a way to convert source files into mobile-ready versions.
Amazon Flexible Payments Service (FPS) provides an interface for micropayments.
Amazon Simple Email Service (SES) provides bulk and transactional email sending.
Amazon Simple Queue Service (SQS) provides a hosted message queue for web applications.
Amazon Simple Notification Service (SNS) provides a hosted multiprotocol "push" messaging for applications.
Amazon Simple Workflow (SWF) is a workflow service for building scalable, resilient applications.

AWS 101 by Amazon: http://www.youtube.com/watch?v=uwhHSosGF0w (Jul 2013)

Introduction to AWS:http://www.youtube.com/watch?v=CaJCmoGIW24 (Oct 2011)

Good talk about the benefits to move to AWS cloud: http://www.youtube.com/watch?v=yMJ75k9X5_8 (Jan 2013)

Auto-Scaling: Launch configuration, group, policy(metrics based or schedule based)

Why adopting cloud computing?
- Variable expense: replace capital expenditure with variable expense: CapEx -> OpEx
- Economics of scale: lower variable expense than companies can achieve themselves; the more usage, the cheaper cost per unit;
- Elastic capacity: No need to guess capacity requirements and over-provision: demand/time chart;
- Speed and agility: infrastructure in minutes not weeks;
- focus on business: not undifferentiated heavy IT lifting ( Data Center, Power, Cooling, Cabling, Racks, Servers, Storage/Labour, buy and install new hardware, setup and config new software)
- Global Reach, Go global in minutes and reach a global audience

image

2013-06-28T02:22:00.001-07:00

http://en.wikipedia.org/wiki/ISO_image

Disk image file formats

Optical discs	CCD/IMG/SUB CUE/BIN CSO DAA IBP/IBQ ISO ISZ MDS/MDF MDX NRG UIF

Hard disks	DMG IMG ISO NDIF QCOW UDIF VDI VHD VMDK WIM

Floppy disks	ADF/ADZ DC42/DART DCF DMS IMG/IMA/IMZ VFD

Flash drives	ISO

CD-DA	Disc Description Protocol

http://en.wikipedia.org/wiki/Disk_image TO BE READ!!!

talk about virtualization.

Ubuntu Cloud Archive

2013-06-28T02:15:00.000-07:00

to support new releases of openstack on Ubuntu 12.04 LTS release.

official site from Ubuntu

How to Enable and Use

1, You'll first need to add the cloud archive gpg key into your ubuntu-keyring by running the following command:

sudo apt-get install ubuntu-cloud-keyring

2, Next, to get access to the Ubuntu Cloud archive, please add the following entries to your /etc/apt/sources.list: (Grizzly)

 # The primary updates archive that users should be using
 deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/grizzly main

 # Public -proposed archive mimicking the SRU process for extended testing.
 # Packages should bake here for at least 7 days. 
 #deb  http://ubuntu-cloud.archive.canonical.com/ubuntu precise-proposed/grizzly main

Now run:

sudo apt-get update

to update your package listings and then proceed to install/upgrade your openstack packages.

Storage / Volume / Instance

2013-06-28T00:37:00.000-07:00

[two types of storage]
The cinder-volume service which provide persistent block storage vs. the ephemeral storage provided by the instance flavor.

- Ephemeral
Rebooting the VM or restarting the host server will not destroy ephemeral data. data will be deleted if the instance is terminated;
In the typical use case an instance's root filesystem is stored on ephemeral storage.
In addition to the ephemeral root volume all flavors except the smallest, m1.tiny, provide an additional ephemeral block device varying from 20G for the m1.small through 160G for the m1.xlarge by default.
This is presented as a raw block device with no partition table or filesystem. Cloud aware operating system images may discover, format, and mount this device. For example the cloud-init package included in Ubuntu's stock cloud images will format this space as an ext3 filesystem and mount it on /mnt.

- Persistent (Volume)
Volume storage is independent of any particular instance and is persistent. Volumes are user created and within quota and availability limits may be of any arbitrary size.
When first created volumes are raw block devices with no partition table and no filesystem. They must be attached to an instance to be partitioned and/or formatted. Once this is done they may be used much like an external disk drive.
It is possible to configure a volume so that it is bootable and provides a persistent virtual instance similar to traditional non-cloud based virtualization systems. In this use case the resulting instance may still have ephemeral storage depending on the flavor selected, but the root filesystem (and possibly others) will be on the persistent volume and thus state will be maintained even if the instance it shutdown.

Volumes do not provide concurrent access from multiple instances. For that you need either a traditional network filesystem like NFS or CIFS or a cluster filesystem such as GlusterFS.

OpenStack Operation Guide - Provisioning & Deployment

2013-06-27T02:37:00.000-07:00

Automatic Deploylment: two ways.

Way 1 - An automated deployment system installs and configures operating systems on new servers, without intervention, after the absolute minimum amount of manual work (physical racking, MAC to IP assignment, power configuration, and so on).

[Installation]
solutions rely on wrappers around PXE boot and TFTP servers for the basic operating system install, then hand off to an automated configuration management system.

- PXE boot

The Preboot eXecution Environment (PXE, also known as Pre-Execution Environment; sometimes pronounced "pixie") is an environment to boot computers using a network interface independently of data storage devices (likehard disks) or installed operating systems.

[configuration]

Ubuntu and Red Hat Linux both include mechanisms for configuring the operating system, including preseed and kickstart, that you can use after a network boot.

- Preseed (DebianInstaller)

Preseeding provides a way to set answers to questions asked during the installation process, without having to manually enter the answers while the installation is running. This makes it possible to fully automate most types of installation and even offers some features not available during normal installations.

Most of the questions asked by DebianInstaller can be preseeded by setting the answers in the debconf database.

- kickstart

The Red Hat Kickstart installation method^[1] is used primarily (but not exclusively) by the Red Hat Enterprise Linux operating system to automatically perform unattended operating system installation and configuration. Red Hat publishes Cobbler as a tool to automate the Kickstart configuration process.

A Kickstart file contains a series of options, to be passed to the Anaconda installer, that describes how to set up the system. It may also include custom scripts to be run before or after the installation.

A Kickstart installation^[3] follows these four steps:

The machine is booted from a CD/DVD, a USB device or over a network using PXE and the network protocols Dynamic Host Configuration Protocol and Trivial File Transfer Protocol.
The Kickstart file is downloaded from the boot media or network (most common).
An Anaconda installation is automatically launched and reads the Kickstart file for the location of the Installation Tree. The tree can reside on the boot media or network.
After accessing the Installation Tree, the installer attempts an unattended installation. If any required information is missing from the Kickstart file, or the file is configured incorrectly, the installer may prompt the user for additional information.

Way 2 - Alternatively, you can use an image based approach for deploying the operating system, such as systemimager.

- systemimager

SystemImager is software that automates Linux installs, software distribution, and production deployment.

SystemImager makes it easy to do automated installs (clones), software distribution, content or data distribution, configuration changes, and operating system updates to your network of Linux machines. You can even update from one Linux release version to another!
It can also be used to ensure safe production deployments. By saving your current production image before updating to your new production image, you have a highly reliable contingency mechanism. If the new production enviroment is found to be flawed, simply roll-back to the last production image with a simple update command!
Some typical environments include: Internet server farms, database server farms, high performance clusters, computer labs, and corporate desktop environments.

areas to be careful"

- Disk Partitioning & RIAD

- Network configuration

NIC Bonding

Automated Configuration

- Puppet

It's coming from Puppet Labs.

Puppet is a tool designed to manage the configuration of Unix-like and Microsoft Windows systems declaratively. The user describes system resources and their state, either using Puppet's declarative language or a Ruby DSL (domain-specific language). This information is stored in files called "Puppet manifests". Puppet discovers the system information via a utility called Facter, and compiles the Puppet manifests into a system-specific catalog containing resources and resource dependency, which are applied against the target systems. Any actions taken by Puppet are then reported.

Remote Management

- Out-of-band management

In computing, out-of-band management (sometimes called lights-out management or LOM) involves the use of a dedicated management channel for device maintenance. It allows asystem administrator to monitor and manage servers and other network equipment by remote control regardless of whether the machine is powered on, or if an operating system is installed or functional.

By contrast, in-band management like VNC or SSH is based on software that must be installed on the remote system being managed and only works after the operating system has been booted. This solution may be cheaper, but it does not allow access to BIOS settings, or the reinstallation of the operating system and cannot be used to fix problems that prevent the system from booting.

Both in-band and out-of-band management is usually done through the network connection, but an out-of-band management card can use a physically separated network connector if preferred. A remote management card usually has an at least partially independent power supply, and can power the main machine on and off through the network.

- IPMI

The Intelligent Platform Management Interface (IPMI) is a standardized computer system interface used by system administrators for out-of-band management of computer systems and monitoring of their operation.

The development of this interface specification was led by Intel Corporation and is supported by more than two hundred computer systems vendors.^[1] Cisco, Dell, Hewlett-Packard, Intel, and NEC Corporation announced IPMI v1.0 on 1998-09-16, v1.5 on 2001-03-01, and v2.0 on 2004-02-14.

Curl

2013-06-26T20:15:00.002-07:00

use the following one highlighted with red color for Windows:

Win32 - Generic
Win32	7.30.0	binary			Dirk Paehl
Win32	7.30.0	binary	SSL	SSH	Dirk Paehl
Win32 2000/XP	7.31.0	binary	SSL		Günter Knauf	2.01 MB
Win32 2000/XP	7.31.0	binary	SSL	SSH	Günter Knauf	2.39 MB
Win32 2000/XP	7.31.0	libcurl	SSL		Günter Knauf	3.32 MB

Also need download openssl

Platform	Version	URL
Windows	various	shininglightpro.com

The followin version can be used.

Win32 OpenSSL v1.0.1e

16MB Installer

Installs Win32 OpenSSL v1.0.1e (Recommended for software developers by the creators of OpenSSL). Note that this is a default build of OpenSSL and is subject to local and state laws. More information can be found in the legal agreement of the installation.

if we need proxy, e.g. use it in office. here is the parameters to use:

curl -x proxy[:port ] -I www.baidu.com

OpenStack - session 1

2013-06-26T20:07:00.004-07:00

Here are couple of good videos to start with:

- OpenStack 101 from Rackspace. 6 mins
a quick introduction about Openstack. Worth watching if you are not familiar of OpenStack.

- Introduction to OpenStack from Sandy Walsh Nov 2012 32 mins
Good history overview and concept overview.

- OpenStack Grizzly Architecture 101 from Ken Pepple Apr 2013(Summit) 43 mins
a good explanation of the architecture of Grizzly release of OpenStack.

- OpenStack Grizzly Dashboard Demo from Mark Collier 2013 7 mins
COO of OpenStack Foundation

- OpenStack 101 from Joshua Mckenty Oct 2012 43 mins
has anology between linux kernel and openstack. quite interesting. advocate guys not download from openstack directly, and should try some distributions from other companies. kind of advertisement. ;)

You can find a good writing about the Grizzly architecture from Ken Pepple:

http://www.solinea.com/2013/06/15/openstack-grizzly-architecture-revisited/

- writen in Python
- deamon/service using WSGI Paste
- Dashboard is a Django app

1, Identity (Keystone) <-> AWS IAM
includes back-end of identity, catalog, token, policy
2, Dashboard (Horizon) Django app <-> AWS deployment/Management services like CloudWatch/CloundFormation
3, Object Storage (Swift), object basically is files <-> AWS S3
not FTP server, not mount as file share, not file server.
should be accessed (via API or HTTP) through swift-proxy
account / container implemented on SQLlite;
object store implemented on disk;
swift-proxy is the deamon
4, Image service (Glance) <-> no
glance-api is the deamon
upload/download images(VM images), query/assign meta-data to images;
glance-registry stores only meta-data about images, not images itself;
actual images have plugable backend to support the images storage,
usually via swift(as actual image repository)
misc: caching images, pre-fetching images;
5, Compute (Nova)
initiates most of the orchestration activities;
policy - quota checks;
nova-compute orchestrate hypervisor;(is a worker deamon that creates and terminates virtual machine instances via hypervisor's API)
<<openstack is not hypervisor, it's not virtualization, it's a framework to control virtualization>>
support VMware(VMware API), Hyper-V, KVM(libvirt), Xen(Xen API).
all below interactions are via queue services(messages)
nova-api -> nova-scheduler -> nova-compute (->glance-api) -> nova-conductor
nova-scheduler determine which nova compute server to use;
nova-compute retrieve image, talk to hypervisor to create the instance;
nova-conductor mediate the access to database for nova;
database stores the all of the state of your current running cloud;
queue can be implemented with different technologies;
nova-consoleauth provides console service;
6, Block Storage(Cinder)
moved out from Nova;
Cinder-Volume; interacts with a variety of storage providers;
Cinder-scheduler;
7, networking (quantum)
allows to create layer2 or layer3 networks;
plugins & agents from different providers;

DevStack setup

2013-06-26T19:44:00.002-07:00

1, if you met the issue/error related to keystone:

keystone-manage can stopped unexpectedly;

This is due to MySQL root password. If you have MySQL installed before the devstack setup. The root password may not be the same as the ADMIN_PASSWORD in localrc config file.

use the following command to change the root password for MySQL:

$ mysqladmin -u root -p'oldpassword' password newpass

refer to the following article:

http://www.cyberciti.biz/faq/mysql-change-root-password/

2, by default swift and quantum are not installed by the stack.sh.

refer to wiki of QuantumDevstack for installation of Quantum.

disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service quantum
# Optional, to enable tempest configuration as part of devstack
enable_service tempest

to enable swift, add followings to localrc
enable_service swift

3, if met quantum server can not be started, if may be caused by an old version of oslo.
refer to following bug fixing on launchpad
https://bugs.launchpad.net/neutron/+bug/1194807

openstack/common/db/sqlalchemy/session.py", line 283, in <module>
    deprecated_opts=[cfg.DeprecatedOpt('sql_connection',
AttributeError: 'module' object has no attribute 'DeprecatedOpt'

Seems that something was merged in quantum that depends on trunk unreleased oslo components.

Oslo version I've installed: oslo.config-1.1.1

TODO list

2013-06-26T19:35:00.001-07:00

- curl

- git

- GitHub

- MySQL

- lanuchpad

- openstack

- vmware

- hipervisor

- Python
- sed
- Ubuntu

- GAE, HeroKu ,Engine Yard (PaaS)
- Sina SAE: PaaS;
- Ali Cloud: IaaS;

- Salesforce / Workday;
- TechCrunch, Terremark;