The STE||AR Group

New Mailing Lists

Hartmut Kaiser — Tue, 09 Apr 2013 21:33:07 +0000

All of our HPX mailing lists will be migrated to new mailing addresses. These mailing lists are the heart of the HPX community. You may read the lists via full-content email, email digests, or via newsgroup reader.

The hosting for the mailing lists is donated by the Center of Computation and Technology at Louisiana State University.

We created three new mailing lists:

HPX Users Mailing List

This list is oriented toward casual users of the HPX libraries. It is a good place to start if you are having trouble getting started with HPX. Feel free to post both “newbie” and more challenging questions. This list is relatively low volume. Subscribe or unsubscribe at the HPX Users list home page. To post to the list send mail to
hpx-users@stellar.cct.lsu.edu. We will migrate all subscribers of the current GoPX mailing list to this new list.

Main HPX Mailing List

This is the main HPX mailing list. It has a higher volume, is very technical, and oriented toward HPX library developers. It is also read by many other members interested in watching the HPX library development process. Virtually all decisions, major or minor, technical or otherwise, are reached via public discussion on this mailing list. Subscribe or unsubscribe at the HPX list home page. To post to the list send mail to
hpx-devel@stellar.cct.lsu.edu.

HPX Github Commit Messages (read only)

The HPX Commits mailing list receives messages about commits to the HPX Github repository. All current subscribers of the px-commit mailing list will be migrated to this new list.

Rating: 5.0/5 (1 vote cast)

Developing HPX with Git: A Tutorial

Shuangyang Yang — Mon, 25 Mar 2013 20:18:29 +0000

EDIT: Heller pointed out that “newer subversion versions come with a single .svn directory in the top level directory; the single .git is not the reason why you can have multiple local branches”. And along with his advise to use Git over SSH instead of HTTPS and to deploy the public key with github.com.

EDIT2: revise the “feature change” section and improve workflow.

The HPX code base has moved from SVN to Git at github.com. This post is meant to serve as a tutorial to help with the transition.

SVN vs. Git – the Repo Conceptual Difference

From git-scm.com:

Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.

The distributed nature is reflected in the differences of repo models. As summarized by Ole Morten Amundsen, with a few edits:

Subversion: 2 levels:

local checkout – Files on your computer, in the file system ( svn checkout https://svn.xx.yy/projectxyz)
server, the central subversion repo – The server which every user checks in to (svn commit) and checks out from (svn up)

Git: 3+ levels:

local file system of a clone (git clone git@github.com:oma/projectxyz as checkout in svn). As above, yet different… Must be seen in context of the local index, below.
local index of a checkout (clone) in the same file system. This is similar to server above. You mess around with the local files as always (level 1), then you commit to the local index (git commit -a -m “msg”). The git repo server is still unchanged. Then make more changes and commit and so on. Any commits you make, are written to the index, stored in the directory .git, locally. Unlike subversion, which creates a .svn directory within each director, git has only one .git directory. This enables you to create multiple branches in the same directory in the file system! It deletes and adds files automatically when switching. OK, make as many commits you want locally. When satisfied, push them all to the server. Read on below.
server, the central git repo (or one of the many!). When you’re satisfied with your local changes and committed them locally (git commit) then you perform a git push which pushes the changes to the repo you cloned from). To get changes from this repo/server, you perform git pull.
the “4th” level is the git server repo of each developer, which your server need to pull from and vica versa.

Install Git and Get HPX

Install Git clients for your platform. github.com has Win and Mac clients. Linux should have Git package in the standard distributions, check here. After Git is installed, it is beneficial to setup the username and email, which will be marked on your commits.

$ git config --global user.name "Ste||ar" # set up username
$ git config --global user.name # verify username

$ git config --global user.email "stellar@cct.lsu.edu" 
  # set up email
$ git config --global user.email # verify email

To get HPX, run the following commands in your working directory.

$ git clone https://github.com/STEllAR-GROUP/hpx.git

$ git clone git@github.com:STEllAR-GROUP/hpx # over ssh instead of https
  # check github:help on generating SSH keys

Then a copy of the HPX code base will reside under the folder hpx/. Whenever someone has made commits on the server repo, the changes can be pulled to the local repo.

$ cd hpx/  # hpx/ contains the .git folder
$ git pull # update local repo

To check the status of the working directory, type

$ git status

Then it says 'On branch master'. It is the default branch name. Also the remote server repo has a name origin. In other literature, the remote server repo may be called upstream repo. In the following context, server repo is used most of the time.

Simple Changes: Single-Branch Workflow

If only simple changes are made and does not deserve a branch, these changes can be made to the working directory and committed to the local repo and then pushed to the server repo. As an example,

$ vi README.rst # did some changes
$ git status # will show modified file
$ git commit -a -m "changes on README.rst" # commit to local repo
$ git status # will show 'nothing to commit'

Rest assured, all things done so far only happen in the local repo. The server is not aware of any of these. Then finally after some more changes and commits, it is ready for submission.

$ git push # push to server repo

If nobody pushed new commits while you are editing, it will go smoothly providing the github.com username and password are correct. Otherwise, a big error shows and indicates that "Maybe you are not up-to-date and need to pull first?". Here comes the trick, it is preferable to user the following command

$ git pull --rebase  # instead of just 'git pull' for clean commits
$ git push  # should make it if no conflict occurs

To explain the reasons, first of all, Git is distributed and local repo contains all histories. The local repo can be seen as a branch of the server repo, and 'git pull' is about merging them. It will create on the local repo, a new commits indicating

Merge branch 'master' of https://github.com/STEllAR-GROUP/hpx,

which contains all the pushes made by others after your last pull. It is nothing wrong, totally standard for a merge. However, it is just a little bit clumsy on the server repo after the push. Because after the push, the 'merge' commit is actually empty since the server repo already has these changes before your push. It is still harmless, but annoying when many empty 'merge' commits show up.

On the other end, when using '--rebase', it is acting like these two–local and server–were one branch, like in SVN. Git does three things,

Un-commits all of your local commits that are not already in server repo.
Stuffs the new commits from the server repo onto the stack
Plays your changes back on top of them

You no longer merge–you actually commit on top of the server repo. This will result in a cleaner commit history on both local and server repos. This is also where the name comes from, single-branch workflow. This post has a graphical comparison of pull vs. rebase.

Feature Changes: Two-Branches Workflow

If adding a new feature or involving a complicate task, it is encouraged to create a branch and merge/rebase back to the master branch, the default name of the main branch. The name, two-branches workflow means to create a new branch asides the master branch. Git keeps the whole commit history locally, thus creating a branch is very cheap. Let’s create a branch ‘newfeature’ as an example.

$ git checkout -b newfeature  # new branch
Switched to a new branch "newfeature"

[.. changed some files, made a few commits ..]

During the development, if there are commits on master and need to be merged to the newfeature, the following command can be issued while on newfeature branch.

git rebase -i master
   # bring the branch up-to-date with master
   # check Interactive rebase for more information
   # will put branch changes on top of master

If conflicts are found while attempting to play back your changes, it throws you into an unnamed branch and gives you a chance to merge everything together. One article about resolving merge conflicts might help. After correcting the conflict(s) you would type

$ git rebase --continue

as directed by the conflict message.

Probably you want to share these changes with colleagues for code review. One way is to push the branch to the server repo.

$ git push -u origin newfeature 
  # push branch to the server repo
  # -u tells git that you want to be able to easily 
  # push and pull changes to that branch in the future

Afterwards, a colleague can get the branch with

$ git fetch origin 
  # update local repo with changes from server repo
$ git checkout --track origin/newfeature
  # creates a local branch named newfeature that 
  # matches the origin/newfeature branch 
  # and tells git that he wants to be able to 
  # easily push and pull from the branch

After the code review is done, it is time to rebase/merge into the master branch. Here rebase is preferred with the same reason mentioned in the previous section.

$ git rebase -i master
   # first bring the branch up-to-date with master

$ git checkout master 
Switched to branch "master"

$ git rebase -i newfeature 
   # interactively put newfeature commits on top of master

Thus the changes of newfeature have successfully merged to master, nice and clean.

Finally the newfeature branch can be thrown away and the commits pushed to server repo.

$ git branch -d newfeature # delete locally
$ git push origin :newfeature # delete on server repo, mind the ":"

$ git push # merge with server repo

This completes a two-branch workflow.

One thing to mention, although rebase keeps things tidy, it could be destructive and potentially dangerous if you don’t know what you are doing. Details can be found in the git-rebase manual page. Jarrod Spillers proposes a simple rule.

Don’t rebase branches you have shared with another developer.

It is always good to ask around if uncertain.

Acknowledgment

This post is not possible without the kind discussion with falcou@#ste||ar IRC channel. For more information, check the Pro Git book and other good sources. Happy Gitting.

Rating: 0.0/5 (0 votes cast)

A Scalable Backend for True MMORPGs – HPX at GTC

Thomas Heller — Tue, 19 Mar 2013 09:38:03 +0000

Currently the GPU Technology conference is ongoing. Together with the HPX Backend for LibGeoDecomp Andreas Schäfer submitted a poster about a scalable MMORPG design which eventually will use HPX to make it scale. The title of the poster is A Scalable Backend for True MMORPGs.

Also, don’t miss Andreas’ talk S3299 – From Notebooks to Supercomputers: Tap the Full Potential of Your CUDA Resources with LibGeoDecomp on thursday in room 211A form 16:00 to 16:25 if you happen to attend the conference.

Rating: 0.0/5 (0 votes cast)

Solving Combination Puzzles – An example HPX application – Part 1

Thomas Heller — Thu, 07 Feb 2013 19:19:10 +0000

HPX is great for developing applications that run both in a shared memory and distributed memory environment. This is accomplished by leveraging the Active Global Address Space (AGAS). By creating components in AGAS we gain the ability to seamlessly write parallel object oriented applications without the need to manually care about passing messages to different localities of explicitly creating threads. While this idea sounds great it is difficult to think about an implementation which achieves exactly that. As such this blog post is trying to walk you through the development of a recursive back tracking brute force solver for combination puzzles and you will discover that recursion allows us in general to exploit parallelism.

This is the first post in a series. This article series will walk you through the complete lifecycle of an HPX application. From the first basic idea, which is covered in this post to a full fledged HPX application exploiting the unique features of HPX to write programs with a unified semantic for local and possibly remote access to objects. The idea to develop such an application was given by Andreas Schäfer who challenged me to beat his MPI implementation. We’ll see how we fared in the last post of this series.

The basic idea and algorithm

Solving combination puzzles can be done a variety of ways. In order to demonstrate some functionalities of HPX we decided to implement a naive recursive back tracking algorithm which essentially does a depth search over all possible moves for a puzzle and terminates whenever a solution was found, or the maximum search depth of the recursion was reached and no solution was found. The basic algorithm therefore reads as follows:

template
bool solve(Puzzle const & p, std::size_t depth)
{
    if(p.is_solution()) return true;
    if(max_depth == depth) return false;

    std::vector next_moves = p.next_moves();
    for(Puzzle const & next : next_moves)
    {
        if(solve(next, depth+1))
        {
            return true;
        }
    }
    return false;
}

What p.next_moves() is supposed to do is probably best shown with a picture:

The next moves of an example 15 puzzle configuration.

The image shows the possible next moves for the 15 puzzle which is an example puzzle from the category of combination puzzles.

Recursion is parallelism!

So far so good. Sounds easy enough! But where is the parallelism you might ask. The answer is: Recursion is parallelism!
Let me prove that statement by presenting the following code:


template
bool solve(Puzzle const & p, std::size_t depth)
{
    if(p.is_solution()) return true;
    if(max_depth == depth) return false;

    std::vector next_moves = p.next_moves();

    std::vector solve_futures;
    solve_futures.reserve(next_moves.size());
    for(Puzzle const & next : next_moves)
    {
        bool (*solve_fun)(Puzzle const &, std::size_t) = solve;
        solve_futures.push_back(
            hpx::async(
                hpx::util::bind(
                    solve_fun
                  , next
                  , depth+1
                )
            )
        );
    }

    while(!solve_futures.empty())
    {
        hpx::util::tuple
            wait_result = hpx::wait_any(solve_futures);

        std::size_t idx = hpx::util::get(wait_result);
        std::size_t res = hpx::util::get(wait_result).get();

        solve_futures.erase(solve_futures.begin() + idx);
        if(res)
        {
            if(!solve_futures.empty())
            {
                hpx::wait(solve_futures);
            }
            return true;
        }
    }

    return false;
}

Alright. That was too much. The simple and basic recursion just grew almost three times in size! However, the gain is massive.
Let me walk you through the various parts that changed:

Line 13 to 22:
Instead of calling the function directly, we just issue an call to async. This will have the effect that a new thread invoking our recursion will get scheduled eventually. We push the result of the async call, which is a future, into a vector. A future is a object representing a value that will be computed eventually. A future can have various representations. In our case, the future is represented by a result of a asynchronously spawned function.
Line 25 to 42:
This is the new meat of our logic. Instead of getting the results back sequentially, we will get results of completed recursions in any order. As such we wait on any of the futures (Line 19). As a result we get a tuple containing the index to the future in our vector, and the actual future that was completed. We erase that future from our vector since we know it was completed (Line 24). If the result of this function was “true”, we can bail out, wait on any remaining function invocations and return true. Otherwise, if all futures returned with a result of “false”, this specific function invocation was not successful in finding a solution.

A full and working example demonstrating this solver can be found here. You can compile it by following the instructions given in the HPX documentation.

We made it! We turned the inherently sequential execution into a fully parallel version! This version has several problems. The first problem is that it only works on one single locality (locality is the HPX term for node). That is, no distributed computing yet. This will be covered in part two. However, we are already able to exploit the full power of multi-core CPUs.
Another problem is that this algorithm as is, searches every possible branch for a solution, which is not exactly what we want. We’d like to stop the recursion if any other thread has already found a solution. This will be discussed in the third part.

Rating: 0.0/5 (0 votes cast)

HPX V0.9.5 Released

Hartmut Kaiser — Thu, 17 Jan 2013 01:09:52 +0000

The STE||AR Group is proud to announce the fifth formal release of HPX (V0.9.5). This release has been made possible by the hard, dedicated, and diligent work of everybody involved. This is the actual release we provide after the release candidate we published last December. Please report any issues you encounter through our issue tracker. HPX (High Performance ParalleX) is a general C++ runtime system for parallel and distributed applications of any scale. It is the first freely available, open source, feature-complete, modular, and performance oriented implementation of the ParalleX execution model. HPX is targeted at conventional architectures and, currently, Linux based systems, such as SMP nodes and conventional clusters. We invite everybody interested to participate in the discussions and the development of this rapidly evolving project.

Why is HPX is so special?

HPX exposes an uniform, standards-oriented API for ease of programming parallel and distributed applications.
It enables programmers to write fully asynchronous code using hundreds of millions of threads.
HPX provides unified syntax and semantics for local and remote operations.
HPX makes concurrency manageable with dataflow and future based synchronization.
It implements a rich set of runtime services supporting a broad range of use cases.
It is designed to solve problems conventionally considered to be scaling-impaired.
HPX has been designed and developed for systems of any scale, from hand-held devices to very large scale systems.

We are very pleased with this release as it is continuing along the lines of code and API consolidation, and overall usability improvements. We dedicated much attention to performance and we were able to significantly speed up the threading and networking subsystems. Please see here for a list of changes and new features.

You can download the release files from the downloads page or directly from Github. The release notes can be found here. Please feel free to try the examples and let us know what you think. The best way to get in contact with us is to leave a comment on this page or to send a mail to gopx@cct.lsu.edu.

Rating: 5.0/5 (3 votes cast)

Clang and OSX support

Thomas Heller — Mon, 07 Jan 2013 07:03:32 +0000

Thanks to Pyry Jahkola (pyrtsa) we start the new year with an exciting new platform support. He submitted a bunch of patches (see his pull request) that allows us to build HPX on OSX with clang! So far he was able to build with a vanilla version of clang. The tests and examples run just fine.

This pull request highlights once again that we can keep our promise to deliver wide platform support. As with the Android support reported last year (HPX on Android), the changes to the core of HPX are very minimal and mostly address some deficiencies of the platforms compiler.
Additionally, the support of Clang outlines our commitment to standard C++ and enables a new variety of different static code analysis to further improve our code base. Once again, thanks pyrtsa!

P.S.: A happy and successful year to everyone!

Rating: 5.0/5 (2 votes cast)

HPX at MeetingC++ 2012

Thomas Heller — Mon, 31 Dec 2012 16:11:44 +0000

Even though the Meeting C++ 2012 Conference happened almost two months ago now. I would still like to point out to this great conference. It was a great success, not only for HPX, but also the conference made a great debut. HPX was represented with one talk!

This talk will demonstrate the novel HPX runtime system and will talk about how current and future versions of C++ will benefit from the new ways of parallelism exposed by these approaches.

The HPX runtime system is a modular, feature-complete, and performance-oriented representation of the ParalleX execution model targeted at conventional parallel computing architectures such as SMP nodes and commodity clusters. As an alternative to MPI, HPX incorporates routines to manage lightweight user-threads in addition to providing an Active Global Address Space (AGAS). HPX is implemented in C++11 and utilizes over 20 Boost and candidate Boost libraries.

Links to videos and slides are available here

The audio is not the best quality. Enjoy!

Rating: 0.0/5 (0 votes cast)

HPX V0.9.5 (RC1) Released

Hartmut Kaiser — Mon, 17 Dec 2012 19:26:01 +0000

We are proud to announce the fifth formal release of HPX (V0.9.5). This release has been made possible by the hard, dedicated, and diligent work of everybody involved. This is a release candidate (RC1) for the actual release which is planned for mid January 2013. Please report any issues you encounter through our issue tracker.

HPX (High Performance ParalleX) is a general C++ runtime system for parallel and distributed applications of any scale. It is the first freely available, open source, feature-complete, modular, and performance oriented implementation of the ParalleX execution model. HPX is targeted at conventional architectures and, currently, Linux based systems, such as SMP nodes and conventional clusters. We invite everybody interested to participate in the discussions and the development of this rapidly evolving project.

Why is HPX is so special?

HPX exposes an uniform, standards-oriented API for ease of programming parallel and distributed applications.
It enables programmers to write fully asynchronous code using hundreds of millions of threads.
HPX provides unified syntax and semantics for local and remote operations.
HPX makes concurrency manageable with dataflow and future based synchronization.
It implements a rich set of runtime services supporting a broad range of use cases.
It is designed to solve problems conventionally considered to be scaling-impaired.
HPX has been designed and developed for systems of any scale, from hand-held devices to very large scale systems.

You can download the release files from the downloads page. The release notes can be found here. Please feel free to try the examples and let us know what you think. The best way to get in contact with us is to leave a comment on this page or to send a mail to gopx@cct.lsu.edu.

Rating: 4.7/5 (3 votes cast)

SC12 Demo: Highly Parallel Interactive Image Flow

Thomas Heller — Sun, 18 Nov 2012 16:29:50 +0000

The Highly Parallel Interactive Image Flow Application (HPIIF) is an interactive visualization built with HPX and LibGeoDecomp in close collaboration with the Chair for Computer Architecture at the Friedrich-Alexander University in Erlangen (Germany).

Based on an N-Body algorithm, we show a visualization of particles interacting in an otherwise static image. The user is able to interact with the image through a high-definition camera and an Android-based tablet. In addition to the visualization, one can introspect the different performance characteristics through HPX’s performance counter framework. LibGeoDecomp is a generic library for stencil-based simulations. Users program against a common API, while the library leverages a number of optimized components to parallelize the simulation and adapt it to the system. Together with the new HPX backend, this demo showcases automatic load-balancing and dynamic removal/addition of hardware resources.

Rating: 4.7/5 (3 votes cast)

HPX on Android!

Thomas Heller — Fri, 12 Oct 2012 13:21:13 +0000

It’s been a while since the last post on this blog representing new development in the HPX world. We are not dead and are still working on providing a new and unique experience to the world of parallel computing! One of our latest efforts was to bring HPX to Android devices. Android is an exciting platform which grew very popular recently.

With this post I am proud to announce the first “Hello World!”-like Android application running HPX:

HPX on Android – Hello World

When pressing the button, 10 HPX threads will be spawned. Additionally, 4 performance counter showing the current value of HPX threads residing in the system. The application was compiled in debug mode, as such, the timings shown in the screenshot are definitely not all you can expect from HPX on ARM.

You can download the APK here. (It has been tested on a Samsung Galaxy Note 10.1 tablet running Android version 4.0.4) For more information and the source code visit the HPXAndroid github repository. Please bear with use, the port is very new, and unfortunately we have to smooth out some things in order to guarantee an optimal experience for Android users. The HPX<->Android Java Bridge is also kinda experimental and lacks lots of features. If you want to be part of this exciting new development visit us on the #ste||ar IRC channels at freenode or drop us a comment here.

Rating: 4.7/5 (3 votes cast)