Apolemia

My wrap-up after a NLP machine learning competition

2018-03-24T15:51:00.001+00:00

I recently participated in a natural language processing (NLP) machine learning competition, and there were some interesting learning’s that I think are worth sharing. NLP is a bit different from my usual topics but I think machine learning is the most interesting thing currently happening in supply chain software and working with text is an important part of machine learning.

The competition was Jigsaw Toxic Comment, with over 4500 teams competing and my team got a great 5th place. The goal was to build a classification system able to classify different comments with given types "toxic/insult/obscene/etc" or as clean.

The best results were 0.9885 AUC which is impressive, it is another task were the machine is at the same level as humans.

To achieve these results all top teams used deep learning, the best models were almost always recurrent neural networks (LSTM or GRU) using pre-trained word vectors. In case you don't know, word vectors are mappings between words and large dimensional vectors (300 dimensions is common), where these vectors result from training in very large collections of text (like wikipedia). In the word vector space, words with similar meaning have close vectors and to some extend the distance between words in the space relate to the concepts. One typical example is that if we take the vector for the work "King" and we add the vector for "Woman" we get a vector that is the closest to the word "Queen". This last example is known to be a bit cherry-picked and word vectors are not yet so perfect but it is for sure one of the best tools in NLP. And there are many good vectors to choose from, so as always in machine learning, combining all of them leads to a better result.

One interesting fact from the data is that even though some word vectors have more then 2 million words, we could find in the comments texts a very large number of words missing in the vectors (around 30%). It is because of bad spelling, using foreign language words and in many cases it is on purpose that people write "heeeyyyy", "d0n't" or "bs'ing". One thing that helps for these cases are subword embeddings like Fasttext that do training in parts of words and based on that can build vectors for unknown words by using the smaller pieces.

Since the comments are full of misspelled words and non-letter characters it could be thought that doing a lot of pre-processing to clean the text and fix the misspellings would improve the results. This was not the case in my experience and other teams also reported the same conclusion. Combination of subword embeddings and the ability of neural networks to internally learn the necessary filtering seem to be better.

Deep learning systems are hungry for data, and if we give them more data we will probably get better results. An interesting trick I learned was that we could easily use the translation technology to get some small variations of the texts. For example using the TextBlob Python package it is as easy as doing

new_text = TextBlob(text).translate(to="de").translate(to="en")

And we get a new text that resulted from translating the original English text to German, and then translating it back to English (using Google webservice). In few cases the result is exactly the same but in most cases there is some small change, different words but hopefully the same meaning.

Data generated this way can be used for additional training of models or can be used for what is called test time data augmentation. This last concept is quite simple to explain, it is often used in image classification where rotating an image sometimes makes it easier for the model to make better predictions. So doing several rotations and then averaging predictions will lead to a better result. With text it also worked quite well by using the different translations variants.

From my experience the language that worked best for this trick was German, then next one was Spanish. I also tried a few other European languages like Portuguese, French or Swedish and also something much different as Japanese.

I did not try it but other teams reported that just doing translations and training with the word vectors for the other language also improved the results.

It was great fun this competition. My team was absolutely amazing.

Testing SAP in the age of the robot

2017-09-13T12:51:00.004+01:00

Like everywhere else in software industry, in SAP testing is the key factor to have things working in a stable and robust way. The main difference is that while some evolve their systems based on a large set of automated tests, in SAP projects it is common to depend on people performing tests.

We can try to dig some historical and technical reasons for doing things manually in SAP, but that is just looking for excuses. Reality is that we can and we should build testing robots for SAP.

I believe the best way to explain how this can be so effective is to show some examples. I will start with a master data test example.

    def test_atp_group(self):
        wrong = marc[(marc. != 'X1')&(marc.matnr.isin(mara[mara.mtart=='FERT'].matnr))]

So what is it so special here. First the code is very short, it takes advantage of python data science tools to write very compact code. Being able to write tests in very few lines allows not only to create large number of tests but also to be easy to rewrite tests as many times as needed. Most of my tests are 1 to 5 lines of code and that is important. With a big screen I can see a few hundred tests without much need to scrolling and that makes it much easier to go back from test failures to the testing rules. Also it does not feel bad to delete a line of code when a rule changes, while deleting 100 lines would feel a bit depressing.

And why testing master data is good? My experience shows that master data tests have the best return of investment. Tests take little time to write (5 or 10 minutes) and will run dozens of times and will find hundreds (or thousands) of errors. Same tests will run in development, regression, pre-prod and production systems. Most projects will create master data in waves, so tests will also run for multiple waves. And sometimes we need to ask others more than once to get something properly fixed. All this multiplied makes many checks, many errors found and a lot of saved time. During a project making master data tests early is like investing in bitcoins when it started, we get a huge return on the time invested.

In the code above it shows mara and marc variables, you may wonder how it gets there. These two database tables for material master and stored as a special data structure called data frames. To get these data frames from SAP data I use an API inspired on Django, that we built at Cognitiva, that works like this:

    marc = ReadTable(con).table('marc').filter('werks',['P001','P002']).all()
    mara = ReadTable(con).table('mara').filter('matnr',marc.matnr.unique()).all()

Again it takes only one line of code for each table, so quite easy to get data from 50 tables or more.

After master data I think the next most useful test robots (larger ROI) are the ones that test large calculation that depend on complex master data setups. One such example is delivery scheduling that dependes on multiple condition tables and calendars. Another is pricing calculation when there are multiple and complex discounts schemes. Whenever these complex tasks can be isolated, it is easier to build a specific test robot than to try to test as part as end to end testing. This is how it looks like a test on delivery scheduling, where the call to schedule is just wrapper call on BAPI_APO_SCHEDULING. It is also very quick to write tests so it is easy to have an extensive coverage.

    def test_std_10_03(self):
        """standard request before cut-off on Wednesday
           GI Wednesday at 18h, delivery Friday at 18h
        """
        items = [('ZDMODE','STD'),('LOCFR','7309'),
                 ('ALAND','DE'),('ZCARRIER','0100530000')]
        times = self.schedule(20140423100000, items)
        self.assertSchedEqual(times['WADAT'], 20140423180000)
        self.assertSchedEqual(times['LFDAT'], 20140425180000)

And finally end-to-end (E2E) test robots are also quite useful. Some years ago E2E testing was mainly about testing the GUI that users would use to perform the actions on the system. But nowadays what we see in SAP is that large usage of the system happens through interfaces (EDI, ecommerce, point of sales, external warehouse systems, etc) so automated testing the interfaces and batch jobs exactly as in production and replacing the GUI with some equivalent RFC calls is a good strategy. An example of this would be replacing the action the user would take in VA02 to remove the delivery block of a sales order with a call to the sales order BAPI to do the same change in delivery block. The last option is technically simpler to automate and good enough to catch errors.

To build E2E test robots some time is spent to create building blocks, like a DESADV IDOC for external warehouse, or a TLB queue for orders created from APO transport load builder. But after having these building blocks writing E2E tests is also quick, a example test looks like:

    def vmi_sales(self, plant, mats, country, routedays=2, channel='04'):
        transport_size = self.data.transport_size(country)
        sold_to, ship_to = self.data.get_customer(country)
        items = self.data.get_items(mats, transport_size)
        mdoc = self.load_stock(items, plant)
        self.check_material_mov(mdoc, plant, country)
        order_id = self.create_tlb(sold_to, plant, items, ship_to=ship_to, 
                   routedays=routedays,channel=channel)
        time.sleep(15) # wait for CIF processing
        self.check_sales(plant, country, order_id)
        dlv = self.create_delivery(order_id)
        self.check_delivery(dlv, plant, country)
        self.delivery_pgi(dlv, plant)
        return {'order_id':order_id, 'dlv':dlv, 'mdoc':mdoc, 'plant':plant, 
                'country':country}

This test would load stock, create a VMI sales through the CIF queue, check sales correctness, create and check the delivery and PGI the delivery by creating an external WMS IDOC. Then finally invoice is created by a batch job. The extension to more complex E2E with transfers between plants and multiple invoices is just a few more lines of code. Then for each E2E writing variants can be just one line of code.

    def vmi_sweden(self):
        return self.vmi_sales('P001',[123,321,112,221],'SE',routedays=14)

Because it is so quick to build variants, robots can test multiple plants, combinations of transfers between sequence of plants and final sales, multiple types of materials, partial deliveries, batch splits, etc.

Although the time investment in E2E test robots is higher, in my opinion it is well worth. Running hundreds or thousands of E2E tests is the best way to be sure we have a robust system or to know which areas need improvement.

Robots are good. We need more robot testing in SAP projects.

Using Machine Learning in a SAP project

2017-06-03T12:43:00.001+01:00

SAP projects can be a very complex task of data mapping and data cleaning. People in data science spend most of their time doing this task called data wrangling. There is no doubt a lot of effort is needed at this stage. Most times in SAP projects this task is done manually using extractions and Excel files in a process that is very likely to be poluted by human errors. As data size and complexity increases final data quality decreases. This happens always, no matter how strong people are with Excel.

I think there is a lot from data science that can be very effective in SAP projects. Not only the tools for building the data wrangling tasks but also the machine learning tools that can be used to extract knowledge from data.

In a recent project the product pricing master data consisted of about 50 thousand entries. This is still a small dataset, but already too large for the user to validate line by line.

With machine learning it was possible to identify the errors in the data using algorithms for outlier detection. These methods work quite well when there is a lot of data. In this pricing data there were many very similar prices for the same material and different customers, and many similar prices for products from same product group. This allows the algorithm to identify areas in space with high density of entries and outliers are elements outside those areas.

The picture below illustrates what the algorithm does, the areas with high density of points are what the algorithm sees as normal values and everything else outside are the outliers.

In this specific case machine learning unconvered a large number of outliers and after analysis it was possible to identify several reasons:

prices can be maintained with quantity factor (eg. it can be a price for 1 unit, or a price for 100 units); wrong quantity factors originated prices that were order of magnitude different
prices can be maintained in different currencies (which were all converted to a base currency before using the algorithm) and there were cases where the price was calculated in a currency but then maintained in a different currency
there were software bugs in reading the upload data format

Using machine learning allowed to quickly extract the few hundred cases of errors from the dataset and this simplified the correction activity. Because it is such a generic tool it can be used with any data to find entries that are outliers. If after inspection the outliers are correct entries, then we can have more confidence on the data quality.

Master data quality is a big problem. Machine learning will not magically solve all master data issues, but it is a strong tool to help on that.

Using deep learning models on Nespresso images

2016-08-20T19:39:00.003+01:00

One great resource in machine learning is pre-trained neural networks for image processing. While training a deep network is complex and needs large amounts of data, using pre-trained models is as easy as using functions from a software library.

Just for some fun I picked a few images from the Nespresso webshop and used the VGG19 pre-trained network with the goal of finding a way to sort the images by similarity. This just needs two steps, the first is to get the network layer outputs for each image.

    base_model = VGG19(include_top=True, weights='imagenet')
    model = Model(input=base_model.input,
            output=base_model.get_layer('block5_pool').output)
    img_features = np.vstack([model.predict(img).flatten() for img in imglst])

The network returns a vector for each image, then using PCA the vector is reduced to a single dimension, which is then the sort order.

    pca = PCA(n_components=1)
    img_score = pca.fit_transform(img_features)

The result is the following sort order for the images (sorted from top to bottom, left to right).

Not perfect but very close, don't you think?

SAPYard

2016-01-19T11:18:00.000+00:00

I recently published a post in SAPYard about the SAP Debugging course that I created for Academy OnSap. The SAPYard is a great site, full a useful information for SAP consultants and users. I am very grateful for their help spreading the word about the SAP debugging course and I also want to highlight they have great content.

For example, they have a series of tutorials about Web Dynpro, a topic that nowadays is fundamental for programmers. Still on ABAP, you will also find interesting this more advanced post about delete on internal tables. And they also have a lot of content and expertise on SAP HANA (eg, HANA Views).

There are amazing SAP resources outside the walls of the SAP company. Official SAP portals are good and important but in my opinion the quality of the independent community is what makes the SAP ecosystem so great.

An online course on SAP debugging

2015-06-15T23:05:00.000+01:00

I think debugging is one of the most important skills to do advanced work on SAP. At least from my experience, when things get hard it is either the debugger or data analysis that will help find the solution. As far as I know there are not many resources to learn and improve the practice of debugging. So, as an attempt to improve this, I worked on assembling a mini course on SAP Debugging, following the patterns of the modern MOOCs.

I think it turned out quite OK and it was fun, probably I will be doing more of these. You can find the debugging course here.

The course is for beginners, but it also includes some advanced topics that can be interesting to experienced people. And it has some assignments that should not take much time and (hopefully) will be fun.

Looking forward to see you in the class!

SAP Unit Tests actually exist, report of a sighting

2015-05-18T18:01:00.000+01:00

One of most impressive things in SAP, and impressive in a bad way, is the huge effort of people executing tests to validate the software after upgrades or some larger developments. It should not be like this. We now look back and we think it is funny a room full of people performing calculations like in this picture from the 40s.

But almost 100 years later this is more or less how a SAP upgrade is tested.

The reason is historic. The "old" SAP code is large, monolithic and highly integrated (a nicer way to describe a crazy web of dependencies). Not surprisingly this older code comes without automated tests, these would be hard to implement in such architecture. But not all SAP is coded the same way. Components developed in the last 10-15 years follow the typical object-oriented modular software best practices. And SAP also includes a complete framework for unit tests. My expectation was that SAP would start shipping more and more unit tests, and this would help reduce the need of manual testing and reduce the risk of regressions when installing SAP updates.

So for many years I asked myself: Where are the SAP unit tests???

Many times I tried the option to see the unit tests in SAP objects, just to find out ... there was none.

Anyone remembers something similar? This documentation option in functions. At least the experience is consistent, it almost never shows any documentation.

It actually shows a message to make you feel there could be something in other language. But, of course, there was nothing.

But now I finally saw some unit tests in SAP code. It was in /SAPAPO/CL_ALL_LOCK_DELTA class (in SAP SCM software).

So unit tests exists, it is worth searching for them. Looking at the unit tests is also a good way to understand how to use that part of the code.

And now I have some SAP example to show when trying to bring unit testing best practices to the custom developments world. SAP recommends using unit tests. You know those words can take you a long way.

SAP S/4

2015-02-08T22:40:00.000+00:00

Last week SAP released S/4, the evolution of their most important product the R/3.

I think this is a good occasion to talk about what makes SAP a successful software. Does S/4 have the magic ingredient that made R/3 so successful? So why was R/3 a success?

When I started working with R/3 I was very impressed with the development environment, the tools in SAPGUI to develop the code running in the server, and how easy it was to start debugging a running program. At that time I thought SAP was great because it was based on this flexible development platform.

But of course some time later I realized it was not the platform. The ABAP language is a pain and lacks in many areas, for example it is quite bad for numerical algorithms. Concurrency is not very efficient, the web layer is slow and the many web frameworks are obsolete even before having some real adoption. Forget it, it was not the platform.

What is really cool in SAP is the applications, having so many applications, for so many areas, industries and processes. But usability is bad, performance also not so good, and there are so many custom applications in SAP projects that something must be wrong with what comes out-of-the-box.

The applications helped a lot, but something else was very important in R/3, the expensive price tag. I'm not joking here, a very expensive price is a powerful feature. When a software costs 1k either it works or it will be scrapped, no client will pay 10k or more in consulting time to adjust and integrate the software. But if the software costs 1000k then 100k in consulting is very reasonable. And if out-of-the-box things don't fit well, after 100k of consulting a lot can be done to make the client happy.

Will S/4 will have the R/3 magic? Yes I think we can trust it will be quite expensive ;-)

SAP code base

2014-10-06T22:37:00.001+01:00

Hasso Plattner, founder of SAP, is also a teacher in an open online course about in-memory databases. In one of the lessons he mentions experiments done by SAP, that by moving some logic to database stored procedures (math logic as he calls it) it was possible to reduce as much as 80% of the code size in the application server. This is of course one more chapter in the SAP HANA is great story. But I cannot help but comment this from the other point of view. SAP was able to become world leader in enterprise software with up to 80% bloated code base. One more case where "good enough" was the best decision.

More values in SAP domains

2013-12-06T22:07:00.001+00:00

Some many years working with the SAP server and it was just recently that I learned it was possible to add more values to standard domains that have a fixed list of values. It is not a big secret, it is just an option in the menu and then it is like append structures in tables.

This was useful to add the missing APO store location 1040 to the CRM domain used in the configuration of location/business partner mapping.

Nice SAPUI5 quick demo (for someone that knows js/html5)

2013-12-04T22:42:00.001+00:00

I found this demo from DJ Adams on SAPUI5 very useful. I know some HTML5 but I have yet to dig into the details of SAPUI5 and with this video it was possible for me to get some feeling of this framework. The first impression is not the best. The XML views, the long namespaces and many small files looks a bit like Java coded in JS.

I still have the feeling that nobody has yet found "the simple way" of doing UI in the browser. Probably it still needs one more iteration of browser core technologies and maybe ideas like polymer-project.org will do it. In any case SAPUI5 is by far the one of the most interesting SAP technologies.

Just fill in the blanks

2013-07-18T22:09:00.001+01:00

Reading the SAP source code is always full of surprises. In this case it was in the SCM system, after finding some useful functionality that was not documented, some additional search on how to maintain the values leads to the following procedure.

It feels like the school books, one just needs to fill in the blanks :-)

Being gamified by "Gamification At Work"

2013-06-12T22:15:00.001+01:00

I have received a pre-release copy of "Gamification At Work". It is not the first time I am contacted to make a blog post about some new book. And so far this was the first time I actually bothered to do something. The marketing was well done, and I actually liked the trick of receiving the mail twice, the first time with some formatting problems and the second time with an apology and without formatting problems. True, it was very likely that I would delete the first email without reading (actually it was not the case). And receiving a link to the online book with apolemia in the background was also a good trick.

So I was gamified, but that is good. I like to be gamified and somehow I do it to myself every day (task lists with colors, targets in the number of commits, etc). If some number or visual thing helps me do more, then it is good.

Anyhow that is was not main the reason I started reading the book and wrote this blog post. The reason was that one of the authors is Mario Herger and I remember some of his work on the SAP community. For example this video (below) or this.

OK those videos are nothing special in the context of the typical startup, but the SAP world 10 years ago was not like that. The SAP world was boring. It was efficient, it was rewarding, it was full of vapourware and it was completely boring. Nowadays it is still rewarding, efficient and full of vapourware but not so boring anymore. And some nonsense like those videos was quite important for that change.

So my expectations of the book are quite high. And I hope there will be a movie release ;-)

PS. No need search the torrents, the book is freely available here.

Interview with Ole Andre Haugen and Njål Stabell

2012-06-12T19:56:00.000+01:00

I recently got to know an interesting product for mobile SAP development from Neptune Software, a Norwegian software shop. Two things caught my attention, they are a startup in the SAP ecosystem, and they are building SAP HTML5 designer, an ABAP development framework for mobile devices. I am quite interested in both topics so I asked Ole-Andre, the CEO of Neptune Software, if he would be interested in a short interview for the blog and he kindly accepted.

[OAH] Ole Andre Haugen – CEO Neptune Software
[NST] Njål Stabell – CTO Neptune Software

Can you tell us more about you and how you started with SAP?

[OAH] I started with SAP back in 1995 after serving as an officer in the Norwegian armed forces as Instructor and Field Engineer. I have since worked with SAP either as consultant or manager in consultancy firms.
[NST] Actually I started out as a Java developer in the late 90’s and was lured into the SAP world through integration projects and java development on the SAP Internet Sales solution. It took me some years to fully embrace and accept the ABAP language but I now cherish its performance, simplicity and suitability for enterprise business logic.

Can you tell us about your company products and goals?

[OAH] Neptune Software is the result of years and years of real-life SAP projects and experience and not a lab product. We have tried to make our product the optimal development tool that we ourselves would have wanted to use as project developers. The result is a tool that combines the best of the features of the SAP Developer Workbench with the functionality we have always desired. As an example we have reused the idea of data binding from Webdynpro and added far better language support in our solution.

[NST] I would like to add that the trigger of this specific product was the emergence of HTML5 as the “code once run anywhere” de facto standard for mobile development. We could not have delivered the kind of quality we do without the open source development communities producing great frameworks like jQuery Mobile and Phonegap(Cordova).

[OAH] So, our product is an SAP certified HTML5 designer within the SAP Netweaver platform. We provide a server, a “drag & drop” designer and tools for creating Apps for multiple devises based on existing SAP functionality. We have no ambitions to move out of this space - meaning we are not a MEAP but a MADP (Mobile development Application Platform)

Is this your first venture in product development?

[NST] No, I started my career as a product developer for a Scandinavian e-procurement company called Proceedo. The move into project based development was quite a shock when I realized that object orientation and inheritance was kind of useless when you code a user-exit or similar “small” fixes within the SAP environment. Luckily I can capitalize on my former experience now that we have created a huge application in SAP and where ABAP Object really shines.

[OAH] I started a company that delivered a procurement HUB between SAP installations (Customers) in 2002 called LogNet. Even if I have started a lot of SAP consultancy firms, it has always been my passion to create software products rather than just delivering manpower to SAP customers.

I suppose it is not easy for you to market a solution outside the official SAP strategy. How do convince customers that is better to use a non-SAP solution to solve the mobile equation?

[OAH] Not being owned by SAP doesn’t mean we are not an SAP solution. We are actually the only ABAP SAP development product for mobility on the market. And I would like to stress that our solution is certified by SAP. In addition we do not provide a MEAP that is competing with the SAP Mobility platform but only a means for ABAP developers to capitalize on their existing skill sets moving into a more and more mobile reality.

[NST] The new partnerships announced by SAP last month also shows that SAP wants more development options for their platform. Especially the Adobe Phonegap partnering was important for us. Now our customers can use applications built with our tool on the SAP Mobile Platform and benefit from the MDM, Security and functionality like push notifications.

Is SAP platform is a good foundation for building a company that develops applications? What major strengths and weaknesses did you find?

[OAH] SAP has a great platform when it comes to performance. We are especially impressed with the ICF (Internet Communication Framework) for performance and scalability. We also utilize the standard transport system and security of the SAP system.

[NST] We are of course limited to the SAP customers in the world, and as we use the ABAP stack the look and feel of the designer is based on the standard SAP GUI. We find that the familiarity between our product and standard SAP ABAP tools really helps ABAP developers getting started.

Do you code ABAP?

[BOTH] Yes, a lot.

Have you tried SAPUI5? What do you think of it? Do you think it will be the building block for all new SAP user interfaces?

[NST] We actually imported the SAPUI5 framework into our solution and made a few SAPUI5 plugins to test how it would play along our solution. It worked perfectly and we are now in touch with the managers of SAPUI5 in Waldorf and are awaiting the licensing decisions before we move any further.

[OAH] For desktop solutions it will definitely have a more appealing and modern look than Webdynpro. I believe that when the browsers in the enterprise have decent HTML5 support – HTML5 will be the preferred choice for most UI development.

I can see you choose JQuery Mobile to use in your product. I also see that SAP uses JQuery Mobile in several places, at least in the SUP and in Portal on Devices (do you know more?). Do think SAP will also adopt JQuery Mobile more officially or it will move to SAPUI5 once this toolkit supports mobile?

[NST] SAPUI5 actually packages jQuery Mobile for mobility development. SAP made a partnering announcement with Sencha Touch so we’ll see what that means for jQuery Mobile in the SAP context in the coming year. For our solution, the jQuery Mobile framework is a better fit. It is more HTML oriented and less Java script dependent when it comes to development.

[OAH] I believe SAPUI5 will have most impact on the desktop. SAP really needs a facelift for their portal and business client.

Are there some new SAP technologies do you think can be particularly interesting to independent developers?

[NST] Apart from the SAPUI5 the new Netweaver Neo is interesting. If we have had a push for mobility and HANA this year, my predictions is that Cloud computing will be the hot thing next year.

Can you tell us more about your marketing channels? Do you get many inquiries from the website?

[OAH] We get great help from SAP through the Eco Hub and they help with setting up webinars with SAP partners. Our trial version strategy has been a success and we have our solution installed on more than 200 SAP systems worldwide.

[NST] Also channels like Linkedin are great for marketing. These days it is a lot easier to find the right audience than it was only a few years ago.

Have you participated in SAP events? Which kind of events you found better to show your products?

[OAH] We participated on Teched and Sapphire in Madrid last year and will attend Teched in Las Vegas in October. Other than that we have been to the local SAP user group events in Scandinavia which has resulted in actual contracts.

How do you see the SAP independent developers space?

[NST] For ABAP developers we can highly recommend the Assembly kit that comes with the certification. It gives you the tool you need for packaging and delivering ABAP add-ons in a smooth and professional manner.

[OAH] SAP provides their Software and Technology partners with both the tools and the marketing assistance you will need starting up. So if you
have a great idea and some means of funding I say go for it.

Thanks guys for this interview. I think you are one of the coolest startups in the SAP space and I hope you have a lot of success with your product.

Simple PhoneGap Logon App for SAP mobile applications

2012-04-04T19:24:00.002+01:00

This post is to share a simple PhoneGap application that manages the logon before calling a Netweaver URL with basic authentication. This app can be used as a container to run any HTML5 application from Netweaver and it is also open-source, so it can be useful for other people working on the SAP mobility topic (direct link to the code).

About PhoneGap

PhoneGap is toolkit to build containers for running HTML5 mobile applications. With PhoneGap it is possible to access hardware APIs like GPS, accelerometer, camera, etc, and still code in cross-platform HTML5/Javascript. For example, PhoneGap has a very useful barcode reader plugin, and reading barcodes from standard browser based HTML5 apps is not possible.

About the problem this app tries to solve

The typical pattern for development of PhoneGap applications is to package the application with the HTML and Javascript code and when the application needs data from a server it requests that data using ajax. This pattern has some advantages, the files are in the mobile device so the app loads faster and needs less network transfers.
But I don't think this is a good pattern for enterprise applications. With this pattern deployment is hard, because when code changes it needs to be synchronized to all devices. And because we cannot be sure that all devices get updated at the same time, the data APIs from the server needs to support the different versions of the applications that might be running. This is complex to maintain. Enterprise mobile solutions must be easy to change and deploy and that can be achieved by having all HTML5/Javascript code in the Netweaver server. For that to work the PhoneGap application needs to be enhanced to have configuration data for the login, since currently it does not have a login dialog like the one found in the mobile browsers. So this Logon App provides a way to connect PhoneGap directly to Netweaver using basic authentication.

The Logon App

The application is ultra simple. Starts with a username/password screen (screenshot left). In the menu there is a settings screen where the server connection details and the URL path to the starter application must be saved (screenshot middle). When the login button is pressed the app calls the Netweaver URL using the Basic Authentication credentials and displays the remotely loaded starter application (screenshot right). After the login all UI and logic is in the server side.

Source code and instructions

Code is available in this repository with some minimal build instructions. A binary apk is also available in the downloads section. Currently it is only for Android (doing for other PhoneGap platforms should be easy, at least an iOS version will be done sooner or later).

Note: this blog entry was originally published in SCN

SAP development with jQuery Mobile

2012-02-02T16:18:00.003+00:00

One of the hot topics for me last year was doing SAP development for mobile platforms. There is a huge quantity and quality of enterprise information inside SAP servers and the new wave of mobile devices opens so many new ways of interacting with that data and business processes. I did not write much on that topic in the blog but on onsap you can find many questions and answers related to the technologies I was using.

There are many platforms and toolkits for mobile development (and that is a good topic for another blog post that I hope to write one day). My choice of platform is jQuery Mobile for the frontend development and business server pages (BSP) for the backend development. With this stack it is possible to develop solutions using the platforms companies already have (ECC and other solutions based on Netweaver) and using an open toolkit that is evolving quickly and at the same time is stable and tested in a large number of devices.

As a result of some of that work we, at Cognitiva, are releasing an open-source package with some utilities to help development of jQuery Mobile on SAP. It includes sample applications like the one show in action in the video below, and it also includes a search help widget that is described in detail in this SDN blog post.

If you have interest in mobile apps give this technology a try. The source code is available in this repository and it is possible to run a demo application with the free Netweaver developer edition.

Interview with Tamás Holics

2012-01-16T17:47:00.007+00:00

Tamás Holics is manager at Hovitaga, a company in Budapest specialized in SAP development that is also bringing to market some SAP add-ons and applications. I asked Tamás if he could share his experience in a short interview, as I think this kind of experience is interesting for many of us running consulting companies or working as freelancers and also trying to market SAP applications or enhancements.

[PL] Hi Tamás, can you tell us more about you, how you started with SAP and what areas of SAP you currently focus?

[TH] I joined SAP Labs Budapest in 2006 as a junior ABAP developer. There I was working on IS-Oil projects for about one and a half years. Since then I've gained experience in the insurance industry, especially with the FS-CD module. Currently I work for a manufacturing company where I do ABAP development within core ERP modules like warehouse management, logistics, sales and distribution etc.

[PL] In parallel with consulting your company is developing SAP applications. What fraction of the time do you invest in app development versus consulting work?

[TH] That changes very often. There were some months when I could spend 10-12 days on the development of my apps, but lately only a few hours per day in my free time. This is also due to the fact that the activities have shifted from development to marketing.

[PL] Can you tell us more about the products you are bringing to the market?

[TH] Our first product was the Hovitaga OpenSQL Editor, a tool which can execute SQL commands on the fly. The results of SQL queries are displayed immediately by an ALV grid or a hierarchical ALV tree. We've packed a lot of features that help developers to create Select statements quickly. The second product is based on the first one, it is called Hovitaga Report Generator. The great feature of it is that it can generate standard SAP reports based on an SQL query. Depending on generation settings, 200-1000+ lines of code can be generated. You can imagine how much time you can spare with this. These tools can cut SAP IT costs by significantly reducing ABAP development, testing and maintenance time.

Our current development is a radically new kind of business intelligence tool called Hovitaga Data Visualizer. It is unique because it is based on Microsoft Silverligt, but it actually runs in the traditional SAP Gui. It can visualize data coming from SQL statements, ALV reports, SAP Queries (SQ01) or tables and views. The added business value comes from the way how data is presented: instead of a simple ALV list, you can analyze business data with interactive charts, maps, pivot grids, treemaps etc. This product is still in development, we currently seek a pilot customer and/or a development company who would join the development. We have just uploaded the first video to youtube.

[PL] How did you come up with the idea of building those applications?

[TH] The SQL editor was missing from the ABAP workbench:) I decided to write one. Again from a personal experience I wanted to reduce the amount of repetitive coding so we came up with the Report Generator. As for the Data Visualizer, I was on the opinion that SAP and SAP customers have put so much effort in developing highly complex business logic to extract data from the database (in forms of reports, queries) that it's a shame that the results of them are analyzed with a simple ALV list. Actually none of these products were suggested by any customer. That would have been much easier if a company would have sponsored the development.

[PL] Do you think the SAP platform is a good foundation for building a company that develops applications? What major strengths and weaknesses did you find?

[TH] In general it is all right. However if you want to gain exposure, you have to invest a lot of money. Presenting at TechEd or SAPPHIRE Now costs a tens of thousands (USD), advertising in SAP related magazines and websites is also expensive.

[PL] How do you distribute your applications? Do you use transports, saplink or something else?

[TH] This depends on what we want to deliver. I think we are the first company in the world (of course I can't be sure) who offer a totally free trial version of their SAP software. Anybody can download a free version of the OpenSQL Editor and Report Generator tools. In order to make the installation even more simple, we have made them "self installing", which means that all you have to do is to copy and paste the downloaded sourcecode into a new report with SE38, activate and execute. It will create the necessary screens, GUI status, messages etc automatically. We consider making a SapLink version also available for download. As for the full version of the products, my experience shows that customers are fine with transport files.

[PL] Are you planning to use WebDynpro for new developments?

[TH] No, not at all. I believe not so many customers use it, and it's already an out-of-date technology.

[PL] That's interesting. How do you see the future of GUI in SAP? Do you think classic Dynpro will be improved or do you think something else will become the SAP GUI of the future?

[TH] I think the UI will not be unified in the short or middle-term future. The traditional SAP GUI will stay for a very long time, and SAP will continue coming up with new technologies. However I'm not sure if those will get the desired popularity. Maybe if they do something with HTML5. But anyway, new technology is only accepted if it adds something to the business.

[PL] Are there some new SAP technologies do you think can be particularly interesting to independent developers?

[TH] Yes, the Netweaver Gateway and mobile applications in general, for example. Nowadays everybody has a smartphone and tablets are very popular these days.

[PL] What has been your experience with selling the applications to the customers? Do you think companies are open to evaluate applications from independent developers?

[TH] Our free products have been downloaded more than 500 times since mid of September 2011. This makes me optimistic. Around 20% of website visitors end up downloading a trial product.

[PL] Can you tell us more about your marketing channels? Do you get many inquiries from the website?

[TH] Since we've made the free trials available we have received around ten quotes from the website, but I know the purchasing departments of big companies (big enough to run SAP) usually work slowly. If there is a download in January, I expect a quote no sooner than February and an actual purchase not before April. We are considering to place our products in online webshops, but I'm not sure how many visitors and quotes would that add. Naturally this again would cost thousands of Euros per year per product.

[PL] I know many networks of companies focused on SAP consulting but so far I did not find any network of companies working on SAP applications. Do you know such networks?

[TH] No, please tell me if you know one :)

I will keep looking and let you know if I find one :-). Thanks, Tamás for this interview, I think it will be very interesting to many people and it is a great first post of 2012 for apolemia. I hope to have more post related to independent SAP developers during this year.

Using ZAKE to export objects to the filesystem

2011-10-04T10:48:00.001+01:00

A script to use ZAKE to export a package to the filesystem. I'm starting to use ZAKE and SAPLink to synchronize the ABAP repository with the mercurial version control system, so the default SVN object was not good for my goal.

onsap.com update

2011-09-22T11:30:00.002+01:00

I'm again using onsap.com to record questions and answers. Now the site allows users to subscribe to email updates of individual questions or tags so it is more useful as a way to follow some topic. I still think it can be a good tool for SAP people. I will keep using it to record my questions (and perhaps later update it with answers) and to learn a bit more from trying to answer other questions.

You can follow my activity here and you're very welcome to join.

Random element from a table in ABAP

2011-09-20T11:27:00.003+01:00

One function I find myself using a lot in python is random.choice that gives me a random element from a list. Quick example in python:


>>> import random
>>> for i in range(20):
...    print random.choice(['A','B','C','D']),
...
C A D C C D A D C B B B B C B C A C D B

It turns out it is not that difficult to do the same in ABAP.


class lcl_random definition.
 public section.
   class-methods choice importing it_table type standard table
                        exporting es_row type any.
endclass.

class lcl_random implementation.
 method choice.
   data: l_size type i,
         l_index type i.

   describe table it_table lines l_size.

   call function 'QF05_RANDOM_INTEGER'
     exporting
       ran_int_max   = l_size
       ran_int_min   = 1
     importing
       ran_int       = l_index
     exceptions
       invalid_input = 1
       others        = 2.
 
   if sy-subrc = 0.
    read table it_table into es_row index l_index.
   endif.
 endmethod.                   
endclass.                   

form test.
 data: lt_test type table of char1,
       l_test type char1.

 l_test = 'A'.
 append l_test to lt_test.
 l_test = 'B'.
 append l_test to lt_test.
 l_test = 'C'.
 append l_test to lt_test.
 l_test = 'D'.
 append l_test to lt_test.

 do 20 times.
   lcl_random=>choice( exporting it_table = lt_test importing es_row = l_test ).
   write: l_test.
 enddo.
endform.

Test functions in SE37 with commit execution

2011-04-12T11:57:00.007+01:00

A very simple trick that I only learned recently. A couple of times I faced the situation that I had a function that I could call to solve my problem. I would like to do it in SE37, but the function would need a COMMIT to be called after execution. I thought it was not possible to do in SE37 but it actually is. There is an option to test sequences:

So I just need to put my function and to perform the commit I can use the standard function BAPI_TRANSACTION_COMMIT.

How to see the number of objects in the Live Cache

2010-10-05T21:24:00.003+01:00

A quite useful report for checking the number of objects in the live cache is /SAPAPO/OM_LVC_OBJECTS. And since I only use it from time to time it's better to write something here so that I don't spend so much time searching for it the next time :-). More on the report in OSS 537210.

Issue with scheduling in delivered sales items

2010-10-05T21:09:00.008+01:00

Just a note to myself as I know I will have to answer this again. After the sales order item is delivered (even partially) the item is no longer scheduled. So what happens: the material availability date is calculated and all other dates are set equal to the delivery date (see FAQ OSS note 547508 item 2).

For example this was the item with correct schedules (the item was partially delivered):

And after a change in the item the scheduling becomes:

So the goods issue date is not the most reliable date, better use the delivery date or material availability date. The code responsible for this behavior is in FV45VFMV_MVERF_AUFBAUEN_WMENG line 763.


  [...]
    XVBEP-MBDAT = DA_DATUM.
  ENDIF.                                    
  XVBEP-LDDAT = DA_XVBEP-EDATU.
  XVBEP-TDDAT = DA_XVBEP-EDATU.
  XVBEP-WADAT = DA_XVBEP-EDATU.
  [...]

There is a possible fix by applying OSS note 833272, but this note (at least currently) is a special development not something going into support packages.

Funny SAP: Report /SAPAPO/SDORDER_DEL

2010-05-06T15:10:00.000+01:00

The program is useful and executing it in a systematic basis will make the database/livecache cleaner and faster.

No mercy with old data in SCM the system!

ATP requirement type configuration flowchart

2010-04-19T22:59:00.004+01:00

Time after time I find myself googling for some of my old posts. Most of the times for ABAP code related posts to copy the program, but I also look often for this one on the configuration of the requirement type in ATP. I recently created a flowchart with that information for a presentation, so I am now posting it here to make my life easier in the future (click to enlarge).