Financial Systems and Business Intelligence

IOS Printing

noreply@blogger.com (Paul Steynberg) — Tue, 02 Oct 2012 09:26:00 +0000

If you are like my household and have invested in tonnes of technology over the years you don't like to keep replacing it every time a new idea comes out. We are quite heavily invested in Apple and have an iMac, Macbook Pro, 2 x iPads, 2 x iPhones, 2 x Apple Tv's, so you want them to all work seamlessly. When it came to printing from my iPhone all roads led to an AirPrint solutions which would require me replacing my 5 year old Canon Pixma iP6600D. This printer has served me well over the years and is hooked up to my iMac.

A bit of searching around and testing and the solution is an application from Netputing called HandyPrint. You download it and use it for 14 days before it requires an activation key which can be obtained by donating via PayPal.

Once installed it appears in System Preferences under "Other" and automatically picks up the printer attached to the iMac. Now I can print from any IOS device in the house directly to the printer.

- Paul Steynberg

Correlate

noreply@blogger.com (Paul Steynberg) — Sun, 04 Sep 2011 06:39:00 +0000

The Google guys have been cooking up some very interesting stuff in the labs again. I took a look at the new trending service called Google Correlate today. Not entirely sure of its commercial value as an online free tool but it appears quite nifty.

One has to have a Google login in order to access the following site: http://www.google.com/trends/correlate but once in it is quite easy to navigate. One is able to trend web search data the Google has been collecting for many years. You can draw a line graph and the software will then match that profile to the closest profile it can find. One can also type in a web search string and it then finds the closest trend to that. For example, being in the Insurance industry, I typed in "Motor Vehicle Accidents" and the top 4 returned search trends are "vehicle accidents" (duh), "city planning", "questionnaires" and "battered women". The correlation between MVA's and battered women is strong (0.8744). I guess the big question is why? One is also able to upload a data file of time stamped data in order to find the closest correlation.

I guess one just needs to be mindful of the fact that these trends are not actual events but trends of search patterns. I would be interested in hearing from anybody who has used this technology in order to bring real life business use cases to life.

- Paul Steynberg

Controlled Use of Excel for BI

noreply@blogger.com (Paul Steynberg) — Tue, 23 Aug 2011 15:12:00 +0000

1. Introduction

In most Financial Institutions the use of Excel is so embedded that any project to curb this is almost doomed to failure from the start.

This blog is an attempt to elicit discussions and to explore the appetite to address some of the inherent risks with Excel spreadsheets and to see how one can deliver supportable MI using Excel. It should be stressed that this proposal should not be seen as a replacement for strategic tools or processes.

2. Benefits and Risks of Using Excel

The use of Excel comes with a multitude of pros and cons. In days past the downsides were considered to be worth the trade-off against the benefits but his is starting to change with the introduction of more stringent governance via legislation such as Solvency II, Basel II and SOX.

Below is a list of benefits and risks of using Excel in your core data streams.

2.1. Benefits
• The skills base for Excel is large and practically every person within a company will be familiar with it.

• Excel is relatively cheap and most users in an organization will have it.

• Excel lends itself to very easily creating financial models in a relatively short time span which normally could not be done in a more structured environment. Less reliance on IT means more flexibility and less time.

• The user experience from creating datasets to graphs is very intuitive and rich.

2.2. Risks
• Any VBA code or macros are specific to the workbook and are not held in a central repository to be reused. This raises concerns about version and change control over the code.

• If the required data set is sourced from any application/database within the IT estate, no way exists to document data lineage or do impact assessments when designing changes to these sub systems.

• Although Excel skills are widespread, most users will develop the model based on easy to develop rather than efficiency of code. No standard way of developing models makes it difficult to support/hand over.

• Any links to external data are also held in the workbook or specific to the PC. Often they contain T-SQL specific to the use-case and this is also not subject to version or change control.

• Excel has restrictions on the amount of data that it can consume. (1 million rows for Excel 2010, 2007 and 65,000 rows for Excel 2003)

• Security to the underlying database is not controlled via a centrally maintained application account but rather based on individual users. This requires users to be added and maintained at Database level.

• Access to spreadsheet models in not restricted via permissions or passwords as they are mostly stored in shared drives accessible to large groups of people.

• Users often save versions of the spreadsheet at various points during a process cycle and between process cycles for archiving. This leads to an explosion of redundant data and cuts of the code being held leading to increased storage costs. This can also lead to confusion as to what is the most recent and correct version. These issues are compounded by users sharing files by e-mailing them to each other.

• The models can become complex in nature and often process large volumes of data. These models can take a long time to run on a PC and in some cases can crash.

• Distributed development and locations lead to key person dependencies.

3. Risk Mitigation

In order to mitigate the risks outlined above one could put the following processes in place. Refer to Annexure A for the risk mitigation matrix.

• Set up standards on structure and development of Excel models.

• Ensure all models are developed against these standards, peer reviewed, tested, documented and appropriately transitioned into production.

• Store the models centrally in a controlled and identifiable location.

• Convert macros and VBA into add-ins and store them centrally.

• Implement security around the models for both access and changes.

• Implement appropriate change control and monitoring of the models to ensure that they are not changed without authorization.

In addition to the new or changed processes, technology can be used to mitigate some of the risks. Based on the risk matrix in Annexure B, a combination of Excel Services 2010 with PowerPivot and Prodiance, in theory could mitigate all the identified risks.

Below is a discussion around these technologies and how they could mitigate the risks.

4. Excel Services

Excel Services is the Excel engine delivered through Sharepoint and has the following key advantages:

• As it is delivered via Sharepoint on a server you can leverage the availability of larger memory, many more processors and 64 bit technology. This should reduce the time it takes to run larger and complex data models.

• Security is controlled via standard Sharepoint functionality.

• Centrally used data connections can be used thus eliminating the need for users to be granted direct access to the data.

• When published to Sharepoint one can only make certain parts of the model visible to the users thus hiding any business logic and underlying assumptions, enhancing security even further.

• Automatic scheduling of data updates can be implemented thus making refreshed reports instantly available to the users.

• As workbooks are versioned you can always restore to a previous one should you it be required.

Excel services does have some disadvantages listed below:

• Not everything that is native to the desktop version will work in Excel services such as add-ins and certain controls. Also some functionality becomes limited such as Pivot Tables and screen split and zoom functions.

• In order to modify the models you still need to pull them into the desktop version first and then republish them.

• Sharepoint is not a particularly easy technology to deploy and maintain and in order for Excel Services to be installed you have to have the Enterprise version.

5. PowerPivot

PowerPivot is an Excel add-in which has been developed by the Microsoft SQL Server Analysis Services Team and uses in-memory column compression technology branded as Vertipaq. This technology is very similar to that used by such market leaders as Qlikview. PowerPivot allows you to bring in large data sets into Excel and even join these sets to each other and then use the resulting data to report off. It is very efficient at consuming large amounts of data and when used in x64 bit mode with large amounts of memory is quite astonishing in its response.

PowerPivot also extends to Sharepoint and when combined with Excel Services in Sharepoint the uses and power suddenly start to become apparent.

6. Prodiance

In one of my blogs a while back I mentioned 4 companies that provide Excel Spreadsheet Control Software. We ultimately decided to back Prodiance as our choice, a decision that has now been ratified by Microsoft.

Up until 7th June 2011, Prodiance was an independent company specializing in risk and control software for Excel. It has now been purchased by Microsoft and is a wholly owned subsidiary. With Microsoft now purchasing the company more integration with the Office suite and Sharepoint is anticipated.

Advantages of implementing Prodiance over your spreadsheets are as follows (as per Prodiance pdf):

• Electronic sign-off and optional eSignatures

• Email notification of significant or unauthorized changes (e.g. exceptions, policy violations)

• Extensive cell-by-cell, file level and workflow audit trails

• Side-by-side comparison of changes between versions

• Management reports, dashboards and drill-down into detailed reports

• Automated document versioning

• Check-in/check-out (optional)

• Web based access to all historical versions

• Association of parent/child versions and unified view across all audit trails and reports

• Robust document security model integrated with Active Directory/LDAP users and groups

• Permissions to grant appropriate folder and document access levels (e.g. view, add, update, delete, approve, etc.).

• Microsoft Information Rights Management (IRM) encryption for spreadsheets containing sensitive information

• Workbook, worksheet and cell level protection

• Optional lock-down for cell level input control with data validation

• Excel Services for displaying and publishing read-only versions of critical spreadsheets and BI reports

• Extensive cell-by-cell, file level, system level and process level audit trails

• Tracking of changes to key inputs, outputs, spreadsheet data, formulas, macros and queries

• Tracking of changes to queries and data connections to external data sources

• Auditing support for spreadsheets and Access databases

• Optional lock-down for cell level input control with data validation

• Interactive link/dependency diagrams

• Auditing of specific input ranges (including named ranges)

• Automated email alerts upon changes to input ranges

• Color scheme tool to highlight (used and unused) input cells

• Spreadsheet validation and testing via automated cell, formula and range diagnostics

• Proactive identification of spreadsheet development and structural problems

• Comprehensive document and records management

• Enterprise class workflow management

7. Conclusion

In order to achieve the controlled use of Excel within an organization you will need to go through a change in technology, process and culture. I hope this blog has provided some readers with a spark for debate.

Appendix A

Appendix B

New Features in SQL Server Code-name Denali - CTP3

noreply@blogger.com (Paul Steynberg) — Wed, 27 Jul 2011 15:14:00 +0000

Microsoft are shipping some new technology in the upcoming version of SQL Server. This new technology is code-named "Apollo" and introduces two new features.

Columnstore Indexes
Vector-based query execution

These 2 features claim to speed up Data Warehouse query processing time by a factor of between 10 and 100. Follow this link for a full description of the these features. You can download the CTP3 of Denali here.

- Paul Steynberg

SQL Server Project "Crescent" Demo

noreply@blogger.com (Paul Steynberg) — Mon, 25 Jul 2011 09:51:00 +0000

Follow this link to see a demonstration of SQL Server Project "Crescent". To quote "a new immersive ad-hoc visualization tool that is part of SQL Server Code Name "Denali" Reporting Services. Project "Crescent" is designed with end users in mind to quickly, easily, and visually explore their data and answer ad-hoc questions in just a few clicks."

It sounds like exciting stuff and certainly plugs some gaps in the Microsoft stack.

- Paul Steynberg

Paperless Home

noreply@blogger.com (Paul Steynberg) — Sat, 18 Jun 2011 13:51:00 +0000

The concept of going paperless both at home and at the office has been around for many years. The execution of this idea has not always been easy or successful. I have tried this in the past with flatbed scanners and various methods of storing the output for easy retrieval and indexing. To date my attempts have been somewhat failures. This is until now. A few weeks ago I found the perfect combination of scanner and software that make the entire process a pleasure.

The solution I am using is for the Mac so if you are PC based you may need to do some more research. My requirements were quite simple. The process of scanning should be effortless and the software should be easy to use. I started looking at Evernote but did not like the idea of all my documents being stored in the cloud, especially bank statements, copies of passports, certificates, mortgage statements etc. Evernote is free as a download but in order to get the most out of it you are required to sign up with them on a monthly or annual basis. It is not expensive, $5 per month or $45 per annum and is very handy for ensuring your documents are held in offsite storage for backups. The application is great but the security issue tipped the scales for me.

After a lot of research and testing out various products I finally decided on DevonThink. Similar concepts to Evernote but a once off fee ($49.99) for the product and you store the documents locally in a database. This would mean that I would have to make sure that I am backed up but the Mac does this automatically through Time Machine.

DevonThink is easy to use comes with some great features such as side bar tray to which you can just drag documents from anywhere and an add-in to Safari that allows you to capture the open web page directly.

I then turned by attention to document scanners and opted for the Fujitsu ScanSnap S1300. This great little scanner (280mm long and 100mm wide) allows you to scan up to 10 pages at a time and can do either 1 side or both sides of the document. The other great feature I found is the ability to set up profiles for scanning which then call DevonThink and drop the document into the inbox for categorisation. It also does its own OCR and orientates the pages based on character recognition. Once the document is in DevonThink you can then search within the scanned documents for key words.

Now when the post arrives or we get back from shopping we just put the invoices/documents through the scanner with one touch of the button and then categorise them later in DevonThink when we have some quiet time. In the mean time all those documents can be destroyed and no more filing.

- Paul Steynberg

iXBRL - LinkedIn Group

noreply@blogger.com (Paul Steynberg) — Thu, 25 Feb 2010 09:40:00 +0000

Hi Everyone

I recently attended a product sponsored session on iXBRL and HMRC/Companies House submissions for the UK. I will put together an updated paper on new developments this week but would like to just highlight that I have created a group on LinkedIn called "iXBRL in the UK". This group is specifically designed to address issues that companies are having with the submission of accounts in the format and to discuss not only the process being followed to comply but also a forum to discuss any software solutions that start to make themselves available. Just search for it on LinkedIn and I look forward to any contributions.

- Paul Steynberg

PowerPivot Storage

noreply@blogger.com (Paul Steynberg) — Tue, 16 Feb 2010 15:20:00 +0000

For you hard core techies out there, stop reading now. This post is for those people in the world who had no idea that the Office 2007 (and upwards) files are actually just a bunch of files zipped up. I was curious to see how PowerPivot Data was stored with the Excel spreadsheet and was amazed to find out that a number of my associates were unaware of the make up of the new Excel formats.

In order to illustrate the concept take an Excel (or Word) 2007/2010 file and just to be safe make a copy of it. Then change the file extension from XLSX to ZIP and then open in Winzip. Shock and horror we find a whole bunch of files (mostly XML) in various directory structures. This was done in order to comply with Office Open XML Format. Read all about it here.

Back to the original question about how the PowerPivot Data is stored. It is stored in a directory called xl\customData with the extension DATA within the Excel (zip) file. Here is a bit more on how the data interacts with your memory.

- Paul Steynberg

PowerPivot

noreply@blogger.com (Paul Steynberg) — Mon, 01 Feb 2010 16:53:00 +0000

Microsoft promised BI for the masses and PowerPivot is the first real taste of this promise. Originally code named Gemini this Excel Add-in was developed by the SQL Server Team. In a nutshell it is SQL Server Analysis Services on your PC. In order to get it up and running you have to have Office 2010 (Beta) installed and XP should be on SP3. Here is a full list of requirements.

I am not going into the full technical bits of the product as this has been done many times before and here is a full list of links to get you up to speed:

www.powerpivot.com

www.powerpivotpro.com

www.powerpivot-info.com

www.powerpivotgeek.com

www.powerpivottwins.com

PowerPivot Team Blog

Kasper de Jonge

Take a look at the demo videos as they are quite quick and really show the power of the product. To illustrate I took a normalised data source and imported 3.5 million rows into PowerPivot in under 2 minutes. I then created a Pivot Table from the data with the new look and feel of having slicers across the top and down the left hand side. The results were blistering fast.

This is a first release and the product is full of bugs. As with most Microsoft products wait for V2 before releasing into your production environment. More to come later.

- Paul Steynberg

iXBRL - Don't Panic

noreply@blogger.com (Paul Steynberg) — Mon, 12 Oct 2009 14:25:00 +0000

Introduction

Companies in the UK will have to submit financials to Her Majesty’s Revenue and Customs (HMRC) via an online process within the next 2 years. I had a session with Ernst & Young about the language to be used called XBRL. I have also attended a workshop at HMRC titled “Company Tax Returns and Online Filing”. This is a summary of my findings and my personal views on the impact.

Overview of XBRL

XBRL stands for eXtensible Business Reporting Language. It is one of a family of "XML" languages which is becoming a standard means of communicating information between businesses and on the internet. (1). iXBRL is a slight derivative of this language which stands for InLine eXtensible Business Reporting Language and allows the data to be read by computers and viewed by humans all from the same file.

Impact

All companies filing tax after 31^st March 2011 will have to comply with the requirements of submitting all the data online. This will affect 3 submissions as follows:

CT600 – Currently can be submitted via XML and will not change.
Accounts – Must be submitted online in iXBRL format.
Tax Computations – Must be submitted online in iXBRL format.

What will not change is WHAT you submit, WHO is responsible for submitting it and WHEN it must be submitted, only HOW you submit it will change.

4 Taxonomies will be supported by HMRC but for the first 2 years of implementation only a limited subset of each taxonomy will be required.

Taxonomy	Full Tag Set	Limited Tag Set
UK GAAP	4375	1182
UK IFRS	3400	<1600
UK Common Data	900	900
CT Computational	4548	1350

One is not restricted to the limited tag set but you are only required to at least submit these for the first 2 years. Thereafter the full tag set will be required.

The current versions of the taxonomies can be downloaded from this website.( http://www.xbrl.org/uk/Taxonomies/)

The Companies House and HMRC issued a joint statement on 1^st September 2009 stating that they are working together and that both their filing services are being aligned so that a single point of filing can be used.

Software

As with any technology solution we will find many ways of solving this challenge ranging from ERP Supplier based solutions right through to stand-alone file converters. Unfortunately iXBRL is not yet widely used so I have not yet managed to find anything worth taking a look at yet. HMRC is busy reviewing a number of products and certifying them but when pushed they did not want to yield any names or costs just in case they were seen to favour one supplier over another. They did assure us that a number of suppliers would be ready for market in Q1 of 2010.

Here is a list of companies who I believe are working on a solution. Do not take this as gospel, just through my googling.

www.savanet.net

www.corefiling.com

www.semansys.com

www.rivetsoftware.com

www.ibs.nl

www.edgarfilings.com

www.allocationsolutions.com

www.justsystems.com

Approach

Awareness of XBRL and its impact is at an acceptable level within the UK market. Based on the Q&A session with HMRC the first 2 years are really going to be a phased/teething process and companies will not be penalised or prosecuted if they don’t get it quite right. We also do not have any products to work with as they are still being developed.

I would suggest a measured approach to the online filing which starts with Finance obtaining the taxonomies that they intend to use for filing and to start manually mapping the figures across from the financials. This in my opinion is going to be the largest portion of the work required. A deadline of the mapping tables by end Q2 2010 should be achievable.

Get your IT Department to keep tabs on the development of the various alternatives in the software space and then organize demonstrations and evaluations as and when possible. I estimate that we should be able to find something by Q2 2010. Based on all the input I would at this point suggest that we look to purchasing an Office Add-In that will allow you to continue with our current method of producing the financials and thus reduce impact on the business.

Works Cited

1. XBRL International. What is XBRL. [Online] [Cited: October 9, 2009.] http://www.xbrl.org/WhatIsXBRL/.

- Paul Steynberg

Solvency II - Data Requirements

noreply@blogger.com (Paul Steynberg) — Thu, 08 Oct 2009 09:17:00 +0000

Overview of Solvency II

Solvency II is a piece of legislation (directive to be exact) adopted by the European Parliament on 22nd April 2009 and is a fundamental review of the capital adequacy regime for European insurers and reinsurers. Planned effective date is October 2012 and it aims to establish a revised set of EU-wide capital requirements, valuation techniques and risk management standards that will replace the current Solvency I requirements. (2).

The full text of this legislation can be found here. Powers have been granted to CEIOPS in order to produce consulting papers and to engage with the industry in order ensure uniformity and clarity.

IT Impact

A section of the proposed framework deals with Standards of Data Quality which is outlined in a consultation paper from CEIOPS, referred to as CP43. (3). Data, according to this paper, is used to refer to all the information which is directly or indirectly needed in order to carry out a valuation of technical provisions, in particular enabling the use of appropriate actuarial and statistical methodologies, in line with the underlying (re)insurance obligations, undertaking’s specificities and with the principle of proportionality. In the context of this Paper, data comprises numerical, census or classification information but not qualitative information. Assumptions are not regarded as data, but it is noted that the use of data is an important basis in the development of actuarial assumptions.

Whereas this Paper is focused on setting out advice in the context of a valuation of technical provisions, it is noted that the issue of data quality is also relevant in other areas of a solvency assessment, for example for the calculation of the Solvency Capital Requirement (SCR) using the standard formula or internal models.

From the materials I have been reading and through a breakfast (1) I attended here are some issues that I would like to flag up. The CP papers expand on the concept that data should be Accurate, Complete and Appropriate. In doing so they highlight the following potential gaps.

Data Governance

Most IT policies focus on security. Substantial changes to policies will be required in order to ensure that we focus on data quality.
As a group you will need to understand what data quality means within your context and how to measure it. This also requires documenting.
Clear responsibility versus ownership of the data is required which also requires documenting. In my view this would be split between the business units and IT.
You need to start looking at the technologies/tools required within your Group in order to ensure data quality.

Monitoring

The subtleties of the text indicate a focus on both transactional and non transactional data.
Both regular and adhoc types of data will require a degree of monitoring and appropriate controls in place to deal with both types.
A move to focusing away from the accuracy of the data but rather its relevance and content is required.

Documentation

Additional documentation required with regular updating.
Decisions to data quality deviations now require specific documenting and approvals.

In order to ensure accuracy, any data deficiencies should be rectified, with each adjustment justified and documented and should not overwrite the raw data. It was also hinted that a more detailed review of the data is required to ensure that it is valid and appropriate. In other words the fact that data made it from your store system through a multitude of layers to a warehouse does not absolve you from the responsibility of ensuring that the original data was correct. This, in reality, leads me to believe that data profiling and review of individual pieces of data with appropriate monitoring tools is required. This applies to both external and internal data.

Addressing the issue of data quality will go to the core of your IT department and if the hype is to be believed will require substantial investment in ensuring that your processes and tools are up to the task. This will have an impact on your IT strategy both from a technology and approach view.

Audit Impact

Article 46 of the Directive Consolidated Text referred to as Insurance and reinsurance (Solvency II)(recast) defines the responsibility of the Internal Auditors (IA) within the scope of Solvency II. (4). This article requires the usual stance of being independent and specifically requires that IA provide an effective internal audit function which includes an evaluation of the adequacy and effectiveness of the internal control system and other elements of the governance system. (This gets a bit circular as the resolution passed under (18a) says that the governance system includes the risk management function, the compliance function, the internal audit function and the actuarial function, which implies that they audit the same system that they are part off).

CP33 from CEIOPS deals with the advice on Governance and makes reference to the functions of Internal Audit. (5).

Although the requirements do not appear to be any different from any IA business as usual functions they are advised to submit a written report on its findings to the administrative or management body at least annually. It makes sense then to assume that IA have a full understanding of the Solvency II requirements.

Conclusions

Solvency II is going to have quite an impact on the way we do business as an Insurance Company and is something that should be driven from the highest possible position in the company as it will require substantial resourcing. This initiative should not be driven from within one department.

Works Cited

1. Addressing the data and technology challenges of Solvency II. PriceWaterhouseCoopers. London : s.n., 2009.

2. Financial Services Authority. Insurance Risk Management: The Path to Solvency II. s.l. : Financial Services Authority, 2008.

3. CEIOPS. Consultation Paper No.43 - Technical Provisions - Article 85 f Standards for Data Quality. Frankfurt : CEIOPS, 2009.

4. European Parliment. Insurance and reinsurance (Solvency II) (recast). Strasbourg : s.n., 2009.

5. CEIOPS. Consultation Paper No.33 - Draft CEIOPS Advice for Level 2 Implementing Measures on Solvency II: System of Governance. Frankfurt : s.n., 2009.

Who Still Trusts the Gorilla?

noreply@blogger.com (Paul Steynberg) — Thu, 01 Oct 2009 12:01:00 +0000

Where does a 500 pound gorilla sleep? Anywhere it wants. Enter left stage - Microsoft.

At the beginning of the year Microsoft canned development on PerformancePoint Server Planning and laid off a whole pile of people. Some of them by e-mail, I know, one of them was on site with me in the UK when he got the news via an e-mail. To me this was a pivotal moment in our relationship with Microsoft. A year ago nobody questioned Microsoft's commitment to software development and products. Today a very different story. A few weeks ago I was discussing the way forward in our ETL architecture with my current employer and we had 2 roads, either IBM DataStage or Microsoft SSIS. The most senior person in the room posed this question "Are we sure that Microsoft SSIS will still be around in the future given the demise of PPS?". A year ago that question would have elicited chuckles from the boardroom table as an obvious joke. Not anymore, it was a serious question which required follow up.

I don't think that Microsoft has any idea how much damage they have done to their reputation in the market.

What do you think? If you have time please complete my on-line quick poll on this.

- Paul Steynberg

Gartner on Mid Market ERP Systems

noreply@blogger.com (Paul Steynberg) — Mon, 28 Sep 2009 14:56:00 +0000

Those of you who have followed my blog will know that prior to me leaving my South African employer I implemented Dynamics AX 2009 (Axapta 5.0). The project was a raging success and done within budget for considerably less than the alternatives on the list. It is always a good thing to have your actions supported in the future and I came across this Gartner report which put Dynamics AX in the top right hand quadrant of the famous Gartner Magic Quadrant.

This report is definitely worth a read and was published in June 2009.

- Paul Steynberg

A Novel Way of Dealing with Excel Models

noreply@blogger.com (Paul Steynberg) — Sat, 29 Aug 2009 11:01:00 +0000

Excel is both heaven and hell in the business world. It's a great calculator, modeler, reporter, you name it. With this incredible flexibility comes a trade off, your data is stored in multiple locations throughout the company in an unstructured and often unsecured manner. This data is not visible to the Enterprise and in most cases is not integrated into your reporting stack. Loads of capturing and data copying results in huge inefficiencies and in many cases errors.

My current employer embarked on a project 18 months ago to "black box" a number of these high profile models. Without going into the gory details let's just say that it did not work. I joined them in April of this year when my family and I made the move to London and was tasked with firstly evaluating the models to give an opinion on whether they were fit for purpose and could they be supported into the future. Sadly they could not and it was my unfortunate responsibility to stop the bleeding and bring the project to a conclusion. That was the easy part, the hard part is what do you do now as the original issues around the Excel Models were still relevant.

The obvious choice would be to take a look at the short listed products again that were selected at great expense and find the next suitable candidate. I was not comfortable with this approach and we did not want to try and rebuild them in PerformancePoint as all development for the product has been discontinued by Microsoft. (Don't even get me started on this subject, I have bitten my tongue since they shook the market with this announcement in January 2009).

I have come up with a rather "out of the box" approach which my team is in the process of designing at the moment. We needed to achieve a few objectives from the project being:

The models should be flexible
They should have access control
Changes should follow appropriate change control
They should be auditable
The data should be stored centrally and available to other business systems

Package these requirements up with the new Solvency II requirements and SOX etc, and you have quite a tall order.

Off course the most flexible model is still Excel. So what if you could keep Excel and also apply security, access control, change control, auditing etc. This led me into a journey of trying to find software that would give us this functionality. And guess what, it does exist, albeit a software concept in its infancy. Gartner have even released a paper on it (March 11, 2008) under the heading "MarketScope for Spreadsheet Control Products, 2008". It was quite comforting to find out that we had made contact with all the major players prior to this report and that our findings were in line with Gartners. Here are the major players in this space:

They all have a presence in London. Cimcon and Prodiance are US based suppliers whereas ClusterSeven and Finsbury are UK based.

Now they all do pretty much the same thing but in slightly different ways. Essentially each product has 3 major functionality groups:

Discovery and Risk Assessment - Basically trolling through all your locations looking for spreadsheets, categorising them and creating a baseline.
Monitoring - Keeping track of your spreadsheets and keeping a full audit trail of all changes and versions.
Development Tools - Some are add-ins, others applications that assist in developing your spreadsheets to minimise risk, errors etc.

So in a nutshell you can see who did what to which spreadsheet over time and set up alerts and workflows etc. You will be able to see broken links between spreadsheets and a whole host of interesting things that are out of the scope of this paper. We are in the process of getting RFP's from the suppliers and although I have some opinions will reserve them until a later date.

This piece of software should take care of most of the issues surrounding the Excel Models. Now for centrally storing the data and making it available to the rest of the business. My team is in the process of requirements gathering for a framework that will allow us to upload many different data sets from these models into a SQL Server database and thus make it available to our reporting layer. This may sound quite simple but to build a framework that is scalable. supportable and flexible is no easy task.

Watch this space!

- Paul Steynberg

Management Reporter/Frx Roadmap

noreply@blogger.com (Paul Steynberg) — Fri, 13 Mar 2009 07:50:00 +0000

The big question has finally been answered, what is the future of Management Reporter, FRx Reporter, FRx Forecaster and Enterprise Reporting? Well MS has released this roadmap.

2010 - MR to Replace FRx Reporter as the reporting tool of choice for Dynamics AX. This will coincide with the release of AX2010 (6.0) and it will be called MR V2.
2012 - MR to be released as V3 with AX2012 (7.0) and will now include Forecaster.
2014 - MR to be released as V4 with AX2014 (8.0) and will include functionality from Enterprise Reporting.

Personally I would not move from FRx Reporter until 2012 having learned a very hard lesson over the last year or so with the current version of MR.

Regards

Paul

Cannot Install Management Report Directly to SQL Server 2008

noreply@blogger.com (Paul Steynberg) — Wed, 11 Feb 2009 09:33:00 +0000

I found a bug some time back that did not allow one to install Management Reporter while linking directly to a SQL Server 2008 database. One had to install RTM and point to a database which was SQL Server 2005 and only once SP1 for MR (SP2 for PPS) was applied could you then change the database to SQL Server 2008.

Just had confirmation from Microsoft that this will be fixed in SP2 for Management Reporter.

SP1 for MR was actually packed with SP2 for PPS so we will get this in SP3 for PPS which will be released in the Summer.

- Paul Steynberg

AIF Updated Doc for AX 2009

noreply@blogger.com (Paul Steynberg) — Sat, 31 Jan 2009 06:40:00 +0000

The documentation surrounding the use of the Application Integration Framework (AIF) with BizTalk has been updated to now include AX 2009. Here is the link. http://www.microsoft.com/downloads/details.aspx?familyid=edc62433-5b21-4f74-b065-b075ba6dc86d&displaylang=en&tm

- Paul Steynberg

PerformancePoint Server Management Reporter and SQL 2008

noreply@blogger.com (Paul Steynberg) — Wed, 14 Jan 2009 21:14:00 +0000

Does Management Reporter (MR) work with SQL Server 2008? Yes, if you apply SP2 which has been recently released by Microsoft. I have tested this with huge improvements in performance. But that is not the reason for this blog. Something a lot more sinister is afoot.

I started the MR rollout to the business after upgrading all our pre-sp2 installs. We migrated the ManagementReporter database from the SQL Server 2005 environment to the SQL 2008 environment with no problems.

Here is the BUT. When you install MR on a clean machine you MUST, during the install process give it a valid ManagementReporter database to connect to. Now in order to install SP2 you must firstly install RTM. SP2 is designed to work with 2008 but RTM does not and lets you know in no uncertain terms. So you sit with a chicken/egg story. You want to install SP2 to make it work with 2008 but because the DB is 2008 you cannot firstly install RTM.

Work around. DO NOT delete a copy of MR DB which you have anywhere on the network that is already on SQL 2000/2005. Or just install a DB from the RTM version on to any 2005 SQL box. During the client install you must point to this DB in order to get RTM complete. Apply SP2 and then create a new connection to the 2008 DB and delete the old connection. Simple but unfortunately necessary.

This has been raised as a bug and is in production.

- Paul Steynberg

PPS Planning - Current Period

noreply@blogger.com (Paul Steynberg) — Thu, 18 Dec 2008 07:46:00 +0000

Over the past 2 years one of the biggest mysteries to a lot of PPS Planning users is the storage of the current period which is set in the Business Planning Modeler. This has been discussed and a number of posts on the PPS forum have clearly demonstrated how one can get to it. However I have yet to see a full set of code to bring this setting into your environment for use. I use the current period for my hourly PPS updates from our ERP system so as to make sure that we do not reload old data and speed up the process. So part of my ETL process interrogates the XML blob in the table BizAppNodes and returns the current period ID and Label for each model. These records are inserted into a table that I keep up to date and then use this in my ETL process. Here is the code:

DECLARE @xmlblob xml
SELECT @xmlblob = CAST(CAST ( ba.xmlblob AS varbinary(MAX))
as xml)
FROM BizAppNodes ba
WHERE VersionEndDateTime = '9999-12-31
00:00:00.000'
AND BizAppNodeLabel = 'FinanceModel'

SELECT distinct ModelLabel

,CurrentPeriodId

,T.[MonthLabel]

FROM

(

SELECT DISTINCT

tab.col.value('../../@Label', 'varchar(30)') as ModelLabel,

tab.col.value('@CurrentPeriodId', 'varchar(30)') as
CurrentPeriodId,

tab.col.value('@EffectiveDate', 'varchar(30)') as EffectiveDate

FROM @xmlblob.nodes
('/BizModelSite/Models/ArrayOfBizModel/BizModel/EffectiveDatedCurrentPeriods/EffectiveDatedCurrentPeriod')
as tab(col)

) AS TABLE1
LEFT OUTER JOIN [dbo].[D_Time] T ON T.[Month] =
TABLE1.[CurrentPeriodId]

The output is the model name, period id and period label for each model in the application.

- Paul Steynberg

PPS Up Your Street

noreply@blogger.com (Paul Steynberg) — Fri, 12 Dec 2008 06:53:00 +0000

I have added a new link to another PPS blog by David Street. It makes for some good reading.

- Paul Steynberg

PerformancePoint Server SP2 - Feedback

noreply@blogger.com (Paul Steynberg) — Wed, 10 Dec 2008 06:57:00 +0000

Nick Barclay, Tim Kent and Sacha Tomey have all posted some detailed blogs on what you can expect from PerformancePoint Server SP2 so I will not labour the point. However here are some real live bits and bobs as I installed SP2(Beta) over a month ago under the following configuration:

PPS Server, Windows 2008 Server and SQL Server 2008. X64, 4x6 core processors, 64 Gb Memory.
SQL Server, Windows 2003 Server and SQL Server 2008. X64, 2x4 core processors, 64 Gb Memory.

We put a new server in for PPS but used an existing one for SQL. This will be upgraded within the next 3 months.

The install did not give any surprises. You must firstly install PPS RTM and then apply SP2 as was expected. Something that did bite us was the fact that you could use the PBM on the server but NOT from your desktop. My guys in networks are convinced that it is something to do with the way credentials are double hopped from the desktop to the PPS Server to the SQL Server. (They mumbled a whole bunch of stuff about Kerberos and AD and it would take days to figure out the problem.) To fix this we just changed the PPSPlanningWebService “ASP.NET impersonation” from Enabled to Disabled. Here is the error message you will find in your event viewer:

Date 14/11/2008 15:38

Log SQL Server (Current - 14/11/2008 15:38)

Source Logon

Message

Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'. Reason: Token-based server access validation failed with an infrastructure error. Check for previous errors. [CLIENT: [Ip address here]]

Date 14/11/2008 15:38

Log SQL Server (Current - 14/11/2008 15:38)

Source Logon

Message

Error: 18456, Severity: 14, State: 11.

Considerations when Installing on Windows 2008.

You need to make sure that Web Server (IIS) is installed. If not open Server Manager, click on “Add Roles” and Install Web Server(IIS).

Once installed open up Server Manager and click on Roles.

Click on “Add Role Services”.

Install the following services:

Application Development, all except “Server Side Includes”.

Under Management Tools, “IIS Management Console” and “IIS 6.0 Management Capability”.

Under Security “Basic Authentication” and “Windows Authentication”.

Now install the PPS RTM 64 bits. DO NOT run the Configuration Manager.

Install the PPS SP2 64 bits.

Now you can run the Configuration Manager for the PPS Planning. You must have SQL Server Cumulative Update 7 installed. We needed to install these files to ensure that we met the requirements:

SQLServer2005-KB949095-x64-ENU.exe
sqlncli_x64.exe
SQLServer2005_ADOMD_x64.exe
SQLServer2005_XMO_x64.exe

To register ASP.NET 2.0 Web Service Extensions in IIS:

Open cmd : Start -> Run -> cmd
Navigate to the correct folder and run:
C:\Windows\Microsoft.NET\Framework64\V2.0.50727\aspnet_regiis –ir

You can then start the Configuration Manager again and from here on it is stock standard as per normal.

We needed to update to SP2 so that we could get the benefits on the Management Reporter in terms of performance. The improvements in performance are massive HOWEVER they are just not good enough. It still takes way too long to produce our reports. So back to the drawing board for Microsoft. They are now performing a full evaluation of the product and are working out how to improve it.

- Paul Steynberg

PerformancePoint Server - An ETL Tip

noreply@blogger.com (Paul Steynberg) — Fri, 05 Dec 2008 15:01:00 +0000

Anyone running a Financial Model within PPS Planning in all likelihood updates the actuals from an ERP System within the business. Under normal circumstances this will entail (amongst a myriad of other things) the following:

Synchronize MG Tables to the Stage Area.
By way of some ETL tool (Normally SSIS) bring in your actuals.
All Deletes, Inserts and Updates are then written into the MG Table with the appropriate BizSystemFlag. 200 for Inserts, 300 for updates and 400 for deletes.
The data is then loaded from staging to the RefDB.
The model is processed.

Now one of the biggest problems in this entire process is the time it takes to Synchronize the MG tables to the Staging database and then if you have an enormous number of records, the inserting of these to the RefDB. (We have 22 million records in our MG table).

So we have decided to shortcut the process. We created our own ETL SSIS packages to move the data into the MG tables in the Stage database. This method completely negates the effort of synchronizing the MG table to the staging database as we join across the 2 databases in order to detect any updates or inserts. As the data always comes from a LOB system we never do any deletes. For new records we just insert them into the Stage database with the BizSystemFlag of 200. For updates we fetch the existing record from the RefDB into the StageDB and insert an additional record with the BizSystemFlag of 300. You cannot insert a record of type 300 or 400 without the accompanying 100 record. If you do the load will fail and you will get errors on those records.

By adopting this methodology we have reduced our update of PPS to under 5 minutes and it is run every half hour so that our reporting will be up to date. Another advantage of not synchronizing is that your indexes on the MG table in the staging DB are not dropped. Saves a lot of time.

- Paul Steynberg

PerformancePoint Server - Balancing Your Actuals

noreply@blogger.com (Paul Steynberg) — Fri, 05 Dec 2008 11:48:00 +0000

If like me, you are loading your actuals from your ERP system in PerformancePoint Server Planning, it helps to check your figures to make sure that they always balance. Now every good accountant will tell you that your trial balance must always balance to zero. This goes for each period and off course year to date. I have written some reports that self balance our system but in general here is the manual leg work behind it.

Remember that when loading from your ERP system all Balance Sheet Items are loaded as “CLO” for closing balance and all your Income Statement items are loaded as “NONE”. Your rules will calculate the Opening Balance “OPE” records and the Movement “MVT” records prior to you processing the model. In order to balance you then just bring the data into a pivot table (Standard connection to analysis services) and then check that your totals come to zero. But it can get a bit confusing if you do not get your combinations of Flow and TimeDataView correct. So to check you Year to Date figure you select TimeDataView as “YTD” and multi-select “NONE” and “CLO” for the flows. To check the period movement only change the TimeDataView to “Periodic” but then set flow to “NONE” and “MVT”. Voila, it should all balance as per the example below. In this example the 1 and 2 series are Balance Sheet and 3-9 series Income Statement. Also note that if you just add up the numbers you will not get to zero but the Pivot Table understands that some of them are signed and that they should be subtracted. Take a look here and here for a quick overview on this subject.

- Paul Steynberg

PPS Management Reporter - Some Key Tables

noreply@blogger.com (Paul Steynberg) — Thu, 04 Dec 2008 09:28:00 +0000

While authoring some reports using the Management Reporter Designer from PerformancePoint Server I had reason to start digging through the tables in order to make sense of the row definitions. During this little excavation I identified some tables that thought might just be useful for somebody else if they knew this information upfront.

Firstly it would appear as though the report meta data is stored in a set of tables with the prefix “Control”. So for example the row definitions are stored in a number of tables but the one that resembles the layout in the designer is “ControlRowDetail”. Listed below are a list of tables that I needed to look into.

ControlCompany

When one has connected to a Management Reporter database (known as a connection) you are then required to connect to an entity. One creates an Entity and when doing so you have 2 “out of the box” sources being the Financial Data Mart 7.0 or PerformancePoint Server. This data is stored in the ControlCompany table. The connection data is stored in XML format and looks something like this:

EntitySetting Name="Connection"; Value xsi:type="xsd:string" /; /EntitySetting; EntitySetting Name="Model"; Value xsi:type="xsd:string";Financial Reporting/Value; /EntitySetting; EntitySetting Name="FunctionalCurrency"; Value xsi:type="xsd:string";ZAR/Value; /EntitySetting; EntitySetting Name="Calendar"; Value xsi:type="xsd:string" /; /EntitySetting; EntitySetting Name="Address"; Value xsi:type="xsd:string";http://servername:46787/Value; /EntitySetting; EntitySetting Name="Application"; Value xsi:type="xsd:string";TheGroup/Value; /EntitySetting; EntitySetting Name="ModelSite"; Value xsi:type="xsd:string";FinanceModel/Value; /EntitySetting; EntitySetting Name="OLAPServer"; Value xsi:type="xsd:string";OlapServerName/Value; /EntitySetting; EntitySetting Name="OLAPDatabaseName"; Value xsi:type="xsd:string";TheGroup_FinanceModel/Value; /EntitySetting; EntitySetting Name="Cube"; Value xsi:type="xsd:string";Financial Reporting/Value; /EntitySetting; EntitySetting Name="CalendarHierarchy"; Value xsi:type="xsd:string";Financial Calendar/Value; /EntitySetting; /ArrayOfEntitySetting;

Associated with the Entity is also a Building Block Group. This is stored in the field SpecificationSetID.

ControlSpecificationSet

The ControlSpecificationSet table stores the Id, Name and Description of the Building Block Groups referred to in the ControlCompany table.

ControlRowMaster

By using the SpecificationSetId from the ControlSpecificationSet table one can then return the Id’s and descriptions of the Row Definitions for the Building Block Group selected.

Now in order to get to the data that I needed (ie what accounts make up the lines in the Row Definition) one has to look at several tables being the ControlRowLinkMaster which gives you the ID for the ControlRowCriteria field called RowLinkId. You obtain the RowDetailId from the ControlRowDetail. This is used to build a query on the ControlRowCriteria, see example below:

select * from dbo.ControlRowCriteria where
rowlinkid='76DCEA67-9069-46E1-9704-2F42A3E0BC72' (Obtained from
ControlRowLinkMaster) and rowdetailid='E430DB82-081B-475A-B7E4-81AF5ECC3725'
(Obtained from ControlRowDetail)

You now have the dimensions and exactly what the criteria for each of those dimensions are for the row being looked at.

I needed to do this in order to generate a list of lowest level accounts from a report to make sure that EVERY account in my hierarchy was represented in the report at some point.

I am sure that the same logic above can be applied to Column Definitions etc etc.

Other tables worth mentioning are those surrounding security which all start with the word “Security”. So to find your users you look at SecurityUser etc.

- Paul Steynberg

Dynamics AX 2009 - Inconsistent Data When Exporting to Excel from Grid

noreply@blogger.com (Paul Steynberg) — Wed, 03 Dec 2008 12:35:00 +0000

Sometime back I listed some bugs that we had found in Dynamics AX 2009. One of the bugs was exporting large record sets from the journal lines grids into Excel. See here.

Although we have not managed to get this fixed here is a workaround until Microsoft releases the fix.

1) Go to AOT – Forms - LedgerTransAccount [this is the form that corresponds to Chart of Account Details - Transactions ]

2) Expand the node to view the datasources. Select the datasource ‘LedgerTrans’

3) Click on the ‘Properties’ icon or alternatively, press ‘Alt+Enter’, to open up the Properties window.

4) The property ‘StartPosition’ by default is marked ‘Last. This results in the cursor being at the last whenever the particular form opens up. Change this property to ‘First’ so that the cursor will always be at the top of the grid for the mentioned form

So whenever one opens this grid it will automatically navigate to the first record and will thus export correctly to Excel.

- Paul Steynberg