Posted by: Francois Koster
As report creation gets ever more flexible, how can we ensure that reports are correct?
Modern reporting systems are pushing the boundaries of technology, making reports easier to produce than ever. These systems rarely take into account the control and compliance requirements of a report. This can expose a company to financial, reputational or regulatory risk. However, even in the traditional reporting model, the control process is often out dated or inadequate and usually is disassociated from the report.
The Problem Zones
7 Problem zones can be identified:
1. Improvements in technology neglect the control requirements.
In a traditional scenario, reports are analyzed, specified, built and verified. Control processes are put in place to verify that all the pre-requisites of the report are met.
With the improvements of technology, many reports can be built on the fly. While this is a good thing, there is usually no consideration in regard to vulnerabilities in the report and little or no control of the reports correctness.
2. The control process is too simplistic
Even if a control process is defined, it rarely does more than check some basic data requirements and some minor quality checks. Commonly this is driven a lack of business knowledge by the developers, a lack of technical knowledge by the business and limited resources in an aggressive timeline.
3. The control process is disassociated
This is, probably, the most common and fundamental problem with control processes for reports. The process is created with the initial report creation and does not remain linked to the report itself. As the report evolves the control process stays static and new or changed elements in the report create vulnerabilities.
An additional problem with disassociation is that the execution of the process occurs independently from the report generation. So even if the process is perfect, you are still exposed to someone executing a report before the control process is completed.
4. Report distribution further disassociates the report from its control process.
The department producing a report may be fully aware of problems within a report. They will often inform the second party of these issues, but will the second party inform the third party?
This is an issue when people require early indication reports as the control process is, by necessity incomplete at the time, but people need to know the rough numbers. Along the way, the information that this is a first cut report is lost and the figures are assumed correct.
5. Too much trust in the reporting system.
In a similar vein, when someone receives a report, they assume that the person before has checked and validated it already. Unfortunately, as trust in a reporting system increases the manual control of the report decreases.
When someone has run the same report for years without issues, they start to take for granted that everything is okay.
6. Over time the knowledge carriers are no longer the report creators.
As a report matures and trust in its completeness and correctness increases, generation of the report is migrated away from the experts to free them for new tasks.
The problem is, though, that these experts provide an additional control step that is often underestimated. Any IT person who has worked with reporting over many is bound to have heard the comment "That can't be right". The familiarity that experts have with their data gives them an immediate feel for whether something is going wrong on a report. Sometimes these are simple signs, like the number of portfolios being wrong or the returns being too low. Sometimes these are very complex judgements in contribution or attribution figures, where familiarity creates an expectation of results and significant deviation causes the expert to be suspicious of the report. Once these people are no longer running there eyes over the report before it is distributed, this simple but valuable control is gone.
7. The small inconsequential change
This is one of the nastiest but frequently occurring problems in real world reporting. Significant technical changes or business process modification will mostly get caught in a test phase. If not, they cause a high impact in the production environment and are immediately addressed.
The small inconsequential change can sneak through test phases and only cause oddities in some reports under certain conditions. The result is that reports are distributed before the error is found.
Next week part 2 of 3
Do you have anything to say about this, please share it!