Worldpay were implementing a greenfield Siebel implementation as part of a much larger project to replace the majority of their IT systems. Mark was employed as a Siebel developer to compliment a large team (approx 10-12 developers). Initially mark worked on number of small enhancements and defect fixing before getting involved in a number of mini projects.

One such mini project was required to receive a large XML file of unknown structure, extract certain key elements from the file, make some of the data available for editing in the Siebel front-end, before having to reconstruct the XML after updates had been applied in Siebel, and then returning the modified data to the source system. There were a number of complexities around this, mainly due to the volume of data in each XML record and the flexible nature of the file, meaning that we could not store the data in dedicated files/fields in Siebel, and storing data in name/value pairs would present significant performance issues, not to mention make the editing of the data rather cumbersome for the users.

The final solution was to break the XML into a number of component parts, reformat the data such that it used less characters (and hence required less storage space), store the component parts in CLOB columns, and then use VBC's to present the data to the user in a user-friendly format.

Mark also used his extensive experience and skills with Excel COM loaders to tackle Worldpay’s own data loaders. Worldpay had a large volume of data entities that were loaded via a single spreadsheet, with each entity having it's own custom script to transfer the data between the spreadsheet and Siebel. The size of the spreadsheet was an issue in itself as it would take a long time to load, especially over the network, and on occasion it would simply refuse to work at all. Having all loaders in a single sheet also created management issues when more than one developer wanted to update it at the same time. And much of the script behind the spreadsheet was of such a poor quality that many of the loaders simply didn't work, or even worst, they would load data incorrectly resulting in data corruption.

Mark therefore developed a framework within the Excel loaders that meant different spreadsheets could use the same scripts to load data, thereby improving code reuse and hence maintainability, allowed multiple developers to work on different data loads at the same time, and reduced the size of each spreadsheet improving the time it took to open them. It also meant that all the data loaders were standardised in the way they worked, the code they used and the way they reported progress and issues. This reduced commonly made mistakes, improved the reporting of progress and, more importantly, reporting of errors so that problems with the data could be more easily and quickly identified. This standardisation of loaders also meant that new data loaders could be created with only a few lines of code, thereby massively reducing the development time - a new loader could literally be created in just a few minutes, and since the loader would reuse tried-and-trusted code it would almost certainly work first time.

As with most things in life there were a few exceptions to the afore mentioned data loaders that required a more dedicated solution. One example of this was for Views and Responsibilities. Previously the business would manage the responsibilities, and their assigned views in a matrix. When the time came to update Siebel with a new "matrix", developers would spend a lot of time filtering the matrix and copying/pasting data into a more standard data loader, before loading the data into Siebel. Typically, this approach only took care of new responsibilities or new view associations (basically just doing an upsert), any instances where a view had to be removed would be managed by manually deleting the view from the responsibility. This was an incredibly time consuming and clunky process, and had a high risk of something being missed.

Mark's solution was simple, to create a data loader that imported the data directly from the matrix, thereby removing the time-consuming copying/pasting. In addition, the loader would run various checks prior to loading to ensure that view names were correct (it is very common for business analysts to enter the wrong view name), highlight which responsibilities were new or would be updated, as well as indication which view/responsibility associations were new or required updating. In addition, as the full matrix was to be processed each time the loader could identify view associations that were no longer on the spreadsheet and remove these from Siebel.

In modifying all the data loader spreadsheets Mark also took the opportunity to review and improve the documents and procedures used to manage the deployment of data. This was to ensure an accurate record was maintained of all the data items to be updated in each release, and to monitor the status of each.

Being a large project there was a lot of information to manage, many environments to keep track of, many different spreadsheets for all sorts of reasons, etc. When Mark joined the project, everyone had their own means for keeping track of all this, and although some team members were happy to share, others were not so. And even those who did share would often do so by sending an email that other team members then lost amongst the myriad of junk in their inbox. So Mark set about creating a very simple website (#ShareTheKnowledge) where developers could post any useful titbits and share them with other team members. This included details of all the environments that were used, with links to the various login pages, details of the servers they were hosted on, database names, etc. The site also contained links to key documents, and details of certain procedures that were relevant to developers (e.g. deployment notes). This web site was nothing more than simple MS Word documents saved as HTML pages meaning that everyone could maintain and add content with ease, and without any web building knowledge.