The US Energy Information Administration collects, analyzes, and disseminates data from energy producers of all sectors (including coal, electricity, nuclear, petroleum, and biodiesel) to educate policymakers and promote a thorough public understanding of the energy market and how it influences (and is influenced by) the economy and the environment. Delivering reliable, impartial information to both policymakers and the public is the EIA’s primary mission, and their analyses directly affect global energy markets. For example, their weekly petroleum reserves report drives the market for oil commodities. Because the information they collect and distribute bears so much weight, accuracy, efficiency, and timeliness are paramount.
The EIA’s surveys and database are completely disparate systems – collecting and storing information with a variety of survey methods and formats. The lack of integration results in a manual, fragmented structure that is difficult for respondents – the energy producers – to navigate and costly for the EIA to update and maintain. The inconsistent processes also prevent the EIA from efficiently measuring data across surveys to glean insights and leads to a lag in reporting. Months may have elapsed between data collection, assessment, and publication.
Qlarion, working in concert with Informatica, has designed a data management architecture that meets all of the client’s requirements (data ingestion via web form, PDF, or XML, data cleansing, data integration, workflow, etc.). The end goal of the program is a single, flexible platform that not only accommodates and streamlines existing processes and rules but also provides a scalable, adaptable solution for future developments.
Qlarion developed ETL mappings for smoother integration of responses from the various energy companies and implemented the Informatica Metadata Manager to manage the metadata and data validation rules across the data management infrastructure. Qlarion went on to model the data warehouse to store the data and make it available for analysis and reporting. Once the new architecture was in place, Qlarion worked closely with the EIA to help them adapt their business processes accordingly. Finally, Qlarion has been closely examining previous data to uncover and purge inaccuracies. The team continues to conduct extensive testing and documentation to improve processes.