Data Quality Assessment

Background

At the 5 th Conference on Joint Oil Data Initiative in Bali in October 2004, the participants agreed that before the release to the public of the JODI database, JODI data had to be of good quality and as accurate as possible .

A Review Committee comprised of representatives from each organization and chaired by the IEF was set up to assess the quality of the data, particularly for the top 30 producer, consumer and stock holder countries. Moreover an independent oil analyst was hired to help the Review Committee in its task and to ensure the impartiality of the assessment.

Data quality comprised several items: timeliness, completeness, sustainability and comparability. The assessment on participation in JODI (on timeliness, completeness and sustainability) is conducted bi-annually.

The basic quality of the data is assessed through regular checks. However in addition, the Review Committee collected data from secondary and national sources for comparison. A colour coding giving an indication of comparability is provided for the data.

Furthermore comments and inputs were received from the participating countries regarding outstanding issues. As a result Metadata is now available.

Timeliness

The JODI database is expected to be updated regularly. The Timeliness indicates whether submissions were submitted at the expected deadline. Ratings are as follows (over a six months period):

• "good" when 6 submissions received within two months after the end of the reference month
• "fair" when 4 or 5 submissions received
• "less reliable" when less than 4 submissions received

Completeness

Completeness measures the number of expected data points out of the maximum 42 in the JODI questionnaire which are filled in. Ratings are as follows:

• "good" when more than 90% of the data are given for production, stock change/closing and demand
• "fair" when between 60% and 90% of the data are given
• "less reliable" when less than 60% of the data are given

Sustainability

Sustainability is the number of the JODI data submission within a given period. Ratings are as follows (over a six-month period):

• "good" if the 6 questionnaires have been submitted
• "fair" if 4 or 5 questionnaires have been submitted
• "less reliable" when less than 4 questionnaires have been submitted

Comparability Assessment Methodology

The assessment was carried out on different levels:

• Comparability of the JODI data with other sources: monthly data from national and secondary sources has been assessed.
• JODI data have also been compared with annual data (when available) in order to check whether the levels and trend over the years could be confirmed.
• When no other sources were available for comparison with the JODI data, internal consistency and balance check have been carried out.

Examples of internal consistency checks: the sum of all the reported products with reported figure for Total Products is compared. When both, closing and stock changes data have been submitted, the consistency of the reported changes with the calculated ones is compared.

Example of balance check : the JODI questionnaire does not collect full balance information, however some basic checks for reasonableness can be carried out e.g. supply + import - export + stock change should have a relation with demand.

Remark: for IEA /OECD countries, data in the JODI database are the MOS data for all months except data shown for M-1. Comparability for the last month has been derived from comparison with MOS data. This methodology is applied using a rolling 12 month period.

Colour Coding

Sustainability is the number of the JODI data submission within a given period. Ratings are as follows (over a six-month period):

• Blue: A blue background indicates that results of the assessment show reasonable levels of comparability
• Yellow: A yellow background indicates that the metadata should be consulted
• White: A white background indicates that data has not been assessed
• Purple: data under verification