Workflows

Assessment for Suitability of Deposit

When a prospective depositor has made contact with the ADA, the deposit request is assessed by the director or deputy director of the ADA for suitability (see Deposit Appraisal & Collection Policy).

Upload of Data and Documentation

Deposit Shell

Once the deposit has been provisionally accepted, an ADA archivist will set up a deposit shell on the ADA Deposit Dataverse site. The Deposit Dataverse is the first of three instances of Dataverse used in the archival process by the ADA. The three Dataverse installations (Deposit/Test/Production) are isolated from one another, with only the Production Dataverse, the third instance, publicly accessible. See Storage & Integrity on how and where data is stored.

The deposit shell simply looks like an empty version of a dataset on the Production Dataverse. Other secure file sharing solutions are allowed, however, this should be discussed with the ADA first. For security reason do not send data files by email.

Data upload

To the Deposit Dataverse, Data-rights holder (or their authorised data depositor) uploads the data files and supporting documentation (e.g. questionnaires, technical reports). The ADA archivist will prompt the depositor to fill in the DDI metadata fields in the Deposit Dataverse as well. The ADA will contact the depositor if further information is needed to create complete documentation for their data (See Quality Assurance for more details).

Data processing

From here, ADAPT assigns each draft deposit a unique six-digit ADA Identification (ADAID) number. The complete draft submission of the data is then stored by ADAPT to an archive folder structure with the same unique ADAID number hosted by the National Computational Infrastructure (NCI) as the Submission Information Package (SIP). Within the SIP, the initial draft deposit remains unchanged so that a complete end-to-end audit trail can be maintained at all times. The archivists use a copy of the data to perform updates and amendments to the material as required. The NCI storage and working areas are accessed via a Remote Desktop Service (RDS) that is managed by the NCI.

Trained ADA archivist staff (see Expertise & Guidance) can perform various levels of curation as agreed with the data owner/depositor. The level of curation may depend on the type of dataset (quantitative or qualitative) deposited, the importance of the dataset and its confidentiality (government of longitudinal data), or other factors as determined in consultation with the data depositor. For all types of data the ADA archivist will check for privacy risks and liaise with the depositor about how best to mitigate them. For qualitative data, this could mean that only the transcript of an interview is published (without the recording) or even just an interview summary, depending in the level of sensitivity in the data. For quantitative data, variables with a particularly high re-identification risk can be relegated to a separate file, which will be published with additional access conditions. All data will be checked for re-usability, e.g. clear labels. In addition to that, tabular data is exported to SPSS, STATA, SAS und CSV formats for publication. All proposed changes to the data are captured in a Processing Report for the deposit. This report is sent to the Data Owner’s for approval prior to the changes being made. All agreed changes are recorded in the curation syntax (SPSS or R).

Review and Publication

Once all agreed changes to the data and metadata have been made, the ADA archivist will set up a preview page on the second instance of Dataverse, the Test Dataverse, that reflects the current state of the metadata and files. The data owner/depositor will be provided with a private URL to review the data. Once the data owner/depositor has approved the preview version, it is copied to the third instance of Dataverse, the Production Dataverse. On this instance of Dataverse, the data is published and can be requested by external users. For all datasets, the metadata is freely available for viewing. Data itself can be downloaded subject to the Data Owner’s licensing agreement and the user fulfilment of the data access criteria (see Setting Access Conditions. This usually involves providing a verified email address and answering a number of guestbook questions. The access criteria for each dataset are formalised as Business Rules and are updated and stored on the ADA’s internal wiki site (These pages are not publicly available).

Changes or updates to the data files of an already published deposit are treated like a new deposit, i.e. a new SIP, AIP and DIP are created. Changes to published datasets are also automatically version controlled through the Dataverse application. Major changes, that is a change to the data, result in a new version release (i.e. Version 0.0 becomes Version 1.0), whilst a minor change such as the addition of metadata results in a sub-version uplift (i.e. from Version 0.0 to Version 0.1), see Provenance and authenticity.

Preservation

After a dataset is successfully published on the ADA Dataverse for access, preservation versions of the DDI metadata are exported using the Dataverse export functionality. The metadata is stored in the preservation directory, along with a copy of the published SPSS data file(s) and SPSS syntax. The Preservation Plan outlines how ADA manages long term preservation of data and metadata for reuse.

Data Curation Process

Submission Information Package

Draft deposits are each assigned a unique six-digit ADA Identification (ADAID) number. The complete draft submission is then saved to an archive folder structure with the same unique ADAID number hosted by the National Computational Infrastructure (NCI) as the Submission Information Package (SIP). Within the SIP, the initial draft deposit remains unchanged so that a complete end-to-end audit trail can be maintained at all times. The archivists uses a copy of the data to perform updates and amendments to the material as required. The NCI storage and working areas are accessed via a Remote Desktop Service (RDS) that is managed by the NCI.

Data Processing

Trained ADA archivist staff can perform various levels of curation as agreed with the data owner/depositor. The level of curation may depend on the type of dataset (quantitative or qualitative) deposited, the importance of the dataset and its confidentiality (government of longitudinal data), or other factors as determined in consultation with the data depositor. All proposed changes to the data are captured in a Processing Report for the deposit. This report is sent to the Data Owner’s for approval prior to the changes being made. All agreed changes are tracked and retraceable in the curation syntax (SPSS or R). The processed data and supporting documentation files are converted to preservation formats suitable for long term storage and are saved in the archive file structure as the Archival Information Package (AIP). The Processing Reports are also retained in the archive and form part of the AIP. Approved changes can also be made to the data, supporting information and metadata, by the Data Owner (or if authorised the data depositor) when the information is still in a draft format in the Deposit Dataverse if required. All copies of syntax and superseded data/documents are also retained in an archival form as part of the AIP.

Review of Data and Metadata

Cross Check

Once all agreed changes to the data and metadata have been made, the ADA archivist will set up a preview page on the second instance of Dataverse, the Test Dataverse, that reflects the current state of the metadata and files. The data owner/depositor will be provided with a private URL to review the data.

License, Terms & Conditions, Access conditions

Before a dataset can be published, the data owner or data rights holder has to sign the license forms, see Rights Management. In these documents to terms & conditions for the data a specified and the access conditions are set out.

Publication

Once the data owner/depositor has approved the preview version, it is copied to the third instance of Dataverse, the Production Dataverse. On this instance of Dataverse, the data is published and can be requested by external users.

Changes made to published datasets are version controlled and stored within the NCI File structure as part of the AIP. Changes to published Datasets are also automatically version controlled through the Dataverse application. Major changes, that is a change to the data, result in a new version release (i.e. Version 0.0 becomes Version 1.0), whilst a minor change such as the addition metadata results in a sub-version uplift (i.e. from Version 0.0 to Version 0.1).

For all datasets, the metadata is freely available for viewing. Data itself can be downloaded subject to the Data Owner’s licensing agreement and the user fulfilment of the data access criteria. This usually involves providing a verified email address and answering a number of guestbook questions. The access criteria for each dataset are formalised as Business Rules and are updated and stored on the ADA’s internal wiki site (These pages are not publicly available).