Storage & Integrity - Revision history

Mmcgale at 01:42, 4 December 2025

2025-12-04T01:42:03Z

← Older revision		Revision as of 01:42, 4 December 2025
Line 42:		Line 42:
	::::* Generated AIP Data		::::* Generated AIP Data

	::c. ~~dataverse-test.ada.edu~~ [95]		::c. Dataverse TEST [95]
	::::* DDI Metadata (copied from deposit.ada.edu.au)		::::* DDI Metadata (copied from deposit.ada.edu.au)
	::::* Data (from 2b.)		::::* Data (from 2b.)

Mmcgale at 01:04, 4 December 2025

2025-12-04T01:04:18Z

Show changes

JMcDougall at 02:05, 18 February 2025

2025-02-18T02:05:46Z

← Older revision		Revision as of 02:05, 18 February 2025
Line 1:		Line 1:
	The ADA archival workflow [34] outlines processes to manage the integrity of the data and metadata flow through the Archive. [https://docs.ada.edu.au/images/f/f4/CTS_2024_ADA%2BNCI_-_RDS%2BStorage_solutions_JM_%28V4%29_2024-09-06_wiki.jpeg The ADA Workflow and Storage Diagram] [47] reflects the distinct deposit, ingest, curation, access, and storage locations for each of the archival phases.		The ADA archival workflow [34] outlines processes to manage the integrity of the data and metadata flow through the Archive. [https://docs.ada.edu.au/images/f/f4/CTS_2024_ADA%2BNCI_-_RDS%2BStorage_solutions_JM_%28V4%29_2024-09-06_wiki.jpeg The ADA Archival Workflow and Storage Diagram based on OAIS Reference Model] [47] reflects the distinct deposit, ingest, curation, access, and storage locations for each of the archival phases.

	==ADA Archive Training ==		==ADA Archive Training ==

JMcDougall at 22:59, 28 October 2024

2024-10-28T22:59:58Z

← Older revision		Revision as of 22:59, 28 October 2024
Line 1:		Line 1:
	The ADA archival workflow [34] outlines processes to manage the integrity of the data and metadata flow through the Archive. The ADA Workflow and Storage Diagram [47] reflects the distinct deposit, ingest, curation, access, and storage locations for each of the archival phases.		The ADA archival workflow [34] outlines processes to manage the integrity of the data and metadata flow through the Archive. [https://docs.ada.edu.au/images/f/f4/CTS_2024_ADA%2BNCI_-_RDS%2BStorage_solutions_JM_%28V4%29_2024-09-06_wiki.jpeg The ADA Workflow and Storage Diagram] [47] reflects the distinct deposit, ingest, curation, access, and storage locations for each of the archival phases.

	==ADA Archive Training ==		==ADA Archive Training ==

RPerry at 04:14, 13 September 2024

2024-09-13T04:14:12Z

Show changes

JMcDougall: /* Data & Storage Management */

2024-09-09T00:29:37Z

Data & Storage Management

← Older revision		Revision as of 00:29, 9 September 2024
Line 13:		Line 13:
	==Data & Storage Management==		==Data & Storage Management==
	The Dataverse software [49] supports reporting, data management and auditing through user accounts for access, authentication, and permissions; to edit, upload, and download data.		The Dataverse software [49] supports reporting, data management and auditing through user accounts for access, authentication, and permissions; to edit, upload, and download data.
	ADA previously developed the ADA Data Processing Tool (ADAPT) [6] based on the OAIS Reference Model [42]. ADAPT enables archivists to programmatically manage movement of data and metadata between Dataverse instances and archival storage to manage data integrity.		ADA previously developed the ADA Data Processing Tool (ADAPT) [6] based on the OAIS Reference Model [42]. ADAPT enables archivists to programmatically manage movement of data and metadata between Dataverse instances and archival storage to manage data integrity:
	* Each user actioned ADAPT function creates or appends to a log using a standard provenance ontology (PROV-O) [3]. The log is stored as part of the AIP to support auditing of archival activities actioned through ADAPT, if required.		* Each user actioned ADAPT function creates or appends to a log using a standard provenance ontology (PROV-O) [3]. The log is stored as part of the AIP to support auditing of archival activities actioned through ADAPT, if required.
	ADAPT is moving towards its third version as ADA continues to develop strategies to minimise the risk of manual data and metadata management versus building management into a software application.		ADAPT is moving towards its third version as ADA continues to develop strategies to minimise the risk of manual data and metadata management versus building management into a software application.
	Data and metadata are stored with each Dataverse instance, and versioned at publication of a dataset. ADAPT is used to move data and metadata between the Dataverse instances as required at each archival phase.		Data and metadata are stored with each Dataverse instance, and versioned at publication of a dataset. ADAPT is used to move data and metadata between the Dataverse instances as required at each archival phase.
	Dataverse exports metadata in a number of formats. The JSON export format of the DDI metadata also includes Dataverse system metadata such as fixity checks on uploaded data files (checksum MD5). The JSON metadata formats are exported by archivists during the Ingest phase and Publish phase. The JSON export is copied to archival storage for preservation of the original (SIP) metadata and data files, and the published (DIP) metadata and data files, to ensure that the integrity of digital objects from deposit to access can be verified against any changes to the data.		Dataverse exports metadata in a number of formats. The JSON export format of the DDI metadata also includes Dataverse system metadata such as fixity checks on uploaded data files (checksum MD5). The JSON metadata formats are exported by archivists during the Ingest phase and Publish phase. The JSON export is copied to archival storage for preservation of the original (SIP) metadata and data files, and the published (DIP) metadata and data files, to ensure that the integrity of digital objects from deposit to access can be verified against any changes to the data.
	The archivist Data Curation Process [34] encapsulates data processing, including superceded and new versions of data, in conjunction with support from Dataverse software versioning control.		The archivist Data Curation Process [34] encapsulates data processing, including superceded and new versions of data, in conjunction with support from Dataverse software versioning control.

	==The Repository's Strategy For Multiple Copies ==		==The Repository's Strategy For Multiple Copies ==

JMcDougall at 00:27, 9 September 2024

2024-09-09T00:27:37Z

← Older revision		Revision as of 00:27, 9 September 2024
Line 1:		Line 1:
	The ADA archival workflow [34] outlines processes to manage the integrity of the data and metadata flow through the Archive. The [https://~~scholar~~.~~harvard~~.edu/~~mercecrosas~~/~~presentations~~/~~fair~~-~~guiding~~-~~principles~~-~~implementation-dataverse~~ ADA Workflow and Storage Diagram] [47] reflects the distinct deposit, ingest, curation, access, and storage locations for each of the archival phases.		The ADA archival workflow [34] outlines processes to manage the integrity of the data and metadata flow through the Archive. The [https://docs.ada.edu.au/images/f/f4/CTS_2024_ADA%2BNCI_-_RDS%2BStorage_solutions_JM_%28V4%29_2024-09-06_wiki.jpeg ADA Workflow and Storage Diagram] [47] reflects the distinct deposit, ingest, curation, access, and storage locations for each of the archival phases.

	==ADA Archive Training ==		==ADA Archive Training ==

JMcDougall at 00:26, 9 September 2024

2024-09-09T00:26:44Z

← Older revision		Revision as of 00:26, 9 September 2024
Line 1:		Line 1:
	The ADA archival workflow [34] outlines processes to manage the integrity of the data and metadata flow through the Archive. The ADA Workflow and Storage Diagram [47] reflects the distinct deposit, ingest, curation, access, and storage locations for each of the archival phases.		The ADA archival workflow [34] outlines processes to manage the integrity of the data and metadata flow through the Archive. The [https://scholar.harvard.edu/mercecrosas/presentations/fair-guiding-principles-implementation-dataverse ADA Workflow and Storage Diagram] [47] reflects the distinct deposit, ingest, curation, access, and storage locations for each of the archival phases.

	==ADA Archive Training ==		==ADA Archive Training ==

JMcDougall at 05:27, 6 September 2024

2024-09-06T05:27:21Z

← Older revision		Revision as of 05:27, 6 September 2024
Line 68:		Line 68:

	For datasets that have been published on ADA’s production Dataverse, ADA does not delete them but rather deaccessions them. This process results in the dataset being labelled as “Deaccessioned” in Dataverse and renders its files accessible only to users with the correct permission levels. The files remain in the Dataverse server /files/zzzzzz/ directory.		For datasets that have been published on ADA’s production Dataverse, ADA does not delete them but rather deaccessions them. This process results in the dataset being labelled as “Deaccessioned” in Dataverse and renders its files accessible only to users with the correct permission levels. The files remain in the Dataverse server /files/zzzzzz/ directory.

			==References==

JMcDougall: /* Risk Management Techniques Used To Inform The Strategy */

2024-09-06T05:26:55Z

Risk Management Techniques Used To Inform The Strategy

← Older revision		Revision as of 05:26, 6 September 2024
Line 53:		Line 53:
	The OAIS model is implemented with data stored for each of the SIP, AIP, DIP NCI <ADAID> directories, and on the Dataverse backend servers. The archive team are trained to strictly follow documented archive procedures to manage the number of copies across the NCI and Dataverse directories. The ADA workflow procedures [34] minimises the probability of data copies becoming unsynchronised.		The OAIS model is implemented with data stored for each of the SIP, AIP, DIP NCI <ADAID> directories, and on the Dataverse backend servers. The archive team are trained to strictly follow documented archive procedures to manage the number of copies across the NCI and Dataverse directories. The ADA workflow procedures [34] minimises the probability of data copies becoming unsynchronised.

	~~PROCEDURES FOR HANDLING~~ & ~~MONITORING DETERIORATION OF STORAGE MEDIA~~		== Procedures for Handling & Monitoring Deterioration of Storage Media ==
	ADA Archival Storage and Dataverse instances [47] are provisioned, hosted and backed up on NCI servers. NCI has procedures for monitoring bit level integrity against the deterioration of their storage media. NCI notifies ADA if there are plans for any necessary upgrades and when those upgrades will take place.		ADA Archival Storage and Dataverse instances [47] are provisioned, hosted and backed up on NCI servers. NCI has procedures for monitoring bit level integrity against the deterioration of their storage media. NCI notifies ADA if there are plans for any necessary upgrades and when those upgrades will take place.

	ADA’s technical team works with NCI to keep server operating systems updated to supported versions:		ADA’s technical team works with NCI to keep server operating systems updated to supported versions:
	- NCI informs the ADA technical team when a Virtual - Machine’s (VM) operating system (OS) is no longer going to be supported.		* NCI informs the ADA technical team when a Virtual - Machine’s (VM) operating system (OS) is no longer going to be supported.
	- NCI provisions new VMs when necessary, and the ADA technical team moves Dataverse installations to these new VMs		* NCI provisions new VMs when necessary, and the ADA technical team moves Dataverse installations to these new VMs.

	~~PROCEDURES TO ENSURE DATA~~ & ~~METADATA ARE ONLY DELETED THROUGH APPROVED PROCESSES~~		== Procedures to Ensure Data & Metadata Are Only Deleted Through Approved Processes==
	Data in the SIP or AIP state can be deleted by the ADA archivist team if directed by depositors, or for legal reasons, with the archiving team maintaining records relating to the request for deletion. Data in the DIP state cannot be deleted.		Data in the SIP or AIP state can be deleted by the ADA archivist team if directed by depositors, or for legal reasons, with the archiving team maintaining records relating to the request for deletion. Data in the DIP state cannot be deleted.