ADA Public Wiki - User contributions [en-gb]

Main Page

2025-10-22T03:53:14Z

MMitrev: /* ADA Archival Workflow Diagram */

The [https://ada.edu.au/ Australian Data Archive (ADA)] provides a national service for the collection and preservation of digital research data. ADA disseminates this data for secondary analysis by academic researchers and other users.

= Deposit Process Overview =
If you are interested in depositing data with the Australian Data Archive, please write an email to ada@ada.edu.au outlining the topic of your research and the nature of the data, e.g. sensitivity, data type, and quantity of files and size. The ADA will assess each deposit request against a set of criteria, see [[Deposit Appraisal & Collection Policy]]. Once the data has been accepted, the ADA will create a deposit shell on a secure system for the depositor to upload the data, the documentation files and enter metadata (see [[Quick Deposit Guide]] for instructions on what to upload). As part of the curation process, the ADA will view the deposit and make suggestions for changes if necessary (see [[Workflows]] for details). When the curation is complete, the depositor has the opportunity to preview the final version of the deposit on a designated review page. Once the depositor approves the review, the deposit is published on the [https://dataverse.ada.edu.au/|dataverse: ADA dataverse] and will receive a permanent DOI.

[[File:Deposit_graph_4.png|1200px|center]]

==Archival Steps explained once data has been approved for deposit==

===Deposit Shell===
* ADA creates a deposit dataset shell on ADA's DEPOSIT Dataverse whete the depositor/data custodian uploads data, and is able to enter study and file level metadata into the metadata fields (e.g. Title, Author...).
* As part of this process the archivist and depositor(s) discuss the ADA Data License agreement.
* Once the depositor confirms they have completed the file uploads and metadata they inform the archivist that their deposit is ready for ADA's archival processing. This entails the deposit internally being ingested into the archival and storage management system.

===Archival Data Curation Process===
* Archivists check the data for confidentiality and completeness.
* An archive Processing Report is sent to the depositor/data custodian to respond to any recommended changes. The archivist then makes any amendments and uploads the dissemination (publication) version of the data to ADA's test Dataverse.

===Review of data & metadata===
* The archivist provides the depositor/data custodian a link to the test Dataverse to verify that the dataset metadata and data are correct.

===Publication===
* Once approval of the test version has been received from the depositor/data custodian, the data and metadata are migrated to the [https://dataverse.ada.edu.au/ production Dataverse], and the dataset is published for user access, based on the ADA Data License agreement.

===ADA Data License Agreement===
* The archivist will provide the ADA Data License agreement and confirm suitable options with the data custodian throughout the deposit and curation process.

== To get started, go to [https://docs.ada.edu.au/index.php/Quick_Deposit_Guide Quick Deposit Guide]==

For more detailed information on all the steps in the process see [https://docs.ada.edu.au/index.php/Workflows Workflows]

ADA website, please go to https://ada.edu.au/

= ADA Archival Workflow Diagram =
ADA bases its archival workflow on the Open Archival Information System (OAIS) Reference Model (2012).

Diagram:[https://docs.ada.edu.au/images/e/eb/CTS_ADA-NCI_RDS-Storage_V8_2025_10_21_wiki.png ADA Archival Workflow based on OAIS Reference Model].

= Australian Data Archive Overview =

*[[Background Information and Context]]

Organisational Infrastructure

*[[Mission & Scope]]
*[[Rights Management]]
*[[Continuity of Service]]
*[[Legal & Ethical]]
*[[Governance & Resources]]
*[[Expertise & Guidance]]

Digital Object Management

*[[Provenance and authenticity]]
*[[Deposit Appraisal & Collection Policy]]
*[[Preservation plan]]
*[[Quality Assurance]]
*[[Workflows]]
*[[Discovery and Identification]]
*[[Reuse]]
*[[Setting Access Conditions]]

Information Technology & Security

*[[Storage & Integrity]]
*[[Technical Infrastructure]]
*[[Security]]

= Australian Data Archive Projects =

See the [[ADA Projects]] page for outlines in detail ADA's involvement in multiple projects etc

= Superseded page =
* [[ADA Self-Deposit - To Documentation Guides]]
* [[1. ADA Collection Policy Criteria Assessment]]
* [[2. Deposit Preparation]]

File:CTS ADA-NCI RDS-Storage V8 2025 10 21 wiki.png

2025-10-22T03:50:10Z

MMitrev:

Main Page

2025-10-21T04:56:02Z

MMitrev: /* ADA Archival Workflow Diagram */

The [https://ada.edu.au/ Australian Data Archive (ADA)] provides a national service for the collection and preservation of digital research data. ADA disseminates this data for secondary analysis by academic researchers and other users.

= Deposit Process Overview =
If you are interested in depositing data with the Australian Data Archive, please write an email to ada@ada.edu.au outlining the topic of your research and the nature of the data, e.g. sensitivity, data type, and quantity of files and size. The ADA will assess each deposit request against a set of criteria, see [[Deposit Appraisal & Collection Policy]]. Once the data has been accepted, the ADA will create a deposit shell on a secure system for the depositor to upload the data, the documentation files and enter metadata (see [[Quick Deposit Guide]] for instructions on what to upload). As part of the curation process, the ADA will view the deposit and make suggestions for changes if necessary (see [[Workflows]] for details). When the curation is complete, the depositor has the opportunity to preview the final version of the deposit on a designated review page. Once the depositor approves the review, the deposit is published on the [https://dataverse.ada.edu.au/|dataverse: ADA dataverse] and will receive a permanent DOI.

[[File:Deposit_graph_4.png|1200px|center]]

==Archival Steps explained once data has been approved for deposit==

===Deposit Shell===
* ADA creates a deposit dataset shell on ADA's DEPOSIT Dataverse whete the depositor/data custodian uploads data, and is able to enter study and file level metadata into the metadata fields (e.g. Title, Author...).
* As part of this process the archivist and depositor(s) discuss the ADA Data License agreement.
* Once the depositor confirms they have completed the file uploads and metadata they inform the archivist that their deposit is ready for ADA's archival processing. This entails the deposit internally being ingested into the archival and storage management system.

===Archival Data Curation Process===
* Archivists check the data for confidentiality and completeness.
* An archive Processing Report is sent to the depositor/data custodian to respond to any recommended changes. The archivist then makes any amendments and uploads the dissemination (publication) version of the data to ADA's test Dataverse.

===Review of data & metadata===
* The archivist provides the depositor/data custodian a link to the test Dataverse to verify that the dataset metadata and data are correct.

===Publication===
* Once approval of the test version has been received from the depositor/data custodian, the data and metadata are migrated to the [https://dataverse.ada.edu.au/ production Dataverse], and the dataset is published for user access, based on the ADA Data License agreement.

===ADA Data License Agreement===
* The archivist will provide the ADA Data License agreement and confirm suitable options with the data custodian throughout the deposit and curation process.

== To get started, go to [https://docs.ada.edu.au/index.php/Quick_Deposit_Guide Quick Deposit Guide]==

For more detailed information on all the steps in the process see [https://docs.ada.edu.au/index.php/Workflows Workflows]

ADA website, please go to https://ada.edu.au/

= ADA Archival Workflow Diagram =
ADA bases its archival workflow on the Open Archival Information System (OAIS) Reference Model (2012).

Diagram:[https://docs.ada.edu.au/images/8/84/CTS_ADA-NCI_RDS-Storage_V8_2025_10_21_wiki.jpeg ADA Archival Workflow based on OAIS Reference Model].

= Australian Data Archive Overview =

*[[Background Information and Context]]

Organisational Infrastructure

*[[Mission & Scope]]
*[[Rights Management]]
*[[Continuity of Service]]
*[[Legal & Ethical]]
*[[Governance & Resources]]
*[[Expertise & Guidance]]

Digital Object Management

*[[Provenance and authenticity]]
*[[Deposit Appraisal & Collection Policy]]
*[[Preservation plan]]
*[[Quality Assurance]]
*[[Workflows]]
*[[Discovery and Identification]]
*[[Reuse]]
*[[Setting Access Conditions]]

Information Technology & Security

*[[Storage & Integrity]]
*[[Technical Infrastructure]]
*[[Security]]

= Australian Data Archive Projects =

See the [[ADA Projects]] page for outlines in detail ADA's involvement in multiple projects etc

= Superseded page =
* [[ADA Self-Deposit - To Documentation Guides]]
* [[1. ADA Collection Policy Criteria Assessment]]
* [[2. Deposit Preparation]]

File:CTS ADA-NCI RDS-Storage V8 2025 10 21 wiki.jpeg

2025-10-21T04:53:46Z

MMitrev:

ADAPT

2025-10-20T05:24:33Z

MMitrev:

ADA Deposit and Preservation Tool (ADAPT) is a web-based tool developed by the Australian Data Archive to ensure that data and metadata in the Archive are programmatically moved between Dataverse instances and ADA’s archival storage.

<blockquote>
See ADAPT's implementation within ADA technical architecture - click on the link to the diagram [https://docs.ada.edu.au/index.php/Main_Page#ADA_Archival_Workflow_Diagram ADA Archival Workflow based on OAIS Reference Model].
</blockquote>

Prior to version 3.0.0 of ADAPT, the scope of the application was strictly only for handling the programmatic moving of data and metadata. The major downside to these prior versions was that the application relied upon the archivists to adhere to the archival workflow as the business rules were not strictly enforced by the application. In addition to this, the state of the dataset was tracked externally to the application with much of the references to dataverse instances, assigned archivists, depositor contacts and related internal ticket numbers having to all be maintained manually. In an effort to amend these manual processes and to introduce enforceable business rules as steps which correlate to the OAIS reference model, ADA completed a major rewrite of the application in Mar 4 2025.

With the focus of ADAPT version 3 being for the application to be a direct representation of the OAIS model, the team developed application enforced steps which each dataset must follow when being processed by the archive. These steps must be followed according to the application's defined order and cannot be skipped, or ignored, thus ensuring adherence to the model and reducing any risks. For the application to know what processing options are available for the archivist to use on a given dataset, this new version extends the datasets by implementing them as objects with a dynamic reference to where within the OAIS reference model the dataset is up to. By tying the processing options with the dataset's state ADA can ensure that data and metadata is consistently and only programmatically moved between instances and storage when appropriate. The additional benefit of this is that it allows for tracking the progression of the datasets in an automated and verifiable process. Furthermore, all of the aforementioned references to dataverse instances, assigned archivists, depositor contacts and related internal ticket numbers are all tracked in application too. To ensure all this information is secure, the ADAPT implements OAuth2 login to ensure only approved ADA staff are able to view and manipulate the datasets.

== Web Ontology Language (OWL) ==

The Web Ontology Language (OWL) is a W3C-recommended standard designed as a foundational language for the Semantic Web, which aims to make internet data machine-readable by providing a formal system for knowledge representation. At its core, OWL allows for the explicit, formal specification of an ontology, defining the terminology and the complex relationships within a specific domain of knowledge. For ADAPT, OWL is leveraged not just for static modeling but also for dynamic auditing and version control: the ontology includes specialized classes and properties to define events, actors, and timestamps, effectively creating a formal change log schema. Any modification to the underlying datasets—such as adding, deleting, or altering entities—is automatically captured as a new set of RDF triples conforming to this change log schema and is then persisted to an rdflib graph. This technique ensures that a formal, machine-readable history of all dataset changes is maintained, utilizing the semantic rigor of OWL to provide a structured, queryable audit trail that is then human readable for each dataset in ADAPT.

== Roles ==

{| class=wikitable
|+ style="text-align: left;" | ADAPT's internal roles and their equivalent permissions within ADAPT
|-
! scope=col | Role !! scope=col | Permissions
|-
| scope=row | User
| Process a dataset depending on available archival rules.
|-
| scope=row | Admin
| Manage users, and all User powers.
|-
| scope=row | Super
| Manage instances, manage archival steps, and all Admin & User powers.
|}

== Software Architecture ==

* [https://www.postgresql.org/ PostgreSQL] Database
* [https://www.python.org/ Python] Backend
** Web framework: [https://fastapi.tiangolo.com/ FastAPI]
** Database toolkit & ORM: [https://sqlmodel.tiangolo.com/ SQLModel]
* [https://react.dev/ React] Frontend
** Bundler: [https://vite.dev/ Vite]
** Component library: [https://ui.shadcn.com/ shadcn]
* [https://traefik.io/traefik Traefik] Proxy

ADAPT

2025-10-20T05:23:58Z

MMitrev:

== Background ==

ADA Deposit and Preservation Tool (ADAPT) is a web-based tool developed by the Australian Data Archive to ensure that data and metadata in the Archive are programmatically moved between Dataverse instances and ADA’s archival storage.

<blockquote>
See ADAPT's implementation within ADA technical architecture - click on the link to the diagram [https://docs.ada.edu.au/index.php/Main_Page#ADA_Archival_Workflow_Diagram ADA Archival Workflow based on OAIS Reference Model].
</blockquote>

Prior to version 3.0.0 of ADAPT, the scope of the application was strictly only for handling the programmatic moving of data and metadata. The major downside to these prior versions was that the application relied upon the archivists to adhere to the archival workflow as the business rules were not strictly enforced by the application. In addition to this, the state of the dataset was tracked externally to the application with much of the references to dataverse instances, assigned archivists, depositor contacts and related internal ticket numbers having to all be maintained manually. In an effort to amend these manual processes and to introduce enforceable business rules as steps which correlate to the OAIS reference model, ADA completed a major rewrite of the application in Mar 4 2025.

With the focus of ADAPT version 3 being for the application to be a direct representation of the OAIS model, the team developed application enforced steps which each dataset must follow when being processed by the archive. These steps must be followed according to the application's defined order and cannot be skipped, or ignored, thus ensuring adherence to the model and reducing any risks. For the application to know what processing options are available for the archivist to use on a given dataset, this new version extends the datasets by implementing them as objects with a dynamic reference to where within the OAIS reference model the dataset is up to. By tying the processing options with the dataset's state ADA can ensure that data and metadata is consistently and only programmatically moved between instances and storage when appropriate. The additional benefit of this is that it allows for tracking the progression of the datasets in an automated and verifiable process. Furthermore, all of the aforementioned references to dataverse instances, assigned archivists, depositor contacts and related internal ticket numbers are all tracked in application too. To ensure all this information is secure, the ADAPT implements OAuth2 login to ensure only approved ADA staff are able to view and manipulate the datasets.

== Web Ontology Language (OWL) ==

The Web Ontology Language (OWL) is a W3C-recommended standard designed as a foundational language for the Semantic Web, which aims to make internet data machine-readable by providing a formal system for knowledge representation. At its core, OWL allows for the explicit, formal specification of an ontology, defining the terminology and the complex relationships within a specific domain of knowledge. For ADAPT, OWL is leveraged not just for static modeling but also for dynamic auditing and version control: the ontology includes specialized classes and properties to define events, actors, and timestamps, effectively creating a formal change log schema. Any modification to the underlying datasets—such as adding, deleting, or altering entities—is automatically captured as a new set of RDF triples conforming to this change log schema and is then persisted to an rdflib graph. This technique ensures that a formal, machine-readable history of all dataset changes is maintained, utilizing the semantic rigor of OWL to provide a structured, queryable audit trail that is then human readable for each dataset in ADAPT.

== Roles ==

{| class=wikitable
|+ style="text-align: left;" | ADAPT's internal roles and their equivalent permissions within ADAPT
|-
! scope=col | Role !! scope=col | Permissions
|-
| scope=row | User
| Process a dataset depending on available archival rules.
|-
| scope=row | Admin
| Manage users, and all User powers.
|-
| scope=row | Super
| Manage instances, manage archival steps, and all Admin & User powers.
|}

== Software Architecture ==

* [https://www.postgresql.org/ PostgreSQL] Database
* [https://www.python.org/ Python] Backend
** Web framework: [https://fastapi.tiangolo.com/ FastAPI]
** Database toolkit & ORM: [https://sqlmodel.tiangolo.com/ SQLModel]
* [https://react.dev/ React] Frontend
** Bundler: [https://vite.dev/ Vite]
** Component library: [https://ui.shadcn.com/ shadcn]
* [https://traefik.io/traefik Traefik] Proxy