Awesome Digital Preservation

Carefully curated list of awesome digital preservation resources.

This Awesome List is one a suite of community-owned resources for digital preservation. See digipres.org or the digipres.org discussion forum for more information.

Contributions are welcome. Please add links through pull requests, or create an issue to start a discussion. Please refer to CONTRIBUTING.md for detailed guidance. And if obsolescence claims something awesome, there's always the Archive.

The text of an annual reminder email about these resources is also held here, in reminder.md. This will be sent around various mailings list once per year, ahead of World Digital Preservation Day.

Contents

• Get Started
• Save Digital Stuff Right Now
• Learn About Digital Preservation
• Find Formats
• Experiment with Tools
• Engage Stakeholders
• Become Part Of The Digital Preservation Community
• Store Digital Content
• Create Preservation Metadata
• Find Test Files
• Multi-format Corpora
• Format-specific Corpora
• Building Corpora
• Sourcing test files from web archives
• Sourcing test files
• Find More Tools
• Build Workflows
• Improve The Tools
• Improving Identification
• Improving Characterisation/Metadata Extraction

Get Started

Save Digital Stuff Right Now

Spotted digital data at risk, but don't know who can save it?

• Save web pages via:
  • Internet Archive Nominations
  • archive.is - Also known as archive.today.
  • perma.cc
  • webcitation.org
  • UK Web Archive Site Nomination - Suggest URLs for the UK Web Archive. Note that UKWA is offline at present.
• Alert the Archive Team, and help them save digital stuff

Learn About Digital Preservation

• The Getting Started chapter of the Digital Preservation Handbook is a great place to start.
• The Digital Preservation Handbook Glossary - Introduces a lot of the core terminology.
• For material that describes the broader issues, you can refer to Digital Preservation on Wikipedia, and consider contributing to the Digital Preservation Wikipedia Project.
• Build your roadmap, guided by the NDSA Levels of Digital Preservation
• Use the Digital Preservation Business Case Toolkit to help get funding.
• Understanding your costs can help to plan your preservation work more effectively. The Curation Costs Exchange allows you to compare your costing data with that of many other organisations.
• Learn about Preserving digital Objects with Restricted Resources
• Learn how to Authenticate, Manage, and Preserve Video -- WITNESS trains activists to archive and preserve their video so that human rights abuses cannot be denied or forgotten over time.
• Explore and contribute to the DP Requirements and Solutions wiki
• Brainscape Digital Preservation Flash Cards - See https://github.com/ross-spencer/brainscape-digital-preservation#readme for more information.

Find Formats

We need to understand the file formats of the resources we care for, and the software they depend on.

• Search across format registries
• Find or add formats to the File Formats Wiki
• Understand file format risks (e.g. JP2)
• Game File Format Central (archived version) - Community project documenting over 1300 game related file-formats.
• Just Solve It - File Formats Wiki - Community project documenting a wide-variety of file formats.

If you have good examples of digital resources and their risks, please consider adding them to a test corpus.

Experiment with Tools

There are a lot of tools out there (see the tools section below), but some tools are particularly great for early experimentation. These tools can be used right in your web browser, so you can get started without installing software locally.

Remote Services

These tools are accessed using your browser, and work by sending a copy of your files to a remote server.

• Siegfried - You can use the side bar to upload a file for Siegfried to identify the format.
• Online TrID File Identifier - A web service that identifies files using TrID.

In-Browser Tools

These tools run entirely in your web browser, so no data is sent anywhere.

• Siegfried JS - This runs the Siegfried format identification tool on your files in your browser.
• CyberChef - The Cyber Swiss Army Knife. Capable of running lots of basic data operations on text or files, including computing things like MD5 or SHA hashes.
• warc-analyser - Proof-of-concept that analyses WARC files in your browser. See https://github.com/edsu/warc-analyzer for more information.
• Demystify Lite - This runs Siegfried WASM on your files in your browser and outputs a Demystify formatted report profiling your collection and highlighting files that might require specific attention during appraisal, such as duplicates; and through various preservation activities, such as caring for file names encoded using specific character-encodings.

Engage Stakeholders

• Visual examples of digital preservation challenges, such as graphic corruption, can be incredibly useful in communicating the digital preservation message. That's why we built the Atlas of Digital Damages Gallery and website. Please add your own images of a digital preservation challenge, failed rendering, encoding damage, corrupt data, or visual evidence documenting to the Atlas of Digital Damages.
• Use the POWRR One Pagers to educate stakeholders about the issues.
• Working with your IT department (some responses arising from this question on twitter:
• Backup versus preservation: "I had useful discussions about 'for the long-term' and what issues that might throw up as a starting point."
• "often it comes down to language - avoid the word archiving. means 1 thing to IT & another DP professionals"
• Ten IT skills you need to have to work with digital preservation - written by Dave Thompson
• "Am going to go all Bruce Lee 'adjust to the object'. What drives IT? Delivery, efficiency, user needs etc."

Become Part Of The Digital Preservation Community

Advance digital preservation by pooling our experience, sharing our stories and finding the answers to the big questions.

• Q&A:
• Ask and answer digital preservation questions
• We tried to run a Digital Preservation Stack Exchange, but it didn't work out. The content is available here
• Forums
  • Discussion forums and active blogs provide the opportunity to share informal advice and war stories, get recommendations and discuss the finer points of digital preservation. By sharing both your intentions for digital preservation work and your results, you can ensure your work benefits from a wealth of community experience.
  • Discuss preservation issues on the Digital Curation forum
  • Share war stories on OPF blogs
  • Mastodon - Join these federations with a digital preservation or general GLAM focus:
    • https://digipres.club
    • https://glammr.us
    • https://ausglam.space
    • https://code4lib.social
  • Twitter - Use these lists to find people to follow:
    • https://twitter.com/DavidUnderdown9/lists/digipres1
    • https://twitter.com/An_Old_Hand/lists/digi-pres-rdm
    • https://twitter.com/digipresnews/followers
    • https://twitter.com/NKrabben/lists
  • r/DataHoarder - "We are digital librarians."
  • r/Archiveteam - "Archive Team is a loose collective of rogue archivists, programmers, writers and loudmouths dedicated to saving our digital heritage."
  • Join the Digital POWRR Slack
• Face-to-Face communities/support groups:
  • Digital Cultural Heritage DC
  • Mid Michigan Digital Practitioners
  • XFR Collective: NYC-based media preservation membership organization preserving at-risk and obsolete audiovisual artwork by providing low-cost migration services
• Collaborations (inc. groups that build things together):
  • Association of Moving Image Archivists (AMIA) Open Source Committee on GitHub
  • Zenodo Digital Preservation Community - Building a comprehensive bibliography of publications, presentations, instructions and data sets related to digital preservation.
• Models, Standards & Certification:
  • The Reference Model for an Open Archival Information System (OAIS)
  • CoreTrustSeal - CoreTrustSeal offers to any interested data repository a core level certification based on the Core Trustworthy Data Repositories Requirements.
• Conferences:
  • PASIG
  • iPres
• Membership organizations:
  • DPC (International)
  • OPF (International)
  • NCDD (Netherlands)
  • NDSA (USA)
  • NESTOR (Germany)

Store Digital Content

• A history of storage media
• File system conventions:
• BagIt
  • BagIt - BagIt's page on the File Format Wiki.
  • Unexpected errors in the BagIt area - Detailed advice on how to approach testing your Bags.
• Oxford Common File Layout (OCFL)

Create Preservation Metadata

• The PREMIS Data Dictionary for Preservation Metadata
• Metadata Encoding & Transmission Standard (METS)
• Portland Common Data Model (PCDM)
• Coalition for Content Provenance and Authenticity (C2PA) - A related, emerging industry standard.
  • [WITNESS and th