Aviary User Guide

How-to articles

Importing Content in Bulk

Aviary enables you to import Resource Metadata, Media Files, Thumbnails, Indexes, and Transcripts in bulk through three methods: Aviary Package Import, OHMS XML Import, and Archivematica DIP Import. If you already have your descriptive metadata, media file paths, and transcript and/or index data packaged into OHMS, then you’ll find OHMS XML Import to be a quick and easy way to upload. If not, don’t worry — Aviary Package Import lets you upload your content through four or fewer spreadsheets for descriptive metadata, media files, transcripts, and indexes.

⁠

Importing Using the Aviary Package Import⁠

⁠

Prepare your content (media files, thumbnail images, indexes, transcripts)⁠

⁠

Prepare your metadata import CSV files (resource, media, index, transcript)⁠

⁠

Resource CSV file⁠

⁠

Media CSV⁠

⁠

Transcript CSV⁠

⁠

Index CSV⁠

⁠

Import your files⁠

⁠

Remove and Undo Imports⁠

⁠

OHMS XML Import⁠

⁠

Adding a New Import - OHMS XML⁠

⁠

Archivematica DIP Import⁠

⁠

File Size Note:

The maximum file size for any files (e.g., .zip, or individual DIP files) uploaded to Aviary during a bulk import is 50GB. Also note that uploading media files as ZIP or DIP files requires stable network connections with sufficient bandwidth. Considerations about the size of these packages should be made with an understanding of your individual environment.

⁠

Importing Using the Aviary Package Import

Aviary allows you to import multiple media files at once by providing a ZIP of media files from your local computer, providing URLs for media on the web, or providing HTML media embed codes. You can also import any associated Thumbnail Image, Transcript, or Index files, if available, by including them in the ZIP of media files. You will need to prepare CSV files as outlined below to import any associated metadata and to tell Aviary where to find your media files, Indexes, and Transcripts. On import, Aviary will create new Resources for each row in the Resource CSV and will then associate any connected media files to each Resource. It also optionally allows you to associate any number of Thumbnails, Transcripts, and/or Index files with each media file.

Sample Aviary package can be downloaded here:

aviary-sample-bulk-import-package_2022-12-16.zip

23.3 MB

⁠

Prepare your content (media files, thumbnail images, indexes, transcripts)

Aviary accepts the following file types:

Audio: mp3, m4a, ogg, wav, wma

Video: mp4, m4v, ogv, webm, wmv

360 Video: mp4

Streaming and Embed Codes: YouTube, Vimeo, SoundCloud, Avalon Media Player, m3u8 media streams

Thumbnail: jpg, png

Transcript: .txt, .doc/.docx, .vtt, .xml (OHMS)

Index: .vtt, .xml (OHMS)

If you are importing files from your local environment, compress them into a single ZIP file. (You may find it helpful to organize your files into separate subdirectories for media, transcripts, and indexes before zipping them all into a single file.) Note: your ZIP file cannot be larger than 50GB in total size. Also note that uploading media files as ZIP or DIP files requires stable network connections with sufficient bandwidth. Considerations about the size of these packages should be made with an understanding of your individual environment.

If you are importing media from the Web using URLs or embed codes, follow the instructions below under ‘Media CSV file.’

Aviary supports m3u8 streams from a Local Media Server. Use the "Embed Code" option to add a m3u8 stream to a resource, and please provide the direct URL to the m3u8 stream.

Aviary also supports 360 video in an MP4 (h.264) format. If you are providing such a file you will have an option to select "360 Video" to ensure that Aviary's media player responds appropriately to the video file. If you are importing, you can add a column to your Media.csv (see below) file called "360 Video" and add the value "yes" for any files that need to be 360 enabled.

Prepare your metadata import CSV files (resource, media, index, transcript)

Bulk upload using CSV files requires all data to be formatted following the Aviary metadata template (provided as a download from the Imports page. If no resources exist yet, at minimum, you will need to create a Resource CSV file describing your resources. This minimum would create resources with metadata, but no content. You may also include CSV files describing their corresponding media files, thumbnail images, transcript files and index files. Media, Transcript, and Index files are all connected to the Resource through the use of user keys (described in more detail below). These keys are used only for the import process, so they need only be unique for each import. Aviary does not store these temporary keys.

Resource CSV file

Using the template package — provided as a download from the Imports page — open the Aviary-resource-csv-template.csv file and enter metadata for your resources, one resource per row. Leave the “aviary ID” blank for now. Aviary IDs will be generated automatically by the system on import.

The order of the columns in the CSV is irrelevant but the Field Names must match to existing fields in the system. It is also possible to add Custom Fields to this CSV by adding column headers for new metadata fields. Column headers that currently don't exist in this particular Collection will be created as new Custom Fields by Aviary upon import. Fields that already exist (including previously created custom fields) will be populated based on the metadata provided in the CSV.

Important Note

Note that while the Aviary Package Import template contains all standard resource metadata fields, it is not necessary to include all columns in the Resource CSV (only the required fields listed below). You can delete any columns for which you do not have data.

Below are the standard column headers that Aviary expects in the Resource.csv file:

aviary ID: [required] Leave blank.

Resource User Key: [required] Enter an identifier of your choice (unique to this CSV file) to identify the resource. In the Media File CSV you will use this to connect your media files to the resource. Note: Please only use numeric values as your key values; no special characters, no alpha characters, no spaces.

Title: [required] The title of your resource.

Public: [required] Enter ‘yes’ (Public) or ‘no’ (Restricted) to let Aviary know how broadly to share the resource. If you would like no one to know this resource exists, leave this field blank to make the resource Private.

Featured: [required] Enter ‘yes’ or ‘no’ to show this resource in the Featured lane of your organization’s homepage and on the collection homepage. If this resource is also “public”, making it featured will make it a candidate to be found on the main Aviary homepage, too.

Description: A description of the resource. You may add a label to your description metadata by including the label text at the beginning of the entry and separating it from the main description text with two successive semicolons, e.g., "Scope Note;; This is a scope note that describes my Resource." Enter multiple descriptions separated by the pipe character, e.g., "Scope Note;; This is a scope note that describes my Resource.| Summary;; This is a summary of the contents of the resource.")

Date: A date or dates associated with the resource in YYYY-MM-DD format (e.g., ‘2009-02-13’). You may specify a date type by entering it in front of the date, followed by two successive semicolons(ex. ‘created;; 2009-02-13’). Enter multiple dates separated by the pipe character (e.g., ‘created;; 2009-02-13| modified;; ‘2015-03-09’).

Agent: People or organizations associated with the resource. You may specify a role for each agent by entering it in front of the name, followed by two successive semicolons (e.g., 'producer;; Michelle Obama’). Enter multiple agents separated by the pipe character (e.g., ‘interviewer;; Studs Terkel| interviewee;; Muhammad Ali’).

Coverage: Terms related to the geographic or temporal nature of the resource. Specify the type of coverage (e.g., ‘spatial’ or ‘temporal’) followed by two successive semicolons before each term. Separate multiple terms with the pipe character (e.g., ‘spatial;; Kentucky| temporal;; 20th century’).

Language: The language(s) of the resource. You specify use (e.g., ‘primary,’ ‘secondary,’ etc.) before the language followed by two successive semicolons. Separate multiple languages with the pipe character (e.g., ‘primary;; Chinese| secondary;; English’).

Format: A free-text description of the format of the audio or video. Separate multiple formats with the pipe character (e.g., ‘videocassettes (3/4" U-matic)| audiocassettes’)

Type: The category of the resource. Separate multiple types with the pipe character (e.g., ‘oral history| interview’).

Subject: Terms about the resource. You may specify a type of subject before the term followed by two successive semicolons (e.g., ‘topical;; Owls’). Separate multiple subject terms with the pipe character (e.g., ‘Basketball| Coyotes| geographic;; Madison, Wi.’).

Identifier: An external identifier associated with the resource. Specify the type of identifier before the value and followed by two successive semicolons (e.g., ‘Catalog number;; 38496736’). Separate multiple identifiers with the pipe character (e.g., ‘LCCN;; 2018601597| Barcode;; 786936836554’).

Custom Unique Identifier: Aside from standard Aviary URLs, Aviary supports the ability for organizations to provide custom unique identifiers for resources in order to support predictable Aviary URLs that do not require knowledge of internal Aviary identifiers. These will be usable in constructing URLs to access your content. Each unique identifier must be unique across all resources in your Aviary site. Using this unique identifier for your resource, you can construct a direct URL as follows: https://[yourSubdomain].aviaryplatform.com/c/[custom-unique-identifier]. Note: Because this value becomes part of a URL, Aviary does not support the use of the following special characters: \ " ! * ' ( ) ; : @ & = + $ , / ? % # [ ].

Relation: A relationship a resource has with other resources or constituent parts. You may specify the type of relationship before the value followed by two successive semicolons (e.g., ‘part of;; 2017 National Book Festival’). Separate multiple relations with the pipe character.

Source: The work, either print or electronic, from which the resource is delivered.

Publisher: The publisher of the resource. Separate multiple publishers with the pipe character.

Rights Statement: A free-text statement describing copyright status of the resource. Separate multiple statements with the pipe character.

Keyword: Uncontrolled access terms for the resource. Separate multiple keywords with the pipe character.

Preferred Citation: The preferred citation for the resource.

Source Metadata URI: A link to the source metadata for the resource.

Resource metadata can include a limited set of HTML tags within the metadata fields. The allowable tags are: <b>, <i>, <em>, <strong>, <sub>, <sup>, <p>, <br>, <a href>

⁠

Media CSV

For each Resource listed in the resource CSV file, you will describe the associated media files, one per row, in a separate CSV found in the same download package called Aviary-media-csv-template.csv. Leave the “aviary ID” blank for now. Aviary IDs will be generated automatically by the system on import.

Below are the standard column headers that Aviary expects in the Media.csv file.

Media User Key: [required] Enter an identifier of your choice unique to this CSV file. If you are importing associated transcript or index files, you will use this key in other CSV files to connect a transcript and/or index to this file. Note: Please only use numeric values as your key values; no special characters, no alpha characters, no spaces.

Resource User Key: [required] Enter the Resource User Key you specified in the resource CSV to connect this file to the resource in Aviary. Note: Please only use numeric values as your key values; no special characters, no alpha characters, no spaces.

Public: [required] Enter ‘yes’ or ‘no’ to let Aviary know how broadly to share the media file.

In order to allow you to add more than one video to a given Resource, you will specify the order in which your files appear within the Resource on Aviary with a sequence number:

Sequence #: [required] Enter the order in which the file should appear on the Resource’s Aviary page. If only one file is associated with the resource, enter ‘1’.

And, then, select one of the following column headers for each row, depending on your file import method:

Path: If you are uploading files from your local environment, enter the path to the unzipped media file, relative to the root directory of the zip file (ex. /aviary-sample-bulk-import-package/media/party.mp4).

URL: If you are importing files from an external location, enter the URL for the media file (ex.

https://ia800209.us.archive.org/24/items/WildlifeSampleVideo/Wildlife.mp4⁠

). Make sure this links to the media file itself and not the webpage.

Note: If you want to use Dropbox files as link uploads, right click to get the link to the file, replace "www" with "dl" and delete the "?dl=0" at the end.

Embed Code: If you are importing the file from an embed code or from a Local Media Server, enter the code (ex. ‘<iframe width="560" height="315" src="

https://www.youtube.com/embed/RTfbOR16AR0⁠

" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>’

Embed Source: Enter the source that generated the embed code (i.e., ‘Youtube’, 'Local Media Server', 'Avalon', 'Vimeo', 'SoundCloud')

Target Domain: if using 'Avalon' embed source, be sure to add this column to enter your target domain.

If you want to manually assign thumbnails to your media files, you can include them in the import package and you can point to them by including one of the following two columns in your media CSV file:

Media Thumbnail URL: If you are importing files from an external location, enter the URL for the thumbnail file (ex.

https://d2hx64tshfdz71.cloudfront.net/organizations/logo_images/000/000/007/original/AVP_logo.png⁠

). Make sure this links to the thumbnail image file itself and not the webpage.

Media Thumbnail Path: If you are uploading files from your local environment, enter the path to the unzipped thumbnail image file, relative to the root directory of the zip file (ex. /aviary-sample-bulk-import-package/thumbnails/thumbnail01.png).

Finally, if you are providing any 360 video files (currently Aviary only supports MP4/h.264 360 video) you can add a column to your Media.csv (see below) file called "360 Video" and add the value "yes" for any files that need to be 360 video.

360 Video: If you are providing a 360 video, please enter 'yes' in this column. For all non-360 videos, you can leave the value blank or enter 'no'. This column is optional.

⁠

Transcript CSV

Aviary currently accepts transcript files in OHMS XML, WebVTT or plain txt format. If you are importing transcripts, you will describe each transcript file associated with a media file, one per row, in the Transcript CSV. (Currently, transcripts may only be bulk imported at the same time as Resource and Media file import. Bulk import of transcripts and indexes to existing Aviary resources will be available in a future release.)

aviary ID: Leave blank

Media User Key: [required] Enter the Media User Key you specified in the Media CSV to connect this transcript to the media file in Aviary.

Public: Enter ‘yes’ or ‘no’ to let Aviary know how broadly to share the transcript. If no value is specified, visibility will default to ‘Restricted’’

Sequence #: [required] Enter the order in which the transcript should appear with the file on the resource’s Aviary page. If only one transcript is associated with the resource, enter ‘1’.

Language: [required] Enter the language(s) of the transcript. Separate multiple languages with the pipe character.

Title: Enter a title for your transcript. This title will differentiate multiple transcripts for a single file and appear to your users in the dropdown menu under the Transcript tab on the Aviary resource page.

Description: Enter a description for your transcript. This description will display in the transcript window just above your transcript display so that users can see it.

Path: [required] Enter the path to the unzipped transcript file, relative to the root directory of the zip file (ex. /aviary-sample-bulk-import-package/transcripts/wildlife.txt.xml).

Optionally, if you are providing WebVTT transcripts or captions, you can add the following two columns if you want to ignore the WebVTT title entry for each timecode element, or if you want to designate the WebVTT data to also be used as a Closed Caption.

Captions: Enter 'yes' in this field if you would like to set your WebVTT transcript to be enabled as a Closed Caption simultaneously.

Ignore Title: Enter 'yes' in this field if you would like Aviary to ignore the title entry values for each timecode element in the WebVTT.

⁠

Index CSV

Aviary currently accepts index files in OHMS XML or WebVTT format. If you are importing indexes, you will describe each index file associated with a media file, one per row, in the Index CSV. (Currently, index files may only be bulk imported at the same time as Resource and Media file import. Bulk import of transcripts and indexes to existing Aviary resources will be available in a future release.)

aviary ID: Leave blank

Media User Key: [required] Enter the Media User Key you specified in the Media CSV to connect this index to the media file in Aviary.

Public: Enter ‘yes’ or ‘no’ to let Aviary know how broadly to share the index. If no value is specified, visibility will default to ‘Restricted’’

Sequence #: [required] Enter the order in which the index should appear with the file on the resource’s Aviary page. If only one index is associated with the resource, enter ‘1’.

Language: [required] Enter the language(s) of the transcript. Separate multiple languages with the pipe character.

Title: Enter a title for your index. This title will differentiate multiple indexes for a single file and will appear to your users in the dropdown menu under the Index tab on the Aviary resource page.

Description: Enter a description for your index. This description will display in the index window just above your index display so that users can see it.

Path: [required] Enter the path to the unzipped transcript file, relative to the root directory of the zip file (ex. /aviary-sample-bulk-import-package/transcripts/wildlife.txt.xml).

⁠

Import your files

On the Imports page, click the “New Import” button.

On the “Import Resources” page, select “Aviary Package Import” and then browse and select the CSV files you prepared. If you are importing media files from your local environment, browse and select the zip file you created.

Create a new collection or select an existing collection where the new resources will be created.

If Media, Transcript or Index CSV or a combination of them is being uploaded without a Resource CSV, the files will get attached to the respective Resources irrespective of what Collection the Resource belongs to.

Click “Add to Queue” and you will then be redirected to the Manage Imports page, where you should see your import awaiting action. Here you can preview the first resource of your import to check everything looks good to go.

When you’re ready to import, click “Start Import” to start your import. This may take quite a while.

If importing media files from your local environment, you must stay on this page until the processing completes.

If importing media files via URL or Embed Code, the import processing will run in the background, so it’s okay to move freely about the site while you’re waiting.

If your import was successful, the status for your import on the Manage Imports page will say “Complete.” (You may have to refresh the page to see the updated status, if you haven’t navigated away after starting the import.) You should now see your imported resources on the Resources page.

⁠

Remove and Undo Imports

If you noticed any errors while previewing your import, you can remove the import from the queue by clicking on "Remove". It is also possible to Undo a completed import. When doing so, all Resources and Collections created during the import will be deleted from Aviary. Note that if you are making changes to your import files, you will need to remove the old import (with the “Remove” button) and create a new import for the changed files.

Note about CSV formatting and common errors with importing:

We see common errors when CSVs have an extra three hidden bytes at the beginning of the file (see screenshot below showing a hex editor view).

⁠

Screen Shot 2021-08-17 at 8.53.10 AM.png

110.1 kB

⁠

A couple of points about the issue:

1. When you save a file in excel as "UTF-8" CSV, excel adds these three byte marks to the start of the file. It's normal. Aviary should handle these bytes better during import, but we're not actually handling them properly at this point and they get added into the first column label and then Aviary doesn't recognize that label and so the import fails. We are going to be adding improved support for ignoring these bytes when they are present in the future but it may be a few weeks before we get it into the pipeline and into production.

2. Unless you have to, don't save the file as UTF-8 CSV, just save as regular CSV files for the time being. If you have to save as UTF-8, then you can easily open the CSV in a hex editor and delete the first three bytes (as seen in the screenshot) and save the file and then Aviary will not have issues with it.

contact

aviary@⁠

⁠

weareavp.com⁠

if you have issues with CSVs during Aviary Package imports.

⁠

OHMS XML Import

Adding a New Import - OHMS XML

Prepare your OHMS Files

Aviary allows you to import media files referenced in OHMS XML in two ways: from URLs on the web, or from an HTML media embed code. Your media files, if listed as URLs in OHMS, should be accessible to Aviary and listed in one of the OHMS XML elements listed in the OHMS-to-Aviary resource mapping.

OHMS metadata mappings can also be found in the

OHMS-to-Aviary resource mapping⁠

. See “Resource CSV” below for more detail on Aviary descriptive metadata elements.

Import your files

To import OHMS XML, select the “New Import” button from the Imports page.

Select the OHMS XML Import tab, and click Browse to select your OHMS XML files.

⁠

Screen Shot 2019-05-03 at 4.32.40 PM.png

356 kB

⁠

Once you've added resources for import, you have the option to set visibility for each file associated with your OHMS XML. Make resources and their associated media, transcript and indexes public or private depending on your needs:

Once you've selected your privacy settings, you can either create a new Collection or select an existing Collection in Aviary where the new Resources will be created.

Click “Add to Queue” and you will then be redirected to the Manage Imports page, where you should see your import awaiting action. Here you can click on "Preview" to see the Resources that will be created by your import to check everything looks good to go. To navigate through the resources click on the blue arrow in the "Import File Value" column.

⁠

Screen Shot 2019-06-12 at 1.23.04 PM.png

53 kB

⁠

When you’re ready to import, return to the Imports management page, and then click “Start Import”. This may take quite a while, but the import will run in the background, so it’s okay to move freely about the site while you’re waiting. Your Resources will start showing in the Resources page as they are being created.

Load content from aviaryplatform.atlassian.net?

Loading external content may reveal information to 3rd parties. Learn more

Allow

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.