From NSDLWiki

Contents [hide] 1 Summary of the NSDL OAI production Harvest / Ingest Process 1.1 Activity name 1.2 Description 1.3 Who 1.4 Funding 1.5 Version 1.6 Dependencies 1.7 Software Development and Version Control 1.8 Work in Progress 1.9 Work planned

Summary of the NSDL OAI production Harvest / Ingest Process

Activity name

OAI Harvest / Ingest

Description

The processing of harvest requests.

• Managing harvest parameters - adjusting for changes
• Follow-up on all harvests:
  • There are successful harvests for which some records fail or trigger warnings. These need investigation & may need action.
  • Failed harvests - always need action. Changes to harvest parameters, conversations with OAI administrators or collection providers.
• Verification that the fedora proai (OAI service) has successfully captured updates.
  • Two proai instances are updated for each NDR system (leader and followers), so one leader & two followers = 6 proai instances.
  • This has been sometimes problematic in the past, but has been quite stable recently (past 4-5 months) - as of 10/08.

Who

• Tim C.

Funding

• NSDL (Tim C)

Version

• No versioning specific to the process.

Dependencies

• Harvest manager web application written in java
• mySQL database containing log entries

Software Development and Version Control

• CVS: OAI harvester written in perl
• CVS: UTF8 conditioner written in C
• CVS: Harvest / ingest application written in java

Work in Progress

• record counts for full ingest: DONE
• record counts for all OAI harvests.

Work planned

• modularize insertion to handle single records, and accomodate any metadata types as "native"