From NSDLWiki

Jump to: navigation, search



Summary of the NSDL OAI production Harvest / Ingest Process

Activity name

OAI Harvest / Ingest


The processing of harvest requests.

  • Managing harvest parameters - adjusting for changes
  • Follow-up on all harvests:
    • There are successful harvests for which some records fail or trigger warnings. These need investigation & may need action.
    • Failed harvests - always need action. Changes to harvest parameters, conversations with OAI administrators or collection providers.
  • Verification that the fedora proai (OAI service) has successfully captured updates.
    • Two proai instances are updated for each NDR system (leader and followers), so one leader & two followers = 6 proai instances.
    • This has been sometimes problematic in the past, but has been quite stable recently (past 4-5 months) - as of 10/08.


  • Tim C.


  • NSDL (Tim C)


  • No versioning specific to the process.


  • Harvest manager web application written in java
  • mySQL database containing log entries

Software Development and Version Control

  • CVS: OAI harvester written in perl
  • CVS: UTF8 conditioner written in C
  • CVS: Harvest / ingest application written in java

Work in Progress

  • record counts for full ingest: DONE
  • record counts for all OAI harvests.

Work planned

  • modularize insertion to handle single records, and accomodate any metadata types as "native"
Personal tools