Populate CMS database with existing collections

[benoit74]

We need a maintenance script to populate CMS database with existing books.

This maintenance script needs to:

• automatically create missing titles and associate them with their collection
• automatically create missing books and associate them with their title

It hence assumes that all ZIM in production have a correct title.

The script can assume that:

• usual environment variables are set: DATABASE_URL, LOCAL_WAREHOUSE_PATHS, QUARANTINE_WAREHOUSE_ID, QUARANTINE_BASE_PATH, STAGING_WAREHOUSE_ID, STAGING_BASE_PATH
• DB is reachable, ZIM files are locally available
• DB is properly configured with warehouses and collections (including their paths)
• Every title has only one collection configured (the case where we wanna publish a single title in two distinct collections does not exist)

The script will explore all warehouses for ZIMs, create titles, associate them with proper collection, create books.

The script will hence needs to use the python-libzim to open the ZIM and retrieve ZIM metadata.

The script will be idempotent, i.e. it can be ran twice in a row without creating duplicate records in the DB.

Zimcheck results will be left empty, no need to recompute them for the time being.

Title and book events will be populated with clear indication that its the maintenance script which performed operations.

The script will not need to be packaged in any form, it will be manually deployed and configured on proper server.

[elfkuzco]

automatically create missing titles and associate them with their collection

Could you clarify more on this? I get that for associating books with titles, we would want to use the Name and find the title with matching name. How would the collection association work?

[benoit74]

Association with collection is indeed a good question and a bit tricky one since we've said that we do not store available paths in collections (which is what I assumed when writing this issue).

In order to have a versatile script, let's say that we will already have (fake) titles for all potential paths in the DB, so first thing the script will do is it will build a list of folders per warehouse, and for each check we do have one (and only one) collection association through existing collection titles, keeping this in mapping folder => collection in memory.

It will then use this mapping to explore all folders and associate title with correct collection. If a title ends-up in multiple collection, we should probably consider it is an error and move on to next file (until we know how to fix this error at least ^^)