Hampson: Created page with "A database is a collection of ZWI files, usually hosted by an aggregator. More documentation can be found at [https://docs.encyclosphere.org/#/databases this link]. == Structure == Here's what the file tree of a typical ZWI database looks like: `database/ └── '''en <————''' ├── examplepedia │ ├── en.examplepedia.org │ │ └── wiki#Example_article.zwi │ └── en.examplepedia.org...."`


2024-02-21T21:26:00Z
Created page with "A database is a collection of ZWI files, usually hosted by an aggregator. More documentation can be found at [https://docs.encyclosphere.org/#/databases this link].  == Structure == Here's what the file tree of a typical ZWI database looks like:  <code>database/  └── '''en <————'''      ├── examplepedia      │   ├── en.examplepedia.org      │   │   └── wiki#Example_article.zwi      │   └── en.examplepedia.org...."
New page
A database is a collection of [[ZWI]] files, usually hosted by an [[aggregator]]. More documentation can be found at [https://docs.encyclosphere.org/#/databases this link].



== Structure ==

Here's what the file tree of a typical ZWI database looks like:

 <code>database/

 └── '''en <————'''

     ├── examplepedia

     │   ├── en.examplepedia.org

     │   │   └── wiki#Example_article.zwi

     │   └── en.examplepedia.org.csv.gz

     ├── trash

     │   └── examplepedia

     │       ├── en.examplepedia.org

     │       │   └── wiki#Deleted_article.zwi

     │       └── en.examplepedia.org.csv.gz

     └── index.csv.gz</code>

There's an <code>en</code> folder in the top level of this database. <code>en</code> is the 2-letter ISO language code for English. (Here's a handy list of language codes.) In the top level of the database, there is a folder for each 2-letter language code, containing all of the articles in the corresponding language.

 <code>database/

 └── en

     ├── '''examplepedia <————'''

     │   ├── en.examplepedia.org

     │   │   └── wiki#Example_article.zwi

     │   └── en.examplepedia.org.csv.gz

     ├── trash

     │   └── examplepedia

     │       ├── en.examplepedia.org

     │       │   └── wiki#Deleted_article.zwi

     │       └── en.examplepedia.org.csv.gz

     └── index.csv.gz</code>

In each language folder (e.g. <code>en</code>), there is a folder for each publisher, containing the articles from that publisher.

 <code>database/

 └── en

     ├── examplepedia

     │   ├── en.examplepedia.org

     │   │   └── wiki#Example_article.zwi

     │   └── en.examplepedia.org.csv.gz

     ├── '''trash <————'''

     │   └── examplepedia

     │       ├── en.examplepedia.org

     │       │   └── wiki#Deleted_article.zwi

     │       └── en.examplepedia.org.csv.gz

     └── index.csv.gz</code>

Also in the language folder, there's a special <code>trash</code> folder, containing ZWI files marked for deletion. The <code>trash</code> folder has the same structure as the parent language folder.

 <code>database/

 └── en

     ├── examplepedia

     │   ├── '''en.examplepedia.org <————'''

     │   │   └── wiki#Example_article.zwi

     │   └── en.examplepedia.org.csv.gz

     ├── trash

     │   └── examplepedia

     │       ├── en.examplepedia.org

     │       │   └── wiki#Deleted_article.zwi

     │       └── en.examplepedia.org.csv.gz

     └── index.csv.gz</code>

In each publisher folder (e.g. <code>examplepedia</code>), there is a folder for each domain used by the publisher, containing the articles hosted on that domain. In this example, the only domain is <code>en.examplepedia.org</code>.

 <code>database/

 └── en

     ├── examplepedia

     │   ├── en.examplepedia.org

     │   │   └── '''wiki#Example_article.zwi <————'''

     │   └── en.examplepedia.org.csv.gz

     ├── trash

     │   └── examplepedia

     │       ├── en.examplepedia.org

     │       │   └── wiki#Deleted_article.zwi

     │       └── en.examplepedia.org.csv.gz

     └── index.csv.gz</code>

In the domain folders (e.g. <code>en.examplepedia.org</code>) are the ZWI files themselves.



The names of the ZWI files start with the part of the <code>SourceURL</code> after the domain, with the slashes replaced by pound signs (<code>#</code>). (<code>SourceURL</code> is a field in the ZWI file's <code>metadata.json</code>.)



Any existing pound signs in the URL are escaped with another pound sign: <code>##</code>



Finally, <code>.zwi</code> is added.
Database - Revision history
