OEDP Automation Strategy - Revision history

Lsanger: /* How to automatically ZWI-ify simpler encyclopedias */

2024-02-22T20:52:25Z

How to automatically ZWI-ify simpler encyclopedias

← Older revision		Revision as of 16:52, 22 February 2024
Line 16:		Line 16:
	Completed and partially completed tasks:		Completed and partially completed tasks:

	* <code>text_cleaner.rb</code> inputs <code>foo.txt</code> and outputs <code>foo-cu.txt</code> (for "cleaned-up"). This takes a TXT file (''not'' HTML) and fixes some common problems with OCR output, such as spaces between quotation marks and the words they govern. It also creates a new <code>/foo</code> directory if one does not yet exist. Since it is HTML only, it does not have any italics, bold, or small caps.		* <code>text_cleaner.rb</code> inputs <code>foo.txt</code> and outputs <code>foo-cu.txt</code> (for "cleaned-up") in the <code>foo/</code> directory (where 'foo' is the metadata's Publisher code). This takes a TXT file (''not'' HTML) and fixes some common problems with OCR output, such as spaces between quotation marks and the words they govern. It also creates a new <code>/foo</code> directory if one does not yet exist. Since it is HTML only, it does not have any italics, bold, or small caps.
	* <code>firstline_finder.rb</code> inputs a <code>foo-cu.txt</code> file and outputs <code>foo-mu.txt</code> (for "marked-up"). This takes the cleaned-up TXT file and adds ~~both~~ <code><article></code> and ~~(soon)~~ <code><nowiki><span class="title"></nowiki></code> tags.		* <code>firstline_finder.rb</code> inputs a <code>foo-cu.txt</code> file and outputs <code>foo-mu.txt</code> (for "marked-up"), also in the <code>foo/</code> directory. This takes the cleaned-up TXT file and adds <code><article></code> and<code><nowiki><span class="title"></nowiki></code> and <code><nowiki><p></nowiki></code> tags.
			* <code>htmlizer.rb</code> takes the <code>foo-mu.txt</code> (marked-up) file and splits them into individual html files, placing them in a new <code>foo/html/</code> directory. For example, one of 3000+ HTML autogenerated this way was saved at <code>ccrk/html/Aachen.html</code>.
			* <code>zwify.rb</code> iterates over the contents of <code>foo/html/</code>, prepares <code>article.html</code>, <code>article.txt</code>, <code>media.json</code>, <code>metadata.json</code>, and <code>signature.json</code> for each article individual article, putting the results in a subdirectory of <code>foo/zwicontent/</code>; for example, <code>ccrk/zwicontent/Aachen/</code> . It then iterates over the <code>zwicontent/</code> subdirectories and produces ZWI files out of them, placing the files in <code>foo/zwi/</code>; for example, <code>ccrk/zwi/Aachen.zwi</code>.
	Tasks to do:		Tasks to do:

	* ~~Actually do~~ the ~~<code><nowiki><span class="title"></nowiki></code> markup. I seem~~ to ~~have made an excellent set of rules for this purpose~~, ~~so I'm all but ready to add this to <code>firstline_finder.rb</code>~~.		* Improve the accuracy of heuristics used to individuate articles, and other small things.
	* ~~Do a test run through the entire ccrk-mu.txt output, make necessary changes~~ to ~~ccrk.txt, and see if the changes made by hand have the preferred effect~~.		* Actually push to Oldpedia.
	* Examine ZWIFormat again and decide whether

Lsanger: /* How to automatically ZWI-ify simpler encyclopedias */

2024-02-19T21:23:38Z

How to automatically ZWI-ify simpler encyclopedias

← Older revision		Revision as of 17:23, 19 February 2024
Line 18:		Line 18:
	* <code>text_cleaner.rb</code> inputs <code>foo.txt</code> and outputs <code>foo-cu.txt</code> (for "cleaned-up"). This takes a TXT file (''not'' HTML) and fixes some common problems with OCR output, such as spaces between quotation marks and the words they govern. It also creates a new <code>/foo</code> directory if one does not yet exist. Since it is HTML only, it does not have any italics, bold, or small caps.		* <code>text_cleaner.rb</code> inputs <code>foo.txt</code> and outputs <code>foo-cu.txt</code> (for "cleaned-up"). This takes a TXT file (''not'' HTML) and fixes some common problems with OCR output, such as spaces between quotation marks and the words they govern. It also creates a new <code>/foo</code> directory if one does not yet exist. Since it is HTML only, it does not have any italics, bold, or small caps.
	* <code>firstline_finder.rb</code> inputs a <code>foo-cu.txt</code> file and outputs <code>foo-mu.txt</code> (for "marked-up"). This takes the cleaned-up TXT file and adds both <code><article></code> and (soon) <code><nowiki><span class="title"></nowiki></code> tags.		* <code>firstline_finder.rb</code> inputs a <code>foo-cu.txt</code> file and outputs <code>foo-mu.txt</code> (for "marked-up"). This takes the cleaned-up TXT file and adds both <code><article></code> and (soon) <code><nowiki><span class="title"></nowiki></code> tags.
			Tasks to do:

			* Actually do the <code><nowiki><span class="title"></nowiki></code> markup. I seem to have made an excellent set of rules for this purpose, so I'm all but ready to add this to <code>firstline_finder.rb</code>.
			* Do a test run through the entire ccrk-mu.txt output, make necessary changes to ccrk.txt, and see if the changes made by hand have the preferred effect.
			* Examine ZWIFormat again and decide whether

Lsanger: Where I'm at with AutoZWI

2024-02-19T20:59:46Z

Where I'm at with AutoZWI

← Older revision		Revision as of 16:59, 19 February 2024
Line 1:		Line 1:
			== The steps to turn old encyclopedias into ZWI-driven web pages ==
	There are five steps to go from old paper encyclopedias to encyclopedia articles on Oldpedia (or other [[Encyclosphere]] sites).		There are five steps to go from old paper encyclopedias to encyclopedia articles on Oldpedia (or other [[Encyclosphere]] sites).

Line 9:		Line 10:

	The only type of encyclopedia that is a candidate for full automation is the sort with relatively short articles and relatively few images (although fewer than 50 or 100 plates seems could be processed by hand. It is these encyclopedias that the rest of this page will deal with.		The only type of encyclopedia that is a candidate for full automation is the sort with relatively short articles and relatively few images (although fewer than 50 or 100 plates seems could be processed by hand. It is these encyclopedias that the rest of this page will deal with.

			== How to automatically ZWI-ify simpler encyclopedias ==
			With [[AutoZWI]], there are several tasks complete, several to do, and several open questions.

			Completed and partially completed tasks:

			* <code>text_cleaner.rb</code> inputs <code>foo.txt</code> and outputs <code>foo-cu.txt</code> (for "cleaned-up"). This takes a TXT file (''not'' HTML) and fixes some common problems with OCR output, such as spaces between quotation marks and the words they govern. It also creates a new <code>/foo</code> directory if one does not yet exist. Since it is HTML only, it does not have any italics, bold, or small caps.
			* <code>firstline_finder.rb</code> inputs a <code>foo-cu.txt</code> file and outputs <code>foo-mu.txt</code> (for "marked-up"). This takes the cleaned-up TXT file and adds both <code><article></code> and (soon) <code><nowiki><span class="title"></nowiki></code> tags.

Lsanger: Draft save

2024-02-19T20:16:23Z

Draft save

← Older revision		Revision as of 16:16, 19 February 2024
Line 1:		Line 1:
	There are ~~''n''~~ steps to ~~going~~ from old paper encyclopedias to encyclopedia articles on Oldpedia (or other [[Encyclosphere]] sites).		There are five steps to go from old paper encyclopedias to encyclopedia articles on Oldpedia (or other [[Encyclosphere]] sites).

	# Scan the books: go from paper to image-only PDF. This has already been done for well over a hundred volumes, more than enough to keep us busy.		# Scan the books: go from paper to image-only PDF. This has already been done for well over a hundred volumes, more than enough to keep us busy.
Line 5:		Line 5:
	# Output text or HTML: go from text-recognized PDF to TXT and HTML. Another AFR step.		# Output text or HTML: go from text-recognized PDF to TXT and HTML. Another AFR step.
	#Markup: go from either TXT or HTML to a file marked up with <code><article></code> as well as markup for title. This step is performed by a combination of hand and code (in the old [[ZWIFormat]] system) or, for one encyclopedia so far, by code (in the new [[AutoZWI]] system).		#Markup: go from either TXT or HTML to a file marked up with <code><article></code> as well as markup for title. This step is performed by a combination of hand and code (in the old [[ZWIFormat]] system) or, for one encyclopedia so far, by code (in the new [[AutoZWI]] system).
	#Make ZWI: go from marked up file to ZWI file. This step is performed by a combination of hand and code (in the old [[ZWIFormat]] system) or, for one encyclopedia so far, by code (in the new [[AutoZWI]] system).		#Make ZWI (and upload): go from marked up file to ZWI file. This step is performed by a combination of hand and code (in the old [[ZWIFormat]] system) or, for one encyclopedia so far, by code (in the new [[AutoZWI]] system).
			While a fully automated system system certainly sounds better, steps 1-3 cannot be automated yet, and encyclopedias with longer articles, and especially those with complex features such as more complex styling, footnotes, tables, and pictures, cannot be handled fully automatically.

			The only type of encyclopedia that is a candidate for full automation is the sort with relatively short articles and relatively few images (although fewer than 50 or 100 plates seems could be processed by hand. It is these encyclopedias that the rest of this page will deal with.

Lsanger: Finished describing five steps

2024-02-19T20:08:19Z

Finished describing five steps

← Older revision		Revision as of 16:08, 19 February 2024
Line 4:		Line 4:
	# OCR the PDFs: go from image-only PDF to text-recognized PDF. An ABBYY FineReader OCR Editor (AFR) step.		# OCR the PDFs: go from image-only PDF to text-recognized PDF. An ABBYY FineReader OCR Editor (AFR) step.
	# Output text or HTML: go from text-recognized PDF to TXT and HTML. Another AFR step.		# Output text or HTML: go from text-recognized PDF to TXT and HTML. Another AFR step.
	#		#Markup: go from either TXT or HTML to a file marked up with <code><article></code> as well as markup for title. This step is performed by a combination of hand and code (in the old [[ZWIFormat]] system) or, for one encyclopedia so far, by code (in the new [[AutoZWI]] system).
			#Make ZWI: go from marked up file to ZWI file. This step is performed by a combination of hand and code (in the old [[ZWIFormat]] system) or, for one encyclopedia so far, by code (in the new [[AutoZWI]] system).

Lsanger: Initial save

2024-02-19T19:51:56Z

Initial save

New page

There are ''n'' steps to going from old paper encyclopedias to encyclopedia articles on Oldpedia (or other [[Encyclosphere]] sites).

# Scan the books: go from paper to image-only PDF. This has already been done for well over a hundred volumes, more than enough to keep us busy.
# OCR the PDFs: go from image-only PDF to text-recognized PDF. An ABBYY FineReader OCR Editor (AFR) step.
# Output text or HTML: go from text-recognized PDF to TXT and HTML. Another AFR step.
#