Old Encyclopedia Digitization Project (OEDP) - Revision history

Lsanger: + OEDP Automation Strategy

2024-02-19T19:45:35Z

+ OEDP Automation Strategy

← Older revision		Revision as of 15:45, 19 February 2024
Line 15:		Line 15:
	We began with this thought: we’ve made a file format for encyclopedia articles (the ZWI file format); wouldn’t it be awesome if we digitized a bunch of old, public domain encyclopedias and put them into ZWI files?		We began with this thought: we’ve made a file format for encyclopedia articles (the ZWI file format); wouldn’t it be awesome if we digitized a bunch of old, public domain encyclopedias and put them into ZWI files?

	'''The project's top priority''' is to speed up the process 100x, mainly by writing software (possibly with AI help) for ''automatically'' generating ZWI files correctly from pure OCR output. This is harder than it sounds; an only partly automated software solution was written by Larry Sanger and is [https://gitlab.com/ks_found/zwiformat/ here]. We have experimented around trying to get an LLM API to help with the tricky parts, and with some progress, but without usable output yet.		'''The project's top priority''' is to speed up the process 100x, mainly by writing software (possibly with AI help) for ''automatically'' generating ZWI files correctly from pure OCR output. This is harder than it sounds; an only partly automated software solution was written by Larry Sanger and is [https://gitlab.com/ks_found/zwiformat/ here]. We have experimented around trying to get an LLM API to help with the tricky parts, and with some progress, but without usable output yet. See [[OEDP Automation Strategy]].

	Our ongoing priority is simply to do the hard work of preparing ZWI versions of old encyclopedia articles. This is semi-technical work, done partly in old documents, partly on the command line, so it is a niche thing.		Our ongoing priority is simply to do the hard work of preparing ZWI versions of old encyclopedia articles. This is semi-technical work, done partly in old documents, partly on the command line, so it is a niche thing.

Hampson: The LinkTitles extension automatically added links to existing pages (https://github.com/bovender/LinkTitles).

2024-02-16T19:17:05Z

The LinkTitles extension automatically added links to existing pages (https://github.com/bovender/LinkTitles).

← Older revision		Revision as of 15:17, 16 February 2024
Line 7:		Line 7:
	\|date_started=Summer 2022		\|date_started=Summer 2022
	}}		}}
	The '''Old Encyclopedia Digitization Project''' aims to digitize old encyclopedias in the ZWI file format. We will also make the results available via [https://oldpedia.org Oldpedia.org].		The '''Old Encyclopedia Digitization Project''' aims to digitize old encyclopedias in the [[ZWI file format]]. We will also make the results available via [https://oldpedia.org Oldpedia.org].

	How much knowledge is buried in old encyclopedias?		How much knowledge is buried in old encyclopedias?

Hampson at 18:53, 2 February 2024

2024-02-02T18:53:10Z

← Older revision		Revision as of 14:53, 2 February 2024
Line 2:		Line 2:
	\|title=Old Encyclopedia Digitization Project (OEDP)		\|title=Old Encyclopedia Digitization Project (OEDP)
	\|url=https://encyclosphere.org/the-old-encyclopedia-digitization-project		\|url=https://encyclosphere.org/the-old-encyclopedia-digitization-project
	\|description=A project to digitize public domain encyclopedias in the ZWI file format.		\|description=A project to digitize public domain encyclopedias in the [[ZWI]] file format.
	\|project-leads=[[Larry Sanger]], [[Nancy Hildebrandt]]		\|project-leads=[[Larry Sanger]], [[Nancy Hildebrandt]]
	\|biggest-challenge=Autogenerating usable ZWI files from OCR text file output.		\|biggest-challenge=Autogenerating usable ZWI files from OCR text file output.

Hampson at 18:52, 2 February 2024

2024-02-02T18:52:56Z

← Older revision		Revision as of 14:52, 2 February 2024
Line 7:		Line 7:
	\|date_started=Summer 2022		\|date_started=Summer 2022
	}}		}}
	The '''Old Encyclopedia Digitization Project''' aims to digitize old encyclopedias in the ZWI file format. We will also make the results available via [[Oldpedia.org]].		The '''Old Encyclopedia Digitization Project''' aims to digitize old encyclopedias in the ZWI file format. We will also make the results available via [https://oldpedia.org Oldpedia.org].

	How much knowledge is buried in old encyclopedias?		How much knowledge is buried in old encyclopedias?

	We aim to find out—and to make it available via modern search engines, readers, and free files available via the Encyclosphere, with the Old Encyclopedia Digitization Project.		We aim to find out—and to make it available via modern search engines, readers, and free files available via the [[Encyclosphere]], with the Old Encyclopedia Digitization Project.

	We began with this thought: we’ve made a file format for encyclopedia articles (the ZWI file format); wouldn’t it be awesome if we digitized a bunch of old, public domain encyclopedias and put them into ZWI files?		We began with this thought: we’ve made a file format for encyclopedia articles (the ZWI file format); wouldn’t it be awesome if we digitized a bunch of old, public domain encyclopedias and put them into ZWI files?

Lsanger: Content additions

2024-01-24T21:32:41Z

Content additions

← Older revision		Revision as of 17:32, 24 January 2024
Line 7:		Line 7:
	\|date_started=Summer 2022		\|date_started=Summer 2022
	}}		}}
	The [[Old Encyclopedia Digitization Project]] aims to digitize old encyclopedias in the ZWI file format. We will also make the results available via [[Oldpedia.org]].		The '''Old Encyclopedia Digitization Project''' aims to digitize old encyclopedias in the ZWI file format. We will also make the results available via [[Oldpedia.org]].

	How much knowledge is buried in old encyclopedias?		How much knowledge is buried in old encyclopedias?
Line 14:		Line 14:

	We began with this thought: we’ve made a file format for encyclopedia articles (the ZWI file format); wouldn’t it be awesome if we digitized a bunch of old, public domain encyclopedias and put them into ZWI files?		We began with this thought: we’ve made a file format for encyclopedia articles (the ZWI file format); wouldn’t it be awesome if we digitized a bunch of old, public domain encyclopedias and put them into ZWI files?

			'''The project's top priority''' is to speed up the process 100x, mainly by writing software (possibly with AI help) for ''automatically'' generating ZWI files correctly from pure OCR output. This is harder than it sounds; an only partly automated software solution was written by Larry Sanger and is [https://gitlab.com/ks_found/zwiformat/ here]. We have experimented around trying to get an LLM API to help with the tricky parts, and with some progress, but without usable output yet.

			Our ongoing priority is simply to do the hard work of preparing ZWI versions of old encyclopedia articles. This is semi-technical work, done partly in old documents, partly on the command line, so it is a niche thing.

			'''What we are doing'''

			[[Nancy Hildebrandt]] has done the most work preparing articles, from an encyclopedia of musical biography (as of 2023-24). [[Larry Sanger\|Larry]] has made some ZWI files from an old encyclopedia of Ohio. They both use subscriptions to ABBYY FineReader, which we can make available to anyone who is interested in this sort of work. Please let us know.

			[[John Hampson]] has been busy (in 2023-24) grading the scan qualities of old encyclopedia PDF files; if that work would help anyone, let us know.

	More introductory info (from late 2022) can be found at [https://encyclosphere.org/the-old-encyclopedia-digitization-project encyclosphere.org/the-old-encyclopedia-digitization-project].		More introductory info (from late 2022) can be found at [https://encyclosphere.org/the-old-encyclopedia-digitization-project encyclosphere.org/the-old-encyclopedia-digitization-project].

Lsanger: New page

2024-01-24T21:19:23Z

New page

← Older revision		Revision as of 17:19, 24 January 2024
Line 1:		Line 1:
	~~Information about~~ the ~~'''~~Old Encyclopedia Digitization Project~~'''~~ (~~OEDP~~) can be found at [https://encyclosphere.org/the-old-encyclopedia-digitization-project encyclosphere.org/the-old-encyclopedia-digitization-project~~]. It creates content for [https://oldpedia.org/ Oldpedia.org~~].		{{ProjectInfo
			\|title=Old Encyclopedia Digitization Project (OEDP)
			\|url=https://encyclosphere.org/the-old-encyclopedia-digitization-project
			\|description=A project to digitize public domain encyclopedias in the ZWI file format.
			\|project-leads=[[Larry Sanger]], [[Nancy Hildebrandt]]
			\|biggest-challenge=Autogenerating usable ZWI files from OCR text file output.
			\|date_started=Summer 2022
			}}
			The [[Old Encyclopedia Digitization Project]] aims to digitize old encyclopedias in the ZWI file format. We will also make the results available via [[Oldpedia.org]].

			How much knowledge is buried in old encyclopedias?

			We aim to find out—and to make it available via modern search engines, readers, and free files available via the Encyclosphere, with the Old Encyclopedia Digitization Project.

			We began with this thought: we’ve made a file format for encyclopedia articles (the ZWI file format); wouldn’t it be awesome if we digitized a bunch of old, public domain encyclopedias and put them into ZWI files?

			More introductory info (from late 2022) can be found at [https://encyclosphere.org/the-old-encyclopedia-digitization-project encyclosphere.org/the-old-encyclopedia-digitization-project].

Tim: first draft

2024-01-21T10:57:30Z

first draft

New page

Information about the '''Old Encyclopedia Digitization Project''' (OEDP) can be found at [https://encyclosphere.org/the-old-encyclopedia-digitization-project encyclosphere.org/the-old-encyclopedia-digitization-project]. It creates content for [https://oldpedia.org/ Oldpedia.org].