What Does Morsulus Herald Do?

Notes from a session with Herveus d’Ormonde at the Known World Heralds and Scribal Symposium, June 9, 2018.

Who serves as Morsulus Herald?

Herveus d’Ormonde has held this position since Apr 2000.

His predecessors in the role were:

  • Master Renfield Wanderscribe (1980–1985) created first society armorial and ordinary.
  • Alban St. Albans (Jul 1985–) and Ianthus (sp?) set up a searchable database.
  • Iulstan Sigewealding (early 90s– Mar 2000) wrote most of the current codebase.

What is the job of the Morsulus Herald?

The primary task of the Morsulus Herald is maintaining the database of heraldic registrations, including indexing new entries, making corrections, and maintaining the software that powers the oanda.sca.org web interface.

Also responsible for publishing the LoARs to the heraldry.sca.org/loar web site, which  only takes a few minutes each month. When the LoAR is published, Morsulus downloads these in HTML format and uploads them to the web site.

Where does the master database and code live?

All of the data processing happens on Herveus’s local workstation, a Mac laptop. This machine is archived on a external backup device in the same location.

The code is written in Perl and run from the command line. There are multiple scripts which perform different parts of the process. Morsulus has a text file that shows examples of the various script invocations required and works through each of the steps as required every month, figuring out which commands need to be run and making minor changes to the file names or options as needed.

The code is tracked in Git, and exposed to the public at github.com/herveus, although the GitHub version lags somewhat behind the current working version used by Herveus.

The master database is maintained in a SQLite database file. The headings used internally correspond to the ones in the mike.cat file described below.

How is the database updated for new LoARs?

Every month, the Laurel team sends Morsulus a batch of XML files corresponding to the new LoAR. Actually, this happens three times, the first two times corresponding to the two proof passes, and then with the final version. The XML files are generated by OSCAR, but then undergo manual revision during the editing process. If problems with the XML files interfere with processing the data, Morsulus reports this to the Laurel team which makes corrections in the next pass.

A Perl script converts the XML file to a delimited-text “action” file which includes all of the changes for each name or armory as a single line.

Another Perl script applies those actions to the master database, after doing some logical consistency checks — eg, confirming that you don’t add armory without a name, that you don’t add two pieces of armory with identical blazons, that you can’t do a name change unless the old name is already registered, etc.

Lines in the action file which can’t be successfully applied to the database are flagged as errors, and Morsulus comments them out and hand-writes a revised version of the line called a “temporary edit” which can be applied, and notifies Laurel about the issue so it can be fixed in the next revision of the XML.

Then Morsulus runs a program in which he can perform the indexing of armorial headings and features for each blazon. This is a Perl TK app which runs in the X11 environment. It has a point-and-click interface that allows you to add multiple armorial descriptions for each blazon, picking from lists of headings and features. The descriptions are then copied to the local batch and the master database, indexed by blazon. Because the descriptions are indexed by blazon, this process can be done when the first proof pass is received, and does not need to be repeated for the next two passes except for any items whose blazons are changed during editing.

In the course of processing a month’s batch, Morsulus may need to add items to the category file. It’s fairly common to add new category cross-references (eg, “bay leaf – see leaf”) and less common to add a new category with a heading (eg “manacle|MANACLE”). When this happens, related changes must be made to several files in parallel.

  • The my.cat file is exposed to the public and corresponds to values visible in the oanda.db.
  • There is a separate mike.cat file which is used for indexing and which contains some additional headings and in some places uses different heading names. The mike.cat file also includes a list of which feature sets are applied to a given heading.
  • There is a tprint.cat file which lists all of the categories in the order they should be listed to the public. (They used to be sorted alphabetically by heading, which is why some of the heading names are strange.) It also specifies which categories to break down into smaller sections by tincture, number, or group; this is used to generate the web ordinary interface as well as the print ordinary.

Then Morsulus runs a consistency-check script that scans the SQLite database and summarizes various kinds of totals, and compare the output to a copy saved from the previous month, to see what’s changed and confirm that the number of changes match the expectation based on the number of items in the LoAR.

A Perl script exports a copy of the database from SQLite to a flat file format called “classic format.” This includes converting some internal headings from mike.cat to the public equivalents used in my.cat; for example, “BIRD-PELICAN” gets exported as “BIRD:pelican.” These changes cause the exported file to line up with the combined categories used for the ordinary display and for the complex search form.

A set of “DB Diffs” are created that compare the new oanda.db file with the one that was generated last month. This simplifies the process of loading new data into another search tool built by another developer, as they only have to import the changes rather than the entire file. (I am not sure which of the other search tools use this, but I think it’s Hirsch von Henford’s Golden Stag app.)

The updated my.cat and oanda.db are then uploaded to the oanda server and the search daemon is restarted so that it will re-index the file; this only takes a couple of seconds.

How is the database accessed by others?

Most people use the oanda web site. Among other things, log files show that some local kingdom OP sites crawl the oanda web site to automatically collect armorial blazons for people in their region.

You can get a copy of the software needed to run the oanda web site by downloading and running the config.web script, which unpacks a copy of the Morsulus-search package.

Some folks, especially in the West Kingdom, are heavily invested in using a “print” version of the ordinary for lookups and conflict checks. Rather than actually being printed on paper, this is a set of documents accessible from oanda.gigo.com which are built from the oanda.db in formats such as epub and PDF using some third-party software. That process uses a file named templ.cat which is similar to my.cat but has print-specific notes such as “do not print” added to some categories.

(There was an old print version which was built by Morsulus using some custom C code which used the database to generate PostScript, but the last time this was used was for Pennsic 2011 and it has now been retired.)

(Although not discussed in detail during this meeting, there are also a couple of other search tools which can import the oanda.db file, including two Windows apps and a Lotus Notes interface.)

How are errors corrected?

People write in to Morsulus when they notice a problem with an armorial listing.

Morsulus checks the reports against the original LoARs and against an archive of scans of the original registration files to figure out at what stage of the process the error appeared.

In some cases the error originates in the LoAR; for those, people need to go through an LoAR errata process, get Laurel signoff, and then the correction is processed as part of the corresponding LoAR update.

In other cases the problem is due to a transcription error, in which the LoAR entry was garbled when it was transferred to the database or the indexing of categories and features was done incorrectly. In those cases Morsulus can simply correct the problem to accurately reflect the LoAR and push that change out in the next update cycle.

Those changes are made via a command-line tool that can search and edit the SQLite master database. This has various internal-use-only functions like looking up records by regid (a synthetic primary key) and directly editing raw database fields.

What are some random factoids about the database?

The database contains approximately 110K registrations. It takes up 120 MB in SQLite with the various indexes, but the flat-file version is only 16 MB.

There are a few cases of where there are two name registrations for the same name.

Jointly owned items have a primary name that they’re attached to, plus a second entry that just references that they’re a co-owner.

A SQL Interface for the SCA Armorial

As powerful as the Morsulus O-and-A software is, it’s not particularly easy to install and get running on your own computer, and it’s a bit daunting to extend with new functionality.

I had an idea that it would be easier to work with the armorial data if it was loaded into a regular SQL database, and spent my weekend putting together a proof-of-concept implementation.

The results are available as Clerk-0.1.tar.gz, a pair of Perl scripts designed to work with a MySQL database.

The first, make.pl, will fetch the armorial data from the SCA’s web site, create database tables to hold it, import the data, and do a bit of cross-indexing to make searching easier.

$ ./make.pl
No changes made; you are up-to-date.

The second, search.pl, accepts command-line arguments that parallel the various options supported by the Morsulus O-and-A interface, and outputs the matching registration records.

$ ./search.pl --armory "PFESS:pl:argent:~and vert" --armory BEAST-BEAR:1:primary --armory BEAST-BEAR:gules:primary --armory "BEAST-BEAR:passant to dexter:primary" --armory "BEAST-BEAR:spa" --close
Found 2 results

Here is one matching item with an exact match:

Mathghamhain Ua Ruadháin
  This device was registered January of 2017 via the East.
  Per fess argent and vert, a bear passant gules.

Here is one matching item with one difference:

Halldor Rauthbjorn
  This badge was registered May of 2002 via the Middle.
  (Fieldless) A bear passant guardant gules maintaining on its back a castle argent.

Both include usage information you can view by running them with --help.

The search results generally match up to those from the original Morsulus search, although I don’t yet have all of the character-code support in place for broad-vs-narrow name searches, and I haven’t yet implemented date/kingdom search criteria.

I’ve also added a couple of features that aren’t available in the baseline tool.

One that would be easy to add to the Morsulus code is a --min-score parameter, including shortcuts for --exact, which only returns items that match all of the provided search terms, and --close, which includes items that have at most one difference; both of these just set the minimum score based on the maximum possible weight.

A new feature that would be more difficult to add to the Morsulus scripts is the ability to search for any of several headings at once, for example --armory "ROSE|FOIL-6", or even to search for any heading at all, such as looking for any sable tertiary with --armory "*:sable:tertiary".

The code is available under a permissive open-source license so feel free to use or modify it as you would like.

Interfaces to the Armorial Database

I’m working on some possible improvements to the web interface used to search the SCA’s heraldic database, known colloquially as the “O and A,” short for “Ordinary and Armorial.”

(Traditionally, an armorial is a printed listing of armory registrations with their blazons and the names of their holders; mundane armorials typically include all, or all of the notable, registrations within a certain heraldic jurisdiction. An armorial is typically organized alphabetically by the holder’s name; in contrast an ordinary is a specialized index used for looking up armory based on its blazon or appearance, grouping registrations under their primary charges.)

For decades the SCA’s armorial database has been maintained and distributed as a delimited text file, but searching it by hand in this format is inconvenient.

Periodically the database has been converted to a PDF file that could be printed for reference by heraldic consultation tables at events without internet access, but the alphabetical armorial and indexed ordinary both grew to over a thousand pages long, which makes it impractical to print up-to-date versions on a regular basis.

There is an e-book version in ePub format that is generated from the database every month by Jason Fesler; it includes both an alphabetical armorial and an indexed ordinary.

However, in practice, most searches of the database are done using the web interface at oanda.sca.org, which provides searches by name and blazon, as well as a clickable ordinary and a powerful “complex search” interface that allows for filtering registrations with multiple simultaneous criteria, which greatly speeds the process of searching for potential conflicts with new heraldic submissions.

The software that powers that web interface is available on GitHub as part of the Morsulus-Tools package, maintained by Master Herveus d’Ormonde of the East, who has served as Morsulus Herald since 1999. It’s written in Perl and includes numerous CGI scripts the power the various search features.

There are several mirror sites which run copies of the same Morsulus oanda software, although you’ll want to double-check that their copies of the database are up-to-date.

The HeraldStick project by Tanczos Istvan provides instructions for setting up a copy of the Morsulus oanda software, along with mirrors of some useful reference works, as a web server that runs inside a virtual machine that can be packaged on a USB memory stick and attached to a Windows computer. The setup process is not fully automated, so a fair amount of technical ability is required to get it running.

The HeraldsPoint hot spot project by Master William of Meridies is an effort to package the HeraldStick software as a pre-configured package running on a WiFi access point that can be made accessible to multiple heralds working at an event’s consultation table. These devices seem promising but are not yet available for purchase.

A few third-party applications have been created that allow browsing and searching the database.

• Aspilogia is a Windows 8/10 application built by Oddr Þiálfason of An Tir. This includes a complex search tool, along with a fuzzy search for names and blazons.

• There is a Windows desktop application built using dBASE by Hirsch von Henford of the West Kingdom.

• Stentorian.us has a Lotus Notes interface to the database, affiliated with the Meridies College of Heralds.

Heraldic Search Shortcuts

I’ve put together an integrated search page that allows you to easily run search queries against a number of different websites that contain useful heraldry information.

It was inspired by similar collections of search shortcuts posted by Morsulus Herald and Sofya la Rus, but I’ve added a bit of JavaScript to make the interface less repetitive. I’ve extended the list of search targets with some other useful sites including OSCAR, some of the onomastic reference books posted by Wenyeva atte grene, and some old armory books I have been using as sources for images for the Book of Traceable Heraldic Art.

I hope it may prove useful to others — if there are websites that seem of general interest that I have omitted please feel free to point them out and I’ll see if I can add them as well.