MUMPS routines have been written that generate XML
Document Type Definitions (DTDs) for queries and responses to/from any MUMPS
database built with the Department of Veterans Affairs FileMan software.
Additional routines parse XML queries that conform to the query DTD, perform
the requested selection, sorting and projection, and generate an XML response
that conforms to the response DTD. Selections can be based on any top-level
field of the database (multiples and pointers are not supported in this
version) using any of seven logical operators to compare field values to a
target value. Selections can be combined using logical operators (AND, OR, NOT)
to generate larger or smaller sets of matches. The optional sort criteria are
independent of selection criteria, and can be based on any top-level field, in
either direction. Any number of sort criteria can be used. Any subset of
top-level fields can be projected in the XML output. If no projection is
specified all top-level fields are presented in the XML output. Both selections
and sorts take advantage of existing cross-references (indexes) in the database
whenever possible.
Problem Statement and General Description
Design and Implementation Details
EQ, NE, LT, GT, LE, GE, SA, SB^ABS383D
Example 1: Populous Coastal States
Example 3: Mottoes of the Original
13 Colonies
Example 4: Entire Contents of the
Database
MUMPS is an ANSI and ISO standard programming language (MDC/ANSI X11.1-1995) that provides a very rich multi-user database environment. MUMPS enjoys wide popularity in a number of fields, notably health-care related fields. Much of this attraction stems from the freely available (through the Freedom of Information Act) software developed by the Department of Veterans Affairs. The most widely used portion of this software is the “File Manager” (or FileMan, as it is often called), which provides a data dictionary and a great number of tools that make MUMPS data a true database system. All databases built with FileMan share a number of common characteristics, and can be manipulated with the tools provided as part of FileMan.
An XML-based interface to MUMPS/FileMan databases will significantly extend the utility of these databases, especially when interacting with non-MUMPS interfaces and systems. Some XML interfaces to databases have been developed already, though these are generally application-specific interfaces using hand crafted document type definitions. Rather than take this limiting and labor-intensive approach, this project shows that useful document type definitions can be developed automatically for any database from the information in the data dictionary.
There are three main entry points to the routines provided for this project. The first entry point (GenQDTD^ABS383C) takes a FileMan File Number that uniquely identifies a particular “file” (equivalent to a “relation” or “table” in a relational database), and a file name as input. It generates in the named file an XML Document Type Definition for queries to the specified database file. Queries conforming to this document type specify the file (required), selection criteria (required), sort criteria (optional) and fields to project (optional). Selection criteria can be combined using AND, OR and NOT relationships, and any number of (nested) sorting criteria can be provided.
The second entry point (GenRDTD^ABS383B) also takes a FileMan File Number and file name as input, and generates a document type definition (in the named file) that corresponds to a response to a query as described above. Responses conforming to this document type contain zero or more records from the file, each of which has one or more fields for which data is provided (if it exists in the database).
The third entry point (ReadQry^ABS383D) takes a file name as input. This file is expected to be a valid XML file conforming to a query DTD generated by GenQDTD. This routine reads this file, and performs the indicated selection, sorting and projection operations, and generates a valid response XML file (named identically to the query except for a prepended “R”) conforming to the DTD generated by GenRDTD.
The parsing of the XML query is not as robust as it could be. Most valid XML queries should be handled properly, but it is possible to generate invalid queries that nevertheless generate responses. It is also possible to produce valid XML queries that the parser does not handle properly. In particular, comments and processing instructions are valid anywhere a tag can occur in an XML file. This parser allows these only before the <Query> tag. Furthermore, only white space may follow the </Query> tag, and the entire XML query cannot exceed 16KB. Future effort will make the parsing more robust, allowing additional errors to be caught.
The selection criteria are applied as the query file is parsed. An attribute grammar for the common portions of the query was developed, which allows semantic rules to be applied when start tags (<Tag>) or end tags (</Tag>) are found. As future optimizations are developed, it is likely that the actual database searching will be done after the parse is completed so that more thorough semantic analysis and optimizations can be applied to the parse tree before applying selection criteria. Currently the only optimization in the selection criteria themselves is the use of existing cross-references (defined in the data dictionary) whenever possible to avoid traversing the entire database. A simple semantic optimization is also performed: NOT operations on selection criteria are pushed down to the actual selection atoms (field relop expression) by the use of DeMorgan’s laws[1]. This removes the need to produce “all records in the database except these” at any step. AND and OR operators are simply the intersection and union (respectively) of the records found by each child criterion. No lazy evaluation of AND criteria is used at this time.
In this project, the XML queries are hand generated (they are quite short and straightforward), though it is expected that future work will produce one or more utilities to allow a user to easily generate queries given the DTD. Similarly, the XML responses are viewed using the “generic” XML viewer in Internet Explorer, which simply shows nested structures of the XML. Again, future work will produce more sophisticated views of the data represented in the XML response.
For simplicity in debugging and demonstrating this project, actual disk files are used for the DTDs and the XML queries and responses. In a more functional application, a client would send an XML query through a dedicated TCP port to the MUMPS server that would then respond over the same port with XML response. This could be done with only minor modifications to the existing routines.
An additional simplification (again for debugging and demonstration purposes) was to store all intermediate results in a fixed database location (MUMPS global), which is retained (with all intermediate results) after the response is generated. No effort was made to ensure that multiple simultaneous accesses don’t interfere with each other, nor to conserve space used. Rather the fixed structure and lack of “clean-up” allow the parsing, selection, sorting and projection processes to be observed directly. Support of multiple simultaneous uses and garbage collection will require only trivial modifications to the existing routines.
The routine that reads the XML Query and produces an XML Response also generates diagnostic output to the screen as it runs, indicating its progress. This may be eliminated or made optional in future versions, but was left in for demonstration purposes.
The implementation presented here is limited in that only fields that are direct members of the FileMan file can be used for selection, sorting and projection criteria. Fields in “sub-files” (used to store multiply-occurring attributes of a record), “super-files” (the reverse of sub-files) and linked fields from other databases are not available for use. Again, future work is expected to add this functionality. There are some difficult issues associated with sub-files and linked files that will need to be examined carefully (e.g., “any” vs. “all” vs. “at least n” matches on subfile-fields).
Other future work will provide additional logical operators for selection criteria (e.g., MUMPS pattern-match or some equivalent regular expression match), selection and projection of fields from linked files (join operations), and further optimization of the searching and sorting operations.
The project design was relatively easily divided into seven components. First the DVA FileMan Data Dictionary had to be read (MkTree^ABS383A). Using this structure, the Document Type Definitions (DTDs) for the query (GenQDTD^ABS383C) and response (GenRDTD^ABS383B) can be generated fairly easily. The XML Queries must be read (ReadQry^ABS383D) and parsed (BldTree^ABS383D). The Selection criteria are applied as they are parsed by the application of semantic attributes (routines following BldTree in ^ABS383D). Following the selection, the sort criteria are applied (Sort^ABS383E), and the projected fields are output in the XML Response (GenResp^ABS383E).
There are three entry points (user calls) for the project. The first, GenQDTD^ABS383C is used to generate the DTD for a query. It invokes the data dictionary reader (MkTree^ABS383A) and then traverses this tree to produce the DTD (main body of GenQDTD^ABS383C). The DTD for queries is constructed to allow the use of subfiles and superfiles. It does not currently incorporate any facility for using fields or subfiles from linked files, nor does it support the “any/all/at-least-n” criteria for multiple fields in subfiles.
The second entry point, GenRDTD^ABS383B is used to generate the DTD for a response. It can be called as a standalone routine, though it is also called when processing a query to ensure that an up-to-date DTD is referenced in the XML file. This routine is very similar to (and simpler than) the routine to generate the Query DTD.
The third entry point, ReadQry^ABS383D is used to process a query and generate a response. It reads in the entire XML query and does a very cursory check on the well-formedness and validity of the query (Prolog^ABS383D). It then creates the nodes for the semantic rules (in a production version, this would be constant data, not set each time). It then parses the query and applies the selection criteria (BldTree^ABS383D), applies the sort criteria (Sort^ABS383E) and applies the projection criteria to produce the XML response (GenResp^ABS383E). BldTree, Sort and GenResp are each successively called by ReadQry^ABS383D.
All the routines are written in MUMPS. To the greatest extent possible, all MUMPS routines conform to MDC/ANSI X11.1-1995 (the current ANSI/ISO MUMPS standard) and do not use any vendor-specific extensions. The only exception to this is the inevitable use of parameters to specify file names and modes on OPEN statements and the use of a vendor-provided function ($ZC) to identify an end-of-file condition when reading the query. The actual MUMPS implementation used was Micronetics Standard MUMPS (MSM-95 v. 4.4.0) produced by InterSystems.
The Document Type Definitions for both queries and responses, and the Response XML all conform to REC-xml-19980210 (the current W3C Recommendation for the Extensible Markup Language (XML) v. 1.0). Note that actual data output (content) in the XML Response uses the “&”, “<”, “'”, and “"” entities as needed.
Because XML NameChars are a proper subset of the valid characters in FileMan File and Field Names, it is necessary to convert all the non-NameChar characters to a valid NameChar when constructing the element names. All of these names are converted to an underscore (“_”), which could conceivably lead to non-unique file or field names if their names in the FileMan data dictionary only differed by these non-alphanumeric characters. This seems unlikely enough in practice to not warrant additional code to ensure uniqueness.
This short routine is used to read the data dictionary for a given FileMan file (specified by the FileNum parameter). After verifying that the file actually exists and initializing the root of the global (MUMPS persistent storage) tree that will contain the extracted data dictionary information it makes the initial call to the recursive routine SubTree^ABS383A which process the file and each of its subfiles[2]. The format of the resulting tree is described in the block comment of the routine (q.v.).
This routine reads the attribute dictionary (^DD) that
FileMan uses to store information about all FileMan files. In particular, it
determines the hierarchical relationship among all the (sub-)files and fields,
the type of each field and its optionality. It stores this information, along
with the name and composited file/field number. The composited file/field
number starts with the top-level file number (the FileMan unique identifier for
the file), and appends each (sub)file and field number down to the level of
this element. The numbers are separated by a “z” (a valid XML NameChar).
SubTree calls itself recursively to handle sub-files. When a field of type “Pointer
to file” — a link to another file, equivalent to a “role” or “relationship” in
conventional databases — is encountered no special processing takes place,
though there is a comment that future versions might want to recursively call
MkTree^ABS383A on those files to allow use of fields in these related files.
This routine calls MkTree^ABS383A to ensure that the information on the file is up-to-date, and opens the specified destination file and writes the “boilerplate” of the XML prolog, and the top level Response tag. The Response element is defined as being zero or more occurrences of any of the (sub)file elements. It then traverses the tree generated by MkTree and creates elements for each file and field. The (sub)file elements are defined as a list of optional field elements (not an optional list, but a required list of optional elements). The field elements are all defined as having #PCDATA content (the actual values of the fields). The only attributes specified are a required IENS attribute for each (sub)file element. The IENS (Internal Entry Number Structure) is used by FileMan to uniquely identify the specific record or sub-record.
This is simply a utility called by GenRDTD that is used to generate the element names for fields and files. The names are constructed by a single letter indicating the depth in the hierarchy (the top level file is “A”, its fields (including sub-files) are “B”, sub-file fields are “C”, etc.), followed by a hyphen, followed by the composited file number (see MkTree^ABS383A, above), another hyphen and finally the file/field name with all non-NameChar characters transformed to underscores (“_”). This produces a legal element name that contains the necessary information (the composited file numbers) as well as human-readable information (the depth and file/field name) that makes the DTD very approachable by computers and people alike.
This is very similar to GenRDTD^ABS383B (q.v.). The difference lies in the generic form of an XML Query. The Query itself is defined as a file element followed by the selection criteria, optional sort criteria and optional projection criteria.
The file elements are exactly as in the response (q.v.).
The selection criteria are either an OR, an AND, a NOT or an SAtom (selection atom). OR and AND elements each contain at least two elements, each of which is an OR, AND, NOT or SAtom. A NOT element consists of just a single OR, AND, NOT or SAtom element. SAtom elements consist of a Field element, one of seven relational operators and an “Expression” (actually a string constant without delimiters). The relational operators are EQ (equal to), NE (not equal to), GT (numerically greater than), LT (numerically less than), GE (numerically greater than or equal to), LE (numerically less than or equal to) and SA (sorts after (MUMPS string sort)).
The Sort element is defined as one or more occurrences of a Field followed by either UP or DOWN (empty elements indicating direction of sort). The Project element is defined as one or more Field elements.
This is the main entry point to read a query and generate a response. This routine reads the query into memory (limit of 16KB), converts all white space into spaces (ASCII 32), and then does a very cursory check on the structure of the query with the extrinsic function $$Prolog^ABS383D (q.v.). It then sets up the parser to start reading at the beginning of the “<Query>” tag (the XML prolog is ignored). It then sets the ^ABS383(“START”|”END”,<Tag>) global nodes to the appropriate MUMPS code to maintain the inherited (START) or synthesized (END) attributes. This code is executed (using the MUMPS XECUTE command for code indirection) when the start tag or end tag is encountered. An empty tag (e.g., <Tag/>) executes both the START and END semantic attributes (if defined).
To process these attributes the application uses XECUTEd code. The parser invokes this dynamic code when it encounters either the start tag (for inherited attributes) or the end tag (for synthesized attributes). The semantic attributes are stored in ^ABS383(“START”|”END”, <Tag Name>) by ReadQry^ABS383D, and are invoked as needed by BldTree^ABS383D. The remainder of ^ABS383D is the routines that implement the actions specified by the semantic attributes.
Finally, ReadQry calls BldTree^ABS383D (q.v., parse the query and do the select operations), Sort^ABS383E (q.v., sort the results of the select) and GenResp^ABS383E (q.v., do the project operations and generate the XML Response), sequentially.
This extrinsic function is used to apply a simple pattern match to an XML Query. It will detect basic errors in the prolog section of the XML (without processing the DTD at all, other than to see that a DOCTYPE is specified). It also checks that there are a <Query> and </Query> tags, and that </Query> is the last tag. This will assure that <Query> is the outermost tag, because BldTree^ABS383D (q.v.) checks for proper nesting of tags.
This short routine is the engine that drives the parsing of the XML query and the searching of the database according to the Select criteria! It sequentially reads tags and performs the designated semantic attribute actions as described above. As it reads the tags, it builds a tree in memory corresponding to the XML element tree. The various inherited and synthesized attributes are generally stored on the tree, also, by the action code. This includes all intermediate and ultimate search results. Currently all of these attributes are left in place for diagnostic and demonstration purposes. In a production version of this code, the end tag for internal nodes should fire an action that deletes the intermediate results from its children after it has synthesized its own attributes. This relatively easy step will greatly reduce the final size of the global (and may accordingly improve speed) in many cases.
Each node of the tree is really a pair of subscripts. The first is used to preserve the tag order, and the second is the tag itself. Other constructions were possible, but this seemed as simple as any.
This routine makes heavy use of subscript indirection and related aspects of indirection in MUMPS. Without this late binding of subscripts (which is virtually unique to MUMPS), this routine would be much longer and would require recursion.
This is a utility extrinsic function called by BldTree to read a tag. It returns both the tag name (through a call-by-reference parameter) and a “mode.” The mode is 1 for a start tag (e.g., <Tag>), 2 for an end tag (e.g., </Tag>), or 3 for an empty tag (e.g., <Tag/>). Other modes that may be returned are 0 (end of file encountered — no more tags) and –1 (improperly constructed tag).
This is a utility extrinsic function called by semantic action code to read the content of a tag (it is called when the start-tag is encountered). The only element that can have content in an XML Query is the Expr element, so this routine is called only when an <Expr> tag is found.
This utility extrinsic function extracts the file number from the long form of the file element. It is called by semantic action code when a </File> tag is encountered.
This utility extrinsic function extracts the field number from the long form of the field element. It is called by semantic action code when a </Field> tag is encountered.
This utility simply copies the child’s attributes up to the parent. This is called in a couple of semantic actions where the processing all occurs lower down, and a synthesized attribute identical to its child is required.
This utility brings the union of the attributes of its children (minimum two) up to their parent. It is used to do the list union when an </OR> tag is encountered.
This utility brings the intersection of the attributes of its children (minimum two) up to their parent. It is used to do the list intersection when an </AND> tag is encountered.
N.B.
There is no function for the handling of NOT elements. The NOT tag does only changes the Boolean sense of a flag (NotFlag) for its child. When an <AND> or <OR> tag is encountered, if the NotFlag is set, the tag is changed in the tree (to <OR> or <AND>, respectively) and the not flag is preserved. The <SAtom> tag also preserves the NotFlag. When a relational operator (EQ, NE, GT, LT, GE, LE, SA) is encountered, if the NotFlag is set, the relational operator is changed in the tree to its opposite (NE, EQ, LE, GE, LT, GT, SB, respectively). Note that SB (Sorts Before) is not available as a relational operator to the user, but semantic actions are defined for it in case it is created by a NOT. The behavior described above implements DeMorgan’s Laws1 and pushes the negation down to the select atoms without changing the logical result. This provides enormous time and space efficiency!
This routine is called when the </SAtom> tag is found. It extracts the field number, the relational operator (possibly modified by a NOT) and the target value from its children and then calls the appropriate routine for the relational operator (the routines have the same name as the relational operators) to perform the actual database Select operation. The matches are stored as a synthesized attribute on the <SAtom> node.
EQ, NE, LT, GT, LE, GE, SA, SB^ABS383D
These eight routines actually perform the database select operations on the FileMan file. In every case except NE (where no advantage can be gained), the routine checks for the existence of a cross-reference (index) for the file on field being selected. If a cross-reference is available, it is used to greatly speed up the selection (especially on an EQ operation). If no cross-reference is available (or if the relational operator is NE), the entire file is traversed to perform the selection.
This deceptively short routine performs all the sorting of the results of the selection criteria. It actually generates a series of lines of MUMPS code (in Task(Step)) that are then called sequentially for each record in the result of the Select operation. These lines of code cause an array to be generated with subscripts from each sort criterion (in order). This is the way sorting is done in MUMPS, as array nodes are always kept sorted by MUMPS. Traversal of a MUMPS array is always sorted by subscript order. Because these are string subscripts, there is no easy way to reverse the ordering for “<UP>” sorts (as could be done by negating a numeric value). Thus the direction of traversal of each subscript level is left up to GenResp^ABS383E (q.v.).
Again, this approach to producing a sorted array of arbitrary depth (i.e., arbitrary number of sort criteria) is very specific to MUMPS. Again, longer code requiring recursion would be required without the late-binding characteristics of MUMPS.
This routine actually produces the XML Response. It first calls GenRDTD^ABS383B (q.v.) to produce an up-to-date DTD for the response, sets a flag for whether a Project operation is defined (if no fields are explicitly projected, all top level fields will be output), and writes the XML prolog and opening <Response> tag for the response file.
At this point, GenResp must traverse the array set up by Sort^ABS383E (q.v.), going in the appropriate direction (up or down) at each subscript level. The routine generates a single line of MUMPS code that contains multiple nested FOR loops (one for each Sort criterion) that traverse the Sort array appropriately. For each record that is extracted, it calls Output^ABS383E to actually write out that records data. Once again, this uses the MUMPS late-binding to simplify the code and eliminate the need for recursion.
This routine actually generates the output for each record in the XML Response. It writes the tag corresponding to the file this record is from (with the IENS attribute set to indicate which record). It then calls the FileMan routine GETS^DIQ to extract the data for all the desired fields (either from the Project operation if specified, or all top-level fields if no Project is specified). It then traverses the results of the GETS call and generates the appropriate field tags and content for each field. The content is checked for “&”, “<”, “'” or “"” characters and they are first transformed to “&”, “<”, “'” or “"”, respectively, to avoid problems with the resulting XML. Finally the closing file tag is written.
Within the limitations inherent in the design (i.e., only allowing selection on top-level fields, no use of sub-files, no use of linked files) and the implementation (i.e., a not-very robust XML parser) this project has been a successful “proof of concept.” It allows database queries and responses to be made using an XML interface (for both query and response) without any database-specific code. These generic utilities will work on any VA FileMan database and allow it to be queried efficiently and effectively. Response time to the queries compares favorably with print requests done entirely within FileMan. It should be notable that this package is more flexible (at a cost in efficiency) than current FileMan routines, which requires that the sort and selection criteria match.
There is obviously a lot of room to expand on this project. In its most general sense, it becomes an entirely separate interface for FileMan databases. In addition to queries and responses, database sets and kills and edits should be available through an XML interface. The ability to use fields from sub-files, super-files and linked files would also extend the capabilities significantly. There are a number of optimizations that can be made at the level of semantic analysis of the parsed Query tree, as well. I see no impediment (other than time) to doing all of this, and hope to undertake this expanded project in the near future. When completed this may be the first database system with a complete and generic (not application-specific) XML interface — I am aware of no other database system that offers this at such a generic level.
As a measure of the merit of the project, I should note that I have discussed this with a (recently former) member of the FileMan development team at the VA and a couple of other high-level programmers working for the VA, and they are very excited about the prospects of this project. I am hopeful that there may even be some grant money available to pursue this project.
A description of the database used for all the samples is given below. Following that, each example includes a brief description of the query and response, and links to the XML query (both as the XML file itself for those with an XML capable browser, and as an HTML representation of the XML) and the resulting XML response (ditto). Also included is a (lightly annotated) listing of the contents of the ABS383 global where all the processing takes place. The XML queries were all hand crafted (a fairly trivial exercise). The XML response was generated in each case by a call to ReadQry^ABS383D.
For demonstration purposes, we are using a simple database of various data about the 50 states and the District of Columbia. Not every state contains every field (some states do not have a State Song, for instance, and the District of Columbia has relatively few of the fields defined). Not every field is unique (seven different states have the cardinal as their State Bird). Some fields are free text, some are numeric, and one (coastline) is a set-of-codes (Atlantic/Gulf, Pacific, or Landlocked). The data dictionary (FileMan's condensed view) shows the fields and cross-references. This same database is used for all of the samples. All of the queries are based on the same Query DTD and Response DTD that were generated by GenQDTD^ABS383C and GenRDTD^ABS383B, respectively.
This example shows how you can query the database to find out how many of the ten most populous states are on the coast, and how does their population ranking compare to their areal (total area of the state) ranking? The selection criteria are that the POPULATION RANK field is less than or equal to ten, and that the COASTLINE field is either ATLANTIC/GULF or PACIFIC. The latter criterion could as easily have been COASTLINE not equal to LANDLOCKED or even a NOT of COASTLINE equal to LANDLOCKED. All would produce the same results. The query is most efficient as written (particularly if there are a lot of records in the database), because the COASTLINE field is cross-referenced, and an equality test can use the cross-reference to only access matches, whereas a non-equality test (which the other criteria would produce) requires a traversal of the database. The sorting criterion is simply the state name in alphabetical order. The fields projected are name, population rank, areal rank and coastline.
Contents of ^ABS383 after response
This example addresses the following conjecture: The cardinal and the western meadowlark are the most common state birds. Do any of the states with these birds also have the same state tree? The selection criteria are BIRD equal to Cardinal or to Western Meadowlark or to Western meadowlark (because the data was inconsistently entered). This example shows the use of OR with more than two children the result is the union of all of the children. The sorting criteria are the state bird, and within in that, the state name. This demonstrates the use of nested sorting criteria. The projected fields are the state name, state bird and state tree. Note that cardinals apparently like tulip poplars, and those western meadowlarks like cottonwoods!
Contents of ^ABS383 after response
This example answers the question “What are the State Mottoes of the 13 original colonies?” The selection criterion is RATIFY ORDER is less than or equal to 13 (i.e., the original 13 states). The sort criterion is the state name. Note that there is no common field between the select and sort criteria. This is something not allowed by conventional FileMan output routines, which closely ally the selection and sorting. The projected fields are name and motto.
Contents of ^ABS383 after response
This example shows how one can transport an entire database with this tool. The selection criterion is that the NAME sorts after the empty string. In MUMPS, every string sorts after the empty string, so all states with names will pass the selection criterion. Since NAME is a required field, this means every entry in the database. The sort criterion is to list by names in alphabetical order. There is no projection criteria specified, so the default of listing every top-level field (i.e., every field in this database) is used. Thus every field of every record is listed.
Contents of ^ABS383 after response
[1] DeMorgan’s laws state that ~ (A⋀B) ≡ (~A)⋁(~B) and similarly ~ (A⋁B) ≡ (~A)⋀(~B).
[2] FileMan “files” are hierarchical databases, not relational databases. In particular, fields need not be single-valued (as in a relational database). Multi-valued fields are handled as “subfiles” — a field that is itself a FileMan file. Note that fields of subfiles may, in turn, be (sub-)subfiles, without explicit limit on the depth of the hierarchy. In general practice a depth of two or three is fairly common, but depths beyond five are almost unknown.