Discussion: Parent file format

[adrybakov]

Topic: Human-readable or binary? Structured or custom?

[adrybakov]

Here is a list of programming languages that shall be considered, based on the surveys

• C
• C++ (in particular C++20)
• Fortran (in particular Fortran90)
• Go
• Matlab
• Python (probably only Python3)
• Rust

Please mention your language if it is not listed

[mducle]

I think that a structured plain text format is preferable because:

• Most of the language listed have libraries which can deserialise / parse such a format (such as json or xml) and this will make implementation much easier than using either a structured binary format like hdf5 or a custom binary format.
• The file can be inspected without any software even if it is not designed to be human readable.

The disadvantage of a text format is that:

• It will be slightly slower / less performant to read in.
• It might not be able to store floating point numbers to the full 64-bit precision (~16 significant digits), depending on string truncation.

However, I don't think that any programs will need to read in the format as part of a tight loop so the format's performance will not be on the critical path; and we can specify that the reference reader/writer write out full-precision numbers.

[adrybakov]

Reflecting on some of your arguments:

Speed

I agree that the speed of reading (or writing) is not the main concern compare to the time of calculations for most of the tools. Perhaps there are use-cases that I am not aware of.

Size

Size of the file might be an issue if one would like to compose a database of spin Hamiltonians. In that case we could optimise in two places: file format itself (.something) and how the parameters are stored by themselves (for instance whether to store i -> j only or both i -> j and j -> j in case of bi-linear interaction). Note that for tri-linear interaction there are 6 equivalent "bonds", 24 for quad-linear and so on (n! for n-linear).

Higher order parameters

The problem of representation of the multidimensional arrays in any txt based-format. I am not aware how well is that handled in the .hdf5, for instance, though.

Floating point

I agree that it is cumbersome, but still any writer can support arbitrary amount of significant digits. Moreover, in my opinion this is out of the scope of the format, as we need to specify how many numbers and in which order to print, but the precision should be decided on the tool's (that produce the file) or user's side. Some might want to default to machine precision, while some might want to base it on the physical relevance, etc.

[adrybakov]

• Most of the language listed have libraries which can deserialise / parse such a format (such as json or xml) and this will make implementation much easier than using either a structured binary format like hdf5 or a custom binary format.

On the other side a structured text format requires a dependency library for reading it in the same way as a structured binary one requires. From that perspective they are identical.

[adrybakov]

• The file can be inspected without any software even if it is not designed to be human readable.

That seems to be the main concern, based on the surveys. In that case if we decide for the format to be human-readable, then the choice is between structured txt or custom txt, the former requires a dependency library in each language. The latter can be read directly by any language, knowing the format specs alone (i.e. without any additional dependency).

Implementation-wise, I think that it is easier to use custom txt format, as then more tools will be ready to use it (does not require the third-party dependency) and the reference implementations of reading/writing will not need to be concerned about the versions and potential bugs in the third-party libraries for the structured txt formats.

Moreover, if the human readability is a secondary goal, then the custom format can be optimised for that as well.

[richard-evans]

Implementation-wise, I think that it is easier to use custom txt format

I agree with this route (keeps things simple), as json etc are a nuisance to read/write. Also once the format is specified we can do a binary equivalent and conversion tool for performance, which works as long as staying within mac/linux/windows.

[lucas-wilkins]

Easy bit first: binary or text

There's a couple of things to consider here:

Speed of reading/writing: compared to the computational expense of the calculations that will produce/consume the information in the files, speed not a huge consideration. Even if the files get really big, the computational expense of working with them will also grow (and grow faster). I consider this to be a very low ranking consideration.

File size: The amounts of data we're considering are relatively small, I'd say that util we're talking about gigabytes of data its not an important consideration. Practically, we are thinking about kilobytes for most purposes.

Given how low priority speed and size are in this application, I think it is fairly safe to rule out binary files for the foreseeable future. Their opacity makes them very hard to work with for both users and developers, and their advantages are not really relevant to our use case. Should we have a need for binary files in the future, it will be because the scope of the work being done with them has shifted radically, and it would probably be time to think about a new file format anyway.

This leaves only the choice of text file to consider.

When talking about the "base format" we're talking about the syntax used throughout the file, this will include how different fields are deiminated, how to represent numbers, how strings are treated (including awkward things like special/escape characters, line endings and quotations), how the grouping and sizes of fields is determined, and how additional contextual information is linked to individual fields. For example, JSON uses curly brackets and string keys to denote structured groupings of data, square brackets for lists, etc. A file format like the 3D model data format .PLY uses a header that specifies the sizes of fields, so that the location in the file can be inferred from these sizes along with further rules about ordering and formatting (a strategy more common in binary files). The former is what I would consider "structured" in the context of this conversation, and the latter "custom". Text based 3D model formats provides a number of other examples of what I would consider "custom text" (an their number is somewhat indicative of the problems with this approach).

My own opinion is that it would be very unwise to adopt an ad hoc syntax. We should adopt a tried and tested standard that assures maintainability and extensibility in the future.

"custom text" files suffer from various issues:

1. As the complexity increases, parsing the files gets messier and less reliable.
2. Individual implementations can potentially be quite esoteric.
3. Everyone them needs to implement the same, low level functionality, which would probably be best handled by well established, standardised, libraries.
4. A consequence of the previous, and the most concerning: As individuals write the their own readers and writers from the ground up, there is a growth of badly formed files. People tend to only support the parts of files that are relevant to their own particular use case, and ignore the parts that don't matter. Different people have different approaches to sanitisation, and dealing with edge cases. This often means that parts of files are not properly checked and leads to incompatibility between different software packages that should putatively support the same files. When designing to support a community, rather than in a competitive commercial setting, this can be particularly destructive.

Structured files using well established formats mitigate these risks by having well tested parsers for the core syntax, committees of people working on assuring correct implementation, and large user bases that provide real-world testing and feedback. They also typically have tools for validating files, and schemas that be used to describe and check the validity of specific usages.

Assuming I've talked everyone out of the "custom text" idea, the question is then which "structured text" format to use.

The obvious choices to consider are XML, JSON, YAML, TOML and STAR.

STAR only gets a mention because its what .CIF files use, but I don't consider it to be a serious candidate. It's a niche and dated format, and by the standards of other formats it is poorly supported and hard to read both by humans and computers. The others have a couple of further advantages

1. There are schema languages for describing files that use them, and accompanying validation tools.
2. There's support in all the languages we might want to use, even FORTRAN.
3. It's easy to find people who are familiar with them, and there are lots of easily obtainable resources for people who are not.

So, XML, JSON, YAML and TOML are what's left, and can be split into two groups:

XML and JSON: These formats and parsers are used by literally billions of people every day and have existed for a long time. Whilst there might be concerns about bugs in third-party libraries, is that really a risk compared to whatever new problems we would create with our own system? Both XML and JSON are natively supported by many modern languages, including python and go. For C++, they're supported by Boost, which might as well be its standard library.

YAML and TOML: Both are relatively new, though now standard in many places. They don't have the widespread use of the web standards XML and JSON, but they're still very widely used. They both are a bit easier to hand craft than JSON or XML because they don't have open/close tags/braces, instead using location and whitespace.

Some specifics of each technology:

XML: In favour of XML, it has some very nice features for extensibility through its namespace system. Consider how (X)HTML, SVG, and other SVG variants interact. It's possible to embed different kinds of structures in other formats and extend it for new applications without affecting existing standards. They all have to XML standards though. Against it is its verbosity, overhead, and the annoyance of having to open and close lots of tags. If future extensibility is a serious consideration, XML might be worth considering.

JSON: Very human readable, widely supported and established. It plays nicely with object oriented languages and is very lightweight.

YAML: Has a cross-referencing system that would be very useful for the kinds of data we're working on, this is absent from all the others mentioned.

TOML: More minimal than YAML, and lacks said cross-referencing. Probably best suited to its main use case, configuration files.

If I was choosing myself I think I would go with YAML, probably followed by JSON as close second, XML, and lastly TOML.

Other things:

Ease of writing readers/writers: This is both highly subjective and language dependent. Personally, I find having the gruntwork of serialisation and deserialization done by a package to make things much easier. As I mentioned before, "custom text" file formats often start out looking simple, but then turn into a complicated mess later on. It's not that "custom text" is simpler, it just seems so because its unexamined. All the problems that are solved by the seeming complexity of "structured text" are still there, they're just something we'd have to firefight in the future.

Another thing: Given the current evolutionary trajectory of scientific computation, we should probably also factor in the ability for the data stored in files to be integrated with databases in the near future.