Comments on: Developer Question: External Content in Software Repository?

By: neb

neb — Tue, 17 Jun 2008 17:28:57 +0000

Storing the full latest copy would be good for recoverability. However, Subversion stores its diffs the other way around. According to the (excellent) user manual:

"FSFS writes the latest version of a file as a delta against an earlier version..."

By: Todd

Todd — Tue, 17 Jun 2008 16:48:03 +0000

Interesting. As you have described it there are benefits and trade-offs to each approach. Storing incremental data saves space, but I imagine that reconstructing a file incrementally would become process intensive as the history grew.

It probably stores the most recent version as a whole copy and then each historical reference describes what you change in the current version to get back to that historical version. Access would be slow for historical versions, but good for current versions.

I’m sure looking it up would answer the question definitively, but it is more fun to speculate.

By: neb

neb — Tue, 17 Jun 2008 14:07:17 +0000

Good points. Regarding binary storage space, ClearCase is certainly a beast of a revision control system; I have no idea how it actually stores data. CVS just keeps a copy of each new binary file version.

Subversion uses a binary diff algorithm to more efficiently store just the bytes that have changed. The efficiency is dependent on how the binary changes from one version to the next.

For example, does a small change to a large image allow Subversion to store just a few new bytes, or does the image’s file format cause extensive restructuring of the data, forcing Subversion to store (almost) a whole new copy?

I wonder if there are any statistics on how well this performs…

By: Todd

Todd — Mon, 16 Jun 2008 18:28:01 +0000

As illustrated very nicely by the ClearCase trainer that came here a few years ago, CM systems are basically three-dimensional filesystems, where the third dimension is time. Your content has a direct chronological tie to your implementation, so I see nothing wrong or impure about putting content in the repository.

The only objections would be:
1. Philosophical – from a purist standpoint it doesn’t seem the right place for data. A database seems more appropriate, but ultimately not a good solution. However, a filesystem is, in fact, a database. A 3-D database is an good place for data that changes over time.
2. Storage – Keeping every revision of data could eat up disk space.

I would not consider these strong enough objections to rule out your simple solution: put it all in the repository.