Scala Bazaars Reference Manual

1. Overview

Scala Bazaars supports Scala enthusiasts in sharing the software they create with each other. The community is strengthened when people can help each other out by sharing code. A programming language needs an archive of library code to be its most useful.

Scala Bazaars has several features specialized for this kind of community:

• Since programming efforts in the community are distributed around the world, the system must allow loose, decoupled collaboration. Participants are not required to wait for each other nor for any centralized authority.

• Since subgroups of the community have their own needs—just consider groups doing commercial development!—the system should support subgroup-specific access control policies.

• Since many open source projects are updated frequently, the system must allow conveniently updating components as new versions become available.

• Since, even in the common open-source group, there are different tolerances for stable versus new offerings, the system supports having multiple public servers with different update policies

Scala Bazaars is most closely related to Debian's APT, though it shares spirit with systems including: YUM, FreeBSD ports, CPAN, CTAN, SqueakMap, and Fink. The main difference from APT is that Bazaars tries to make it not only possible, but convenient, for subgroups to run their own servers with their own access policies.

The present document is a reference manual for Scala Bazaars. It is complete at the expense of readable. It does include full specifications for all components of the system, including the command-line interface, all file formats, and the network protocol. Those wanting to use the system should first check whether a tutorial exists for the specific need. Those wanting a more philosophical look at the system, including comparisons to other systems and to the literature, should look at the Package Universes Architecture document. It is available at the Scala Bazaars home page:

http://lamp.epfl.ch/~spoon/sbaz/

2. Architectural Concepts

This section describes the architecture as a whole. Each component of the architecture is described in detail in a later section.

Users of the system have a local managed directory that includes content supplied by the system. In general, users should not modify content in a managed directory except via a Scala Bazaars system tool. The one exception is the config directory, which is explicitly intended for user modification and should not be modified by the Scala Bazaars tool.

Sharable content is held by packages. A package can be installed into a local managed directory, in which case all of its included files are extracted into the managed directory. A package can also be removed from a local managed directory, causing all of the files it installed to be removed.

Each package has a name and a version. Packages with the same name may be substituted for each other, but packages with larger version numbers are preferable in some sense.

A bazaar is an evolving set of packages. Each managed directory is associated with one bazaar. The system makes it convenient to choose and install packages from a managed directory's own bazaar, but not from other bazaars. The set of available packages evolves as the community associated with the bazaar shares and retracts packages over time.

Packages may depend on other packages. All dependencies are resolved within a single bazaar. The system does not allow installing or removing packages in a way that a managed directory's dependencies become unsatisfied.

On the whole, this approach gives convenient upgrades (simply grab the newest version of everything), loose coupling (you can wait for a while to upgrade), and convenient dependency management (requesting the install of a package automatically installs the packages it depends on). The cost of this approach is that packages must be posted separately to each bazaar they are useful in, but as argued in the architecture document, that cost appears to be ephemeral.

Bazaars may be combined in a variety of ways. Each bazaar has its own access policy, and thus combinations of bazaars with different access policies can yield a wide variety of useful configurations.

Bazaars do not hold the packages themselves. Instead, a bazaar holds package advertisements, and the packages themselves must be posted separately. Each package advertisement includes a URL referencing the associated package.

3. Bazaar Definitions

There are a number of kinds of bazaars. They are each described here along with the XML used to formally describe each kind.

4. Access Control

Many organizations that use Scala Bazaars do not want to allow arbitrary accesses from unknown users. Scala Bazaars includes a simple access control system to limit usage as appropriate for the organization.

The approach is based on keys. A bazaar server has a list of keys that it considers valid. Each request to a server must either include a valid and sufficient key, or it must be in the subset of requests configured as open access for the particular server.

Each key is associated with a message pattern denoting the subset of requests it can authorize. The following message patterns are supported:

<read/>
    

<edit nameregex="sbaz.*">
    

<keyedit/>
    

A key includes three pieces of information: a request pattern, a short, descriptive text, and a string of decimal digits. The descriptive text is human-readable and allows keys to be associated with principles in an external access control system. Typical uses are names, email addresses, usernames, and employee ID's. The string of decimal digits is typically randomly generated and must be unguessable in order for the server to remain protected.

In XML, a key looks like:

<key>
  <messages>
    <edit nameregex="lex-.*"></edit>
  </messages>
  <description>lex@lexspoon.org</description>
  <data>26405971450520721508638067086623258803</data>
</key>

The keys-based access control system of Scala Bazaars is both usable directly and supports, it is hoped, a variety of useful access policies. For simple access policies, the server administrator (or someone to whom they delegate) can manually manage and distribute keys. Note that key data can be transmitted via email and web sites, thus allowing distribution of keys to piggyback on existing secured infrastructure.

For organizations that have too many users for manual management of access, the keys approach supports writing a simple web server that can manage and distribute keys in an arbitrary fashion. For example, one could write a web server that gives out keys to people who have a valid login according to an LDAP server; nightly, the server could revoke keys for any users whose LDAP entry has disappeared.

5. Common Configurations

The available bazaar types and access-control policies described above can be combined to form a number of useful configurations. This section outlines several of them.

6. Packages and Package Advertisements

7. Versions and Dependencies

Every package has a version. A version is a string that may contain ASCII numbers, digits, and the following symbolic characters:

.-+/?,&!@#$%^&*

Versions are totally ordered by the following algorithm. To compare two versions, first break them into a number of subsequences, each maximally long, where each subsequence only contains decimal digits, only contains alphabetic characters, or only contains symbolic characters. Compare the two sequences lexicographically. Two numeric subsequences are compared numerically, two alphabetic subsequences are compared in ASCII order, and two symbolic subsequences are compared in ASCII order as well. For two subsequences with different kinds of characters, the order is: alphabetic, then numeric, then symbolic. For some examples, the following versions are sorted in order:

1. (empty string)

2. abc

3. abcd

4. abd

5. 1

6. 1.1

7. 1.1a

8. 1.1a2

9. 1.1a100

10. 1.1.5

11. 1.2

12. 1.2.

13. 2

14. 12

A package dependencies can currently only have hard dependencies to specific other package names. For those dependencies to be satisfied, a package must be installed for each of the package names specified. Here is package description for a package that depends on package "scala2-library" and "sbaz":

<availablePackage>
  <package>
    <name>base</name>
    <version>1.7</version>
    <depends>
       <name>scala2-library</name>
       <name>sbaz</name>
    </depends>
    <description>This package depends on the basic packages that all
managed directories must include.  Each of these packages is either
essential or is very commonly used.  </description>
  </package>
  <link>http://lamp.epfl.ch/~spoon/scbaztmp/base-1.7.sbp</link>
</availablePackage>

A somewhat richer set of dependencies is planned, including provides, suggests, alternative dependencies, and minimum version numbers.

8. Command-Line Interface

9. Suggested Directory Layout

Each sbaz repository has its own informal standards for the directory layout within a managed directory. This section documents the emerging layout used in the main Scala bazaar. It is the standard for that repository, and it might serve as a guideline for other repositories.

• lib — Any jar file(s) associated with the package, especially those that are meant as libraries to be usable by other programs in the bazaar. Jars placed in this directory are particularly easy to access, because both the generic scala script and most of the tool-running scripts in bin will automatically load classes from any jars in lib. Normally, jar filenames in this directory do not include any version number. A typical filename is sbaz.jar.

• src — Source code for the package. This source code should be presented in a way that IDE's can find the code easily. Thus far, packages install directories under src that parallel the dotted package paths from the Scala code. For example, class sbaz.clui.CommandLine is found in a file named src/sbaz/clui/CommandLine.scala.

• bin — Command-line runnable scripts. These are most easily created via the Scala ant tasks. As a special case, the sbaz tool will make files within bin be executable on platforms where that makes sense.

• config — Configuration files. Packages should not include any files in this directory! They should look in this directory for optional user configuration. If there is a single configuration file, it can be included directly in the config directory, e.g. with a name like config/sbaz.properties. If there is more than one configuration file for a package, then the files should be located in a subdirectory of config named after the package name. For example, the sbaz package could include its configuration files in a directory named config/sbaz/.

• misc — Arbitrary files not included in any of the above. All such files for a package should be included in a directory named after the package. For example, the sbaz package includes miscellaneous files in the directory misc/sbaz/.

10. The scala.home property

By convention, the scala.home system property is used to point at the root of the current managed directory. Command-line scripts installed in bin directories should normally set this variable. The conventional choice is to set scala.home to the SCALA_HOME environment variable if it set, or otherwise to the parent directory of the bin directory the script is located in.

Programs intended to run within a sbaz directory can use this property to locate any files they may need. For example, sbaz itself uses the following code to find a user-specified settings file:

val home = System.getProperty("scala.home", ".")
val propFile = new File(new File(new File(home),
                        "config"), "sbaz.properties")

11. Managed Directory Layout

Most of a managed directory is populated by the contents of packages. The single directory meta is reserved for the system to track information about the managed directory. The meta directory has the following contents:

• universe — a file holding the XML description of the bazaar.

• available — a list of package advertisements available within the bazaar. The file is an XML document whose top level node is <availableList> and whose subnodes are package advertisements.

• cached — a subdirectory holding a cache of package files that the tool has downloaded.

• installed — information about installed files. The format is described below.

The installed file holds information about the packages that are currently installed in the managed directory. Its format is mostly straightforward:

<installedlist>
  <installedpackage>
    <package>
      <name>base</name>
      <version>1.7</version>
      <depends>
        <name>sbaz</name>
        <name>scala2-library</name>
      </depends>
      <description>A package that depends on the basic, necessary packages</description>
    </package>
    <files>
      <filename isAbsolute="true" isFile="true">
        <pathcomp>lib</pathcomp>
      </filename>
    </files>
  </installedpackage>

  <installedpackage>
    <package>
      <name>sbaz</name>
      <version>1.10</version>
      <depends></depends>
      <description>The command-line interface to Scala Bazaars.</description>
    </package>
    <files>
      <filename isAbsolute="true" isFile="true">
        <pathcomp>bin</pathcomp> 
        <pathcomp>sbaz</pathcomp>
      </filename>

      <filename isAbsolute="true" isFile="true">
        <pathcomp>bin</pathcomp>
        <pathcomp>sbaz.bat</pathcomp>
      </filename>

      <filename isAbsolute="true" isFile="true">
        <pathcomp>misc</pathcomp> 
        <pathcomp>sbaz</pathcomp>
        <pathcomp>scala2-library.jar</pathcomp>
      </filename>

      <filename isAbsolute="true" isFile="true">
        <pathcomp>misc</pathcomp> 
        <pathcomp>sbaz</pathcomp> 
        <pathcomp>smartrun.mswin.template</pathcomp>
      </filename> 

      <filename isAbsolute="true" isFile="true">
        <pathcomp>misc</pathcomp> 
        <pathcomp>sbaz</pathcomp> 
        <pathcomp>smartrun.unix.template</pathcomp>
      </filename> 

      <filename isAbsolute="true" isFile="true">
        <pathcomp>lib</pathcomp>  
        <pathcomp>sbaz.jar</pathcomp>
      </filename></files>
    </installedpackage>
  <installedpackage>
</installedist>

Notice that an installed package is the combination of a package plus a list of files. Each file in the list of files is represented with a filename whose attributes determine whether it is an absolute file (never) and whether it is a file (versus a directory). The contents of the filename are the sequence of path components of the file, relative to the managed directory's root.

12. Common Problems