API Descriptions

 

Introduction

libpsirp provides an interface towards the in-kernel blackboard functionality. For maximum compatibility, applications should use libpsirp rather than calling syscalls directly. However, the API is still constantly changing as new features are developed.

This page describes the API for C, Python, and Ruby.

Please note that Blackhawk is a research prototype, so the documentation on this page is not complete and not always up to date. Reading the libpsirp.h file, for example, is therefore also highly recommended! Questions can be asked on the psirp-code mailing list.

Quick API overview

Publication objects
  • Placeholders for pointers to data and metadata
  • E.g.: psirp_pub_t and Publication
Blackboard operations
  • Create, publish, and subscribe publication objects
    • create(), publish(), subscribe()
Accessors
  • Getting data and attributes of publication objects
  • E.g.: identifiers, length, etc.
Events
  • Enable dynamically updated publications
    • kqueue, kevent()
Auxiliary functions
  • Conversions, debugging, etc.

C API

Data types and defintions

Psirp identifiers

Currently all psirp identifiers are defined to be 32 bytes long (256 bits).

#define PSIRP_ID_LEN  32
#define PSIRP_FID_LEN PSIRP_ID_LEN

struct psirp_id {
  unsigned char id[PSIRP_ID_LEN];
};
typedef struct psirp_id psirp_id_t, psirp_fid_t;

Psirp publications

From user space point-of-view, all publications are represented by a data blob (binary large object). The application receive only a pointer to this blob and will have to use libpsirp’s accessor functions to retrieve and/or alter information in the data blob.

struct psfs_pub; // contents not defined for user space
typedef psfs_pub *psirp_pub_t;

NOTE: psirp_pub_t is a pointer, whereas psirp_id_t is not!

Blacboard functions

Create publication

int psirp_create(int size, psirp_pub_t *pub);

Allocates memory for the calling process and initializes publication structure.

Arg pub: Pointer to psirp_pub_t. If create succeeds, the argument will point to a newly allocated and initialized structure.
Arg size: Size of the data part of the publication to be allocated.

Ret 0: Allocation successful
Ret EINVAL (22):
(…)

Allocated memory must be freed by calling psirp_free() or by using NOTE_UNMAP with kevents. psirp_free() also frees the psirp_pub_t data structure, while NOTE_UNMAP only unmaps the blob from the subscribing process’ memory space.

Publish publication

int psirp_publish(psirp_id_t *sid, psirp_id_t *rid, psirp_pub_t pub);

Stores (= publishes) the publication to the local blackboard. If an old publication with the same RId and SId exists, the new publication will be a assumed to be a new version of the original publication. Processes listening to new publications (in correct scope) will receive notifications (NOTE_PUBLISH).

Arg sid: Scope Identifier
Arg rid: Rendezvous / P:L identifier
Arg pub: Publication structure allocated by psirp_create()

Ret EINVAL (22): …
Ret 0: Publishing succeeded.

Subscribe publication

Non-blocking:

int psirp_subscribe(psirp_id_t *sid, psirp_id_t *rid, psirp_pub_t *pub);

Blocking:

int psirp_subscribe_sync(psirp_id_t *sid, psirp_id_t *rid, psirp_pub_t *pub, struct timeval *timeout);

For subscribing a publication we offer two alternatives; non-blocking and blocking functions. They both deliver the same results, but non-blocking is guaranteed to fail if local rendezvous cannot immediately be accomplished (i.e. publication does not exist). The blocking variant of subscribe will block at most for the duration specified by the user. If no timeout value is provided, the blocking subscribe will sleep forever or until the publication appears locally.

Once the publication has been successfully subscribed, the calling process may proceed to access the data. Currently, the data should be considered READ ONLY. Remember to call psirp_free() or use NOTE_UNMAP to get rid of the publication.

Arg sid: Scope Identifier
Arg rid: Rendezvous / P:L identifier
Arg pub: Pointer to publication structure. This will be allocated and filled by the function.
Arg timeout: Maximum amount of time the blocking subscribe will wait for publication. If a NULL pointer, forever is assumed. If the structure is initialized as zeros, the semantics of non-blocking function are followed.

Ret ENOENT (2): rid was not found in sid scope. I.e. publication does not exist in the requested scope.
Ret ESRCH (3): sid scope was not found. I.e. scope does not exist.
Ret 0: Subscrbing to publication successful.

Free publication

void psirp_free(psirp_pub_t pub);

Frees publication structure and unmaps memory mappings made by the blackboard.

NOTE! Freeing publication does not imply removal of related information from the blackboard! Currently there is no way of deleting publications from the blackboard.

Accessors

caddr_t psirp_pub_data(psirp_pub_t pub);

Access to publication data.

u_int64_t psirp_pub_data_len(psirp_pub_t pub);

Length of the publication data.

int psirp_pub_fd(psirp_pub_t pub);

File descriptor assigned for the publication. This is to be used together with kevent() to listen for notifications related to the publication. Negative file descriptor value indicates an error.

psirp_id_t *psirp_pub_sid(psirp_pub_t pub);

Scope identifier used to subscribe/publish the publication. Note: This does not reveal in which scopes the publication has been published, only what the calling process already knows.

psirp_id_t *psirp_pub_rid(psirp_pub_t pub);

The identifier of a publication that was used during earlier calls to psirp_subscribe() or psirp_publish().

psirp_id_t *psirp_pub_current_version(psirp_pub_t pub);

Returns current version of the publication. Subscribing to version RIds is not yet supported.

enum psirp_pub_type { PSIRP_PUB_UNINITIALIZED=0, PSIRP_PUB_UNKNOWN,
                      PSIRP_PUB_SCOPE, PSIRP_PUB_DATA };
typedef enum psirp_pub_type psirp_pub_type_t;

psirp_pub_type_t psirp_pub_type(psirp_pub_t pub);

Type of the publication. Mainly useful to check whether the publication is a scope or not.

int psirp_pub_current_version_index(psirp_pub_t pub);

Returns the index of the current version. Usage of this function is not recommended.

int psirp_pub_version_count(psirp_pub_t pub);

Returns the total version count for the publication. Usage of this function is not recommended.

int psirp_scope_rid_count(psirp_pub_t scope);
int psirp_scope_get_rids(psirp_pub_t scope,
                         psirp_id_t **rids, int *rid_count);
int psirp_pub_get_vrids(psirp_pub_t pub,
                        psirp_id_t **rids, int *rid_count);
int psirp_version_get_prids(psirp_pub_t version,
                            psirp_id_t **rids, int *rid_count);

Additional functions

int psirp_idcmp(psirp_id_t *id1, psirp_id_t *id2);
int psirp_idcpy(psirp_id_t *dst, psirp_id_t *src);
void psirp_idzero(psirp_id_t *id);

Conversion functions

These functions convert identifiers in ASCII format to binary format. See below for syntax of ASCII format. The identifier structures need to be preallocated.

int psirp_atoid(psirp_id_t *rid, const char *str);
int psirp_atoids(psirp_id_t *sid, psirp_id_t *rid, const char *str);

Example:

int ret;
psirp_id_t myid;
char mystr[] = "CAFE::ABBA";

ret = psirp_atoid(&myid, mystr);
if (ret < 0) {
   error();
}

Reverse conversion (binary to ASCII):
This functions are NOT re-entrant, i.e. they are not multithread-safe! The ASCII strings that these functions provide are from statically allocated memory that these function overwrite all the time.

char *psirp_idtoa(psirp_id_t *rid);
char *psirp_idstoa(psirp_id_t *sid, psirp_id_t *rid);

Example:

printf("My publication's RID is: %s\n", psirp_idtoa(psirp_pub_rid(pub)));

Example of undefined and erroneous usage:

printf("My publication's SID is: %s and RID is: %s\n",
       psirp_idtoa(psirp_pub_sid(pub)), psirp_idtoa(psirp_pub_rid(pub)));

It is quite likely that the code above will print either the SId or RId twice.

Example of how to do the above right (since you will likely want to print both SId and RId):

printf("My publication's IDs: %s\n", psirp_idstoa(psirp_pub_sid(pub), psirp_pub_rid(pub)));

Some rules of ASCII format for identifiers:

  1. Can be condenced similarly to IPv6 addresses, otherwise 64 hexadecimal digits are expected.
  2. The conversion functions will try to find longest run of double zeros and replace them with ::.
  3. The conversion functions understand a SId-RId separator: /, in which case there MUST be two valid identifiers on both sides of the separator.
  4. Digits (or letters) in identifiers MUST appear in even numbers.

Examples of valid identifiers:

::          (= 0000000000000000000000000000000000000000000000000000000000000000)
25::        (= 2500000000000000000000000000000000000000000000000000000000000000)
CA::FE      (= CA000000000000000000000000000000000000000000000000000000000000FE)
25::/CA::FE (= SID being the second from above, and RID being the third from above)

Examples of invalid identifiers:

58374       (-> too short, must be 64 characters)
1::         (-> uneven amount of digits. Try 01:: or 10::)
/25::       (-> missing SID identifier)

Events

int listen_to_new_versions(psirp_pub_t pub) {
    struct kevent kev;
    int kq;
    int err;

    kq = kqueue();
    if (kq < 0)
        return -1;

    EV_SET(&kev, psirp_pub_fd(pub), EVFILT_VNODE, EV_ADD|EV_CLEAR,
           NOTE_PUBLISH|NOTE_UNMAP, NULL, pub);

    /* Register new event listener */
    err = kevent(kq, &kev, 1, NULL, 0, NULL);
    if (err < 0) {
        return -1;
    }

    /* Listen to new events */
    while (1) {
        err = kevent(kq, NULL, 0, &kev, 1, NULL);
        if (err < 0) {
            return -1;

        /* check kev (e.g. ident, fflags, data) and do something with the pub */
    }

Debugging

To see under the hood of a publication, we provide a debugging access to publication’s meta data. Applications should not use this interface for gathering information, as
the internal structures are subjected to lots of changes. Do not expect these functions to exist later.

void psirp_debug_meta(psirp_pub_t pub);
void psirp_debug_meta2(psirp_pub_t pub);

The former function will try to pretty print the contents of the meta data, while the latter one implements a simple hexdump of the first 1024 bytes of the meta data.

Python API

Python version

Python 2.6 (or greater) is required due to its support for kevents. (Note that the default in FreeBSD 7.x is still Python 2.5.)

install-prereqs.sh installs Python 2.6 by default. But if needed, it can also be installed manually:

pkg_add python26 || pkg_add ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages/Latest/python26.tbz

Starting the interpreter:

python2.6

Executable scripts should start with this line:

#!/usr/local/bin/python2.6

Module usage

Import the module (psirp.libpsirp):

import psirp.libpsirp

or:

from psirp.libpsirp import *

Note: Importing libpsirp_py provides a “raw” API, and importing libpsirp_py_, with a trailing underscore, provides the v0.1 API (to be removed in future releases).

To get information about all classes and functions in the API, the following Python interpreter commands are very useful:

from psirp import libpsirp
dir(libpsirp)
help(libpsirp)
help(libpsirp.Publication)
help(libpsirp.PubSubKQueue)
#...

Especially the new features in v0.3 are currently only documented like this! Please also check the examples provided with the source code.

Classes and functions

The psirp.libpsirp module contains an abstract class called Publication, which includes the following members:

  • buffer: Publication data as a read-write Buffer object (similar to a string or array)
  • len: Publication length
  • sid: SId as a read-only buffer
  • rid: RId as a read-only buffer
  • vrid: Version-RId as a read-only buffer
  • version_index: Current version index
  • version_count: Total version count
  • publish(sid, rid): Publishes the publication
  • publish_(sidstr, ridstr): Same as above, but does ID conversion
  • resubscribe(): Returns a new object with the same SId and RId, but possibly a later version (i.e., the latest one)
  • republish(): Re-publishes the publication with the same SId and RId as before
  • get_vrids(): Returns a list of all (local) version-RIds

Publication instances are either DataPublication, ScopePublication, or VersionPublication objects.
DataPublication objects have the same methods and members as the abstract Publication class.

ScopePublication objects also have the additional accessors listed below, but the publish() method and buffer member have been disabled.

  • rid_count: Number of RIds in the scope
  • get_rids(): List of RIds in scope

Moreover, the API includes the following module-level functions:

  • create(len): Creates a new publication
  • subscribe(sid, rid): Subscribes to a publication
  • subscribe_(sidstr, ridstr): Same as above, but does ID conversion
  • subscribe_sync(sid, rid [, timeout=None]): Subscribes to a publication synchronously. The timeout parameter is optional.
  • subscribe_sync_(sid, rid [, timeout=None]): Same as above, but does ID conversion
  • atoid(a): Hexadecimal string to binary identifier conversion
  • idtoa(id): Binary identifier to hexadecimal string conversion
  • idstoa(sid, rid): Binary identifier to hexadecimal string conversion

Note: These functions raise exceptions (PubSubError and subclasses such as ScopeNotFoundError and PubNotFoundError) if the operations fail.
Note: Publication objects acquired via create() and subscribe() automatically call psirp_free() when they are destroyed (garbage collected).

Because kevents are a very important means of getting notified about publication updates, a class called PubSubKQueue is provided with the API:

  • register(pub [, update=False, unmap=False]): Registers to publication update events for pub
  • unregister(pub): Unregisters pub
  • listen([n=1, timeout=None]): Listens to events. Returns a list containing (ev, pub) elements
  • custom_command(...): Register/unregister any type of kevent on the same kqueue

By using the update and unmap flags together, a re-subscription after an event can be avoided.
In that case, the newest version is automatically mapped to the application and can be accessed through the existing and/or returned pub object.
Note, however, that the old publication is unmapped already when register() is called, so its data or metadata must not be accessed after that.

Simple usage example

from psirp.libpsirp import *     # import the module

p = create(8)                    # create a publication
p.buffer[:] = "12345678"         # fill it with data
p.publish_("::aa", "::bb")       # publish it (SId/RId: 0x00..0xaa/0x00..0xbb)

q = subscribe_("::aa", "::bb")   # subscribe to the publication
print(q.buffer[:])               # print its data
q.buffer[1:4] = 'abc'            # modify it
q.republish()                    # re-publish it with the same SId and RId

Ruby API

The Ruby API is currently less developed than the ones for C and Python.

More information:

require 'libpsirp_rb'
Libpsirp_rb.methods
Libpsirp_rb.constants

Simple usage example:

require 'libpsirp_rb'                                                                                               # import the module

Libpsirp_rb.psirp_rb_publish_string("\0aa"*Libpsirp_rb::PSIRP_ID_LEN, "\0bb"*Libpsirp_rb::PSIRP_ID_LEN, "12345678") # publish a string

spub = Libpsirp_rb.psirp_subscribe("\0aa"*Libpsirp_rb::PSIRP_ID_LEN, "\0bb"*Libpsirp_rb::PSIRP_ID_LEN)[1]           # subscribe to the publication
s = Libpsirp_rb.psirp_rb_buffer(spub)                                                                               # get its data as a read-only string
p s                                                                                                                 # print the string