Hardware Map

Introduction

The Python objects described in Object Model are used in running Python sessions. They must be instantiated somehow. When writing short, standalone Python scripts, these objects can be created directly:

>>> import pydfmux

>>> d = pydfmux.Dfmux(hostname='iceboard004.local')
>>> m1 = pydfmux.MGMEZZ04(mezzanine=1, serial='fmc2_001')
>>> m2 = pydfmux.MGMEZZ04(mezzanine=2, serial='fmc2_002')
>>> d.mezzanines = [ m1, m2 ]

>>> hwm = pydfmux.HardwareMap()
>>> hwm.add(d)
>>> hwm.commit()

>>> d = hwm.query(pydfmux.Dfmux).one()
>>> print(d.mezzanine[2])
Dfmux(u'iceboard004.local').MGMEZZ04(2,u'fmc2_002')

This code block is already somewhat clumsy, with just a single dfmux and 2 mezzanines. In an experiment, we typically need to create a large number (thousands) of a much wider variety of objects:

  • LCBoards containing LCChannels,
  • Wafers containing Bolometers,
  • SQUIDs, and
  • ChannelMappings to associate these structures and readout channels.

Formerly, experiments used combinations of CSV, JSON, XML, and Python documents to generate complete hardware descriptions for experiments. In this document, we introduce a similar scheme using YAML documents.

A Simple Example

To reproduce the above hardware map in YAML, create a file called “hwm.yaml” with the following contents:

!HardwareMap [
   !Dfmux {
      hostname: "iceboard004.local",
      mezzanines: [!MGMEZZ04 {serial: "fmc2_001"}, !MGMEZZ04 {serial: "fmc2_002"}]
   }
]

This hardware map may be loaded in Python as follows:

>>> import pydfmux
>>> hwm = pydfmux.load_session(open('software/examples/example1.yaml'))

This hardware map is identical to the one created in the introduction:

>>> d = hwm.query(pydfmux.Dfmux).one()
>>> print(d.mezzanine[2])
Dfmux(u'iceboard004.local').MGMEZZ04(2,u'fmc2_002')

A few notes on YAML syntax:

  • YAML is a superset of JSON. So, { key1: value1, key2: value2 } expresses a mapping; [ value1, value2, ...] expresses a sequence. The YAML above has been deliberately formatted to look familiar to JSON users; below, we’ll introduce other ways to format mappings and sequences that are probably better suited to describing large hardware maps.

  • Tokens beginning with exclamation marks (!HardwareMap, !Dfmux, MGMEZZ04) are tags, which tell the YAML parser to treat the following element specially. In the case of !Dfmux and !MGMEZZ04, the parser will automatically convert dictionaries into pydfmux.Dfmux and pydfmux.GMEZZ04 objects. In the case of !HardwareMap, the parser converts a list of hardware map entries (here, a single pydfmux.Dfmux) into a fully-fledged HardwareMap object.

  • The top-level object in our “hwm.yaml” was a HardwareMap. It could have been anything (a dictionary or list, perhaps containing !HardwareMap-tagged objects). This can be useful when describing more than just a HardwareMap and the objects it can contain. For example:

    validity: [
   !!timestamp "2014-10-05t21:59:43.10-05:00",
   !!timestamp "2014-10-06t21:59:43.10-05:00"
]
hardware_map: !HardwareMap [
   !Dfmux {
      hostname: "iceboard004.local"
      mezzanines: [!MGMEZZ04 {serial: "fmc2_001"}, !MGMEZZ04 {serial: "fmc2_002"}]
]

    As with !Dfmux, the !!timestamp tag announces a particular type and triggers special behaviour during parsing; in this case, the !!timestamp tag is part of the YAML specification and automatically converts the datestring into a Python datetime.DateTime object.

    This suggests YAML documents can be used to describe more than just hardware maps, e.g. test scenarios, quality-control configurations, et cetera.

Mappings and Sequences

The following two YAML snippets are identical. We begin with YAML code that is deliberately formatted to look like JSON:

!HardwareMap [
   !Dfmux {
      hostname: "iceboard004.local"
      mezzanines: [!MGMEZZ04 {serial: "fmc2_001"}, !MGMEZZ04 {serial: "fmc2_002"}]
   }

Particularly in larger hardware maps, the square and curly brackets can clutter up the presentation. YAML provides a number of alternative ways to specify mappings and sequences, such as:

# YAML also lets us write comments!
!HardwareMap
   - !Dfmux
      hostname: iceboard004.local
      mezzanines:
         - !MGMEZZ04 { serial: fmc2_001 }
         - !MGMEZZ04 { serial: fmc2_002 }

Note the following:

  • Unlike JSON, YAML does not require strings to be quoted unless they would otherwise be ambiguous. So, strings like the dfmux hostname iceboard004.local may be written plainly.
  • Mappings are now indicated with key: value pairs and scoped using indentation, instead of curly brackets.
  • Sequences are now indicated using a single dash, and scoped using indentation instead of square brackets.
  • The second encoding (which does not surround sequences or mappings with delimiters [] or {}) may produce fewer merge collisions with SCM tools like git.

YAML is a medium-sized standard; please refer to the YAML homepage for details. It should take a short time to read and write simple YAML documents, and a few days to become relatively proficient.

Aliases

YAML has one last trick up its sleeve: it’s hard to put together a complete hardware map without defining an element and needing to refer to it elsewhere. For this purpose, YAML provides anchors and aliases.

Anchors define a label:

- &my_mezz1 !MGMEZZ04
   serial: fmc2_001

- &my_mezz2 !MGMEZZ04
   serial: fmc2_002

...anywhere after that, we can use an alias to refer to the object:

- !Dfmux
   hostname: iceboard004.local
   mezzanines: [ *my_mezz1, *my_mezz2 ]

For the hardware map above, aliases allow us to group like elements together (rather than mixing dfmuxes with nested mezzanine definitions.) For a larger hardware map, a single element might be referred to multiple times and aliases are more important.

Building a Complete Hardware Map

So far, we have introduced YAML and described !tags that we’ve added to generate specific parts of the hardware map. We now provide a complete list of tags, and introduce a few extra helpers used to pull in data from different sources.

Creating Experiment-Specific Subclasses

Let’s say we want to instantiate a Wafer object from YAML. We can quickly do so using the following YAML example_subclass.yaml:

!HardwareMap
- !Wafer { name: foo }

We can then load this HardwareMap as follows:

>>> import pydfmux

>>> hwm = pydfmux.load_session(open('software/examples/example_subclass.yaml'))

>>> w = hwm.query(pydfmux.Wafer).one()
>>> print(w.__class__)
<class 'pydfmux.core.dfmux.Wafer'>

As the print statement shows, this snippet produces a pydfmux.core.dfmux.Wafer object. However, in Object Model, we suggested subclassing the classes in pydfmux.core.dfmux classes in order to allow experiments to add custom columns. We need the YAML loader to create these subclasses, instead of their ancestors.

To do so, we tell the SessionLoader what module to use by supplying a !flavour tag:

!HardwareMap
- !flavour pydfmux.mcgill.dfmux
- !Wafer { name: foo }

>>> import pydfmux

>>> hwm = pydfmux.load_session(open('software/examples/example_subclass2.yaml'))

>>> w = hwm.query(pydfmux.Wafer).one()
>>> print(w.__class__)
<class 'pydfmux.mcgill.dfmux.Wafer'>

The loader will create objects from the module specified as a “flavour”. If the flavour module does not implement the specified class, you will get an error. (This is preferrable to falling back on pydfmux.core.dfmux, since it’s quite easy to provide e.g. pydfmux.mcgill as a flavour by accident. An error message is quicker to debug than a HWM with the wrong classes in it.)

Note

Because HWM elements are parsed in order, you may switch back and forth between different flavours (although it’s not clear this is a good idea.)

Including external YAML/JSON documents

Because it permits things like comments, YAML is a great way to describe the top level of an experiment’s hardware map. (Later on, we’ll see how to pull in CSV data as well.) While YAML is machine-readable, it is not suitable for “round-tripping” — that is, being loaded, modified, and saved back to a YAML file by Python code. It’s easy to see why:

>>> import yaml

>>> y = yaml.safe_load('''
...   # This is a simple piece of vanilla YAML data. We wish we could load it
...   # in Python and write it back without altering its formatting.
...   top_level_is_a_mapping: True
...   with_some_other_elements: [ 3.0, ~, { another_mapping: true } ]
... ''')

>>> print(yaml.dump(y))
top_level_is_a_mapping: true
with_some_other_elements:
- 3.0
- null
- {another_mapping: true}

The contents of the YAML data are unchanged, but it has been reformatted!

  1. Comments are discarded during YAML parsing. Since they are not present in the Python representation of the YAML data, they are not reproduced when it is re-serialized.
  2. The style of formatting (e.g. the encoding used for sequences and mappings) has been changed. (For example, the [3.0,~,{...}] list has been replaced with a dashed encoding.)
  3. In YAML, JSON, and Python, key ordering is not guaranteed. Although the data emerged here in the same order as they were created, they might not have. It is not safe to assume the output data order resembles the input data order at all.

These restrictions mean we cannot easily round-trip an entire YAML hardware map without imposing major restrictions on the type of YAML we write. Rather than accept these restrictions, we permit a hardware map to include external YAML/JSON data that can be round-tripped (by accepting that it may be arbitrarily reformatted when it is saved.) To do so, we use the !include tag:

# This is "include_top.yaml"

hardware_map: !HardwareMap
    - !Dfmux { hostname: iceboard004.local }
    # ...

included_data: !include "include_child.yaml"

# vim: sts=4 ts=4 sw=4 tw=80 smarttab expandtab

This file includes a subsidiary YAML file, called include_child.yaml. We’ve given it some fairly arbitrary contents:

parameters: [1, 2, 3]

We can load both files as follows:

>>> import pydfmux

>>> y = pydfmux.load_session(open("software/examples/include_top.yaml"))

We can inspect the included data directly:

>>> print(y['included_data'])
IncludedYAMLValue({'parameters': [1, 2, 3]})

We can also (optionally) alter included_data and round-trip the data back to the original YAML file:

>>> y['included_data']['parameters'] = [4,5,6]
>>> y['included_data'].save()

Note

Let’s say we wanted to replace the entire contents of included_data. The following would not work as expected:

>>> y['included_data'] = {'parameters': [4, 5, 6]}
>>> y['included_data'].save()
AttributeError: 'dict' object has no attribute 'save'

This attempt failed because we replaced the entire IncludedYAMLValue class (which defined the save() method) with a plain Python dictionary. Instead, we need to ensure the wrapper class is also replaced:

>>> y['included_data'] = pydfmux.core.session.IncludedYAMLValue(
...   {'parameters': [4,5,6]},
...   filename=y['included_data'].filename)
>>> y['included_data'].save()

Including CSV Files

YAML is a good way to specify an experiment’s top-level structure. However, for large collections of objects (e.g. bolometers or LC channels), it’s much more convenient to store data in an external CSV document. To do this, we provide the !csv tag.

- &wafer-arg1a !Wafer
    name: arg1a
    bolometers: !CSVBolometers "wafer_arg1a.csv"

Assuming the YAML loader has been configured to generate pydfmux.mcgill.dfmux classes (see above), this example produces a pydfmux.mcgill.dfmux.Wafer element with the name attribute arg1a. This Wafer has attached pydfmux.mcgill.dfmux.Bolometer classes taken from the CSV file named hwm_complete_arg1a.csv.

The contents of this CSV file could be as follows:

name    lc_board_pad    lc_board_index  y_coord x_coord observing_band  polarization_angle
1A.6.X  45      1       -35.846 105.988 90GHz   -21.6
1A.6.Y  90      1       -35.846 105.988 90GHz   68.4
1A.5.X  4       1       -31.629 96.242  90GHz   -66.6
1A.5.Y  3       1       -31.629 96.242  90GHz   23.4

The first line in the CSV file specifies the column headings, which are translated into columns in the Bolometer objects that are constructed. (The names must match exactly! If not, you will see a semi-informative error message.)

CSV data must be tab-separated. (It is probably easy to pass arguments to the CSV parser, but it’s simpler if we can standardize on tab-separated CSV.)

Looking Up HWM Data

Above, we showed how to import HWM elements from CSV files using the !csv tag. Although this permits compact HWMs, it leaves us with a problem: how do we refer to parts of the HWM, e.g. when constructing a ChannelMapping, when we don’t have a YAML anchor to use? For example, let’s say we want to create a ChannelMapping for bolometer 1A.6.X (see above). How might a ChannelMapping be created with YAML data?

 - &wafer-arg1a !Wafer
     name: arg1a
     bolometers: !CSVBolometers "hwm_complete_arg1a.csv"

 - &lc-003 !LCBoard
     name: LC003
     channels: !CSVLCChannels "hwm_complete_lc003.csv"

- !ChannelMapping
    lc_channel: ??? # How do we get e.g. lc-003.channel[0]?
    bolometer: ??? # How do we get e.g. arg1a.bolometer['1A.6.X']?
    readout_channel: ??? # ...and readout channels aren't in the YAML at all!
    squid: *Sq1SBpol03 # Let's at least pretend we have one of these.

To resolve these three missing links, we provide a !HWMLookup tag. This tag annotates a sequence and walks through HWM links. The following YAML:

!HWMLookup [*lc-003, channel: 1]

...is equivalent to the following Python code:

>>> lc003.channel[1]

Likewise, the lookups can be much deeper. Suppose *d is an alias for a Dfmux object. Then, we can access a pydfmux.core.dfmux.ReadoutChannel as follows:

!HWMLookup [*d, mezzanine:1, module:1, channel:1]

This is equivalent to the following Python code (if d is a Dfmux object):

>>> d.mezzanine[1].module[1].channel[1]

A Complete Example

The following YAML HWM demonstrates “one of everything.” The top-level YAML hwm_complete.yaml contains the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
# Hardware Map
---

# Start time and end time for this HWM.
validity:
    - !!timestamp "2013-10-05t21:59:43.10-05:00"
    - !!timestamp "2013-10-06t21:59:43.10-05:00"

# Bias properties are written by Python code. Include it here, so we can access
# it during tuning (that includes updating the file's contents and writing it
# back to disk.)
default_bias_properties: !include "hwm_complete_bias_properties.json"

# Create a hardware map with the following contents
hardware_map: !HardwareMap

    # Switch to 'mcgill' flavour
    #- !flavour pydfmux.mcgill.dfmux

    # You can instantiate IceBoards in IceCrates as follows.
    - !IceCrate
        serial: icecrate_001
        slots:
            3: &iceboard004 !Dfmux
                hostname: iceboard004.local
                serial: "004"
                mezzanines:
                    1: ~
                    2: !MGMEZZ04
                        serial: FMC2_001

    # You can instantiate Wafers and Bolometers as follows.
    #- !Wafer
    #    name: arg1a
    #    bolometers: !Bolometers "hwm_complete_arg1a.csv"

    # You can instantiate LCBoards and LCChannels as follows.
    #- !LCBoard
    #    name: LC003
    #    channels: !LCChannels "hwm_complete_lc003.csv"

    # Last but not least -- channel mappings! If you use only one part of the
    # hardware map, this is it. Entries in this file can actually imply Wafers,
    # LCBoards, IceCrates, Dfmuxes, et cetera -- so the definitions above are
    # only necessary if you want to specify properties belonging to those
    # objects.
    #
    # !ChannelMappings must currently come last in your HWM, since the other
    # object loaders can only create objects (they do not expect them to exist
    # already.)
    - !ChannelMappings "hwm_complete_sbpol03.csv"

# vim: sts=4 ts=4 sw=4 tw=80 smarttab expandtab

This YAML file references several additional files:

hwm_complete_bias_properties.json:

This file contains JSON (or YAML) data with contents that may be useful for system tuning. It contains the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  carrier_gain: 2, 
  dan_gain: 0.0040687412935323372, 
  demod_gain: 0, 
  fb_mode: "5K", 
  fmaxp2p: 0.9, 
  is_dan: true, 
  nuller_gain: 2, 
  operating_frac_res: 0.73, 
  overbias_amplitude: 0.08,
  dv_del: 0.016
}
hwm_complete_arg1a.csv:

This file contains bolometers associated with the wafer “arg1a”, stored in CSV format. The contents are as follows:

1
2
3
4
5
name            lc_board_pad    lc_board_index  y_coord         x_coord         observing_band  polarization_angle
1A.6.X          45              1               -35.846         105.988         90GHz           -21.6
1A.6.Y          90              1               -35.846         105.988         90GHz           68.4
1A.5.X          4               1               -31.629         96.242          90GHz           -66.6
1A.5.Y          3               1               -31.629         96.242          90GHz           23.4
hwm_complete_lc003.csv:

This file contains LC channel definitions associated with the LC board “lc003”, in CSV format. The contents are as follows:

1
2
3
4
5
6
channel         frequency
1               573.8975528
2               282.1257644
3               625.8574363
4               345.532083
5               678.1459397

We load the hardware map as follows:

>>> import pydfmux

>>> s = pydfmux.load_session(open('software/examples/hwm_complete.yaml'))
>>> hwm = s['hardware_map']

We can now query data from the hardware map:

>>> bolo = hwm.query(pydfmux.Bolometer).filter(pydfmux.Bolometer.name=='1A.6.X').one()

>>> print(bolo)
Wafer(u'arg1a').Bolometer(u'1A.6.X')

>>> print(bolo.polarization_angle)
-21.6

Tag Reference

The “stock” Python YAML parser has been augmented with the following tags.

Hardware Map Objects

!HardwareMap [...]
creates a core.hardware_map.HardwareMap() from a list of suitable objects.
!HWMLookup [*some_reference, key:value, ...]
returns a single HWM object, retrieved by accessing attributes from an object already in the hardware map.

Direct Class Instantiation

!IceCrate {...}

creates a IceCrate object from a mapping. Key/value pairs are passed directly to the object’s constructor. For example:

- !IceCrate
   serial: foo-bar-001
   slots: [ *my_dfmux1 ]
!Dfmux {...}

creates a Dfmux object from a mapping. Key/value pairs are passed directly to the object’s constructor. For example:

- &my_dfmux1 !Dfmux
   hostname: iceboard004.local
   serial: 004
   mezzanines: [ *my_mezz1, *my_mezz2 ]
!MGMEZZ04 {...}

creates a MGMEZZ04 object from a mapping. Key/value pairs are passed directly to the object’s constructor. For example:

- &my_mezz1 !MGMEZZ04
   serial: fmc2_001
   squid_controller: *my_sqc
!SQUIDController {...}

creates a SQUIDController object from a mapping. Key/value pairs are passed directly to the object’s constructor. For example:

- &my_sqc !SQUIDController
   serial: 06-01
   squids: [ *squid1, *squid2, *squid3, *squid4 ]
!Wafer {...}

creates a Wafer object from a mapping. Key/value pairs are passed directly to the object’s constructor. For example:

- !Wafer
   name: c1
   bolometers: [...]
!Bolometer {...}

creates a Bolometer object from a mapping. Key/value pairs are passed directly to the object’s constructor. For example:

- !Bolometer { name: C1.B1.16.X }
!LCBoard {...}

creates a LCBoard object from a mapping. Key/value pairs are passed directly to the object’s constructor. For example:

- !LCBoard
    name: LC027
    channels: [...]
!LCChannel {...}

creates a LCChannel object from a mapping. Key/value pairs are passed directly to the object’s constructor. For example:

- &lc027-1 !LCChannel { }
!ChannelMapping {...}

creates a ChannelMapping object from a mapping. Key/value pairs are passed directly to the object’s constructor. For example:

- !ChannelMapping
    readout_channel: !HWMLookup [*fmc2_001, module: 1, channel: 1]
    lc_channel: *lc027-1
    bolometer: *c1-b1-16-x
    squid: *Sq1SBpol03

External Files

!CSVBolometers "filename.csv"
loads Bolometer objects from the file filename.csv. This file must be tab-separated, and must have column headings that exactly match the attributes of the bolometers to create. The tag generates a list of Bolometers, suitable for attachment to a Wafer object.
!CSVLCChannels "filename.csv"
loads LCChannel objects from the file filename.csv. This file must be tab-separated, and must have column headings that exactly match the attributes of the bolometers to create. The tag generates a list of LCChannels, suitable for attachment to a LCBoard object.
!include "filename.yaml"
includes YAML or JSON data from filename.yaml. If you don’t mind having it reformatted (i.e. losing comments, in the case of YAML data), you can “roundtrip” this data by calling .save() on its handle in Python.