Troubleshooting

Troubleshooting Tools

There are a few tools baked into PyDFMUX to help with troubleshooting. By default the console (print-to-screen) logging output level is set to “INFO” to avoid clutter. However, in every algorithm there are “DEBUG” level messages that may be useful for troubleshooting. These are always written to the text files which can be found in the directory saved in the PYDFMUX_OUT_DIR environment variable.

To view these messages in real-time, set the console logger to “DEBUG” before running your algorithms as follows:

>>> from pydfmux.core.utils.logging_utils import LoggingManager
>>> LoggingManager().set_console_level('DEBUG')

Additionally, in some (though not yet all – this is ongoing) algorithms, when an exception is raised any useful partial-data-products are pickled and dumped to disk in the same directory as the log text files.

For example: if DAN goes unstable or a DAC rails while dropping bolometers into the transition, and the algorithm throws an exception and halts, the Resistance, Current, and Voltage data taken for all bolometers up to that point will be stored in intermediate data product that is dumped to disk. Inspection of that may reveal one bolometer which began oscillating, or railed, ahead of the others, or otherwise help reconstruct a picture of what happened.

When uncaught exceptions are raised no output documents or plots will be made, but the data in the pickle file that is stored should be named reasonably enough to understand the fields, and in most cases will have an identical structure as the final output file but with incomplete fields.

A Note About Real-Time Timestream Data

It is absolutely invaluable to have real-time insight into what the channels are doing when trying to debug anything.

PyDFMUX has a (Linux-only, at the moment) tool built in to allow this called the parser. It is similar to the PyWTL parser and lives (for now) in firmware/mcgill/x86_64/parser The parser streams the multicasted timestreams from all IceBoards into dir-files which may be opened using KST2 or a similar visualization tool.

Syntax:

$ cd firmware/mcgill/x86_64
$ git-annex get parser
$ ./parser -d DESIRED/PATH/TO/DIRFILE

Note that you may have to CHMOD to make executible (git-annex can sometimes trip over permissions)

If it is not standard for you to operate with KST or a similar tool open, it should absolutely be step 3 in your debugging procedure (after consulting the log files and any intermediate data dumps) to attempt to reproduce the issue while noting what is happening in KST.

Note

This tool has been superceded by the spt3g_software “legderman” tool. Ledgerman is (very enthusiastically) supported, and runs well with more than one board. The parser is no longer supported and has issues with large numbers of boards.

mDNS configuation (Crate-wise hostnames)

When the iceboards are installed into an icecrate they should be accessible not just using iceboard<serial>.local, but also as slot<slot#>.icecrate<crateserial>.local (ie, slot1.icecrate003.local)

If that is not happening, then your computer may have a default nss configuration that is blocking it.

Step 1:
in /etc/nsswitch.conf if you have a line that includes mdns4_minimal, change to mdns4
Step 2:

create a file “/etc/mdns.allow” with two lines in it:

.local.
.local

iPython Idiosyncrasies

Greedy Completion

For iPython users, you will notice:

In [1]: dfmuxes = hwm.query(pydfmux.core.dfmux.Dfmux)
In [2]: dfmuxes[0].<tab>
.DS_Store .git/ .gitignore

will not tab-complete – instead it will show files in the currect directory that begin with ‘.’ What we really want is for that to show methods of the dfmux hardware map object being referenced:

In [1]: dfmuxes = hwm.query(pydfmux.core.dfmux.Dfmux)
In [2]: dfmux = dfmuxes
In [3]: dfmux.<tab>
dfmux.detect dfmux.hwm dfmux.serial
dfmux.flush dfmux.metadata dfmux.set_fpga_bitstream
dfmux.hold dfmux.mezzanine dfmux.tuber_uri
dfmux.hostname dfmux.mezzanines

The default behavior is the result of a configuration in iPython called “greedy completion.” To turn this behavior off, allowing for tab-completion of list-like objects, enable greedy completion either in an individual session:

In [1]: %config IPCompleter.greedy = True

or as default in your ipython_config.py file:

c.IPCompleter.greedy = True

Be warned, however, that this behavior will make it impossible for you to tab-complete within the arguments-position of in-line statements. The following will not tab-complete into ‘CARRIER’ with greedy-completion enabled:

In [1]: d.mezzanine[1].module[1].channel[1].set_amplitude(0.5, d.UNITS.HZ, d.TARGET.C<tab>)

Unfortunately, this seems to be a lesser-of-two-evils scenario here. The tab-completion code in ipython is reasonably compact, so if you are interested in hacking it, we’d love to hear about it.

Logging Uncaught Exceptions

In the logging module PyDFMUX identifies whether the environment is python or ipython and uses the appropriate method to insert its own custom exception hook that logs any uncaught exception (aside from Keyboard Interrupts) before raising the typical stack-trace.

Ipython has a bug, however, which prevents this exception hook from being successfully implemented if a script is launched from the commandline rather than interactively (with a .py extension):

$ ipython some_dfmux_tuning_scripy.py   ## Bad -- will NOT work
$ ipython some_dfmux_tuning_scripy.ipy  ## Bad -- will work (but don't do this. Really)
$ python some_dfmux_tuning_scripy.py    ## Good -- this will work
More details may be found here:

Serializing Data

Pydfmux uses pickle files to store dictionaries of data output to disk. There are several issues with doing this reliably.

First, serializing the SQLAlchemy data types is difficult, because these objects are typically dynamically generated from the database. To enable storing data keyed by (e.g.) ReadoutModule, use the make_pickleable function in pydfmux.core.utils.save_returns. To write data returned by an algorithm to a particular output directory setup by the algorithm logger, use the save_returns function in the same module.

Second, the default compression protocol used in python3 is not readable in python2, and extra care must be taken to distinguish between unicode and bytestrings when reading in pickle files in python3. To make this whole process less complicated, use the load_pickle and save_pickle functions in the pydfmux.core.utils.pickle_compat module, also imported for convenience as pydfmux.load_pickle and pydfmux.save_pickle.