Setup Guide For Quality Control (Cancino)

This document will walk you through the process of setting up a DFMUX Quality Control machine for performing SQUID Controller Board and Mezzanine quality control and uploading the results to bitbucket. The setup includes networking through running the software itself.

It is tailored specifically for setting up the Cancino Quality Control machine, but may also be used as a general guide.

Note

Look out for BOLD or NOTE elements in this document. These are things that may sneak up on you, or should be done soon.

Physical Equipment Necessary

You will need:
  • PC computer running supported Linux (Ubuntu 15.04 or 14.04, Debian 8 recommended) with either:
    1. Two network cards (two ethernet plugs) OR One network card, and one USB-to-ethernet plug
  • 1 Gigabit switch
  • 2 ethernet cables
  • 1 IceBoard with:
    1. 2 Mezzanines
    2. SFP connector
    3. short ethernet loopback
    4. power cable
    5. CF card
  • 37-pin to 37-pin Cable (I recommend just making another ribbon cable with the press-fit connectors. I would prefer not to give them one of our CAT-7 cables)
  • SQUID Controller Board Dongles:
    1. Pass-Through dongle (37-pin to 37pin with breakout 9-pin and 15-pin. Looks a little like something Dr. Frankenstein would make)
    2. #1 DC Dongle (single-moded PCB board 37-pin DSUB. Should be labeled)
    3. #2 AC Dongle (37-pin DSUB soldercup dongle, has a labeled backshell but may not be in it. On the table in the RF room. Has 4 transformers.)
    4. #3 Heater Dongle (37-pin DSUM soldercup dongle to 15-pin DSUB that plugs into the Pass-Through dongle)
  • Mezzanine Board Dongle (37-pin DSUB, plastic backshell, labeled)
  • Either anti-static matt, or stand-offs on the board to lift it like feet (preferably both)

Computer Dependencies and setup

This assumes you are logged into your Linux machine as a user with sudo priviledges

Install pydfmux dependencies

$ sudo apt-get install python-yaml python-sqlalchemy \
python-nose python-sphinx git-annex rst2pdf libavahi-compat-libdnssd1 \
git-python

There are a fiew version requirements which may be problematic for older linux distributions, specifically:

  • SQLAlchemy >= 0.8 (for the in-memory hardware map)

If you have an older linux distribution, you may have to manually upgrade the SQLAlchemy package to meet these requirements. You will be notified if there are version issues the first time you import pydfmux in python (or when you first try to run the quality control scripts). These can be fixed with:

$ pip install --upgrade SQLAlchemy

Install networking dependencies

Our control computere needs to communicate with both the outside world AND the IceBoards on separate networks. On the IceBoard (Local) network, it will additionally need to serve as the DHCP server. To do that we need the following packages:

$ sudo apt-get install avahi-browse isc-dhcp-server

Create the Local Area Network for your IceBoards

This section of the guide will not be as precisely step-by-step as the others.

Plug the ethernet cable into a gigabit switch that is powered on (this will be your iceboard network). Also connect an ethernet cable to the internet to the PC using a USB-to-ethernet adaptor.

Then set up both of your network connections using the GUI network manager. In Ubuntu you can click the upper right hand antenna icon which will produce a dropdown menu. In the configuration you will probably have a network adaptor already filled in, and will have to create a new network adaptor to use the USB connection.

Configure the network adaptors

I will refer to the USB-ethernet outside-world adaptor as “outside-network” and the ethernet adaptor (which probably appears as “eth0” as “DFMUX”.

The following settings should be used:

“outside-network”
IP address DHCP assigned (nothing special here). I will assume that these operate on the subnet 192.168.0.x. You can check this by successfully configuring it and connecting to the internet, and then using $ ifconfig or the network manager to view the IP address the PC is using.
“DFMUX”
Static IP address. Exactly which subnet you use will depend on the default that “outside-network” uses. If the “outside-network” uses 192.168.0.x, give yourself a static IP address on 192.168.1.x. Let’s assume you use the static IP address 192.168.1.47

Configure DHCP

Open up the DHCP configuration file at /etc/dhcp/dhcpd.conf ($ sudo nano /etc/dhcp/dhcpd.conf) and edit it to conform to the following example, where I have specified that the DHCP network is only allowed to assign addresses in the .200 - .250 block. This is arbitrary, but allows static IP addresses below that to be untouched.

#
# Sample configuration file for ISC dhcpd for Debian/Ubuntu at /etc/dhcp/dhcpd.conf
#
# Attention: If /etc/ltsp/dhcpd.conf exists, that will be used as
# configuration file instead of this file.

ddns-update-style none;
default-lease-time 600;
max-lease-time 7200;
log-facility local7;

shared-network 200-250{
 subnet 192.168.1.0 netmask 255.255.255.0 {
   option routers 192.168.1.47;
 }
 pool {
  range 192.168.1.200 192.168.1.250;
  allow unknown-clients;
 }
}

Once that is saved, go ahead and fire up the server with:

$ sudo /etc/init.d/isc-dhcp-server restart

You can take a look at the log messages with

$ sudo tail -F /var/log/syslog

It should look something like this initially:

Jul  8 17:59:25 warwick dhcpd: All rights reserved.
Jul  8 17:59:25 warwick dhcpd: For info, please visit
Jul  8 17:59:25 warwick dhcpd: https://www.isc.org/software/dhcp/
Jul  8 17:59:25 warwick dhcpd: Wrote 35 leases to leases file.
Jul  8 17:59:25 warwick dhcpd:
Jul  8 17:59:25 warwick dhcpd: No subnet declaration for eth0 (192.168.0.24).
Jul  8 17:59:25 warwick dhcpd: ** Ignoring requests on eth0.  If this is not what
Jul  8 17:59:25 warwick dhcpd:    you want, please write a subnet declaration
Jul  8 17:59:25 warwick dhcpd:    in your dhcpd.conf file for the network segment
Jul  8 17:59:25 warwick dhcpd:    to which interface eth0 is attached. **
Jul  8 17:59:25 warwick dhcpd:

Where the warning is telling you that it noticed there is a network segment on eth0, which in this case is the “outside-network” adaptor that it is ignoring. This is good. We only want it to serve addresses on the “DFMUX” segment.

If you have anything on that network already (including an IceBoard) you will then see names get registered like so:

warwick dhcpd: DHCPREQUEST for 192.168.1.233 from 08:00:27:5d:1a:f4 (adam-VirtualBox) via eth1
warwick dhcpd: DHCPACK on 192.168.1.233 to 08:00:27:5d:1a:f4 (adam-VirtualBox) via eth1
warwick dhcpd: DHCPREQUEST for 192.168.1.211 from 84:7e:40:6d:f4:88 via eth1
warwick dhcpd: DHCPACK on 192.168.1.211 to 84:7e:40:6d:f4:88 via eth1
warwick dhcpd: DHCPREQUEST for 192.168.1.228 from 84:7e:40:66:39:a4 via eth1
warwick dhcpd: DHCPACK on 192.168.1.228 to 84:7e:40:66:39:a4 via eth1

Where in this instance “eth1” is the “DFMUX” network.

Checking Your Work

With that working, plug an IceBoard into the “DFMUX” network and boot it up. Watch the log message above to see it get its IP address, and once it boots navigate to http://iceboard<serial>.local/ . If you see the IceBoard webpage, DHCP is set up correctly and mDNS is functioning.

Generate an SSH RSA public/private key pair and uploading the key to bitbucket

The bitbucket user for all of the pydfmux and dfmux-qc repo work is “dfmuxqualitycontrol” with the usual password. In order to automate pushing the quality control outputs do the bitbucket repository we want to remove the requirement of asking for a password. We do this by generating SSH keys and giving the public key to bitbucket.

First, ensure that you aren’t going to override an existing ssh key (id_rsa, id_rsa.pub):

$ ls ~/.ssh

Generate an SSH key with no passphrase:

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/USERNAME/.ssh/id_rsa):
[core]
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/USERNAME/.ssh/id_rsa.
Your public key has been saved in /home/USERNAME/.ssh/id_rsa.pub.

Check that an SSH agent is running:

$ ps -e  | grep [s]sh-agent

If not, start it:

$ ssh-agent /bin/bash

Add the key to the SSH agent:

$ ssh-add ~/.ssh/id_rsa
Identity added: /home/USERNAME/.ssh/id_rsa (/home/USERNAME/.ssh/id_rsa)

Check that it is saved:

$ ssh-add -l

Display the key and then copy the text:

$ cat ~/.ssh/id_rsa.pub

and add it to the dfmuxqualitycontrol keys https://bitbucket.org/account/user/dfmuxqualitycontrol/ssh-keys/

Clone pydfmux and dfmux-qc repositories

Finally, cd somewhere you like...say, ~/Desktop and clone the repos.

Note

This is a slightly different clone command than you may be used to– the repo should be cloned via SSH rather than HTTPS.

$ cd ~/Desktop
$ git clone git@bitbucket.org:winterlandcosmology/pydfmux.git
$ git clone git@bitbucket.org:winterlandcosmology/dfmux-qc.git

Configure the output path in pydfmux to point to dfmux-qc

Because the modified scripts we will be using were made specifically for cancino, there are two hardcoded paths to the dfmux-qc repo on that machine. If you cloned into the Desktop as above, you can do it this way:

Edit line 34 in pydfmux/tests/sqc_qc/mgngsq04_cancino.py to be the path you copy from above:

output_dir: /some/path/to/dfmux-qc

So the full string-block looks like this:

    configfile = '''
# This is the YAML configuration file for the DFMUX Mezzanine Quality Control
# testing suite. Modify the parameters as required. DO NOT commit your changes
# to git, unless everyone should see them!

output_dir: /some/path/to/dfmux-qc
output_subdir: sqcb_qc


# A HardwareMap tells us how to communicate with the right mezzanine.
hardware_map: !HardwareMap
    - &iceboard !Dfmux
        hostname: iceboard{0}.local
        mezzanines:
            {1}: !MGMEZZ04
                squid_controller: !SQUIDController
                    serial: "{2}"
'''.format(ib_serial, mezz_num, sqcb_serial)

Do this same to line 37 of pydfmux/tests/mezzanine_qc/mgmezz04_cancino.py

output_dir: /some/path/to/dfmux-qc

So the full string-block likes like this:

    config_file='''
# This is the YAML configuration file for the DFMUX Mezzanine Quality Control
# testing suite. Modify the parameters as required. DO NOT commit your changes
# to git, unless everyone should see them!

output_dir: /some/path/to/dfmux-qc
output_subdir: mezzanine_qc

# Do we upload results to cloud storage?
s3_enabled: false
s3_bucket: winterland-mgmezz04-qc
s3_uid: AKIAIUK4J3HNPJEDPTNA
s3_key: nugonQuP0WwRWO3EG7w6j1oLYrZP3+yGSmhr34oX

# A HardwareMap tells us how to communicate with the right mezzanine.
hardware_map: !HardwareMap
    - &iceboard !Dfmux
        hostname: iceboard{0}.local
        mezzanines:
            {2}: !MGMEZZ04 {3}

# EEPROM initialization
write_ipmi: true    # Write IPMI on an uninitialized mezzanine?
clobber_ipmi: false # Do we *overwrite* valid IPMI data?

# This data is written to the mezzanine's EEPROM, so it's crucial that it's
# correct. Otherwise, it'll be a nightmare tracing what's plugged in where.
ipmi_data:
    part_number: MGMEZZ04
    serial_number: {1}
    product_version:

# What parts of QC do we run?
test_basics: true
test_dc_tf: true
test_ac_tf: true
test_noise: true

# The resistance from Nuller + (Carrier +) to Demod +.
dongle_resistance: 350

# Disabled by default because it can be VERY large (700+MB)
pickle_output_data: False
'''.format(ib_serial, mezz_serial, mezz_num,'{}')

Running the SQCB and Mezzanine quality control routines

This is the easy part.

Ensure that you have all of the relevant dongles, and the IceBoard you intend to use is plugged in, booted up, and accessible (easy check is the webpage).

The two scripts we will be using are:

SQUID Controller Board Quality Control
pydfmux/tests/sqc_qc/mgngsq04_cancino.py
Mezzaning Quality Control
pydfmux/tests/mezzanine_qc/mgmezz04_cancino.py

Running the Mezzanine Quality Control

You will need the Mezzanine QC dongle, as well as a (known good) SQUID Controller Board and 37-pin DSUB cable to connect them.

Plug in the SQUID Controller Board to the Mezzanine you intend to QC. Then execute the python script. You will be prompted for the IceBoard serial number, mezzanine slot number, and mezzanine serial number, and then to plug in the SQCB.

$ cd ~/Desktop/pydfmux/tests/mezzanine_qc
$ python mgmezz04_cancino.py
2015-07-09 10:16:23 | WARNING | dfmux_logging.py | __init__ |
    /home/daquser/pydfmux/core/utils/dfmux_logging.py:107:
    UserWarning: Logging to pydfmux base folder
    Please edit your pydfmux/logging.yaml file

2015-07-09 10:16:24 | ERROR | dfmux_logging.py | set_console_level |
    Console logging output level switched to 40

Enter IceBoard serial number as 4-digit number (XXXX)
0035
Enter which slot the Mezzanine board to be QCed is in (1 or 2)
1
Enter the 3-digit serial number of the Mezzanine board (XXX)
008
Mezzanine to be QCed is: None.Dfmux(hostname=iceboard0035.local).MGMEZZ04(1,'008')

Plug in a (known good) SQUID Controller Board to the Mezzanine output and hit enter.

Note

The error is not an error, it is just informing you that the logging has been changed to only display errors.

It will then perform the basics test, before prompting you to unplug the SQUID Controller Board and plug in the Mezzanine QC dongle.

Plug in a (known good) SQUID Controller Board to the Mezzanine output and hit enter.
Tests we are within FMC spec tolerances for the 12V rail ... ok
Tests we are within FMC spec tolerances for the 2.5V rail ... ok
Tests we are within FMC spec tolerances for the 3.3V rail ... ok
Tests that the EEPROM reads back, and looks right ... ok
Test [module 1 carrier] write/read gain IIC address at all gains ... ok
Test [module 2 carrier] write/read gain IIC address at all gains ... ok
Test [module 3 carrier] write/read gain IIC address at all gains ... ok
Test [module 4 carrier] write/read gain IIC address at all gains ... ok
Test [module 1 nuller] write/read gain IIC address at all gains ... ok
Test [module 2 nuller] write/read gain IIC address at all gains ... ok
Test [module 3 nuller] write/read gain IIC address at all gains ... ok
Test [module 4 nuller] write/read gain IIC address at all gains ... ok
Tests IIC communication with the SQUID Controller Board via the temp sensor ... ok
Tests the ability to ask for the mezzanine temperature, and that it is sane ... ok
Writing output document and dumping data to disk (if requested) ...
Generating reStructuredText document at
/home/daquser/dfmux-qc/mezzanine_qc/mezz_008/test_mezzanine_basics_1436451636/mezz_008.rst

ok

----------------------------------------------------------------------
Ran 15 tests in 26.754s

OK
Unplug the SQUID Controller Board, plug in the resistive QC Dongle, and hit enter.

After that no additional user input is required. The test suites will take approximately 30 minutes, and generate an output PDF at dfmux-qc/mezzanine_qc/mezz_008/ and uploaded to bitbucket at https://bitbucket.org/winterlandcosmology/dfmux-qc/src.

Running The SQUID Controller Board Quality Control

You will need all of the SQCB dongles (Pass-Through, DC, AC, Heater) in addition to a SQCB plugged into a known-good mezzanine and IceBoard.

Make note of the IceBoard serial number, Mezzanine Slot Number, and SQCB serial number, you will be prompted for these.

$ cd pydfmux/tests/sqc_qc
$ python mgngsq04_cancino.py

Please Wait. Initializing PyDFMUX.

2015-07-09 10:56:14 | WARNING | dfmux_logging.py | __init__ | /home/daquser/pydfmux/core/utils/dfmux_logging.py:107:
    UserWarning: Logging to pydfmux base folder Please edit your pydfmux/logging.yaml file

2015-07-09 10:56:15 | INFO | dfmux_logging.py | set_console_level | Console logging output level switched to 20
Enter IceBoard serial number as 4-digit number (XXXX)
0035
Enter which mezzanine slot the SQUIDController Board is attached (1 or 2)
1
Enter the serial number of the SQUIDController board (XX-XX)
00-00
Install Pass-through Dongle in between SQUID controller board and cable to mezzanine with 9-pin dsubs connected
 Install Dongle #1 (DC): Press Enter to Continue, press s to skip this step:

Once you hit enter, it will proceed with the tests and prompt you to change the dongles when appropriate:

2015-07-09 10:57:25 | INFO | QC_sqctrl_10mhz.py | VOffsetVsFbias | Voffset vs Flux Bias: board 00-00; Module 1 begins
2015-07-09 10:57:26 | INFO | QC_sqctrl_10mhz.py | VOffsetVsFbias | Voffset vs Flux Bias: board 00-00; Module 1 ends
2015-07-09 10:57:26 | INFO | QC_sqctrl_10mhz.py | VoffsetVsCurrentBias | Voffset vs Current Bias: board 00-00; Module 1 begins
2015-07-09 10:57:28 | INFO | QC_sqctrl_10mhz.py | VoffsetVsCurrentBias | Voffset vs Current Bias: board 00-00; Module 1 ends
2015-07-09 10:57:28 | INFO | QC_sqctrl_10mhz.py | VadcVsVoffset | Vadc vs Voffset: board 00-00; Module 1 begins
2015-07-09 10:57:28 | INFO | QC_sqctrl_10mhz.py | VadcVsVoffset | Vadc vs Voffset: board 00-00; Module 1 ends
2015-07-09 10:57:28 | INFO | QC_sqctrl_10mhz.py | Integrator | Integrator Test: board 00-00; Module 1 begins
2015-07-09 10:57:48 | INFO | QC_sqctrl_10mhz.py | Integrator | Integrator Test: board 00-00; Module 1 ends
2015-07-09 10:57:48 | INFO | QC_sqctrl_10mhz.py | VOffsetVsFbias | Voffset vs Flux Bias: board 00-00; Module 2 begins
2015-07-09 10:57:50 | INFO | QC_sqctrl_10mhz.py | VOffsetVsFbias | Voffset vs Flux Bias: board 00-00; Module 2 ends
2015-07-09 10:57:50 | INFO | QC_sqctrl_10mhz.py | VoffsetVsCurrentBias | Voffset vs Current Bias: board 00-00; Module 2 begins
2015-07-09 10:57:51 | INFO | QC_sqctrl_10mhz.py | VoffsetVsCurrentBias | Voffset vs Current Bias: board 00-00; Module 2 ends
2015-07-09 10:57:51 | INFO | QC_sqctrl_10mhz.py | VadcVsVoffset | Vadc vs Voffset: board 00-00; Module 2 begins
2015-07-09 10:57:51 | INFO | QC_sqctrl_10mhz.py | VadcVsVoffset | Vadc vs Voffset: board 00-00; Module 2 ends
2015-07-09 10:57:52 | INFO | QC_sqctrl_10mhz.py | Integrator | Integrator Test: board 00-00; Module 2 begins
2015-07-09 10:58:12 | INFO | QC_sqctrl_10mhz.py | Integrator | Integrator Test: board 00-00; Module 2 ends
2015-07-09 10:58:12 | INFO | QC_sqctrl_10mhz.py | VOffsetVsFbias | Voffset vs Flux Bias: board 00-00; Module 3 begins
2015-07-09 10:58:13 | INFO | QC_sqctrl_10mhz.py | VOffsetVsFbias | Voffset vs Flux Bias: board 00-00; Module 3 ends
2015-07-09 10:58:13 | INFO | QC_sqctrl_10mhz.py | VoffsetVsCurrentBias | Voffset vs Current Bias: board 00-00; Module 3 begins
2015-07-09 10:58:15 | INFO | QC_sqctrl_10mhz.py | VoffsetVsCurrentBias | Voffset vs Current Bias: board 00-00; Module 3 ends
2015-07-09 10:58:15 | INFO | QC_sqctrl_10mhz.py | VadcVsVoffset | Vadc vs Voffset: board 00-00; Module 3 begins
2015-07-09 10:58:15 | INFO | QC_sqctrl_10mhz.py | VadcVsVoffset | Vadc vs Voffset: board 00-00; Module 3 ends
2015-07-09 10:58:15 | INFO | QC_sqctrl_10mhz.py | Integrator | Integrator Test: board 00-00; Module 3 begins
2015-07-09 10:58:35 | INFO | QC_sqctrl_10mhz.py | Integrator | Integrator Test: board 00-00; Module 3 ends
2015-07-09 10:58:35 | INFO | QC_sqctrl_10mhz.py | VOffsetVsFbias | Voffset vs Flux Bias: board 00-00; Module 4 begins
2015-07-09 10:58:37 | INFO | QC_sqctrl_10mhz.py | VOffsetVsFbias | Voffset vs Flux Bias: board 00-00; Module 4 ends
2015-07-09 10:58:37 | INFO | QC_sqctrl_10mhz.py | VoffsetVsCurrentBias | Voffset vs Current Bias: board 00-00; Module 4 begins
2015-07-09 10:58:38 | INFO | QC_sqctrl_10mhz.py | VoffsetVsCurrentBias | Voffset vs Current Bias: board 00-00; Module 4 ends
2015-07-09 10:58:38 | INFO | QC_sqctrl_10mhz.py | VadcVsVoffset | Vadc vs Voffset: board 00-00; Module 4 begins
2015-07-09 10:58:39 | INFO | QC_sqctrl_10mhz.py | VadcVsVoffset | Vadc vs Voffset: board 00-00; Module 4 ends
2015-07-09 10:58:39 | INFO | QC_sqctrl_10mhz.py | Integrator | Integrator Test: board 00-00; Module 4 begins
2015-07-09 10:58:59 | INFO | QC_sqctrl_10mhz.py | Integrator | Integrator Test: board 00-00; Module 4 ends
Install Dongle # 2 (Loopback), Press Enter to Continue, Press s to skip this step:

The whole test should take under 10 minutes, and an output is produced in dfmux-qc/sqc_qc/SQUID_ctrl_board_00-00/ and pushed to bitbucket as above.

BONUS: SSH access to the machine from the outside world

I don’t actually know this procedure well enough for a step-by-step guide, but this is a placeholder. Gilbert may be able to help.