Organization of SINQ Data Acquisition
Client Server Data Acquisition
SINQ uses SICS as its data acquisition program. SICS is a client server system. Thus at any given time at least two programs are required to operate SICS:
- A SICS client. SICS clients can run on any computer
- A SICS server. The SICS server runs on the instrument computer.
The two programs communicate with each other through the TCP/IP network and a simple ASCII command protocol.
The Instrument Computer
For each instrument there is an instrument computer. This computer is named like the instrument. For example, for the instrument AMOR the instrument computer is amor.psi.ch. Instrument computers are HP workstations equipped with two disks set up in RAID 1 (mirroring) configuration. For most instruments the instrument computer is located in the instruments cabin. The instrument computers run linux. At the time of writing Scientific Linux 6.
Instrument Computer Accounts
On each instrument computer there are two important local user accounts:
- The instrument user account
- The instrument guest account
The instrument account is used to run the SICS server. Its name is the same as the name of the instrument. The password follows a scheme: instrument name + lns, all in capitals. Thus for amor the credentials for this account are: amor/AMORLNS. Ordinary users are not meant to touch this account.
The instrument guest account is named inst + lnsg. The password is the last two digits of the year + lns1. Thus, again for the amor example, the credentials are amorlnsg/14lns1. The instrument guest account supposed to be used by SINQ users and holds user data such as batch files and else. Permssions and umasks are set appropriately that the instrument account can read and write the guest account and that the guest account can read data files.
This is the theory, in practice some instrument do everything through the instrument account.
This scheme allows us to accommodate external SINQ users without going through the pains of allocating AFS accounts for each user group. So far, security posed no problem. The local accounts on the instrument computer are not easily accessible from outside of PSI. To access those local instrument accounts the following steps are required:
- Logon to balrog.psi.ch with your AFS username
- Logon to any other linux machine, for example, lnsl15.psi.ch, with your AFS username
- From there logon to the local accounts on the instrument computer
This is so complicated because each instrument has its own instrument network segment.
The Instrument Computer in the Instrument Network
For each instrument in the SINQ hall there is a separate network segment, each with space for 32
TCP/IP devices. Each instrument network is separated from the main PSI network through a firewall
maintained by PSI. The purpose of this setup is to separate the instruments from each other and
protect them against malicious access. This setup has a number of consequences:
- Devices in the instrument network are only accessible from the instrument computer
- The instrument computer is the gateway to the instrument. To this purpose, ssh, AFS and the SICS ports have been openened in the firewall. If additional services are required, the corresponding ports must be openened in the firewall by AIT.
- Ping is one of the blocked protocols; thus instrument computers cannot be pinged from the outside.
There is no DNS in the instrument network. TCP/IP addresses are managed locally by way of the /etc/hosts
file on the instrument computer. Inspect this file for TCP/IP addresses of instrument devices. As yet
another complication, the PSI SL6 standard setup tries to manage the /etc/hosts file automatically.
In order to prevent overwriting of the /etc/hosts file an obscure ext3 file system feature, file attributes,
is used. Thus, when a change to /etc/hosts is required the following procedure must be adhered too:
- Become root
- Remove the immutable attribute from /etc/hosts: chattr -i /etc/hosts
- Edit /etc/hosts to your hearts delight
- Set the immutable attribute again: chattr +i /etc/hosts
The SICS Server in the Instrument Account
The instrument account contains the necessary files to run the SICS server. The basic command to
start the SICS server is:
Where inst.tcl is a program in the server side scripting language Tcl which configures the SICS server for
this particular instrument. All instruments use the same SICS server executable. Inst.tcl is named after
the instrument, for example amor.tcl. Inst.tcl will load further Tcl files as required. I will use the
string inst as the place holder for the instrument name in the following discussion.
The instrument account has to contain the following directories:
- The instrument specific Tcl programs and the SICS server executable. For example: amor_sics
- The Tcl files shared among many instruments
- Log files
- Data files
- Files for the monit system, see later
The rest is user generated obfuscation.....
Some important files in the inst_sics directory:
- the main configuration program for the instrument
- the definition of instrument specific commands
- the programs for writing data files.
- The dictionary file which controls the placement of data items in the data file
The data directory contains the data collected at the instrument. There is a directory for each year
of operation. Such year directories contain:
- a file to persist the current data file number. Do not touch
- a directory holding the SICS parameter history files
may collect more then hundred thousand experimental data files per year. When these are all in one
directory file system operations break down.
The SICS server and its associated processes are managed by a supervisor program called monit. This is
described in more detail in the section on starting and stopping SICS. The directory monitfiles contains
script belonging to this system.
- directories containing 1000 data files each. This organisation was choosen because an instrument
All hardware is controlled by the SICS server from the instrument computer. Some hardware communicates
directly via TCP/IP. But a great many devices speak only RS232 and are connected to the network through
terminal servers. There is a terminal server per instrument.
Data Replication and Backup
The experimental data files are the most valuable commodity produced by a SINQ instruments. The following
scheme is in place to keep them safe:
- The SICS server writes the data initially to the local disk. This works even when the network breaks down.
- The SICS server also sends a UDP message to a little java server running on the instrument computer. This
server essentially starts an rsync process which makes sure that the data file is duplicated to the AFS
The AFS storage area for data files is: /afs/psi.ch/project/sinqdata There are further subdirectories per
year which in turn contains subdirectories per instrument. Which in turn contain the 000-0nn directories
containing the data files, the DataNumber and status history files. The directory for the current year
is covered by a nightly backup by AIT.
Older data lives on a non backuped area on AFS. But the old data file years are also
archived in the PSI archiving system.
For the local instrument accounts the following scheme is in place:
- The sicscommon and inst_sics directories are under version control using git.
- There is a nightly rsync backup to lnsl15 triggered by a cron job. This protect against disk crashes;
it is no full journaled backup.
We do not give guarantees that user files on the instrument computers can be recovered at all times.
This page last changed on 01-Jul-2014 15:10:30 CEST by MarkKoennecke.