Synchronisation to AFS Fails

Symptom

Data files are written on the instrument computer but do not appear on /afs/psi.ch/project/sinqdata/year/inst

Solutions

File synchronisation happens via rsync. It occurs that rsync get stuck, especially when there was a problem with the AFS servers. Resolve this with the folwoing steps:

1. Loging to the instrument account on the instrument computer
2. Check for the condition with: ps waux | grep rsync. There should be no long running rsync processes
3. To fix it:
1. Stop synchronisation: monit stop sync
2. Kill all rsync processes: kilall rsync
3. Restart file synchronisation: monit start sync
4. Wait a couple of minutes and check if this fixed the issue.

Another reason for synchronisation to fail is that the AFS file system is full. In order to check for this:

1. cd to /afs/psi.ch/project/sinqdata/year/inst. Replace year with the current year and inst with your instrument
2. Type fs examine at the unix prompt. There will be a message showing the quota and how much of it is used. If the quota is exceeded the AFS volume needs to be expanded. Normally this has to be run through the LDM computing staff. But if they are not available, the AIT helpdesk can help.

This covers most synchronisation issues. If this does not help, synchronisation itself has to be debugged. The following steps:

1. Login to the instrument account on the instrument computer and cd into inst_sics.
2. Type: cat fs.ini at the unix prompt
3. This prints the configuration file for the synchronisation server. Towards the bottom of the output there is a line reading shellcommand.
4. Cut and paste the content of shellcommand onto a unix prompt and try to execute it.
5. Look at any error messages and resolve them.

Synchronisation uses a convoluted system to get credentials for writing to /afs/psi.ch/project/sinqdata. Changes by AIT might break the scheme. Consult Achim Gsell if this is the case.