Poor performance using Intel MPI on Rackham

milou2 rebooted on Monday 2017-08-28 at 19:51

milou2 rebooted on Saturday 2017-08-19.

Intelmpi performance issues

Intelmpi performance issues.

We have observed and several users have reported issues with running X11 applications on Milou. We are investigating it.

The login nodes milou2.uppmax.uu.se and milou-b.uppmax.uu.se were rebooted 15:00 today (29th of May) due to some issues with the kernel NFS module.

Some project volumes on pica are very heavily loaded and slow/next to unusable for interactive use. We're doing what we can to resolve this but can not promise any set time for when things will behave as normal again.

UPDATE: We've had some continuing issues with this due to some nodes not realizing when resources behave better, we're working on these issues but this may have caused disturbances like failed jobs or missing output.

The UPPMAX system group hosts the spring 'SONC' conference where administrators from all SNIC-centers meet and discuss how to improve our centers. With many UPPMAX adminstrators being out of office during the conference (Thursday 11th and Friday 12th) the support will likely be less responsive.

Due to a misconfiguration active on a certain number of nodes around 12AM today, some jobs that were launched on milou could not start.

If you have jobs that were victims of this, they will likely show up as completed although with a very short run time (a few seconds).

We have gotten reports that the Intel license server is not responding. We are investigating it. This might manifest itself with hangs or freezes during compilations.

We have identified a problem with the Slurm account database. If you just got added or created a new project you might get the following message when scheduling jobs "Invalid account or account/partition...". It affects primarily Rackham and Milou.

Resetting your password from this page is currently not working. If you need to reset your password please contact support@uppmax.uu.se

Update 2017-04-18: This issue should now be fixed.

Some of the shared funk-accounts used on Irma and Milou might stop working due to the IP-address change.

Smog will be decommissioned on Wednesday 5th of April. As previously mentioned the SNIC Cloud Team is currently working on bringing up a new cloud to replace Smog and join the other two regions in the SNIC Science Cloud project.

For questions ,please contact support@cloud.snic.se (and not the UPPMAX support queues).

We have idenfied performance issues when using Intel MPI on Rackham. In some cases you see a 10x slowdown (or worse) using Intel MPI compared to Open MPI. We are investigating this issue and hope to have it solved soon. For now, please use Open MPI.

An issue exist on Rackham affecting projects of the form "p123456". The projects are not allowed to run due to the monthly core allocation incorrectly being set to 0 hours. We are investigating why this happens.

Update 2017-03-10: The issue should now be fixed.

All active Tintin projects (exception Tintin-Fysast1, please see below) have been migrated to Rackham. All UPPMAX users should now have access to Rackham.
 

Many Tintin users have missed that Rackham will replace Tintin. We are currently migrating all projects from Tintin to Rackham and when this is done, all users will get access to Rackham. We will announce this per email and on our homepage.

1st of March 2017 is the day we decommission Tintin. It will be replaced by the Rackham cluster. All projects on Tintin will be moved to our new Rackham cluster.

We have identified a problem with the UPPMAX Account Request which unfortunately causes some delay before you can login to UPPMAX. We hope to complete the registration this week. You do not have to resubmit your Accounts.
 

There is currently a problem sending in support tickets to support@uppmax.uu.se. We are investigating and hope to have it fixed soon.

We are happy to announce that UPPMAX's cluster Rackham is now available!

Milou, Tintin, and Fysast-1 are back in production. Bianca is back in test production. Still working on Smog.
 

The degraded RAID now fixed

Lustre file system problem
 

Totally inresponsible. Lustre file system problem (wich will be decommissioned tomorrow....)

We have now quota on the number of files in /scratch (/tmp).

100000 is maximum (per user). If you need more you have to use a compute node.

Milou2 hasn't worked well for a while. We will give him a fresch restart.
 

Milou1 rebooted Thursday am11.

Problems with lustre file system.

Fysast1 down Wednesday before lunch.

One power supply broken and the fuses for half the cluster was blown.

Milou1 rebooted Wednesday am11.00

Problems with lustre file system

We have a maintenance window coming up on January 11 from 9:00. Due to physical work, we need to shut down the system during the maintenance window this time so jobs won't run.

We will also likely be required to rebuild virtual nodes and will probably lose information about queued jobs.

Update 21:10: Maintenance is now finished and the system should be available again.

Milou2 rebooted at 06:01 due to a problem with the lustre filesystem