How are you moving the data through your HPC system?
NFS? Specialized protocols? MPI? Parallel filesystems self contained in the cluster? SMB? Written on paper slips like the nigglywiggly in kiss (or does it just feel that way?)?
We tend to not move data through anywhere. We are using Isilon for file storage so data tends to be ingested on some type of bench side workstation running Windows or OS X and saved to the file shares over SMB then the cluster accesses the data where it resides via NFS to crunch or process placing the results in the same storage then analysis is done via other Windows/Mac systems over SMB or Linux workstations over NFS.
So as I said the data doesn’t so much move as we bring the appropriate compute resources to the data to interact with it for each step.
Today we house the HPC storage on HP’s X9000 aka IBRIX. Approximately 16 NAS heads mounting the IBRIX filesystem that has 65 x 7.4TB EVA LUNs across 4x8400s & 1 P6500.
The HPC environment has infiniband connections on the Voltaire Director.
The HPC environment are all HP C-Class Blades that run RHEL5.5
Data flow from most of the researchers log onto their workstations and execute batch jobs on the HPC clusters (selective nodes).
NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!